EPA/600/R-14/307
February 2007
Environmental Technology
Verification Report
ENDETEC TECTA™ B-16 BY
PATHOGEN DETECTION SYSTEMS, INC.
Prepared by
Battelle
Business of Innovation
Under a cooperative agreement with
EPA
ET1/ET1/ET1/
-------
Environmental Technology Verification
Report
ETV Advanced Monitoring Systems Center
ENDETEC TECTA™ B-16 BY
PATHOGEN DETECTION SYSTEMS, INC.
by
Ryan James, Dan Lorch, Betsy Cutie, and Amy Dindal, Battelle
Doug Grosse, U.S. EPA
-------
Notice
The U.S. Environmental Protection Agency, through its Office of Research and Development,
funded and managed, or partially funded and collaborated in, the research described herein. It
has been subjected to the Agency's peer and administrative review and has been approved for
publication. Any opinions expressed in this report are those of the author (s) and do not
necessarily reflect the views of the Agency, therefore, no official endorsement should be inferred.
Any mention of trade names or commercial products does not constitute endorsement or
recommendation for use.
11
-------
Foreword
The EPA is charged by Congress with protecting the nation's air, water, and land resources.
Under a mandate of national environmental laws, the Agency strives to formulate and implement
actions leading to a compatible balance between human activities and the ability of natural
systems to support and nurture life. To meet this mandate, the EPA's Office of Research and
Development provides data and science support that can be used to solve environmental
problems and to build the scientific knowledge base needed to manage our ecological resources
wisely, to understand how pollutants affect our health, and to prevent or reduce environmental
risks.
The Environmental Technology Verification (ETV) Program has been established by the EPA to
verify the performance characteristics of innovative environmental technology across all media
and to report this objective information to permitters, buyers, and users of the technology, thus
substantially accelerating the entrance of new environmental technologies into the marketplace.
Verification organizations oversee and report verification activities based on testing and quality
assurance protocols developed with input from major stakeholders and customer groups
associated with the technology area. ETV consists of six environmental technology centers.
Information about each of these centers can be found on the Internet at http://www.epa. gov/etv/.
Effective verifications of monitoring technologies are needed to assess environmental quality
and to supply cost and performance data to select the most appropriate technology for that
assessment. Under a cooperative agreement, Battelle has received EPA funding to plan,
coordinate, and conduct such verification tests for "Advanced Monitoring Systems for Air,
Water, and Soil" and report the results to the community at large. Information concerning this
specific environmental technology area can be found on the Internet at
http://www.epa.gov/etv/centers/centerl.html.
in
-------
Acknowledgments
The authors wish to thank Jim Sinclair, Sandhya Parshionikar, Jennifer Best, Keya Sen, and
Mark Rodgers of the U.S. EPA, and Rick Sakaji of the East Bay Municipal Utility District for
their review of the test/quality assurance (QA) plan and Mark Rodgers and Stanley States of the
Texas A&M Engineering Extension Service for their review of this verification report. QA
oversight was provided by Holly Ferguson, U.S. EPA, and Betsy Cutie, Zachary Willenberg, and
Rosanna Buhl, Battelle.
IV
-------
Contents
Page
Foreword iii
Acknowledgments iv
List of Abbreviations vii
Chapter 1 Background 1
Chapter 2 Technology Description 2
Chapters Test Design and Procedures 4
3.1 Introduction 4
3.2 Test Overview 4
3.3 Experimental Design 5
3.3.1 Verification Test Sample Preparation 5
3.3.2 Sample Analysis 7
3.3.3 Detection of Additional Concentration Levels in Continuous Operating Mode 8
Chapter 4 Quality Assurance/Quality Control 9
4.1 Quality Control Samples 10
4.2 Audits 10
4.2.1 Technical Systems Audit 10
4.2.2 Data Quality Audit 11
Chapters Statistical Methods 12
5.1 False Positive Rates, False Negative Rates, Sensitivity, and Specificity 12
5.2 Method Comparability 12
Chapter 6 Test Results 14
6.1 TECTA B-16 Confirmed Results 14
6.2 Method Comparability 15
6.3 Detection of Additional Concentration Levels in Continuous Operating Mode 16
6.4 Operational Factors 16
Chapter 7 Performance Summary 19
Chapter 8 References 21
v
-------
APPENDIX
Appendix A: Raw Data from Reference Methods, TECTA B-16, and Confirmation Analyses
FIGURES
Figure 2-1. TECTA B-16 during analysis (left); Operator holding test cartridge next to
TECTA B-16 ready for insertion of sample cartridges (right)
Figure 2-2. Sample cartridge exhibiting positive (top) and negative (bottom) test results..
Figure 2-3. Red and green indicate results around sample chambers and on touch screen.
2
2
3
TABLES
Table 3-1. Methods, Equipment, and Results for the Characterization of the Drinking
Water Sample 6
Table 3-2. Quality Control Strains and Expected Results 7
Table 3-3. Replicate Samples by each Analysis Method 7
Table 6-1. TC and EC Positive Results 14
Table 6-2. TECTAB-16 TC and EC Data Summary - Positives 15
Table 6-3. TECTA B-16 TC and EC Data Summary - Negatives 15
Table 6-4. TC and EC Data Summary - Confirmations 15
Table 6-5. TC and EC Results 16
Table 6-6. Results of Analysis of Additional Concentrations in Continuous
Operation Mode of the TECTA B-16 17
Table 7-1. Results Summary for Positive TECTA B-16 Results for TC and EC 19
Table 7-2. Confirmed Result Summary of TECTAB-16 20
VI
-------
List of Abbreviations
ADQ Audit of Data Quality
AMS Advanced Monitoring Systems
ATCC American Type Culture Collection
CDW Columbus Division of Water
CFU colony forming unit
cm centimeters
COC chain of custody
DDW dechlorinated drinking water
DQA data quality audit
DW drinking water
EA Enterobacter aerogenes
EC Escherichia coli
EPA U.S Environmental Protection Agency
ETV Environmental Technology Verification
FN false negative
FP false positive
h hour(s)
L liter
MB method blank
MCL maximum contaminant level
MCLG maximum contaminant level goal
min minute(s)
mL milliliter
MUG 4-methyllumbelliferyl-p-D-glucorinide
N number
NA not applicable
NPDWR National Primary Drinking Water Regulation
NRMRL National Risk Management Research Laboratory
PA Pseudomonas aeruginosa
PDS Pathogen Detection Systems
QA quality assurance
QC quality control
QMP Quality Management Plan
RTCR revised Total Coliform Rule
TC total coliforms
TCR Total Coliform Rule
TN true negative
TP true positive
TQAP Test Quality Assurance Plan
TSA technical systems audit
USB Universal Serial Bus
vn
-------
Chapter 1
Background
The U.S. Environmental Protection Agency (EPA) supports the Environmental Technology
Verification (ETV) Program to facilitate the deployment of innovative environmental
technologies through performance verification and dissemination of information. The goal of the
ETV Program is to further environmental protection by accelerating the acceptance and use of
improved and cost-effective technologies. ETV seeks to achieve this goal by providing high-
quality, peer-reviewed data on technology performance to those involved in the design,
distribution, financing, permitting, purchase, and use of environmental technologies.
ETV works in partnership with recognized testing organizations; with stakeholder groups
consisting of buyers, vendor organizations, and permitting agencies; and with the full
participation of individual technology developers. The program evaluates the performance of
innovative technologies by developing test plans that are responsive to the needs of stakeholders,
conducting field or laboratory tests (as appropriate), collecting and analyzing data, and preparing
peer-reviewed reports. All evaluations are conducted in accordance with rigorous quality
assurance (QA) protocols to ensure that data of known and adequate quality are generated and
that the results are defensible.
The EPA's National Risk Management Research Laboratory (NRMRL) and its verification
organization partner, Battelle, operate the Advanced Monitoring Systems (AMS) Center under
ETV. The AMS Center recently evaluated the performance of the ENDETEC™ TECTA B-16, a
bench top incubator/analyzer/data logger system for the analysis of total coliforms (TC) and
Escherichia coli (EC) manufactured by Pathogen Detection Systems, Inc. (PDS), a subsidiary of
Veolia Water Solutions & Technologies.
-------
Chapter 2
Technology Description
This report provides results for the verification testing of the ENDETEC™ TECTA B-16 by
PDS (hereafter referred to as the TECTA™ B-16). The following is a description of the TECTA
B-16, based on information provided by the vendor.
The TECTA B-16 is a bench top detection and data logging system for the analysis of TC and
EC in water samples. It utilizes an enzyme substrate test to simultaneously detect the presence
of TC (p-galactosidase enzyme) and EC (p-glucuronidase enzyme). The system consists of
single-use cartridges that contain pre-measured reagents and an embedded optical sensor. A 100
mL water sample is added
to the cartridge and then up
to 16 of the cartridges are
incubated in and analyzed
by the TECTA B-16 which
is shown in Figure 2-1.
Figure 2-1. TECTA B-16 during analysis (left); Operator holding test
cartridge next to TECTA B-16 ready for insertion of sample cartridges
(right).
The enzymes produced by
TC and EC bacteria cleave
the fluorogenic substrates in
the growth media, resulting
in the release of fluorescent products. The fluorescent product
molecules rapidly accumulate in an optical sensor formed from a
polymeric material embedded at the center of the test cartridge base
(see Figure 2-2), which is continuously illuminated by an ultraviolet
light source in the bottom of each TECTA B-16 sample chamber. The
light emitted by the polymer when fluorescent indicator products are
present is detected at wavelengths specific to each fluorescent product
that are in turn specific to detection of TC and EC bacteria. Optical
detection is performed automatically by a charge-coupled device.
Test management software within the TECTA B-16 interprets these
optical signals continuously throughout the test cycle, and provides an
alert of a positive sample detection for both EC and TC as soon as a
threshold level of fluorescence is detected. Samples not displaying
detectable fluorescence at the wavelengths of interest are determined
Figure 2-2. Sample
cartridge exhibiting
positive (top) and
negative (bottom) test
results.
-------
I
^f=^~^ ^^^^ /" Six S~ \
OCXOQ
to be absent of bacteria after 18 hours. As shown in Figure 2-3, positive results are indicated by
a red light surrounding the symbol for
each sample chamber on the touch
screen (green indicates negative)
when the presence of TC or EC is
detected. The results are stored on the
TECTA B-16 and can be downloaded
with a universal serial bus (USB)
drive for viewing with the TECTA B-
16 software or any internet browser.
Due to its continuous monitoring
capability, positive sample results can
be detected in less than 18 hours.
Figure 2-3. Red and green indicate results around sample
chambers and on touch screen.
In continuous mode, the TECTA B-16
can analyze up to 16 samples in 18
hours (h). PDS provided three units
for testing, providing simultaneous sample analysis capacity of 48 samples. The TECTA B-16
can also be operated in read mode where the detector in the sample chamber can be used to make
an instantaneous measurement of light emission from the polymer. Read mode can be used to
confirm results obtained in continuous mode or if a sample cartridge has been incubated under
appropriate conditions outside of the TECTA B-16. In read mode, the results display on the
screen in the same way as for the continuous measurements, except that a "time-to-result"
indication is not available for samples processed in read mode.
The TECTA B-16 has dimensions of 48 cm wide x 62 cm deep x 34 cm high (18.8 inches wide
x 24.5 inches deep x 13.5 inches high) and weighs approximately 28 kilograms (61.7 pounds).
The TECTA B-16 is completely self contained and does not require any additional equipment or
materials to perform analyses.
-------
Chapter 3
Test Design and Procedures
3.1 Introduction
The ETV AMS Center Water Stakeholder Committee identified the use of coliform detection
technologies for the monitoring of drinking water (DW) as an area of interest for technology
verification. Fecal pollution can introduce disease-causing (pathogenic) bacteria, viruses, and
parasites into receiving waters, which may serve as private/public DW supplies. Utilities fully
recognize the possibility of this waterborne pollution and take every precaution (filtering,
treatment with disinfectants such as chlorine and chloramines, and regulatory compliance
sampling and analysis) to avoid fecal contamination in DW. Assessment of this health risk is
based on the detection and enumeration of fecal indicator bacteria, such as TC and EC, where its
presence indicates a potential pathway for contamination (e.g., sewage or animal waste) of the
distribution system which is designed to provide a physical barrier to contamination of DW. To
evaluate the ease of use of the TECTA B-16, as well as the applicability of the instrument to
non-microbiologist users under field / in-plant conditions, the testing was conducted by a
Columbus, OH water utility staff member who did not have any prior training or certification in
microbiological analyses.
In February 2013, EPA revised the 1989 Total Coliform Rule (TCR)1, a national primary
drinking water regulation (NPDWR). The revised rule establishes a maximum contaminant level
goal (MCLG) of zero for E. coli, a more specific indicator of fecal contamination and potential
harmful pathogens than total coliform. EPA has removed the 1989 MCLG and maximum
contaminant level for total coliform. In the revised TCR, total coliforms serve as an indicator of
a potential pathway of contamination into the distribution system. A PWS that exceeds a
specified frequency of total coliform occurrence must conduct an assessment to determine if any
sanitary defects exist and, if found, correct them.
3.2 Test Overview
This verification test was conducted according to procedures specified in the Test/QA Plan for
Verification of Coliform Detection Technologies for Drinking Water2 (TQAP) and adhered to
the quality system defined in the ETV AMS Center Quality Management Plan (QMP)3.
In order to comply with the revised TCR (RTCR), water utilities need coliform detection
technologies that are able to detect EC at concentrations of one colony forming unit (CFU) per
100 milliliters (mL). While it is difficult to determine if a single target organism is present in
-------
100 mL of water, when approximately half of the analyzed replicates are positive and half are
negative, the density of the organism has become adequately low so that a positive result can be
considered a single organism detection. Therefore, for the purpose of this verification test, the
objective was to prepare spiked DW dilution sets that provided 50 ± 25% positive results for EC
using the Colilert-18 reference method. However, because of the very small number of
organisms in the suspension, the mixture is not homogeneous, and therefore, representative
sampling cannot be done reliably from a bulk solution. The implication is that replicate 100 mL
samples cannot be compared. That is, the ratio of positive and negative results in 20 samples
analyzed by the TECTA B-16 and 20 samples analyzed by Colilert-18 cannot be compared
directly to confirm the performance against an accepted reference method. In order to confirm
presence or absence in each sample, the positive and negative spent media from each replicate
analyzed by the TECTA B-16 was inoculated into sterilized water for analysis by Colilert-18.
The Colilert-18 results confirmed the presence or absence of TC and EC for the samples that had
been analyzed by the TECTA B-16.
The overall ETV test of the TECTA B-16 was conducted from August 21-23, 2013 at the City of
Columbus Division of Water (CDW) laboratory in Columbus, Ohio with the reference method
analyses being performed at Superior Laboratories in Galloway, Ohio (which is a 15 minute
drive from the CDW laboratory). Technology operation and sample handling and analysis were
performed according to the operating documentation and method description provided by the
vendor. Both reference method and TECTA B-16 sample analysis results were reported in
presence/absence format, consistent with the requirements of the RTCR.
Sample analysis results from the TECTA B-16 were evaluated by calculating the true positive
(and true negative) results through confirmation analyses as described above. These calculations
include the comparison of false positive rate (or specificity) and false negative rate (or
sensitivity). In addition, statistical testing was performed on the initial reference method and
TECTA B-16 results. Sustainable operational factors such as ease of use, required reagents,
analysis time, and laboratory space and utilities required are reported.
3.3 Experimental Design
3.3.1 Verification Test Sample Preparation
The preparation of verification test samples included the collection of the DW sample, the
inoculation of the DW sample with target organisms, and the dilution of samples for analysis. A
detailed description of the sample preparation steps is provided in the TQAP .
3.3.1.1 Drinking Water Sample Collection
A single DW sample was collected from the tap at a Battelle laboratory as directed in the TQAP.
The DW sample was collected by first removing the faucet screen and decontaminating the
surface with 70% isopropanol. Next the line was purged for 5 minutes with cold water and 18
liters (L) of DW were collected from the tap into a sterile (autoclaved) carboy equipped with a
spigot. Once collected, an aliquot (several hundred milliliters) was collected from the carboy
-------
and used to measure the pH, free chlorine, and total chlorine. The analysis methods and
subsequent results are provided in Table 3-1.
Table 3-1. Methods, Equipment, and Results for the Characterization of the
Drinking Water (DW) Sample
Parameter
PH
free chlorine
total chlorine
Equipment/Media
calibrated pH meter
HACH Chlorine test kit
HACK Chlorine test kit
SOP/Method
SOPGEN.V-003-104
HACH Method 8021
HACH Method 8 167
DW Results
7.24
1.51mg/L
1.54mg/L
3.3.1.2 Preparation of Samples for Verification Testing
To test the coliform technologies, separate DW samples of EC containing concentrations of
approximately 1 organism per 100 mL were prepared. To ensure that these concentrations would
be attained, a range of concentrations were prepared. Two separate aliquots, approximately 5 L
each, of dechlorinated DW (DDW) were added to carboys (sterilized by autoclaving) containing
stir bars and spiked with a suspension of EC (American Type Culture Collection [ATCC] 25922)
to generate target suspensions of 0.5 CFU/100 mL and 5 CFU/100 mL. Each dilution was mixed
on a stir plate for 5 to 10 minutes, and then 100 mL aliquots were dispensed into sterile 100 mL
bottles using 50 mL and/or 100 mL graduated pipettes (sterile, individually wrapped, and
disposable). Twenty replicate samples were prepared at each concentration level. Each bottle
was labeled with a unique sample identification number. Once all forty 100 mL aliquots were
dispensed for technology verification, they were immediately transported on ice to CDW where
verification testing was conducted upon receipt. All laboratory work was performed within
certified Class II biological safety cabinets by analysts wearing disposable laboratory coats and
non-latex gloves to minimize the potential for inadvertent contamination. Polypropylene sample
bottles used were either sterilized via autoclaving or purchased sterile.
In addition to the samples to be used for verification, a second set of 100 mL aliquots were
prepared in the same manner as described above (forty samples in total; twenty from each
carboy) for the reference method analysis. Immediately after being dispensed and labeled with
unique sample identification numbers, all reference samples were transported by car in coolers
packed with ice packs to Superior Laboratories, Inc.
Quality control samples (listed in Table 3-2) were also prepared. Positive and negative ATCC
control cultures were purchased from MicroBioLogics. Control organisms included the EC
negative control Enterobacter aerogenes (EA) (ATCC 13048), EC (ATCC 25922), and the non-
coliform Pseudomonas aeruginosa (PA) (ATCC 10145). All control cultures were prepared on
tryptic soy agar and incubated overnight. The QC samples were then prepared by inoculating
triplicate 990 mL filter sterilized DDW aliquots each with 10 mL of a 100 colony forming unit
(CFU)/mL suspension prepared from the agar cultures in DDW. Control samples were used to
confirm the accurate response (positive response for positive control and negative response for
the negative controls) of the TECTA B-16 and reference methods at relatively high
concentrations. The QC samples were shipped with the test samples that went to both
laboratories. This resulted in 48 samples (Table 3-3) prepared and shipped to each laboratory.
-------
Sample custody for all samples transferred to CDW and Superior Laboratories were documented
using a chain-of-custody (COC) form following Battelle SOP ENVS-6-055 for Chain of
Custody5. The COC form was signed once receipt of all samples was confirmed. Reference
method analysis was initiated using Colilert-18 on the same day as arrival at the laboratory,
within 2 h of initiation of the TECTA B-16 sample analysis.
3.3.2 Sample Analysis
The ability of the TECTA B-16 to determine the presence of EC was challenged using 20
replicates of the two concentrations of EC in DW samples. Positive/negative control samples
spiked with quality control (QC) cultures listed in Table 3-2 as well as method blank samples
were included during testing. PDS provided three TECTA B-16s to perform testing of the
replicate samples shown in Table 3-3. Each of the TECTA B-16s contained 16 sample chambers
for incubating and measuring the fluorescence from the sample cartridges. Therefore, all 48
samples in the primary technology evaluation were analyzed simultaneously in continuous
detection mode. In continuous mode, the sample cartridges were inserted into the TECTA B-16s
at the start of the incubation and remained in the unit for the full 18 h analysis period. All of the
samples were assayed by the Colilert-18 reference method and the TECTA B-16 concurrently.
Table 3-2. Quality Control Strains and Expected Results
Targeted Coliform
Sterilized Water
Pseudomonas aeruginosa (PA)ATCC
10145
Enterobacter aerogenes (EA) ATCC
13048
Escherichia coli ATCC 25922
Description
Method Blank
TC and EC negative control
TC positive control
EC negative control
TC positive control
EC positive control
Expected Result
TC- and EC-
TC- and EC-
TC+ and EC-
TC+ and EC+
Table 3-3. Replicate Samples by each Analysis Method
Sample Description
Dilution A
Target conc.= 5 CPU/ 100 mL
Dilution B
Target conc.= 0.5 CPU/100 mL
Method Blank (sterilized water)
EC Positive control
Target conc.=100 CPU/100 mL
TC Positive control
Target conc.= 100 CPU/100 mL
Negative control
Target conc.= 100 CPU/100 mL
Total Replicate Analyses
Replicate Analyses by
TECTA B-16
20
20
3
3
1
1
48
Replicate Analyses by
Colilert-18
20
20
3
3
1
1
48
-------
3.3.2.1 Confirmation of Results
As described in Section 3.2, each TECTA B-16 result was confirmed as definitively positive or
negative with the Colilert-18 reference method to verify the result obtained TECTA B-16. In
summary, 1 mL of each 100 mL sample resulting from the 18 h incubation during TECTA B-16
analysis was inoculated into 99 mL of filter sterilized water, dechlorinated tap water and
analyzed using Colilert-18. The Colilert-18 result provided definitive confirmation of presence
or absence of EC in the initial samples.
3.3.3 Detection of Additional Concentration Levels in Continuous Operating Mode
Another component of the ETV test was performed to verify the capability of the TECTA B-16
to detect EC ATCC 25922 at various concentration levels in continuous operating mode which
provides positive results as soon as determined by the TECTA B-16. A target inoculation was
prepared in DDW that contained approximately 104 EC per 100 mL, and then a serial dilution
(1:10, 1:100, and 1:1,000) of the stock was prepared to obtain four separate samples for testing
(10, 100, 1,000 and 10,000 EC per 100 mL). The data from these tests were intended to identify:
(1) whether or not the TECTA B-16 detected the presence of EC at higher concentrations and (2)
to evaluate the time required for detection of positive results. Triplicate aliquots at each
concentration level were analyzed using a quantitative spread plate enumeration method for EC
to confirm the concentration of the samples.
-------
Chapter 4
Quality Assurance/Quality Control
QA/QC procedures were performed in accordance with the TQAP for this verification test1 and
the QMP for the AMS Center2. QA/QC procedures and results are described in the following
sections.
During testing, there were five minor deviations from the TQAP. These included:
• Adjusting the number of QA samples due to the limited number of sample chambers (48)
within the TECTA B-16 instrument package provided for the trial on-site. The number
of QA samples was adjusted so the total number of test and QA samples combined did
not exceed 48. The original plan called for two of each QA sample. The number was
adjusted to three method blanks and three EC positive controls (therefore there was one
sample per TECTA B-16 unit) and one EC negative control and one EC positive control
overall. Analysis of the planned number of QA samples would have been preferable.
However, inclusion of triplicate samples of the EC positive control provided positive
control results for both TC and EC. Also, the EC negative control and the negative
control had densities of greater than 100 CFU/lOOmL, a rather high concentration of
possible interfering organisms that would be more likely to cause a false positive than a
lower density. Lastly, confirmation analysis also adds certainty to the correct result of
the control samples.
• Changing the SM 9226 (nutrient Agar-4-methyllumbelliferyl- p-D-glucorinide [MUG])
method to the standard spread plate enumeration method because EC source was a pure
culture and therefore the differential enumeration technique was not required.
• Changing target suspensions of EC to 0.5 CFU/100 mL and 5 CFU/100 mL from
1 CFU/100 mL and 10 CFU/mL based on preliminary results from use of the methods
planned to attain the target presence/absence ratios.
• Using Pseudomonas aeroginosa ATCC 27853 as the negative control rather than ATCC
10145 because of supplier availability.
• Correcting the phone number of Superior Laboratories.
Each of these deviations was judged by the Battelle Verification Test Coordinator to not result in
any adverse impacts on the quality of the data generated. The deviation was reviewed and
approved by the EPA ETV AMS Center Project Officer and EPA ETV AMS Center Quality
Manager.
-------
4.1 Quality Control Samples
The Colilert-18 reference method required the inclusion of method blanks (MBs) and positive
and negative control organisms. Three MB samples were taken across the 48 total test samples.
The MB samples consisted of 100-mL dechlorinated, sterilized tap water processed as a sample.
MB samples were exposed to identical handling and analysis procedures as the rest of the test
samples. These samples were used to ensure that no sources of contamination were introduced
in the sample handling and analysis procedures. All three MB samples analyzed by the TECTA
B-16 as well as the reference method were negative, indicating the absence of TC and EC.
Positive and negative control samples were also analyzed using each method. The control
cultures were enumerated by membrane filtration. The EC positive control was 116 CPU
EC/100 mL (three replicates), the EC negative control was 146 CPU EA/100 mL (one replicate),
and the negative control was 276 CPU PA/100 mL (one replicate).
The EC negative control was determined to be negative (TC+EC-) using the reference method
and the TECTA B-16 (and confirmed with the Colilert-18 confirmation analysis). In addition, all
three EC positive controls were determined to be positive (TC+EC+) using the reference method
and the TECTA B-16 (and confirmed with the Colilert-18 confirmation analysis). The non-
coliform negative control resulted in only negative results which were also confirmed.
4.2 Audits
Two types of audits were performed during the verification test; a technical systems audit (TSA)
of the verification test procedures, and a data quality audit (DQA). Audit procedures for the
TSA and the DQA are described further below.
4.2.1 Technical Systems A udit
The Battelle AMS Center Quality Auditor performed a TSA on August 21, 2013 at CDW's water
quality laboratory in Columbus, OH and at the reference laboratory, Superior Laboratories
located in Galloway, OH. The TSA consisted of interviews with Battelle and Superior
Laboratories personnel, observations of test sample preparation and testing at Battelle and
Superior Laboratories, and observation of sample analysis. The purpose of the audit was to
verify that:
• Sample preparation procedures were performed by Battelle according to the TQAP
requirements
• Reference laboratory methods for analyzing test samples conformed to the TQAP and
reference method requirements
• Technology testing was performed according to the TQAP and vendor instructions
• Test documentation provided a complete and traceable record of sample preparation and
analysis
10
-------
• Equipment used in the test was calibrated and monitored according to TQAP
requirements and standard laboratory procedures.
The ISA revealed no findings and just one observation. The observation involved an
inconsistency with the reference method procedure being followed by the reference laboratory
and a description of the reference method included in the reference method given in the appendix
of the TQAP. The method summary provided in the appendix gave an incubation time range of
24 to 28 h rather than the correct range of 18 to 22 h. As the reference lab followed the proper
procedure, there was no negative impact to the test results. No action was required.
A TSA report was prepared and distributed to the Verification Test Coordinator, the Battelle
AMS Center Manager, the EPA AMS Center Project Officer, and the EPA AMS Center Quality
Manager.
4.2.2 Data Quality Audit
Records generated in the verification test were reviewed by a second verification staff member
before these records were used to calculate, evaluate, or report verification results. The person
performing the review added his/her initials and the date to a hard copy of the record being
reviewed. In addition, an audit of data quality (ADQ) was conducted on October 17-18, 2013.
During the audit, laboratory data generated at the reference laboratory, Superior Laboratories,
Inc., and data generated by the TECTA B-16 were reviewed and verified for completeness,
accuracy and traceability. The verification of coliform detection technologies was determined by
the EPA AMS Center Project Officer to be Category III test. Accordingly, at least 25% of the
results for each of the testing scenarios were verified versus the raw data, and 100% of the QC
sample results were verified. The data were traced from the initial acquisition, through reduction
and statistical analysis, to final reporting to ensure the integrity of the reported results. All
calculations performed on the data undergoing the audit were checked.
The ADQ revealed no findings and two observations. The first observation was that the TQAP
specified grouping results to determine proportions of results that are positive, negative, false
positive, and false negative, then running chi-square tests for determination of significance.
Instead, the chi-squared test was modeled in SAS using the FREQ procedure for determining
significance (without presenting the specified groupings). While a completely appropriate
approach to completing a chi-squared test, a deviation will be completed to document this slight
change in data treatment. The second observation is that all project entries could be linked to
personnel, but there was no signature page to link initials and signature to a specific individual.
A signature page will be added to the data binder to link the initials to individual performing the
work. None of these had an adverse impact on the test results.
A data audit report was prepared, and a copy was distributed to the EPA AMS Center Quality
Manager.
11
-------
Chapter 5
Statistical Methods
The statistical methods used to evaluate the quantitative performance factors are presented in this
chapter. Qualitative observations were also used to evaluate verification test data.
5.1 False Positive Rates, False Negative Rates, Sensitivity, and Specificity
False positive (FP) and false negative (FN) rates of the TECTA B-16 were evaluated when
assessing comparability. During this test, true positives (TPs) were those positive results from
the TECTA B-16 that were confirmed as positive by the reference method, and false positives
were those positive results from the TECTA B-16 that were not confirmed by the reference
method. Conversely, true negative (TN) results were those negative results that were confirmed
as negative, and false negative results were those negative results that were shown to be positive
by the confirmatory method.
Sensitivity is defined as the percent of positive samples correctly identified as positive and
specificity is defined as the percent of negative samples correctly identified as negative.
Estimates of sensitivity, specificity, false positive rates, and false negative rates as percentages
for the two methods were calculated as follows:
TP
Sensitivity = x 100%
J TP+FN
TN
Specificity = x 100%
F J TN+FP
FP f TN \
False positive rate = x 100% = (1 - TN+Fp) x 100% = 1 - Specificity
FP
TN+FP
FN
False negative rate = x 100% = (1 - ^f^) x 100% = 1 - Sensitivity
5.2 Method Comparability
In order to assess whether the proportion of positive and negative samples was significantly
different between the TECTA B-16 and the reference method, chi-square tests for independence
were conducted. The chi-squared test was modeled in SAS® (ver. 9.1.3), using the FREQ
12
-------
procedure. If the calculated chi-square value is less than the critical value, the sample results
between the two methods are not significantly different (95% confidence, alpha = 0.05, p-value >
0.05). If the Chi-square value is greater than the critical value (based on a significance of 0.05),
the results between the two methods are significantly different, and it is concluded that there is a
difference between the two methods.
Prior to testing, a power analysis was conducted to determine the number of replicates required
to determine possible significant differences between the technologies being tested and the
reference method. Conducted using the POWER procedure in SAS, the power analysis
determined the number of replicate tests (across both test types) that would be necessary to
detect a specified difference in proportions of a specified size with 80% power, given a specified
value of the proportion for the reference test (the acceptable range of reference test positive
proportions was 25% to 75% for this test), and a significance level of 0.05 for the test. To
summarize, the power analysis shows that for approximately 20 replicates, if the reference
method was approximately 50% positive (10 positive results and 10 negative results), then the
technology being tested would be required to be 90% positive (18 positives and two negative
results) or 90% negative (two positives and 18 negatives) to have a significant difference. The
TECTA B-16 results are discussed in the context of this power analysis.
In summary, the smallest difference that is able to be determined with 20 replicates is
approximately a 30% to 40% change in positive results. The power analysis revealed that
differences of 5% or 10% of positive results could be determined, but between 150 and 1,250
replicates may be required.
13
-------
Chapter 6
Test Results
This verification test included both quantitative and qualitative evaluations. The quantitative
evaluation was conducted to confirm the presence/absence TECTA B-16 results with those
generated by the presence/absence result from the reference method in the same replicate sample
as well as in parallel sample sets. The qualitative evaluation was performed to document the
operational aspects of the TECTA B-16 when it was used during verification testing. The
following sections provide the results of the quantitative and qualitative evaluations. The TC
and EC results are presented together in this section because (1) a pure EC culture was used as
the source of EC for this test (rather than sewage), (2) positive TC results under the RTCR no
longer trigger an acute MCL violation (but can trigger the requirement for sanitary surveys), and
(3) the results for TC and EC were the same in each sample. Tables presenting the raw data
presence/absence results for the TECTA B-16, the Colilert-18 initial reference method result, and
the Colilert-18 confirmation analysis results are provided in Appendix A.
6.1 TECTA B-16 Confirmed Results
The positive TC and EC test results for the TECTA B-16 and reference method (Colilert-18) are
presented in Table 6-1. One of the two dilutions (0.5 CFU/100 mL) yielded the target 50 ± 25%
split in responses from the reference method. The other dilution generated results that were
100% positive.
Table 6-1. TC and EC Positive Results
Dilution
(target concentration)
A
(5CFU/100mL)
B
(0.5 CFU/100 mL)
TECTA B-16
TC and EC
N
20
6
% of total
samples
100%
30%
Colilert-18
N
20
11
% of total
samples
100%
55%
N - Number of replicates
Because the reference method results were between 25% and 75% positive, the reference method
results suggested that the 0.5 CFU/100 mL solutions prepared for the evaluation were at the
14
-------
single organism per 100 mL concentration level. Therefore the TECTA B-16 results were able
to be used along with the reference method confirmation data to determine the effectiveness of
the TECTA B-16 in detecting such low concentrations. Specifically, following analysis using
the TECTA B-16, 1 mL of the resulting suspension was inoculated into 99 mL of sterilized water
and analyzed by the reference method. The result of these analyses provided confirmation of the
presence/absence results for each replicate sample. Tables 6-2 and 6-3 summarize the confirmed
true positive and true negative TC and EC results for the TECTA B-16.
Table 6-2. TECTA B-16 TC and EC Data Summary - Positives
Dilution
(target concentration)
A
(5CFU/100mL)
B
(0.5CFU/100mL)
N
20
6
Confirmed
(True Positive)
20
6
Difference
(False Positive)
0
0
Table 6-3. TECTA B-16 TC and EC Data Summary - Negatives
Dilution
(target concentration)
A
(5CFU/100mL)
B
(0.5CFU/100mL)
N
0
14
Confirmed (True
Negative)
0
14
Difference
(False Negative)
0
0
The sensitivity, specificity, false-positive, and false-negative rates for the TECTA B-16 results
for both the 0.5 CFU/100 mL and 5 CFU/100 mL dilutions were determined as described in
Section 5.1 and are presented in Table 6-4.
Table 6-4. TC and EC Data Summary - Confirmations
Incubation Time (h)
Sensitivity
Specificity
False Positive
False Negative
0.5 CFU/100 mL
100%
100%
0%
0%
5 CFU/100 mL
100%
NA
NA
0%
NA - not applicable because zero in denominator of calculation
6.2 Method Comparability
Table 6-5 shows the results from the chi-square test for independence that was performed to
compare the TC results from the TECTA B-16 for each incubation time period against the initial
results (not the confirmation results) of the reference method (Colilert-18). Because only the
15
-------
Table 6-5. TC and EC Results
Dilution
(target
concentration)
A
(5CFU/100mL)
B
(0.5 CPU/ 100 mL)
TECTA B-
16
+
20
6
_
0
14
Colilert-18
+
20
11
_
0
9
Chi-
Square
Degrees
of
freedom
p-Value
Critical Limits
(p=0.05)
NA
2.558
1
0.110
3.841
0.5 CFU/100 mL dilution had both positive and negative results, the chi-squared analysis was
only performed for that solution. This analysis generated a p-value that was greater than 0.05
indicating that the TECTA B-16 results were not significantly different from the initial Colilert-
18 results (at the 95% confidence interval), a result that is consistent with the confirmatory
analyses described above (which indicated identical results between the TECTA B-16 and
Colilert-18 confirmatory analysis of each TECTA B-16 replicate).
These results are consistent with the power analysis performed before testing and described in
Section 5.2. For TC and EC, the reference method generated 55% positive results for Dilution B.
When referencing the power analysis when the reference method was 50% positive, significant
differences could only occur with TECTA B-16 results of one or two positive results and the rest
negative results or one or two negative results with the rest positive results. The TECTA B-16
result of six positive and 14 negative samples did not meet that requirement. Based on the power
analysis, a significant difference perhaps could have been determined with an additional 70 or 80
replicates. However, because of the small concentrations involved, confirmation analysis on
each replicate will always be the best route of determining the technology performance.
6.3 Detection of Additional Concentration Levels in Continuous Operating Mode
The objective of this component of the testing was to verify the TECTA B-16 capability of
reporting analysis results as soon as determined by the TECTA B-16 rather than waiting for the
end of an 18 h incubation time period. Table 6-6 gives the results for the analysis of various
concentrations of EC ATCC 25922, including the result provided and the time of result. Without
exception, the TECTA B-16 generated positive TC and EC responses at all four concentration
levels. Four replicate samples of each concentration were analyzed and the TC and EC positive
results were reported between 12-14 h for 8 EC CFU/100 mL, 11-13 h for 100 EC CFU/100 mL,
10-12 h for 1,000 EC CFU/100 mL, and 9-11 h for 8,600 EC CFU/100 mL. As there is no
requirement for the TECTA B-16 to run a complete 18-hour cycle, once a positive result has
been detected, the detection time represents the test endpoint and operator alerts are generated.
6.4 Operational Factors
Following testing, the CDW operator (a CDW staff member who was a part of the field water
sampling team) commented that the TECTA B-16 was a very user friendly technology. They
noted that the software interface was extremely straight forward and that the 2 hour training
session with PDS staff was more than adequate to train them on the operation. The TECTA
16
-------
Table 6-6. Results of Analysis of Additional Concentrations in Continuous
Operation Mode of the TECTA B-16.
EC Cone.
(CFU/100mL)a
8
100
1,000
8,600
TC
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Incubation Time
TC Detected
(h:min)
13:29
13:38
13:09
13:35
12:18
12:12
12:29
11:53
10:59
11:12
11:05
10:59
10:12
10:18
10:10
10:18
EC
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Incubation Time
EC Detected
(h:min)
12:56
12:56
12:39
13:04
12:00
11:42
11:52
11:25
10:29
10:41
10:37
10:22
9:38
9:44
9:38
9:38
"Calculated concentrations
X=Presence; O= Absence
min = minute
B-16 was set up by plugging into standard 110 volt power and powering up. For training
purposes, several samples were prepared at a target concentration of 100-200 CFU/100 mL (EA,
PA, and EC) and analyzed in triplicate along with an MB (all control samples were prepared and
handled by a trained microbiologist). The results were all as expected (EC samples were
TC+EC+, PA samples were TC-EC-, method blank samples were TC-EC-) except for two of the
EA samples (182 CFU/100 mL) which produced TC+EC+ results and not the expected TC+EC-
result. When the sample was immediately repeated in read mode, the EC result became negative.
This behavior was also observed in a repeated sample set of EA only (171 CFU/100 mL). PDS
assisted in the troubleshooting of the TECTA B-16 and after review of raw data (not available to
a standard user) PDS determined that the algorithm that monitors the absorbance of the
wavelength produced by the EC was incorrectly producing a EC+ result due to an electronic
signal artifact in the presence of high concentrations of EA. The testing was continued and the
result was not observed again in the EC negative control samples (146 CFU/100 mL EA). PDS
noted that a revision to the algorithm will be made to account for this observation.
Upon first use of a TECTA B-16, analysis of a validation cartridge by each sample chamber was
required (following first use, validation is recommended weekly). The validation cartridge does
17
-------
not contain any liquid, but when inserted into a sample chamber, fluoresces at the proper
wavelengths to check for the proper functioning of the TECTA B-16. Once each sample
chamber had been validated (each chamber took approximately 30 seconds) with the result
reported as "passed", the TECTA B-16 units were ready for the analysis of samples. To further
verify the function of the TECTA B-16, the same validation procedure was also successfully
completed after the completion of ETV testing.
As previously described, the TECTA B-16 was operated in continuous measurement mode for
the simultaneous measurement of TC and EC using PDS 100 mL sample cartridges containing
the required reagents. The samples were loaded into the sample chambers and the 18 h
incubation/analysis was started by closing the lid of the TECTA B-16. Once a sample was
added, the operator swirled the contents (being careful not create bubbles in the bottom of the
cartridge) and set the sample down while preparing the rest of the samples. Full dissolution
required approximately 5 minutes for each sample. The samples (16 at a time) were incubated
within the TECTA B-16 at 35°C and results were reported on the screen as soon as the TECTA
B-16 was able to make a conclusive positive determination of TC and/or EC based on the
fluorescence measurement. A positive result could have been reported at any point during the 18
h analysis, while a negative result would not occur until the end of the incubation time. During
the continuous measurement, a countdown timer appeared on the touch screen nearest the sample
chambers that were used for the analyses.
Following testing, the CDW operator also noted that the TECTA B-16 does not require staff
available after hours and on weekends to read the results of samples after an analysis set of up to
16 had been started during working hours (the incubation function is automatically shut down
after 18 hours and results are stored for later evaluation). In addition, according to PDS, the
TECTA B-16 can be connected to a network for e-mail or text message alerts upon positive
samples.
The result of each measurement was displayed on the screen and the operator recorded the result
on a sample data sheet. Each result could also be downloaded for review and viewed
individually on a computer containing the TECTA B-16 software or any standard web browser,
but the results from a group of samples could not be exported as a spreadsheet.
18
-------
Chapter 7
Performance Summary
In order to comply with the RTCR, water utilities need coliform detection technologies that are
able to detect EC at concentrations of one organism per 100 mL samples. This ETV test verified
the performance of the TECTA B-16 at that level of detection. While it is difficult to determine
if a single target organism is present in 100 mL of water, when approximately half of the
analyzed replicates are positive and half are negative, the density of the organism has become
adequately low so that a positive result can be considered single organism detection. Therefore,
for the purpose of this verification test, DW dilution sets (inoculated with a pure culture of EC)
were prepared to provide 50 ±25% positive results for TC and EC with the reference method.
These reference method results confirmed single organism detection. The results from each
replicate sample analyzed on the TECTA B-16 were then confirmed with the reference method
for definitive presence/absence determination. In addition, the initial results from the reference
method were compared through statistical testing with the TECTA B-16. The results of the
verification of the TECTA B-16 are summarized in Table 7-1.
Table 7-1. Results Summary for Positive TECTA B-16 Results for
TC and EC
Dilution
(target concentration)
A (5 CPU/ 100 mL)
B (0.5 CFU/100 mL)
TECTA TC and EC
N
20
6
% of total
samples
100%
30%
Colilert-18
N
20
11
% of total
samples
100%
55%
It should be noted that for Dilution B that the observed differences in positive detection rate
(30% for the TECTA B-16 and 55% for Colilert-18) are due to statistical differences in organism
content in the original samples (i.e., the non-uniform distribution of positive and negative
samples from taking 100 ml aliquots from the original sample source) rather than a difference in
sensitivity between the methods. Method sensitivity was determined by confirmation - see
Sensitivity., below.
Specificity, Sensitivity, FPrate, and FN rate. Table 7-2 summarizes the specificity, sensitivity,
FP rate, and FN rate for TC and EC for the TECTA B-16 results. Sensitivity is defined as the
percent of positive samples correctly identified as positive and specificity is defined as the
percent of negative samples correctly identified as negative.
19
-------
Table 7-2. Confirmed Result Summary of TECTA B-16
EC target concentration:
Sensitivity
Specificity
False Positive
False Negative
0.5 CFU/100 mL
100%
100%
0%
0%
5 CFU/100 mL
100%
NA
NA
0%
NA = undefined results because of zero in denominator
Comparability. In another approach of comparison between the TECTA B-16 and the reference
method, a chi-square test for independence was performed. Results from each dilution of EC
were tested separately. The chi-square value for the EC solution was less than the critical limit
in each case; therefore, for EC and TC, the chi-square test did not detect any differences between
the results of the TECTA B-16 and the reference method. In addition, the corresponding p-value
was greater than 0.05, indicating that the data did not show a statistically significant difference
between the two methods for the detection of EC or TC at the 95% confidence interval. These
results were consistent with the power analysis performed before testing and described in
Section 5.2.
Additional Concentrations in Continuous Operation. The objective of this component of the
testing was to verify the TECTA B-16 capability of reporting analysis results as soon as
determined by the TECTA B-16 rather than waiting for the end of an incubation time period such
as 18 or 24 h. Four concentrations of EC ATCC 25922 (8, 100, 1,000, and 8,600 CFU/100 mL)
were analyzed four times each. The TECTA B-16 generated positive TC and EC responses for
all of the samples. The required analysis time for TC ranged from 10 to 14 h and for EC ranged
from 9 to 13 h. The amount of time until detection for the TC and EC samples decreased with
each increasing concentration level and generally the EC took about 30 to 50 minutes less time
for detection.
Operational Factors. The TECTA B-16 was operated in continuous measurement mode for the
simultaneous measurement of TC and EC in up to 16 different samples. To initiate analysis, 100
mL of each individual water sample were dispensed into each cartridge and the cartridge was
snapped firmly shut, then swirled to dissolve the contents. The cartridges were loaded into the
TECTA B-16 in the same manner and the 18 h incubation/analysis was started by closing the lid.
The samples (16 at a time) were incubated within the TECTA B-16 at 35 °C and results were
reported on the screen (and available to the operator as electronic alerts) as soon as the TECTA
B-16 was able to make a conclusive positive determination of TC and/or EC based on the
fluorescence measurement. The result of each measurement was displayed on the screen and the
operator recorded the result on a sample data sheet. Each result could also be downloaded for
review and viewed on a computer containing the TECTA B-16 software or a standard web
browser. The CDW operator noted that the technology was very user friendly and eliminated the
need for a technician to be present outside of working hours to read the results.
20
-------
Chapter 8
References
1. Total Coliform Rule, United States Federal Register, 54 FR 27544-27568, June 29, 1989,
Vol. 54, No. 124
2. Test/QA Plan for Verification of Coliform Detection Technologies for Drinking Water,
Battelle, Version 2.0, July 16, 2013.
3. Quality Management Plan for the ETV Advanced Monitoring Systems Center, Version 7.
U.S. Environmental Technology Verification Program, Battelle, November 2008.
4. SOP GEN.V-003-10. Standard Operating Procedure for the Use of pH meters to Measure
pH. Battelle.
5. SOP ENVS-6-055, Sample Handling, Receipt, and Custody. Battelle, April 2013.
21
-------
Appendix A
Raw Data from Reference Methods, TECTA B-16,
and Confirmation Analyses
-------
Dilution
(calculated
concentration)
A
(5 CFU/lOOml)
Sample
No.
XX01
XX04
XX06
XX07
XX08
XX09
XX13
XX20
XX22
XX25
XX27
XX29
XX30
XX31
XX32
XX33
XX34
XX36
XX38
XX40
Ratio of Positive
Percent Positive
Endetec
TC
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
20 of 20
100%
EC
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
20 of 20
100%
Colilert-18
TC
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
20 of 20
100%
EC
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
20 of 20
100%
Confirmation of
Endetec via Colilert-18
TC
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
20 of 20
100%
EC
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
20 of 20
100%
A-l
-------
Dilution
(calculated
concentration)
B
(O.SCFU/lOOml)
Sample
No.
XX02
XX03
XXI 1
XXI 2
XX14
XX16
XX17
XX18
XX19
XX21
XX24
XX26
XX37
XX39
XX41
XX42
XX43
XX44
XX45
XX47
Ratio of Positive
Percent Positive
Endetec
TC
O
o
O
o
o
o
o
X
o
o
X
o
o
X
X
X
o
o
o
X
6 of 20
30%
EC
O
O
o
o
o
o
o
X
o
o
X
o
o
X
X
X
o
o
o
X
6 of 20
30%
Colilert-18
TC
O
O
o
X
X
X
o
X
X
o
X
X
X
o
o
X
X
o
X
o
11 of 20
55%
EC
O
O
o
X
X
X
o
X
X
o
X
X
X
o
o
X
X
o
X
o
11 of 20
55%
Confirmation of
Endetec via Colilert-18
TC
O
O
o
o
o
o
o
X
o
o
X
o
o
X
X
X
o
o
o
X
6 of 20
30%
EC
O
O
o
o
o
o
o
X
o
o
X
o
o
X
X
X
o
o
o
X
6 of 20
30%
X= Presence
O= Absence
A-2
------- |