September 2010
Environmental Technology
Verification Report
PATHOGEN DETECTION SYSTEMS, INC.
AUTOMATED MICROBIOLOGY PLATFORM
Prepared by
Battelle
Battelle
The Business of Innovation
Under a cooperative agreement with
U.S. Environmental Protection Agency
ET1/ET1/ET1/
-------
September 2010
Environmental Technology Verification
Report
ETV Advanced Monitoring Systems Center
PATHOGEN DETECTION SYSTEMS, INC
AUTOMATED MICROBIOLOGY PLATFORM
by
Ryan James, Dan Lorch, Stacy Pala, Amy Dindal, Battelle
John McKernan, U.S. EPA
-------
Notice
The U.S. Environmental Protection Agency, through its Office of Research and Development,
funded and managed, or partially funded and collaborated in, the research described herein. It
has been subjected to the Agency's peer and administrative review and has been approved for
publication. Any opinions expressed in this report are those of the author (s) and do not
necessarily reflect the views of the Agency, therefore, no official endorsement should be inferred.
Any mention of trade names or commercial products does not constitute endorsement or
recommendation for use.
11
-------
Foreword
The EPA is charged by Congress with protecting the nation's air, water, and land resources.
Under a mandate of national environmental laws, the Agency strives to formulate and implement
actions leading to a compatible balance between human activities and the ability of natural
systems to support and nurture life. To meet this mandate, the EPA's Office of Research and
Development provides data and science support that can be used to solve environmental
problems and to build the scientific knowledge base needed to manage our ecological resources
wisely, to understand how pollutants affect our health, and to prevent or reduce environmental
risks.
The Environmental Technology Verification (ETV) Program has been established by the EPA to
verify the performance characteristics of innovative environmental technology across all media
and to report this objective information to permitters, buyers, and users of the technology, thus
substantially accelerating the entrance of new environmental technologies into the marketplace.
Verification organizations oversee and report verification activities based on testing and quality
assurance protocols developed with input from major stakeholders and customer groups
associated with the technology area. ETV consists of six environmental technology centers.
Information about each of these centers can be found on the Internet at http://www.epa.gov/etv/.
Effective verifications of monitoring technologies are needed to assess environmental quality
and to supply cost and performance data to select the most appropriate technology for that
assessment. Under a cooperative agreement, Battelle has received EPA funding to plan,
coordinate, and conduct such verification tests for "Advanced Monitoring Systems for Air,
Water, and Soil" and report the results to the community at large. Information concerning this
specific environmental technology area can be found on the Internet at
http://www.epa.gov/etv/centers/centerl.html.
in
-------
Acknowledgments
The authors wish to thank Jim Sinclair, Sandhya Parshionikar, Jennifer Best, Keya Sen, and Mark
Rodgers of the U.S. EPA, and Rick Sakaji of the East Bay Municipal Water District, for their review
of the test/QA plan and/or this verification report. Quality assurance (QA) oversight was provided by
Michelle Henderson, U.S. EPA, and Rosanna Buhl, Battelle.
IV
-------
Contents
Page
Foreword iii
Acknowledgments iv
List of Abbreviations vii
Chapter 1 Background 1
Chapter 2 Technology Description 2
Chapters Test Design and Procedures 4
3.1 Introduction 4
3.2 Test Overview 4
3.3 Experimental Design 5
3.3.1 Verification Test Sample Preparation 5
3.3.2 Sample Analysis 7
3.3.3 Detection of Additional Concentration Levels in Continuous Operating Mode 8
Chapter 4 Quality Assurance/Quality Control 11
4.1 Quality Control Samples 11
4.2 Audits 12
4.2.1 Technical Systems Audit 12
4.2.2 Data Quality Audit 13
Chapters Statistical Methods 15
5.1 False Positive Rates, False Negative Rates, Sensitivity, and Specificity 15
5.2 Method Comparability 16
Chapter 6 Test Results 17
6.1 TCData 17
6.2 EC Data 18
6.3 Method Comparability 19
6.4 Detection of Additional Concentration Levels in Continuous Operating Mode 21
6.5 Operational Factors 22
Chapter 7 Performance Summary 24
Chapter 8 References 27
-------
Figures
Figure 2-1. PDS Polymer Partitioning Technology 2
Figure 2-2. Positive (top) and negative (bottom) test results 2
Figure 3-1. Flowchart describing confirmation analyses for both the the PDS AMP and
SM9221BandF 10
Tables
Table 3-1. Methods, Equipment, and Results for the Characterization of Sewage and Drinking
Water Samples 6
Table 3-2. Quality Control Strains 8
Table 3-3. Replicate Samples by each Analysis Method 8
Table 6-1. TC Positive Results 17
Table 6-2. TC Data Summary -Positives 18
Table 6-3. TC Data Summary -Negatives 18
Table 6-4. TC Data Summary - Confirmations 18
Table 6-5. EC Positives 18
Table 6-6. EC Summary -Positives 19
Table 6-7. EC Summary - Negatives 19
Table 6-8. EC Data Summary - Confirmations 19
Table 6-9. TCs-18h 20
Table 6-10. TCs-24h 20
Table 6-11. EC-18h 21
Table 6-12. EC-24h 21
Table 6-13. Results of Analysis of Additional Concentrations in Continuous Operation Mode 22
Table 7-1. Results Summary for Positive PDS AMP Results for TC and EC 24
Table 7-2. Results Summary of PDS AMP for 18 and 24 h Incubation Times 25
Appendix Raw Data from Reference Methods, PDS AMP, and Confirmation Analyses 28
VI
-------
List of Abbreviations
ADQ audit of data quality
AMP Automated Microbiology Platform
AMS Advanced Monitoring Systems
ATCC American Type Culture Collection
BGLB brilliant green lactose bile
cm centimeters
COC chain of custody
DDW dechlorinated drinking water
DTU desktop testing unit
DW drinking water
EC Escherichia coli
EPA U.S. Environmental Protection Agency
ETV Environmental Technology Verification
FN false negative
FP false positive
h hour(s)
in inch
LTB lauryl tryptose broth
MB method blank
min minute(s)
mL milliliter
MUG 4-methyllumbelliferyl-p-D-glucorinide
N number
n/a not applicable
NA nutrient agar
NRMRL National Risk Management Research Laboratory
org organisms
PDS Pathogen Detection Systems
ppm parts per million
QA quality assurance
QC quality control
QMP Quality Management Plan
SM Standard Methods
SOP Standard Operating Procedure
SSDW spiked, stressed drinking water
SWTP Southerly Wastewater Treatment Plant
TQAP Test Quality Assurance Plan
TC total coliform
TCR Total Coliform Rule
TN true negative
TP true positive
TSA technical systems audit
TSB trypticase soy broth
USB Universal Serial Bus
vn
-------
Chapter 1
Background
The U.S. Environmental Protection Agency (EPA) supports the Environmental Technology
Verification (ETV) Program to facilitate the deployment of innovative environmental
technologies through performance verification and dissemination of information. The goal of the
ETV Program is to further environmental protection by accelerating the acceptance and use of
improved and cost-effective technologies. ETV seeks to achieve this goal by providing high-
quality, peer-reviewed data on technology performance to those involved in the design,
distribution, financing, permitting, purchase, and use of environmental technologies.
ETV works in partnership with recognized testing organizations; with stakeholder groups
consisting of buyers, vendor organizations, and permitters; and with the full participation of
individual technology developers. The program evaluates the performance of innovative
technologies by developing test plans that are responsive to the needs of stakeholders,
conducting field or laboratory tests (as appropriate), collecting and analyzing data, and preparing
peer-reviewed reports. All evaluations are conducted in accordance with rigorous quality
assurance (QA) protocols to ensure that data of known and adequate quality are generated and
that the results are defensible.
The EPA's National Risk Management Research Laboratory (NRMRL) and its verification
organization partner, Battelle, operate the Advanced Monitoring Systems (AMS) Center under
ETV. The AMS Center recently evaluated the performance of the Automated Microbiology
Platform (AMP) by Pathogen Detection Systems, Inc. (PDS), a bench top incubator/analyzer/data
logger system for the analysis of total coliforms (TC) and Escherichia coll (EC).
-------
Chapter 2
Technology Description
The objective of the ETV AMS Center is to verify the performance characteristics of
environmental monitoring technologies for air, water, and soil. This report provides results for
the verification testing of the PDS Automated Microbiology Platform (hereafter referred to as
the PDS AMP). The following is a description of the PDS AMP, based on information
provided by the vendor.
The PDS AMP is a bench top
incubator/analyzer/data logger system for the
analysis of TC and EC. It utilizes an enzyme
substrate test to simultaneously detect the presence of
TC (p-galactosidase enzyme) and EC (P-
glucuronidase enzyme). The system consists of
single-use cartridges that contain pre-measured
reagents and a chemical-optical sensor. A 100 mL
water sample is added to the cartridge and then
incubated in and analyzed by the PDS Desktop
Testing Unit (DTU), which is the major hardware
component of the PDS AMP. The PDS DTU is
the blue unit shown in Figure 2-1.
Figure 2-1. PDS Desktop Testing Unit
The enzymes produced by TCs and EC cleave the
fluorogenic substrates in the growth media,
resulting in the release of fluorescent products.
The fluorescent product molecules rapidly
accumulate into a proprietary, polymer-based
optical sensor embedded in the test cartridge,
which is continuously illuminated by an ultraviolet
light source. The light emitted by the optical
sensor is detected at wavelengths specific to each
fluorescent product. Therefore, the presence of
TC and EC can be determined simultaneously by a
light detector containing a charge-coupled device.
Test management software within the computer accompanying the PDS AMP automatically
interprets these optical signals continuously throughout the test cycle and when the PDS AMP is
operated in continuous mode, a positive result is reported on the screen when the presence of TC
Figure 2-2. Positive (top) and negative
(bottom) test results
-------
or EC is detected, regardless of the amount of time that has passed The results are stored on the
computer provided with the PDS AMP and can be downloaded with a universal serial bus (USB)
drive for viewing with the PDS AMP software.
In continous mode, the PDS AMP can analyze two samples in 24 hours (h). PDS provided four
units for testing, limiting the sample capacity to eight samples per 24 h. The large number of
samples required for this verification test exceeded that capacity. Therefore, the PDS AMP was
used mostly in manual mode. In manual mode, following the addition of a water sample to the
cartridges, the cartridges were incubated in laboratory incubators for the specified time (18 and
24 h before being inserted into the PDS DTU for a 30-second fluorescent measurement. The
results were displayed on the screen in the same manner as for the continuous measurements.
The PDS DTU (not including corresponding desktop computer) has dimensions of 20
centimeters (cm) wide x 30 cm deep x 15 cm high (8 inches (in) wide x 16 in deep x 12 in high)
and weighs approximately 5 kilograms (11 pounds). The PDS DTU, computer, and all required
software costs approximately $10,000. Sample cartridges can be purchased for approximately
$10 per cartridge. The PDS DTU is completely self contained and does not require any
additional equipment or materials to perform analyses.
-------
Chapter 3
Test Design and Procedures
3.1 Introduction
The ETV AMS Center Water Stakeholder Committee identified the use of coliform detection
technologies for the monitoring of drinking water (DW) as an area of interest for technology
verification. Fecal pollution can introduce disease-causing (pathogenic) bacteria, viruses, and
parasites into receiving waters, which may serve as private/public DW supplies. Utilities fully
recognize the possibility of this waterborne pollution and take every precaution (filtering,
treatment with disinfectants such as chlorine and chloramines, and regulatory compliance
sampling and analysis) to avoid fecal contamination in DW. Based on the 1989 Total Coliform
Rule (TCR)1, assessment of this health risk is based on the detection and enumeration of fecal
indicator bacteria, such as TC and EC, whose presence indicates the presence of a potential
pathway for contamination (e.g., sewage or animal waste) of the distribution system which is
designed to provide a physical barrier to contamination of DW. It is important to note that this
verification test was not being conducted to provide data to be used to approve technologies for
use in meeting regulatory requirements for the detection of TC or EC as required by either the
1989 TCR or the anticipated revision to the TCR. It was conducted, based on feedback from
ETV AMS Center stakeholders, to provide a verification test that is similar in requirements to the
current TCR approval protocol2, such that technologies that are not already approved have an
opportunity to be tested under a similar set of test conditions.
3.2 Test Overview
This verification test was conducted according to procedures specified in the Test/QA Plan
(TQAP) for Verification of Coliform Detection Technologies for Drinking Water3 and adhered to
the quality system defined in the ETV AMS Center Quality Management Plan (QMP)
4
The TCR sets both goals and legal limits for the presence of TC and EC in DW. To summarize,
the TCR states that the objective is for zero TC organisms in DW samples and the rule (for large
water systems) is that no more than 5% of all DW samples collected by a utility can be positive.
In order to comply with the TCR, water utilities need coliform detection technologies that are
able to detect TC and EC at concentrations of one organism (org) per 100 milliliters (mL).
While it is difficult to determine if a single target organism is present in 100 mL of water, when
approximately half of the analyzed replicates are positive and half are negative, the density of the
organism has become adequately low so that a positive result can be considered single organism
detection. Therefore, for the purpose of this verification test, the objective was to prepare spiked
-------
DW dilution sets that provided 50 ±25% positive results for both TC and EC with the reference
method(s) and then compare the results from the reference method with the detection technology
being tested.
In this report, results from the PDS AMP were compared to the results obtained from the
reference method analyses which were presence/absence methods for TC and EC, specifically,
Standard Methods (SM)5 922IB (TC) and 922IF (EC). These methods utilize selective and/or
chromatogenic liquid growth media to detect TC and EC. The verification test of the PDS AMP
was conducted from July 19 through July 27, 2010 at Battelle in Columbus, Ohio with the
reference method analyses being performed at Superior Laboratories in Galloway, Ohio (a 20
minute drive from Battelle). Additional testing was performed on August 16-18, 2010 at these
facilities. Technology operation and sample handling and analysis were performed according to
the vendor's instructions. Both reference method and PDS AMP sample analysis results were
reported as presence/absence.
Sample analysis results from the PDS AMP were evaluated by comparing the proportion of
positive (and negative) results to the proportion of positive (and negative) results produced by
the reference methods, which includes the comparison of false positive rate (or specificity) and
false negative rate (or sensitivity). In addition, sustainable operational factors such as ease of
use, required reagents, analysis time, and laboratory space and utilities required are reported.
3.3 Experimental Design
3.3.1 Verification Test Sample Preparation
The preparation of verification test samples included the collection of raw sewage as the source
of the target organisms, collection of the DW sample, the fortification of the DW sample with
target organisms, and the chlorine stressing and dilution of samples for analysis. A detailed
description of the sample preparation steps is provided in the TQAP1. A summary of the sample
preparation activities and timeline is provided below.
3.3.1.1 Sewage and Drinking Water Sample Collection
A single raw sewage sample (approximately 1 liter [L]), was collected at 7 a.m. on July 19, 2010
at the Southerly Wastewater Treatment Plant (SWTP) in Columbus, Ohio. The sewage sample
was a 24 h composite sample collected automatically over a 24 h period (midnight July 18 -
midnight July 19). The SWTP automated system collects 50 to 100 mL aliquots at
approximately 5-minute intervals directly into a refrigerated carboy. The sewage sample was
collected from this carboy. The sampling approach was a deviation from the TQAP, which had
implied that the sample would be collected without compositing. Battelle does not believe that
there was an adverse impact to the results of the evaluation due to this deviation because the
coliform levels were adequate for the purposes of testing.
Upon sampling, the sewage sample was immediately stored on wet ice, and transported by
Battelle staff to Battelle laboratories. Upon receipt, the sewage sample was filtered through a
Whatman No. 2 filter (11 micron pore-size) under vacuum using a Buchner funnel to remove
-------
excess solids, shaken vigorously for 1 minute to ensure homogeneity, and then immediately
characterized for total culturable heterotrophic bacteria, TCs, and EC.
A single DW sample was collected from the tap at the Battelle laboratory the same day the
sewage sample was collected. The DW sample was collected by first removing the faucet screen
and decontaminating the surface with 70% isopropanol. Next, the line was purged for 5 minutes
with cold water and 67 L of DW was collected from the tap into multiple sterile (autoclaved)
carboys equipped with a spigot and containing large stir bars. Once collected, aliquots from each
carboy were pooled and then used to characterize the DW using the methods and standard
operating procedures provided in Table 3-1. Table 3-1 also gives the results of the initial
characterization of the sewage and DW samples.
Table 3-1. Methods, Equipment, and Results for the Characterization of Sewage and
Drinking Water Samples
Parameter
PH
temperature
free chlorine
total chlorine
total, culturable
heterotrophic
bacteria
TC
EC
Units
n/a
°C
mg/L
mg/L
org/100 mL
org/lOOmL
org/lOOmL
Equipment/Media
calibrated pH meter
calibrated thermometer
HACH Chlorine test kit
HACK Chlorine test kit
R2A agar
m-Endo
NA-MUG
SOP/Method
SOPGEN.V-003-106
SOPGEN.V-013-0477
HACH Method 8021
HACH Method 8 167
AOAC's
Bacteriological
Analytical Manual8
SM 9222B
SM 9222G
Sewage
n/a
n/a
n/a
n/a
8.5 xlO8
5.7 xlO6
8.0 xlO5
DW
6.9
23
0.80
0.80
n/a
n/a
n/a
n/a - not measured
NA - Nutrient agar
MUG4-methyllumbelliferyl-p-D-glucorinide
3.3.1.2 Chlorine Stressing and Preparation of Samples for Verification Testing
The PDS AMP was tested with chlorine stressed TC and EC. The chlorination stressing step was
started within 4 h from the time Battelle received the sample, or approximately 11 h from the
time the last automated sample was collected and 35 h from the time the first automated sample
was collected. This multi-step stressing process was accomplished on the same day as DW
sample collection by adding approximately 37 L of the unspiked DW sample to one 50 L carboy.
The DW was adjusted to a free chlorine concentration of 2 parts per million (ppm) using a 4%
hypochlorite solution, after which 10.5 L was dispensed into three 10 L aliquots containing stir
bars. Each aliquot was then spiked with TC and EC by adding 200 mL of filtered sewage
(amount of sewage providing enough TC and EC to bring the DW sample to a starting
concentration of approximately 105 TC org/100 mL and 104EC org/100 mL). The three aliquots
were chlorinated for 2.5, 5, and 10 minutes, respectively, after which time the samples were
dechlorinated with sodium thiosulfate and subsequently enumerated using SM9222 B and G.
The results determined the log reduction of TC and EC due to the chlorine stressing that had
occurred in each aliquot. This chlorine stressing step was considered adequate if the number of
organisms in the spiked DW samples was reduced by two to four orders of magnitude.
During the testing of the PDS AMP, the 5-minute chlorine stressing attained a two log reduction
in both TC and EC; therefore, after being refrigerated overnight, that aliquot of spiked, stressed
-------
drinking water (SSDW) was used to prepare the diluted samples for analysis. To test the
coliform technologies, separate SSDW samples of TC and EC containing concentrations of
approximately 1 organism per 100 mL were prepared. EC concentrations are typically 10 times
less than the TC concentrations. To ensure that these concentrations would be attained for both
TC and EC, a range of concentrations was prepared. Three separate aliquots, approximately 10
L each, of dechlorinated DW (DDW) were added to carboys and spiked with a calculated volume
of SSDW sample to generate target suspensions of 0.1 org/100 mL, 1 org/100 mL, and 10
org/100 mL. Each dilution was mixed on a stir plate for 5 to 10 minutes, and then 100 mL
aliquots were dispensed into sterile 100 mL bottles using 50 mL and/or 100 mL graduated
pipettes. Twenty replicate samples were prepared at each concentration level. Once all 100 mL
aliquots were dispensed for technology verification (20 at each dilution level for a total of 60
replicates per technology), verification testing was initiated. All samples were stored
refrigerated during the day of preparation until the analysis was initiated that same day.
In addition to the samples to be used for PDS verification, a set of twenty 100 mL aliquots were
prepared for the reference method analysis. Immediately after being dispensed, all reference
samples were transported by car in coolers packed with ice packs to Superior Laboratories, Inc.
Sample custody for all samples transferred to Superior Laboratories were documented using a
chain of custody (COC) form following Battelle Standard Operating Procedure (SOP) ENV-
ADM-009 for Chain of Custody9. The COC form was signed once receipt of all samples was
confirmed. Reference method analysis was initiated on the same day as arrival at the laboratory,
within 2 h of initiation of the PDS sample analysis.
3.3.2 Sample Analysis
The ability of the PDS AMP to determine the presence of TC and EC was challenged using 20
replicates of the three concentrations of SSDW samples. Positive/negative control samples
spiked with quality control (QC) cultures listed in Table 3-2 as well as method blank samples
were included during testing. PDS provided four DTUs to perform testing of the replicate
samples shown in Table 3-3. Each of the PDS DTUs contained two sample chambers for
incubating and measuring the fluorescence from the sample cartridges. However, as mentioned
in Chapter 2, because of the large number of concurrent sample analyses required during this
verification test, 66 out of the 72 samples were incubated apart from the PDS DTUs and then
inserted into the PDS DTUs at the end of the incubation periods (18 and 24 h) for fluorescent
measurement. Six samples were analyzed in continuous detection mode (i.e., they were inserted
into the PDS DTU at the start of the incubation and left there for the full 24 h analysis period).
All of the samples were assayed by the reference methods and the PDS AMP concurrently.
-------
Table 3-2. Quality Control Strains
Targeted Coliform
TC
EC
Method Blank
Sterilized DW
Sterilized DW
Positive Control
Enterobacter aerogenes
ATCC 13048
Escherichia coli
ATCC 8739
Escherichia coli
ATCC 8739
Negative Control
Pseudomonas
aeruginosa
ATCC 10145
Enterobacter aerogenes
ATCC 13048
Pseudomonas
aeruginosa
ATCC 10145
ATCC - American Type Culture Collection
Table 3-3. Replicate Samples by each Analysis Method
Sample Description
Dilution A - 10 org/100 mL
Dilution B - 1 org/100 mL
Dilution C - 0.1 org/100 mL
Method Blank
TC Positive control
EC Positive control
Negative control
Total Replicate Analyses
Replicate
Analyses by
PDS AMP
20
20
20
3
3
3
3
72
Replicate
Analyses by
SM9221B
20
20
20
3
3
3
3
72
Replicate
Analyses by
SM9221F
20
20
20
3
3
3
3
72
3.3.2.1 Confirmation of Results
All reference and PDS results were confirmed with more definitive tests to adequately verify the
PDS AMP. Confirmation for the SM 922IB and 922IF reference methods, as well as the PDS
AMP, is described in detail in the TQAP. In summary, for the PDS AMP analyses, 1 mL of each
100 mL sample resulting from the 24 h incubation during PDS analysis was inoculated into 9 mL
of lauryl tryptose broth (LTB) and analyzed using SM 9221 B and F. Following the LTB step,
TCs were confirmed using brilliant green lactose bile (BGLB) broth, and EC were confirmed
using EC-MUG. Figure 3-1 illustrates the process by which all positive and negative samples
from the PDS AMP and SM 922IB and 922IF were confirmed. As an additional, optional
confirmation, a complete test for TC was performed for three samples by inoculating
MacConkey media and then selecting suspected TC colonies and inoculating into LTB, as
described by SM9221B.
3.3.3 Detection of Additional Concentration Levels in Continuous Operating Mode
An optional component of the ETV test was performed to verify the capability of the PDS AMP
to detect EC ATCC 8739 at various concentration levels in continuous operating mode, which
-------
provides positive results as soon as determined by the PDS DTU. Four target inoculations were
prepared in DDW that contained approximately 104 EC per 100 mL, and then a serial dilution
(1:10, 1:100, and 1:1,000) of the stock was prepared to obtain four separate samples for testing
(10, 100, 1,000 and 10,000 EC per 100 mL). The data from these tests were intended to identify:
1) whether or not each technology detects the presence of EC and 2) the time required for
detection. Triplicate aliquots at each concentration level were analyzed using a quantitative
method for EC (SM 9222G - NA-MUG) to confirm the concentration of the samples.
-------
TC Dilution
50 ±25% )
Positive rNegative J
-—-— —- ^
LTB(+)
Replicates
(SM9221B)
v J
SM Reference
Method
Technology
Being Verified
Positive (+)
Replicates
CAN E, colt
Dilutions
Inocu
BGL
Results
Innru
ECM
Test for E. co//
lc9MUGl "* Ruore
Confirm (+) Results inoculate
False + Specificity ^ mocwate
Negative (-)
Replicates
/ii-
Confirm (-)Resulte moruiate
False - Soecificitv ~* moculate
Sample Confirmation
ate
B
TC
ate UV366
JG Fluorescence
/366 Record E. coli p ,
scence Reference Results
Inoculate Tr.
^ BGLB * U
_TB |—
Inoculate UVsee
EC MUG " Fluorescence
If +. Inoculate Tr
"* BGLB
TR
If +. Inoculate UV366
EC MUG * Fluorescence
E, colt
E. coti
Figure 3-1. Flowchart describing confirmation analyses for both the PDS AMP and SM9221B and F
10
-------
Chapter 4
Quality Assurance/Quality Control
QA/QC procedures were performed in accordance with the TQAP for this verification test1 and
the QMP for the AMS Center2. QA/QC procedures and results are described in the following
sub chapters.
During testing, there was one deviation from the TQAP (described in Section 3.3.1.1 involving a
change in collection method for the sewage sample). The TQAP had implied that the sewage
sample would be sampled directly and not composited over two days. This deviation was judged
by the Battelle Verification Test Coordinator to not result in any adverse impacts on the quality
of the data generated. The deviation was reviewed and approved by the EPA ETV AMS Center
Project Officer and EPA ETV AMS Center Quality Manager.
4.1 Quality Control Samples
The reference method required the use of method blanks (MBs), positive and negative control
organisms, and result confirmation. One MB was performed during the analysis for every 20
samples analyzed. The MB consisted of 100-mL dechlorinated, sterilized tap water processed as
a sample. MB samples were exposed to identical handling and analysis procedures as the rest of
the test samples, including the addition of all reagents. These samples were used to help ensure
that no sources of contamination were introduced in the sample handling and analysis
procedures. All three MB samples analyzed by the PDS AMP as well as the reference method
were negative, indicating the absence of TC and EC.
Three positive and negative control samples were also analyzed using each method. Positive and
negative ATCC control cultures were purchased from MicroBioLogics. Control organisms
included the TC Enterobacter aerogenes (ATCC 13048), EC (ATCC 8739), and the non-
coliform Pseudomonas aeruginosa (ATCC 10145). All control cultures were prepared onto
tryptic soy agar and incubated overnight. The QC samples were then prepared by inoculating
triplicate 100 mL-filter, sterilized DDW aliquots each with 1 mL of a slightly turbid culture
suspension prepared from the agar cultures in DDW. Control samples were used to confirm the
accurate response (positive response for positive control and negative response for the negative
controls) of the PDS AMP and reference methods at relatively high concentrations. The control
cultures were not enumerated, but were estimated to be approximately 105 to 106 org/100 mL
based on the degree of turbidity observed in the sample.
11
-------
All three TC positive controls were determined to be positive using the reference method and the
PDS AMP (and confirmed to be positive during the confirmation analysis). In addition, all three
EC positive controls were determined to be positive (for both TC and EC) using the reference
method and the PDS AMP (and confirmed to be positive during the confirmation analysis). One
out of three TC negative control samples was found to be positive (the other two were properly
negative) for TC during the PDS AMP and reference analysis; however, this sample was also
found to be positive during the PDS AMP confirmation analysis. Apparently there was TC
contamination in the sample container causing positive results. However, there was no other
indication of contamination throughout the rest of the test. All of the other negative control
samples generated expected negative results and the method blank samples all produced negative
results. While the organism was not isolated and identified, it seems as though this was an
isolated instance of TC contamination of the sample container or of the negative control culture
and did not adversely impact the results of the test.
4.2 Audits
Two types of audits were performed during the verification test; a technical systems audit (TSA)
of the verification test procedures, and an audit of data quality (ADQ). Audit procedures for the
TSA and the ADQ are described further below.
4.2.1 Technical Systems Audit
The Battelle AMS Center Quality Manager performed a TSA on July 20, 21, and 22, 2010 at
Battelle's microbiology laboratory in Columbus, OH and at the reference laboratory, Superior
Laboratories in Galloway, OH. The EPA AMS Center Quality Manager participated in the
Battelle and Superior Laboratories audits on July 21. The TSA consisted of interviews with
Battelle and Superior Laboratories personnel, observations of test sample preparation and testing
at Battelle and Superior Laboratories, and observation of sample analysis. The purpose of the
audit was to verify that:
• Sample preparation procedures were performed by Battelle according to the TQAP
requirements;
• Reference laboratory methods for analyzing test samples conformed to the TQAP and
reference method requirements;
• Technology testing was performed according to the TQAP and vendor instructions
• Test documentation provided a complete and traceable record of sample preparation and
analysis;
• Equipment used in the test was calibrated and monitored according to TQAP
requirements and standard laboratory procedures.
Seven (7) findings, six (6) observations, and three (3) remarks were identified during the TSA.
The findings involved training records, reference method requirements, sewage sample
collection, sample custody, and traceability of critical reagents. It was determined by Battelle
that none of these had an adverse impact on the test results and all findings received a
satisfactory response.
12
-------
In response to this audit report, the following actions were taken:
• Documentation of reference laboratory microbiology training;
• Generation of a deviation to more accurately describe the collection of the sewage water
sample;
• Clarification and additional detail to document the sewage sample collection on the
custody form.
A TSA report was prepared and distributed to the Verification Test Coordinator, the Battelle
AMS Center Manager, the EPA AMS Center Project Officer, and the EPA AMS Center Quality
Manager.
4.2.2 Data Quality Audit
Records generated in the verification test received a one-over-one review before these records
were used to calculate, evaluate, or report verification results. Data were reviewed by a Battelle
technical staff member involved in the verification test. The person performing the review added
his/her initials and the date to a hard copy of the record being reviewed.
In addition, ADQs were conducted on August 4-6, 2010 and August 24-26, 2010. During the
audits, laboratory data generated at the reference laboratory, Superior Laboratories, Inc. and data
generated by the PDS AMP were reviewed and verified for completeness, accuracy and
traceability. Because this verification testing could potentially be referenced by the Office of
Water, it was decided to establish the testing as a Quality Category II, requiring a QA review of
25% of the test data, and a minimum of three peer-reviewers. Accordingly, at least 25% of the
results for each of the testing scenarios were verified versus the raw data, and 100% of the QC
sample results were verified. The data were traced from the initial acquisition, through reduction
and statistical analysis, to final reporting to ensure the integrity of the reported results. All
calculations performed on the data undergoing the audit were checked.
Two (2) findings and three (3) observations were identified during the ADQs. The two findings
involved documentation of failed QC samples and spreadsheet version control. Findings and
observations from the audits were addressed through the following actions.
• In one instance, a trypticase soy broth (TSB) positive control tested negative for EC.
Upon review of all available documentation, the negative result was unable to be
explained as it was confirmed that the lot of TSB had been used previously to
successfully grow EC. It is suspected that the control was inadequately inoculated with
EC at the time. There was no adverse impact as TSB was only used in the test as the
growth medium to verify the sterility of the PBS used to serial dilute and enumerate the
filtered sewage sample.
• Two transcription errors from the original data sheet into a spreadsheet were corrected
and a spreadsheet formula pertaining to the percent positive results was also corrected.
• Documentation of the reference laboratory reagent controls was added to the project file.
13
-------
None of these had an adverse impact on the test results and all findings received a satisfactory
response. A data audit report was prepared, and a copy was distributed to the EPA AMS Center
Quality Manager.
14
-------
Chapter 5
Statistical Methods
The statistical methods used to evaluate the quantitative performance factors are presented in this
chapter. Qualitative observations were also used to evaluate verification test data.
5.1 False Positive Rates, False Negative Rates, Sensitivity, and Specificity
False positive (FP) and false negative (FN) rates of the PDS AMP were evaluated when
assessing comparability. During this test, true positives (TPs) were those positive results from
the PDS AMP that were confirmed, and FPs were those positive results from the PDS AMP that
were not confirmed by the reference method. Conversely, true negative (TN) results were those
negative results that were confirmed as negative, and FN results were those negative results that
were shown to be positive by the confirmatory method. Performance of the PDS AMP was
tested by comparing the proportion of true positive results from those technologies to the
proportion of positive results from the SM 922 IB and F reference methods.
Sensitivity is defined as the percent of positive samples correctly identified as positive and
specificity is defined as the percent of negative samples correctly identified as negative.
Estimates of sensitivity, specificity, FP rates, and FN rates as percentages for the two methods
were calculated as follows:
TP
Sensitivity = TP + FN x 100%
TN
Specificity = TN + FP x 100%
FP
False positive rate = TN + FP x 100% = T TN + FPI x 100% = 1 - Specificity
FN r TP
False negative rate = TP + FN x 100% = I TP + FNJ x 100% = 1 - Sensitivity
15
-------
5.2 Method Comparability
In order to assess whether the proportion of positive and negative samples were significantly
different between the PDS AMP and the reference method, chi-square tests for independence
were conducted. The chi-squared test was modeled in SAS® (ver. 9.1.3), using the FREQ
procedure. If the calculated chi-square value is less than the critical value, the sample results
between the two methods are not significantly different (95% confidence, alpha = 0.05, p-value >
0.05). If the chi-square value is greater than the critical value (p-value < 0.05), the results
between the two methods are significantly different, and it is concluded that there is a difference
between the two methods.
Prior to testing, a power analysis was conducted to determine the number of replicates required
to determine possible significant differences between the technologies being tested and the
reference method. Conducted using the POWER procedure in the SAS system, the power
analysis determined the number of replicate tests (across both test types) that would be necessary
to detect a specified difference in proportions of a specified size with 80% power, given a
specified value of the proportion for the reference test (the acceptable range of reference test
positive proportions was 25% to 75% for this test), and a significance level of 0.05 for the test.
To summarize, the power analysis shows that for approximately 20 replicates, if the reference
method was 25% positive (five positive results and 15 negative results), then the technology
being tested would be required to be 65% positive (13 positives and seven negative results) to
have a significant difference. PDS AMP results with a higher percentage of positive results out
of 20 replicates would be considered similar to the reference method. Similarly, if the reference
method was 50% positive, then a significant difference could be determined with PDS AMP
results that were 11% or less positive or negative (less than two positives and 18 negatives or
more than 18 positives and two negatives). Finally, if the reference method was 65% positive,
then a significant difference could be determined with at most a 32% positive result. The PDS
AMP results are discussed in the context of this power analysis.
In summary, the smallest difference that is able to be determined with 20 replicates is
approximately a 30% to 40% change in positive results. The power analysis revealed that
differences of 5% or 10% of positive results could be determined, but between 150 and 1,250
replicates may be required.
16
-------
Chapter 6
Test Results
As mentioned previously, this verification test included both quantitative and qualitative
evaluations. The quantitative evaluation was conducted to assess the comparability of results
generated by the presence/absence results for the PDS AMP with those generated by the
presence/absence result from the reference methods. The qualitative evaluation was performed
to document the operational aspects of the PDS AMP when it was used during verification
testing. The following sections provide the results of the quantitative and qualitative evaluations.
Tables presenting the raw data presence/absence results for the reference methods, the PDS
AMP, and the confirmation analyses are provided in the Appendix.
6.1 TCData
The positive TC test results for the PDS AMP and reference method (SM 922IB) are presented
in Table 6-1. One of the three dilutions yielded the target 50 ± 25% split in responses for the
reference method. However, a second dilution generated results that were similar to the targeted
range (85% positive, 15% negative). Therefore, results for both of these dilutions (Dilutions A
and B) are reported. Table 6-1 summarizes the positive TC test results for the PDS AMP
incubated for 18 and 24 h.
Table 6-1. TC Positive Results
Dilution
A (10 org/100 mL)
B(lorg/100mL)
PDS 18 h
N
13
2
% of total
samples
65%
10%
PDS 24 h
N
19
5
% of total
samples
95%
25%
SM 9221B
N
17
5
% of total
samples
85%
25%
N - Number
17
-------
Tables 6-2 and 6-3 summarize the TP (confirmed) and TN (confirmed) TC results for the PDS
AMP (18 and 24 hour incubations). The reference method data are also presented.
Table 6-2. TC Data Summary - Positives
Dilution
A (10 org/100 mL)
B(lorg/100mL)
PDS 18 h
N
13
2
Confirmed
TP
13
2
Difference
(FP)
0
0
PDS 24 h
N
19
5
Confirmed
TP
17
4
Difference
(FP)
2
1
SM 9221B
N
17
5
Table 6-3. TC Data Summary - Negatives
Dilution
A (10 org/100 mL)
B(lorg/100mL)
PDS 18 h
N
7
18
Confirmed
TN
3
14
Difference
(FN)
4
4
PDS 24 h
N
1
15
Confirmed
TN
1
13
Difference
(FN)
0
2
SM 9221B
N
3
15
The sensitivity, specificity, FP, and FN rates for the PDS AMP 18 hour and 24 hour TC results
were determined as described in Section 5.1 and are presented in Table 6-4.
Table 6-4. TC Data Summary - Confirmations
Incubation Time (h)
Sensitivity
Specificity
False Positive
False Negative
18 h
65%
100%
0%
35%
24 h
91%
82%
18%
9%
6.2 EC Data
Table 6-5 summarizes the positive EC test results for the PDS AMP incubated for 18 and 24 h
according to the manufacturer's directions. The results for the reference method (SM 922IF) are
also presented.
Table 6-5. EC Positives
Dilution
A (10 org/100 mL)
PDS 18 h
N
8
% of total
samples
40%
PDS 24 h
N
8
% of total
samples
40%
SM 9221F
N
6
% of total
samples
30%
18
-------
Tables 6-6 and 6-7 summarize the confirmed TP and TN EC results for the PDS AMP (18 and 24
hour incubations). The reference method data are also presented.
Table 6-6. EC Summary - Positives
Dilution
A (10 org/100 mL)
PDS 18 h
N
8
Confirmed
TP
8
Difference
(FP)
0
PDS 24 h
N
8
Confirmed
TP
8
Difference
(FP)
0
SM 9221F
N
6
Table 6-7. EC Summary - Negatives
Dilution
A (10 org/100 mL)
PDS 18 h
N
12
Confirmed
TN
8
Difference
(FN)
4
PDS 24 h
N
12
Confirmed
TN
8
Difference
(FN)
4
SM 9221F
N
14
The sensitivity, specificity, FP, and FN rates for the PDS AMP 18 hour and 24 hour EC results
were determined as described in Section 5.1 and are presented in Table 6-8.
Table 6-8. EC Data Summary - Confirmations
Incubation Time (h)
Sensitivity
Specificity
False Positive
False Negative
18 h
67%
100%
0%
33%
24 h
67%
100%
0%
33%
6.3 Method Comparability
Tables 6-9 and 6-10 show the results from the chi-square test for independence that was
performed to compare the TC results from the PDS AMP for each incubation time period against
the reference method (SM 922 IB). For TC, data from each of the two dilutions were tested
separately and together. The chi-square value for each of the TC dilutions, as well as the
additive chi-square value, was less than the critical limits; therefore, the chi-square test did not
detect any differences between the results of the PDS AMP and the reference method at 18 or 24
h. The calculated p-values were greater than 0.05, indicating that the data did not show a
statistically significant difference between the two methods for the detection of TCs.
19
-------
Table 6-9. TCs-18h
Dilution
A (10 org/100 mL)
B(lorg/100mL)
PDS AMP
+
13
2
.
7
18
SM9221B
+
17
5
.
3
15
Additive Result
Chi-
Square
2.13
1.56
3.69
Degrees
of
freedom
1
1
2
p-Value
0.144
0.212
0.164
Critical Limits
(p=0.05)
3.841
3.841
5.991
Table 6-10. TCs-24h
Dilution
A (10 org/100 mL)
B(lorg/100mL)
PDS
AMP
+
19
5
_
1
15
SM9221B
+
17
5
_
3
15
Additive Result
Chi-
Square
1.11
0.00
1.11
Degrees
of
freedom
1
1
2
p- Value
0.292
1.00
0.574
Critical
Limits
(p=0.05)
3.841
3.841
5.991
These results are consistent with the power analysis performed before testing and described in
Section 5.2.
For TC, the reference method generated 85% positive results for Dilution A. While the power
analysis was only performed for reference method proportions between 25% and 75%, when the
reference method was 75% positive (15 positive and five negative), the technology being tested
was required to be approximately 65% negative (seven positive and 13 negative) to be
considered different from the reference method. Had the power analysis been applied to the
reference method being 85% positive, a result that was significantly different from the reference
would be similar in proportion, but slightly less negative (possibly eight or nine positive and 12
or 11 negative). Given the Dilution A result of 13 positive and seven negative at 18 h and 19
positive and one negative at 24 h, neither result was more than 50% negative, confirming that the
chi-square result was consistent with the power analysis. It is possible that smaller differences
between the reference method and the PDS AMP could be determined if more replicates were
included in the experimental design.
For Dilution B, the TC reference method results were 25% positive. This was the same
proportion as negative Dilution A. Therefore, the evaluation of the results in the context of the
power analysis was the same as for Dilution A, only with the opposite sign. Since the reference
method was 25% positive, the power analysis showed that a result with approximately 65%
positive (13 positive and seven negative) would be required to exhibit a significant difference.
The PDS results for Dilution B were 10% positive (two positive and 18 negative) after 18 h,
slightly less positive than the reference method and not close to the difference required to
determine a significant difference. The result after 24 h was an exact match to the reference
method. Both of these results confirmed that the chi-square result was consistent with the power
analysis.
20
-------
Tables 6-11 and 6-12 show the results from the chi-square test for independence that was
performed to compare the EC results from the PDS AMP for each incubation time period against
the reference method (SM 922IF). For EC, the chi-square value was also less than the critical
limits; therefore, the chi-square test did not detect any differences between the results of the PDS
AMP and the reference method at 18 or 24 h. The calculated p-values were also greater than
0.05, indicating that the data did not show a statistically significant difference between the two
methods for detection of EC.
Table 6-11. EC-18H
Dilution
A (10 org/100 mL)
PDS
AMP
+
8
-
12
SM9221F
+
6
-
14
Additive Result
Chi-
Square
0.440
0.440
Degrees
of
freedom
1
1
p-Value
0.507
0.507
Critical Limits
(p=0.05)
3.841
3.841
Table 6-12. EC - 24 h
Dilution
A (10 org/100 mL)
PDS
AMP
+
8
-
12
SM9221F
+
6
-
14
Additive Result
Chi-
Square
0.440
0.440
Degrees
of
freedom
1
1
p-Value
0.507
0.507
Critical
Limits
(p=0.05)
3.841
3.841
As was the case for TC, the EC results are consistent with the power analysis performed before
testing. The proportion of positive results from the reference method was 30% (six positive and
14 negative). According to the power analysis, approximately a 75% positive result (15 positive
and five negative) would be required from the PDS AMP for a significant difference to be
determined between the reference method and the PDS AMP. The PDS AMP result after both
18 h and 24 h was 40% positive (eight positive and 12 negative), slightly more positive than the
reference method and not close to the difference required to determine a significant difference,
confirming the chi-square results as being consistent with the power analysis. The determination
of significant differences in EC results was also limited by the number of replicates as was
described above.
6.4 Detection of Additional Concentration Levels in Continuous Operating Mode
The objective of this component of the testing was to verify the PDS AMP capability of
reporting analysis results as soon as determined by the PDS DTU rather than waiting for the end
of an incubation time period such as 18 or 24 h. Table 6-13 gives the results for the analysis of
various concentration of EC ATCC 8739 including the result provided and the time of result. In
general, the PDS AMP did not generate positive EC responses except for two of the 10,000
EC/100 mL samples. However, all of the samples were reported as positive for TC. Four
replicate samples of each concentration were analyzed and the TC positive results were reported
21
-------
between 14 and 16 h for 10 EC 7100 mL, 13and 15 h for 100 EC 7100 mL, 11.5 and 13.5 h for
1,000 EC 7100 mL, and 10 and 11.5 h for 10,000 EC 7100 mL. The two positive EC results were
reported in approximately 23 h, just before the end of the 24 h analysis period.
Table 6-13. Results of Analysis of Additional Concentrations in Continuous Operation
Mode
EC Cone.
(org/lOOmL)
10
100
1,000
10,000
TC
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Incubation
Time TC
Detected
(h:min)
15:45
15:08
14:48
14:02
14:53
14:43
13:22
13:02
13:17
12:46
12:07
11:48
10:55
10:18
11:21
10:34
EC
O
o
O
o
o
o
o
o
o
o
o
o
o
o
X
X
Incubation
Time EC
Detected
(h:min)
ND
ND
ND
ND
ND
ND
ND
ND
ND
ND
ND
ND
ND
ND
23:16
22:59
X=Presence; O= Absence
6.5 Operational Factors
The verification staff found that the PDS AMP was easy to use. A PDS representative came to
Battelle to set up the equipment and train the verification staff in the operation of the PDS AMP.
The PDS AMP was set up by plugging the PDS DTU and desktop computer into standard 110
volt power and powering up. For operation in continuous mode, no special laboratory facilities
were required. In manual mode, laboratory incubators were required. Following an
approximately 30-minute training session, the operators (consisting of Battelle microbiology
technicians) were comfortable operating the PDS AMP without assistance.
Prior to use for water samples, PDS required the analysis of three control cartridges that do not
contain any liquid, but contain the same polymer bottom (as the standard sample cartridges) that
fluoresces at the proper wavelengths to indicate the presence of TC only with one cartridge, TC
and EC with a second cartridge, and neither with the third cartridge. Once these control
22
-------
cartridges had been analyzed and reported the proper results, the PDS DTUs were ready for the
analysis of samples.
As previously described, the PDS AMP was operated in both manual and continuous
measurement mode for the simultaneous measurement of TC and EC using the same 100 mL
cartridge. In manual mode, 100 mL of the water sample was dispensed into each cartridge and
the cartridge was snapped firmly shut and swirled to dissolve the contents. The cartridges were
then placed in an incubator that was held between 35 and 36 °C for 24 h. After 18 h, the
cartridges were removed from the incubator and inserted into the PDS DTU for an initial
measurement. The cartridges were measured two at a time by clicking on a "start" button on the
computer screen. The measurement of two samples took approximately 30 seconds and the
cartridges were immediately returned to the incubator for the remaining 6 h to complete the 24 h
incubation. The measurement step was repeated after the 24 h incubation was complete.
In continuous operation mode, the samples were loaded into the cartridges in an identical fashion
and the 24 h incubation/analysis was started. The samples (two at a time) were incubated within
the PDS DTU and results were reported on the screen as soon as the PDS AMP was able to make
a conclusive determination of TC and/or EC based on the fluorescence measurement. A positive
result could have been reported at any point during the 24 h analysis, while a negative result
would not occur until the end of the 24 h incubation. During the continuous measure, a
countdown timer appeared on the computer screen to indicate the time remaining for the
analysis. The continuous operation mode eliminates the need for a technician to be present to
read the sample result. Also, the PDS AMP method calls for a 18 h or 24 h analysis, shortening
the analysis time from the 48 to 72 required by the standard methods, increasing the efficiency
and decreasing the amount of reagents and manpower expended performing the reference
methods.
During the measurement step in both modes, the result of each measurement was displayed on
the screen and the operator recorded the result on a sample data sheet. Each result could also be
downloaded for review and viewed on a computer containing the PDS AMP software, but the
results from a group of samples could not be exported as a spreadsheet. The PDS DTU (not
including corresponding desktop computer) has dimensions of 20 cm wide x 30 cm deep X15 cm
high (8 in wide x 16 in deep x 12 in high) and weighs approximately 5 kilograms (11 pounds).
The PDS DTU, computer, and all required software costs approximately $10,000. Sample
cartridges can be purchased for approximately $10 per cartridge.
23
-------
Chapter 7
Performance Summary
To comply with the TCR, water utilities need coliform detection technologies that are able to
detect TC and EC at concentrations of one organism per 100 mL samples. This ETV test
verified the performance of the PDS AMP at that level of detection. While it is difficult to
determine if a single target organism is present in 100 mL of water, when approximately half of
the analyzed replicates are positive and half are negative, the density of the organism has become
adequately low so that a positive result can be considered single organism detection. Therefore,
for the purpose of this verification test, spiked DW dilution sets were prepared that provided 50
±25% positive results for TC and EC with the reference methods and then the results from the
reference method were compared with the PDS AMP. The results of the verification of the PDS
AMP are summarized below:
Positive Results. Table 7-1 summarizes the positive TC test results for the PDS AMP incubated
for 18 and 24 h.
Table 7-1. Results Summary for Positive PDS AMP Results for TC and EC
TCor
EC
TC
EC
Dilution
A (10 org/100 mL)
B(lorg/100mL)
A (10 org/100 mL)
PDS 18 h
N
13
2
8
% of total
samples
65%
10%
40%
PDS 24 h
N
19
5
8
% of total
samples
95%
25%
40%
SM 9221B/F
N
17
5
6
% of total
samples
85%
25%
30%
Specificity, Sensitivity, FPrate, and FN rate. Table 7-2 summarizes the specificity, sensitivity,
FP rate, and FN rate for TC and EC for 18 and 24 h incubations. Sensitivity is defined as the
percent of positive samples correctly identified as positive and specificity is defined as the
percent of negative samples correctly identified as negative.
24
-------
Table 7-2. Results Summary of PDS AMP for 18 and 24 h Incubation Times
Sensitivity
Specificity
False Positive Rate
False Negative Rate
TC
18
65%
100%
0%
35%
24
91%
82%
18%
9%
EC
18
67%
100%
0%
33%
24
67%
100%
0%
33%
Comparability. In another approach of comparison, a chi-square test for independence was
performed to compare the PDS AMP for each incubation time period against the reference
methods (SM 922IB and F). For the EC and TC results, data from each dilution was tested
separately and for the TC only, two dilution levels were tested together. The chi-square value for
the EC solution and each of the TC dilutions, as well as the additive chi-square value, was less
than the critical limit in each case; therefore, for EC and TC, the chi-square test did not detect
any differences between the results of the PDS AMP and the reference method at 18 or 24 h. In
addition, the calculated p-values were also greater than 0.05, indicating that the data did not
show a statistically significant difference between the two methods for the detection of EC or TC
at the 95% confidence interval. These results were consistent with the power analysis performed
before testing and described in Section 5.2. This power analysis showed the number of replicate
samples required for significant differences at a minimum of 80% power. It showed that the
smallest difference that is able to be determined with 20 replicates was approximately a 30% to
40% change in positive results for each dilution. The power analysis also revealed that
differences of 5% or 10% of positive results could be determined, but between 150 and 1,250
replicates may be required.
Additional Concentrations in Continuous Operation. The objective of this component of the
testing was to verify the PDS AMP capability of reporting analysis results as soon as determined
by the PDS AMP rather than waiting for the end of an incubation time period such as 18 or 24 h.
Four concentrations of EC ATCC 8739 (10, 100, 1,000, and 10,000 EC/100 mL) were analyzed
four times each. The PDS AMP did not generate positive EC responses except for two of the
10,000 EC/100 mL samples. However, all of the samples were reported as positive for TC in an
average time of approximately 13 hours. An observation was made that the amount of time until
detection for the TC samples decreased with each increasing concentration level.
Operational Factors. The PDS AMP was operated in both manual and continuous measurement
mode for the simultaneous measurement of TC and EC using the same 100 mL cartridge. In
manual mode, 100 mL of the water sample was dispensed into each cartridge and the cartridge
was snapped firmly shut and swirled to dissolve the contents. The cartridges were then placed in
an incubator that was held between 35 and 36 °C for 24 h. After 18 h, the cartridges were
removed from the incubator and inserted into the PDS DTU for an initial measurement. The
cartridges were measured two at a time by clicking on a "start" button on the computer screen.
The measurement of two samples took approximately 30 seconds and the cartridges were
immediately returned to the incubator for the remaining 6 h to complete the 24 h incubation. The
measurement step was repeated after the 24 h incubation was complete.
25
-------
In continuous operation mode, the samples were loaded into the cartridges in an identical fashion
and the 24 h incubation/analysis was started. The samples (two at a time) were incubated within
the PDS DTU and results were reported on the screen as soon as the PDS AMP was able to make
a conclusive determination of TC and/or EC based on the fluorescence measurement. The
continuous operation mode eliminates the need for a technician to be present to read the sample
result. Also, the PDS AMP method calls for a 18 h or 24 h analysis, shortening the analysis time
from the 48 to 72 required by the standard methods, increasing the efficiency and decreasing the
amount of reagents and manpower expended performing the reference methods.
During the measurement step in both modes, the result of each measurement was displayed on
the screen and the operator recorded the result on a sample data sheet. Each result could also be
downloaded for review and viewed on a computer containing the PDS AMP software, but the
results from a group of samples could not be exported as a spreadsheet.
The PDS DTU (not including corresponding desktop computer) has dimensions of 20 cm wide x
30 cm deep x 15 cm high (8 in wide x 16 in deep x 12 in high) and weighs approximately 5
kilograms (11 pounds). The PDS DTU and computer and all required software costs
approximately $10,000. Sample cartridges can be purchased for approximately $10 per
cartridge.
26
-------
Chapter 8
References
1. Total Coliform Rule, United States Federal Register, 54 FR 27544-27568, June 29, 1989,
Vol. 54, No. 124
2. EPA Microbiological Alternate Test Procedure Protocol for Drinking Water, Ambient
Water, and Wastewater Monitoring Methods, EPA 821-B-03-004, April 2004.
3. Test/QA Plan for Verification of Coliform Detection Technologies for Drinking Water,
Battelle, Version 1.0, July 14, 2010.
4. Quality Management Plan for the ETV Advanced Monitoring Systems Center, Version 7.
U.S. Environmental Technology Verification Program, Battelle, November 2008.
5. American Public Health Association, American Water Works Association, and Water
Environment Federation. 2005. Standard Methods for the Examination of Water and
Wastewater. 21st Edition.
6. SOP GEN.V-003-10. Standard Operating Procedure for the Use of pH meters to Measure
pH. Battelle.
7. SOP GEN.V-013-04. SOP for the Calibration and Maintenance of Thermometers.
Battelle.
8. Bacteriological Analytical Manual, 8th Edition. 1995. "Chapter 3: Aerobic Plate Count".
AOAC International..
9. SOP ENV-ADM-009, Standard Operating Procedure for Sample Chain-of-Custody
Battelle, September 2007.
27
-------
Appendix
Raw Data from Reference Methods, PDS AMP, and Confirmation Analyses
28
-------
Dilution
A
(10
org/lOOmL)
Sample
No.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Percent Positive=
SM 9221B/F
TC
X
X
X
X
X
X
X
X
X
X
X
X
X
X
o
o
o
X
X
X
85%
EC
O
X
o
o
o
o
o
o
o
X
X
X
o
X
o
o
o
o
X
o
30%
PDS 18H
TC
X
X
X
o
X
o
X
o
o
o
X
X
X
X
X
X
X
o
X
o
65%
EC
X
O
X
o
o
o
X
o
o
o
X
o
X
X
o
o
X
o
o
X
40%
PDS 24H
TC
X
X
X
o
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
95%
EC
X
O
X
o
o
o
X
o
o
o
X
o
X
X
o
o
X
o
o
X
40%
PDS
Confirmation
via SM9221B/F
TC
X
X
X
o
X
X
X
X
X
o
X
X
X
X
X
X
X
o
X
X
85%
EC
X
X
X
o
o
o
X
X
o
o
X
X
X
X
X
o
X
o
o
X
60%
X= Presence
O= Absence
29
-------
Dilution
B
(1
org/lOOmL)
Sample
No.
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
Percent Positive=
SM 9221B/F
TC
O
X
X
O
X
O
X
O
O
O
O
O
O
O
O
O
O
O
O
X
25%
EC
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
0%
PDS 18H
TC
O
O
O
O
O
O
X
O
O
X
O
O
O
O
O
O
O
O
O
O
10%
EC
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
0%
PDS 24H
TC
O
O
O
O
O
X
X
O
O
X
O
X
X
O
O
O
O
O
O
O
25%
EC
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
0%
PDS
Confirmation
via SM9221B/F
TC
O
O
O
X
O
O
x
O
X
X
O
X
X
O
O
O
O
O
O
O
30%
EC
O
O
O
O
O
O
o
O
o
X
o
o
o
o
o
o
o
o
o
o
5%
X= Presence
O= Absence
30
-------
Dilution
C
(0.1
org/lOOmL)
Percent
Controls
Method
Blank
TC Positive
(Ea)
EC Positive
(EC)
TC Neg/EC
Neg (Pa)
Sample
No.
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
Positive=
M
61
62
63
64
65
66
67
68
69
70
71
72
SM 9221B/F
TC
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
0%
M
O
O
O
X
X
X
X
X
X
O
O
O
EC
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
0%
M
O
O
O
O
O
O
X
X
X
O
O
O
PDS 18H
TC
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
0%
M
O
O
O
X
X
X
X
X
X
O
O
X
EC
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
0%
M
O
O
O
O
O
O
X
X
X
O
O
O
PDS 24H
TC
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
0%
M
O
O
O
X
X
X
X
X
X
O
O
X
EC
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
0%
M
O
O
O
O
O
O
X
X
X
O
O
O
PDS
Confirmation
via SM9221B/F
TC
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
0%
M
O
O
O
X
X
X
X
X
X
O
O
X
EC
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
O
0%
M
O
O
O
O
O
O
X
X
X
O
O
O
rese
= Absence
31
------- |