Batreiie
The Business of Innovation
Environmental Technology
Verification Program
Advanced Monitoring
Systems Center
    Test/QA Plan for Verification of
  Coliform Detection Technologies for
           Drinking Water

-------
      TEST/ QUALITY ASSURANCE PLAN
                       for
                  Verification of
Coliform Detection Technologies for Drinking Water

                     Version 1.0


                    July 14, 2010
                    Prepared by
                      Battelle
                   505 King Avenue
               Columbus, OH 43201-2693

-------
                                                     Coliform Detection Technologies
                                                                 Test/QA Plan
                                                                   Page 3 of 3
                                                                   Version 1.0
                                                                Date: 7/14/2010
                                SECTION A

                         PROJECT MANAGEMENT
Al    VENDOR APPROVAL PAGE
                  ETV Advanced Monitoring Systems Center

                      Test/QA Plan for Verification of
             Coliform Detection Technologies for Drinking Water
                               APPROVAL:
               Name
               Company

               Date

-------
                                                          Coliform Detection Technologies
                                                                        Test/QA Plan
                                                                         Page 4 of 4
                                                                         Version 1.0
                                                                      Date: 7/14/2010
A2    TABLE OF CONTENTS
Section

SECTION A PROJECT MANAGEMENT	3
Al    Vendor Approval Page	3
A2    Table of Contents	4
A3    List of Acronyms	6
A4    Distribution List	7
A5    Verification Test Organization	8
A6    Background	16
A7    Verification Test Description and Schedule	18
A8    Quality Objectives and Criteria for Measurement Data	20
A9    Special Training/Certification	21
A10   Documentation and Records	22

SECTION B MEASUREMENT AND DATA ACQUISITION	24
Bl    Experimental Design	24
B2    Sampling Method Requirements	40
B3    Sample Handling and Custody Requirements	41
B4    Laboratory Reference Methods	41
B5    Quality Control Requirements	42
B6    Equipment Testing, Inspection, and Maintenance	43
B7    Calibration Verification	45
B8    Inspection/Acceptance of Supplies and Consumables	45
B9    Non-Direct Measurements	46
BIO   Data Management	46

SECTION C ASSESSMENT AND OVERSIGHT	48
Cl    Assessments and Response Actions	48
C2    Reports to Management	50

SECTION D DATA VALIDATION AND USABILITY	52
Dl    Data Review, Validation, and Verification Requirements	52
D2    Validation and Verification Methods	52
D3    Reconciliation with User Requirements	52

SECTION E REFERENCES	54
El    References	54

-------
                                                              Coliform Detection Technologies
                                                                             Test/QA Plan
                                                                               Page 5 of 5
                                                                               Version 1.0
                                                                           Date: 7/14/2010

List of Figures

Figure 1. Organization Chart for the Verification Test	9
Figure 2. Experimental Design for Sample Dilution and Analysis	31
FigureS.  Flowchart describing confirmation analyses for both the technologies being verified
andSM9221B	37
List of Tables

Table 1. Proposed Verification Test Schedule	20
Table 2. Quality Control Strains	21
TableS. Summary of Data Recording Process	23
Table 4. Sample Preparation Timeline	25
Table 5. Volume of Drinking Water Required for Verification Testing	27
Table 6. Power Analysis Results for Determination of Replicates	33
Table 7. Sample Analysis Timeline	34
Table 8. False Negative Rate Comparison	39
Table 9. False Positive Rate Comparison	39
Table 10.  Reference Methods	42
Table 11.  Quality Control Samples for Reference and Technology Methods	44

-------
                                                             Coliform Detection Technologies
                                                                           Test/QA Plan
                                                                             Page 6 of 6
                                                                             Version 1.0
                                                                         Date: 7/14/2010
A3    LIST OF ACRONYMS
ADQ        audit of data quality
AMS        Advanced Monitoring Systems
ATCC       American Type Culture Collection
BGLB       brilliant green lactose bile
COC        chain of custody
DANETV    Danish ETV
DDW        dechlorinated drinking water
DQA        data quality audit
DW         drinking water
E. coli       Escherichia coll
EPA         U.S Environmental Protection Agency
ETV         Environmental Technology Verification
FN          false negative
FP          false positive
h            hours
LRB         laboratory record book
LTB         lauryl tryptose broth
MB         method blank
MCL        Maximum Contaminant Level
MCLG       Maximum Contaminant Level Goal
MUG        4-methyllumbelliferyl-p-D-glucorinide
N           number
NA         nutrient agar
PA          presence/absence
PDF         portable document format
PO          project officer
QA         quality assurance
QC          quality control
QMP        Quality Management Plan
RB          reagent blank
RMO        Records Management Office
SM         Standard Methods
SSDW       spiked, stressed drinking water
SWTP       Southerly Wastewater Treatment Plant
TC          total coliform
TCR         Total Coliform Rule
TSA         technical systems audit

-------
A4    DISTRIBUTION LIST

Vendors
James Berg
Colifast AS
Strandveien 33, P.O.Box 31
1324 Lysaker, Norway

Dr. Peter Gallant
Pathogen Detection Systems, Inc.
116 Barrie Street, Suite 4697
Kingston, Ontario, Canada K7L 3N6
EPA ETV
John McKernan
Michelle Henderson
U.S. Environmental Protection Agency
26 West Martin Luther King Drive
Cincinnati OH 45268

Battelle
Ryan James
Daniel Lorch
Amy Dindal
Battelle
505 King Ave.
Columbus, OH 43201

Stacy Pala
Rosanna Buhl
Battelle
397 Washington Street
Duxbury, MA 02332
               Coliform Detection Technologies
                             Test/QA Plan
                               Page 7 of 7
                               Version 1.0
                           Date: 7/14/2010

Peer Reviewers
Jim Sinclair
Sandhya Parshionikar
Jennifer Best
Key a Sen
U.S. Environmental Protection Agency
Office of Water
26 West Martin Luther King Drive
Cincinnati OH 45268

Rick Sakaji
East Bay Municipal Water District
375 Eleventh St, MS 705
Oakland, CA 94607-4240

Mark Rodgers
U.S. Environmental Protection Agency
National Risk Management Research
Laboratory
26 West Martin Luther King Drive
Cincinnati OH 45268

Cooperating ETV Programs
John Neate
Mona El Hallak
ETV Canada
2070 Hadwen Road, Unit 201A
Mississauga, Ontario
L5K 2C9

Christian Gron
Mette Tjener Andersson
Claus J0rgensen
DANETV
Agern Alle 5
DK-2970 H0rsholm
Denmark

-------
                                                               Coliform Detection Technologies
                                                                              Test/QA Plan
                                                                                Page 8 of 8
                                                                                Version 1.0
                                                                            Date: 7/14/2010
A5    VERIFICATION TEST ORGANIZATION
       The verification test will be conducted under the U.S. Environmental Protection Agency
(EPA) Environmental Technology Verification (ETV) Program. It will be performed by
Battelle, which is managing the ETV Advanced Monitoring Systems (AMS) Center through a
cooperative agreement with EPA.  The scope of the AMS Center covers verification of
monitoring technologies for contaminants and natural species in air, water, and soil.  This
verification testing is also being conducted in cooperation with the Canadian and Danish
(DANETV) ETV programs as a possible ETV verification in those countries. The criteria for
ETV cooperation are outlined in a cooperative verification process document prepared by the
respective cooperating ETV programs. At least one of the technology vendors has agreed to
participate in the verification with the possibility that their technology be considered for
verification.  It should be noted however that neither U.S. ETV verification, nor the cooperation
with the Canadian or Danish ETV programs, represents an approval of methods for regulatory
compliance.  This verification test is being conducted to provide a verification of technologies
that are not already approved for compliance monitoring in the U.S., or elsewhere, such that the
vendors have an opportunity to have their technologies tested under similar set of test conditions
that might be required for method approval.
       The day to day operations of this verification test will be coordinated and supervised by
Battelle, with the participation of the vendors who will have the performance of their
technologies verified. Testing of the technologies being verified will be conducted at Battelle
laboratories in Columbus, Ohio. Each vendor will provide Battelle with their respective
technologies and will train the verification staff.  Staff from Battelle will operate the
technologies during verification testing.
       Quality Assurance (QA) oversight will be provided by the  Battelle Quality  Manager and
also by the EPA AMS Center Quality Manager, at her discretion.  The organization chart in
Figure 1 identifies the responsibilities of the organizations and individuals associated with the
verification test. Roles and responsibilities are defined further below.

-------
                                                                 Coliform Detection Technologies
                                                                                Test/QA Plan
                                                                                  Page 9 of 9
                                                                                  Version 1.0
                                                                              Date: 7/14/2010
       This test/QA plan will be subjected to review by the vendors, EPA, and expert peer
reviewers referred to as verification test stakeholders in this test/QA plan. The reviews of this
test/QA plan will help to improve the design of the verification test and the resulting report(s)
such that they better meet the needs of potential users of these technologies.  After all peer
review comments are received, revisions will be made to the test/QA plan, and it will be
submitted to EPA for approval.
                                       AMS Center
                                       Stakeholders
              Battelle
            Management
         Battelle AMS Center
          Quality Manager
           Rosanna Buhl
 Battelle AMS
Center Manager
                                       Amy Dindal
                                     Verification Test
                                       Coordinator
                                       Ryan James
                                      Battelle Testing
                                          Staff
EPA AMS Center
 Project Officer
                           John McKernan
                       EPA AMS Center Quality
                         Assurance Manager
                          Michelle Henderson
                               Vendor
                           Representatives
Figure 1. Organization Chart for the Verification Test
A5.1   Battelle
       Dr. Ryan James is the AMS Center's Verification Test Coordinator for this test.  In this
role, Dr. James will have overall responsibility for ensuring that the technical, schedule, and cost
goals established for the verification test are met.  Specifically, Dr. James will:
       •   Prepare the draft test/QA plan, verification reports, and verification statements
       •   Establish a budget for the verification test and manage staff to ensure the budget is
           not exceeded

-------
                                                                 Coliform Detection Technologies
                                                                                 Test/QA Plan
                                                                                 Page 10 of 10
                                                                                  Version 1.0
                                                                              Date: 7/14/2010
       •  Revise the draft test/QA plan, verification reports, and verification statements in
          response to reviewer comments
       •  Assemble a qualified technical staff to conduct the verification test
       •  Direct the staff in performing the verification test in accordance with this test/QA
          plan
       •  Hold a kick-off meeting approximately one week prior to the start of the verification
          test to review the critical logistical, technical, and administrative aspects of the
          verification test.  Responsibility for each aspect of the verification test will be
          reviewed to ensure each participant understands his/her role
       •  Ensure that all quality procedures specified in this test/QA plan and in the AMS
          Center Quality Management Plan1 (QMP) are followed
       •  Maintain real-time communication with the Battelle AMS Center Manager and EPA
          AMS Center Project Officer and Quality Assurance Manager on any potential or
          actual deviations from the test/QA plan
       •  Serve as the primary point of contact for vendor representatives
       •  Ensure that confidentiality of sensitive vendor information is maintained
       •  Assist vendors as needed during verification testing
       •  Become familiar with the operation and maintenance of the technologies through
          instruction by the vendors
       •  Direct the 100% validation and verification of data by technical staff prior to submission for
          data quality audit
       •  Respond to any issues raised in assessment  reports, audits, or from verification staff
          observations, and institute corrective action as necessary
       •  Coordinate distribution of the final test/QA plan, verification reports, and verification
          statements.

       Ms. Amy Dindal is Battelle's Manager for the AMS Center.  As such, Ms. Dindal will
oversee the various stages of verification testing. Ms. Dindal will:
       •  Review the draft and final test/QA plan
       •  Review the draft and final verification reports and verification statements

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 11 of 11
                                                                                 Version 1.0
                                                                             Date: 7/14/2010
       •  Ensure that necessary Battelle resources, including staff and facilities, are committed
          to the verification test
       •  Ensure that confidentiality of sensitive vendor information is maintained
       •  Support Dr. James in responding to any issues raised in assessment reports and audits
       •  Maintain communication with EPA's technical and quality managers
       •  Issue a stop work order if Battelle or EPA QA staff discovers adverse findings that
          will compromise test results.

       Mr. Daniel Lorch and other technical staff in Battelle's microbiological laboratories will
support Dr. James in planning and conducting the verification test. They will:
       •  Assist in planning for the test, and making arrangements for the receipt of and
          training on the technologies
       •  Attend the verification test kick-off meeting
       •  Assist vendor staff as needed during technology receipt and training
       •  Conduct verification testing using each participating technology, following all aspects
          of the ETV AMS Center QMP as well as the test/QA plan for this verification
       •  Coordinate activities of Superior Laboratories, Inc., the laboratory performing the
          reference measurements
       •  Confirm that Superior Laboratories, Inc. is completing all required  QC samples and
          corresponding documentation
       •  Support Dr. James in the preparation of the test/QA plan and reports, as necessary
       •  Support Dr. James in responding to any issues raised in assessment reports and audits
          related to technical performance, statistics, or data reduction as needed.

       Ms. Rosanna Buhl is Battelle's Quality Manager for the AMS Center.  Ms. Buhl will:
       •  Review the draft and final test/QA plan
       •  Attend the verification test kick-off meeting and lead the discussion of the QA
          elements of the kickoff meeting checklist

-------
                                                                 Coliform Detection Technologies
                                                                                Test/QA Plan
                                                                                Page 12 of 12
                                                                                  Version 1.0
                                                                              Date: 7/14/2010
       •  Prior to the start of verification testing, verify the presence of applicable training
          records for Battelle and Superior Laboratories,  Inc. staff, including any vendor
          training on test equipment
       •  Contact Superior Laboratories, Inc. before the verification test to confirm adequate
          quality practices are in place
       •  Conduct a technical  systems audit at least once during the verification test, or
          designate other QA staff to conduct the audit (Section Cl.l)
       •  Conduct audits of verification data quality or designate other QA staff to conduct the
          data audit (Section C1.2).
       •  Prepare and distribute an assessment report for each audit
       •  Verify that audit responses for each audit finding and observation are appropriate and
          that corrective action has been implemented effectively
       •  Communicate to the test coordinator and/or technical staff the need for immediate
          corrective action if an audit identifies test/QA plan deviations or practices that
          threaten data quality
       •  Review and approve test/QA plan amendments, deviations and audit reports
       •  Verify implementation of any necessary corrective action
       •  Request that Battelle's AMS Center Manager issue a stop work order if audits
          indicate that data quality is being compromised
       •  Provide a summary of the QA/QC activities and results for the verification reports
       •  Review the draft and final  verification reports and verification statements.
       •  Delegate QA  activities to other Battelle quality staff as needed to meet project
          schedules

A5.2  Technology Vendors

       The responsibilities of the technology vendors are as follows:
       •  Review and provide comments on the draft test/QA plan
       •  Accept (by signature of a company representative) the final test/QA plan prior to test
          initiation

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 13 of 13
                                                                                 Version 1.0
                                                                             Date: 7/14/2010
       •  Provide their technology for evaluation during the verification test
       •  Provide all other equipment/supplies/reagents/consumables needed to operate their
          technology for the duration of the verification test
       •  Supply training on the use of the technology, and provide written consent and
          instructions for verification staff to carry out testing, including written instructions for
          routine operation of their technology
       •  Provide maintenance and repair support for their technology, on-site if necessary,
          throughout the duration of the verification test
       •  Review and provide comments on the draft verification report and statement for their
          respective technology
       •  Recognize that results in the verification report will be reviewed by the cooperating
          ETV programs in order to determine if verification is applicable in  those countries
       •  Allow for publication of the verification results by the cooperating  ETV programs, if
          vendors are considered for verification in those countries.

A5.3   EPA

       EPA's responsibilities in the AMS  Center are based on the requirements stated in the
"Environmental Technology Verification Program Quality Management Plan" (ETV QMP)2.
The roles of specific EPA staff are as follows:

       Ms. Michelle Henderson is EPA's AMS Center Quality Manager. For the verification
test, Ms. Henderson will:
       •  Review the draft test/QA plan
       •  Perform at her option an external technical systems audit during the verification test
       •  Notify the EPA AMS Center Project Officer of the need for a stop work order if the
          external audit indicates that data quality is being compromised
       •  Prepare and distribute an assessment report summarizing results of the external audit
       •  Perform at her option audits of data quality

-------
                                                                Coliform Detection Technologies
                                                                                Test/QA Plan
                                                                                Page 14 of 14
                                                                                 Version 1.0
                                                                              Date: 7/14/2010
       •  Notify the EPA AMS Center Project Officer of the need for a stop or modify work
          order if the audit of data quality indicates that data quality is being compromised
       •  Review draft verification reports and verification statements.

       Dr. John McKernan is EPA's Project Officer for the AMS Center. Dr. McKernan or
designee will:
       •  Review the draft test/QA plan
       •  Approve the final test/QA plan
       •  Be available during the verification test to authorize any test/QA plan deviations by
          phone and provide the name of a delegate to the Battelle AMS Center Manager
          should he not be available during the testing period
       •  Review the draft verification reports and verification statements
       •  Oversee the EPA review process for the test/QA plan, verification reports, and
          verification statements
       •  Coordinate the submission of verification reports and verification statements for final
          EPA approval
       •  Post the test/QA plan, verification reports, and verification statements on the ETV
          web site.

A5.4   Verification Test Stakeholders

       This test/QA plan and the verification report(s) and verification statement(s) based on
testing described in this document will be reviewed by experts in the fields related to coliform
detection. The following experts have been providing input to this test/QA plan and have  agreed
to provide a peer review:
       •  Rick Sakaji, East Bay Municipal Water District
       •  Jim Sinclair, Sandhya Parshionikar, Jennifer Best, and Keya Sen, EPA Office of
          Water

-------
                                                               Coliform Detection Technologies
                                                                              Test/QA Plan
                                                                              Page 15 of 15
                                                                                Version 1.0
                                                                            Date: 7/14/2010
The responsibilities of verification test stakeholders (or expert peer reviewers) include:
       •  Participate in technical panel discussions (when available) to provide input to the test
          design
       •  Review and provide input to the test/QA plan
       •  Review and provide input to the verification report(s)/verification statement(s).

In addition, the general approach for verification testing was reviewed with the broader AMS
Center Stakeholder Committee as a presentation during regular stakeholder teleconferences,
including the November 12, 2009 meeting, and input from the committee was solicited.

A5.5   Cooperating ETV Programs

       This test/QA plan will be reviewed by the cooperating ETV programs to confirm that the
experimental design meets the requirements for possible verification in those countries.  In
addition, the verification report(s) based on testing described in this document will be reviewed
to determine if verification is applicable.  The following people from the cooperating ETV
programs will  review these documents and provide input
       •  John Neate and Mona El Hallak, ETV Canada
       •  Mette Tjener Andersson, Claus J0rgensen, and Christian Gran, DANETV

Representatives of the cooperating ETV programs will:
       •  Participate in technical discussions during development of the test/QA plan to provide
          input to the test design
       •  Review and provide input to the test/QA plan
       •  Review and provide input to the verification report(s)
       •  Draft the verification statements for their respective program if results are determined
          to be acceptable.

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 16 of 16
                                                                                Version 1.0
                                                                             Date: 7/14/2010
A6    BACKGROUND
A6.1   Technology Need
       The ETV Program's AMS Center conducts third-party performance testing of
commercially available technologies that detect or monitor natural species or contaminants in air,
water, and soil. The purpose of ETV is to provide objective and quality assured performance
data on environmental technologies, so that users, developers, regulators, and consultants can
make informed decisions about purchasing and applying these technologies. The ETV Water
Stakeholder Committee, made up of buyers and users of such technologies recommend
technology categories, and technologies within those categories, as priorities for testing.  Among
the technology categories recommended for testing are coliform detection technologies.  In
particular, the use of coliform detection technologies for the monitoring of drinking water (DW)
was identified as an area of interest for technology verification.  It is important to note that this
verification test is independent of any activities related to approval of technologies for use in
meeting regulatory requirements for the detection of total coliforms (TC) or Escherichia coll (E.
coli).
       Fecal pollution can introduce disease-causing (pathogenic) bacteria, viruses, and parasites
into receiving waters, which may serve as private/public DW supplies.  Through the
consumption of these waters, enteric and other pathogens can be transmitted to humans, resulting
in gastrointestinal illness. Utilities fully recognize the possibility of this waterborne pollution
and take every precaution (filtering,  treatment with disinfectants such as chlorine and
chloramines, and regulatory compliance sampling and analysis) to avoid serving fecally polluted
water to consumers. Based on the 1989 Total Coliform Rule (TCR), assessment of this health
risk is based on the detection and enumeration of fecal indicator bacteria, such as TC and E. coli,
whose presence indicates that the water may be contaminated with sewage or animal wastes. In
this study, TC are defined as all facultative anaerobic, gram-negative, non-spore forming, rod-
shaped bacteria that ferment lactose  with gas and acid formation within 48 hours (h) at 35°C, and
that develop red colonies with  a metallic (golden) sheen within 24 h at 35°C on a Endo-type
medium containing lactose.3 E. coli  is defined as a species of coliform bacteria that possesses the
enzyme p-glucuronidase and is capable of cleaving the fluorogenic substrate 4-
methyllumbelliferyl-p-D-glucorinide (MUG) with the release of fluorogen when grown in a

-------
                                                               Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 17 of 17
                                                                                Version 1.0
                                                                             Date: 7/14/2010
MUG-based medium at 44.5°C within 24 ± 2 h or less, producing a blue fluorescence within the
media and/or around the periphery of the colony.3
       The 1989 TCR sets both goals and legal limits for the presence of TC in DW.  The rule
also details the type and frequency of testing that all public water systems must implement. The
Maximum Contaminant Level Goal (MCLG1) is no detectable TC in DW.  The Maximum
Contaminant Level (MCL11) is no more than 5% of samples positive for TC in a month. For
water systems that collect fewer than 40 routine  samples per month, no more than one sample
can be total coliform-positive per month. If a sample is positive for TC, each total coliform-
positive routine sample must be tested for the presence of fecal coliforms (fecal coliforms will be
removed from the revised TCR) or E. coli, and repeat  samples must be collected and analyzed
for TC.  As mentioned above, this verification test is not being conducted to provide data that
will be used to approve technologies for use in meeting regulatory requirements for the detection
of total coliform or E.  coll as required by either the 1989 TCR or the upcoming revision to the
TCR. It is being conducted, based on feedback from ETV AMS  Center stakeholders, to provide
a verification test that is similar in requirements to the current TCR approval process, such that
technologies that are not already approved have an opportunity to be tested under similar set of
test conditions.
       Similar to the TCR in Europe, the Official Journal of the European  Communities
published Council Directive 98/83/EC in 1998 that provided directives on the regulation of
public water systems within the European Union. Annex II and III within that document
includes the standard of zero E. coli per 100 mL and suggests the International Standards
Organization Method  9308 as the preferred method for regulatory E.coli analysis.  More
recently, the Colilert-18 method has been shown to be equivalent to the suggested method and
therefore will be used  to meet the DANETV requirement of European acceptance.
1 Maximum Contaminant Level Goal (MCLG) - Level of a contaminant in drinking water below which there is no
known or expected risk to health. MCLGs allow for a margin of safety and are non-enforceable public health goals.
"Maximum Contaminant Level (MCL) - The highest level of a contaminant that is allowed in drinking water. MCLs
are set as close to MCLGs as feasible using the best available treatment technology and taking cost into
consideration. MCLs are enforceable standards.

-------
                                                               Coliform Detection Technologies
                                                                              Test/QA Plan
                                                                              Page 18 of 18
                                                                                Version 1.0
                                                                            Date: 7/14/2010
A6.2   Technology Description
       The coliform detection technologies to be tested use chromatogenic and fluorogenic
growth media to detect coliforms and E. coli based on the enzymatic activity of these organisms.
The systems consist of single-use sample containers that contain pre-measured reagents and can
be used in conjunction with a bench top incubator/analyzer/data logger system.  In short, a 100
mL water sample is added to the containers and then incubated in and analyzed by the bench top
instrument. In order to accommodate the large number of samples required for this test, the
containers can be incubated in an external incubator, rather than in the bench top instrument, and
then returned to the instrument for analysis.  The enzymes produced by TC and E. coli cleave the
chromatogenic or fluorogenic substrates in the growth media, resulting in the release of colored
or fluorescent products.  The bench top instruments include optical sensors and/or
spectrophotometers that take readings of the sample at regular intervals to quantify the bacteria.
The sample data and instrument calibration are stored on the instrument's internal computer until
it is retrieved with a flash drive or some other portable data transfer device.
       If no target bacteria are present in the sample, the sample color remains unchanged or
fluorescence is not produced. If the target organisms are present, there is a progressive change in
the color or fluorescence produced in the sample.  For some technologies, results may be
available within 60 minutes after sample collection/test initiation, depending on the number of
bacteria present in the sample. However, the typical analyses are not completed until after 14 to
24 hours  of incubation.

A7    VERIFICATION TEST DESCRIPTION AND SCHEDULE

       This verification test will assess the performance of the coliform detection technologies
relative to key verification parameters, including specificity and sensitivity. In performing the
verification test, Battelle will follow the technical and QA procedures specified in this test/QA
plan and will comply with the data quality requirements in the AMS Center QMP (refer to Table
11 in Section B).

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 19 of 19
                                                                                Version 1.0
                                                                             Date: 7/14/2010
A7.1   Verification Test Description
       The objective of this verification test is to evaluate the technologies' performance
detecting the presence or absence of TC and E. coli against spiked concentrations of target
organism in DW. This verification test will take place at Battelle in Columbus, Ohio.
Technologies undergoing verification will be used to analyze spiked DW samples for TC and E.
coli. The presence/absence results from the technologies will be compared to the results from
the reference methods,  Standard Methods (SM) 922IB and F3. These reference methods are
recognized in the U.S. and Canada as the standard for regulatory purposes. Each technology in
the test will be considered for possible verification in Canada.  For those vendors that select the
optional verification in Denmark, their results will also be compared with Colilert  -18 as this
product has recently been shown to be equivalent to the European Union approved method and
therefore will be used to meet the DANETV requirement of European acceptance.
       Subsequent to the verification test, verification reports describing the test will  be drafted.
These reports will be reviewed by the vendor and by peer reviewers, revised, and submitted to
EPA for final approval.  In performing the verification test, Battelle will follow the technical and
QA procedures specified in this test/QA plan and will comply with the data quality requirements
in the AMS Center QMP. As a point of clarification, an ETV verification does not imply EPA
approval for use to meet regulatory requirements by current or future regulations (such as the
TCR).

A7.2   Proposed Verification Test Schedule

       Table 1  shows the proposed schedule of testing, auditing, and data analysis/reporting
activities to be conducted during this verification.  The verification of coliform detection
technologies is planned to be completed over the course of approximately three weeks after the
start of laboratory testing.  The verification test is expected to be performed in July-August 2010.
The technical systems audit (TSA) will take place during testing; the data quality audit (DQAs)
will take place after the data are reviewed by the Test Coordinator, or designee.  A daily
overview of the technical activities to be performed during the verification test is presented in
Section B 1.2.

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 20 of 20
                                                                                Version 1.0
                                                                             Date: 7/14/2010
Table 1. Proposed Verification Test Schedule
Approximate Date(s)
January - June 2010
July and August 20 10
August 20 10
August 20 10
September 20 10
Testing Activities
Test/QA plan design and approval
Verification testing
Perform Technical Systems Audit
Audit of data (1st batch)
Reference analysis
Audit of data (remaining data)
Prepare draft verification reports and
statements
Coordinate reviews of draft verification
reports and statements
Prepare final verification reports and
statements
Data Analysis and Reporting
Not applicable
Compile data from technologies
Review and summarize testing staff
observations
Compile reference method results
Perform data analysis
Begin draft reports
Complete draft verification reports and
statements
Complete peer review and vendor review
of draft reports
Revise draft verification reports and
statements
Submit final reports for EPA approval1
 Indicates EPA approval of ETV reports for release and does not imply approval for regulatory use
A8    QUALITY OBJECTIVES AND CRITERIA FOR MEASUREMENT DATA

       In performing the verification test, Battelle will follow the technical and QA procedures
specified in this test/QA plan and will comply with the data quality requirements in the AMS
Center QMP. QA Category II, Highly Visible Project, has been specified for this test by the
EPA Project Officer.
       To ensure that this verification test provides suitable data for a robust evaluation of
performance, a variety of quality control (QC) samples will be incorporated for this test.  Their
acceptance criteria are defined in Section B5.  The acceptance criteria for the QC samples
indicate the minimum required performance to meet the objectives of the verification test and are
used as the QC criteria for laboratory measurements.  The QC samples and acceptance criteria
for this verification test were established to assess the performance of the coliform detection
technologies relative to the reference method.  In order to provide a suitable benchmark for
comparison, the reference measurements must meet the minimum requirements of the QC
sample acceptance criteria.
       Method and reagent blank samples will be required to demonstrate freedom from
contamination, and positive and negative control cultures (American Type Culture Collection

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 21 of 21
                                                                                Version 1.0
                                                                             Date: 7/14/2010
[ATCC]) will be processed to demonstrate the laboratories can accurately identify the target
organism and that the media and confirmation testing are providing appropriate results. ATCC
strains shown in Table 2, which are commonly used for water quality testing, will be acquired
from MicroBioLogics®. A Certificate of Analysis is provided with each strain that demonstrates
culture purity based on macroscopic, microscopic, and biochemical (phenotypic) testing.

Table 2. Quality Control Strains
Targeted Coliform
Total Coliform
Fecal Coliform
Positive Control
Enterobacter aerogenes
ATCC 13048
Escherichia coll
ATCC 8739
Negative Control
Pseudomonas aeruginosa
ATCC 10145
Enterobacter aerogenes
ATCC 13048
       The Battelle Quality Manager or designee will perform a technical systems audit (TSA)
at least once during this verification test to augment these QC criteria. The EPA Quality
Manager also may conduct an independent TSA, at her discretion.

A9    SPECIAL TRAINING/CERTIFICATION

       Documentation of training related to technology testing, data analysis, and reporting is
maintained for all Battelle technical staff in training files at their respective Battelle location.
Prior to the start of verification testing, the Battelle Quality Manager will verify the presence of
applicable training records, such as documentation, chain-of-custody, reference and technology
test methods, and equipment training. The vendors will be required to train the Battelle technical
staff prior to the start of testing. Battelle will document this training with a consent form, signed
and dated by the vendor, which states which Battelle technical staff have been trained to use their
technology and can train other staff. In the event that other staff members are required to use the
technologies, they will be trained by the operators that were trained by the vendors. All technical
staff will have a minimum of a bachelor's degree in science or equivalent work experience (e.g.,
experience performing bacterial analysis).

-------
                                                                 Coliform Detection Technologies
                                                                                Test/QA Plan
                                                                                Page 22 of 22
                                                                                 Version 1.0
                                                                              Date: 7/14/2010
       Battelle will conduct all verification testing following the biosafety guidelines4
established at the Battelle laboratories (Battelle Building 20 Biosafety Manual, Version 4.0).
Exposure to biological organisms will be minimized, personal protective equipment will be
worn, and safe laboratory practices will be followed.

A10   DOCUMENTATION AND RECORDS

       The documents for this verification test will include the test/QA plan, vendor instructions,
reference methods, verification reports, verification statements, and audit reports.  The project
records will include laboratory record books (LRB) and data collection forms, supporting
laboratory records, training records, electronic files (both raw data and spreadsheets), and QA
audit files.  Table 3 summarizes the types of data to be recorded.  Documentation of Battelle staff
training by vendors and copies of other project specific training will also be included  in the
project files.  All of these records will be maintained in the Verification Test Coordinator's office
during the test and will be transferred to permanent storage at Battelle's Records Management
Office (RMO) at the conclusion of the verification test.  All Battelle LRBs are stored  indefinitely
with the project files by Battelle's RMO. Section BIO further details the data management
practices and responsibilities.
       All data generated during the conduct of this project will be recorded directly, promptly,
and legibly in ink. All data entries will be dated  on the date of entry and signed or initialed by
the person entering the data. Any changes in entries will be made so as not to obscure the
original entry, will be dated and signed or initialed at the time of the change and shall indicate
the reason for the change. Project-specific data forms will be developed prior to testing to ensure
that all critical information is documented in real time. The draft forms will be provided to the
Battelle QA Manager for review.

-------
                                                                  Coliform Detection Technologies
                                                                                  Test/QA Plan
                                                                                  Page 23 of 23
                                                                                   Version 1.0
                                                                                Date: 7/14/2010
Table 3. Summary of Data Recording Process
Data to Be
Recorded
Dates, times of test
events
Test parameters
Sample collection
data
Reference sample
data
Reference method
sample analysis,
chain of custody, and
results
Responsible
Party
Battelle
Battelle
Battelle
Battelle
Superior
Laboratories, Inc.
Where
Recorded
Laboratory record
books or data
collection sheets
Laboratory record
books or data
collection sheets
Laboratory record
books or data
collection sheets
Laboratory record
books or data
collection sheets
Laboratory record
books, chain of
custody forms,
data collection
sheets, or data
acquisition system,
as appropriate
How often
recorded
Start/end of test, and
at each change of a
test parameter
When set or changed,
or as needed to
document test
notable details during
testing
During each
sampling event
When test samples
are aliquotted for the
reference analysis
Throughout sample
handling and analysis
process
Disposition of
Data
Used to organize/check
test results; manually
incorporated in data
spreadsheets as
necessary
Used to organize/check
test results; manually
incorporated in data
spreadsheets as
necessary
Used to document
collection, handling,
and storage of sewage
and drinking water
samples
Used to organize/check
test results; manually
incorporated in data
spreadsheets as
necessary
Transferred to
spreadsheets/agreed
upon report; project
files. Retained for
documentation of
reference method
performance

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 24 of 24
                                                                                Version 1.0
                                                                             Date: 7/14/2010
                                      SECTION B
                     MEASUREMENT AND DATA ACQUISITION

Bl     EXPERIMENTAL DESIGN

       Technologies undergoing verification will be used to analyze spiked DW samples for TC
and E. coli.  Technology operation and sample handling and analysis will be performed
according to the vendor's instructions. The results from the coliform detection technologies will
be evaluated by comparing the proportion of positive results to the proportion of positive results
produced by SM9221B and F which includes the comparison of false positive rate (or
specificity) and false negative rate (or sensitivity. In addition, operational factors such as ease of
use, required reagents, analysis time, and laboratory space and utilities required will be
documented and reported.

Bl.l   Verification Test Sample Preparation

       The preparation of verification test samples includes the collection of raw sewage as the
source of the target organisms, collection of the DW sample, the fortification of the DW sample
with target organisms, and the chlorine stressing and dilution of samples for analysis. The
sample preparation steps and timeline are illustrated in Table 4 and described in detail in the
sections B1.1.1 through B1.1.3.

-------
                                                                     Coliform Detection Technologies
                                                                                      Test/QA Plan
                                                                                      Page 25 of 25
                                                                                        Version 1.0
                                                                                    Date: 7/14/2010
Table 4. Sample Preparation Timeline
Day
Process
Battelle Key Activities
         Collect Sewage
                          Collect sewage in early AM from wastewater treatment plant
                          Remove excess solids by filtration
                          Characterize sewage
                              Enumerate total coliform (TC) and E. coli via Standard Method (SM)
                              9222B/G
                              Enumerate total heterotrophic bacteria via standard spread plating
                              (R2A agar)
                          Measure pH (SOP GEN.V-003-10), free and total chlorine (HACH
                          Method 8021 & 8167)	
         Collect Drinking Water
                          Collect approximately 50 L drinking water from tap in Battelle laboratory
                          Measure pH (SOP GEN.V-003-10), free and total chlorine (HACH
                          Method 8021 & 8167)
                          Adjust 30 L DW to 2.0-2.5 mg/L free chlorine for chlorine stressing
         Chlorine Stressing
         (stress bacteria; reduce
         TC/£. coli by 2-4
         orders of magnitude)
         (prepare spiked,
         stressed TC or E. coli
         DW)	
                          Spike three 10 L DW samples with 200 mL filtered sewage (targeting 10 "
                          org./lOO mL for TC or E. coli)
                          Dechlorinate after 2.5, 5, and 10 minutes
                          Enumerate each spiked, stressed, DW sample (SSDW)
                          Refrigerate SSDW
         Prepare dilutions for
         sample analysis
                          Based on Day 1 enumeration results, prepare three 10 L dilutions of
                          SSDW targeting 0.1, 1, and 10 TCorE1. coli/WO mL from one of the three
                          SSDW 10 L aliquots	
         Complete Sample
         Preparation; Initiate
         Verification Testing
                          Dispense 20 replicate 100 mL aliquots of each dilution into sterile
                          reservoirs (total of 60 samples plus 1 negative matrix control, 1 negative
                          TC control,, 1 positive TC/negative E. coli, and 1 positive E. coli for a
                          total of 64 samples per method)
                          Promptly transfer required samples to the reference laboratory3 on wet ice
                          to begin Reference testing (SM 9221B/F) and Colilert®-18 testing
                          Initiate vendor analysis at Battelle
  Superior Laboratories, Inc., 1075 Amity Road, Galloway, OH 43119-9304, (614- 793-2172).  Attn: James Gossard
B1.1.1  Sewage and Drinking Water Sample Collection
        A single raw sewage sample (approximately 1 L), will be collected in the morning hours

from the influent sewage line entering the Southerly Wastewater Treatment Plant (SWTP) in

Columbus, OH by SWTP staff.  The sewage will be immediately stored on wet ice, and

transported by Battelle staff to Battelle laboratories. Upon receipt, the sewage will be filtered

through a Whatman No. 2 pleated filter under vacuum using a Buchner funnel  to remove excess

solids, shaken vigorously for 1 minute to insure homogeneity, and then immediately

characterized as follows:

           •   Measure pH using calibrated pH meter (SOP GEN.V-003-10)5

-------
                                                               Coliform Detection Technologies
                                                                              Test/QA Plan
                                                                              Page 26 of 26
                                                                               Version 1.0
                                                                            Date: 7/14/2010
          •   Measure free chlorine and total chlorine using a HACK Chlorine test kit [(HACK
              Method 8021 (free) and Method 8167 (total); procedures are equivalent to U.S.
              EPA Standard Method 4500-CL G for drinking water]
          •   Quantify the total, culturable heterotrophic bacteria by enumerating onto R2A
              agar using a standard spread plating method as described in the AOAC's
              Bacteriological Analytical Manual6.
          •   Quantify TC and E. coli (organisms per 100 mL) using quantitative methods for
              TC (SM 9222B - m-Endo) and E. coli (SM 9222G - NA-MUG)3.
                 -   Based on recent analysis of sewage samples collected from SWTF, the
                     concentration of total coliform and E. coli is expected to be in the range of
                     106"7 organisms/100 mL.
                 -   To obtain readable filters, serial dilutions of the sewage will be prepared
                 -   Total coliform concentrations will be based on the number of typical
                     coliform colonies (pink to dark-red with metallic surface sheen) observed
                     on m-Endo.
                 -   To determine E. coli concentrations, the filter membranes will be
                     transferred from m-Endo onto NA-MUG, incubated for 4 hours at
                     35±0.5°C, and then held under a long-wave length ultraviolet light.
                     Colonies that fluoresce blue are counted as E. coli.

       A single DW sample (approximately 62 L) will be collected into multiple, sterile carboys
the same day the sewage sample is collected.  This volume will accommodate the preparation of
a complete set of 100 mL aliquots for one round of verification testing for TC and E. coli.  Table
5 shows that approximately 62 L of DW will be required to prepare three 10.5 L volumes for
chlorine stressing (32 L) and to prepare the three dilutions for sample analysis from one of the
chlorine stressed samples (30 L). These diluted samples will be the source of the  100 mL
samples that will be analyzed by the technologies being tested as well as the reference methods.
       The DW sample will be collected as follows:
       (1) Remove faucet screen if present, and surface decontaminate with 70% isopropanol.
       (2) Purge line for 2 to 3 minutes with cold water.

-------
                                                               Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 27 of 27
                                                                                Version 1.0
                                                                             Date: 7/14/2010
       (3) Collect approximately 62 L DW from the tap into multiple sterile (autoclaved)
          carboys equipped with a spigot and containing large stir bars.
       Once collected, a 100 mL aliquot will be collected from the carboy's spigot and used to
characterize the DW by measuring the pH (SOP GEN.V-003-10) and the concentration (mg/L)
of free and total chlorine using a HACK Chlorine kit (HACK Methods 8021 and 8167). Since
recent chlorine tests of the tap water indicate the water generally contains 0.2 to 1.5 mg/L free
chlorine, sodium thiosulfate will not be added prior to the chlorination step. The DW sample
will be used the same day as collected for the chlorine stressing and stored at 2°C to 8°C and
used the following day to prepare the diluted samples for analysis. DW used to dilute the stress
samples will be dechlorinated prior to use.
Table 5. Volume of Drinking Water Required for Verification Testing
Step
Chlorine Stressing
Dilution and Sample
Preparation
Description
Prepare three 10.5 L spiked aliquots for
three timed exposures
Prepare three 10.5 L dilutions from one
of the 10.5 L Chlorinated aliquots
above ; 2 L required for each method
(60 x lOOmL);
-control preparation
-replicate 100 ml samples will be
generated from these three dilutions
Total=
DW Volume Needed
32L
Approx. 30 L
(Volume required for prep, of
three dilutions depends on
concentration of chlorinated
sample selected for sample
prep.)
62L
Bl.1.2 Chlorine Stressing and Preparation of Samples for Verification Testing

       The goal of the chlorine stressing step is to reduce the number organisms in the spiked
DW samples by two to four orders of magnitude,  starting at approximately 105 target
organisms/100 mL. In addition, this step also stresses the organisms with chlorine. Three spiked
DW samples will be chlorinated for 2.5, 5, or 10 minutes, after which time they will be
dechlorinated with sodium thiosulfate and subsequently enumerated to determine the reduction
of target organisms in each spiked,  stressed drinking water (SSDW) sample.  Chlorine stressing
will be initiated the day the sewage and DW samples are  collected in order to minimize the
storage time between sewage and DW collection and the  start of sample analysis. Following

-------
                                                               Coliform Detection Technologies
                                                                              Test/QA Plan
                                                                              Page 28 of 28
                                                                                Version 1.0
                                                                            Date: 7/14/2010
chlorine stressing, one of the three 10 L dilutions of SSDW will be selected to prepare a set of
three 10 L dilutions prepared into dechlorinated drinking water targeting a range of 0.1  to 10
target organisms/100 mL.  The chlorine stressing and sample preparation steps are detailed
below:
       Chlorine Stressing Procedure:
          (1) Dispense 10.5 L of the original DW sample into each of three, sterile 10 L
              carboys containing stir bars. Place carboys on stir plates and mix at medium
              speed.
          (2) Measure the total residual and free residual chlorine concentration using a HACK
              Chlorine test kit (approved for SM 4500-C1- G3) in each carboy. If necessary,
              adjust the free residual chlorine level to 2.0-2.5 mg/L using a 4% solution (40,000
              mg/L) of sodium hypochlorite.  Since the spiked DW sample will generally have a
              low chlorine demand, overstressing or killing the organisms by prolonged
              exposure to free chlorine must be avoided.  The total residual chlorine level
              should not exceed 2.5 mg/L.
          (3) Measure pH by collecting a 10 mL aliquot from each carboy and measure using a
              calibrated pH meter, following SOP GEN.V-003-105.
          (4) Measure temperature of water in each carboy using a calibrated thermometer.
              Thermometers are calibrated according to SOP GEN.V-013-047.  [Surface
              decontaminate thermometer with  70% isopropanol before each use].
          (5) Start chlorine stressing by spiking each 10.5 L DW sample with 200 mL filtered
              sewage to achieve a concentration of approximately 105 target organisms/100 mL.
              The concentration of TC or E. coli will be assumed to be 104-105 org./mL (as
              indicated by analysis of sewage collected over the past several weeks leading up
              to the verification study).
          (6) At each of three time points (2.5,  5, and 10 minutes after spiking with sewage):
                 a.  Add 8.6 mL 10% (w/v) sodium thiosulfate solution (i.e., 0.8 mL  per L) to
                    stop chlorine oxidation (dechlorinate). Allow to mix for at least 1 minute.
                 b.  Measure pH and temperature of sample.
                 c.  Measure free residual chlorine and total chlorine.

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 29 of 29
                                                                                Version 1.0
                                                                             Date: 7/14/2010
          (7) Enumerate the target organism densities of each SSDW via SM 9222B for TC and
              SM9222Gfor£. coli. .
                 a.  Enumerate triplicate 100 mL aliquots of undiluted sample, and triplicate
                     100 mL aliquots of 10"1 and 10"2 dilutions of the sample; using phosphate
                     buffered water as the diluent.
                 b.  Calculate TC and E. coli/100 mL in each 10 L sample.
          (8) Calculate the reduction of TC and E. coli based on the estimated spiking (starting)
              concentration and the concentrations after each exposure time.

       If none of the exposure times used was adequate to achieve the desired reduction of two
to four orders of magnitude, the entire process will be repeated using a freshly collected sewage
sample and DW. The measured density of chlorine-stressed target organisms confirms that a two
to four order of magnitude reduction has been attained, and can then be used to prepare three
dilutions that target a density of 0.1 to 10 target organisms/100 mL. While it is difficult to
determine if a single target organism is present in 100 mL of water, when approximately half of
the analyzed replicates are positive and half are negative, the density of the organism has become
adequately low so that a positive result can be considered single organism  detection.  Therefore,
the ultimate objective is to have one of the prepared dilution sets provide 50  ±25% positive
results for TC or E.  coli (depending on which is being targeted) with the reference method(s).
       Based on the results of the chlorine stressing step, three 10.5 L volumes of DW with TC
or E.coli concentrations of 0.1 org./lOO mL, 1 org./lOO mL, and 10 org./lOO  mL, respectively,
will be prepared, and replicate 100 mL aliquots of each dilution will be subsampled for
verification testing.  Preparation of the dilution samples will be performed as follows:
       (1) Calculate the volume of the original DW needed to prepare 10.5 L of each of three
          dilutions, and dispense this volume into one or more carboys. Measure the free
          chlorine concentration, and dechlorinate if the free chlorine concentration is in the
          range of 0.2 - 0.5 mg/L by adding 0.8 mL 10% sodium thiosulfate per 1 L. Use this
          dechlorinated DW (DDW) to prepare the dilutions.
       (2)  Accurately dispense the appropriate volume of DDW to each of three sterile carboys
          having spigots and containing large stir bars.

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 30 of 30
                                                                                Version 1.0
                                                                             Date: 7/14/2010
       (3) Spike the DDW with the appropriate volume of SSDW sample to each of the three
          carboys to generate suspensions of 0.1 org./lOO mL, 1 org./lOO mL, and 10 org./lOO
          mL.
       (4) Each dilution will be treated as follows:
              a.  Mix on stir plate for 5-10 minutes.
              b.  During continuous mixing on the stir plate, dispense 100 mL aliquots into
                 sterile 100 mL bottles using 50 mL and/or 100 mL graduated pipets. All
                 bottles will be pre-labeled with a unique sample ID.
              c.  Store bottles at 2 °C to 8 °C as they are generated.
              d.  Immediately after all samples have been dispensed for the reference methods,
                 transport them in coolers packed with ice packs to Superior Laboratories, Inc.,
                 which is approximately a 15 to 20 minute drive from the sample preparation
                 laboratory. Reference method analysis following SM9221B/F and Colilert -
                 18 will  commence the same day  as arrival at the laboratory.
Once all 100 mL aliquots are dispensed for technology verification (20 at each dilution level for
a total of 60 replicates per technology), verification testing will begin. Figure 2 illustrates the
experimental design for the sample dilution. Not shown in Figure 2 are the QC samples which
will be prepared and analyzed at the same time.

-------
                                                                                                      Coliform Detection Technologies
                                                                                                                    Test/QA Plan
                                                                                                                    Page 31 of 31
                                                                                                                      Version 1.0
                                                                                                                  Date: 7/14/2010
  Reference
   Methods
                            Dilution 1
                        (0.1 org./100ml_)
                       20 replicates SM9221B
                               (LIB)
                            20 replicates
                             Colilert-18
                                                      Spiked, Stressed
                                                           Sample
                                                       (TC or E. coll)
                                   Dilution 2
                                   org./100mL)
                               20 replicates SM 9221B
                                      (LIB)
                                   20 replicates
                                    Colilert-18
                                    Dilutions
                                 (10org./100ml_)
                               20 replicates SM 9221B
                                       (LIB)
                                    20 replicates
                                     Colilert-18
                          TC&
                         E co//
                         E co//
                          only
Technologies
Being Verified
     20 replicates
Each Verified Technology
     20 replicates
Each Verified Technology
     20 replicates
Each Verified Technology
TC&
E co//
              Figure 2.  Experimental Design for Sample Dilution and Analysis

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 32 of 32
                                                                                 Version 1.0
                                                                             Date: 7/14/2010
B1.2   Sample Analysis

       The number of replicates to be analyzed by each technology and reference method was
selected following a power analysis based on a Pearson chi-square test for equality of
proportions associated with two independent populations, where the test is two-sided (i.e., test
for equality versus any type of non-equality). The populations being compared are the two test
types (reference and technologies being verified), and the proportion corresponds to the
proportion of positive results out of the total tests performed.  Conducted using the POWER
procedure in the SAS System, the power analysis determined the number of replicate tests
(across both test types) that is necessary to detect a specified difference in proportions of a
specified size with 80%  power, given a specified value of the proportion for the reference test
(the acceptable range of reference test positive proportions will be 0.25 to 0.75 for this test), and
a significance level of 0.05 for the test. Table 6 gives the power analysis results for reference
method positive proportions of 0.25, 0.5, and 0.65. The results indicate that approximately 20
replicate samples  of the technologies being tested will be adequate to determine significant
differences in the  proportion of positive results generated by the technologies being tested and
the reference method. The absolute differences in proportion relative to the reference test that is
shown through this power analysis to be detectable as significantly difference were 0.45, 0.4, and
0.325 (shaded results) for the three reference method positive proportions of 0.25, 0.50,  and 0.65,
respectively.  Differences in positive proportion less than those may be detectable, but not with
80 percent power under the assumptions of the test.
       As shown in Table 6, the use of approximately 20 replicates provides a likelihood that a
significant difference can be determined between the reference method and the technologies
being tested.  If a  significant difference is determined based on the results of this ETV test, the
result means that the reference method and technology being tested are at least different by the
shaded ratios shown in the table. Upon discussion with the peer reviewers for this verification
test, it was determined that use of 20 replicates would provide data that was useful to the end
user of these technologies.  The reviewers determined that, because this test does not have
regulatory implications,  the result will be valuable as a screening tool to help with purchase
decisions.

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 33 of 33
                                                                                 Version 1.0
                                                                             Date: 7/14/2010
Table 6. Power Analysis Results for Determination of Replicates
Reference
Method Ratio of
Positive Results
0.25
0.50
0.65
Reference
Method
N
71
152
1,251
1,251
152
58
31
19
15
20
39
93
388
388
93
39
20
22
39
189
804
189
39
22
Tested
Technology
Ratio of
Positive
Results
0.025
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.05
0.10
0.20
0.30
0.40
0.60
0.70
0.80
0.90
0.325
0.390
0.520
0.715
0.780
0.910
0.975
Difference in
Positive
Proportion
Between
Reference and
Technology
0.225
0.15
0.05
0.05
0.15
0.25
0.35
0.45
0.45
0.4
0.3
0.2
0.1
0.1
0.2
0.3
0.4
0.325
0.26
0.13
0.065
0.13
0.26
0.325
Required Tested
Technology N
for Significant
Difference
71
152
1,251
1,251
152
58
31
19
15
20
39
93
388
388
93
39
20
22
39
189
804
189
39
22
Test
Method
Positives
2
15
251
375
61
29
19
13
1
2
8
28
233
155
28
31
18
7
15
98
575
147
36
21
Test
Method
Negatives
69
137
1,001
876
91
29
12
6
15
18
31
65
155
233
65
8
2
15
24
91
229
42
o
J
1
       The ability of each coliform detection technology to determine the presence of TC and E.
coli will be challenged using spiked stressed DW samples.  Positive/negative control samples
spiked with quality control cultures listed in Table 2 will be included during testing. The
technologies will be operated according to the manufacturer's instructions and vendor will
provide the necessary equipment and supplies to analyze up to sixty 100 mL samples at a given
time.  The dilution samples generated in Section Bl.l will be assayed by the reference methods
and the technology methods concurrently.  The reference analyses will be conducted as described

-------
                                                                   Coliform Detection Technologies
                                                                                    Test/QA Plan
                                                                                    Page 34 of 34
                                                                                     Version 1.0
                                                                                 Date: 7/14/2010

in Section B4. Table 7, which is continued from Table 4, presents an overview of the daily

verification test activities during days 3 through 7. The details of the activities are described in

sections B 1.2.1 and B 1.2.2.
Table 7.  Sample Analysis Timeline
      Day
                                                Battelle and Superior Key Activities
     9221B
   (Superior)
  9221F
(Superior)
Colilert®18
 (Superior)
  Vendor
Technologies
 (Superior)
  Vendor
Technologies
  (Battelle)
1
Sewage and DW collection, and chlorination (refer to Table 4)
2 (Dilutions
already prepared)
                 100 mL SSDW into LTB
                                Start analysis on
                                replicate 100 mL
                                samples
                                           Replicate 100 mL
                                           samples into
                                           vendor supplied
                                           containers; start
                                           analysis
                 24h LTB into
                 BGLB /EC-MUG
                 24 h LTB into
                 EC-MUG
             Complete tests
              Receive
              Vendor
              analyzed
              samples; start
              Confirmed
              phase by
              inoculating
              LTB
             Vendor Results;
             transport all
             samples to
             Superior Labs
                 48h LTB into
                 BGLB /EC-MUG
                 48 h LTB into
                 EC-MUG
                              24h LTB into
                              BGLB /EC-
                              MUG
                 Complete tests
                 from Day 3
                                                 48h LTB into
                                                 BGLB /EC-
                                                 MUG
                 Complete tests
                 from Day 4
                                                 Complete
                                                 tests from
                                                 Day 4
                 (Optional)
                 Additional
                 completed tests if
                 deemed
                 appropriate
                                                 Complete
                                                 tests from
                                                 Day 5
                                                 (Optional)
                                                 Additional
                                                 completed
                                                 tests if
                                                 deemed
                                                 appropriate
       The data used for verification will be from the dilution(s) which produces results closest

to an equal number of positive (50%) and negative (50%) results for the reference method. A 50

± 25% split in responses (in either direction; i.e., 25:75 or 75:25% split) will be sought.  For

comparability, the evaluated results for the coliform detection technologies will be from the same

dilution level as the 50 ± 25% positive reference method results. If none of the dilutions

-------
                                                               Coliform Detection Technologies
                                                                              Test/QA Plan
                                                                              Page 35 of 35
                                                                               Version 1.0
                                                                            Date: 7/14/2010
produces an acceptable split in positive and negative results for the reference method, the process
may be repeated starting with the collection of fresh sewage and DW. The results from the
dilution sets not used for the data evaluation will be documented, but the samples will be
discarded.  Confirmation tests will be conducted on the dilutions to be used for the data
evaluation.

B1.2.1 Confirmation of Results

       All sample results must be confirmed with more definitive tests in order to adequately
compare results between the reference method and the technologies being tested. Confirmation
for both the SM 922IB and F reference methods, as well as the technology methods, is described
below. Figure 3 illustrates the process by which all positive and negative samples from the
verification technologies and SM 922IB and F are confirmed. Although all  samples will be
processed through the confirmation phase, only the results from the dilution  of reference samples
that gives the 50 ± 25% positive result will be used to compare the technologies to the reference
methods.
       Confirmation of reference method (SM 922 IB  and F) results
           1)  All lauryl tryptose broth (LTB) bottles showing acidic reaction (distinct yellow
              color) within 24 ± 2 hours of incubation will be submitted for confirmatory
              testing according to SM 922IB using inoculation into brilliant green lactose bile
              (BGLB) broth. Negative LTB samples  will continue to incubate.
                 o   If acidic reaction in LTB  is displayed at the end of a 48 ± 3 hour
                     incubation period, these samples will also be  submitted to the confirmed
                     phase by simultaneous inoculation into BGLB and EC-MUG broth
                 o   The absence of acidic reaction in LTB at the end of 48 ± 3 hours of
                     incubation constitutes a negative test for TC and E. coli.
           2)  Formation of gas in the BGLB broth fermentation tube at anytime within a 48 ± 3
              hour incubation period constitutes a positive confirmed phase for TC.
           3)  After incubation in EC-MUG, the appearance of a blue fluorescence under long-
              wave (approximately 366 nm) UV light after 24 h incubation at 44.5°C constitutes
              a positive confirmed phase for E. coli.

-------
                                                               Coliform Detection Technologies
                                                                              Test/QA Plan
                                                                              Page 36 of 36
                                                                               Version 1.0
                                                                            Date: 7/14/2010
          4)  As additional, optional confirmation, a complete test for TC may be performed by
              inoculating MacConkey media and then selecting suspected TC colonies and
              inoculating into LIB, as described by SM 9221B.
       No confirmation testing is required for Colilert®-18. Positive Colilert®-18results for TC
are indicated by presence of yellow color equal to or greater than the Colilert®-18comparator
control after 18 h at 35°C.  The appearance of blue fluorescence when exposed to long-wave UV
light represents a positive reaction for E. coli.
       Confirmation of the samples analyzed by the vendor technologies will be performed as
follows:
       (1) Transport all vendor samples to Superior Labs after all vendor analyses have been
          performed.
       (2) .For each vendor samples following proper incubation time and result determination
          (regardless if positive or negative): inoculate  10 mL LTB into test tubes containing
          Durham tubes with 1 mL of vendor sample to initiate confirmation tests.
       (3)  Confirm results as illustrated in Figure 3 and following SM 922IB and SM9221F.

-------

-~-~ " —
f TC Dilution \ /^^ AH c /w,- "^X
/ _n ,_0/ \ f All E. co// X
V D ^ M % 7 V Dilutions J
\ Positive:Negative / ^^^^^~-'^^^f
^--^ ^--^

Inoculate
BGLB

Confirm TC
( \ Inoculat*
LTB-K) L^ eriunir ^
Replicates EC MUG
(SMS^IB) T .....
L J Test for £. co/i
	 i SM9"'"11F 	 > UV366
SM Reference (EC MUGI Fluorescence

Method
Technologies
Being Verified

rosmve I*) uontirm (.+.) results |nrtrui^iTR
Rpplicat^ False + Specificity ^ Inoculate LTB




"
Negative H _^ uonnrm (-) «esuns inoculate I TB
Rpplirates FsA« - .
-------
                                                                 Coliform Detection Technologies
                                                                                Test/QA Plan
                                                                                Page 38 of 38
                                                                                  Version 1.0
                                                                              Date: 7/14/2010
B1.3   Statistical Analysis

Bl.3.1 False Positive Rates, False Negative Rates, Sensitivity, and Specificity
       False positive (FP) and false negative (FN) rates of reference methods will be evaluated
when assessing comparability. During this test, true positives are those positive results from the
technologies being tested that are confirmed, and false positives are those positive results from
the technologies being tested that are not confirmed as directed by the reference method.
Conversely, true negative results are those negative results that are confirmed as negative, and
false negative results are those negative results that were shown to be positive by the
confirmatory method. Performance of the coliform detection technologies will be tested by
comparing the proportion of true positive results from those technologies to the proportion of
positive results from the SM 9221B and F and Colilert®-18.
       Specificity is defined as the percent of negative samples correctly identified as negative,
and sensitivity is defined as the percent of positive samples correctly identified as positive.
Estimates of sensitivity, specificity, false positive rates, and false negative rates as percentages
for the two methods will be calculated as follows:
                       jp.
       Sensitivity; =      Vv x 100%
                        I"r ?i»
       Specificity,  = T^ ^pp; x 100%
                                PP.-               f      77v,    \
       False positive rate.- = -=^-, — Vw x 100% =  1 - -=^-, — LF-=r x 100% 1 - Specificitr.-
                             T^i + rPi            \    T^'i + rP.J
                                                   f       TP.-   \
       False negative rateL - •== - J^T- x i°°°'c =  U ~ ^p — V^rl x: 1°°% = 1 ~ Sensiti
       Where,
              F = False
              N = Negative
              P = Positive
              T = True
              i = specified method (i = 1 for technology being tested, i = 2 for reference
              method)
Bl.3.2 Method Comparability
       The chi-square test will be used to compare the percent of false positives and false
negatives in the coliform detection technologies with the percent of false positives and false

-------
                                                               Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 39 of 39
                                                                                Version 1.0
                                                                             Date: 7/14/2010
negatives produced by the reference method. To perform the chi-square test, the data will first
be arranged in the format shown in Tables 8 and 9:

Table 8. False Negative Rate Comparison

Result



True +
False -
Total
Method
Tested
Technology
TPj
FNi
TPj + FNi
Reference
TP2
FN2
TP2 + FN2

Total
TPj + TP2
FNi + FN2
TPJ+TP2+FNJ + FN2
Table 9. False Positive Rate Comparison

Result



True +
False -
Total
Method
Tested
Technology
FPi
TNi
FPi + TNi
Reference
FP2
TN2
FP2 + TN2

Total
FPi + FP2
TNi + TN2
FPi + FP2+TNi + TN2
       In order to assess whether the false positive and false negative rates differ between
methods, chi-square tests will be run over all samples. For false negative rates, the chi-square
test(s) indicate whether the proportions of negative samples correctly identified as negative by
the two methods are significantly different, and for false positive rates, the chi-square test(s)
indicate whether the proportions of true positive results correctly identified as positive for the
two methods are significantly different.

Bl.3.3 Operational Factors
       Operational factors such as maintenance needs, calibration frequency, data output,
consumables used, ease of use, repair requirements, waste production, and sample throughput
will be documented based on operator and Verification Test Coordinator observations.
B1.4   Additional Concentration Levels

       An optional component of the ETV test will be performed to verify the capability of each
technology to detect E. coli ATCC 8739 at various concentration levels. Four target inoculations
will be prepared in dechlorinated DW. A stock solution will be prepared that contains

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 40 of 40
                                                                                Version 1.0
                                                                             Date: 7/14/2010
approximately 104 E. coli per 100 mL, and then a serial dilution (1:10, 1:100, and 1:1,000) of
the stock will be prepared to obtain four separate samples for testing (10, 100, and 1,000 E. coli
per 100 mL). Four replicates of each sample will be analyzed to establish kinetic data and "early
warning" capabilities of these technologies. The data from these tests will identify: 1) whether
or not each technology detects the presence ofE.coli and 2) the time required for detection.
Triplicate aliquots at each will also be analyzed using a quantitative method for E. coli (SM
9222G-NA-MUG).

B1.5   Reporting

       Separate verification reports and verification statements are planned for each vendor that
is participating in the verification testing.  The statistical comparisons described above will be
conducted separately for each of the technologies being tested, and information on the
operational factors will also be compiled and reported separately for each technology. The
verification reports will present the test procedures, test data as statistical evaluation of those
data, and discuss any deviations from the approved test/QA plan.
       Operational aspects of the technologies will be recorded by the testing staff at the time of
observation during the verification test, and summarized in the verification reports. The
verification report and verification statements will briefly describe the ETV program, the AMS
Center, and the procedures used in verification testing. The results of the verification test
regarding coliform detection technology performance will be stated quantitatively. Each draft
verification report and verification statement will be subjected to review by the vendor, EPA, and
other peer reviewers. The resulting review comments will be addressed in a  subsequent revision
of the report or statement, and the peer review comments and responses will  be tabulated to
document the peer review process and submitted to EPA.  The reporting and review process will
be conducted according to the requirements of the AMS Center QMP.  The cooperating ETV
programs will deem whether the results are acceptable. If acceptable, a separate verification
statement will be prepared by the cooperating ETV program.

B2    SAMPLING METHOD REQUIREMENTS

       Sampling method requirements are described above in Section B 1.1.1.

-------
                                                               Coliform Detection Technologies
                                                                              Test/QA Plan
                                                                              Page 41 of 41
                                                                               Version 1.0
                                                                            Date: 7/14/2010
B3    SAMPLE HANDLING AND CUSTODY REQUIREMENTS

       Sample custody will be documented for the DW, SWTP water, and reference test samples
following Battelle SOP ENV-ADM-009 for Chain of Custody8.  A chain-of-custody (COC) form
will include details about the sample such as the time, date, location, and person collecting the
sample. The COC form will track sample release from the sampling location to the testing
laboratory.  The COC form will be signed by the person relinquishing samples once that person
has verified that the COC form is accurate.  Upon arrival at the testing laboratory, COC forms
will be signed by the person receiving the sample once that person has verified that all samples
identified on the COC forms are present.  Copies of all COC forms will be delivered to the
Verification Test Coordinator and maintained with the test records.

B4    LABORATORY REFERENCE METHODS

       Technology verification will involve comparison of the results from each coliform
detection technology being verified to the results obtained from appropriate reference methods
which are described further in Table  10.  Superior Laboratories, Inc. will perform the reference
analysis, including all QA/QC procedures, and will comply with this test/QA plan and all
requirements of the ETV QMP. Prior to testing, the Battelle QA Manager will contact them and
confirm that adequate quality systems are in place and then a TSA will include that laboratory.
Any options described in the method will be considered and determined prior to testing and
documented in the laboratory records. All laboratory records required by the method will be
maintained. The reference methods chosen for the U.S. verification test are presence/absence
methods for TC and E. coli per SM 922IB (TC) and 922IF (E. coif). These methods utilize
selective and/or chromatogenic liquid growth media to detect TC and E. coli. In  addition, an
optional verification of these technologies for the detection of E. coli will be performed using
Colilert®-18.  The two reference methods (SM and Colilert®-18) will be run in parallel during the
verification test, and the results for both will be reported as presence/absence. Reference data
from the Colilert -18 method will be compared to only those technologies for which the vendor
has agreed to verification of E. coli results.

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 42 of 42
                                                                                 Version 1.0
                                                                             Date: 7/14/2010
Table 10. Reference Methods
Analyte
Total coliforms
E. coli
E. coli
Method Format
P/A
P/A
P/A
Approving Agency
U.S. EPA
U.S. EPA
European Union
Reference Method
SM9221B
SM9221F
Colilert-18
Citation
40 CFR 14 1.2 1111
40 CFR 141.2F
Drinking Water
Directive 98/83/EC
(3 November 1998)
P/A = presence/absence
Colilert®-18method is an optional reference method and will only be included in the verification test for each of the
technologies upon the vendor's agreement.

B5    QUALITY CONTROL REQUIREMENTS

       Quality control procedures will follow the requirements described in this test/QA plan,
ETV QMP, and any vendor specified requirements for the coliform detection technologies.
Some of the technologies may require calibration before testing begins.  If so, the calibration
method and results will be documented.
       The reference method requires the use of method blanks (MB), positive and negative
control organisms, and result confirmation (Table 11). One MB will be performed during the
analysis for every 20 samples analyzed.  The MB will consist of 100-mL dechlorinated tap water
processed as a sample. MB samples will be exposed to identical handling and analysis
procedures as the DW samples, including the addition of all reagents. These samples will be
used to help ensure that no sources of contamination are introduced in the sample handling and
analysis procedures.  The MB should be rejected if the target organisms are detected.
       Three positive and negative control samples will also be analyzed using each method.
Positive and negative control samples will be obtained from ATCC.  Control organisms include
the total coliform Enterobacter aerogenes (ATCC 13048), E. coli (ATCC 8739), and the non-
coliform Pseudomonas aeruginosa (ATCC 10145). All control cultures will be prepared at
Battelle following the manufacturer's instructions. In short, freeze dried cultures will be
rehydrated using the medium and incubation conditions specified in the manufacture's catalog.
The QC control samples will be prepared by  diluting the cultures in sterile deionized water for
11140 CFR 141.21 - Coliform sampling. Code of Federal Regulations - Title 40: Protection of Environment
(December 2005)

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 43 of 43
                                                                                Version 1.0
                                                                             Date: 7/14/2010
analysis. Control samples will be used to determine the specificity and sensitivity of the
coliform detection technologies and reference methods.
       In addition, reference results for one of the three dilutions sets for each target organism
must result in a ratio of 50 ± 25% positive results.  If these requirements are not met, a new set of
samples may be analyzed by the reference method.  If the results are still outside the required
limits, the repeat of the appropriate parts of the verification test may be considered.

B6    EQUIPMENT TESTING, INSPECTION, AND MAINTENANCE

       The equipment used for the reference analyses will be tested and inspected as per the
equipment manuals, the standard operating procedures of the analysis laboratory, or the methods
being used to make each measurement; results will be documented.  Operation of the
technologies during the verification test will be performed by Battelle as directed by the vendor.
       Temperatures for the refrigerators and freezers will be logged on a daily basis while test
samples are in storage. If found to be outside of the specified range above, the samples will be

-------
Table 11. Quality Control Samples for Reference and Technology Methods
                                                                                                                  Coliform Detection Technologies
                                                                                                                                   Test/QA Plan
                                                                                                                                  Page 44 of 44
                                                                                                                                    Version 1.0
                                                                                                                                 Date: 7/14/2010

Performance
Test

Method Blank

TC Positive
control
E. coli Positive
control
Negative Control

Method of
Assessment

Analysis of filter-
sterilize,
dechlorinated tap
water

Analysis of ATCC
E. aerogenes
Analysis of ATCC
E. coli
Analysis of ATCC P.
aemgenes
Dilution


N/A

N/A
N/A
N/A
Applicable
Method

Reference and
Technology

Reference and
Technology
Reference and
Technology
Reference and
Technology

Frequency

Once every 6-0
samples

Once every 60
samples
Once every 60
samples
Once every 60
samples
Total Number
per Analysis
and Target
Organism
o
J

3
o
3
3

Acceptance
Criteria

No growth

Target
organisms
detected
Target
organisms
detected
Target
organisms not
detected

Corrective Action

Review data and analysis for
possible sources of
contamination. Re-analyze QC
sample and document corrective
action.
Recheck controls for evidence
of cross contamination or
deterioration.
Recheck controls for evidence
of cross contamination or
deterioration.
Recheck controls for evidence
of cross contamination or
deterioration.
Note: Sterility control testing on every lot of reagents (buffers and media) used will be performed during each batch (iteration) of testing.

-------
                                                               Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 45 of 45
                                                                                Version 1.0
                                                                             Date: 7/14/2010
transferred to an acceptable refrigerator or freezer and the deviation will be noted in the LRB, in
a deviation report, and in the final verification reports.

B7    CALIBRATION VERIFICATION

       Balances, thermometers, pH meters, autoclaves, incubators, pipettes, etc. used during
sample preparation and analysis will be maintained and calibrated as required by the Battelle or
reference laboratory SOPs.  A list of relevant SOPs is attached to this test/QA plan. Equipment
maintenance logs and calibration records will be reviewed by the Battelle Quality Manager prior
to the verification test.
       One laboratory clock will be used as the standard for all time recording.  Any other time
pieces must be calibrated to it.  In particular, the sewage and DW sample collection time must be
determined based on a calibrated time piece.
       If necessary, the technologies undergoing testing will be calibrated initially by the
respective vendors at the time of set-up, at the vendor's discretion.

B8    INSPECTION/ACCEPTANCE OF SUPPLIES AND CONSUMABLES

       All materials, supplies,  and consumables will be ordered by the Verification Test
Coordinator or designee.  Where possible, Battelle will rely on sources of materials and
consumables that have been used previously as part of ETV verification testing without
problems. Battelle will also rely on previous experience or recommendations from its technical
staff to guide selection of manufacturers and materials. The manufacturer's criteria for
acceptance/purity/sterility will  be required to be met. Supplies must meet the following criteria:
          •   Solvent and reagent grades are based on the intended use. All materials must
              meet the purity requirements of the method.
          •   Equipment used to generate data must provide appropriate sensitivity.
          •   A certificate of analysis must be retained for reagents and standards.
          •   A certificate of sterility must be retained for sample containers and growth  media.
          •   The quality and purity of expendable materials must be adequate to meet the data
              quality objectives of the client.

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 46 of 46
                                                                                 Version 1.0
                                                                             Date: 7/14/2010
These same requirements for quality must be implemented by the reference laboratory. The
laboratory manager is responsible for verifying the quality of supplies used for this test.

B9    NON-DIRECT MEASUREMENTS

       Data published previously in the scientific literature will not be used during this
verification test.

BIO   DATA MANAGEMENT

       Various types of data will be acquired and recorded electronically or manually by
verification staff during this verification test.  Table 3 summarizes the type of data to be
recorded.  All data and observations for the operation of the coliform detection technologies will
be documented by the verification staff on data sheets or in LRBs. Results from the laboratory
analyses will be compiled by laboratory staff in electronic format and submitted to the
Verification Test Coordinator or other verification staff upon obtaining results within 7 days of
test completion. A dedicated shared folder within the ETV AMS Center SharePoint site will be
established for all project records.
       Records received by or generated by any of the verification staff during the verification
test will be reviewed by a Battelle staff member within two weeks of receipt or generation,
respectively, before the records are used to calculate,  evaluate, or report verification results.  The
review will be documented as the dated initials of the reviewer.  Some of the checks that will be
performed include:
           •   QC samples and calibration standards were analyzed according to the test/QA
              plan and the acceptance criteria were met.  Corrective action for exceedances was
              taken
           •   100% hand-entered and/or manually calculated data were checked for accuracy
           •   Calculations performed by software are verified at a frequency sufficient to
              ensure that the formulas are correct, appropriate, and consistent
           •   For each cut and paste function, the first and last data value was verified vs. the
              source data

-------
                                                                 Coliform Detection Technologies
                                                                                Test/QA Plan
                                                                                Page 47 of 47
                                                                                  Version 1.0
                                                                              Date: 7/14/2010
           •   Data are reported in the units specified in the test/QA plan.
           •   Results of QC samples are reported.

       If a Battelle staff member generated the record, this review will be performed by a
Battelle technical staff member involved in the verification test, but not the staff member that
originally received or generated the record. The review will be documented by the person
performing the review by adding his/her initials and date to the hard copy of the record being
reviewed.  In addition, data calculations performed by verification staff will be spot-checked by
Battelle technical staff to ensure that calculations are performed correctly. Calculations to be
checked include any statistical calculations described in this test/QA plan. The data obtained
from this verification test will be compiled and reported for each technology.  Results for
technologies from different vendors will not be compared with each other.
       Battelle will provide technology test data and associated reference data (including
records, data sheets, notebook records) from the first day of testing within one day of receipt to
EPA for simultaneous review. The goal of this data delivery schedule is prompt identification
and resolution of any data collection or recording issues.  These data will labeled as preliminary
and will not have had a QA review before their release.

-------
                                                               Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 48 of 48
                                                                                Version 1.0
                                                                             Date: 7/14/2010
                                      SECTION C
                           ASSESSMENT AND OVERSIGHT

Cl    ASSESSMENTS AND RESPONSE ACTIONS

       Every effort will be made in this verification test to anticipate and resolve potential
problems before the quality of performance is compromised.  One of the major objectives of the
test/QA plan is to establish mechanisms necessary to ensure this. Internal quality control
measures described in this test/QA plan, which is peer reviewed by a panel of outside experts,
implemented by the technical staff and monitored by the Verification Test Coordinator, will give
information on data quality on a day-to-day basis. The responsibility for interpreting the results
of these checks and resolving any potential problems resides with the Verification Test
Coordinator, who will contact the Battelle AMS Center Manager, Battelle AMS Center Quality
Manager, EPA AMS Center Project Officer, and EPA AMS Center QA Manager if any
deviations from the test/QA plan are observed. The Verification Test Coordinator will describe
the deviation in a teleconference or by email, and once a path forward is determined and agreed
upon with EPA, the deviation form will be completed.  Technical staff have the responsibility to
identify problems that could affect  data quality or the ability to use the data. Any problems that
are identified will be reported to the Verification Test Coordinator, who will work with the
Battelle Quality Manager to resolve any issues. Action will be taken by the Verification Test
Coordinator and Battelle testing staff to identify and appropriately address the issue, and
minimize losses and correct data, where possible. Independent of any EPA  QA activities,
Battelle will be responsible for ensuring that the following audits are conducted as part of this
verification test.

Cl.l   Technical Systems Audits

       Battelle Quality Manager will perform a technical systems audit (TSA)  at least once
during this verification test. The purpose of this audit is to ensure that the verification test is
being performed in accordance with the AMS Center QMP and this test/QA plan. The primary
focus of the audit will be the reference test method and specifically the determination of positive
and negative results.  During this audit, the Battelle Quality Manager, or designee, will compare

-------
                                                                Coliform Detection Technologies
                                                                               Test/QA Plan
                                                                               Page 49 of 49
                                                                                 Version 1.0
                                                                             Date: 7/14/2010
actual test procedures to those specified or referenced in this plan, and review data acquisition
and handling procedures. The audit will include a review of the testing facility and equipment
(calibration, maintenance, operation), and observation of testing and records (including custody
forms).  She will also check data acquisition procedures and may  confer with the vendor and
Battelle technical staff. The TSA will be guided by a project-specific checklist based on the
test/QA plan and reference methods.
       The Battelle Quality Manager will prepare an initial TSA  report and will submit the
report to the Verification Test Coordinator within 10 business days after completion of the audit.
A copy of this initial TSA report (with no corrective actions documented) will be provided to the
EPA AMS Center Project Officer.  A copy of the final TSA report (with corrective actions
documented) will be provided to the EPA AMS Center Project Officer within 20 business days
after completion of the audit.  At EPA's discretion, EPA QA staff may also conduct an
independent on-site TSA during the verification test.  The TSA findings will be communicated to
technical staff at the time of the audit and documented in a TSA report.

C1.2   Data Quality Audits

       The Battelle Quality Manager, or designee, will audit at least 25% of the sample results
acquired in the verification test and 100% of the calibration and QC data vs. the test/QA plan
requirements. A checklist based on the test/QA plan will guide the audit. An initial data quality
audit will be conducted on the first batch of test data within 10 business days of when data were
collected to identify errors early in the data reduction process. Given the short duration of this
test, the first batch is defined as the first set of reference test data  released by the Verification
Test Coordinator.  The remaining data will be audited once all data for a technology or method
has been posted on the project SharePoint site and once all statistical analyses are complete.  The
primary focus of the audit will be the reference test method. The  Battelle Quality Manager, or
designee, will trace the data from initial acquisition, through reduction and  statistical
comparisons, to final reporting.  All formulae applied to the data will be verified.  Data for the
technologies will be reviewed for calculation and transcription errors and data traceability.
Results of each audit of data quality (ADQ) will be documented using the checklist and reported
to the Verification Test Coordinator within 10 business days after completion of the audit. The
EPA AMS Center Project Officer will be provided a copy of the initial ADQ report (with no

-------
                                                               Coliform Detection Technologies
                                                                              Test/QA Plan
                                                                              Page 50 of 50
                                                                                Version 1.0
                                                                            Date: 7/14/2010
corrective actions documented). A final ADQ that assesses overall data quality, including
accuracy and completeness of the verification report, will be prepared as a narrative and
distributed to the Verification Test Coordinator within 10 business days of completion of the
audit. The EPA AMS Center Project Officer will be provided a copy of the final ADQ report
(with corrective actions documented).

C1.3  QA/QC Reporting

       Each assessment and audit will be documented in accordance with Section 3.3.4 of the
AMS Center QMP.  The results of the TSA and DQA will be submitted to EPA.  Assessment
reports will include the following:
          •   Identification of Findings  and Observations
          •   Recommendations for resolving problems
          •   Response to adverse findings or potential problems
          •   Confirmation that solutions have been implemented and are effective
          •   Citation of any noteworthy practices that may be of use to others.

C2    REPORTS TO MANAGEMENT

       The Battelle Quality Manager, during the course of any assessment or audit, will identify
to the technical staff performing experimental activities any immediate corrective action that
should be taken. If serious quality problems exist, the Battelle Quality Manager is authorized to
notify the Battelle AMS  Center Manager who will issue the stop work. Once the TSA or data
quality audit report has been prepared, the Verification Test Coordinator will respond to each
Finding and Observation following the timeline defined in Section Cl and will implement any
necessary corrective action.  The Battelle Quality Manager will verify that corrective action has
been implemented effectively.
       In addition to this test/QA plan, a final report and  a verification statement for each
technology verified will be prepared and  reviewed.  The final report  is a comprehensive
document describing the verification test. The verification statement is a two-to-three page
summary of the technology, the test procedures, and the test results.  Each draft report and
verification statement will be submitted to the respective vendor for  review.  They are then

-------
                                                               Coliform Detection Technologies
                                                                              Test/QA Plan
                                                                              Page 51 of 51
                                                                               Version 1.0
                                                                            Date: 7/14/2010
reviewed by EPA AMS Center Quality Manager and the EPA AMS Center program Project
Officer. Upon approval by EPA, each verification statement will be signed by a senior manager
of Battelle and by an EPA laboratory director.  Original signed verification statements will be
provided to the respective vendors for use in marketing their technology. Upon final review and
approval, the final verification report(s) will be submitted to EPA in MS Word and Adobe
portable document format (PDF) and subsequently posted on the ETV website
(www.epa.gov/etv).

-------
                                                               Coliform Detection Technologies
                                                                              Test/QA Plan
                                                                              Page 52 of 52
                                                                                Version 1.0
                                                                            Date: 7/14/2010
                                      SECTION D
                        DATA VALIDATION AND USABILITY

Dl    DATA REVIEW, VALIDATION, AND VERIFICATION REQUIREMENTS

       The key data review requirements for the verification test are the analysis of QC samples
as outlined in the test/QA plan, a comparison of laboratory data sheet comments against final
data to flag any suspect data, and a review of final data to resolve any questions about apparent
outliers.  The QA audits, as described within this document, are designed to assure the quality of
this data.

D2    VALIDATION AND VERIFICATION METHODS

       Section C of this test/QA plan provides a description of the validation safeguards
employed for this verification test. Data validation and verification efforts include the analysis
of QC samples as required in this document, and the performance of a ISA and DQAs as
described in Section C.

D3    RECONCILIATION WITH USER REQUIREMENTS

       The purpose of this verification test is to evaluate the performance of coliform detection
technology for use in DW monitoring. In part, this evaluation will include demonstrations of the
monitoring ability of coliform detection technologies to detect the presence of TC and E. coli.
To meet the requirements of the user community, input on the tests described in this test/QA plan
has been provided by external experts. Additional performance data regarding operational
characteristics of the coliform detection technologies will be  collected by verification test
personnel. To meet the requirements of the user community, these data will include thorough
documentation of the performance of the technologies during the verification test. The data
review, verification, and validation procedures described above will assure that data meeting
these requirements are accurately presented in the verification reports generated from this test,
and will assure that data not meeting these requirements will be appropriately flagged and
discussed in the verification reports.

-------
                                                                  Coliform Detection Technologies
                                                                                 Test/QA Plan
                                                                                 Page 53 of 53
                                                                                   Version 1.0
                                                                               Date: 7/14/2010
       This test/QA plan and the resulting ETV verification report(s) will be subjected to review

by the vendor(s), EPA, and expert peer reviewers. The reviews of this test/QA plan will help to

improve the design of the verification test and the resulting report(s) such that they better meet

the needs of potential users of these technologies.

-------
                                                             Coliform Detection Technologies
                                                                           Test/QA Plan
                                                                           Page 54 of 54
                                                                            Version 1.0
                                                                         Date: 7/14/2010
                                     SECTION E

                                   REFERENCES
El     REFERENCES

1.      Quality Management Plan for the ETV Advanced Monitoring Systems Center, Version 7.
       U.S. Environmental Technology Verification Program, Battelle, November 2008.

2.      Environmental Technology Verification Program Quality Management Plan. January,
       2008 (EPA/600/R-08/009).

3.      American Public Health Association, American Water Works Association, and Water
       Environment Federation.  2005. Standard Methods for the Examination of Water and
       Wastewater. 21st Edition.

4.      Battelle Building 20 Biosafety Manual, Version 4.0.

5.      SOP GEN.V-003-10. Standard Operating Procedure for the Use ofpH meters to
       Measure pH. Battelle.

6.      AO AC International.  1995. Bacteriological Analytical Manual, 8th Edition.

1.      SOP GEN.V-013-04. SOP for the Calibration and Maintenance of Thermometers.
       Battelle.

8.      SOP ENV-ADM-009, Standard Operating Procedure for Sample Chain-of-Custody
       Battelle, September 2007.

-------