A1*A—, tm
at innovation
Environmental Technology
Verification Program
Advanced Monitoring
Systems Center
Generic Verification Protocol for
Rapid Beach Water Quality
Screening Technologies
ETY ETV ET
-------
This page intentionally left blank.
-------
GENERIC VERIFICATION PROTOCOL
for
Verification of
Rapid Beach Water Quality
Screening Technologies
August 31, 2007
Prepared by
Battelle
505 King Avenue
Columbus, OH 43201-2693
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 4 of 48
Version 1.0
Date: 8/31/2007
ACKNOWLEDGEMENTS
This generic protocol was developed by Battelle under a cooperative agreement with the
U.S. Environmental Protection Agency's (EPA's) National Exposure Research Laboratory
(NERL) and includes input from stakeholders, especially from Kenneth M. Hill, Elizabeth Essex,
and Vito Minei of the Suffolk County Department of Health Services. Peer reviewers for this
protocol were Kristen P. Brenner of EPA NERL; Martha Link of the Nebraska Department of
Environmental Quality; and Richard H. Sakaji of the East Bay Municipal Utility District,
Oakland, California. The contributions to this protocol from the EPA Project Officer
Robert Fuerst and Quality Assurance Manager Elizabeth Betz are gratefully acknowledged.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 5 of 48
Version 1.0
Date: 8/31/2007
TABLE OF CONTENTS
Section
SECTION A. PROJECT MANAGEMENT 11
Al VERIFICATION TEST ORGANIZATION 11
Al.l Battelle 12
A1.2 Beach Water Quality Screening Technology Vendors 14
A1.3 EPA 15
A1.4 Technical Staff 16
A1.5 Reference Laboratory 17
A1.6 Test Facility 17
A2 BACKGROUND 17
A3 VERIFICATION TEST DESCRIPTION AND SCHEDULE 19
A3.1 Summary of Technology Category 19
A3.2 Verification Test Schedule 21
A3.3 Test Location 22
A3.4 Health and Safety 22
A4 QUALITY OBJECTIVES AND CRITERIA FOR MEASUREMENT DATA... 22
A4.1 Quality Objectives 23
A4.2 Criteria for Measurement Data 23
A5 SPECIAL TRAINING/CERTIFICATION 24
A6 DOCUMENTATION AND RECORDS 25
SECTIONS. MEASUREMENT AND DATA ACQUISITION 27
Bl EXPERIMENTAL DESIGN 27
Bl.l Test Procedures 28
B1.2 Statistical Analysis 31
B1.3 Reporting 34
B2 SAMPLING REQUIREMENTS 35
B3 SAMPLE HANDLING AND CUSTODY REQUIREMENTS 36
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 6 of 48
Version 1.0
Date: 8/31/2007
B4 LABORATORY REFERENCE METHODS 37
B5 QUALITY CONTROL 38
B6 INSTRUMENT/EQUIPMENT TESTING, INSPECTION, AND
MAINTENANCE 38
B7 CALIBRATION/VERIFICATION OF TEST PROCEDURES 39
B8 INSPECTION/ACCEPTANCE OF SUPPLIES AND CONSUMABLES 39
B9 NON-DIRECT MEASUREMENTS 39
BIO DAT A MANAGEMENT 40
SECTION C. ASSESSMENT AND OVERSIGHT 42
Cl ASSESSMENTS AND RESPONSE ACTIONS 42
Cl.l Performance Evaluation Audits 42
C1.2 Technical Systems Audits 43
C1.3 Data Quality Audits 43
C1.4 QA/QC Reporting 44
C2 REPORTS TO MANAGEMENT 44
SECTION D. DATA VALIDATION AND USABILITY 45
D1 DATA REVIEW, VALIDATION, AND VERIFICATION REQUIREMENTS. 45
D2 VALIDATION AND VERIFICATION METHODS 45
D3 RECONCILIATION WITH USER REQUIREMENTS 46
SECTION E. REFERENCES 48
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 7 of 48
Version 1.0
Date: 8/31/2007
List of Figures Page
Figure 1. Organization Chart 12
List of Tables
Table 1. 1986 Criteria for Bacteriological Indicators 18
Table 2. General Verification Test Schedule 21
Table 3. Data Quality Indicators and Criteria for Critical Measurements for Reference
Method(s) 24
Table 4. Summary of Data Recording Process 26
Table 5. Spike Levels for Indicator PT Samples 29
Table 6. Summary of Test Samples 30
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 8 of 48
Version 1.0
Date: 8/31/2007
DISTRIBUTION LIST
U.S Environmental Protection Agency
Advanced Monitoring Systems (AMS) Center Project Officer
AMS Center Quality Manager
Battelle AMS Center
AMS Center Manager
Verification Testing Leader
Verification Test Coordinator
Quality Manager
Technical Staff
Peer Reviewers
Rapid Technologies for Beach Water Quality Screening Vendors
Reference Laboratory (if applicable)
Test Facility (if applicable)
Test Collaborators (if applicable)
Subcontractors (if applicable)
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 9 of 48
Version 1.0
Date: 8/31/2007
LIST OF ABBREVIATIONS/ACRONYMS
AMS Advanced Monitoring Systems
ASW artificial sea water
cfu colony forming units
COA certificate of analysis
COC chain-of-custody
DI deionized
DQI data quality indicator
E. coli Escherichia coli
EPA U.S. Environmental Protection Agency
ETV Environmental Technology Verification
LCS laboratory control standard
LRB laboratory record book
mTEC membrane-Thermotolerant Escherichia coli Agar
MS matrix spike
NEEAR National Epidemiological and Environmental Assessment of Recreational
NIST National Institute of Standards and Technology
PCR polymerase chain reaction
pdf Adobe portable document format
PE performance evaluation
PT performance test
QA quality assurance
QC quality control
QCS quality control samples
QMP quality management plan
R percent recovery
r2 coefficient of determination
RSD relative standard deviation
S standard deviation
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 10 of 48
Version 1.0
Date: 8/31/2007
SCCWRP Southern California Coastal Water Research Proj ect
SOP standard operating procedure
TC temperature control
TQAP test/quality assurance plan
ISA technical systems audit
w/v weight/volume
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 11 of 48
Version 1.0
Date: 8/31/2007
SECTION A.
PROJECT MANAGEMENT
Al VERIFICATION TEST ORGANIZATION
This protocol provides generic procedures for implementing a verification test for rapid
technologies for beach water quality screening in fresh, brackish, and/or marine/estuarine water.
Verification tests are conducted under the auspices of the U.S. Environmental Protection Agency
(EPA) through the Environmental Technology Verification (ETV) Program. Verification tests of
monitoring technologies are coordinated by Battelle, which manages the ETV Advanced
Monitoring Systems (AMS) Center through a cooperative agreement with EPA. The scope of the
AMS Center covers verification of monitoring technologies for contaminants and natural species
in air, water, and soil. In performing verification tests, Battelle follows the procedures specified
in this protocol and compiles quality requirements in the "Quality Management Plan for the ETV
Advanced Monitoring Systems Center" (QMP).(1)
Verification tests are performed by Battelle in cooperation with EPA and the vendors
whose technologies are being verified. These test procedures may be performed by Battelle, test
facility staff, a qualified collaborator, and/or a qualified subcontractor. The specific staff
members who will perform the test procedures are referred to as "technical staff in this protocol.
Each technology vendor is expected to provide their respective beach water quality screening
technology, including installation, as appropriate, and any supplies required for its operation.
Each vendor will operate, repair and/or maintain their technology during the test, unless they
provide written consent for other technical staff to operate their technology. Battelle will
oversee all testing activities.
Quality Assurance (QA) oversight will be provided by the Battelle Quality Manager and
also by the EPA AMS Center Quality Manager, at his or her discretion. The organization chart in
Figure 1 identifies the responsibilities of the organizations and individuals associated with the
verification test. Roles and responsibilities are defined further below.
-------
Battelle
Management
Battelle
Quality Manager
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 12 of 48
Version 1.0
Date: 8/31/2007
Battelle AMS
Center Manager
Verification
Testing Leader
Verification
Test Coordinator
Technical Staff
EPA AMS Center
Project Officer
EPA AMS Center
Quality Manager
Vendor
Representatives
Figure 1. Organization Chart
Al.l Battelle
The AMS Center's Verification Test Coordinator has overall responsibility for ensuring
that the technical, schedule, and cost goals established for the verification test are met.
Specifically, the Verification Test Coordinator will:
• Prepare a draft test/quality assurance plan (TQAP) based on this protocol and revise it
in response to reviewers' comments.
• Establish a budget for the verification test and manage staff to ensure the budget is
not exceeded.
• Assemble a team of qualified technical staff to conduct the verification test.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 13 of 48
Version 1.0
Date: 8/31/2007
• Direct the team in performing the verification test in accordance with this protocol
and any TQAP based on this protocol.
• Hold a kick-off meeting for technical staff approximately one week prior to the start
of the verification test to review the critical logistical, technical, and administrative
aspects of the verification test and to confirm responsibility for each aspect of the
verification test.
• Ensure that all quality procedures specified in this protocol, in any TQAP based on
this protocol, and in the AMS Center QMP(1) are followed.
• Serve as the primary point of contact for vendor representatives.
• Ensure that confidentiality of sensitive vendor information is maintained.
• Assist vendors as needed during the beach water quality screening technology
installation and verification testing.
• Become familiar with the operation and maintenance of the beach water quality
screening technologies, including through instruction by the vendors, if needed.
• Respond to any issues raised in assessment reports, audits, or from test staff
observations, and institute corrective action as necessary.
• Prepare the draft verification reports and verification statements and revise in
response to reviewer comments.
• Coordinate distribution of the final TQAP, verification reports, and verification
statements.
The Verification Testing Leader for the AMS Center provides technical guidance and
oversees the various stages of verification testing. The Verification Testing Leader will:
• Support the Verification Test Coordinator in organizing the test.
• Review the draft and final verification reports and verification statements.
• Attend the verification test kick-off meeting.
The Battelle AMS Center Manager will:
• Review the draft and final verification reports and verification statements.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 14 of 48
Version 1.0
Date: 8/31/2007
• Ensure that necessary Battelle resources, including staff and facilities, are committed
to the verification test.
• Ensure that confidentiality of sensitive vendor information is maintained.
• Support the Verification Test Coordinator in responding to any issues raised in
assessment reports and audits.
• Maintain communication with EPA's technical and quality managers.
• Attend the verification test kick-off meeting.
• Issue a stop work order if Battelle or EPA QA staff discovers adverse findings that
will compromise test results.
Battelle's Quality Manager for the AMS Center will:
• Attend the verification test kick-off meeting.
• Conduct a technical systems audit once during the verification test, or designate other
QA staff to conduct the audit.
• Audit at least 10% of the verification data or designate other QA staff to conduct the
data audit.
• Prepare and distribute an assessment report for each audit.
• Verify implementation of any necessary corrective action.
• Request that Battelle's AMS Center Manager issue a stop work order if audits
indicate that data quality is being compromised.
• Provide a summary of the Q A/quality control (QC) activities and results for the
verification reports.
• Review the draft and final TQAP, verification reports, and verification statements.
• Assume overall responsibility for ensuring that the test/QA plan is followed.
A1.2 Beach Water Quality Screening Technology Vendors
The responsibilities of the beach water quality screening technology vendors are as
follows:
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 15 of 48
Version 1.0
Date: 8/31/2007
Provide their beach water quality screening technology for evaluation during the
verification test.
Provide all other equipment/supplies/reagents/consumables needed to operate their
technology for the duration of the verification test.
Supply a representative to install and maintain their technology, as appropriate, and
provide training instructions, and written consent for technical staff to carry out
verification testing. Alternatively, the vendor representative may operate the
technology during the verification test.
Provide written instructions for routine operation of their technology including a daily
checklist of diagnostic and/or maintenance activities, if applicable.
Provide maintenance, repair, and/or technical support for their technology, on-site if
necessary, throughout the duration of the verification test.
Review and provide comments on the draft TQAP, verification report, and
verification statement for their respective technology.
A1.3 EPA
EPA's responsibilities in the AMS Center are based on the requirements stated in the
"Environmental Technology Verification Program Quality Management Plan" (EPA QMP).(2)
The roles of specific EPA staff are as follows:
The EPA AMS Center Quality Manager will:
• Perform at his or her option one external technical systems audit during the
verification test.
• Notify the EPA AMS Center Project Officer of the need for a stop work order if the
external audit indicates that data quality is being compromised.
• Prepare and distribute an assessment report summarizing results of the external audit.
• Review draft verification reports and verification statements.
EPA's Project Officer for the AMS Center will:
• Review the draft TQAP, verification reports, and verification statements.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 16 of 48
Version 1.0
Date: 8/31/2007
• Notify the Battelle AMS Center Manager of the need for a stop work order if work of
inadequate quality is discovered.
• Oversee the EPA review process for verification reports and verification statements.
• Coordinate the submission of verification reports and verification statements for final
EPA approval.
A1.4 Technical Staff
Technical staff from Battelle, subcontractor(s), and/or collaborating organization(s) will
support the Verification Test Coordinator in planning and conducting the verification test. The
responsibilities of the technical staff may include:
• Assist in planning for the test, and making arrangements for the receipt of the beach
water quality screening technologies.
• Attend the verification test kick-off meeting.
• Assist vendor staff as needed during the beach water quality screening technology
verification testing.
• Perform experimental procedures specified in this protocol and/or TQAP and acquire
data from the beach water quality screening technologies provided the vendor has
given written consent for the technical staff to conduct testing. Contact the
Verification Test Coordinator if any problems in testing or equipment operation
occur.
• Record observations about the maintenance and operation of the beach water quality
screening technologies during the testing period.
• Perform statistical calculations specified in this protocol and/or TQAP on the beach
water quality screening data, as needed.
• Provide results of statistical calculations and associated discussion for the verification
reports as needed.
• Support the Verification Test Coordinator in responding to any issues raised in
assessment reports and audits related to statistics and data reduction as needed.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 17 of 48
Version 1.0
Date: 8/31/2007
A1.5 Reference Laboratory
One or more analytical laboratories at Battelle, a subcontractor, and /or a collaborating
organization will serve as a reference laboratory to:
• Perform reference analyses of all test and QC samples described in this protocol
and/or TQAP.
• Submit the results of the reference analyses in an agreed-upon format to the
Verification Test Coordinator.
A1.6 Test Facility
Either Battelle or another appropriate facility such as a laboratory that conducts beach
water quality screening will serve as the test facility. The test facility personnel are expected to:
• Identify a point of contact for the test who will serve as the primary interface with the
Verification Test Coordinator.
• Attend the verification test kick-off meeting.
• Ensure that test facility staff and facilities are ready for the verification test.
• Assist Battelle's Verification Test Coordinator in ensuring that verification testing is
conducted in accordance with this protocol and/or TQAP.
• Assist Battelle and beach water quality screening technology vendor staff in the
installation, operation, testing and removal of the beach water quality screening
technologies from the test facility.
• Ensure that necessary test facility resources (e.g., space and power) are committed to
the verification test.
A2 BACKGROUND
The ETV Program's AMS Center conducts third-party performance testing of
commercially available technologies that detect or monitor contaminants and natural species in
air, water, and soil. Stakeholder committees of buyers and users of such technologies recommend
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 18 of 48
Version 1.0
Date: 8/31/2007
technology categories, and technologies within those categories, as priorities for testing. Beach
water quality screening technologies were identified as a priority technology category through
the AMS Center stakeholder process since the screening technologies have the potential to make
beach closures and public health warnings more efficient and timely.
Frequent water quality monitoring of recreational waters is necessary to ensure the
public's safety. Current water quality criteria require that concentrations of pathogen indicator
organisms, such as Enterococci and E. coli, are below specific action levels for marine and
freshwater locations. Current action levels by indicator organism and water type are summarized
in Table l.(3) Beach managers currently use results generated at least 24 hours after samples are
collected to determine if public health warnings should be posted and/or beaches closed for
recreational use. Rapid monitoring technologies that produce same-day results for pathogen
indicator organisms in recreational waters are beginning to emerge on the commercial market.
Independent performance testing of such technologies is imperative to demonstrating to EPA,
state/local regulators, and coastal managers the potential applicability of these technologies to
beach monitoring applications.
Table 1. 1986 Criteria for Bacteriological Indicators
Indicator
Organism
Enterococci
E. coli
Water Type
Fresh water
Marine water
Fresh water
Steady State Geometric
Mean Indicator Density
33 colony forming units
(cfu)/100 mL
35 cfu/100 mL
126 cfu/100 mL
Single Sample Maximum
Allowable Density(a)
61 cfu/100 mL
104 cfu/100 mL
235 cfu/100 mL
(a) Designated Beach Area (upper 75% confidence limit)
Both state and Federal programs aimed at identifying and evaluating new methods for
rapid determination of beach water quality are ongoing. For example, the National
Epidemiological and Environmental Assessment of Recreational Water Study, which is a
collaboration between two EPA laboratories and the Centers for Disease Control and Prevention,
is designed to correlate human health effects with recreational water use, by application of rapid
water quality methods. As part of the study, the water quality at several beaches has been
evaluated using traditional and new rapid methods. The measured bacterial indicator levels are
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 19 of 48
Version 1.0
Date: 8/31/2007
being compared to survey data of gastrointestinal symptoms in swimmers to correlate water
quality with occurrence of swimming-associated gastroenteritis.(4) Additional rapid beach water
quality methods have been evaluated by the Southern California Coastal Water Research Project
(SCCWRP) Authority.(5'6) The SCCWRP studies included initial testing of six technologies that
detect Enterococci or E. coli and more comprehensive "beta" testing of two genetic methods;
beta testing included side-by-side comparisons of the rapid and traditional methods for more than
100 samples. Although several rapid methods have demonstrated promising results in these
tests, the false negative rates, in particular, have exceeded those demonstrated by traditional
methods. Developers of rapid beach water quality screening technologies continue to refine and
improve their methods.
Verification under this protocol will determine the performance characteristics of
commercially available technologies that can provide same-day results (i.e., within 8 hours) to
make beach closures and public health warnings more efficient. Technologies may detect
pathogen indicator organisms in fresh, brackish, and/or marine/estuarine water. Critical
characteristics that will be understood as a result of this testing include the following:
• Accuracy, precision, sensitivity, and comparability of technology results relative to
accepted laboratory-based methods (e.g., membrane filtration)
• Rate of false positive/false negative results relative to current action levels (e.g.,
frequency of technology reporting Enterococci levels above 104 colony forming units
(cfu)/100 mL in marine water when the level is actually below, and vice versa)
• Operational features of each technology, including ease of use and maintenance
• Interference effects and selectivity.
A3 VERIFICATION TEST DESCRIPTION AND SCHEDULE
A3.1 Summary of Technology Category
Technologies to be evaluated in the beach water quality screening technology verification
include rapid monitoring technologies that produce same-day results for pathogen indicator
organisms in recreational waters. Traditional beach monitoring methods involve culture-based
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 20 of 48
Version 1.0
Date: 8/31/2007
methods that can take up to 24 hours to achieve results. Rapid screening technologies provide
same-day results (i.e. in less than 8 hours) and have the potential to expedite the decision making
process for regulators who monitor beach water quality. The analytical techniques employed by
such technologies vary widely, and include immunoassay test strips, flow cytometry, quantitative
polymerase chain reaction (PCR), and others. The mechanism of action of these technologies
may vary; however, a key factor is the technology's ability to generate data on water quality
results on the same day as sample collection (i.e., in 8 hours or less). Rapid technologies for
beach water quality screening, referred to in this protocol as "screening technologies," may be
quantitative, semi- quantitative, or provide a presence/absence response with respect to a pre-
defined limit. The target indicator bacteria detected by the rapid screening technology may be E.
coli, Enterococci, and/or other such appropriate pathogen-indicating bacteria that are part of
beach monitoring programs at the time of testing. In this protocol, "indicator bacteria" refers to
the target pathogen-indicating bacteria detected by the screening technology and/or used by
beach monitoring programs to determine the safety of water for bathing/recreational use.
Rapid screening technologies for determining beach water quality offer a number of
advantages over traditional methods. Most screening methods require fewer man-hours per
analysis, allowing for greater sample throughput compared to traditional methods. Same-day
information on the safety of beaches can reduce the exposure of swimmers to contamination
during the time for analysis of water samples using traditional methods. Thus, beaches can be
closed the same day that samples are collected and determined to be above safe levels rather than
the next day. Similarly, beaches can be reopened more quickly once the bacteria count has
dropped to safe levels. Faster results can also provide more opportunities to track and mitigate
contamination sources, protecting the ecosystem, human health, and promoting environmental
sustainability. Although rapid screening technologies are not currently included in state and
federal recreational water regulations for determining beach closures and openings, they can be
used to augment traditional methods especially for non-regulatory applications, such as tracking
the spread and dilution of known spills and tracking fecal contamination to the source.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 21 of 48
Version 1.0
Date: 8/31/2007
A3.2 Verification Test Schedule
A beach water quality screening verification test following this protocol should take
between six and nine months to complete after the start of testing. Table 2 shows a general
schedule of testing and data analysis/reporting activities to be conducted in this verification.
Test planning and site preparation may take place over a period of three to four months. The
period of operation at the test facility will be approximately two weeks. The test procedures are
described in Section B of this protocol. Subsequent to the field testing, a separate verification
report and verification summary statement for each technology will be drafted, reviewed,
revised, and submitted to EPA for final signature.
Table 2. General Verification Test Schedule
Anticipated
Duration
(months)
3-4
1
1
1
1-2
1-2
1-3
Testing Activities
• Identify test collaborators/
subcontractors and test facility
• Recruit vendors
• Procure necessary standards and
reagents
• Arrange for necessary logistical
infrastructure and supplies at the
test facility
• Develop TQAP
• Vendor to set up/install screening
technologies and train technical
staff on technology use
• Conduct verification tests
• Conduct reference tests
• Return beach water quality
screening equipment to vendors
NA
NA
NA
NA
Data Analysis and Reporting
• Submit TQAP for vendor review
• Revise TQAP and submit final TQAP for
vendor approval
• Distribute final TQAP and post on ETV
website
• Begin preparation of report template
• Review and compile testing data and records
as they become available
• Review and summarize verification testing
staff observations
• Complete common sections of reports
• Evaluate and analyze data generated during
testing
• Complete draft reports and submit for vendor
and peer reviews
• Revise draft reports and submit final reports
for EPA approval
• Distribute finalized, EPA approved reports and
post ETV reports and verification statements
on website
NA = not applicable
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 22 of 48
Version 1.0
Date: 8/31/2007
A3.3 Test Location
The majority of a verification test conducted under this protocol will be performed in a
laboratory setting. The test facility could be laboratory facilities at Battelle or another
organization that can accommodate testing of beach water quality screening technologies and
should be in close proximity to the beach sampling sites selected for the test. Use of a facility
that conducts beach water quality monitoring programs may facilitate collection of water
samples for use in testing and in operation of the screening technologies. Sample collection and
field portability checks will be performed at the selected beach sampling field sites.
A3.4 Health and Safety
All sampling, reference analyses, and verification testing will follow the safety and health
protocols in place for the test facility and the rapid screening technologies undergoing
verification. This includes maintaining a safe work environment and a current awareness of
microbiological pathogens. Exposure to microorganisms will be minimized, personal protective
equipment will be worn, and safe laboratory practices will be followed.
A4 QUALITY OBJECTIVES AND CRITERIA FOR MEASUREMENT DATA
In performing the verification test, Battelle will follow the technical and QA procedures
specified in this protocol and will comply with the data quality requirements in the AMS Center
QMP.(1) The objective of this verification test is to evaluate the performance of beach water
quality screening technologies under realistic operating conditions for each technology being
tested. This evaluation will assess the capabilities of the screening technologies for determining
one or more indicator bacteria in both laboratory-prepared and real-world environmental
samples, and will include a comparison of the beach water quality screening technology results
to those of a reference method such as membrane filtration, which is described in Section B4.
Additionally, this evaluation will rely upon verification testing staff observations to assess other
performance characteristics of the beach water quality screening technologies. Below is a
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 23 of 48
Version 1.0
Date: 8/31/2007
discussion of the quality objectives and the criteria for measurement data that have been
established to ensure that the objectives of this test are met.
A4.1 Quality Objectives
The data quality objectives indicate the minimum quality of data required to meet the
objectives of the beach water quality screening technology verification. The data quality
objectives for this verification test include those for the preparation and analysis of reference
samples, for the operation of the beach water quality screening technologies, as well as for the
documentation of verification testing staff observations. The data quality objectives for the
preparation and analysis of reference samples are based on the requirements of the reference
methods described in Section B4, and are presented in terms of data quality indicator (DQI)
criteria for the critical measurements associated with the reference methods. The data quality of
the reference samples relies, in part, on the proper preparation of the samples and proper
application of the reference method. The data quality of the test samples relies on proper
preparation of the test samples and proper operation of the beach water quality screening
technologies.
A4.2 Criteria for Measurement Data
Table 3 presents the DQIs and criteria for the critical measurements of the reference
method(s). The quality of the reference measurements will be ensured by adherence to these DQI
criteria. The quality of the reference measurements will be monitored by sterility checks of
media, reagents, membrane filters, and sample containers and by inclusion of method blank
samples, negative control samples, and positive control samples for each reference method as
applicable. The quality of the screening technology measurement data will be monitored by any
standard assessments, such as method blanks and positive and negative controls, identified by the
screening technology vendors. The Battelle Quality Manager or his designee will perform a
technical systems audit (TSA) at least once during this verification test to augment these QA/QC
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 24 of 48
Version 1.0
Date: 8/31/2007
requirements. The EPA Quality Manager also may conduct an independent TSA, at his or her
discretion.
Table 3. Data Quality Indicators and Criteria for Critical Measurements for Reference
Method(s)
DQI
Precision
Bias and Accuracy
Selectivity
Completeness
Method
Representativeness
Method of
Assessment
Triplicate analyses
of split samples
Positive Control
Samples
Laboratory Control
Standards (LCS)
Matrix Spikes (MS)
Sterility of media,
reagents and filters
Method Blank
Negative control
Positive control
Amount of valid
data obtained
Performance
Evaluation (PE)
Test
Frequency
As required in
the reference
method
As required in
the reference
method
As required in
the reference
method
Overall
number of
data points
collected for
reference
method
Once, prior to
verification
testing
Minimum
Acceptance Criteria
Refer to reference
method criteria
Refer to reference
method criteria
Refer to reference
method criteria
80% of overall data
points collected should
be valid
Results within ± 25%
of expected value for
standard solutions,
results within certified
limits for standard
reference materials
Corrective Action
Investigate sources of
contamination or
changes in instrument
parameters; perform
instrument maintenance
as needed; reanalyze
fresh standard or sample,
or repeat initial
calibration
If feasible, analyze
additional samples to
meet the acceptance
criterion
Evaluate reference
method performance;
perform maintenance or
recalibrate as required,
repeat performance
evaluation test. If
performance evaluation
test criteria cannot be
met, consider alternative
reference laboratory
A5 SPECIAL TRAINING/CERTIFICATION
Documentation of training related to technology testing, field testing, data analysis, and
reporting is maintained for all Battelle technical staff in training files at their respective
locations. Documentation of the expertise and experience of collaborators and/or subcontractors
must be similarly available. The Battelle Quality Manager may verify the presence of
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 25 of 48
Version 1.0
Date: 8/31/2007
appropriate training records prior to the start of testing. If technical staff operate and/or maintain
a beach water quality screening technology during the verification test, the beach water quality
screening technology vendor will be required to train those staff prior to the start of testing.
Battelle will document this training with a consent form, signed by the vendor, that states which
specific technical staff have been trained on their technology. Battelle technical staff will have a
minimum of a bachelor's degree in science/engineering or have equivalent work experience.
A6 DOCUMENTATION AND RECORDS
The records for this verification test will include this protocol, any TQAP based on this
protocol, chain-of-custody (COC) forms, laboratory record books (LRB), data collection forms,
electronic files (both raw data and spreadsheets), and the final verification reports and
verification statements. All of these records will be maintained in the Verification Test
Coordinator's office or at the test facility during the test and may be transferred to permanent
storage at Battelle's Records Management Office at the conclusion of the verification test. The
location (e.g., specific personal computer, server, or media type and storage location) of final
versions of the electronic files will be noted in the test records. All Battelle LRBs are stored
indefinitely, either by the Verification Test Coordinator or Battelle's Records Management
Office. EPA will be notified before disposal of any files. The QA/QC documentation and results
of the reference measurements made by the reference laboratory will be submitted to Battelle
immediately upon completion of all sample analyses and maintained with the records for this
test. Table 4 has further details regarding the data recording practices and responsibilities.
All written records must be in ink. Any corrections to notebook entries, or changes in
recorded data, must be made with a single line through the original entry. The correction is then
to be entered, initialed, and dated by the person making the correction. In all cases, strict
confidentiality of data from each vendor's beach water quality screening technology, and strict
separation of data from different technology, will be maintained. Separate files (including
manual records, printouts, and/or electronic data files) will be kept for each technology.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 26 of 48
Version 1.0
Date: 8/31/2007
Table 4. Summary of Data Recording Process
Data to Be
Recorded
Dates, times, and
details of test events,
technology
maintenance, down
time, etc.
Beach water quality
screening technology
calibration
information
Beach water quality
screening technology
readings
Sample collection
and reference
method analysis
procedures,
calibrations, QA, etc.
Reference method
results
Where Recorded
ETVLRBsordata
recording forms
ETVLRBs,data
recording forms, or
electronically
Either recorded
electronically by the
technology and
downloaded to an
independent
computer or other
storage media; hard
copy data printed by
the technology and
taped into the ETV
LRB; or hand-
recorded data on data
sheets or in the ETV
LRB
LRBs, chain-of-
custody, or other
data recording forms
Electronically from
analytical method or
hand-recorded on
data recording forms
or ETV LRB
How Often
Recorded
Start/end of test
procedure, and at
each change of a test
parameter or change
of beach water
quality screening
technology status
At technology
calibration or
recalibration
Every sample
analysis
Throughout sampling
and analysis
processes
Every sample
analysis
By Whom
Technical staff
Technical staff or
vendor performing
the calibration.
Technical staff or
vendor for transfer to
Battelle
Technical staff and
Reference laboratory
Reference laboratory
Disposition of
Data
Used to organize and
check test results;
manually
incorporated in data
spreadsheets as
necessary
Incorporated in
verification report as
necessary
Transferred to or
manually entered
into spreadsheet for
statistical analysis
and comparisons
Retained as
documentation of
reference method
performance
Transferred to or
manually entered
into spreadsheets for
calculation of results,
and statistical
analysis and
comparisons
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 27 of 48
Version 1.0
Date: 8/31/2007
SECTION B.
MEASUREMENT AND DATA ACQUISITION
Bl EXPERIMENTAL DESIGN
This test will specifically address verification of screening technologies that provide
quantitative or qualitative measurements of indicator bacteria, such as Enterococci and E. coli.
Screening technologies may provide qualitative results indicating the presence or absence of the
indicator bacteria within a specified concentration interval or quantitative results reporting
concentrations using a digital display or an electronic output signal. The vendor will specify in
which category their technology should be included, which indicator bacteria it is designed to
detect, and in which matrices (i.e., fresh, brackish, and/or marine/estuarine water) the technology
is designed to operate.
The screening technologies will be verified by subjecting them to various concentration
levels of individual indicator bacteria in several matrices: deionized (DI) water and/or artificial
sea water (ASW) (prepared from DI water and a sea salt mixture). The technologies will also be
subjected to two concentrations (including zero) of the indicator bacteria in the presence of
possible interferents (e.g., mixed humic acids, and naturally occurring non-target marine and/or
fresh water bacteria) in the appropriate sample matrix. Samples will be prepared by Battelle, the
reference laboratory, or other qualified collaborating organization. Finally, each technology will
analyze a variety of natural water samples, including samples collected from locations that have
typically exceeded standards for bacterial contamination. These samples may be from fresh,
brackish, or marine/estuarine water locations, as appropriate for the screening technologies
undergoing verification; however, this is not intended to be an exhaustive study or to represent
all possible samples types that could be tested.
The screening technologies will be evaluated for the following parameters, as appropriate
for each technology:
• Accuracy
• Reproducibility
• Linearity
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 28 of 48
Version 1.0
Date: 8/31/2007
• Comparability relative to accepted laboratory-based methods (e.g., membrane
filtration)
• Occurrence of false positive and false negative results relative to current action levels
as determined by the reference method result (e.g., frequency of technology reporting
Enterococci levels above 104 cfu/100 mL in marine water when the level is actually
below, and vice versa)
• Interference effects and selectivity
• Field portability
• Data completeness
• Operational factors, such as ease of use, waste production, maintenance, and
throughput.
Qualitative screening technologies will not be evaluated for linearity or comparability
since they only provide a positive or negative result relative to a specified concentration level.
Technologies that are not designed for use outside of the laboratory will not be evaluated for
field portability.
Bl.l Test Procedures
This verification test will determine the performance capability of the screening
technologies to detect individual contaminants in two types of samples - performance test (PT)
and natural water. The number of each sample type identified in this protocol should be
regarded the minimum number; additional samples should be considered based on the variance
of each method. PT samples will include all the samples prepared in DI water or ASW. The
indicator PT samples will be spiked with indicator bacteria at four target concentrations listed in
Table 5. The results from triplicate analysis of each indicator PT sample and comparison with
the concentrations determined by the reference method will provide information on accuracy,
precision, linearity, comparability, and false-positive/false-negative rates of the screening
technologies.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 29 of 48
Version 1.0
Date: 8/31/2007
Table 5. Spike Levels for Indicator PT Samples
Indicator Organism
Enterococci
E. coli
Water Type
Fresh water
Marine water
Fresh water
Target Spike Concentrations (cfu/100 mL)
0, 35, 104, 1000
0.33.61, 1000
0, 126, 235, 1000
One type of interferent PT sample will consist of mixed humic acid in DI and/or ASW,
both spiked and unspiked with each indicator bacterium. Another type of interferent PT sample
will test the selectivity of the screening technology to the target indicator bacteria in the presence
of natural non-target bacteria. This will be assessed by analyzing samples that contain the non-
target bacteria, both spiked and unspiked with the target bacterial indicator.
Natural water samples will be collected from at least five locations, including fresh,
brackish, and sea water as needed based on the screening technologies being verified (e.g., if
only technologies designed for sea water analysis are being evaluated, only sea water samples
should be tested), so that each technology is subjected to a minimum of thirty (30) natural water
samples in an appropriate matrix. Each natural water sample will be analyzed in duplicate.
These samples will be used to evaluate comparability to accepted laboratory-based methods
(e.g., membrane filtration) and the occurrence of false-positive and false-negative results relative
to current action levels. Samples should be collected using procedures and in locations that are
representative of real-world beach monitoring programs. If known or potential pollution sources
exist, additional samples may be collected as appropriate from those sources. Sample collection
is further described in Section B2. A summary of the test samples is shown in Table 6.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 30 of 48
Version 1.0
Date: 8/31/2007
Table 6. Summary of Test Samples
Sample
Type
Indicator PT
Interferent PT
Natural water
Performance
Parameter
Accuracy
Precision
Linearity (b)
False Positives
False Negatives
Interference
effects
Selectivity
Comparability^
False Positives
False Negatives
Sample Matrix
DI water and/or ASW
DI water and/or ASW with
0.001% (w/v) mixed humic
acids
DI water and/or ASW with
nontarget bacteria at
ambient concentrations
Natural fresh, brackish,
and/or sea water
Bacterial Indicator
Target Concentration
(cfu/100 mL)
0
35
104
1000
0
104
0
104
Unspiked
Approximate total number of samples per indicator bacteria
Minimum
Number
of samples
30
38
Reps(a)
3
3
3
3
3
3
3
3
2
84
(a) Reps = number of replicates
(b) Not determined for qualitative screening technologies
All screening technologies will be tested in the laboratory; applicable screening
technologies will also be evaluated for their performance and ease of use outside of the
laboratory. Field portability will be evaluated by analyzing the natural samples with the
screening technology in the field, if possible, at the time of sample collection in addition to in the
laboratory at the same time as the reference method measurements. It should be noted that the
laboratory-based analyses may be performed up to six (6) hours after sample collection and the
field portability analyses are performed; therefore, some differences in the bacterial
concentrations may occur. (The allowable holding time for recreational water samples is 6
hours.)
No additional test procedures will be carried out specifically to address data
completeness. This parameter will be assessed based on the overall data return achieved by each
technology as a percentage of the maximum possible data return.
Operational factors such as maintenance needs, calibration frequency, data output, ease of
use, repair requirements, and those that impact environmental sustainability (i.e., consumables
used, waste production, analysis time, etc.), will be evaluated based on observations recorded by
the technical staff. A separate LRB will be maintained at the test site for each technology
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 31 of 48
Version 1.0
Date: 8/31/2007
undergoing testing; daily observations on operational factors will be recorded in the LRB or on
data sheets. Examples of information to be recorded in the record books or on data sheets include
the daily status of diagnostic indicators for the technology; use or replacement of any
consumables; the effort or cost associated with maintenance or repair; vendor effort (e.g., time
on site) for repair or maintenance; the duration and causes of any technology down time or data
acquisition failure; and operator observations about ease of use of the technology. These
observations will be summarized to aid in describing the technology performance in the
verification report on each technology.
B1.2 Statistical Analysis
The statistical methods and calculations used for evaluation of the quantitative
performance parameters are described in the following sections.
B1.2.1 Accuracy
For screening technologies that provide quantitative results, PT and natural water sample
accuracy will be assessed relative to the reference method result. The results for the three
replicate analyses will be averaged, and the accuracy will be expressed in terms of a percent
recovery (R) as calculated from Equation 1:
R = (C/CR)xlOO (1)
where C is the average concentration measured by screening technology and CRis the reference
method result for the PT or natural water sample.
For qualitative results, accuracy will be assessed by evaluating how often the screening
technology result correctly reports the presence of the indicator bacteria above the screening
technology's reported limit of detection as determined by the reference method result. An overall
percent agreement will be determined by dividing the number of correctly identified positive
responses to the overall number of analyses.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 32 of 48
Version 1.0
Date: 8/31/2007
Bl.2.2 Reproducibility
For screening technologies that provide quantitative results, the standard deviation (S) of
the results for the replicate samples will be calculated for each sample using Equation 2:
S =
(2)
where n is the number of replicate samples, Ckis the concentration measured for the kth sample,
and C is the average concentration of the replicate samples. The precision for each sample will
be reported in terms of the relative standard deviation (RSD) or coefficient of variation as
calculated using Equation 3:
%RSD =
C
xlOO (3)
For screening technologies that provide qualitative results, precision will be assessed by
calculating the percentage of consistent responses.
Bl.2.3 Linearity
For screening technologies that provide quantitative results, linearity will be assessed by
performing a linear regression with the reference method result from the indicator PT samples as
the independent variable, and the individual screening technology result as the dependent.
Individual replicate results will be used in the linear regression. Linearity will be expressed in
terms of the slope, intercept, and coefficient of determination (r2). Linearity will not be
determined for qualitative screening technologies.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 3 3 of 48
Version 1.0
Date: 8/31/2007
Bl.2.4 Comparability
Comparability between the screening technology results and the reference method results
will be assessed by linear regression using the reference method concentrations from the natural
water samples as the independent variable and results from the screening technologies being
evaluated as the dependent variable. Comparability will be expressed in terms of slope,
intercept, and r2. Comparability will also be calculated using Equation 1 and reported as a
percent recovery. It is expected that the measured indicator bacteria concentrations will vary by
at least a factor of five for the natural water samples included in this test. However, if this
magnitude of variation is not achieved for the reference method results, comparability will be
expressed only as a percent recovery. As appropriate, tests of statistical significance may be
used in addition to or in place of linear regression analysis. Comparability will not be
determined for qualitative screening technologies.
Bl.2.5 False Positive/False Negative Responses
For both quantitative and qualitative screening technologies, a false-positive response is
defined as a positive screening technology response with respect to the appropriate action level
when the reference method result was below the action level. A false-negative response is
defined as a response lower than the appropriate action level when the reference method result
was greater than the action level. Indicator PT, Interferent PT, and natural water samples will be
included in this analysis.
B1.2.6 Interference Effects and Selectivity
For both quantitative and qualitative screening technologies, interference effects and
selectivity will be determined as the percent of samples that correctly report the presence or
absence of the indicator bacteria in solutions containing humic acids and nontarget bacteria,
respectively. For qualitative screening technologies, the percent recovery may also be reported.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 34 of 48
Version 1.0
Date: 8/31/2007
Bl.2.7 Data Completeness
Data completeness will be calculated as the percentage of the total possible data return
that was actually generated by each technology. This calculation will use the total number of
samples with data returned by the technology, divided by the total number of samples included in
the verification test. The causes of any substantial loss of data return will be established from
operator observations or technology records, and noted in the discussion of data completeness
results.
Bl.2.8 Operational Factors
There are no statistical calculations applicable to operational factors. Operational factors
will be determined based on documented observations of the technical staff and the Verification
Test Coordinator.
Bl.2.9 Field Portability
The results obtained from the measurements made on samples in the laboratory and field
setting will be compiled independently for each technology and compared to assess the accuracy
of the measurements under the different analysis conditions. Means and standard deviations of
the endpoints generated in both locations will be used to make the comparison. For qualitative
screening technologies, the number of false positives and false negatives will be used to make
the comparison.
B1.3 Reporting
The statistical comparisons described above will be conducted separately for each of the
technologies being tested, and information on the operational parameters will be compiled and
reported. The data for each technology will be kept separate from data for all other technologies,
and no intercomparison of the technology data will be performed at any time. A separate
verification report will be prepared for each technology tested that presents the test procedures
and test data, as well as the results of the statistical evaluation of those data. Operational aspects
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 3 5 of 48
Version 1.0
Date: 8/31/2007
of the technologies will be recorded by technical staff at the time of observation during the
verification test, and summarized in the verification report. For example, descriptions of the data-
acquisition procedures, use of vendor-supplied proprietary software, consumables used, repairs
and maintenance needed, and the nature of any problems will be presented in the report.
Each verification report will briefly describe the ETV program, the AMS Center, and the
procedures used in verification testing. The results of the verification test will be stated
quantitatively and in comparison to the reference method, but without comparison to any other
technology tested or comment on the acceptability of the technology's performance. Each draft
verification report will be reviewed by the respective technology vendor and by EPA and other
peer reviewers. Comments on the draft report will be addressed in revisions of the report. The
peer review comments and responses will be tabulated to document the peer review process. The
reporting and review process will be conducted according to the requirements of the AMS Center
QMP.(1)
B2 SAMPLING REQUIREMENTS
The following sample design, employed by the Suffolk County Department of Health
Services, is included as an example.(7) If the verification test is being performed in collaboration
with an existing beach monitoring program, its sampling plan may be used.
All samples should be collected at a depth of approximately 18-24 inches (knee depth),
where possible, using 150 mL pre-sterilized plastic (e.g., polypropylene) bottles. Prior to
collecting the sample, individual bottles should be marked with the appropriate beach code on a
pre-fixed label. A COC form should also be completed, noting the beach code; sampling
location; sample matrix (i.e., fresh, brackish, or marine/estuarine water); date and time; current
and past (previous 24-hour) weather; number of bathers, animals, and boats present; and any
unusual conditions noted during sampling. Immediately after collection, samples should be
placed in a cooler with ice and kept at 1°- 4° C. During each sampling event, one additional
sample should be collected to serve as a temperature control (TC) and be labeled accordingly.
Upon arrival at the laboratory, the temperature of this sample will be measured and recorded.
For the samples to be acceptable, the TC must be less than or equal to 10° C.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 36 of 48
Version 1.0
Date: 8/31/2007
Other QC considerations taken during sampling include:
• Take care to avoid contamination by not touching the inside or rim of the sample
bottle and cap.
• Be careful not to disturb sediments when wading out to the appropriate sampling
depth.
• Avoid any surface scum layer by removing the lid after the bottle has been plunged
downward to the appropriate sampling depth.
• If there is a water current present, move to a position downstream to collect the
incoming flow to prevent contamination of or effect on the sample.
• Pour off a small portion of the sample before capping the sample bottle to provide an
air-space that will facilitate later mixing.
• Do not allow the bottle to become immersed in cooler melt-water during transit or
storage.
Samples will be delivered to the test facility as soon as possible, so that the appropriate analyses
can be started within 6 hours of sample collection. Sample analysis should be completed within
8 hours of sample collection.
B3 SAMPLE HANDLING AND CUSTODY REQUIREMENTS
Sample custody will be documented throughout collection, transport, shipping (if
necessary), and analysis of the samples, using COC forms. Each COC form summarizes the
samples collected and analyses requested. The COC form will track sample release from the
sampling location to the test facility and/or reference laboratory. COC forms will be used
regardless of whether the samples are being transferred within the test facility or to an external
location. Each COC form will be signed by the person relinquishing samples once that person
has verified that the COC form is accurate. The original sample COC forms will accompany the
samples; the shipper will keep a copy. Upon receipt at the test facility and/or reference
laboratory, COC forms will be signed by the person receiving the samples once that person has
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 37 of 48
Version 1.0
Date: 8/31/2007
verified that all samples identified on the COC forms are present and intact. Any discrepancies
will be noted on the form and the sample receiver will immediately contact the Verification Test
Coordinator to report missing, broken, leaking, or compromised samples. Copies of all COC
forms will be delivered to the Verification Test Coordinator, and maintained with the test
records.
B4 LABORATORY REFERENCE METHODS
The reference laboratory will perform standard methods for determination of indicator
bacteria (i.e., Enterococci, E. coli, etc.) as the reference methods against which beach water
quality screening technology data will be compared. The following are examples of reference
methods which may be appropriate for beach water quality screening.
For fresh and salt water samples, Enterococci may be determined using EPA Method
1600: Enterococci in Water by Membrane Filtration Using Membrane- Enterococcus Inoxyl-p-
D-Glucoside Agar.(8) This method provides a direct count of bacteria in water based on the
development of colonies on the surface of a 0.45 |j,m pore size membrane filter, which retains the
bacteria. Following filtration, the membrane containing the bacterial cells is placed on a
selective medium, mEI agar, and incubated for 24 hours at 41° ± 0.5 °C. All colonies (regardless
of color) with a blue halo are recorded as Enterococci colonies. Magnification and a small
fluorescent lamp are used for counting to give maximum visibility of colonies.
E. coli may be determined using EPA Method 1603: Escherichia coli (E. coli) in Water
by Membrane Filtration Using Modified membrane-Thermotolerant Escherichia coli Agar
(mTEC).(9) This method provides a direct count of E. coli in water based on the development of
colonies that grow on the surface of a membrane filter. A water sample is filtered through a
0.45 |j,m pore size membrane filter, which retains the bacteria. After filtration, the membrane
containing the bacteria is placed on a selective and differential medium, modified mTEC Agar,
incubated at 35° ± 0.5 °C for 2 hours to resuscitate the injured or stressed bacteria, and then
incubated at 44.5° ± 0.2 °C for 22 hours. The target colonies on modified mTEC agar are red or
magenta in color after both incubation periods.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 3 8 of 48
Version 1.0
Date: 8/31/2007
B5 QUALITY CONTROL
Steps will be taken to maintain the quality of the data collected during this verification
test. As described in Section B4, the reference laboratory will follow standard reference methods
for the determination of pathogen indicator organisms. Quality control samples (QCSs) will
include sterility checks of media, reagents, membrane filters, and sample containers; method
blanks; negative controls; and positive controls, as appropriate for the reference method(s). QCSs
producing results not meeting the reference method or the reference laboratory's standard
requirements will be reanalyzed or reported with qualifiers if reanalysis is not possible. If the
outlying results persist, the affected data will be flagged and a repeat of the affected parts of the
verification test may be considered. Sample results not meeting these requirements will be
flagged and excluded from comparison to the beach water quality screening technology results.
Reference measurements for a minimum of 30 natural water and 8 laboratory-prepared samples
for each bacterial indicator used for evaluating a beach water quality screening technology must
meet these QC requirements for use in this test.
B6 INSTRUMENT/EQUIPMENT TESTING, INSPECTION, AND MAINTENANCE
The equipment used by the test facility and/or reference laboratory will be tested,
inspected, and maintained as per the standard operating procedures of the test facility and/or
reference laboratory and/or the manufacturer's recommendations to meet the performance
requirements established in this document. Examples of such equipment include, but are not
limited to, laboratory balances, autoclaves, microscopes, thermometers, and incubators. When
technical staff operate and maintain technologies undergoing testing, those activities will follow
directions provided by the technology vendor. Otherwise, operation and maintenance of the
technologies will be the responsibility of the technology vendor.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 3 9 of 48
Version 1.0
Date: 8/31/2007
B7 CALIBRATION/VERIFICATION OF TEST PROCEDURES
Systems used for reference analyses will be calibrated as appropriate before any reference
samples are analyzed and recalibrated as needed based on the reference methods and/or reference
laboratory standard operating procedures (SOPs).
Technologies undergoing testing will be calibrated initially by the respective technology
vendor at the time of installation at the test facility, as appropriate. Calibration checks will be
performed upon direction of the vendor. In the event that recalibration is necessary, the
recalibration will be carried out by the technology vendor, or by technical staff under the
direction of the vendor. All calibrations will be documented as appropriate by the technical staff
or vendor.
B8 INSPECTION/ACCEPTANCE OF SUPPLIES AND CONSUMABLES
Required materials, supplies, and consumables will be ordered by the Verification Test
Coordinator or designee. Where possible, Battelle will rely on sources of materials and
consumables that have been used previously without problems as part of ETV verification
testing. Battelle will also rely on previous experience or recommendations from EPA advisors,
stakeholders, test collaborators, subcontractors, or beach water quality screening technology
vendors. When possible, National Institute of Standards and Technology (NIST)-traceable
materials will be used. Upon receipt of any supplies or consumables, the Verification Test
Coordinator or designee will visually inspect and ensure that the materials received are those that
were ordered and that there are no visual signs of damage that could compromise the suitability
of the materials. Certificates of analysis (CO A) or other documentation of analytical purity will
be checked for all reagents and standards to ensure suitability for the verification test and will be
included in the test files. If damaged, unsuitable, or inappropriate goods are received, they will
be returned or disposed of and arrangements will be made to receive replacement materials.
B9 NON-DIRECT MEASUREMENTS
No non-direct measurements or existing data will be used during this verification test.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 40 of 48
Version 1.0
Date: 8/31/2007
BIO DATA MANAGEMENT
Various types of data will be acquired and recorded electronically or manually by
Battelle, vendor, collaborator, and/or subcontractor staff during the verification test. Table 4
summarizes the types of data to be recorded. All maintenance activities, repairs, calibrations, and
operator observations relevant to the operation of the screening technologies will be documented
by technical staff in LRBs or on data sheets. A separate record book will be maintained for each
participating technology. Results from the reference methods, including raw data, analyses, and
final results, will be compiled by the reference laboratory, preferably in electronic format, and
submitted to Battelle at the conclusion of reference method testing.
Records received by or generated by any technical staff during the verification test will
be reviewed by a Battelle staff member within two weeks of generation or receipt, and before the
records are used to calculate, evaluate, or report verification results. If a Battelle staff member
generated the record, this review will be performed by a Battelle technical staff member involved
in the verification test, but not the staff member who originally generated the record. The review
will be documented by the person performing the review by adding his/her initials and date to the
hard copy of the record being reviewed. In addition, any calculations performed by technical
staff will be spot-checked by Battelle QA and/or technical staff to ensure that calculations are
performed correctly. Calculations to be checked include any statistical calculations described in
this protocol and any TQAP based on this protocol. The data obtained from this verification test
will be compiled and reported independently for each beach water quality screening technology.
Results for technologies from different vendors will be compared individually to the reference
method, but will not be compared with each other.
Among the QA activities conducted by Battelle QA staff will be an audit of data quality.
This audit will consist of a review by the Battelle Quality Manager or designee of at least 10% of
the test data. During the course of any such audit, the Battelle Quality Manager will inform the
technical staff of any findings and any need for immediate corrective action. If serious data
quality problems exist, the Battelle Quality Manager will request that Battelle's AMS Center
Manager issue a stop work order. The results of the data quality audit will be included in an
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 41 of 48
Version 1.0
Date: 8/31/2007
assessment report. The Verification Test Coordinator will ensure that a response is provided for
each adverse finding or potential problem, and will implement any necessary follow-up
corrective action. The Battelle Quality Manager will ensure that follow-up corrective action has
been taken.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 42 of 48
Version 1.0
Date: 8/31/2007
SECTION C.
ASSESSMENT AND OVERSIGHT
Cl ASSESSMENTS AND RESPONSE ACTIONS
Every effort will be made in this verification test to anticipate and resolve potential
problems before the quality of performance is compromised. One of the major objectives of this
protocol is to establish mechanisms necessary to ensure this. Internal QC measures described in
this protocol, which is peer reviewed by a panel of outside experts, implemented by the technical
staff and monitored by the Verification Test Coordinator, will give information on data quality
on a day-to-day basis. The responsibility for interpreting the results of these checks and resolving
any potential problems resides with the Verification Test Coordinator. Technical staff have the
responsibility to identify problems that could affect data quality or the ability to use the data.
Any problems that are identified will be reported to the Verification Test Coordinator, who will
work with the Battelle Quality Manager to resolve any issues. Action will be taken to control the
problem, identify a solution to the problem, and minimize losses and correct data, where
possible. Independent of any EPA QA activities, Battelle will be responsible for ensuring that the
following audits are conducted as part of this verification test.
Cl.l Performance Evaluation Audits
A Performance Evaluation (PE) audit will be conducted to assess the quality of the
reference method measurements made in this verification test. The PE audit of the reference
methods will be performed by supplying each reference method a blind sample or standard
reference material containing the bacterial indicator. The PE audit samples will be analyzed in
the same manner as for all other samples and the analytical results for the PE audit samples will
be compared to the nominal concentration. The target criterion for this PE audit is agreement of
the analytical result within 25% of the nominal concentration (by percent difference). If the PE
audit results do not meet the tolerances shown, they will be repeated. If the outlying results
persist, a change in the instrument used for the reference method and a repeat of the PE audit
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 43 of 48
Version 1.0
Date: 8/31/2007
may be considered. This audit will be performed once during the verification test, and will be the
responsibility of the Verification Test Coordinator or designee.
C1.2 Technical Systems Audits
The Battelle Quality Manager or designee will perform a TSA at least once during this
verification test. The purpose of this audit is to ensure that the verification test is being
performed in accordance with the AMS Center QMP,(1) this protocol and/or TQAP, published
reference methods, and any SOPs used by the reference laboratory. In the TSA, the Battelle
Quality Manager, or a designee, will review the reference methods used, compare actual test
procedures to those specified or referenced in this protocol and/or TQAP, and review data
acquisition and handling procedures. The Battelle Quality Manager or designee will tour the test
facility, observe sample collection (if possible), inspect documentation of sample COC; and
review beach water quality screening technology-specific record books. He or she will also
check standard certifications and technology data acquisition procedures, and may confer with
the technology vendors, reference laboratory, and technical staff. The Battelle Quality Manager
may also visit the reference laboratory to review procedures and adherence to this protocol, any
TQAP based on this protocol, and applicable SOPs. A TSA report will be prepared, including a
statement of findings and the actions taken to address any adverse findings. The EPA AMS
Center Quality Manager will receive a copy of Battelle's TSA report. At EPA's discretion, EPA
QA staff may also conduct an independent on-site TSA during the verification test. The TSA
findings will be communicated to technical staff at the time of the audit and documented in a
TSA report.
C1.3 Data Quality Audits
The Battelle Quality Manager or designee will audit at least 10% of the verification data
acquired in the verification test. The Battelle Quality Manager will trace the data from initial
acquisition, through reduction and statistical comparisons, to final reporting. All calculations
performed on the data undergoing the audit will be checked.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 44 of 48
Version 1.0
Date: 8/31/2007
C1.4 QA/QC Reporting
Each assessment and audit will be documented in accordance with Sections 3.3.4 and
3.3.5 of the AMS Center QMP.(1) The results of the technical systems audit will be submitted to
EPA. Assessment reports will include the following:
• Identification of any adverse findings or potential problems
• Response to adverse findings or potential problems
• Recommendations for resolving problems
• Confirmation that solutions have been implemented and are effective
• Citation of any noteworthy practices that may be of use to others.
C2 REPORTS TO MANAGEMENT
The Battelle Quality Manager, during the course of any assessment or audit, will identify
to the technical staff performing experimental activities any immediate corrective action that
should be taken. If serious quality problems exist, the Battelle Quality Manager is authorized to
request that Battelle's AMS Center Manager issue a stop work order. Once the assessment report
has been prepared, the Verification Test Coordinator will ensure that a response is provided for
each adverse finding or potential problem and will implement any necessary follow-up corrective
action. The Battelle Quality Manager will ensure that follow-up corrective action has been taken.
The test protocol and final verification report(s) are reviewed by EPA AMS Center QA staff and
EPA AMS Center program management staff. Upon final review and approval, both documents
will be posted on the ETV website (www.epa.gov/etv).
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 45 of 48
Version 1.0
Date: 8/31/2007
SECTION D.
DATA VALIDATION AND USABILITY
Dl DATA REVIEW, VALIDATION, AND VERIFICATION REQUIREMENTS
The key data review requirements for the verification test are stated in Section BIO of
this protocol. In general, the data review requirements specify that the data generated during this
test will be reviewed by a Battelle technical staff member within two weeks of generation of the
data. The reviewer will be familiar with the technical aspects of the verification test, but will not
be the person who generated the data. This process will serve both as the data review and the
data verification, and will ensure that the data have been recorded, transmitted, and processed
properly. Furthermore, this process will ensure that the beach water quality screening
technology data and the reference method data were collected under appropriate testing
conditions and that the reference method data meet the specifications of the reference method.
The data validation requirements for this test involve an assessment of the data quality
relative to the DQIs and audit acceptance criteria specified for this test. The DQIs listed in
Section B5 will be used to validate the quality of the data. The QA audits described within
Section C of this document, including the performance evaluation audit and audit of data quality,
are designed to validate the quality of the data.
D2 VALIDATION AND VERIFICATION METHODS
Data verification is conducted as part of the data review, as described in Section BIO for
this protocol. A visual inspection of handwritten data will be conducted to ensure that all entries
were properly recorded or transcribed, and that any erroneous entries were properly noted (i.e.,
single line through the entry with an error code and the initials of the recorder and date of entry).
Electronic data from the screening technologies and other instruments used during the test will
be inspected to ensure proper transfer from the datalogging system. Data manually incorporated
into spreadsheets for use in calculations will be checked against handwritten data to ensure that
transcription errors have not occurred. All calculations used to transform the data will be
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 46 of 48
Version 1.0
Date: 8/31/2007
reviewed to ensure the accuracy and the appropriateness of the calculations. Calculations
performed manually will be reviewed and repeated using a handheld calculator or commercial
software (e.g., Excel). Calculations performed using standard commercial office software (e.g.,
Excel) will be reviewed by inspecting the equations used in calculations and verification of
selected calculations by handheld calculator. Calculations performed using specialized
commercial software (i.e., for analytical instrumentation) will be reviewed by inspection and,
when feasible, verified by handheld calculator, or standard commercial office software.
To ensure that the data generated from this test meet the goals of the test, a number of
data validation procedures will be performed. Section C of this protocol provides a description
of the validation safeguards employed for this verification test. Data validation and verification
efforts include the completion of QC activities, and the performance of ISA and PE audits as
described in Section C. The data from this test will be evaluated relative to the measurement
DQIs described in Section B5, and the PE audit acceptance criteria given in Section Cl.l of this
test/QA Plan. Data failing to meet these criteria will be flagged in the data set and not used for
evaluation of the beach water quality screening technologies, unless these deviations are
accompanied by descriptions adequately demonstrating that data quality was not compromised.
An audit of data quality will be conducted by the Battelle Quality Manager to ensure that
data review, verification, and validation procedures were completed, and to assure the overall
data quality.
D3 RECONCILIATION WITH USER REQUIREMENTS
The purpose of a verification test performed following this protocol is to evaluate the
performance of commercial beach water quality screening technologies. In part, this evaluation
will include comparisons of the results from the screening technologies to results from
established reference methods. To meet the requirements of the user community, the data
obtained in such a verification test should include thorough documentation of the performance of
the screening technologies during the verification test. The data review, verification, and
validation procedures described above will ensure that verification test data meet these
requirements, are accurately presented in the verification reports generated from the test, and that
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 47 of 48
Version 1.0
Date: 8/31/2007
data not meeting these requirements are appropriately flagged and discussed in the verification
reports. Additionally, all data generated using reference methods that are used to evaluate beach
water quality screening technology results during the verification test should meet the QA
requirements of the reference methods.
This generic verification protocol and any resulting ETV verification report(s) generated
following procedures described in this protocol will be reviewed by participating beach water
quality screening technology vendors, ETV AMS Center staff, test collaborators, EPA, and
external expert peer reviewers. These reviews will ensure that this protocol, verification test(s) of
beach water quality screening technologies, and the resulting report(s) meet the needs of
potential users and permitters of beach water quality screening technologies. The final report(s)
will be submitted to EPA in Microsoft Word and 508-compliant Adobe Portable Document
Format (pdf) and subsequently posted on the ETV website.
-------
Rapid Beach Water Quality Screening Technologies
Generic Verification Protocol
Page 48 of 48
Version 1.0
Date: 8/31/2007
SECTION E.
REFERENCES
1. Quality Management Plan for the ETV Advanced Monitoring Systems Center, Version 6.0,
U.S. EPA Environmental Technology Verification Program, Battelle, Columbus, Ohio,
November 2005.
2. Environmental Technology Verification Program Quality Management Plan,
EPA/600/R-03/021, U.S. Environmental Protection Agency, Cincinnati, Ohio, December
2002.
3. Water Quality Standards for Coastal and Great Lakes Recreation Waters; Final Rule 40 CFR
Part 131. Federal Register, Vol. 69, No. 220, November 2004.
4. T. J. Wade, R. L. Calderon, E. Sams, M. Beach, K. P. Brenner, A. H. Williams, A. P. Dufour.
Rapidly Measured Indicators of Recreational Water Quality Are Predictive of Swimming-
Associated Gastrointestinal Illness, Environmental Health Perspectives, 114, 24-28, 2006.
5. J.F. Griffith and S.B. Weisberg. Evaluation of Rapid Microbiological Methods for
Measuring Recreational Water Quality. Technical Report 486, Southern California Coastal
Water Research Project. Costa Mesa, CA, 2006.
6. J.F. Griffith, D. Moore, C. McGee, S.B. Weisberg. Beta testing of rapid methods for
measuring beach water quality. Technical Report 506, Southern California Coastal Water
Research Project. Costa Mesa, CA, 2007.
7. Bathing Beach Monitoring and Notification Program Workplan, Suffolk County Department
of Health Services, New York, December 2005.
8. Method 1600: Enterococci in Water by Membrane Filtration Using membrane-Enterococcus
Indoxyl-|3-D-Glucoside Agar (mEI), EPA 821-R-02-022, U. S. Environmental Protection
Agency, Washington DC, September 2002.
9. Method 1603: Escherichia coll (E. coli) in Water by Membrane Filtration Using Modified
membrane-Thermotolerant&c/ZOT'c/z/'a coli Agar (Modified mTEC), EPA-821-R-04-025,
U. S. Environmental Protection Agency, Washington DC, April 2005.
------- |