&EPA
United States
Environmental Protection
Agency
EPA/600/R-09/076
October 2009
        Distribution System
        Water Quality Monitoring:
        Sensor Technology Evaluation
        Methodology and Results
        A Guide for Sensor Manufacturers and
        Water Utilities

-------

-------
                                                       EPA 600/R-09/076
                                                           October 2009
      Distribution System Water Quality Monitoring:
Sensor Technology Evaluation Methodology and Results

                        A Guide for
          Sensor Manufacturers and Water Utilities
                         John S. Hall
                       Jeffrey G. Szabo
            Water Infrastructure Protection Division
          National Homeland Security Research Center

                      Srinivas Panguluri
                        Greg Meiners
           Shaw Environmental & Infrastructure, Inc.
                       Cincinnati, Ohio
           U.S. Environmental Protection Agency

             Office of Research and Development
            Water Infrastructure Protection Division
         National Homeland Security Research Center
                       Cincinnati, Ohio
                                                Recycled/Recyclable
                                                Printed with vegetable-based ink on
                                                paper that contains a minimum of
                                                50% post-consumer fiber content
                                                processed chlorine free

-------
                                       Disclaimer


The U.S. Environmental Protection Agency (EPA) through its Office of Research and Development funded
and managed the research described herein under Contract Nos. EP-C-04-034 and EP-C-09-041 with
Shaw Environmental & Infrastructure, Inc. This document has been reviewed by the Agency but does not
necessarily reflect the Agency's views. No official endorsement should be inferred. The U.S. Environmental
Protection Agency does not endorse the purchase or sale of any commercial products or services.

-------
                                         Foreword
The U.S. Environmental Protection Agency is charged by Congress with protecting the nation's air, water,
and land resources. Under a mandate of national environmental laws, the Agency strives to formulate
and implement actions leading to a compatible  balance between human activities and the ability of
natural systems to support and nurture life. To meet this mandate, the Agency's Office of Research and
Development provides data and science support that can be used to solve environmental problems and
build the scientific knowledge base needed to manage our ecological resources wisely, to understand
how pollutants affect our health, and to prevent or reduce environmental risks.

In September 2002, the Agency announced the formation of the National Homeland Security Research
Center. The Center is part of the Office of Research and Development; it manages, coordinates, supports,
and conducts a variety of research and technical assistance efforts. These efforts are designed to provide
appropriate, affordable, effective,  and validated technologies and methods for addressing risks posed
by chemical, biological, and radiological terrorist attacks. Research focuses on enhancing our ability to
detect, respond (through containment, mitigation, and response to public/media), and stabilize (through
treatment and decontamination) in the event of such attacks.

The Center's team of scientists and engineers is dedicated to  understanding the terrorist threat,
communicating the risks, and mitigating the results of attacks. Guided by the roadmap set forth in the
Agency's Homeland Security Strategy, the Center ensures rapid production and distribution of water
security related research products.

The Center created the Water Infrastructure Protection Division to perform research in water protection
areas including: Protection and Prevention, Detection, Containment, Decontamination and Water Treatment
Mitigation, and Technology Testing and Evaluation. The detection  research can be divided into two main
categories: 1) support for contamination warning systems for timely detection of contamination events
and 2) confirmation of events through sampling and analysis. This document focuses on online detection
technologies evaluated  at the Agency's Test and Evaluation Facility in Cincinnati, Ohio. Additional
information on the Center and its research products can be found at http://www.epa.gov/nhsrc.
                                                Kim R. Fox, Director
                                                Water Infrastructure Protection Division
                                                National Homeland Security Research Center

-------
IV

-------
                                    Table of Contents

Disclaimer	ii
Foreword	iii
List of Tables and Figures	vii
Acronyms and Abbreviations	viii
Acknowledgements	x
Notice of Trademarks and Product Names	xi
Executive Summary	xii
1.0  Introduction	1-1
    1.1  Background	1-1
    1.2  Definitions, Representations and Units	1-1
    1.3  Concept of Operations for Contamination Warning Systems	1-2
    1.4  Research Overview	1-4
    1.5  Report Outline	1-4
2.0  Online Detection Equipment and Testing	2-1
    2.1  Description of Testing Apparatus	2-1
       2.1.1 Recirculating DSS Loop No. 6 	2-1
       2.1.2 Single Pass DSS	2-2
    2.2  Test Contaminants and Water Matrices	2-3
       2.2.1 Tested Contaminants	2-3
       2.2.2 Test Water  Matrix 	2-4
    2.3  Water Quality Measurement	2-4
       2.3.1 Measured Water Quality Parameters	2-4
       2.3.2 DSS Loop  No. 6 Online Instrumentation	2-6
       2.3.3 Single Pass DSS Online Instrumentation	2-6
       2.3.4 Single Pass DSS Online Optical Instruments	2-7
    2.4  Data Collection  and Analysis	2-7
       2.4.1 Data Collection 	2-7
       2.4.2 Data Analysis	2-7
    2.5  Teaming with EPA's Water Security Initiative	2-8
    2.6  EPA's Future Water Quality  Sensor Research 	2-8
3.0  Instrument Setup and Data Acquisition	3-1
    3.1  Site-Specific Requirements	3-1
       3.1.1 Environmentally Protected Housing	3-1
       3.1.2 Access for Servicing the Instrumentation	3-2
       3.1.3 Pressure-controlled Water Supply	3-2
       3.1.4 Drainage Access	3-3
       3.1.5 Power Supply and Electrical Protection	3-3
       3.1.6 Transmission Media Access	3-4
       3.1.7 Source Water Quality Adjustment	3-4
       3.1.8 Instrument-Specific Accessories	3-4
    3.2  Calibration Materials/Reagents and Onsite Accessories	3-5
    3.3  Data Acquisition System	3-5
       3.3.1 4 to 20 Milliamperes Current Output	3-5
       3.3.2 Serial Protocols	3-6
       3.3.3 Data Communication Protocols	3-6
       3.3.4 SCADA Setup and Poll Rate	3-6
       3.3.5 Data Marking	3-6
       3.3.6 Data Transmission and Storage	3-6
    3.4  Best Practices for Instrument Setup and Data Acquisition	3-7

-------
4.0  Testing Procedures and Safety Precautions	4-1
    4.1  Blank/Control Injection	4-1
    4.2  Contaminant Injection Procedures	4-1
       4.2.1  Concentration of the Injected Contaminant	4-1
       4.2.2  Duration of Injection	4-1
       4.2.3  Water Main Flow Rate and Injection Rate	4-1
       4.2.4  Neat Compounds Versus Commercial Off-the-Shelf Products	4-2
       4.2.5  Wastewater and Ground Water Injections	4-2
    4.3  Testing and Analytical Confirmation	4-2
       4.3.1  Testing Confirmation	4-2
       4.3.2  Analytical Confirmation	4-2
    4.4  Flushing and Baseline Establishment	4-2
    4.5  Health and Safety Precautions	4-3
    4.6  Disposal of Contaminated Water From Test Runs	4-4
    4.7  Best Practices for Testing and Safety Precautions	4-4
5.0  Data Analysis	5-1
    5.1  Non-Algorithmic Sensor Response Evaluation	5-2
       5.1.1 Single Pass DSS Data Analysis	5-2
       5.1.2 Recirculating DSS  Loop No. 6 Data Analysis	5-3
       5.1.3 Edgewood Chemical and Biological Center Test Loop Data Analysis	5-5
    5.2  Automated Algorithmic  Evaluation of Sensor Response	5-6
    5.3  Data Analysis Best Practices 	5-8
6.0  Operation, Maintenance and Calibration of Online Instrumentation	6-1
    6.1  Operation and Maintenance Labor Costs	6-1
    6.2  Equipment-Specific Maintenance and Consumable Costs	6-1
    6.3  Total Organic Carbon Instrumentation	6-2
       6.3.1  Hach astroTOC™  UV Process Total Organic Carbon Analyzer	6-2
       6.3.2  Sievers® 900 On-Line Total Organic Carbon Analyzer	6-2
       6.3.3  Spectro::lyzer™/Carbo::lyzer™	6-2
    6.4  Chlorine Instrumentation	6-3
       6.4.1  HachCL-17 Free and Total Chlorine Analyzer	6-3
       6.4.2  Wallace STiernan® Depolox® 3 plus	6-3
       6.4.3  YSI6920DW	6-3
       6.4.4  Analytical Technology, Inc., Model A15/62 Free Chlorine Monitor	6-3
       6.4.5  Rosemount Analytical Model FCL	6-4
    6.5  Conductivity Instrumentation	6-4
    6.6  pH/Oxygen Reduction  Potential Instrumentation	6-4
    6.7  Turbidity	6-4
    6.8  Dissolved Oxygen	6-5
    6.9  Other Conventional Water Quality Parameter/Instrumentation	6-5
    6.10 Online Optical Instrumentation	6-5
       6.10.1  FlowCAM®	6-5
       6.10.2  Hach FilterTrak™ 660 sc Laser Nephelometer and  Hach 2200 PCX Particle Counter....6-5
       6.10.3  BioSentry® 	6-6
       6.10.4  Spectro::lyzer™/Carbo::lyzer™	6-6
       6.10.5  ZAPSMP-1	6-6
    6.11 Best Practices and Lessons Learned	6-7
7.0 Bibliography	7-1

-------
                                      List of Tables

Table 2.1  Test Contaminant Matrix	2-4
Table 2.2  Measured Water Quality Parameters	2-5
Table 5.1  Parameter-Specific Significant Change Thresholds	5-1
Table 5.2  Percent Change in Sensor Parameter Response to Injected Chemical Contaminants in
          Chlorinated Water - Single Pass DSS	5-2
Table 5.3  Normalized, Signal-to-Noise Corrected Sensor Parameter Response to Injected Chemical
          Contaminants in Chlorinated Water - Single Pass DSS	5-3
Table 5.4  Percent Change in Sensor Parameter Response to Injected Biological Contaminants and
          Growth Media in Chlorinated Water - Single Pass DSS	5-4
Table 5.5  Normalized, Signal-to-Noise Corrected Sensor Parameter Response to Injected Biological
          Contaminants and Growth Media in Chlorinated Water - Single Pass DSS	5-4
Table 5.6  Percent Change in Sensor Parameter Response to Bacillus globigii Injection in
          Chlorinated Water- Single Pass DSS	5-4
Table 5.7  Normalized, Signal-to-Noise Corrected Sensor Parameter Response to Bacillus globigii
          Injection in Chlorinated Water- Single Pass DSS	5-4
Table 5.8  Quantitative Sensor Parameter Response Matrix to Contaminants in Chloraminated
          Cincinnati Tap Water	5-5
Table 5.9  Percent Change in Sensor Parameter Response to Injected Warfare Agents in Chlorinated
          Water- Edgewood Chemical  and Biological Center Test Loop	5-6
Table 5.10 Normalized, Signal-to-Noise Corrected Sensor Parameter Response to Injected Warfare
          Agents in Chlorinated Water - Edgewood Chemical and Biological Center Test Loop	5-6
                                     List of Figures

Figure 1.1 Architecture of the EPA Contamination Warning System (EPA, 2007a)	1-3
Figure 2.1 Schematic of DSS Loop No. 6	2-1
Figure 2.2 DSS Loop No. 6 Sensor Manifold and Instrumentation Rack	2-2
Figure 2.3 Schematic of Single Pass DSS	2-3
Figure 2.4 Single Pass DSS - Longitudinal View	2-3
Figure 2.5 Single Pass DSS - Connecting Pipe Elbows	2-3
Figure 2.6 Single Pass DSS - Sampling Ports	2-3
Figure 2.7 DSS Loop No. 6 - Online Instrumentation	2-6
Figure 2.8 Single Pass DSS Instrument Panels	2-7
Figure 2.9 Various Single Pass DSS Optical Instruments	2-7
Figure 2.10 Technical Associates Radiation Monitoring Device	2-7
Figure 2.11 NexSens iSIC Data Acquisition System	2-8
Figure 2.12 First Pilot Utility - Water Security Initiative Instrument Panel Type A	2-8
Figure 2.13 First Pilot Utility - Water Security Initiative Instrument Panel Type B	2-8
Figure 3.1 Single Pass DSS Instrument Panel at 80-foot Sampling Location	3-2
Figure 3.2 Single Pass DSS Instrument Panel at 1,180-foot Sampling Location	3-2
Figure 3.3 Example Constant Head Mechanism for Hach 2200 PCX Particle Counter	3-3
Figure 3.4 Field Communications Enclosure	3-3
Figure 3.5 Hach astroTOC™ UV Process Total Organic Carbon Analyzer	3-4
Figure 3.6 Sievers® 900 On-LineTotal Organic Carbon Analyzer	3-5
Figure 3.7 SCADA Data Flow Schematic	3-5
Figure 3.8 T&E Facility NexSens iSIC Datalogger	3-7
Figure 4.1 Injection Apparatus for the Single Pass DSS	4-1
Figure 4.2 Glyphosate Triplicate Injection Run Results	4-3
Figure 4.3 Glyphosate Injections at Varying Concentrations	4-3
Figure 5.1 CANARY Operation Schematic	5-7
                                                                                              VI!

-------
                            Acronyms and Abbreviations
|jm       Micrometer
|jS/cm     Microsiemens/centimeter
°C        Degrees Centigrade or Celsius
°F        Degrees Fahrenheit
AC       Absolute Change
ANL      Argonne National Laboratory
AWWA    American Water Works Association
AwwaRF  American Water Works Association
          Research Foundation
BOD      Biological Oxygen Demand
Cl"        Chloride ion
CL2       Chlorine
CO2      Carbon dioxide
CWS      Contamination Warning System
DCE      Data Circuit-Terminating Equipment
DCS      Digital Control Systems
DMSO    Dimethyl sulfoxide
DO       Dissolved Oxygen
DOC      Dissolved Organic Carbon
DSS      Distribution System Simulator
DTE      Data Terminal Equipment
ECBC     Edgewood Chemical and Biological
          Center
EDS      Event Detection Software
EIA       Electronic Industries Alliance
EPA      U.S. Environmental Protection Agency
eq.       Equivalent
ETV      Environmental Technology Verification
ft/sec      Foot  per second
FTI       Frontier Technology, Inc.
GAC      Granular activated carbon
GB       G-type Nerve Agent (Sarin or
          isopropyl methylphosphonofluoridate)
GCWW   Greater Cincinnati Water Works
gpm      Gallons per minute
H+        Hydrogen ion (a proton)
HASP     Health and Safety Plan
HMI       Human-Machine Interface
HOCL"    Hypochlorous acid
HPLC     High-Performance (or High-Pressure)
          Liquid Chromatography
hr        Hour
HSPD     Homeland Security Presidential
          Directive
I/O        Input and Output
ICR       Inorganic Carbon Remover
IDLH      Immediately Dangerous to Life or
          Health
I EC       International Electrotechnical
          Commission
IP        Internet Protocol
ISE       Ion Selective Electrode
iSIC       Intelligent Sensor Interface and
          Control
KCN      Potassium cyanide
KHP      Potassium hydrogen phthalate
mA       Milliamperes
MALS     Multi-Angle Light Scattering
MCL      Maximum contaminant level
mg/L      Milligrams per liter
min       Minutes
mL        Milliliter
mm       Millimeter
mNTU     Milli-Nephelometric Turbidity Unit
MSD      Metropolitan Sewer District
MS2      Male-specific 2
mV       Millivolts
N/A       Not available or not applicable
NAREL    National Air and Radiation
          Environmental Laboratory
NEMA     National Electrical Manufacturers
          Association
NH2CI     Chloramine (monochloramine)
NHSRC   National Homeland Security Research
          Center
nm        Nanometers
NTU      Nephelometric Turbidity Unit
O2        Oxygen
O&M      Operation and Maintenance
OCL"      Hypochlorite ion
ODBC     Open Database Connectivity
OGWDW  Office of Ground Water and Drinking
          Water
ORP      Oxidation Reduction Potential
PC        Personal Computer

-------
pH        Potential of hydrogen in standard
          units
PLC      Programmable Logic Controller
ppm      Parts per million
QAPP     Quality Assurance Project Plan
ROC      Receiver Operating Characteristic
RP-570   RTU Protocol based on I EC 57 Part
          5-1 (present I EC 870) Version 0 or 1
RS-232   Recommended Standard 232
RS-485   Recommended Standard 485
RTUs     Remote Terminal Units
Speak      Peak sensor value between first in
          contact of the contaminant with the
          sensor until 15 minutes after the initial
          contact
^baseline   Baseline (mean) sensor value for
          one hour immediately preceding an
          injection test
S/N       Signal-to-Noise
SCADA   Supervisory Control and Data
          Acquisition
SDWA    Safe Drinking Water Act
Shaw     Shaw Environmental & Infrastructure,
          Inc.
SNL      Sandia National Laboratories
T&E      Test & Evaluation
TA        Technical Associates
TCP      Transmission Control Protocol
TEVA     Threat Ensemble Vulnerability
          Assessment
TEVA-SPOT Threat Ensemble Vulnerability
          Assessment - Sensor Placement
          Optimization Tool
TOC      Total Organic Carbon
TTEP     Technology Testing and Evaluation
          Program
DC       University of Cincinnati
UPS      Uninterrupted Power Supply
U.S.      United States
USB      Universal Serial Bus
UV       Ultraviolet
UV-Vis    Ultraviolet-Visible
UV254     Ultraviolet 254 nanometer wavelength
v/v        Volume/Volume Percent
VX        V-series Nerve Agent (S-[2-
          (diisopropylamino)ethyl]-O-ethyl
          methylphosphonothioate)
WATERS  Water Awareness Technology
          Evaluation Research and Security
WDMP    Water Distribution Monitoring Panel
WSD      Water Security Division
WSi       Water Security Initiative
XML      extensible Markup Language
ZAPS     Zero Angle Photon Spectrometer
                                                                                              IX

-------
                                  Acknowledgements


The principal authors of this document, titled "Distribution System Water Quality Monitoring: Sensor
Technology Evaluation Methodology and Results -A Guide for Sensor Manufacturers and Water Utilities,"
are Mr. John S. Hall, Dr. Jeffrey G. Szabo, Mr. Srinivas Panguluri,  RE., and Mr. Greg Meiners.

The authors wish to acknowledge the contributions of the following individuals and organizations towards
the development and review of this document:

Technical reviews of the document were performed by:
   Mr. Steve Allgeier, U.S. Environmental Protection Agency's Office of Ground Water and Drinking
      Water
   Mr. Stanley States,  Pittsburg Water and Sewer Authority
   Mr. Alan Roberson, Director of Security and Regulatory Affairs, American Water Works Association

U.S. Environmental Protection Agency National Homeland Security Research Center quality
assurance review was performed by:
   Ms. Eletha Brady-Roberts, Quality Assurance Manager

Illustrations and publication design were performed by:
   Mr. James I. Scott, Shaw Environmental & Infrastructure, Inc.

Cover Photo Credits are as follows:
   Unknown Rural Town, West Virginia - photograph from archives of Shaw Environmental &
      Infrastructure, Inc.
   Water Tower, Florence, Kentucky- photograph by Ms. Jennifer Panguluri
   Child drinking water from a tap - photograph of Mr. Ravi Panguluri taken by Ms. Jennifer Panguluri

-------
Notice of Trademarks and Product Names
Trademark or Product Name Manufacturer's Name, City, State Web Site
1720DTurbidimeter
Analytical Technology, Inc. Model A15/62
Free Chlorine Monitor
BioSentry®
FlowCAM®
GuardianBlue™ Early Warning System
H2O Sentinel™
Hach 2200 PCX Particle Counter
Hach astroTOC™ UV Process Total Organic
Carbon Analyzer
Hach CL17 Free Chlorine Analyzer
Hach CL17 Total Chlorine Analyzer
Hach Event Monitor™ Trigger System
Hach FilterTrak™ 660 sc Laser Nephelom-
eter
Hach/GLI Model C53 Conductivity Analyzer
Hach/GLI Model P53 pH/ORP Analyzer
Hach Water Distribution Monitoring Panel
(WDMP) sc
Hydrolab® DS5
Real Kill®
Rosemount Analytical Model FCL
Roundup®
Sievers® 900 On-Line Total Organic Carbon
Analyzer
Sievers® RL
Six-CENSE™
Spectro::lyser™ or Carbo::lyser™
SSS-33-5FT
TROLL® 9000
Wallace & Tiernan® Depolox® 3 plus
YSI 6600
YSI 6920DW
ZAPS MP-1
Hach Company, Loveland, Colorado
Analytical Technology, Inc., Collegeville, Pennsylvania
JMAR Technologies, Inc., San Diego, California
Fluid Imaging Technologies, Yarmouth, Maine
Hach Company, Loveland, Colorado
Frontier Technology Inc., Goleta, California
Hach Company, Loveland, Colorado
Hach Company, Loveland, Colorado
Hach Company, Loveland, Colorado
Hach Company, Loveland, Colorado
Hach Company, Loveland, Colorado
Hach Company, Loveland, Colorado
Hach Company, Loveland, Colorado
Hach Company, Loveland, Colorado
Hach Company, Loveland, Colorado
Hach Company, Loveland, Colorado
Realex Corporation, Spectrum Brands, St. Louis, Mis-
souri
Emerson Process Management, Irvine, California
The Scotts Company, LLC or its affiliates, Marysville,
Ohio
GE Analytical Instruments, Boulder, Colorado
GE Analytical Instruments, Boulder, Colorado
CENSAR Technologies, Sarasota, Florida
scan Messtechnik GmbH, Vienna, Austria
Technical Associates, Canoga Park, California
In-Situ® Inc., Ft. Collins, Colorado
Siemens Water Technologies, Kent, United Kingdom
YSI lnc.,Yellow Springs, Ohio
YSI lnc.,Yellow Springs, Ohio
ZAPS Technologies Inc., Corvallis, Oregon
http://www.hach.com
http://www.analyticaltechnology.com
http://www.jmar.com
http://www.fluidimaging.com
http://www.hach.com
http://www.fti-net.com
http://www.hach.com
http://www.hach.com
http://www.hach.com
http://www.hach.com
http://www.hach.com
http://www.hach.com
http://www.hach.com
http://www.hach.com
http://www.hach.com
http://www.hach.com
http://www.spectrumbrands.com
http://www.raihome.com
http://www.scotts.com
http://www.geinstruments.com
http://www.geinstruments.com
http://www.censar.com
http://www.s-can.at
http://www.tech-associates.com
http://www.in-situ.com
http://www.wallace-tiernan.com
http://www.ysi.com
http://www.ysi.com
http://www.zapstechnologies.com
                                                     XI

-------
                                         Executive Summary
      This report, titled "Distribution System Water Quality Monitoring:  Sensor Technology Evaluation
      Methodology and Results - A Guide for Sensor Manufacturers and Water Utilities," provides an overview
      of the U.S. Environmental Protection Agency's (EPA's) research results from investigating water quality
      monitoring sensor technologies that might be used to serve as a real-time contamination warning system
      (CWS) when a contaminant is introduced into a drinking water distribution system. EPA's concept of CWS
      for protecting  water distribution systems is discussed in Chapter 1.0. A principal component of such a
      system is online water quality monitoring.

      Based on a review of available online water quality monitoring sensor technologies, an early determination
      was made that it was not technically feasible to accurately identify and quantify the many different types
      of contaminants that could potentially be introduced into the drinking water supply/distribution system.
      Furthermore, because online sensor technologies need to be economically suitable for mass deployment
      within a distribution system, EPA focused its research on identifying sensor technologies that could be
      used to detect anomalous changes in water quality due to contamination event(s). Once a water quality
      anomaly is detected, the water utility operator is alerted, and further actions (e.g., sampling and analysis)
      could be undertaken by the operator to identify and quantify the contaminant if necessary. This report
      focuses on EPA's research on pilot-scale evaluations of available online water quality monitoring sensor
      instrumentation.

      This report first describes the testing apparatus (the recirculating Distribution System Simulator (DSS)
      - Loop No. 6,  Single Pass DSS, and the online instrumentation) used for the pilot-scale evaluations at
      the EPA Test and Evaluation Facility in Cincinnati, Ohio (Chapter 2.0). The instrument setup and data
      acquisition specifics are described in Chapter 3.0. The detailed testing procedures and safety precautions
      are described in Chapter 4.0. The data analysis procedures are presented in Chapter 5.0. Operation and
      maintenance specifics for selected instruments are provided in Chapter 6.0. In addition, each chapter
      includes a best practices summary at the end with key points that are designed to deliver the "lessons
      learned" through this research. A bibliography of selected references is included as Chapter 7.0.
XII

-------
1
The safety of drinking water supplied to the consumers
by water treatment plant operators is dependent upon
many factors: quality of raw water (surface water and/
or ground water), application of appropriate treatment
technology/disinfection (as needed), and monitoring
of treated/finished water  within  the  water distribu-
tion system network. Appropriate treatment/disinfec-
tion technologies for  both surface and ground water
sources are identified by the U.S.  Environmental Pro-
tection Agency (EPA) in various regulations that were
promulgated pursuant to the Safe  Drinking Water Act
(SDWA) of 1974 and its  amendments. Although the
treated water leaving a treatment plant typically meets
EPA's water quality requirements, the water could un-
dergo transformation  within the  various distribution
system  components (e.g.,  storage tanks and  pipes),
which alters the quality, potentially making it  unsuit-
able for human consumption. To address these issues,
EPA has developed specific regulations that mandate
periodic monitoring of water quality within distribu-
tion systems.

Research related to water quality monitoring  within
the distribution system has increased significantly
since the events of September 11, 2001, when improv-
ing the security of our nation's water infrastructure
became a major priority. Homeland  Security Presi-
dential  Directive 7 (HSPD-7), issued on December
17, 2003, established a national policy for  federal
departments and agencies to  identify and prioritize
United  States  critical infrastructure  and to protect
the infrastructure from terrorist  attacks. Thereafter,
HSPD-9, issued on January 30, 2004, directed EPA to
"develop robust, comprehensive, and fully coordinat-
ed surveillance and monitoring systems, ...  that pro-
vide early detection and awareness of disease, pest,
or poisonous agents."  EPA plays a critical role in this
effort as the lead federal agency for water security.
Subsequent to these directives, in March 2004,  EPA
released the peer-reviewed Water Security Research
and Technical  Support Action Plan (Action  Plan -
EPA,  2004a), which identified important water secu-
rity related issues and outlined research and  techni-
cal support needs to address these issues. In addition,
the EPA Action Plan identified a list of projects to be
undertaken in response to the identified needs.  Fur-
thermore, the Action Plan identified several products
proposed to be developed to enhance the  security of
drinking water and wastewater systems. This report is
one of the products designed to meet the Action Plan
requirements specified under Section 3.3.d.2 - Stan-
dard Operating Procedures and Quality Assurance and
Control Practices to Guide the Evaluation of Monitor-
ing Technologies and Section 3.3.d.5 - Standard Op-
erating Procedures for Evaluating Monitoring Tech-
nologies.

1,1
The analytical methods and water quality sensors used
to address EPA regulations pursuant to SDWA were
not designed to address water security threats. Conse-
quently, data necessary to identify a serious threat to
the water supply caused by either an accidental release
or by an intentional act might not be captured during
routine periodic monitoring at drinking  water treat-
ment plants  and various distribution system locations.
Over the past five years, as part of the overall Water
Awareness Technology Evaluation Research and Secu-
rity (WATERS) program at the EPA Test & Evaluation
(T&E) Facility in Cincinnati, Ohio, EPA investigated
online water quality monitoring technologies that might
be used to achieve the goal of serving as early warning
indicators to detect contaminant introduction into the
drinking water supply. The WATERS program testing
efforts were sponsored by EPA's National Homeland
Security Research Center (NHSRC). During this study,
a variety of  commercially available  online sensors/in-
struments were evaluated.

Based on a  review of available online water quality
monitoring sensor technologies,  an early determination
was made that it was not technically feasible to accu-
rately identify and quantify the many different types of
contaminants that could potentially be introduced into
a drinking water supply/distribution system. Further-
more, these online technologies needed to be economi-
cally suitable for mass deployment within a distribution
system. Therefore, EPA focused its research to identify
online sensor technologies that could be used to detect
anomalous changes in the baseline water quality with-
out specific regard to precision,  accuracy or identifica-
tion of the contaminant. Once an anomaly is detected
and the water utility operator is alerted, further actions
(e.g., grab sampling and analysis) could be undertaken
by the operator to identify and quantify the contami-
nant whenever possible. This report focuses on EPA's
research on  pilot-scale evaluations of available online
water quality monitoring sensor instrumentation.

1,2
For the purposes of this document, a "sensor" is defined
as an electro-mechanical device (e.g., membrane, elec-
trode, or microchip) that measures a physical or chemi-
cal characteristic of water and converts it into a "signal"
or measured value, which is typically processed further
by an instrument.
                                                                                                         1-1

-------
      An instrument is denned as an electro-mechanical device
      (or a collection of electro-mechanical devices) that can
      manipulate (e.g., amplify) a measured output from an as-
      sociated "sensor" and transmit the measured value (e.g.,
      analog or digital output value) to a data acquisition system.

      Some instruments (e.g., optical instruments) perform
      measurements without an associated sensing element (as
      denned in this  section) and contain additional devices
      that transmit data. Therefore, in general, an instrument
      is meant to refer collectively to a sensor that provides the
      overall measurement functionality. Furthermore, a "sen-
      sor"  or "instrument" response is intended to define the
      change in measured or recorded output value of the rel-
      evant water quality parameter. The term "equipment" is
      used to refer collectively to electro-mechanical devices
      that might include one or more sensors, instruments, and
      additional appurtenances such as plumbing, data collec-
      tion and/or recording devices that are necessary to make
      the overall manufactured device functional. Depending
      upon the focus  of the discussion, and to improve docu-
      ment readability, the terms  "sensor," "instrument," and
      "equipment" have been used somewhat interchangeably
      throughout the document.

      The research did not address instrument-specific preci-
      sion, accuracy, or ability to identify contaminants.

      The  term "sensor manufacturer(s)"  is intended to in-
      clude instrument manufacturer(s) and  vendor(s) who
      might simply resell or repackage a manufactured prod-
      uct. A listing of tested sensor/instrument technologies
      and  their associated registered or  unregistered trade-
      marks is  included under the Notice of Trademarks and
      Product Names (page xi) in this report. The tested equip-
      ment referred to in this document was procured over
      time and used for testing during the period of 2003 to
      2008. Subsequent to the testing, there could have been
      design changes and/or improvements to the equipment
      by the manufacturers. These devices might perform dif-
      ferently under the same tested conditions, but could bear
      the same registered or unregistered trademark.

      Neither the authors nor EPA make any representations
      on the usefulness or general performance  of these  de-
      vices outside the context of the testing described in this
      report. The  use of these  manufacturer-specific  names
      and model numbers throughout the document is to pro-
      mote clarity so that the reader  can identify the tested
      equipment. Any rights associated with these registered
      or unregistered trademarks are the sole property of the
      trademark holders. It is recommended that water utilities
      and other researchers apply their own judgment prior to
      choosing any equipment for water quality monitoring.

      English standard units  that are commonly used by the
U.S. water utility personnel have been used throughout
this document. For example, volume is reported in U.S.
gallons and velocity in feet per second (ft/s). However,
in keeping with industry usage, contaminant concentra-
tions are reported in metric units, in milligrams per liter
(mg/L). Unless otherwise stated for computational pur-
poses, the values from the instruments are presented as
reported in the output from the individual instrument(s)
without any conversions provided.

1.3              of                 for


      Systems
Real-time water quality monitoring to control treatment
process operations has been successfully performed
at water treatment plants for many years. EPA's con-
cept for contamination warning systems (CWS) is de-
signed to  extend this monitoring approach to multiple
locations  within a water distribution system (Kessler
et al., 1998; ISLI,  1999; AwwaRF, 2002; Kirmeyer et
al., 2002;  EPA, 2005 a-d; Roberson and Morley, 2005;
Allgeier et al., 2006; Dawsey et al., 2006). Consequent-
ly, baseline water quality conditions can be monitored
continuously  in real-time such that a sudden change in
water quality parameter(s) can trigger a contamination
warning. Monitoring baseline water quality parameters
within the distribution system will also provide mul-
tiple benefits of improved water quality closer to the
point-of-use and additional security for detecting inten-
tional or unintentional contamination events within the
system. The capital, operational, and maintenance costs
for CWS will be difficult to sustain unless multiple ben-
efits are identified. For water utilities,  it is important
to first maximize the security benefits by strategically
placing the selected online monitors in the network and
utilizing suitable techniques to evaluate the online sen-
sor responses. Therefore, in addition to evaluating on-
line water quality monitoring and sensor technologies,
EPA has collaborated with  various research entities to
develop two key software tools that provide these func-
tionalities, described below.

EPA, in  collaboration with  research  organizations
including  the Sandia  National Laboratories  (SNL),
Argonne National Laboratory  (ANL), University of
Cincinnati (UC) and the American Water Works As-
sociation (AWWA), has developed a software program
referred to as the  Threat Ensemble Vulnerability As-
sessment - Sensor Placement Optimization Tool (TE-
VA-SPOT). TEVA-SPOT can be used to determine the
optimum number and locations for monitoring stations
within a water distribution system. The software allows
the user to specify a wide range of performance objec-
tives including:  1) Population-based health measures,
2) Time to detection, 3) Extent of contamination, 4)
1-2

-------
      Volume of contaminated water consumed, and 5) Num-
      ber of contamination events detected. TEVA-SPOT fa-
      cilitates interactive design of a water quality monitor-
      ing system by allowing the user to specify constraints
      to ensure that the performance objective is satisfied.
      For example, a TEVA-SPOT  user can integrate expert
      knowledge during the design process by identifying ei-
      ther existing or unfeasible sensor locations. Installation
      and maintenance costs for sensor placement  can also
      be factored into the analysis.  More information on the
      TEVA Research Program and SPOT can be  obtained
      online at: http://www.epa.gov/nhsrc/water/teva.html.

      EPA,  in collaboration with SNL, also developed the
      CANARY algorithm to evaluate water quality sensor re-
      sponses and identify changes in water quality that could
      indicate a contamination event. The name CANARY is
      not an acronym, but suggests a parallel with the historic
      "canary in the coal  mine" event detection approach in
      which the coal miners used canaries to detect poison gas
      events. Similarly, the CANARY software evaluates real-
      time water quality data obtained from various instru-
      ments and uses mathematical  and statistical  techniques
      to identify the onset of anomalous water quality events.
      The CANARY software allows for the following: 1) the
      use of a standard data format for input and output of
      water quality and operations data, 2) the  ability to se-
      lect different detection algorithms (the program contains
      three  different mathematical approaches for analyzing
      the data), 3) the ability to select various water utility and
      location-specific configuration options, 4) an online op-
      erations mode and an off-line evaluation/training mode,
      and 5) the ability to generate data needed to  establish
      performance metrics (e.g., false alarm rates). This algo-
      rithmic  approach enhances the detection sensitivity of
      the field equipment and simultaneously reduces the false

                  Routine Operation
                   Operational Strategy
positive alarm events. CANARY is freely available for
download through the EPA website. More information
on CANARY can be obtained online at: http://www.epa.
gov/nhsrc/water/teva.html.

Regardless of the approach used by the utility to evalu-
ate the data collected from online sensors, establishing
a protocol to verify and respond to alarms triggered
by the online water quality monitoring instruments is
important.  Note that online water quality monitoring
represents only one component of a holistic CWS. Ad-
ditional data inputs from the utility and public health
agencies should be collected and evaluated to comple-
ment the benefits of online water quality monitoring
(See Figure 1.1).

EPA's Office of  Ground  Water and  Drinking Water
(OGWDW), Water Security Division (WSD), has field-
deployed a pilot project called the Water Security Initia-
tive (WSi), that is based upon the concepts identified in
Figure 1.1. The WSi program is being implemented in
the following three phases:

   •   Phase I: develop the conceptual design of a
      system for timely detection and appropriate
      response to drinking water contamination
      incidents to mitigate public health and economic
      impacts;

   •   Phase II: test and demonstrate CWS
      through pilots at drinking water utilities and
      municipalities and make refinements to the
      design based upon pilot results; and

   •   Phase III: develop practical guidance and
      outreach to promote voluntary national adoption
      of effective and sustainable drinking water CWS.

           Consequence Management
                                                                       Consequence Management Plan
    Online water quality
    Sampling & analysis
                                              Return to routine monitoring & surveillance
Figure 1.1 Architecture of the EPA Contamination Warning System (EPA, 2007a)
                                                                                                                1-3

-------
        Water Utilities and
        Sensor Manufacturers

       Online water quality monitoring alone will not
       provide a holistic CWS.
       Integration of data streams such as consumer
       complaint surveillance, enhanced security
       monitoring, public health surveillance, and
       triggered sampling and analysis with the online
       water quality monitoring is necessary for realizing
       the full benefits from a CWS.
       EPA has developed several guidance and guideline
       documents, a modular response protocol toolbox,
       and other software tools for utilities planning to
       establish a comprehensive CWS. The relevant
       software tools include TEVA-SPOT for locating
       online sensors and CANARY event detection
       software. The bibliography section includes a
       listing of the related EPA documents.
       Manufacturers should design flexibility into the
       sensor equipment to output real-time data streams
       in a variety of formats, which allows for analysis
       by both external and/or internal event detection
       algorithms.
       In addition to helping achieve regulatory
       compliance (e.g., monitoring residual disinfectant
       levels), sustainable online CWS equipment
       can provide other benefits that can lead to
       improvements in: distribution system water quality;
       treatment process control; distribution system
       control; customer service; and overall security.

      Based on information collected from the ongoing Phase
      I and Phase II activities, WSD has developed a variety
      of guidance and interim guidance documents on relat-
      ed topics including: WaterSentinel system architecture
      (EPA, 2005c), planning for CWS deployment  (EPA,
      2007b),  developing an operational strategy for CWS
      (EPA, 2008a),  developing  consequence  management
      plans (EPA, 2008b), and the Cincinnati pilot post-imple-
      mentation system status (EPA, 2008c). In addition, WSD
      had previously developed a modular response protocol
      toolbox to assist water utilities for planning and respond-
      ing to contamination threats  (EPA, 2004[c through j]).
      More information on the EPA WSi can be obtained on-
      line at: http://www.epa.gov/watersecurity.

      1.4  Research Overview
      The vast majority of the research described in this re-
      port was conducted at the EPA T&E Facility in Cincin-
      nati, Ohio.  Since the early 1990's, at this facility, EPA
      has conducted research using simulated drinking water
      distribution systems. A number of pilot-scale distribu-
      tion system simulators (DSSs) are in use at the T&E Fa-
cility. EPA operates, maintains, and modifies the DSSs
as needed to accommodate evolving study designs. For
the research results reported in this document, EPA em-
ployed two types of DSSs at the T&E Facility to inves-
tigate water quality monitoring sensor technologies that
might be used to serve as a real-time early warning sys-
tem when a contaminant is introduced into the drink-
ing water supply.  Only online sensors were evaluated,
because the response time is critical for achieving the
project objective of contamination warning.

To evaluate the selected sensors, a series of test runs was
conducted by injecting known quantities of potential
contaminants into the selected DSSs. After injection,
sensor data were collected continuously and electroni-
cally archived. After injection, grab samples were col-
lected periodically to confirm the sensor results. These
studies were  focused  on providing independent  third
party  data to decision makers in the following areas:

  1.  What water quality parameters will be most
      useful in CWS?

  2.  Can online water quality sensors be used
      to reliably trigger alarms in response to
      contamination events within a water distribution
      system?

  3.  What are the operational and maintenance costs
      associated with online water quality monitoring
      systems?

1.5  Report Outline
The following  chapters of  this  report summarize the
findings related to this research. Chapter 2.0 presents
a summary  of the various online detection sensors/
instrumentation evaluated  and the evaluation-specific
research activities performed at  the EPA T&E Facility
in Cincinnati, Ohio and other field locations. Chapter
3.0 describes general instrument setup and data acqui-
sition. Chapter 4.0 contains a description of the testing
procedures and safety precautions. Chapter 5.0 outlines
the data analysis procedures. Chapter 6.0 describes the
operation and maintenance (O&M) and calibration re-
quirements of the tested instrumentation. At the end of
each chapter (starting in Chapter 3.0), a summary of
applicable best practices is presented for the targeted
audience, which includes sensor manufacturers and wa-
ter utilities.
1-4

-------
2.0  Online  Detection

Equipment  and Testing

The focus of this research was to identify water qual-
ity parameters and  online sensor technologies that
could be used to detect anomalous changes in water
quality due to contamination event(s) within a wa-
ter distribution system. The sections of this chapter
briefly describe the following: testing apparatus, con-
taminants and injected concentrations, disinfectants,
water quality parameters and online instrumentation,
data collection and analysis, event detection, and field
applications.

2.1  Description of Testing
      Apparatus
The first round of testing for online water quality sen-
sor instrumentation  was conducted using recirculat-
ing DSS  Loop No. 6 located at the T&E Facility in
Cincinnati,  Ohio. DSS Loop  No. 6 was essentially
operated as a closed system during the sensor testing
period. At the conclusion of the first round of tests,
some of the research stakeholders expressed concern
that the recirculation mode operation of DSS Loop
No. 6 enhanced the  detection ability of the sensors.
In this mode, the contaminant is  recirculated within
the distribution system, thereby allowing the sensor
to detect the same slug of contaminant multiple times.
Subsequently, later rounds of testing involved the use
                                      of the Single Pass DSS, also located at the T&E Facil-
                                      ity in Cincinnati, Ohio.

                                      Concurrent to the DSS Loop No. 6 and Single Pass
                                      DSS testing, EPA conducted a series of bench-scale
                                      minimum dosing tests. In these tests, the selected con-
                                      taminants in a water matrix (at various concentrations)
                                      were exposed to the online sensors  to establish the
                                      minimum  dosage/concentration of the  contaminant
                                      where a "response"  to various water quality param-
                                      eters was produced by the sensor instrumentation.

                                      2.1.1 Recirculating DSS Loop No. 6
                                      Recirculating DSS Loop No. 6 consists of a 15-year
                                      old, 6-inch-diameter unlined  ductile iron pipe and is
                                      one of six pipe loops within the DSS (Loop Nos.  1
                                      through 6). DSS Loop No. 6 is approximately 75 feet
                                      long and has a total capacity of approximately 150
                                      gallons. DSS Loop No. 6 is equipped with a 3-horse-
                                      power pump capable of circulating water through the
                                      loop at a rate of up to 110 gallons per minute (gpm).
                                      The loop is normally operated at a flow rate of 88 gpm,
                                      which produces a velocity of 1 foot per second (ft/sec)
                                      in the main pipe. The process flow schematic of the
                                      DSS Loop No.  6 used for these tests (including modi-
                                      fications for this research) is presented in Figure 2.1.

                                      For the purposes of this testing, DSS Loop No. 6 was
                                      operated in recirculation  mode  using municipal tap
                                      water supplied by the Greater Cincinnati Water Works
                                      (GCWW). In this mode, the feed tanks and the 100-gal-
           Chlorine
           injection
            pump
                          Total Organic Carbon •
                          Chlorine
                                                       H—»- Drain
                                      Conductivity/Temperature
                                      pH/Oxidation Reduction Potential
                                      Turbidity
                          Biofilm sampling coupon
                             — Water tlow -
  Potable
   water
  30-gallon
     feed
    water
     tank
         Recirculation J
              pump

        I (0-0.25 gpm)
      .J\
a
Feed
water
pump
                                                      Heat
                                                    exchanger
                                                              pH-\
                                                 Oxidation Reduction Potential
                                                       Dissolved Oxygen
                                                                    S,
                                     Contaminant
                                      feed tank
Figure 2.1 Schematic of DSS Loop No. 6
                                                                  Incoming makeup water
                                                                  < Loop recirculation/test water
                                                                  < Drain
                                                                  < Chemical addition
                                                                  Sensor loop instrumentation
                                                                    Biofilm
                                                                   - sampling
                                                                    rack
                                                                  Overflow
                                                                             100-gallon
                                                                             recirculation/mix
                                                                             tank
                                                                                                    2-1

-------
      Ion recirculation tank are kept inline with the system.
      Operation in this mode effectively increases the volume
      of water in the system by 85 gallons, to a total of ap-
      proximately 235 gallons. When operating in recircula-
      tion mode, potable water is added to the system from
      the 30-gallon feed-water tank at a rate of 0.16 gpm. At
      this rate, the entire volume of the system is exchanged
      in 24 hours. However, due to mixing in the recirculation
      tank, the time required to completely exchange the con-
      tents of the system via dilution is considerably longer.

      Injected contaminants reached the sensors in approxi-
      mately 75 seconds and quickly become homogeneously
      mixed with the 250 gallons of water in the system. Dye
      tests were performed to  confirm the travel  time  and
      mixing. The response profiles to injected contaminants
      reflect this design. An initial response after the contami-
      nant first reaches the sensors is recorded for those sen-
      sors capable of detecting the contaminant. The response
      persist as the contaminant becomes dispersed in the
      DSS Loop No. 6, and in the sensor manifold, followed
      by a period of recovery due to dilution or consumption
      of the injected material via hydrolysis or reaction with
      free chlorine present in the tap water or through pipe
      wall reaction.

      DSS Loop No. 6 is equipped with one 10-gallon chemi-
      cal feed tank and a pump used to add treatment chemi-
      cals to the system. The feed tank was used to add chlo-
      rine when establishing baseline conditions prior to the
      addition of contaminants. Chlorine additions continued
      during test runs in order to keep the disinfectant levels
      stable during injections. The DSS Loop No. 6 setup also
      allowed for testing using chloramine as the disinfectant.

      Two hardware modifications to the flow  system of DSS
      Loop  No. 6 were made to support the  sensor evalua-
      tion studies. A 50-gallon feed tank with a delivery line
      to the intake side of the recirculation pump was added
      for the purpose of introducing contaminants into DSS
      Loop  No. 6. Also, a  sensor loop manifold (see Figure
      2.1 and Figure 2.2) was fabricated for  the purpose of
      diverting water flow  from the DSS Loop No. 6 to the
      online monitors under evaluation, and  to collect grab
      samples for field and laboratory analyses.

      DSS Loop No. 6 was equipped with a sensor mani-
      fold incorporating the needed online sensors so that
      the studies could begin quickly. Since DSS Loop No.
      6 was operated in essentially a closed  mode, the ob-
      served sensor responses were typical of a batch reactor
      operation. Essentially, the sensor response seen for the
      duration of a  test run was similar to the case where
      a contaminant slug would travel through the system
      for the entire test duration (assuming minimal disper-
      sion, mixing and general disruption of slug due to flow
Figure 2.2 DSS Loop No. 6 Sensor Manifold and
Instrumentation Rack

variations). The recirculation mode within the  tank
also dilutes the concentration of the contaminant in 24
hours and does not represent a true plug flow system.
Because there are some technically valid differences
as compared to a "real world" distribution system, the
recirculation mode allowed for safer contained tests,
eliminated wastage of water, and allowed for  easy
identification of viable sensors, prior to embarking on
studies using the Single Pass DSS as outlined in the
next section.

2.1.2 Single Pass DSS
The Single Pass DSS was constructed of 3-inch-diam-
eter glass-lined ductile iron pipe and spans the entire
length (150 feet) of the T&E facility high-bay area and
wraps back and forth across this expanse eight times.
The combined length  of this pipe  is  approximately
1,200 feet and the Single Pass DSS has a total capac-
ity of approximately 440 gallons. The pipe is gravity
fed with tap water via a 750-gallon stainless steel tank
mounted near the ceiling of the facility. This tank is
supplied from a floor-mounted 1,000-gallon stainless
steel tank. In-situ chemical feed tanks and mixers can
be used for chlorine dosing, chemical addition, or other
similar purpose. The contaminant injection port was in-
stalled immediately downstream of the 750-gallon feed
tank. In addition, two sampling ports were installed at
80-foot and 1,180-foot distances from the contaminant
injection port. The two sampling ports supply sample
water to multiple  instrumentation racks. Figure 2.3
shows a schematic of the Single Pass DSS within the
T&E Facility.

Figures 2.4 and 2.5 show the Single Pass DSS running
the length of the T&E Facility high bay and wrapping
its length 4 times on the east side of the pipe rack. Fig-
ure 2.6 shows the sampling ports for the inlet located
at the top near the 80-foot mark and the outlet located
directly below this port at the 1,180-foot distance.
2-2

-------


Contaminant
feedr~|
container | |
TCT
Chemica
injection

1000-gallon
|Q — holding tank
heed
water
pump

750-gallon
holding tank _ overflow
Lo Drain
Online Instrument Panel
I- Chlorine
1 	 1 - Conductivity/Temperature
- pH/Oxygen Reduction Potential
Sample Port 1 Dissolved Oxygen
80 ft
. 	 >
1 Sample Port 2
1 180 ft Online Instrument Panel
Drain - Chlorine
I - pH/Oxygen Reduction Potential
I 	 1 - Turbidity
., 	 Tap water - Dissolved Oxygen
feed

Figure 2.3 Schematic of Single Pass DSS

      2.2  Test Contaminants and Water
            Matrices
      Target contaminants  for the  study were selected to be
      representative of broad classes of biological and chemi-
      cal contaminants that could be potentially introduced
Figure 2.4 Single Pass DSS - Longitudinal View
Figure 2.5 Single Pass DSS - Connecting Pipe Elbows
Figure 2.6 Single Pass DSS - Sampling Ports

into the U.S. water supply. Municipal tap water sup-
plied by the GCWW was used as the water matrix for
this testing.

2.2.1 Tested Contaminants
Table 2.1 presents a summary of the broad classes of con-
taminants and specific contaminants tested by EPA along
with the associated test water matrix. The online instru-
mentation used to  measure the  individual water qual-
ity parameter responses during the testing varied due to
various logistical reasons and the evolution of the testing
activity during the course of the research. For example,
most of the advanced optical instruments such as the Bio-
Sentry®, FlowCAM®, Spectra: :lyser™, and Each Fil-
terTrak™ 660 sc Laser Nephelometer were not procured
prior to beginning testing that utilized  the recirculating
DSS Loop No. 6. These optical devices were purchased
later to  evaluate their efficacy in detecting biological
contaminants. The BioSentry® and FlowCAM® instru-
ments are designed to count and identify the injected
biological cells. Therefore, for the purposes of evaluating
these instruments, the following biological contaminants,
surrogates, and growth (or carrier) media such as nutri-
ent broths were injected into the Single  Pass DSS: three
micron beads, Escherichia coli (E. coli), E.  coli (in de-
chlorinated water), bacteriophage male-specific  (MS2),
Bacillus globigii (B. globigii), B. globigii (in dechlorinat-
ed water), secondary effluent from wastewater treatment,
sporulation media, sucrose, Terrific Broth, nutrient broth,
and Trypticase soy™ broth. The biological contamina-
tion tests were performed in three distinct ways: 1) test
cells (centrifuged to isolate the contaminant only) inject-
ed with tap water, 2) test cells in nutrient or broth solu-
tions, 3) test cells in nutrient and broth solutions preceded
by treatment with dechlorinating agents such as  sodium
thiosulfate pentahydride and sodium thiosulfate anhy-
drous. The last test was performed because real-world
contamination events might be conducted in conjunction
with dechlorination in an attempt to make the cells more
                                                                                                                2-3

-------
Table 2.1  Test Contaminant Matrix
Recirculating Single
Contaminant Sperifir L°°P PaSS
Class Contaminant Cl,a NH,Clb Cl,
Biologicals
Insecticides
Herbicides
Culture Broths
Inorganics
Warfare
Agents
Others
Bacillus globigii
Bacteriophage MS2
Escherichia coll
Surrogate beads
Aldicarb
Nicotine
Real Kill®/Malathion
Dichlorvos
Phorate
Roundup® /Glyphosate
Dicamba
Nutrient broth
Sporulation media
Terrific broth
Tryptic soy broth
Arsenic trioxide
Cesium chloride
Cobalt chloride
Lead nitrate
Mercuric chloride
Potassium cyanide
Potassium ferricyanide
Sodium arsenite
Sodium thiosulfate
Sodium fluoride
Ricin
G-type nerve agent
V-series nerve agent
Potassium cyanide
Blank (GAC water)
Secondary effluent
Colchicine
Dimethyl sulfoxide
Dye
Sucrose
Sodium fluoroacetate
Methanol


X



X


X











X



xc
xc
xc
xc

X


X





X


X
X

X
X












X






X
X






X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X



X
X
X
X
X
X
X
X
X
 aChlorine
 bChloramines
 cTesting conducted at the U.S. Army's Aberdeen Proving Ground's Edgewood Chemical
 Biological Center (ECBC) Facility.
       viable. The presence of free chlorine at the typical residu-
       al levels [~1 milligrams per liter (mg/L)] is deleterious to
       many biological organisms and reduces the efficacy of a
       biological attack. The bacteriophage MS2 tests were per-
       formed to simulate a viral threat. To evaluate the impact
of nutrient broth and the dechlorinating agents, the fol-
lowing "control" injections were also performed: sodium
thiosulfate pentahydride, sodium thiosulfate anhydrous,
sucrose, terrific broth, and nutrient broth.

2.2.2 Test Water Matrix
The GCWW water  supply to the T&E Facility comes
from the Miller Plant, which treats water from the Ohio
River. GCWW uses chlorine as the residual disinfectant
for water distribution. The background range of values
for the routinely measured water quality parameters at
the T&E Facility are as follows: free chlorine - 0.8 to
1.1 mg/L, specific conductance - 300 to 600 microsie-
mens per centimeter (uS/cm),  oxidation reduction po-
tential (ORP) - 500 to 700 millivolts (mV), potential of
hydrogen in standard units (pH) - 8.5 to 8.8, turbidity
< 0.1 nephelometric turbidity units (NTU), and total or-
ganic carbon (TOC) -  0.3 to 1.3 mg/L. Only the free
chlorine levels were adjusted as needed (prior to test-
ing) such that the levels were approximately 1 mg/L.

The chloraminated water was  prepared in batches us-
ing a 2,400-gallon  tank. GCWW-supplied tap  water
was collected in a 2,400-gallon tank at the EPA T&E
Facility and tested for total chlorine  residual. Calcu-
lations were made to determine the correct amount of
sodium hypochlorite necessary to raise the  total  chlo-
rine concentration to the desired level, usually 2 mg/L.
When this concentration was achieved and verified by
analysis, ammonium hydroxide was added in sufficient
quantity (chlorine to ammonia ratio of 4:1) to convert
the free chlorine into combined chlorine. The resulting
chloraminated water was mixed for 15 to 20 minutes
and retested for both free and total chlorine.

2.3 Water Quality Measurement
Prior to introduction of contaminants, water-quality
sensors located within the selected test apparatus (i.e.,
DSS Loop No. 6 or Single Pass DSS) were typically
monitored for an hour to establish normal (baseline)
conditions. After contaminant  injection, data from the
various sensors were monitored and recorded. The sen-
sor data were supported by the analysis of grab samples
taken from the test apparatus at discrete intervals. For
experimental control, uncontaminated test water matrix
was injected into the test apparatus. During the testing,
it was  verified  that  the act of injection did not affect
baseline conditions as characterized by sensor response.

2.3.1 Measured Water Quality Parameters
A variety of water  quality parameters was measured
during the  testing period.  The specific instrumenta-
tion used in  individual test runs for both DSS Loop
No. 6 and the Single Pass DSS was dependent on the
availability of instrumentation during the testing pe-
riod. Table  2.2  presents an overall summary of the
2-4

-------
Table 2.2 Measured Water Quality Parameters
Parameter
Ammonia
- nitrogen
Apparent color
Chloride
Conductivity
measured
as specific
conductance3
Dissolved oxygen
(DO)
Fluorescence
(total, humic and
bacterial)
Free chlorine
Multi-angle light
scattering (MALS)
Multi-spectrum (UV-
Vis) absorption
Nitrate - nitrogen
Oxidation-reduction
potential (ORP)
Particle Count
Particle count
and image-based
identification
PH
Measurement Type
Continuous and grab
Grab
Continuous and grab
Continuous and grab
Continuous and grab
Continuous
Spectrophotometric
Continuous and grab
Continuous
Continuous
Spectrophotometric
Continuous, grab, and
Spectrophotometric
Continuous and grab
Continuous
Continuous
Continuous and grab
Online Instrumenta-
tion Tested
YSI 6600, YSI 6920DW
Various laboratory in-
struments, Six-Cense™
YSI 6600, YSI 6920DW
YSI 6600, YSI 6920DW,
Hydrolab® DS5, Troll®
9000, Six-Cense™,
Hach/GLI Model C53
Conductivity Analyzer
YSI 6600, Hydrolab®
DS5, Troll® 9000, Six-
Cense™
ZAPS MP-1
YSI 6920DW, Hydro-
lab® DS5, Troll® 9000,
Six-Cense™, Hach
CL17 Free Chlorine
Analyzer
BioSentry®
Spectra ::lyser™ or
Carb::olyser™
YSI 6600, YSI 6920DW
YSI 6600, YSI 6920DW,
Hydrolab® DS5, Troll®
9000, Six-Cense™,
Hach/GLI Model P53
pH/ORP Analyzer
Hach 2200 PCX Particle
Counter
FlowCAM®
YSI 6600, YSI 6920DW,
Hydrolab® DS5, Troll®
9000, Six-Cense™,
Hach/GLI Model P53
pH/ORP Analyzer
Parameter Applicability
Naturally occurring form of nitrogen in the nitrogen cycle. Dis-
solved ammonia gas is toxic to aquatic life at concentrations
as low as 0.2 milligram per liter (mg/L). Will be converted to
chloramine in chlorinated drinking water.
Visible color resulting from turbidity and dissolved materials
(humic material, dissolved metals, dyes, algae). Potable water
is normally colorless after treatment.
Indicator of salinity. Associated with a secondary maximum
contaminant level (MCL) of 250 mg/L in drinking water.
Ability of water to carry an electrical current. Strong indicator of
dissolved salts. Serves as a surrogate for total dissolved solids.
Concentration of oxygen dissolved in water can serve as an
indicator of chemical and biochemical activity in water.
Instrumental measure of fluorescence at various wavelengths.
Chlorine is added to the DSSb in the form of sodium hypochlo-
rite. Chlorine levels in drinking water are controlled at ~1 mg/L.
Utilizes laser-produced MALS technology to generate unique
bio-optical signatures for classification using JMAR's pathogen
detection library.
UV-Vis excitation that provides a means of estimating absorp-
tion at various wavelengths. Nitrate and/or nitrite concentra-
tion, DOCC, TOC, CODd and BODe (depending on the used
algorithm), and turbidity. Information at nearly any wavelength
between 200 and 750 nm.
Essential nutrient for plants and animals. Nitrate is the most
soluble form of nitrogen. Causes health problems in humans.
Drinking water standard is 10 mg/L.
Indicator of dissolved oxidizing and reducing agents (metal
salts, chlorine, sulfite ion). ORP values above 700 millivolts
(mV) kill unwanted organisms in drinking water. A ground water
incursion may lower ORP by increasing chlorine demand.
Chlorination of drinking water produces an ORP background of
-700 millivolts in GCWW water.
Counts all particles that are between 2 and 750 urn in size.
The counted particles can be subdivided into 32 size ranges
to identify particles of interest. For example, the particle size
ranges could be selected to correspond to biological organisms
such as Giardia (6-10 urn) and Cryptosporidium spp. (2-5 urn).
Measures particle size, count and shape. Images particles
between 2 urn and 3 mm in size. Helps to identify and classify
particles based on library of images.
Indicator of hydrogen ion activity (acidity or alkalinity) of water.
Most chemical and biochemical processes are pH dependent.
Carbon dioxide/bicarbonate/carbonate and ammonia/ammo-
nium equilibria are pH dependent. pH of drinking water is well
established and controlled. A change of more than 0.5 pH unit
indicates a problem.
                                                                                                 2-5

-------
 Table 2.2 (continued) Measured Water Quality Parameters
Parameter Measurement Type Online Instrumenta- Usefulness of Parameter for Water Quality
tion Tested
Temperature
Total cyanide,
malathion, and
glyphosate
Total organic
carbon (TOC)
Transmission
Turbidity
Ultraviolet 254
nanometer
wavelength (UV254)
absorption
Continuous and grab
Grab
Continuous and grab
Continuous
Spectrophotometric
Continuous and grab
Continuous
Spectrophotometric
YSI 6600, YSI 6920DW,
Hydrolab® DS5, Troll®
9000, Six-Cense™,
Hach/GLI Model C53
Conductivity Analyzer,
Hach/GLI Model P53
pH/ORP Analyzer
Various laboratory
instruments
Hach astroTOC™ UV
Process Total Organic
Carbon Analyzer, Siev-
ers® 900 On-Line Total
Organic Carbon Ana-
lyzer, Spectro::lyser™
orCarb::olyser™
ZAPSMP-1,
Spectra ::lyser™ or
Carb::olyser™
YSI 6600, YSI 6920DW,
Hydrolab® DS5, Troll®
9000, Six-Cense™,
1720DTurbidimeter,
Hach FilterTrak™ 660
sc Laser Nephelometer
ZAPSMP-1,
Spectra ::lyser™ or
Carb::olyser™
A measurement indicator of how hot or cold the water is. DO and
specific conductance change with temperature. Biological and
chemical activities are heavily influenced by water temperature.
Compound-specific laboratory analysis for the purpose of deter-
mining the fate of these three contaminants in the DSS.
Dissolved plus particulate organic compounds. Can range from
0.5 to 25 mg/L in drinking water in the U.S. May be correlated to
chemical and biological oxygen demand.
Measure of color based on Beer's Law as measured by photon
transmission through water [800 nanometers (nm) for this study].
Indicator of suspended matter and microscopic organisms. Patho-
gens are more likely to be present in highly turbid waters.
Measure of organic compounds that absorb photons at 254 nm.
Indicative of organic compounds with aromatic chemical structure
and conjugation.
 aSpecific conductance is defined as the raw solution conductivity, compensated to 77°F (25°C).
 bDSS = Distribution System Simulator.
 CDOC = Dissolved Organic Carbon.
 dCOD = Chemical Oxygen Demand.
 eBOD = Biological Oxygen Demand.
      measured water quality parameters and a summary of
      the usefulness of each measurement in terms of water
      quality.

      2.3.2 DSS Loop No. 6 Online Instrumentation
      The following are  online water quality  monitoring
      sensor instruments that were evaluated during the var-
      ious DSS Loop No.  6 test runs: YSI 6600, Hydrolab®
      DSS, Troll® 9000,  Six-CENSE™, Hach Water Dis-
      tribution Monitoring Panel (WDMP), and Zero Angle
      Photon Spectrometer (ZAPS) MP-1. Figure 2.2 (previ-
      ously shown) and Figure 2.7 depict most of the online
      instrumentation evaluated during the DSS Loop No.
      6 testing.

      2.3.3 Single Pass DSS Online Instrumentation
      The following are online water quality monitor-
      ing  sensor instruments that were evaluated  during
      the various Single  Pass DSS test runs: Hach CL17
      free chlorine analyzer; Analytical Technology, Inc.
      Model A15/62 free chlorine monitor; YSI 6920DW;
Wallace & Tiernan® Depolox® 3 plus; Hach astro-
TOC™ UV  process TOC  analyzer;  Hach WDMP;
Sievers® RL; and Sievers® 900 On-Line TOC Ana-
lyzer. Figure 2.8 shows two Single Pass DSS instru-
ment panels.
Figure 2.7 DSS Loop No. 6 - Online Instrumentation
2-6

-------
Figure 2.8 Single Pass DSS Instrument Panels
      2.3.4 Single Pass DSS Online Optical
           Instruments
      The following are online optical instruments that were
      evaluated during the various Single Pass DSS test runs:
      Carbo::lyser™   and  Spectro::lyser™,  BioSentry®,
      FlowCAM®, Each FilterTrak™ 660 sc  Laser Neph-
      elometer, and Hach 2200 PCX  Particle Counter. Fig-
      ure 2.9 depicts the instrument panel that contains the
      controller for the Carbo::lyser™,  controller for the
      Hach Filter/Irak™ 660 sc Laser Nephelometer, and the
      FlowCAM® device.
Figure 2.9 Various Single Pass DSS Optical
Instruments

      In addition to these instruments, EPA is also evaluating
      the  radiation monitor (Technical Associates, Canoga
      Park, California) at the National Air and Radiation En-
      vironmental Laboratory (NAREL) in Montgomery, Ala-
      bama. The results from these tests were not available at
      the time of production of this document. Figure 2.10 de-
      picts the radiation monitor.

      2.4 Data Collection  and  Analysis
      Data collected for each parameter from the online wa-
      ter quality sensor instruments were complemented by
      laboratory analyses of grab samples. To facilitate com-
                                                         parisons between the online monitoring results and lab-
                                                         oratory analyses, sensor responses to contaminants for
                                                         each parameter were plotted along with associated grab
                                                         sample results. These plots  allowed  a graphic inter-
                                                         pretation of the data to 1) evaluate changes in baseline
                                                         conditions due to contaminant introduction, 2) compare
                                                         sensors  (using different technologies to measure the
                                                         same parameter), and 3) recognize false negative/false
                                                         positive responses by visual comparison to the grab
                                                         sample data.
                                                         Figure 2.10 Technical Associates Radiation Monitoring
                                                         Device
2.4.1 Data Collection
Wherever possible, each of the online sensors was
connected to a data acquisition system. The intelli-
gent Sensor Interface and Control (iSIC) system was
connected to the data collection personal computer
(PC) via hardwire or radio (as appropriate). The data
collection PC ran the iChart software program, which
polled the connected iSIC(s) and monitoring devices
every 2 minutes and recorded the data reported by the
instrumentation. The 2-minute data collection cycle
was considered to  be optimum because of the num-
ber of instruments  concurrently tested that needed to
be polled for data and the measurement cycle limita-
tions of some tested devices. The iSIC/iChart system
was selected as the data collection platform because
it  incorporated many pre-built device drivers that
could communicate with the widest variety of online
instrumentation tested at the T&E Facility. A more
detailed discussion of the data collection system is
presented in Chapter 4.0. Figure 2.11  shows the Nex-
Sens iSIC data acquisition system.

2.4.2 Data Analysis
The data plots generated from the tests conducted at
the T&E Facility were analyzed visually to construct a
qualitative response matrix for the contaminants test-

                                                   2-7

-------
Figure 2.11 NexSens iSIC Data Acquisition System

      ed. The criteria for determining a "significant change"
      was subjective at the early stages of the research. The
      sensor responses were plotted over the course of the
      test runs and analyzed for visually significant chang-
      es. Thereafter, a more robust analysis was performed
      where the absolute change, percent change, and sig-
      nal-to-noise (S/N)  ratio for each measured parameter
      was computed. See Chapter 5.0 for further details.

      At the onset of this testing  effort, EPA determined
      that an automated algorithmic analysis of the online
      data was essential. Therefore, concurrent to the test-
      ing, EPA  initiated collaboration with SNL  for the
      development of the CANARY Algorithm (previously
      described  in Chapter 1.0). In addition, EPA contin-
      ues to evaluate other commercial data analysis algo-
      rithms/products as they became available (Umberg et
      al., 2009).

      2.5 Teaming with  EPA's Water
           Security Initiative
      The work conducted at the  T&E Facility assisted
      EPA's Water  Security initiative (WSi - formerly,
      WaterSentinel). As  described  in  Section  1.2, the
      EPA's OGWDW-WSD worked collaboratively with
      NHSRC to deploy a pilot network of water quality
      monitoring instrumentation at GCWW as a part of
      the  WSi pilot in Cincinnati (EPA, 2008c). Figures
      2.12 and  2.13 show two types of instrument panels
      deployed at the first pilot utility. The panels contain
      online instrumentation to measure   free chlorine,
      TOC, pH, ORP, conductivity, temperature and tur-
      bidity. The Type A panels utilize all Hach instrumen-
      tation, whereas the Type B panels utilize instrumen-
      tation from manufacturers other than  Hach. As  will
      be discussed in Chapter 5.0, free chlorine and TOC
      were found to be  most useful trigger parameters in
      chlorinated water systems.
2.6  EPA's Future Water Quality
      Sensor Research
EPA,  through their Technology Testing and Evalu-
ation  Program (TTEP) and testing activities at the
T&E Facility, will continue to identify and evaluate
promising sensor technologies for potential use in
CWS, as funding allows. Radiological and low densi-
ty biological detection equipment testing are the key
current  sensor-related data gaps. New technologies
are needed to reduce the current capital, operational,
and maintenance costs in order for CWS programs to
be sustainable. Information on sensor evaluation pro-
grams can be obtained by contacting Mr. John Hall
via e-mail (Hall.John@epa.gov) or phone (513-487-
2814). Additional information related to EPA's water
protection research can be obtained at: http://www.
epa.gov/nhsrc/aboutwater.html.

WSi is a program designed to address the risk of in-
tentional contamination of drinking water distribution
systems. Initiated by OGWDW in response to HSPD-9,
Figure 2.12 First Pilot Utility - Water Security Initiative
Instrument Panel Type A
                                                        Figure 2.13 First Pilot Utility - Water Security Initiative
                                                        Instrument Panel Type B
2-8

-------
  Water Utilities and
  Sensor Manufacturers
 Online sensors were tested in simulated distribution
 systems using chlorinated and chloraminated
 waters. The simulated systems were injected with
 a variety of target contaminants to evaluate the
 individual sensor/parameter response.
 Grab sampling is critical to verify online sensor
 responses.

 A data polling frequency of two minutes was found
 to be optimal for the wide range of sensors tested,
 but utilities might want to evaluate other polling
 frequencies.

 A robust Supervisory Control and Data Acquisition
 (SCADA) system is needed to fully utilize and
 process in real-time the large volumes of data that
 are generated.

 The EPA T&E Facility distribution system
 simulators attempt to replicate field conditions,
 but the effects of varying water demands were
 not simulated during the tests. In addition, the
 background water quality parameter levels are very
 stable at the T&E Facility. Therefore, for utilities
 with varying background water quality parameters
 compounded with varying demands, the simulated
 tests might result in different sensor/parameter
 response.
                                              .
Instrument manufacturers need to design for
allowing an automated grab sample to be collected
to validate the instrument response as needed.
 Instrument manufacturers should design their
 sensors so that they can be easily interfaced with a
 wide variety of SCADA systems.
the overall goal of WSi is to design and deploy CWS
for drinking water utilities. EPA is implementing the
WSi in three phases: (1) development of a conceptual
design that achieves timely detection and appropriate
response to drinking  water contamination incidents;
(2) demonstration and evaluation of the  conceptual
design in full-scale pilots at drinking water utilities;
and (3) issuance of guidance and conduct of outreach
activities to promote  voluntary national adoption of
effective and sustainable drinking water  CWS. The
initial full-scale pilot was implemented  in  Cincin-
nati, Ohio. EPA-OGWDW plans  to implement more
pilot studies utilizing the CWS concept presented in
Section 1.3. These pilot studies will be conducted at
several utilities to demonstrate that a functional CWS
can be deployed under a variety of real-world condi-
tions. Information on the GCWW pilot study can be
                                                   obtained by contacting Mr. Steve Allgeier via e-mail
                                                   (Allgeier.Steve@epa.gov)  or phone 513-569-7131.
                                                   Additional information related to EPA's water security
                                                   research can be obtained at: http://www.epa.gov/safe-
                                                   water/watersecurity/.
                                                                                                         2-9

-------

-------
3.0  Instrument  Setup

and  Data  Acquisition

As previously mentioned in Section 2.5 (and will be dis-
cussed later in Chapter 5.0), the tests conducted at the
EPA T&E Facility show that free chlorine and TOC are
the most  useful water quality parameters for detecting
changes indicative of contamination in chlorinated wa-
ter systems. The prototype monitoring panels installed
by the WSi pilot utility included online instrumentation
to measure free chlorine, TOC, pH, ORP, conductivity,
temperature and turbidity. Each utility should evaluate
its needs and resources (both capital and labor), and re-
view the test results associated with its water distribution
system before selecting a suite of online parameters and
associated instrumentation. The use of chloramines as
disinfectant should also be taken into account when se-
lecting the parameters and instrumentation. Once the pa-
rameters and instruments have been selected, they should
be set up in accordance with the instructions provided by
the manufacturer for flow, pressure, and sample condi-
tioning requirements. This chapter discusses in detail the
various requirements for setting up online water quality
sensor instrumentation at a specific site.

3.1   Site-Specific Requirements
EPA-developed software  such as TEVA-SPOT should
be used to identify the optimal locations of a fixed
number of sensors. After a potential monitoring site has
been identified using TEVA-SPOT, a site visit should be
performed to ensure that the selected site has:
  •   sufficient environmentally protected secure
      space for housing the selected instrumentation
  •   access and a clear path for transporting and
      servicing the instrumentation to conduct
      installation and maintenance activities
  •   adequate source of pressurized and pressure-
      controlled water supply for the proposed
      instrumentation
  •   drainage access to discharge the water analyzed
      by the online instrumentation
  •   necessary power supply and backup
      (uninterrupted power supply) to power the
      online instrumentation, data collection, and data
      transmission systems
  •   appropriate media (wired or wireless) for
      transmitting the online data in real-time to a
      specified data collection center
  •   water quality characteristics that are suitable (or
      can be appropriately conditioned) for analysis
      by selected online instrumentation
Depending upon the threat and vulnerability analy-
sis, some of the selected sites might not meet all of
the requirements. For such sites, alternate means of
meeting a site-specific requirement should be inves-
tigated. For example, all sites might  be not suited
for deploying a single communication  technology.
In such cases, a combination of wired and  wireless
communication technology should be investigated.
Another example could  involve a situation where
the initial water quality is not suitable for  selected
instrumentation. In this case, either alternate instru-
mentation should be investigated or site/instrument-
specific sample water conditioning could  be per-
formed such as pH buffering, degassing, or removing
iron and salts.

3.1.1  Environmentally Protected Housing
The selected site should be environmentally protect-
ed and secure. Many of the online sensors are  typi-
cally contained in a National Electrical Manufactur-
ers Association (NEMA) class 4- or  4X-compliant
corrosion-proof enclosures and protected from wind-
blown dust, rain, sleet and external icing.  Specifi-
cally,  a NEMA4-compliant enclosure has to pass the
"Hose Test," which  is described as: a 1-inch nozzle,
delivering 65 gpm of water, from a distance of 10
feet, from all directions, for a 5-minute time period,
with no water leak  to the interior. Class 4X enclo-
sures  have additional protection against corrosion.
Preferred materials  for  mounting (or housing) the
online instrumentation are polyester/glass, stainless
steel,  and epoxy coatings. Although the selected en-
closure might be suited for general outdoor applica-
tion, there is an additional need for temperature and
humidity control because the advanced  devices are
equipped with  onboard computers and electronics
that might not withstand the  temperature, humid-
ity, and altitude extremes. The environmental toler-
ances are  instrument-specific and the  manufacturer
instructions should  be followed to ensure the suit-
ability of the selected housing. In general,  it is not
recommended that the instruments be housed in an
environment  where the temperature  exceeds  90°F
(32.2°C) or falls below 40°F  (4.4°C). Appropriate
cooling and/or heating devices should be installed at
the site as needed.

Furthermore, the selected instrumentation might have
humidity specifications, (for example, a range of 5 to
95%). High humidity might result in corrosion of elec-
tronic components and/or could lead to short circuits
and malfunction. Humidity can increase the conductivi-
ty of the embedded electronics, leading to short circuits
and malfunction. Condensation is another problem that
can cause electronic devices to malfunction. For exam-
ple, when an instrument is moved from  a colder place
                                                                                                     3-1

-------
      to a wanner and more humid place, condensation could
      coat circuit boards and other insulators, leading to short
      circuiting  inside  the equipment. Such short circuits
      might cause substantial permanent damage if the equip-
      ment is powered on before the condensation has evapo-
      rated. Electronic equipment should be acclimatized for
      several hours (as specified by the manufacturer)  before
      powering on.

      3.1.2 Access for Servicing the Instrumentation
      The online instrumentation generally requires peri-
      odic servicing, calibration and reagent replacement.
      In a multi-instrument setup, individual instruments
      are mounted on panels that are fabricated to  fit the
      space requirement and also provide easy access for
      servicing  each instrument. The instruments are set
      up such that the servicing need for a single instru-
      ment  does not disrupt the function of other instru-
      mentation. Also,  the  water  intake  and  drain lines
      are configured in a manner such that they are gen-
      erally below the instrumentation so that any water
      line failure does not damage the instrumentation. In
      addition, the power conditioning, data logging, and
      communication equipment are separated from the
      online instrumentation. Power conditioning devices
      are designed to regulate the voltage and improve the
      power quality  (e.g., electrical noise suppression and
      transient impulse protection).  Figures  3.1  and 3.2
      show instrument panels that have been designed spe-
      cifically to facilitate online  monitoring  at the EPA
      T&E Facility. For example, the sensor shown in Fig-
      ures 3.1 and 3.2 has all the water lines at the bottom.
      The sample inlet line to each instrument is isolated.
      The drain lines are  connected to the available floor
      drain. The data logging equipment  is on the back of
      the panel and the power lines are on top.

      The operator should be able to see and service the
      instrumentation/data  collection  components  with
ease. If the instrument panel is improperly designed,
the operator might take  shortcuts while servicing,
which could lead to lower data quality or equipment
malfunctions due to improper servicing.

3.1.3 Pressure-controlled Water Supply
The  majority of the water quality instrumentation
is  sensitive to fluctuations in water supply pressure.
Pressure  changes can create bubbles (degassing) in
the sampled water,  resulting in erroneous data. Pres-
sure  regulator valves are used to allow water from
a high-pressure supply line (or tank) to be reduced
to  a safer  preset level specified by  the  instrument
manufacturer(s). Pressure regulators are also suscep-
tible to changes in  the water supply pressure. Some-
times, it  might be  necessary to have multiple layers
of pressure regulation to dampen any effects of pres-
sure  fluctuations on the instrument readings. Instru-
ments like particle  counters require separate mounted
constant-head overflow weir mechanisms so that the
sample outlet can be raised or lowered  to the height
that will  produce the desired flow. Figure 3.3  shows
the constant head mechanism for a particle counting
device.

By pushing water up a fixed-height column and col-
lecting the sample stream from that column, a constant
Figure 3.1 Single Pass DSS Instrument Panel at 80-foot
Sampling Location
Figure 3.2 Single Pass DSS Instrument Panel at 1,180-
foot Sampling Location
3-2

-------
Figure 3.3 Example Constant Head Mechanism for
Hach 2200 PCX Particle Counter.

      pressure is delivered to the  instrument. This method
      of regulated sample delivery, although simple, is very
      effective in controlling pressure and flow fluctuations.
      Hach uses  this  method  of sample delivery for Hach
      CL-17 free (or total) chlorine analyzers and Hach 2200
      PCX Particle Counters. The BioSentry® unit also em-
      ploys a similar method for delivering constant sample
      flow.

      3.1.4 Drainage Access
      The sampled water drawn from the online instrumen-
      tation panel needs to be discharged appropriately to
      meet local discharge requirements. Generally, access to
      a sanitary sewer line is sufficient. In certain locations,
      such access might not be easy. Care should be taken
      so that water does not pool near the instrumentation,
      causing a slipping hazard. A drain manifold is recom-
      mended for locations with multiple online instruments.
      The drain line should be sized adequately, taking into
      account any instrument and inlet line failures.

      3.1.5 Power Supply and Electrical Protection
      Adequate power supply (preferably  3-phase) with a
      backup device for uninterrupted power supply (UPS)
      intended to provide sufficient power for the online in-
      strumentation, data  collection, and data transmission
      systems is recommended. In addition, the electrical  cir-
                                                          cuits to each instrument should be isolated via a circuit
                                                          breaker, and both the panel and instruments should be
                                                          appropriately grounded. The electrical isolation allows
                                                          for servicing of individual instrumentation without dis-
                                                          rupting the  other equipment installed at the location.
                                                          Circuit breakers protect the instrumentation from elec-
                                                          trical surges and  short circuits. Connection to ground
                                                          is a safety issue designed to protect the personnel ser-
                                                          vicing the instrumentation and the online instrumenta-
                                                          tion. The ground connection also helps limit build-up of
                                                          static electricity on the instrumentation. In areas where
                                                          the line voltage is known to fluctuate, a surge protector
                                                          is also recommended. The surge protector regulates the
                                                          voltage supplied to the instrument by either blocking or
                                                          by shorting  to ground connection when voltages above
                                                          safe instrumentation thresholds are sensed in the circuit.

                                                          A UPS or an inline battery backup lasting four to eight
                                                          hours is recommended, because it continuously  pow-
                                                          ers and protects  the instrumentation from the previ-
                                                          ously described power problems. A UPS is also known
                                                          as a power or line conditioner because of this ability. A
                                                          UPS generally contains a lead-acid battery for storing
                                                          power. During electrical  outages, the energy reserves
                                                          stored in the UPS are used to power the instrumenta-
                                                          tion. Figure 3.4 shows a field data communications
                                                          NEMA 4 enclosure with backup UPS Power.
Figure 3.4 Field Communications Enclosure.
                                                                                                              3-3

-------
      Some instruments provide the option for portable or
      line power. For example, the YSI 6920DW instrument
      can be powered by battery or line power. Line power
      is preferred in installations where the line is backed up
      with an appropriately sized UPS.

      3.1.6 Transmission Media Access
      In order to fully realize the benefits of online instru-
      mentation,  appropriate  media (wired  or wireless)
      should be used in real-time for transmitting the data or
      information to a pre-specified data collection location.
      Wired media generally provide higher bandwidth, but
      could be cost-prohibitive in certain locations. In these
      cases, the use of wireless media (e.g., licensed or un-
      licensed radio, cellular, satellite-based transmission
      media options) should be investigated. In some cases,
      depending upon the location and media used, the data
      transmission might be susceptible to various types of
      interferences. In such cases, additional programmatic
      error control techniques should be applied to mitigate
      the errors during transmission. For example, in a poll-
      based data collection platform,  it might be necessary
      to either increase the number of retries on a failed
      poll event or adjust the data packet reception window
      based on the bandwidth and latency limitations of the
      selected media.

      3.1.7 Source Water Quality Adjustment
      Generally, the selected instruments need to be suitable
      to analyze the source water quality. In some cases, the
      source water quality can be adjusted to meet the instru-
      ment specifications. For example, certain free chlorine
      measuring devices require the pH of the water to be
      below 8.5 standard units. If the pH at the selected lo-
      cation is above 8.5, appropriate buffering agents (e.g.,
      carbon dioxide) should be used to condition the pH of
      the sampled water or an  alternate online monitoring
      instrument should be selected for that parameter. For
      example, the Sievers® RL unit is not appropriate for
      high pH water (> 8.5).

      3.1.8 Instrument-Specific Accessories
      As discussed in the previous section, for some instru-
      ments, there might be  a need for  either peripheral
      support equipment (or accessories)  that precondition
      the sample water or for carrier gases to complete the
      analysis. The  Hach TOC monitor  is another exam-
      ple of an instrument that requires specific accessory
      equipment as identified below.

      In general, TOC monitors are one of the more com-
      plex instruments to operate. Hach uses  the  ultravi-
      olet  (UV) persulfate method; this  method requires
      reagents (sodium persulfate and phosphoric acid) to
      drive the oxidation reaction. These  reagents are sup-
      plied in 5-gallon carboys and are  bulky to  handle.
This instrument also requires a clean, carbon diox-
ide (CO2)-free air source to carry sample flow to the
CO2 detector. The CO2-free air source is supplied
either by a cylinder of liquid nitrogen, or a zero air
generator. If a zero air generator is used, an air com-
pressor is needed to supply a constant stream of air.
A considerable amount of space is required to house
this monitor and its  supporting equipment.  These
units have proven to be fairly labor-intensive to op-
erate and require a highly skilled technician to per-
form maintenance and calibration procedures. Fig-
ure 3.5 shows a Hach  astroTOC™ UV process TOC
analyzer instrument and associated zero air system.

The Sievers® 900 On-Line TOC Analyzer also uses
the UV persulfate method. Similar to the Hach unit,
the Sievers® 900 On-Line TOC Analyzer  can be
fairly labor-intensive to operate and requires a highly
skilled technician to perform maintenance and cali-
bration procedures. However, this instrument and its
reagent packs are more compact than the Hach unit.
Also, the Sievers® 900 On-Line TOC Analyzer does
not require an external zero air system/compressor
or a liquid nitrogen Dewar. This unit  does require
an inorganic carbon remover (ICR) for waters  that
are heavily laden with inorganic carbon. Figure 3.6
shows a  Sievers® 900 On-Line TOC Analyzer  (the
ICR is contained inside the instrument enclosure).

Unless the  utility has  extensive in-house experi-
ence with these instruments, it might be prudent to
procure service contracts for each of the aforemen-
tioned TOC units. Surrogate TOC monitoring equip-
ment using UV-visible (UV-Vis) spectral absorbance
has been found to be less labor intensive, but trade-
offs in its limited ability to detect a variety of po-
tential organic contaminants should be taken  into
consideration.
Figure 3.5 Hach astroTOC™ UV Process Total Organic
Carbon Analyzer.
3-4

-------
Figure 3.6 Sievers® 900 On-Line Total Organic Carbon
Analyzer.

      3.2 Calibration  Materials/Reagents
           and Onsite  Accessories
      During the setup,  instrument  calibration  material,
      reagents and  accessories  ought to  be available (as
      needed) to ensure that the instruments are operating
      as recommended by the manufacturer. Many reagents
      and calibration solutions have expiration dates; there-
      fore, these reagents should be ordered according to the
      instrument-specific maintenance schedule. While per-
      forming calibration and maintenance activities, each
      manufacturer's procedure needs to be followed to en-
      sure that the instrument is performing properly and is
      measuring the water quality in  the designated range.
      Also, proper calibration ensures that the quality of the
      data is reliable.

      3.3 Data  Acquisition System
      Most water utilities implementing  a network of online
      instrumentation generally have some type of SCADA
      system. SCADA systems are also  known as  industrial
      control systems and are capable of monitoring and con-
      trolling a process. Generally, water treatment plants are
      automated with some type of SCADA system. From a
      water utility perspective, a SCADA system  generally
consists of the following four components:

   1. a Human-Machine Interface (HMI), which is a
     combination of computer software and hardware
     that presents information to an operator; the
     operator is able to monitor and control the
     process/instrumentation through this interface.

   2. a supervisory or a central node that gathers data
     from a programmable logic controller (PLC)
     and/or a Remote Terminal Unit (RTU) for
     presentation to the operator through the HMI
     and sends commands to the PLC/RTU based on
     the operator inputs from the HMI.

   3. PLC s/RTUs connected to the online
     instrumentation that convert sensor signals to
     digital data (inputs) and send commands to
     connected automated devices (such as sampling
     devices and pumps) to perform a pre-defined
     task based on operator commands from the
     HMI.

   4. a data communication infrastructure connecting
     the supervisory system to the PLC/RTU.

Figure 3.7 shows a data flow  schematic from field-de-
ployed online instrumentation to the operator in a con-
trol center.

Historically,  SCADA system hardware and software
tend to be proprietary. Water utilities that have invested
in a particular manufacturer's solution might find them-
selves restricted to limited choices for equipment when
considering system expansions or upgrades. However,
most SCADA systems can communicate with  sensors
or instrumentation that can provide their data output in
4 to 20 milliamperes (mA), or through serial protocols
such as Recommended Standard 232 (RS-232)/Recom-
mended Standard 485  (RS-485). The RS-232/RS-485
electrical specifications are defined by the Electronic
Industries Alliance  (EIA) for  a serial communications
channel.
    I/O
   Meters
   Sensors
 Field Devices
    Field
  Devices
  Remote
>   PLC   4
 RTU Controller
Comms
Protocols
Ethernet
 Serial
Wireless
  Master
> SCADA Server
    HMI
                               Control
                               Center
Figure 3.7 SCADA Data Flow Schematic
3.3.1 4 to 20 Milliamperes Current Output
Developed in the 1950s, the 4 to 20 mA instrument out-
puts are still widely used by SCADA and instrument
manufacturers. This output format is ideally suited for
low-cost instruments that provide one or two analog
                                                                                                           3-5

-------
      output values. Generally, the output signal can travel
      distances of around 50 meters. In other words, the PLC/
      RTU  capturing this data output from the instrument
      must  be located within 50 meters of this instrument.
      This output is easy to understand and troubleshoot: a
      signal of 4 mA represents zero percent of the output
      span and 20 mA represents one hundred percent signal
      output span. For example, a chlorine monitor calibrated
      to measure a span between 0 and 5 parts per million
      (ppm) will provide a corresponding analog output be-
      tween 4 and 20 mA when reporting these values. Trou-
      bleshooting the output is simple, requiring only a digi-
      tal voltmeter to read the values inline.

      3.3.2 Serial  Protocols
      Developed in the 1960s, the RS-232 is a serial protocol
      for sending and receiving signals between a Data Ter-
      minal Equipment (DTE) and Data Circuit-terminating
      Equipment (DCE). Prior to the popularization of the
      Universal Serial Bus (USB), the RS-232 serial port was
      commonly available  with all types of personal com-
      puters. The RS-232 connection (at a minimum) needs
      3 wires to communicate where  one wire is dedicated
      to transmitting data, one to receiving data, and one is
      ground. The RS-232 can even use a two-wire connec-
      tion (data and ground) if the data flow occurs one way.
      The RS-232  standard defines the voltage levels that
      correspond to logical one and logical zero levels. Valid
      signals are plus or minus 3 to 15 volts.

      RS-485 (also known as EIA-485) is a multipoint serial
      communications channel that can span distances of up
      to 4,000 feet. The multipoint communication is often
      in a master-slave arrangement when one device dubbed
      "the master"  initiates all communication  activity with
      other  devices in the  network. RS-485 is used as the
      underlying protocol in many standard and proprietary
      SCADA  protocols, including  the most common ver-
      sions  of Modbus.

      3.3.3 Data Communication Protocols
      Typical legacy SCADA communications protocols  in-
      clude Modbus (developed by Modicon), RTU Protocol
      (RP-570) and PROFIBUS. These communication pro-
      tocols are  all proprietary and SCADA-manufacturer
      specific, but are widely adopted and used. During the
      late 1990s, many of the SCADA manufacturers shifted
      toward more open communication  protocols and ad-
      opted the "de facto" open message structure offered by
      Modbus over serial communications protocols such as
      RS-232/RS-485. Since 2000, most  SCADA manufac-
      turers are offering greater open interfacing operability
      by  adopting standards such as Modbus Transmission
      Control Protocol (TCP) over Ethernet and Internet Pro-
      tocol  (IP). Other  standard communications protocols
      include:  International  Electrotechnical  Commission
(IEC) 60870-5-101 or 104, IEC 61850 and Distributed
Network Protocol 3. These protocols are standardized
and recognized by all of the major SCADA manufac-
turers. Similar to Modbus, many of these protocols now
contain extensions to operate over TCP/IP.

3.3.4 SCADA Setup and Poll Rate
Generally speaking, most water utilities embarking on
online monitoring have some type of SCADA system in
place. It is generally cost-effective to expand on existing
SCADA systems to accommodate online water quality
monitoring and integrate it with distribution system and
treatment plant operations as needed. For a large utility
(serving >100,000 persons), it is common to have tens
of thousands of SCADA tags [or SCADA input and out-
put (I/O) values] that are polled by the  SCADA "mas-
ter device" periodically. Depending upon the poll cycle
and available data bandwidth, polling  most of these
SCADA I/O values every one to five minutes is com-
mon. The online water quality instruments themselves
have a sampling cycle, and a vast majority of these have
sampling and reporting cycles of less than one minute.
However, in some cases, the sample cycles for achiev-
ing peak measured values might be between four  and
eight minutes (e.g., Hach astroTOC™ UV process TOC
analyzer and Sievers®  900 On-Line TOC  Analyzer).
The data acquisition system used at the EPA T&E Facil-
ity was set to poll every two minutes. Based on a review
of the data generated during this testing, the researchers
at the T&E Facility conclude that a device poll rate of
every two minutes is sufficient to produce data quality
that can reliably be processed by algorithms to evaluate
significant changes in water quality that is protective of
human health for most locations. However, the utilities
might want to evaluate other polling frequencies.

3.3.5 Data Marking
The SCADA system  should be set up so that calibra-
tion events, bad data, and instrument warnings (low re-
agent) are captured and "marked" within the SCADA
water quality database. This will permit the algorithms
analyzing the data in real-time to exclude these marked
data from further analysis, as any anomalies resulting
from these data are unlikely to be actionable.

3.3.6 Data Transmission and Storage
Data transmission at the T&E Facility and nearby as-
sociated locations use a variety of communication me-
dia, including wired and  wireless  (radio  and cellular)
technologies.

For large SCADA implementations, the majority of the
newer SCADA  software manufacturers recommend
the use of a centralized (or distributed) database as the
back-end data repository. Generally, older data that are
not needed for any real-time  analysis or computations
3-6

-------
      are archived/stored in a database traditionally referred
      to as the "historian."  The real-time and the near real-
      time values are usually run through an event-driven
      calculation engine that either automatically performs a
      task when predefined conditions are met or raises an
      alarm for the operator to intervene or acknowledge the
      SCADA value exception.  Typically,  the commercial
      SCADA systems only  allow the operator to define set
      points (both at high and low levels) for each monitored
      parameter to trigger an alarm.

      The database  storage and retrieval mechanism allows
      for add on algorithm-type programs, which can evalu-
      ate the water quality data in real-time, compare it to the
      baseline data in the database, and raise alerts and alarms
      based on computed values and data trends.

      The T&E Facility water quality data is stored at mul-
      tiple locations.  The iSIC  datalogger and the iChart
      application store the data in proprietary formats. The
      iChart stores the data in encrypted extensible Markup
      Language (XML) and  also pushes the data to a sepa-
      rate Open Database Connectivity  (ODBC)-compliant
      MySQL database for storage and retrieval over the net-
      work using commonly available tools. Figure 3.8 shows
      the T&E Facility NexSens iSIC  datalogger.
Figure 3.8 T&E Facility NexSens iSIC Datalogger


      3.4 Best  Practices for Instrument
           Setup and  Data Acquisition
      Each utility, equipment developer/manufacturer should
      review the site-specific requirements identified in Sec-
      tion 3.1  of this document. For the  utilities, if techni-
      cally feasible, a single standard type of panel mount for
      housing  all of the instrumentation/ SCADA is recom-
      mended. In cases where a one-size-fits-all solution is
      not possible  (due to space constraints), no more than
      two or three types of  standard panel designs are rec-
      ommended for field implementation. Each of the addi-
tional panel types could be designed to eliminate site-
specific length, width or depth constraint(s). The panel
standardization makes the fabrication and maintenance
easier. Within each type of panel, the following design
factors are of prime importance:

   •   Each instrument should be both electrically and
      hydraulically isolated (i.e., each instrument has
      its own circuit breaker, separate water inlet with
      a ball valve).

   •   Flow monitoring devices should be non-
      fouling (i.e., a rotameter without flow control
      or float guide-wire, which tends to accumulate
      biological growth and particle debris) and
      instrument-specific in the correct flow range.

   •   To conserve water, some of the instruments can
      be designed to accept the discharge of another
      non-reagent based instrument.

   •   In cases where the inlet water tends to be colder
      than the environmental housing, degassing
      (bubbles) can negatively impact the performance
      of some instruments. The discharge side can be
      pressurized for some instruments to minimize
      the degassing effect. In other cases, a bubble
      trap or a constant head mechanism could prove
      effective.

   •   The panels should be accessible and  well-lit
      (with an external light source)

   •   The sites should have sufficient space to be
      ergonomically efficient. This will prevent the
      operator from taking shortcuts while performing
      maintenance activities.

   •   There should be a workbench, restrooms, a
      place to store supplies and chemicals onsite to
      maximize operator efficiency.

Equipment manufacturers should try to minimize the
footprint of their device and ensure that  the housing is
NEMA-compliant. Wherever possible, the fluid lines
should include moisture sensors and be below the elec-
trical and  data  acquisition  components to  minimize
damage in  case of a leak. In case of malfunctions, the
instruments should be robust and have an alarm func-
tion and self-restarting capability.

Data acquisition using field RTUs should be standard-
ized by the water utility so that  the programming can
be simplified and replicated across sites. If data trans-
mission at  a particular location is prone to interfer-
ences, programmatic error control techniques should
be applied to mitigate the  errors. The data acquisi-
tion and communication units should be UPS-backed
and  equipped with lightning/surge protection. The
                                                                                                             3-7

-------
      manufacturers should ensure that the sensor is able to
      communicate with the field RTUs, using the common
      SCADA communication protocols.

      Wireless data transmission should use secure protocols
      where possible.  Other SCADA related cyber-security
      recommendations should be implemented whenever
      possible. The data should be stored in databases with
      ODBC connectivity and routinely backed up. The
      ODBC connectivity enables the use of third party data
      analysis  tools/applications or  event detection algo-
      rithms such as CANARY to interface with the data in
      real-time. In addition, data should be marked for instru-
      ment alarms, errors, and calibration events so that they
      can be filtered out by the algorithms while analyzing
      the data for anomalies.
Water Utilities and
Sensor Manufacturers
Trade-offs should be considered when locating
online sensors at the optimal TEVA-SPOT
identified location or at an alternate nearby
utility-owned location that meets site-specific
requirements identified in Section 3.1.

Pressure fluctuations, flow control, bubble
formation, and higher pH values might impact data
quality of many online sensors. Manufacturers
should provide robust non-fouling flow controls
with the sensor and eliminate the potential for
bubble formation in their equipment. Both utilities
and manufacturers should consider the addition of
pressure regulators and constant head devices prior
to sensor elements.

Manufacturers should add alarm output channels to
identify instrument-related problems such as low
reagents, instrument calibration drifts, etc. Also,
manufacturers should provide a variety of interface
options for SCADA communications protocols.
In addition, the instruments should be designed to
have a small footprint with built-in self-restarting
capability in case of malfunctions.
                                                                  Utilities considering setting up a panel of
                                                                  instruments should review the important panel
                                                                  design factors identified in Section 3.4. Utilities
                                                                  should also standardize the data acquisition
                                                                  approach and follow the best practices identified in
                                                                  Section 3.4.

                                                                  Online TOC monitoring equipment employing
                                                                  UV-persulfate methods are expensive and
                                                                  difficult to maintain. Factory service contracts are
                                                                  recommended. One of the TOC instruments, as
                                                                  tested at the T&E Facility, requires a carbon-free
                                                                  air source (i.e., a compressor/generator or nitrogen
                                                                  tanks). Manufacturers should design and fabricate
                                                                  simplified TOC monitoring devices.

3-8

-------
4.0  Testing Procedures

and  Safety Precautions

Prior to evaluating various online sensors at the T&E
Facility, a Quality Assurance Project Plan (QAPP) and
a Health and Safety Plan (HASP) were developed. The
QAPP outlined the experimental and analytical objec-
tives.  The HASP outlined the safety precautions nec-
essary for handling the selected contaminants.  Some
of the critical elements obtained from the QAPP and
HASP that are applicable to the testing program (which
might benefit water utilities and manufacturers of sen-
sors considering such internal testing) are described in
the following sections.

4.1   Blank/Control Injection
Prior to injecting any contaminant into the DSS (Single
Pass or Loop No. 6), a  control run was made to ensure
that there were no significant contributions to the base-
line water quality sensor response from the pumping ac-
tion of the injection apparatus, the DSS itself, or any
associated instrumentation. The blank/control injection
matrix was designed to  match the water matrix for each
specific contaminant and was either Cincinnati tap water
or granular activated carbon-filtered Cincinnati tap wa-
ter. This procedure ensures that significant changes are
not caused by either the absence of the selected disinfec-
tant (chlorine, chloramines) or naturally occurring mate-
rial in the injected water. Figure 4.1 shows the injection
apparatus used for testing on the Single Pass DSS.

4.2  Contaminant Injection
      Procedures
As discussed previously in Section 2.2, various contam-
inants, surrogates, carrier/growth media were selected
to represent a range of chemical, biological and radio-
logical agents that might be  accidentally  or intention-
ally introduced into a water distribution system. This
section provides details on the injection specifics.

4.2.1  Concentration of the Injected
      Contaminant
Several factors were considered while establishing the
injected concentration/dosage of the selected contami-
nants. These included the following: solubility of the
selected contaminant in water, results from the bench-
scale minimum dose sensor response study, and target
concentrations lower than the Immediately Dangerous
to Life or Health (IDLH) level for the selected contami-
nant. Mixing times and solubility observations were
made from beaker tests before performing the injection
event. The bench-scale  minimum dose sensor response
study was performed to determine the "detection limit"
associated with a particular water quality monitoring
sensor for the selected contaminant. The purpose of the
bench-scale and the DSS Loop No. 6 and Single Pass
DSS studies was to determine if it was possible to inject
a contaminant at a concentration that was high enough
to cause health effects, but could not be detected by the
array of sensors. Contaminant concentrations of 1 mg/L
were typical for both the DSS Loop No. 6 and the Sin-
gle  Pass DSS injections.  This concentration was usu-
ally detectable by at least one water quality sensor; yet,
for a vast majority of the contaminants, it represented a
concentration well below the IDLH level. In compari-
son, other EPA-sponsored Environmental Technology
Verification (ETV) studies have been conducted us-
ing  contaminant injection concentrations of 10 mg/L,
which are generally well  within the detection range of
most instruments and suitable for tracking the precision
and accuracy of the test instruments.

4.2.2 Duration of Injection
For the purposes of determining the minimum duration
necessary to detect a water quality  baseline change,
2-minute injections  were  performed. These  short-
duration injections were  successfully detected by the
sensors, even though some of the instrument sampling
durations exceed the 2-minute injection period. To eval-
uate total dosage necessary to cause  potential harm to
humans, a longer 20-minute injection duration was se-
lected. This duration also allowed for stable tests with
a maximum response time long enough to see a change
in baseline that could be detected by automated algo-
rithms. After injection, data from the various sensors
were monitored and recorded for at least 4 hours for
the DSS Loop No. 6 tests, and for at least 1 hour for the
Single Pass DSS tests. The algorithms and data analysis
techniques are discussed in Chapter 5.

4.2.3 Water Main Flow Rate and  Injection Rate
The flow rate through DSS Loop No. 6 was typically
kept at 88 gpm, which also translates to  a velocity of
                                                   Figure 4.11njection Apparatus for the Single Pass DSS

                                                                                                     4-1

-------
      1 foot per second (ft/sec) through the 6-inch pipe. This
      velocity  is commonly  encountered  in  a  distribution
      system. Similarly,  most of the testing that was con-
      ducted on the Single Pass DSS was performed at a flow
      rate of 22 gpm, which also yielded  a velocity  of ap-
      proximately 1 ft/sec in the 3-inch pipe. All of the test-
      ing was conducted under turbulent flow conditions. In
      some of the tests, the flow rate of the Single Pass DSS
      was varied to obtain the desired contaminant dilution
      effect based on available stock concentrations. The fol-
      lowing flow rates were used for testing in the  Single
      Pass DSS: 5, 10.7, 22, and 40 gpm. The typical injec-
      tion rate was 0.5 liter per minute  (0.13 gpm). When-
      ever technically possible (based on solubility and any
      other constraints discussed in Section 4.2.1), a batch of
      10 liters  of contaminant in water was injected  over a
      20-minute period.

      4.2.4 Neat Compounds Versus Commercial
           Off-the-Shelf Products
      For the purposes of evaluating how effective the online
      sensor instrumentation is in detecting  herbicides and
      pesticides, manufacturers of some of the commercially
      available off-the-shelf products were contacted to ob-
      tain the neat (or pure) form of the  active ingredient
      in the product. During the bench-scale studies, it was
      discovered that the inactive ingredient in  commercial
      off-the-shelf herbicides/pesticides might change water
      quality in a detectable manner. For the DSS Loop No.
      6 testing, Real Kill® (pesticide) and Roundup® (her-
      bicide) were used to represent large groups of similar
      commercially available compounds that  are  readily
      available and accessible.

      4.2.5 Wastewater and Ground Water Injections
      Wastewater and ground water injections were con-
      ducted to simulate natural or accidental contamination
      events  such as cross-connections  and broken mains.
      There  have  been  cross-connection and back flow
      events  reported where contaminated wastewater has
      entered the distribution system. Also, it is possible for
      mains under the water table to seep or infiltrate ground
      water.

      4.3 Testing and Analytical
           Confirmation
      In addition to the blank and control injections described
      in the previous section, the DSS tests were repeated
      both for DSS Loop No. 6 (in triplicate) and Single Pass
      DSS (in duplicate) to ensure that the sensor responses
      were valid and repeatable. The EPA ETV studies con-
      ducted at the T&E Facility also evaluated inter-unit re-
      producibility by deploying multiple units concurrently
      for testing  purposes. In addition,  NHSRC's TTEP is
      designed to provide reliable information regarding the
performance of homeland security related technolo-
gies. More information on the TTEP program can be
obtained from the EPA website: http://www.epa.gov/
nhsrc/ttep.html.

4.3.1 Testing Confirmation
The  initial rounds of triplicate  testing in DSS Loop
No. 6 yielded consistent results based on the direction
of parameter-specific change. For the later rounds of
testing, only duplicate runs were performed. In case a
test run yielded inconsistent results due to equipment
malfunction, an additional test  run was performed as
needed. Figure 4.2 shows a sample graph of triplicate
test results for glyphosate conducted on DSS Loop No.
6. Figure 4.3 shows a sample  instrument response with
increasing injected contaminant (glyphosate) concen-
trations, conducted in the Single Pass DSS.

4.3.2 Analytical Confirmation
Bench-top analytical tests were performed to confirm
the water quality parameter readings of the online in-
strumentation. As shown in Figure 4.2, the grab samples
matched the results of the online instrumentation, with
the exception of ORP. The ORP readings are altered
when the sample is exposed to  atmosphere during the
grab sampling event. In addition, for some of the con-
taminants (malathion and glyphosate), to ensure  that
the injected contaminant  was not absorbed/adsorbed
into  the biofilm or pipe material, grab  sampling from
the sample taps of online sensor instrumentation  was
performed. These grab samples  were submitted to an
outside laboratory to perform analytical confirmation.
Although the analytical results confirmed the presence
of these  contaminants, the concentration  levels were
found to vary. The varied results were attributed to the
following: 1) relatively poor analytical methods, which
were chosen by the outside laboratory; 2) injected com-
pounds interacted with free chlorine in the test water,
which might have resulted in the generation  of other
by-products that were not measured; and 3)  possible
adsorption/absorption to the biofilm. However, as the
changes in the measured water quality parameters were
consistent with the injected level of contaminants (as
shown in Figure 4.3), the analytical confirmation  was
abandoned to keep up with the rapid pace of testing and
to reduce project costs.

4.4 Flushing and Baseline
      Establishment
Between the test runs, DSS Loop No. 6 was continuous-
ly operated to flush the system. In addition, prior to the
test runs, DSS Loop No. 6 was sufficiently flushed so
that  the water quality parameters (especially turbidity
and temperature) equilibrated and remained stable dur-
ing the test. This parameter stability was confirmed by
4-2

-------
   1.30
   1.10
   0.90
   0.70
   0.50
   0.30
   0.10
   -0.10
                              T
      >- Free Chlorine (Grab) DPD - colorimetric Run 1
      ^—Free Chlorine (OnLine) DPD - colorimetric Run 1
      I • Free Chlorine (Grab) DPD - colorimetric Run 2
      ^—Free Chlorine (OnLine) DPD - colorimetric Run 2
      k - Free Chlorine (Grab) DPD - colorimetric Run 3
      ^—Free Chlorine (OnLine) DPD - colorimetric Run 3
                                3 min.
                                        T- 60 min.
                                  T-3Hr.
                                  i I  i • i
      -2.00   -1.50   -1.00   -0.50   0.00   0.50   1.00   1.50   2.00   2.50   3.00   3.50   4.00   4.50   5.00   5.50   6.00
                                                   TIME (hours)
Figure 4.2 Glyphosate Triplicate Injection Run Results
evaluating the online instrument baseline data to ensure
that the instrument readings were within the "normal"
range of operation. If the instrument readings deviated
from normal conditions (based on operator experience),
the instrument was recalibrated to ensure accuracy and
repeatability.

4.5  Health and Safety Precautions
Standard laboratory personal  protective equipment
such as laboratory coats, gloves, safety glasses, and
safety  shoes were  required
during  the  experiments. For
chemical  contaminants, ad-
ditional    test/contaminant-
specific protective gear might
be  required  in accordance
with the Material Safety Data
Sheet or contaminant-specific
HASP.
For biological contaminants,
depending upon the contami-
nant,  the biohazards and the
risk  of  infection should  be
minimized. All  of  the bio-
logical contaminants used for
the testing at the T&E Facil-
ity were non-pathogenic. The
surrogates closely  represent
the biological activity of real
                     pathogens (Edberg et al, 2000; Lytle and Rice, 2002;
                     Rice et  al., 2005; Sivaganesan  et al., 2006). How-
                     ever, as part  of good  laboratory practice,  standard
                     Biosafety Level 1 measures were implemented. Per-
                     sonnel had to change gloves after coming in contact
                     with items that might carry biological contaminants.
                     Gloves could not be placed near the face after expo-
                     sure to biological contaminants. Any positive refer-
                     ence materials were handled with gloves in an appro-
                     priate laboratory hood.
             Test 1 - 0.4 mg/L
               st 1 Dup - 0.4 mg/L
           - 'Test 2 - 1.5 mg/L
               st 2 Dup - 1.5 mg/L
           - 'Test 3-3.0 mg/L
               st 3 Dup - 3.0 mg/L
  0.1
  0.0
    -0.40
             -0.20
                      0.00
                               0.20
                                                0.60
                                                         0.80
                                        0.40
                                    Time (hours)
Figure 4.3 Glyphosate Injections at Varying Concentrations
                                                                  1.00
                                                                           1.20
                                                                                                          4-3

-------
      Equipment and supplies that came into contact with
      suspected biohazard materials had to be sterilized prior
      to disposal or reuse. The contaminated equipment/sup-
      plies were sterilized by either standard autoclaving or
      wiping with 0.02% bleach solution, depending on the
      extent of contamination and the type of material consti-
      tuting the equipment/supplies. Waste samples had to be
      autoclaved prior to disposal. Special precautions, such
      as donning heat-resistant gloves, were required for au-
      toclaving.

      4.6  Disposal of Contaminated
            Water From Test  Runs
      The EPA T&E Facility operates under a discharge per-
      mit from the Metropolitan Sewer District (MSD). This
      permit authorizes the direct discharge of specified lev-
      els of contaminants to the local,  publicly owned waste-
      water treatment facility. The aforementioned sensor
      technology testing at the T&E Facility  was conducted
      so that all of the test water could be directly discharged
      to the sewer system. Utilities and sensor manufacturers
      considering such testing at their  facilities should evalu-
      ate their contaminant-specific discharge limits prior to
      initiating a testing program.  If the discharge limits for
      the selected contaminants are too low, it might be nec-
      essary to make alternate arrangements  for disposal of
      the test water (e.g., local treatment before discharge,
      offsite shipment) or modification of the permit.

      4.7  Best Practices for Testing and
            Safety Precautions
      In order to avoid positive bias from any of the injection
      equipment/sampling or monitoring equipment, all test-
      ing components should be disinfected and calibrated so
      that an accurate baseline is established prior to testing.

      Test contaminants (or  surrogates) should  be selected
      so that they represent a broad class of potential threat
      agents. The target concentrations should  be at or be-
      low the levels where human health can be  adversely
      affected, including considerations for sensitive/suscep-
      tible subpopulations. The selected duration of injection
      should be optimized to minimize the use of contami-
      nants for both cost control and waste discharge consid-
      erations. Whenever possible, for instruments measuring
      physical and chemical  parameters, bench-scale testing
      is recommended prior  to pilot-scale or full-scale test-
      ing to determine the levels at which the selected instru-
      ments can detect the selected contaminants.

      A QAPP can  help establish a  detailed experimental
      plan that identifies specific types and quantities of the
      contaminant(s) involved during  the testing. The QAPP
      can also help to define the overall experimental objec-
        Water Utilities and
        Sensor Manufacturers
       Control and blank injections should be performed
       to ensure that the water quality sensors are not
       impacted by the injection apparatus.

       The testing at the T&E Facility revealed that
       the commercially available online water quality
       sensor equipment can generate reproducible data
       responses with duplicate contaminant injections
       and also at varying concentration levels.

       Stable or predictable baseline water quality levels
       are needed to obtain useful data from the online
       water quality sensors. The variation in background
       values should be considered when locating online
       sensors. Also, baseline data should be collected for
       a sufficient time period to capture normal water
       quality variability for each location.

       Without varying water demands, contaminants were
       found to travel as a slug or in plug flow within the
       Single Pass system. Recirculating DSS Loop No. 6
       experienced fully mixed conditions within several
       minutes.
       Utilities and manufacturers considering inhouse
       testing should select contaminants that represent
       a broad class of potential threat agents, develop
       a detailed experimental plan/QAPP/HASP,
       and evaluate potential disposal options prior to
       conducting any test runs.
lives, standardize the experimental procedures, estab-
lish protocols for instrument calibration (prior to test-
ing), and establish data quality that can be technically
defensible when reviewed.

Prior to any testing, a HASP should be developed, re-
viewed, and approved by appropriately trained person-
nel so that the tests can be performed safely. The HASP
will identify minimum job hazards and controls, sample
handling techniques, personal protective  equipment,
work practices and  engineering controls,  and spill/
emergency procedures.  This documentation also helps
in determining if the test water can be directly and safe-
ly discharged (without treatment) based on the facility's
existing discharge permit. Otherwise, it will be neces-
sary  to make arrangements  for appropriate waste-han-
dling procedures. In addition  (if  needed), HASPs can
identify appropriate safety training programs, personal
monitoring needs, and  medical surveillance based on
the contaminant and concentration used.

4-4

-------
5.0  Data Analysis

Water quality sensor  response data generated  at the
T&E Facility enabled EPA to construct a qualitative
sensor parameter response matrix for the contaminants
tested. Analysis of this data required that the concept of
a significant change from baseline water quality  condi-
tions be defined. A significant change is a large enough
deviation from normal water quality parameters that
could be used to trigger an alarm to be transmitted to
the data user (i.e., a drinking water utility). The criteria
for determining a significant change in the sensor pa-
rameter was subjective at the early stages of the investi-
gations. Initially, significant change  determination was
based on visual qualitative inspection (e.g., drop  in free
chlorine and increase  in measured TOC value)  of the
plotted sensor responses over the course of the contami-
nant injection test runs. Later on,  the significant change
determination was based on a quantitative approach: the
maximum change observed within a short time  period
of contact (defined as 15 minutes) of the sensor with
the contaminant was divided by the baseline  value of
the parameter to compute a percent change or deviation
(Hall et al., 2007). Although this method was simple
and  straightforward, it omitted several critical factors
such as slow sensor response times and noisy back-
ground  data. Therefore, the same response data was
also  evaluated using basic statistical methods that are
described in Section 5.1 of this report. As demonstrated
later in  this chapter, the significant change threshold
is dependent upon the variability of the baseline water
quality data at a particular monitoring site. Based on the
testing conducted at the T&E Facility, EPA developed
and utilized the significant change thresholds presented
in Table 5.1 for evaluating contaminant injection sen-
sor response data. These threshold  values are not en-
tirely based on measured or statistically derived values;

Table 5.1 Parameter-Specific Significant Change
Thresholds
Deviation from
Baseline Classified as
Water Quality Parameter "Significant Change"
Temperature
Specific Conductance
Dissolved Oxygen
Oxygen Reduction Potential
Nitrate
Chloride
Ammonia
Turbidity
Free Chlorine
Total Organic Carbon
±0.15°C(~0.27°F)
> 5% increase
± 0.2 milligrams per liter
± 20 millivolts
± 10%
± 15%
± 20%
> 200% increase
> 5% decrease
> 0.1 mg/l increase
the operator's understanding of the variability of water
quality at each location should also be taken into ac-
count while developing these parameter-specific signif-
icant change thresholds. For example, the baseline wa-
ter quality parameters observed at the T&E Facility are
stable with little variance. Therefore, using these values
at a location where the water quality baseline is highly
variable can lead to triggering of an excessive number
of false positive alarms. Keeping these observations in
mind, it is recommended that the end-users employing
this methodology should develop their own site-specific
significant  change thresholds for evaluating real-time
water quality data.

During the course of this research, EPA was aware that
the significant  change  data analysis approach, which
used visual inspection of time series data and percent
change from baseline, could lead to  variable results
caused by  site-specific water  quality  differences and
analyst bias. Therefore, a more sophisticated quantita-
tive approach  was  undertaken:  individual  sensor re-
sponses were analyzed by computing absolute change,
percent change, and S/N ratio. Absolute change and
percent change analysis employ the same mathematical
techniques  described previously in  the development of
significant change thresholds. The S/N ratio analysis is
designed to filter the level of "background noise" caused
by frequent fluctuations in the baseline data. The S/N
ratio is defined as the ratio of a measured value to the
background noise. A low S/N ratio indicates that the
change in measured value of the parameter might be
caused by the background noise (representing routine
fluctuations in measured baseline) rather than resulting
from a real change in water quality due to the presence
of contaminants (Szabo et al., 2006, 2008a, and 2008b).
Furthermore, EPA's field installation experience has in-
dicated that the baseline water quality at certain  loca-
tions (that are  immediately influenced by utility opera-
tions) can change significantly over a short time period.
For example, monitored parameters might fluctuate dra-
matically with  changes in the operation of tanks, pumps,
and valves. The monitored parameters are also affected
by daily and seasonal changes in the source and finished
water quality,  as well as fluctuations in  demand. EPA
collaborated with SNL to build an  automated algorith-
mic data analysis tool that combines and enhances  some
of the previously mentioned qualitative and quantitative
approaches to distinguish between normal variations in
water quality and changes in water  quality triggered by
the presence of contaminants (McKenna, et al., 2006).
These types of tools are often referred to as event de-
tection algorithms, which can read SCADA data (water
quality signals, operations data, etc.), perform analysis
in near real-time, and return a 0/1 result (indicating pres-
ence or absence of an alarm). All of these approaches are
discussed further in this chapter.
                                                                                                          5-1

-------
      5.1  Non-Algorithmic Sensor
            Response Evaluation
      The qualitative and quantitative approaches employed
      to evaluate the data generated at the T&E Facility and
      Edgewood Chemical and Biological Center (ECBC)
      are discussed in the following subsections.

      5.1.1  Single Pass DSS Data Analysis
      The data tabulated and reported in this section was
      generated by  injecting  selected contaminants  (as
      listed in Tables 5.2 through 5.7) into the Single Pass
      DSS using chlorinated tap water available at the T&E
      Facility (as supplied by GCWW). As mentioned pre-
      viously, the background  values are generally stable
      at this location. The range of background values for
      the routinely measured water quality parameters are
      as follows: free chlorine  - 0.8 to 1.1 mg/L, specific
      conductance - 300 to 600 nS/cm, ORP - 500 to 700
      mV, pH - 8.5 to  8.8 standard units,  turbidity  <  0.1
      NTU, and TOC - 0.3 to  1.3 mg/L. Table 5.2 shows
      the percent change of several water  quality param-
      eters for various injected contaminants in the Single
Pass DSS (described in Section 2.1.2). The qualitative
response information  in Table 5.2 is color-coded to
show changes that exceed 10% from the baseline val-
ue. Percent change is calculated by first calculating
the difference between the baseline mean over one
hour before injection,  termed absolute change (AC):
                     peak
                            baseline
Where, S   ... = the peak sensor value between when
the contaminant is in contact with  the  sensor until
15 minutes after and Sbaseline = the mean baseline value
for one hour immediately preceding the  contaminant
injection test.

Percent change is then calculated as follows:
                             AC
             % Change =
                            baseline
Table 5.2 allows for easy identification of online wa-
ter quality monitoring parameters that would poten-
tially respond to the specified injected contaminants.
An  examination of Table 5.2 reveals that free/total
chlorine and turbidity provide a  significant change
Table 5.2 Percent Change in Sensor Parameter Response to Injected Chemical Contaminants in
Chlorinated Water - Single Pass DSS
Aldicarb
Glyphosate
Colchicine
Dicamba
Dimethyl Sulfoxide
Lead Nitrate
Mercuric Chloride
Nicotine
Potassium
Ferricyanide
Sodium Thiosulfate
(Anhydrous)
Sucrose
Control Blank
0.2
1.1
2.2
0.4
1.5
3.0
0.4
1.8
3.6
0.8
1.3
2.6
0.6
2.0
4.0
0.6
0.7
1.4
0.4
1.1
2.2
0.4
1.9
3.8
0.6
1.6
3.2
0.2
1.3
2.6
0.6
1.8
3.6
0
0
AVGa
-9.0%
-43.6%
-87.6%
-34.4%
-77.9%
-95.2%
-2.7%
-4.3%
-5.6%
-2.8%
-3.1%
-1 .9%
-11.8%
-29.2%
-46.9%
-3.9%
-2.9%
-0.3%
-2.5%
-1 .0%
-1.1%
-14.3%
-49.3%
-84.7%
-3.7%
-5.7%
-4.1%
-15.1%
-75.5%
-98.8%
-2.9%
-3.2%
-2.6%
2.4%
1 .0%
1 .7%
-8.8%
-43.5%
-82.9%
-17.1%
-39.3%
-52.4%
0.3%
-3.9%
-4.2%
0.0%
-1.7%
0.5%
-9.5%
-26.0%
-42.6%
1 .3%
-2.0%
-2.0%
-1.2%
-1.5%
0.1%
-7.9%
-28.6%
-47.8%
0.8%
8.0%
21.2%
-13.3%
-72.1%
-95.1%
-0.7%
-0.4%
-0.1%
1 .6%
0.2%
0.9%
2.3%
-2.8%
2.7%
3.0%
2.0%
2.6%
1 .6%
0.1%
-0.6%
-1.4%
-1.3%
-0.6%
-1.5%
-0.1%
0.1%
-5.4%
-0.8%
-0.7%
-1.6%
-1.7%
-2.3%
2.4%
0.5%
2.4%
1 1 .9%
14.5%
26.3%
2.8%
2.2%
4.9%
-1.0%
-0.9%
-0.7%
5.5%
0.6%
3.1%
0.2%
0.0%
0.0%
-0.0%
0.0%
-0.0%
0.1%
0.4%
0.0%
-0.0%
-0.3%
0.2%
0.1%
0.1%
0.1%
0.0%
-0.1%
-0.1%
0.0%
0.3%
0.4%
0.2%
0.0%
0.1%
-0.1%
0.5%
0.8%
0.4%
0.6%
0.8%
0.1%
0.0%
0.0%
0.5%
1.1%
0.8%
0.0%
-1.2%
0.1%
1 .2%
1 .3%
0.6%
3.2%
1 .3%
-5.0%
-0.6%
-0.5%
-0.6%
-0.4%
-0.9%
-1.3%
2.3%
-3.4%
-0.9%
0.0%
-7.3%
-8.0%
1 .4%
1 .5%
-1.5%
-0.4%
-0.6%
-0.1%
-0.1%
-1.2%
-2.6%
1 .4%
-5.7%
-3.8%
0.5%
1 .0%
0.8%
-0.3%
-0.6%
-1.2%
0.1%
-0.9%
-2.6%
0.2%
-0.4%
-0.7%
0.6%
0.6%
0.5%
0.2%
-0.3%
-0.3%
0.2%
-0.1%
-0.9%
0.2%
0.5%
0.8%
0.2%
-2.4%
-4.8%
0.5%
-0.3%
-0.1%
0.1%
-1.1%
-5.3%
3.7%
1.1%
0.4%
1 .8%
2.0%
1 .9%
0.3%
0.0%
-0.1%
-0.1%
-1 .4%
-4.6%
0.3%
0.2%
0.2%
-0.3%
-0.5%
-0.7%
0.2%
0.3%
0.1%
-0.2%
-0.1%
-0.1%
-0.1%
-0.7%
-1 .2%
0.3%
0.7%
1 .0%
0.1%
-0.2%
-0.1%
0.3%
-0.1%
0.1%
0.0%
-0.1%
-0.1%
0.2%
0.1%
0.2%
188.5%
487.0%
-100.0%
1003.5%
329.2%
-100.0%
593.3%
157.0%
111.1%
0.0%
-6.9%
-7.1%
4.4%
1 .0%
2.0%
400.0%
137.2%
538.4%
26.3%
46.9%
-1 .4%
1200.0%
82.0%
143.4%
287.7%
503.2%
50.0%
883.3%
385.4%
645.2%
167.8%
449.1%
433.3%
52.5%
71 .8%
62.1%
I  I indicates that the percent change was >10% of the baseline within 15 minutes.
a Average of two control blank runs. Data presented in this table was not corrected to accommodate for the control blank response.
5-2

-------
      signal for a majority of the contaminants tested.

      Table 5.3  shows further manipulation of the same da-
      taset (presented previously in Table 5.2). The data pre-
      sented in Table 5.3 has been normalized and adjusted to
      correct for the S/N ratio for each monitored parameter.
      The S/N ratio was calculated as follows:
                                  AC
                      S/N= 	
                                 baseline

      where CTbaseline = the standard deviation in the baseline
      for one hour prior to injection.

      In Table 5.3, further manipulation of the original data-
      set (see Table 5.2), reveals that the turbidity parameter
      is no  longer a good indicator of significant change, as
      shown in Table 5.2, because the baseline signal is too
      noisy. The data in Table 5.3 also reveal that there are
      other  water quality parameters that might be good at
      detecting contamination, but  do not  appear so at first
      glance in Table 5.2. For example, pH and ORP changed
      in response to four and seven of the eleven injected con-
taminants,  respectively. Both  parameters have stable
baselines (low standard deviation) and produced small
changes when contaminants were injected. However,
because the baselines were so stable, the normalized
and S/N-adjusted change was relatively  large.  Tables
5.2 and 5.3 provide examples  of how non-algorithmic
analyses of water quality data can be useful. The tables
also identify some pitfalls to be avoided when interpret-
ing online data. Additional data on the response of wa-
ter quality sensors to biological suspensions and culture
broth are presented in Tables 5.4 through 5.7.

5.1.2 Recirculating DSS Loop No. 6 Data
     Analysis
The data tabulated and reported in this  section were
generated  by  injecting  selected  contaminants  (as
listed in the individual tables) into  recirculating DSS
Loop No.  6 (described  in Section 2.1.1)  with chlo-
raminated  water prepared at  the T&E Facility.  The
preparation methodology for the chloraminated water
was described  previously in  Section 2.2.1. For per-
forming these tests, the background chloramine level
Table 5.3 Normalized, Signal-to-Noise Corrected Sensor Parameter Response to Injected Chemical Contaminants in
Chlorinated Water-Single Pass DSS
Aldicarb
Glyphosate
Colchicine
Dicamba
Dimethyl Sulfoxide
Lead Nitrate
Mercuric Chloride
Nicotine
Potassium
Ferricyanide
Sodium Thiosulfate
(Anhydrous)
Sucrose
Control Blank
0.2
1.1
2.2
0.4
1.5
3.0
0.4
1.8
3.6
0.8
1.3
2.6
0.6
2.0
4.0
0.6
0.7
1.4
0.4
1.1
2.2
0.4
1.9
3.8
0.6
1.6
3.2
0.2
1.3
2.6
0.6
1.8
3.6
0
0
AVGa
-31.6
-101.0
-309.2
-90.2
-275.7
-380.3
-4.7
-7.2
-19.6
-5.9
-6.3
-3.3
-23.4
-52.5
-72.7
-3.7
-2.4
-2.1
-6.1
-2.2
-2.7
-26.8
-187.2
-243.6
-7.8
-7.5
-5.3
-38.8
-235.1
-158.4
-8.9
-6.0
-4.8
2.3
2.0
2.2
-20.2
-155.1
-216.6
-40.0
-97.6
-130.4
0.5
-9.8
-9.4
-0.0
-5.4
-0.5
-22.1
-110.8
-115.3
2.1
-11.3
-4.0
-4.8
-4.3
0.6
-10.0
-43.0
-91.7
-0.4
13.1
37.1
-37.0
-233.2
-193.8
-1.6
-2.7
1.0
4.9
0.6
2.7
3.5
-5.3
6.3
4.6
11.7
14.9
0.1
0.1
-3.8
-5.7
-12.7
-3.2
-4.5
0.3
1.9
-7.5
-2.8
-9.5
-4.7
-13.1
-10.0
2.0
-2.1
23.1
12.6
58.3
49.7
3.9
12.2
29.8
-0.5
-4.6
-5.0
3.2
0.5
1.9
1.2
0.0
0.0
-0.0
-5.4
-0.3
1.5
3.2
0.0
-0.4
-3.1
1.6
0.0
2.6
0.4
0.3
-1.9
-0.5
-5.4
5.5
1.7
1.0
0.2
0.6
0.9
4.9
4.8
4.2
7.0
5.9
1.8
0.4
0.2
3.2
2.5
2.9
-0.5
-3.6
0.3
4.1
3.8
2.8
0.8
-0.2
-3.0
-7.4
-8.0
-6.6
-6.2
-8.1
-8.2
2.2
-3.1
0.5
-0.3
-1.7
-2.3
3.6
2.8
-2.4
-2.9
-2.2
-0.9
-1.2
-0.3
-2.5
-0.1
-10.4
-3.4
2.0
1.5
1.8
-2.1
-9.8
-40.6
0.6
-14.4
-34.1
2.7
-4.4
-13.1
7.2
9.9
11.4
4.1
-4.6
0.0
2.4
-0.4
-7.8
3.4
8.9
26.6
-2.4
-20.1
-59.2
3.4
-3.0
-3.6
-1.9
-14.9
-29.5
3.5
6.0
4.0
4.0
2.2
3.1
2.1
0.5
-1.9
-1.3
-28.5
-108.4
4.4
3.6
2.7
-4.9
-9.4
-13.1
4.8
5.2
1.9
-3.1
-1.0
4.9
-1.7
-26.8
-33.0
5.4
8.5
13.5
2.3
-3.1
-5.7
2.6
-6.5
-4.3
0.6
-1.8
-2.5
2.7
0.7
1.7
2.5
4.8
-1.1
6.0
2.2
-0.8
2.6
4.5
2.9
0.0
-2.7
-2.5
0.7
1.9
1.8
8.9
3.9
16.3
1.4
4.9
0.7
5.7
0.6
1.0
2.1
1.4
0.4
3.6
3.4
3.3
0.1
1.0
3.0
5.7
1.9
3.8
I  I indicates that the normalized, S/N corrected change was >10 of the baseline within 15 minutes.
a Average of two control blank runs. Data presented in this table was not corrected to accommodate for the control blank response.
                                                                                                                5-3

-------
Table 5.4 Percent Change in Sensor Parameter Response to Injected Biological Contaminants and Growth Media in
Chlorinated Water - Single Pass DSS
      Biological
  Contaminants and
    Growth Media
Initial In-Pipe
Concentration
   (mg/L)
Nutrient Broth
Trypticase Soy Broth
Terrific Broth
E.coli in Terrific Broth
Control Blank
0.12
0.48
0.96
0.12
0.48
0.96
0.12
0.47
0.97
0.01
0.07
0.14
0
0
AVGa
-1.4%
-9.1%
-21.6%
-6.7%
-13.4%
-20.4%
-3.0%
-13.8%
-25.5%
-20.1%
-70.5%
-89.2%
2.4%
1 .0%
1 .7%
-2.2%
-4.1%
-6.4%
-4.6%
-9.3%
-13.0%
1 .0%
-3.6%
-6.4%
-7.0%
-12.4%
-9.6%
1 .6%
0.2%
0.9%
0.1%
-0.1%
1.1%
-1.1%
-0.3%
0.2%
2.7%
-1.0%
-0.7%
1 .8%
4.6%
10.7%
5.5%
0.6%
3.1%
0.9%
0.2%
0.2%
0.2%
-0.0%
0.1%
0.0%
-0.1%
0.1%
0.1%
0.3%
0.8%
0.5%
1.1%
0.8%
-2.3%
-0.5%
-1 .6%
-0.2%
0.1%
-0.4%
0.8%
-0.1%
-0.2%
-0.9%
-0.7%
-0.9%
0.5%
1 .0%
0.8%
-1.1%
-1 .4%
-1.1%
-0.1%
-2.0%
-4.4%
0.5%
-0.4%
-1 .2%
-1 .9%
-5.8%
-7.6%
1 .8%
2.0%
1 .9%
-0.2%
-0.1%
-0.1%
-0.1%
-0.2%
-0.0%
0.2%
0.2%
0.5%
0.1%
-0.6%
-1.3%
0.2%
0.1%
0.2%
-20.7%
-14.1%
3.3%
60.0%
0.4%
0.0%
15.3%
-19.3%
-3.0%
170.8%
192.5%
296.9%
52.5%
71 .8%
62.1%
 HH indicates that the percent change was >10% of the baseline within 15 minutes.
 a Average of two control blank runs. All data presented in this table was corrected to accommodate for the expected average control blank response.

Table 5.5 Normalized, Signal-to-Noise Corrected Sensor Parameter Response to Injected Biological Contaminants
and Growth Media in Chlorinated Water - Single Pass DSS
      Biological
  Contaminants and
    Growth Media
 Initial In-Pipe
Concentration
   (mg/L)
Nutrient Broth
Trypticase Soy Broth
Terrific Broth
E.coli in Terrific Broth
Control Blank
0.12
0.48
0.96
0.12
0.48
0.96
0.12
0.47
0.97
0.01
0.07
0.14
0
0
AVGa
-9.4
-13.3
-38.2
-8.7
-20.2
-24.0
-6.4
-43.2
-65.6
-21.2
-127.5
-153.6
2.3
2.0
2.2
-4.4
-13.8
-18.6
-12.8
-31.8
-25.8
0.8
-4.8
-9.7
-15.9
-20.2
-29.0
4.9
0.6
2.7
0.5
-0.4
6.1
-3.5
-1.7
0.2
2.6
-3.8
-6.2
2.2
5.8
5.6
3.2
0.5
1.9
4.9
0.6
3.0
0.7
-3.7
0.4
-1.4
-1.0
1.4
1.0
2.5
5.7
3.2
2.5
2.9
-0.6
-1.1
-3.1
-0.1
1.2
-1.1
2.6
1.1
1.1
-2.8
-2.0
-3.2
2.0
1.5
1.8
-4.2
-11.6
-18.8
-1.4
-26.7
-47.7
3.4
-7.4
-16.4
-34.4
-51.0
-59.6
4.0
2.2
3.1
-3.5
-2.9
-3.2
-1.9
-5.5
1.8
1.8
4.0
6.7
0.5
-10.0
-22.6
2.7
0.7
1.7
-2.4
-1.3
1.3
1.8
-0.4
0.0
1.0
0.5
0.7
2.1
2.2
2.8
5.7
1.9
3.8
 Q indicates that the normalized, S/N corrected change was >10 of the baseline within 15 minutes.
 a Average of two control blank runs. All data presented in this table was corrected to accommodate for the expected average control blank response.
Table 5.6 Percent Change in Sensor Parameter
Response to B. globigii Injection in
Chlorinated Water - Single Pass DSS
                                        Table 5.7 Normalized, Signal-to-Noise Corrected Sensor
                                        Parameter Response to B. globigii Injection in
                                        Chlorinated Water - Single Pass DSS
   indicates that the percent change was >10% of the baseline within 15 minutes.
a Average of two control blank runs. Data presented in this table was not corrected to
   accommodate for the control blank response.
b Concentration in Cells/mL.
       was set at 2 mg/L (measured as total chlorine). The
       background values of other measured water quality
       parameters are the same as mentioned previously in
       Section 5.1.1. Table 5.8 shows the quantitative sen-
       sor responses to contaminants injected in DSS  Loop
       No. 6 as  absolute change, percent change, and S/N
                                        O indicates that the normalized, S/N corrected change was >10 of the baseline within 15
                                           minutes.
                                        a Average of two control blank runs. Data presented in this table was not corrected to
                                           accommodate for the control blank response.
                                        b Concentration in Cells/mL.

                                       ratio. Also, Table 5.8 shows that the TOC parameter
                                       responded to each contaminant except sodium arse-
                                       nite, which was expected since sodium arsenite is not
                                       an organic (carbon-containing) compound. The TOC
                                       sensor responses are comparable to  those observed
                                       during the same injections in the chlorinated GCWW
5-4

-------
Table 5.8 Quantitative Sensor Parameter Response Matrix to Contaminants in Chloraminated Cincinnati Tap Water
 Chemical
Contaminants ^^^^^|
Glyphosate
(Roundup®)

Malathion (Real Kill®)


Phorate

Wastewater

Sodium Arsenite


Nicotine


Control Blank

1

1


1

0.8v/vc
(2 gallons)

1


10


0

(mg/L)
0.27a
127.5%
15.7
0.40
38.2%
54.7
0.22
18%
48.4
0.18
13.8%
34.0
0.02
1 .7%
3.8
2.71
963%
269.8
0.02
1 .7%
2.1
(mg/L)
-0.04
1 .6%
2.8
-0.04
1 .8%
2.1
-0.22
9.5%
60.0
-0.04
1 .6%
5.7
-0.31
14.6%
48.6
-0.02
0.8%
5.6
-0.01
0.4%
2.0
(mg/L)
N/Ab
5.2
7.7%
3.8

N/A

N/A
20.7
9.5%
4.7
2.3
1.7%
6.6
0.00
0.0%
0.0
(mg/L)
0.1
1.0%
2.7
0.08
1 .5%
1.5
-0.04
1 .5%
0.4
0.06
2.1%
0.5
0.29
4.8%
2.7

N/A

0.03
1.1%
2.8
(mg/L)
6.33
3.8%
6.1
12.85
6.1%
12.9
8.73
17%
34.1
29.73
11.1%
27.8
4.22
1 .7%
1.7
1.13
9.2%
4.5
0.01
0.1%
0.2
(uS/cm)
5.16
0.9%
10.0
2.11
0.3%
4.8
2.57
0.6%
7.1
18.13
3.7%
28.4
0.94
0.2%
1.7
3.05
0.6%
5.3
4.35
0.9%
8.4
(m\/)
N/A
-5.6
1 .7%
8.0
-7.7
2.2%
16.2
-7.0
4.8%
6.2
-74.2
23.9%
84.2
-10.3
2.8%
18.0
-1.0
0.3%
0.0

-0.02
0.2%
3.1
0.03
0.3%
4.8
-0.02
0.2%
3.4
-0.05
0.5%
7.2
0.05
0.6%
6.9
0.02
0.3%
5.1
-0.02
0.3%
5.3
(NTU)
0.56
111.1%
46.2
0.46
112%
43.1
0.29
74.9%
11.5
0.43
116%
23.1
0.53
139%
9.3
1.55
505%
173.3
0.73
160%
19.5
 a Top values in each cell are the magnitude of the change, middle values are percent change, and bottom values are signal-to-noise ratio.
 b N/A indicates a problem with the probe or SCADA system that rendered the data invalid.
 c Two gallons of wastewater were injected, which represents 0.8 percent of the loop volume.
       water. Responses were also similar for other sensors,
       except total chlorine. Total chlorine, as a measure of
       chloramines, showed little change for the contami-
       nants tested, except for decreases of 0.22 mg/L (9.5
       percent decrease, S/N: 60) and 0.31 mg/L (14.6 per-
       cent decrease, S/N: 48.6) for phorate and sodium ar-
       senite, respectively.

       However, since chloramines (mostly monochloramine)
       react more slowly with the injected organic contami-
       nants than with free chlorine, the changes  reported
       in Table 5.8 occurred over a period of four hours. In
       comparison, the free chlorine values changed almost
       instantaneously when the same contaminants were in-
       troduced into DSS Loop No. 6. So it  is important to
       note that the changes in Table 5.8 are  not necessarily
       comparable to those presented in the previous section.
       Furthermore, the slow changes  in chloramine levels
       (measured  as total chlorine) might prevent the person
       (or automated computer algorithm analyzing this data)
       from detecting a significant change unless  the time
       window for this type of analysis was appropriately ad-
       justed to account for the delay (Szabo et al., 2006 and
       2008a; Kroll and King 2007).
5.1.3 Edgewood Chemical and Biological
      Center Test Loop Data Analysis
The data tabulated and reported in this section was gen-
erated by  injecting  selected contaminants  (as listed in
the individual tables) into the recirculating ECBC Test
Loop with chlorinated water supplied by the City of Ab-
erdeen Department  of Public Works (Aberdeen, Mary-
land). The sensor responses to introduction  of chemi-
cal warfare agents in the recirculating ECBC  Test Loop
are presented in Tables 5.9 (% change) and 5.10 (S/N).
The ECBC Test Loop is similar to DSS Loop No. 6 de-
scribed in Section 2.1.2. Both Tables 5.9 and 5.10 indi-
cate that TOC and free/total chlorine are good detection
parameters for all injected contaminants. Also, the small
changes to pH result in a low percent change, but S/N
data analysis presented in Table 5.10 shows  that pH is
a very good detection parameter for the warfare agent
contaminants, given that the baseline is very stable.

The  data  presented in Sections  5.1.1  through 5.1.3
clearly indicates the usefulness of analyzing data in a
qualitative and/or quantitative fashion, but the analyst
should be aware of the usefulness and pitfalls of each
selected approach. In addition, the professional judg-
                                                                                                                 5-5

-------
Table 5.9 Percent Change in Sensor Parameter Response to Injected Warfare Agents in
Chlorinated Water - Edgewood Chemical and Biological Center Test Loop
V-series Nerve Agent
G-type Nerve Agent
Ricin
Potassium Cyanide
0.02
0.2
2.0
0.02
0.2
2.0
0.25
0.25
0.25
0.2
0.2
2.0
-5.7%
-20.2%
-58.8%
-4.7%
-14.3%
-13.2%
-40.2%
-21 .4%
-29.1%
-90.6%
-77.9%
-98.7%
-9.9%
-19.0%
-48.3%
-4.6%
-7.5%
-7.3%
-18.3%
-13.8%
-14.5%
-68.4%
-63.7%
-98.5%
43.8%
20.4%
135.4%
15.3%
20.4%
82.9%
12.9%
14.3%
20.8%
21 .3%
10.0%
53.4%
1.1%
0.4%
0.9%
0.3%
1 .6%
1 .0%
1 .6%
1 .0%
2.2%
1 .4%
1 .0%
5.6%
N/A
N/A
N/A
N/A
N/A
N/A
-0.2%
N/A
N/A
-8.9%
-11.4%
-59.8%
1.1%
-1.3%
2.5%
3.6%
3.8%
5.7%
-0.7%
1 .2%
3.1%
2.5%
4.0%
17.1%
-14.3%
-7.8%
-4.0%
-6.5%
-12.7%
-9.7%
1 .0%
-0.2%
-8.4%
1 .7%
1 .9%
7.5%
  ] indicates that the percent change was >10% of the baseline within 15 minutes.
Table 5.10 Normalized, Signal-to-Noise Corrected Sensor Parameter Response to Injected Warfare Agents in
Chlorinated Water - Edgewood Chemical and Biological Center Test Loop
V-series Nerve Agent
G-type Nerve Agent
Ricin
Potassium Cyanide
0.02
0.2
2.0
0.02
0.2
2.0
0.25
0.25
0.25
0.2
0.2
2.0
1.4
16.7
47.7
3.3
9.8
7.2
17.6
6.0
21.4
35.5
51.6
56.9
13.4
45.7
45.5
9.1
16.7
7.0
17.8
15.7
16.3
69.7
88.6
119.3
647.4
19.5
41.0
19.7
20.3
152.4
21.3
28.3
41.6
17.0
17.8
184.8
7.1
1.9
3.8
2.5
6.6
6.0
8.6
5.3
8.3
10.1
6.7
25.4
N/A
N/A
N/A
N/A
N/A
N/A
1.5
N/A
N/A
44.8
34.6
89.3
5.9
33.8
7.1
12.2
16.2
21.0
10.2
5.9
14.5
9.7
39.9
80.5
38.3
21.6
8.0
11.0
25.3
30.7
1.4
0.2
12.0
1.8
2.7
6.4
   indicates that the normalized, S/N corrected change was >10 of the baseline within 15 minutes.
      ment of the analyst was used to define the usable sig-
      nificant change thresholds such as absolute change,
      percent change, and S/N ratio.  For example, the peak
      sensor value used in the absolute  change and subse-
      quent calculations was computed based on values ob-
      tained between the time that the contaminant contacted
      the sensor and 15 minutes past that time. This definition
      of "time window"  was not necessary  for parameters
      like pH or conductivity, which respond within a minute.
      However, responses for parameters such as ORP devel-
      op more slowly and the maximum change might not be
      reached for an extended period. Hence, the definition
      of time window is necessary to utilize the full sensor
      response. Even though  free/total chlorine parameters
      are good indicators, they might react slowly depending
      upon the injected contaminant and thus delay the obser-
      vation of significant change in the disinfectant residual.
      This delay is much longer for chloraminated systems.
      Furthermore, the threshold at which a statistical pa-
      rameter such as percent change or S/N ratio becomes
      significant or triggers a change alarm is based on the
      judgment of the analyst and the baseline water quality
      at the monitoring point. In general, absolute changes of
      10% and S/N ratio of 10 were deemed adequate to iden-
tify the parameters best suited for detecting contamina-
tion events at the T&E Facility, but these numbers will
vary depending on the  variability of the baseline water
quality data  for a particular location. Ultimately, data
analysis employing these techniques need to be auto-
mated and performed real-time for realizing any event
detection benefits.

5.2 Automated Algorithmic
     Evaluation  of Sensor  Response
Based on the online data collected during the tests per-
formed at the EPA T&E Facility and at the WSi field
locations, EPA confirmed the need for automated data
analysis tools. Specifically, this includes data analysis
tools that can distinguish between normal variations in
background water quality and changes  in water quality
triggered by  the presence of contaminants (O'Halloran
et al., 2009). Often referred to as event detection algo-
rithms, such  data analysis tools can read SCADA data
(water quality signals, operations data, etc.), perform
analysis in near real-time,  and return an event value
or code 0/1 (indicating the presence or absence of an
alarm).
5-6

-------
EPA collaborated with SNL to develop the open source
CANARY software, which is intended to provide this
capability. CANARY  software is designed to accept
standard water quality data and use mathematical al-
gorithms to identify the onset of periods of anomalous
water quality, while at the same time limiting the num-
ber of false alarms that occur. CANARY is trained on
"normal" baseline water quality data by the user during
the setup, and the configuration parameters are selected
to accommodate the normal site-specific variability of
water quality parameters. Therefore, these configura-
tion parameters could vary from one utility to the next
and might even vary across monitoring locations within
a single utility. CANARY can be set up to receive data
from a SCADA database, and return alarms to the SCA-
DA system. In addition, it can be run "offline" on his-
torical data to help set the configuration parameters (or
train the algorithm) to provide the desired balance be-
tween event detection sensitivity and false alarm rates.

CANARY'S open source code is designed to be custom-
izable, allowing outside researchers to develop new al-
gorithms that can be added to CANARY. In the current
version (Version 4.2), CANARY has three change detec-
tion algorithms:  time series increments, a linear filter,
and a multivariate nearest-neighbor algorithm. These
algorithms identify a background "water quality signa-
ture" for each water quality sensor and compare each
new water quality  measurement to the background to
determine if the new measurement is an outlier (anoma-
lous) or not. The definition of the water quality back-
ground  is  updated continuously as new data become
available. A binomial event discriminator (BED) exam-
ines multiple outliers within a prescribed time window
to determine the onset of either an anomalous event or
a change in the water quality baseline. Figure 5.1 shows
the schematic operation of CANARY software.

In addition to the  CANARY software,  EPA has also
tested the  commercially available Hach Event Moni-
tor™ Trigger System. Hach's patented technology
utilizes  the Hach Event Monitor™ Trigger System to
analyze five commonly measured water quality pa-
rameters monitored from the  Hach WDMP (chlorine,
pH, turbidity, conductivity) and the Hach astroTOC™
UV process  TOC analyzer to estimate a water distri-
bution system's operating baseline (i.e., water quality
under normal operating conditions). Thereafter, every
minute,  the  Hach  Event Monitor™ Trigger System
analyzes the sensor data  and computes a trigger  sig-
nal, which indicates the  level of deviation from the
water quality baseline. If significant deviations oc-
cur, the trigger signal sends  alarms to the operators
in real-time. Once a deviation is detected, the Hach
Event Monitor™ Trigger System signals the (optional)
automatic  water sampler to capture a water sample at

       Water Utilities and
       Sensor Manufacturers
       Online monitoring data should be evaluated both
       qualitatively and quantitatively to identify which
       parameters provide significant change signals and
       have relatively stable baselines with low S/N ratio.
       These criteria should be factored into the selected
       algorithmic approach of data analysis.

       Free chlorine and TOC were found to be the
       most responsive trigger parameters in chlorinated
       systems. Total chlorine was not an effective trigger
       parameter in chloraminated systems. TOC or TOC
       surrogate monitoring should be considered for both
       chlorinated and chloraminated systems.

       To capture and evaluate sensor responses in
       real-time,  SCADA equipment and algorithmic
       analysis are highly recommended. The SCADA
       database should be designed so that it can be easily
       interfaced with one or more automated algorithms
       for real-time analysis of data.


       Algorithms should be designed to learn or predict
       baseline values of parameters when monitoring
       location(s) with relatively unstable baseline
       conditions. Known routine system events (such as
       valve closures or tank fill and discharge cycles)
       need to be incorporated into the algorithmic
       evaluation(s) to reduce false positive events.

       For chlorinated systems, the algorithms and
       sensors should be designed to co-relate free
       chlorine and TOC data. Manufacturers should
       consider providing alarming or algorithm software
       with the online sensor equipment that is capable
       of identifying bad data and other instrument
       operational problems. This will prevent bad data
       from being analyzed by the algorithms, resulting in
       fewer false positive events.
              Read Configuration and Setup
         Wait For
     Next Time Step
                                      Read
                              New Water Quality Data
        Process
Event Detection Algorithms
                                     Report
                                 Probable Events
            When Finished

               Print Results Files and Exit
Figure 5. 1 CANARY Operation Schematic
                                                                                                        5-7

-------
the designated monitoring location. The system sub-
sequently compares the computed algorithmic values
to the "Agent Library" and "Plant Library" to classify
the deviation. The subscription-based "Agent Library"
(Hach  GuardianBlue™ Early Warning System) con-
tains "fingerprints" for a wide variety of threat con-
taminants, ranging from V-series nerve agent (VX) and
ricin to arsenic and herbicides. The site-specific oper-
ator-developed "Plant Library" contains "fingerprints"
of operational and naturally occurring  events specific
to each water distribution system. The plant library can
be used to detect and classify real-world events such as
water main breaks, switching water sources, and caus-
tic overfeeds.

EPA is also evaluating other  commercial event detec-
tion technologies such as the Frontier Technology, Inc.
(FTI) Event Detection Software (EDS) tool called H2O
Sentinel™ for contaminant detection. FTI has devel-
oped a proprietary software to monitor a set of standard
water quality parameters measured by  sensor stations
placed within a utility's water distribution system and
detect  anomalous events  that might be indicative of
possible contamination incidents.

EPA plans to continue evaluating other commercially
available event detection  algorithms as they become
available (Einfeld et al., 2007; Umberg et al., 2009).
Because true contamination events  are rare, the perfor-
mance of event detection systems is difficult to evalu-
ate. It is tempting to set the sensitivity of the algorithm
at a low  level  so that few alarms are generated, since
true contamination events are costly  to  investigate.
However, high sensitivity algorithms can result in the
generation of many alarms, which can result in "alarm
fatigue."  EPA continues to evaluate the alarm predic-
tion accuracy  of  these algorithms  by simulating con-
tamination events. EPA is considering the use of modi-
fied receiver operating characteristic (ROC) curves, a
data classification methodology that can plot the frac-
tion of true positive alarms versus the fraction of false
positive alarms generated by  the individual algorithm.
Modified ROC curves can help determine the efficiency
of these algorithmic approaches.

5.3               ysis
When  the  contaminant (or surrogate)  injection tests
were performed at the T&E  Facility, the  algorithmic
approaches of data evaluation were not available, with
the exception of the previously-mentioned ETV study.
The qualitative approach,  although robust, is not a vi-
able technique for real-time event detection. There
are normal/natural changes in water quality that could
mimic  some of the qualitative changes shown in the
testing. For example, at a monitoring station near a stor-
age tank  or reservoir,  the chlorine levels might change
dramatically depending upon the source of water and
tank operation. Similarly,  quantitative changes have a
drawback: in real life, the concentration of the injected
contaminants is unknown, and the resulting amount of
change does not necessarily correlate with prescribed
quantitative values. Therefore, an algorithmic approach
is the preferred approach for event detection. However,
a utility with existing SCADA systems (which general-
ly allow for high-low alarm set points) in the process of
deploying water quality monitoring stations can utilize
significant  threshold and other non-algorithmic meth-
odologies described in this chapter to alert the local op-
erator for further investigation. These types of prelimi-
nary data evaluations will assist in establishing and fine
tuning parameter-specific change thresholds and time-
windows for the algorithmic approaches. The evalua-
tions of existing algorithms have so far demonstrated
only limited success. Also, in an algorithmic approach,
there is a need to optimize the sensitivity of the algo-
rithm so that the false positives are minimized while
retaining the algorithm's ability to detect contamination
events. EPA's research for fine tuning individual algo-
rithms and the search for new approaches are ongoing.

-------
6.0  Operation,

Maintenance and

Calibration  of  Online

Instrumentation

The capital costs of the equipment tested at the EPA
T&E Facility are readily available from the manufac-
turers and are subject to change. However, the O&M
costs  are not as readily obtainable or well-defined.
The experience  gained at  the T&E Facility  indicates
that, for most online sensors, O&M costs will quickly
exceed capital costs. To keep the costs under control
for the  longer term, manufacturer recommendations
should be followed when performing O&M activities.
Also,  maintenance requirements  vary significantly,
depending upon the parameter and the device. The on-
line instrumentation evaluated at the T&E Facility was
calibrated and maintained as needed for testing efforts
described in the previous chapters. For year-round op-
eration,  maintaining a  tight maintenance schedule  is
necessary to obtain optimal instrument performance.
The scheduled maintenance activity, as well as cost of
consumable(s), depends on the individual sensor.

This chapter describes general O&M activities and as-
sociated costs for the instruments evaluated at the T&E
Facility. For this purpose,  EPA  tracked the labor and
consumable costs required  to operate and maintain the
tested sensor equipment. The O&M  costs presented
here do not include travel time to service the individual
monitoring  locations. The  travel costs will vary sig-
nificantly, depending upon where service personnel are
operating relative to the geographic distribution of the
monitoring sites. Furthermore, the total labor cost will
depend on operator skill and training, sensor complex-
ity, service contract terms, and the number of sensor
stations  deployed. Based on discussions with utilities
participating in the WSi, the optimal  service require-
ment for a sensor station deployed to the field was de-
termined to be one O&M site visit per month, with each
visit taking less than four hours.

6.1   Operation & Maintenance
      Labor Costs
The labor hours  expended for O&M vary significantly,
depending upon the type of instrument used (or parame-
ter that is monitored). As mentioned previously, the goal
of the WSi's pilot implementation in Cincinnati was to
achieve a service level of four labor hours per monitor-
ing station per month. It was well understood  that, dur-
ing the initial phase of installation and shake-down, the
labor costs were going to be much higher. Also, depend-
ing upon the size and complexity of an implementation,
the shake-down period can range from a month  to a
year. The labor hours per instrument also vary signifi-
cantly. Of the conventional instruments (TOC, chlorine,
conductivity, pH/ORP and Turbidity), TOC instruments
were by far the most labor intensive from an O&M per-
spective. They also required the highest technician skill
level to operate and maintain. However, the TOC instru-
ments are being continuously redesigned and the labor
level required is expected to be lower in the coming
years. Overall, the data collected from the WSi initiative
(for a 10-month period between January and September
2008) showed that approximately 1.5 person(s) working
on a full-time basis were needed to operate and maintain
17 monitoring stations. Between 60% and 80% of this
labor cost was associated with the O&M for the TOC in-
struments. Also, as indicated previously, the labor hour
estimate does not include travel time to the monitoring
stations, which could vary significantly depending upon
the geographic configuration of the monitoring network.
Experience  indicates that labor estimates per site can
vary  widely because there might be some sites with ad-
verse water quality (or other site-specific anomaly) that
can cause numerous O&M problems. The best current
estimate for labor hours per site is 1.5 days per month on
average, which is three times the goal of the WSi.

Additional labor costs will be incurred for data anal-
ysis and event detection efforts. Because it is too la-
bor intensive to have the operator monitor the data on
a continuous basis, an automated event detection or
alarm system is necessary. It is expected that the exist-
ing SCADA operator at a water utility can be assigned
the additional duty of checking the operational status of
the data collection system during  every shift to ensure
that data is being collected and  analyzed by automated
tools. To ensure everything is operating normally, the
utility should plan to assign a person to skim through
the historical data and monitor the alarm software on a
daily basis. This task is expected to add approximately
30 to 60 minutes per day or per shift depending upon
how the task is assigned. The reviewer should be a staff
member who  already  understands the operations and
the monitored water quality data  and can make deci-
sions on O&M and alarm response needs as warranted
by the quick review of the data and alarm history.

6.2 Equipment-Specific
     Maintenance and Consumable
     Costs
As mentioned previously, the maintenance requirements
vary  significantly, depending upon the parameter and
the device. Based on the experience gained at the T&E
Facility, EPA identified TOC, free chlorine, conductiv-
ity, pH/ORP, turbidity and temperature as the key online
                                                                                                   6-1

-------
      monitoring parameters. These parameters are listed in
      the order of importance from an event detection per-
      spective. The following sections (Section 6.3 through
      6.7) describe these parameters (except temperature) and
      the instrument-specific O&M activity for the equipment
      evaluated at the T&E Facility. The temperature probes
      are inexpensive and very robust with almost no main-
      tenance requirements and are, therefore, not discussed
      in this report. The prices for consumables for the instru-
      ments mentioned in this chapter are based  on pricing
      information obtained during the years 2007 and 2008.

      6.3 Total Organic Carbon
            Instrumentation
      TOC instrumentation  responded  to a wide range  of
      contaminants tested at the T&E Facility. Especially in
      chloraminated waters, TOC  instrumentation was more
      responsive  to  contaminants  than the  instrumentation
      used for measuring total chlorine/monochloramines. A
      skilled technician was required to reliably operate and
      maintain the TOC instrumentation on a continuous ba-
      sis, making it the most expensive and time-consuming
      equipment to operate. It is highly recommended that the
      person responsible for TOC instrumentation O&M is-
      sues obtain manufacturer-provided training, or the wa-
      ter utility should consider purchasing a manufacturer
      maintenance contract. EPA also evaluated an optical in-
      strument (Carbo::lyser™) that estimates TOC levels by
      measuring UV-Vis spectra between 200 and 750 nano-
      meter (nm) wavelengths.

      6.3.1  Hach astroTOC™ UV Process Total
           Organic Carbon Analyzer
      This instrument requires monthly replenishment of re-
      agents, quarterly scheduled maintenance, and approxi-
      mately $4,000 per year for consumables. The following
      routine maintenance activities need to be performed:

         • Replace reagent/acid (monthly), reagent/oxidizer
           (quarterly)

         • Change the pump tubing (quarterly)

         • Replace the semi-permeable sparger membrane
           (quarterly)

         • Replace the hydrophobic filter (quarterly)

         • Calibrate the infra-red detector with carbon
           dioxide (quarterly)

         • Calibrate the wet-side of the unit with potassium
           hydrogen phthalate (KHP) standard (monthly/
           quarterly)

         • Replace the UV lamps (annually)

         • Replace the consumables in the carrier gas
           generator (annually)
   •  Clean infra-red detector window (annually)

A significant amount of time was required for trouble-
shooting and repairing the Hach astroTOC™ UV Pro-
cess Total Organic Carbon Analyzer, primarily due to
instrument malfunctions.  The types of malfunctions
observed were mostly related to plugged or interrupted
flow (liquid/gaseous):

   •  There was a build-up of silica crystals in the
     resample block, which was rectified by Hach by
     redesigning the resample block

   •  The sparger orifice would get plugged (semi-
     permeable membrane) and result in shut-down
     of the instrument

Monthly calibration is  essential to electronically adjust
for the instrument drift prior to the quarterly mainte-
nance/calibration event.

6.3.2 Sievers® 900 On-Line Total Organic
     Carbon Analyzer
This  instrument requires quarterly scheduled mainte-
nance. Depending upon the initial water quality, main-
tenance costs  can vary between $2,000 and $4,000 per
year  for consumables. The following routine mainte-
nance activities need to be performed:

   •  Replace reagent/acid/oxidizer (quarterly/semi-
     annual)

   •  Change the pump tubing (semi-annual)

   •  Replace the resin bed (semi-annual)

   •  Replace UV lamp (semi-annual)

   •  Replace ICR degasser, chemical trap, and pump
     rebuilt kit (annual)

   •  Replace in-line particulate filter (annual)

   •  Replace oxidizer syringe (annual)

   •  Replace restrictor tubing (annual)

Comparatively, a significant amount of time was re-
quired  for troubleshooting and repairing the Siev-
ers® 900  On-Line TOC Analyzer, primarily due to
instrument malfunctions.  Malfunctions  observed
were mostly related to plugged or interrupted liquid
flow  (e.g., restrictor  tube blockage).  In areas with
high-carbonate water,  reagents need to be replenished
quarterly.

6.3.3 Spectro::lyzer™/Carbo::lyzer™
As indicated  earlier, this  sensor is an optical instru-
ment that  estimates TOC levels by measuring UV-Vis
spectra  between  200 and 750 nm wavelengths. Only
6-2

-------
TOC that has some absorption property in the specified
wavelengths is detected. The instrument has minimal
O&M requirements. At first, EPA procured a unit made
of high-grade aluminum. However, the body of this in-
strument corroded in chlorinated Cincinnati tap water.
A replacement stainless steel  unit was provided by the
manufacturer free of charge. EPA recommends that
only a stainless steel unit be purchased for locations
with water quality that might be aggressive to alumi-
num. The unit in global calibration mode does not re-
quire any calibration. If local TOC values are known,
the  unit can  be calibrated using locally available high
and low values. For obtaining a zero value in the local
calibration mode, a highly-purified High-Performance
(or  High-Pressure) Liquid Chromatography (HPLC)
grade reagent or a high grade distilled water should be
used. Deionized and nanopure waters are not suitable
for this purpose.

6.4 Chlorine Instrumentation
Free chlorine instruments responded to the majority of
contaminants tested at the T&E Facility on chlorinated
Cincinnati tap water. Both free and total chlorine instru-
ments use either a reagent-based (colorimetric) method
or a reagent-free method (amperometric/polarographic/
galvanic - electrode/membrane-based).  The reagent-
free method has varied pH dependence, depending upon
the  use of buffering agents.  The reagent-based method
consumables include buffering and indicator solutions.
The reagent-free method consumables include electro-
lyte solutions (and membranes, if applicable).

6.4.1 Hach  CL-17 Free and Total Chlorine
     Analyzer
Hach CL-17 Free and Total  Chlorine Analyzers both
utilize  the colorimetric  method.  Buffer and reagent
solutions, which adjust the pH and react with chlorine
to produce a color, are added to the sample. The color
depth is proportional to the amount of chlorine. Buffer
and reagent solutions last about a month. The estimated
cost of the buffer and reagent solution consumables is
between $750 and $1,000 a year. In addition, the tubing,
stir-bar,  and plastic-tube connectors need to be replaced
every six months. The colorimeter needs to be cleaned
every six months. This unit requires no calibration.

6.4.2 Wallace &Tiernan®  Depolox® 3 plus
The Wallace & Tiernan® Depolox® 3 plus employs a
reagent  free  (potentiostatic 3-electrode-amperometric)
method for measurement. There is an option for either a
bare electrode or membrane-type measurement. The op-
tion selection is based on water hardness, conductivity,
and variation in pH. The membrane-type instrument is
recommended for a higher pH range (6 to 10 pH usable
range) and low conductivity (10 - 2,500 |aS/cm) waters.
In high pH waters, buffering (e.g., using CO2) is needed.
Free chlorine  consists of chlorine molecules (C12),
hypochlorous  acid (HOC1) and  hypochlorite  ions
(OC1~). The presence of each component is mostly de-
pendent upon pH with some influence of temperature.
The HOC1 component is the most effective compo-
nent of free chlorine for disinfection. The electrodes
typically measure the HOC1 component, and report it
as free chlorine. In membrane-type instruments, the
membrane is designed to allow only the HOC1 acid
through the membrane, which is measured and re-
ported as free chlorine. At a pH of > 8.5, the majority
of the HOC1 is converted to OC1", thus,  interfering
with the accurate measurement of free chlorine. From
a maintenance perspective, the following consider-
ations are noted:

   •   Replace the electrolyte in the electrode
      reservoir (semi-annually)

   •   In the bare electrode model, replace the
      grit (that self-cleans the electrode) (semi-
      annually)

   •   In the membrane type model, clean the
      electrode tip with abrasive paper and replace the
      membrane (every three years)

The cost of consumables ranges between $350 and
$1,000 a year. The calibration of the bare electrode
is performed by turning the sample flow off and set-
ting to zero  (after waiting several minutes). The wa-
ter is  then turned back on,  and after waiting for the
reading to stabilize, a grab sample is collected, a N,
JV-diethyl-p-phenylenediamine measurement is per-
formed and  the span set to match the test result. For
the membrane-type unit, there is zero calibration.

6.4.3 YSI 6920DW
The YSI 6920DW is a multi-parameter (free chlorine,
turbidity, temperature, conductivity, pH, and ORP) in-
strument that uses a reagent-free (amperometric mem-
brane) method for free chlorine measurement (similar
to the instrument described  in Section 6.4.2). From a
maintenance perspective:

   •   Replace the electrolyte and membrane
      (quarterly)

The annual  cost  of consumables is  approximately
$2,100 a year.

6.4.4 Analytical Technology, Inc., Model
      A15/62  Free Chlorine Monitor
The Analytical Technology, Inc.,  Model A15/62 free
chlorine monitor instrument uses a reagent-free (po-
larographic membrane) method for free chlorine mea-
surement (similar  to the instrument described in  Sec-
tion 6.4.2).
                                                                                                      6-3

-------
           Replace the electrolyte and membrane
           (quarterly)
     These probes need to be cleaned and calibrated
     (quarterly)
      The annual cost of consumables is approximately $100
      a year. The consumable costs for this instrument are
      less expensive when compared to the Wallace & Tier-
      nan® Depolox® 3 plus instrument, because there is no
      pH electrode to be replaced.

      6.4.5 Rosemount Analytical Model FCL
      The Rosemount Analytical Model FCL uses a reagent-
      free  (membrane type amperometric)  method for free
      chlorine measurement (similar to the instrument de-
      scribed in Section 6.4.2). In addition, the Rosemount
      Analytical Model FCL-01  (with manual pH  adjust-
      ment) chlorine sensor can measure  both  HOC1 and
      OC1" forms of chlorine. The sensor responds differently
      to each form and compensates for both sample pH and
      temperature. From a maintenance perspective:

         •  Replace the electrolyte and membrane
           (quarterly)

         •  In addition, the membrane needs to be cleaned
           (monthly)

      During the contaminant injections, this membrane ap-
      peared to be more prone to fouling than the other mem-
      brane-type instruments. The  annual cost  of consum-
      ables is approximately $400 a year.

      6.5 Conductivity Instrumentation
      The presence of dissolved mineral substances such as
      chloride, nitrate, sulfate, and phosphate anions (ions
      that carry a negative charge) or  sodium, magnesium,
      calcium, iron, and aluminum cations (ions that carry
      a positive charge) dissolved in water can be measured
      as conductivity (or temperature-compensated specific
      conductance). Conductivity is  basically  a measure-
      ment of the sample water's ability to carry an elec-
      trical current. There are several factors that affect the
      conductivity of water including: concentration of ions;
      mobility of ions; oxidation state (valence); and tem-
      perature. The testing at the T&E Facility indicated that
      a high volume of contaminant injection was needed
      for the  conductivity value  to change significantly.
      Conductivity is typically measured directly by either
      measuring the voltage drop or the current flow through
      a sample.

      The performance of the conductivity probes was similar
      for the probes tested at the T&E Facility. The probes
      evaluated at the facility included the following:  Hach/
      GLI Model C53 Conductivity Analyzer, YSI 6600, YSI
      6920DW,  Hydrolab® DS5 and Troll® 9000. From a
      maintenance perspective:
   •  The probe requires replacement as needed
     (generally lasts at least a year)

The consumables include calibration solutions that cost
approximately $200 per year.

6.6 pH/ Oxygen Reduction
     Potential Instrumentation
The majority of the manufacturers combine the pH/
ORP measurement with a reference electrode. The pH
value is a measure of the activity of hydrogen ions (H+)
in the water sample. Therefore, it a measure of the de-
gree of acidity or alkalinity of the water sample. ORP
is a measure of the tendency of the water sample to
oxidize or reduce another chemical substance. Typi-
cally, ORP is measured using an inert metal electrode
(platinum), which will donate electrons to an oxidizing
agent or accept  electrons from a reducing  agent. The
ORP electrode continues to accept or donate electrons
until it  develops a potential that is equal to the ORP of
the solution.  ORP is sometimes utilized for estimating
the concentration of chlorine in water.  However, ORP
measurement is affected by many factors  and might
not be a good surrogate for chlorine. The testing at the
T&E Facility indicated that the ORP probes took longer
than pH probes to return to baseline and grab samples
for ORP are  not reliable. The performance of the pH/
ORP probes was similar for all of the probes tested at
the T&E  Facility. The probes evaluated at the facility
included the following: Hach/GLI Model P53 pH/ORP
Analyzer, YSI 6600, YSI 6920DW, Hydrolab® DS5
and Troll® 9000. From a maintenance perspective:

   •  These probes need to be cleaned and calibrated
     (quarterly)

   •  These probes require replacement as needed
     (generally lasts at least a year)

   •  The Hach/GLI Model P53 pH/ORP Analyzer
     requires annual replacement of the salt-bridge
     and electrolyte solution, which extends the
     instrument life to about 3 years.

The consumables include pH calibration solutions and
probes  that cost approximately $1,000 per year.

6.7 Turbidity
Turbidity is a measure of cloudiness or haziness of the
water sample caused by the suspended particles. Tur-
bidity is typically determined by shining a light beam of
wavelengths between 830 and 890 nm into  the sample
solution and then measuring the light  (at 90-degrees)
scattered  by  the suspended particles. Online nephelo-
6-4

-------
metric turbidity measuring devices were evaluated. The
turbidity sensors evaluated include the Hydrolab® DS5
and Hach FilterTrak™ 660  sc Laser Nephelometer.
From a maintenance perspective:

   •   The Hach FilterTrak™ 660 sc Laser
      Nephelometer requires quarterly cleaning/
      calibration and annual replacement of the light
      source

   •   Hydrolab® DS5 turbidimeters require semi-
      annual replacement of the wiper

The Hydrolab® DS5 optical turbidimeter probes last
about three years  and cost approximately $1,500 per
unit. The annualized  cost for consumables, including
the optical probe and calibration solutions, is approxi-
mately $1,000.

The annualized consumables  cost for the Hach Filter-
Trak™ 660 sc Laser Nephelometer, including the bulb
replacement and calibration solution, is approximately
$300.

6.8  Dissolved  Oxygen
Dissolved oxygen probes  are  used  to  measure the
amount of gaseous oxygen (O2) dissolved in the water
sample. Dissolved oxygen  levels did not significantly
change after  the  initial contaminant injections and
therefore was removed from further evaluation.

6.9  Other Conventional
      Water Quality Parameter/
      Instrumentation
During the initial phases of testing at the T&E Facility,
various other  ion-selective electrodes (ISEs) were also
evaluated for  their efficacy and usefulness in detecting
changes in water quality. The other ISEs evaluated for
their ability to measure  the following parameters: am-
monia, nitrate, and chloride.  Although some of these
parameters  changed in  the presence  of the contami-
nants, free chlorine interfered with ISE calibration and
prevented the repeatability of the measurement. In addi-
tion, some of the ISEs burned out too quickly and were
expensive to replace. Therefore, these parameters were
excluded from further testing.

6.10  Online Optical
      Instrumentation
The  following optical  instruments (listed alphabeti-
cally) were evaluated mostly  for their ability to detect
biological agents  and growth media  at the T&E Fa-
cility: FlowCAM®, Hach  2200 PCX Particle Coun-
ter, Hach FilterTrak™  660  sc Laser Nephelometer,
BioSentry®, Spectro::lyzer™ and Carbo::lyzer™ and
ZAPS MP-1. In addition, because of their ability to
detect contaminants in addition to  biological organ-
isms, the Spectro::lyzer™ and Carbo::lyzer™ and the
ZAPS MP-1 were tested using various chemical injec-
tions. Generally, the biological organism tests were per-
formed by injecting 100, 600, 1,000, and 25,000 cells/
mL (both in chlorinated  and dechlorinated waters).
Based on previous testing, EPA had discovered that the
online chlorine  and TOC  monitors generated reliable
responses at injections of  100,000 cells/mL and some
responses at 25,000 cells/mL, but the response faded
below this level.

Generally, these optical instruments were evaluated as
they became available for the research and, therefore,
evaluations utilizing all contaminants with each instru-
ment were not performed. The following is a brief sum-
mary of  their capability and equipment performance
observation.

6.10.1   FlowCAM®
FlowCAM® is  an online particle imaging and flow-
cytometry system that takes high-resolution digital
images of particles and cells in the water sample. The
images are analyzed by a proprietary software program
based on Microsoft®  Office Excel® that captures and
analyzes  the parameters of the particles such as count,
size, length, shape, and equivalent spherical diameter.
In addition, the instrument captures the intensity, trans-
parency,  color,  bio-volume, compactness,  roughness,
and elongations of the particles.

At the T&E Facility, Ankistodesmiis  (20 to  100 mi-
crometers (|om)), Selenestmm (10 |jm), and Saccha-
romyces  yeast (1.5 jam) were injected. The unit was
able to recognize all of the particles. In order to cap-
ture individual images, the flow cytometry setup sig-
nificantly reduces the flow  volume to the unit. This
restrictive flow path  causes significant delays  in the
measurement from the injection time. At concentra-
tions below 1,000 cells/mL, the instrument is unable
to differentiate baseline noise from injected contami-
nants. Sub-micron particles such as E. coli (0.8 to 0.9
|om), bacteriophage MS2 (0.02 to 0.03  jam) and B. glo-
bigii (0.5 to 0.9 jam -  spores are smaller than the cells)
were not identifiable  with the existing camera optical
resolution and flow cell. However, at sufficiently high
concentrations,  the instrument is able to show an in-
creased count.

6.10.2 Hach FilterTrak™ 660 sc Laser
       Nephelometer and Hach 2200 PCX
       Particle Counter
The Hach FilterTrak™ 660 laser nephelometer uses
a collimated light source with high beam density
and a distinct wavelength to detect baseline turbid-
                                                                                                      6-5

-------
ity change as low as 0.5 milli-nephelometric turbidity
units (mNTU) (0.0005 NTU). The instrument is opti-
mized to detect particles in the 0.1 to 0.5 jam range.
The Each 2200 PCX Particle Counter is a laser-di-
ode-based particle counter designed for drinking wa-
ter applications. The instrument is optimized to detect
particles in the range of 2-750 |om.

The instruments were challenged with the following
biological injections performed at the T&E Facility:
E. coli, MS2 and B. globigii. The Each Filter/Trak™
660 sc Laser Nephelometer was able to detect only E.
coli and B. globigii at injection levels of 25,000 cells/
mL. The Each 2200 PCX Particle Counter was un-
able to detect any of the biological injections at these
levels. The particle counter is designed for detecting
larger biological particles such as  Cryptosporidium
spp. and Giardia  lamblia.


The BioSentry® system is a laser-based, continu-
ous, online, real-time monitoring device for detecting
microorganisms in water. The unit utilizes laser-pro-
duced, multi-angle light scattering (MALS) tech-
nology to generate unique bio-optical signatures for
classification using BioSentry®'s pathogen detection
library. BioSentry® can be  set up to detect microor-
ganisms and identify suspected pathogens.

The biological injections performed at the T&E Fa-
cility  included the following:   3-micron beads (sur-
rogates for Cryptosporidium spp.), B. globigii and E.
coli. In the current design (as tested), the unit has to
be programmed to identify a specific contaminant,
whereas all others (even when detected) are identified
as unknowns. Prior to any injection(s), the unit should
be programmed to recognize the injected particle, i.e.,
the unique bio-optical MALS signature should be de-
veloped and put to use (using the local water and pure
form of the injected particle). The  unit tested at the
T&E Facility was able to reliably recognize injection
events and identify injected particles between 1,000
and 10,000 cells/mL level.  At lower cell concentra-
tions, the detection was not consistent across the test
runs.
These units are based on UV-Vis spectroscopic absorp-
tion measurements. Contaminants that respond to the
UV-Vis absorption spectra can be detected by this in-
strument.  The Spectro::lyser™ was programmed  to
measure the optical equivalents of turbidity, nitrate,
TOC, and dissolved organic carbon (DOC). In addition,
the Spectro::lyzer™ was connected to the  con::stat™
process control terminal, which ran the proprietary soft-
ware to compute four pre-set alarm parameters based
on computations of spectral channels that were consid-
ered to be important by the manufacturer.

At the T&E Facility, both chemical and biological in-
jections were performed using the Single Pass DSS.
The chemical injections included: humic acid, sodium
fluoroacetate, aldicarb, dicamba, and gasoline. The in-
strument was able to detect the water quality changes
for all of the contaminants, except sodium fluoroace-
tate. The biological injections included the following:
sucrose, E.  coli and B. globigii. The instrument was
able to detect changes at injection levels of approxi-
mately 25,000 cells/mL.

Results at the T&E Facility indicated that this device
was capable of serving as a good surrogate for traditional
TOC measurement. The operation and maintenance re-
quirements for this device were minimal when compared
to  the traditional UV-persulfate  method-based TOC
measuring devices. Also, the size of this  device is much
smaller than the traditional TOC measuring devices. As
mentioned previously in Chapters 2, 3,  and 5, TOC is
a critical water quality trigger parameter. EPA recom-
mends the use of this type of device especially at loca-
tions where  the traditional TOC devices are difficult to
deploy, either due to size or due to ongoing operational
and maintenance costs. Subsequently, the Carbo::lyzer™
was successfully  deployed at two locations under the
WSi pilot study in Cincinnati alongside the traditional
TOC measuring devices. Furthermore, this instrument
and associated software are capable of analyzing the full
spectrum of UV-Vis absorbance, which if fully exploited,
can yield additional information about the water quality
changes that are not captured by the other devices evalu-
ated by EPA at the T&E Facility.


The ZAPS MP-1 is an online water quality  monitor-
ing device that can be programmed to  measure up to
100 slices of optical wavelengths (using optical filters)
between 200 and 800 nanometers. The optical data that
can be captured includes absorption, fluorescence, and
total reflection  bands. The ZAPS MP-1 configuration
at the T&E Facility was set up to measure the follow-
ing parameters:  dark counts, pinhole, nitrate, ultraviolet
254 nanometer wavelength (UV254) absorbance, bac-
terial fluorescence, humic fluorescence, total fluores-
cence, rhodamine, and transmission. The individual ex-
citation and response wavelengths were pre-set by the
manufacturer to these parameters.

At the T&E Facility, nitrate, formazin, Saccharomyces
cerevisiae (yeast) and E. coli were injected to evaluate
responses on this instrument. The instrument respond-
ed well to nitrate injections at 0.14 mg/L.  The instru-
ment responded well to the injection of formazin (tur-

-------
bidity standard) at test levels of 13.3 NTU and 26.6
NTU; a linear response was observed in UV254, total
fluorescence, bacterial fluorescence, and nitrate chan-
nels. However, for Saccharomyces cerevisiae (yeast)
and E. coli injections, the instrument showed no re-
sponse below 100,000 cells/mL. Therefore, no further
testing was performed with the biological agents.

6.11   Best Practices and Lessons
      Learned
As presented in Sections 6.1 through 6.9, many con-
ventional instruments are relatively easy to operate
and  maintain. However, the TOC  instruments  are
more complex and require a relatively higher level of
technical skill. A formal training class provided by
the manufacturer for each type of instrument is highly
recommended. Just by following the instruction man-
uals,  experienced instrumentation technicians and
field engineers who have several years of experience
are usually capable of installing, setting up, calibrat-
ing, and operating new or unfamiliar instrumentation.
Less experienced staff will require assistance from the
instrument manufacturer to avoid some of the pitfalls
that can cause serious damage to an instrument and
result in improper/inefficient operation.  For example,
the Sievers® 900 On-Line TOC Analyzer requires the
de-ionized loop reservoir to  be filled prior to power
up; otherwise, air is trapped in the measurement mod-
ules, which will require factory service.  Furthermore,
factory default settings for an instrument might not be
suitable for the location; the operator should be aware
of the exact settings for each location prior to service.
As another example, when installing a probe to an
analyzer such as  Wallace & Tiernan®  Depolox® 3
plus, it is necessary to complete a formal setup in the
instrument analyzer to recognize the probe correctly.
It is always advisable to attend training  workshops
offered by most instrument manufacturers. Adequate
staff training for setup and maintenance activities is
essential for optimal operations. In addition, for more
complex instruments (such as the optical instrumen-
tation), it is essential for the maintenance technician
to receive formal manufacturer training.

The operation of  a single or a  few instruments is a
relatively straightforward process; however, the oper-
ation of a multi-station network or multiple networks
can be  a  logistical nightmare. Loss of  data, false
alarms, and other malfunctions  can lead to improper
analysis of data by algorithms and inappropriate ac-
tions. The operator should consult with the instrument
manufacturer(s) and develop a thorough understand-
ing of the instrument outputs during power outages
and other malfunctions such as the loss of reagents. It
is important to develop a monitoring plan for  sched-

       Water Utilities and
       Sensor Manufacturers
       Over the life of the equipment operation, in general,
       the O&M cost will exceed capital cost for most
       online sensing equipment. At a minimum, the
       water utilities should set aside a budget for annual
       labor and consumable costs, based on the detailed
       information presented in this chapter.
       The evaluations at the EPA T&E Facility and
       during the WSi pilot study revealed that almost
       60% to 80% of the O&M labor cost associated with
       the water quality monitoring sensors was related to
       TOC instruments.

       Instrument technicians should be appropriately
       factory-trained for optimal operations.

       O&M activities should be appropriately scheduled.
       Consumables should be purchased in a timely
       manner so that they do not expire before they can
       be used up.

       Ideally, sensor manufacturers need to develop
       reagent-free sensors that result in lower labor and
       consumable costs.

       In the absence of better TOC instrumentation, TOC
       surrogate monitoring (such as the S::CAN - see
       section 6.10.4) should be seriously considered for
       both chlorinated and chloraminated systems as the
       maintenance requirement for this type of device is
       minimal.
uled maintenance, order expendable  supplies in a
timely manner, maintain calibration standards, and
schedule sufficient manpower to cover network op-
erations. Following a good monitoring plan will en-
sure the collection of high-quality data that meets the
monitoring requirements.

In general, most instrument problems are related  to
flow (sample and/or reagent issues). The sample flow
problem could be related to restrictions in the flow
manifold or restriction of flow through the instrument.
Reagent flow blockage can result in diminished or un-
stable readings. If the reagents run out completely, the
readings typically drop to zero and are easy to spot.
Some instruments are factory-set to hold the last good
reading; if the numbers do not change over a signifi-
cant period, it is an indication of instrument failure.
For membrane-type probes, the reading usually drifts
downwards as the membrane is clogged or nearing its
useful life. For electrodes,  failure  is generally indi-
cated by problems during calibration where either the
slope  or the gain or the cell constant is outside of the
manufacturer-recommended tolerance range.
i

                                                                                                      6-7

-------
      Degassing  or bubbles can cause improper  readings.
      Degassing  is generally an issue where  the incoming
      sample water is colder than the environmental housing
      of the instrument.

      When purchasing consumables, one needs to be sure to
      use them prior to their expiration dates. For some ISEs,
      the shelf life begins from the date of manufacture and
      not the date of installation. When planning to purchase
      spare parts and consumables, the shelf life and project-
      ed use by date should be taken into consideration. Water
      utilities are encouraged to work with manufacturers to
      negotiate purchase price of equipment based on volume
      of purchase and any negotiated long-term service con-
      tracts.
8-8

-------
The references included in this bibliography contain
additional detailed information for readers who wish to
pursue, in greater detail, the specific topics discussed
in this guide. Many of these references (especially the
EPA references) are freely available on the Internet.
The references are listed alphabetically, based on the
last name of the first author(s). In cases where there
are two or more works by the same author (e.g., EPA),
the entries are listed chronologically.

Allgeier, S.C., J. Pulz, and R. Murray, 2006. Con-
ceptual design of a contamination warning system.
Proceedings of the AWWA Water Security Congress,
Washington, DC.

AwwaRF, 2002. Online monitoring for drinking water
utilities. Editor, Erika Hargesheimer, Awwa Research
Foundation and CRS PRO AQUA, American Water
Works Association, Denver, CO.

Dawsey, W.J., B.S. Minsker, and V.L. VanBlaricum,
2006. Bayesian belief networks to integrate monitoring
evidence of water distribution system contamination.
Journal of Water Resources and Planning, ASCE,
132:4:234.

Department of Homeland Security, 2006. National
Infrastructure Protection Plan.

Edberg, S. C., E.W Rice, R. J. Karlin, and M. J. Allen,
2000. Escherichia coli: The Best Biological Drinking
Water Indicator for Public Health Protection. Journal
of Applied Microbiology, 88 (S6).

Einfeld, W., S.  McKenna, and M.P. Wilson, 2008.
A Simulation Tool to Assess Contaminant Warning
System Sensor Performance Characteristics. AwwaRF,
Denver, CO.

EPA, 2003.  Contamination threat management guide -
Module 2. EPA/817/D-03/002, Washington, DC.

EPA, 2004a. The Water Security Research and Techni-
cal Support Action Plan. EPA/600/R-04/063, Washing-
ton, DC.

EPA, 2004b. Public Drinking Water Systems Pro-
grams:  http://www.epa.gov/safewater/pws/pwss.html.

EPA, 2004c. Response Protocol Toolbox Overview.
EPA/817/D-03/007, Washington, DC.
EPA, 2004d. Response Protocol Toolbox: Water Util-
ity Planning Guide - Module 1. EPA/817/D-03/001,
Washington, DC.

EPA, 2004e. Response Protocol Toolbox: Con-
tamination Threat Management Guide - Module 2.
EPA/817/D-03/002, Washington, DC.

EPA, 2004f. Response Protocol Toolbox: Site
Characterization and Sampling Guide - Module 3.
EPA/817/D-03/003, Washington, DC.

EPA, 2004g. Response Protocol Toolbox: Analytical
Guide - Module 4. EPA/817/D-03/004, Washington,
DC.

EPA, 2004h. Response Protocol Toolbox:
Public Health Response Guide - Module 5.
EPA/817/D-03/005, Washington, DC.

EPA, 2004L Response Protocol Toolbox: Remediation
and Recovery Guide - Module 6. EPA/817/D-03/006,
Washington, DC.

EPA, 2004j. Response Protocol Toolbox: Planning
for and Responding to Drinking Water Contamina-
tion Threats and Incidents Response Guidelines.
EPA/817/D-04/001, Washington, DC.

EPA, 2004k. Homeland Security Strategy. Washington,
DC.

EPA, 2005a. Evaluation of Water Quality Sensors as
Devices to Warn of Intentional Contamination in Water
Distribution Systems. EPA/600/R-05/105, Washington,
DC. Available only through WaterlSAC.

EPA, 2005b. WaterSentinel Online Water Quality
Monitoring as an Indicator of Drinking  Water Con-
tamination. EPA/817/D-05/002, Washington, DC.

EPA, 2005c. WaterSentinel System Architecture.
EPA/817/D-05/003, Washington, DC.

EPA, 2005d. Technologies and Techniques for Early
Warning Systems to Monitor and Evaluate Drink-
ing Water Quality: A State-of-the-Art Review.
EPA/600/R-05/156, Washington, DC.

EPA, 2007a. Water Quality Sensor Response to Con-
tamination in a Single Pass Water Distribution System
Simulator. EPA/600/R-07/001, Washington, DC.
Available only through WaterlSAC.

-------
EPA, 2007b. Water Security Initiative: Interim Guid-
ance on Planning for Contamination Warning System
Deployment. EPA/817/R-07/002, Washington, DC.

EPA, 2008a. Water Security Initiative: Interim Guid-
ance on Developing an Operational Strategy for
Contamination Warning Systems. EPA/817/R-08/002,
Washington, DC.

EPA, 2008b. Water Security Initiative: Interim Guid-
ance on Developing Consequence Management Plans
for Drinking Water Utilities. EPA/817/R-08/001,
Washington, DC.

EPA, 2008c. Water Security Initiative: Cincinnati Pilot
Post-Implementation System Status - Covering the
Pilot Period: December 2005 through December 2007.
EPA/817/R-08/004, Washington, DC.

Hall, J.S., A.D. Zafnro, R.B. Marx, PC. Kefauver,
E.R. Krishnan, R.C. Haught, and J.G. Herrmann,
2007. On-line  water quality parameters as indicators of
distribution system contamination. Journal of Ameri-
can Water Works Association, 99:1:66.

Homeland Security Presidential Directive-7/HSPD-7,
2003. Critical  Infrastructure Identification, Prioritiza-
tion, and Protection: http://www.dhs.gov/xabout/laws/
gc_1214597989952.shtm.

Homeland Security Presidential Directive-9/HSPD-9,
2004. Defense of United States Agriculture and Food:
http://www.dhs.gov/xabout/laws/gc_1217449547663.
shtm.

ISLI, 1999. Early warning monitoring to detect haz-
ardous events  in water supplies. ISLI Press, Washing-
ton, DC.

Kessler, A., A. Ostfeld, and G. Sinai, 1998. Detect-
ing accidental  contaminations in municipal water
networks. Journal of Water Resources Planning and
Management-ASCE, 124:4:192.

Kirmeyer, G., M. Friedman, and K. Mattel, 2002. Guid-
ance manual for monitoring distribution system water
quality. Awwa  Research Foundation, Denver, CO.

Kroll, D., and  K. King, 2007. To switch or not to
switch. Potential security considerations in the free
chlorine versus monochloramine debate for drinking
water disinfection. Proceedings of the ASM Bio-
defense and Emerging Diseases Research Meeting,
Washington, DC.
Lytle, J., and E.W. Rice, 2002. A Systematic Compari-
son of the Electrokinetic Properties of Environmen-
tally Important Microorganisms in Water. Colloids and
Surfaces B: Biointerfaces, 24, 91-101.

McKenna, S.A., K.A Klise, and M.R Wilson, 2006.
Testing water quality change detection algorithms.
Proceedings of the 8th Annual Water Distribution Sys-
tems Analysis (WDSA) Symposium, Cincinnati, OH.

O'Halloran, R., R. Jarrett, G. Robinson, and P. Toscas,
2009. Data Processing and Analysis for Online Distri-
bution System Monitoring, AwwaRF, Denver, CO.

Rice, E. W., N. J. Adcock, M. Sivaganesan, and L. J.
Rose, 2005. Inactivation of spores of Bacillus anthra-
cis Sterne, Bacillus cereus, and Bacillus thuringiensis
subsp. israelensis by chlorination. Applied and Envi-
ronmental Microbiology. 9: 5587-5589.

Roberson, J.A. and K.M. Morley, 2005, Guidance
document, contamination warning systems for water:
An approach for providing actionable information to
decision-makers, AWWA.

Sivaganesan, M., N.J. Adcock, and E.W. Rice, 2006.
Inactivation of Bacillus globigii by chlorination: A
Bayesian model. Journal of Water Supply Research
Technology -Aqua, 55.1: 33-43.

Szabo, J.G., J.S. Hall, and G.C. Meiners, 2006. Water
quality sensor responses to injected contaminants in a
chloraminated pipe loop. Proceedings of the AWWA
Water Security Congress, Washington, DC.

Szabo, J.G., J.S. Hall, and G.C. Meiners, 2008a. Sen-
sor responses to contamination in chloraminated drink-
ing water. Journal of American Water Works Associa-
tion, April 2008, Denver, CO.

Szabo, J.G., J.S. Hall, and G.C. Meiners, 2008b.
Detection of biological suspensions using online
detectors in a model drinking water distribution system
simulator. AWWA Water Security Congress, Cincin-
nati, OH.

Umberg, K., N. Mix, B. Pickard, and C. Dangel,
2009. Event Detection System Evaluation, Selection,
and Optimization. AWWA Water Security Congress,
Washington, DC.

-------

-------
v>EPA
      United States
      Environmental Protection
      Agency


      Office of Research and Development
      National Homeland Security
        Research Center
      Cincinnati,  OH 45268

      Official Business
      Penalty for Private Use
      $300


      EPA/600/R-09/076
      October 2009
      www.epa.gov
PRESORTED STANDARD
 POSTAGE & FEES PAID
        EPA
   PERMIT No. G-35
                                           Recycled/Recyclable
                                           Printed with vegetable-based ink on
                                           paper that contains a minimum of
                                           50% post-consumer fiber content
                                           processed chlorine free

-------