&EPA
United
Environmental Protection
Agency
WaterSentinel System Architecture
Draft, Version 1.0
December 12, 2005
-------
-------
U.S. Environmental Protection Agency
Water Security Division
Ariel Rios Building, Mail Code 4601M
1200 Pennsylvania Avenue, N. W.
Washington, DC 20460
EPA817-D-05-003
-------
-------
WS System Architecture
Disclaimer
The Water Security Division, of the Office of Ground Water and Drinking Water, has reviewed and
approved this draft document for publication. This document does not impose legally binding
requirements on any party. The word "should" as used in this Guide is intended solely to recommend or
suggest and does not connote a requirement. Neither the United States Government nor any of its
employees, contractors, or their employees make any warranty, expressed or implied, or assumes any
legal liability or responsibility for any third party's use of or the results of such use of any information,
apparatus, product, or process discussed in this report, or represents that its use by such party would not
infringe on privately owned rights. Mention of trade names or commercial products does not constitute
endorsement or recommendation for use.
Questions concerning this document or its application should be addressed to:
Steve Allgeier
U.S. EPA Water Security Division
Threat Analysis, Prevention, and Preparedness Branch
26 West Martin Luther King Drive
Cincinnati, OH 45268-1320
(513)569-7131
Allgeier. Steve @epa. gov
DRAFT-121205
-------
WS System Architecture
Acknowledgements
The Water Security Division would like to recognize the following organizations and individuals for their
support in the design of the WaterSentinel system architecture and development of supporting
documentation:
Steve Allgeier
Elizabeth Hedrick
Colm Kenny
Latisha Mapp
Office of Water -
Water Security Division
Cindy Simbanin
Irwin Silverstein
Ashley Smith
David Travers
Office of Research and Development -
National Homeland Security Research Center
Dominic Boccelli
Kathy Clayton
Kim Fox
John Hall
Rob Janke
Eric Koglin
Alan Lindquist
Matthew Magnuson
Regan Murray
Cindy Sonich-Mullin
JeffSzabo
Contractor Support
Zaileen Alibhai, CSC
John Chandler, CSC
Kevin Cornell, CSC
Tom Fieldsend, CSC
Ruth Grunerud, CSC
Adrian Hanley, CSC
Neal Jannelle, CSC
Ken Miller, CSC
Misty Pope, CSC
Jon Prawdzik, CSC
Jessica Pulz, CSC
Erin Salo, CSC
Doron Shalvi, CSC
Sarah Tater, CSC
Lynn Walters, CSC
Sandra Davis, CH2MHILL
Bill Desing, CH2MHILL
Todd Elliott, CH2MHILL
Yakir Hasit, CH2MHILL
Alan Ispass, CH2MHILL
Gary Jacobson, CH2MHILL
Bill Phillips, CH2MHILL
Tim Ellis, Emergint
Craig Stanley, Emergint
DRAFT-121205
-------
WS System Architecture
Executive Summary
Through the assessment of vulnerabilities to drinking water systems, water security experts have
identified the distribution system as one of the most vulnerable components in a drinking water utility.
This finding was further supported through additional studies and analyses. For example, a Government
Accounting Office (GAO) survey of a panel of nationally recognized water security experts identified
distribution systems as among the most vulnerable physical components of a drinking water utility due to
the large number of access points, ease of access, and the inability to detect contamination in a timely
manner due to the absence of integrated and reliable monitoring and surveillance systems (GAO-04-29).
Strengthening of key relationships between water utilities and other federal agencies in terms of
preparedness, detection, and response activities was another important issue highlighted by the GAO
study, and was addressed to some extent through an inter-agency effort, directed by the Homeland
Security Council (HSC) of the White House, to assess the threat of drinking water contamination. This
effort resulted in a report prepared by the U.S. Environmental Protection Agency (EPA), for the HSC that
identified a number of contaminants, which if introduced into a drinking water distribution system could
produce consequences upwards of 10,000 fatalities (USEPA, 2004c). This same report concluded that in
the absence of a contamination warning system (CWS), many of these contamination incidents would go
undetected until weeks following the attack when the first cases of disease would begin to appear in the
population, at which time it may be difficult or impossible to find even a trace of contamination in the
drinking water distribution system.
Contamination of the drinking water distribution system - whether it is accidental or intentional - can
have devastating consequences for public health, critical infrastructure, the economy, and the
environment. Drinking water distribution systems may be accidentally contaminated through cross-
connections with non-potable water, permeation of contaminated water through pipes in areas of the
distribution system subject to low pressures, or chemical reactions or microbial growth within the
distribution system pipes. Such unintentional events that result in degradation to distributed water quality
may occur with some regularity. Furthermore, intentional contamination, or even the threat of
contamination can have significant impacts. Drinking water utilities occasionally receive threats or
indications of possible contamination. These contamination threat warnings can be a direct threat or an
unusual observation or discovery that indicates the potential for contamination and initiates actions to
investigate and potentially respond. However, these threat warnings are not standardized and are difficult
to corroborate in the absence of an integrated monitoring and surveillance system and close coordination
with response partners including, but not limited to public health, emergency responders, and law
enforcement.
In recognition of the contamination threat and the importance of early detection, the Administration
issued HSPD 9 - Defense of United States Agriculture and Food. This directive was for EPA and other
Agencies, using existing authorities, to build upon and expand current monitoring programs, to:
'develop robust, comprehensive, and fully coordinated surveillance and monitoring systems . . .
for ... water quality that provide early detection and awareness of disease, pest, or poisonous
agents," and
'develop nationwide laboratory networks for ... water quality that integrate existing Federal and
State laboratory resources, are interconnected, and utilize standardized diagnostic protocols and
procedures.'
By its authority under section 300i-3 of the Safe Drinking Water Act (42 USC section 1434) and to
address the monitoring and surveillance requirements of HSPD 9, EPA intends for WS to build on
DRAFT-121205 in
-------
WS System Architecture
existing Agency and utility efforts to enhance the ability to detect and respond to contamination threats
and incidents through the use of a CWS.
What is a Contamination Warning System?
The key to an effective response to a water contamination threat is minimizing the time between
indication of a contamination incident and implementation of effective response actions to minimize
further consequences. Implementation of a robust CWS can achieve this objective by providing an earlier
indication of a potential contamination incident than would be possible in the absence of a CWS; thus, the
core component of the WS program is a CWS. A CWS is a proactive approach to managing threat
warnings that uses advanced monitoring technologies/strategies and enhanced surveillance activities to
collect, integrate, analyze, and communicate information to provide a timely warning of potential water
contamination incidents and initiate response actions to minimize public health and economic impacts.
Components of the WS-CWS that should be implemented and evaluated through a pilot demonstration
project include the following:
Online water quality monitoring. Online monitors for water quality parameters, such as
chlorine residual, total organic carbon, pH, conductivity, turbidity, etc., should be used to
establish expected levels for these parameters (a 'baseline'). Anomalous changes from the
established water quality baseline should be used as an indicator of potential contamination in the
WS-CWS.
Sampling and analysis. Water samples should be collected at a predetermined frequency and
analyzed to establish a baseline through the use of an 'unknowns' protocol. This 'unknowns'
protocol would target specific, priority contaminants, but may also detect some non-target
analytes if the analytical techniques used in the routine monitoring program are sufficiently
robust and if the analysts are trained and encouraged to investigate tentatively identified
contaminants. In addition, water samples should be collected in response to triggers from water
quality monitors or other information streams to identify the potentially unknown contaminants in
the sample.
Enhanced security monitoring. Security breaches, witness accounts, and notifications by
perpetrators, news media, or law enforcement should be monitored and documented through
enhanced security practices. This component of the WS-CWS has the potential to detect a
tampering event in progress, potentially preventing the introduction of a harmful contaminant into
the drinking water.
Consumer complaint surveillance. Consumer complaints regarding unusual taste, odor, or
appearance of the water are often reported to water utilities, which document the reports and
conventionally use them to identify and address water quality problems. Occasionally, water
quality complaints are reported to local agencies other than the water utility, such as 911 call
centers, the health department or a city's general information number. Using an appropriate
methodology that compiles and tracks the information provided by consumers, the utility can
consider these complaints along with data from other CWS components to identify unusual trends
that may be indicative of a contamination incident.
Public health surveillance. Syndromic surveillance conducted by the public health sector,
including information such as over-the-counter sales of medication, as well as reports from
emergency medical service logs, 911 call centers, and poison control hotlines may serve as a
warning of a potential drinking water contamination incident. Information from these sources
should be integrated by developing a reliable link between the public health sector and drinking
water utilities.
A CWS is not merely a collection of monitors and equipment placed throughout a water system to alert of
intrusion or contamination. Fundamentally, it is an exercise in information acquisition and management.
DRAFT-121205 iv
-------
WS System Architecture
Different information streams should be captured, managed, analyzed, and interpreted in time to
recognize potential contamination incidents in time to respond effectively. As discussed in Section 2.0 of
this document and further evaluated in WaterSentinel Contamination Incident Timeline Analysis, each of
these information streams can independently provide some value in terms of more timely initial detection
(USEPA, 2005b). However, when these information streams are integrated and used to evaluate a
possible contamination incident, the credibility of the incident can be established more quickly and
reliably than if any of the information streams were used independently. While the primary purpose of a
CWS is to detect contamination incidents, accidental or intentional, implementation of a CWS is expected
to result in dual-use benefits for drinking water utilities that should help to ensure sustainability of the
system.
Although many utilities are currently implementing some monitoring and surveillance activities, these
activities would not be likely to detect a wide range of possible contamination events. For example, while
many utilities currently track consumer complaint calls, WS proposes to develop a robust spatially based
system that, when integrated with data from public health surveillance, online water quality monitoring,
and enhanced security surveillance, should provide specific, reliable, and timely information for decision
makers to establish credibility and respond in an effective manner. Beyond each individual component of
the WS-CWS, WS should facilitate coordination and planning between the utility and local public health
agency to develop a robust consequence management plan that involves the appropriate local officials,
law enforcement, emergency responders, etc., to ensure that appropriate actions should occur in response
to various triggers/alarms. An advanced and integrated laboratory infrastructure to support baseline
monitoring as well as analysis of samples collected in response to triggers from the CWS monitoring and
surveillance activities is critical to timely response. In the absence of a reliable and sustainable CWS, a
utility's ability to respond to contamination threats and incidents in a timely and appropriate manner is
limited.
What is WaterSentinel?
WS is a program developed by EPA in close partnership with drinking water utilities and other key
stakeholders in response to HSPD 9. The program involves designing, deploying, and evaluating a model
CWS for drinking water security as part of a demonstration project, or pilot. The overall goal of WS is to
design and demonstrate an effective system for timely detection and appropriate response to drinking
water contamination threats and incidents that would have broad application to the nation's drinking
water utilities. The systematic approach to design of the WS-CWS should reduce the time between
indication of potential contamination incidents, evaluation of the possible threat, and implementation of
consequence management and response actions. More specifically, EPA's objectives for the WS program
are to design a CWS that:
Provides timely detection of contamination;
Has broad coverage of priority contaminant classes;
Is the most protective of public health using currently available and well-characterized
technologies;
Is sustainable through benefits to the water utility independent of enhanced water security (dual-
use benefits);
Is implementable, cost-effective, and reliable; and
Is ultimately applicable to utilities nationally.
To meet these objectives, EPA intends to test a number of broad hypotheses that are critical to
understanding the efficacy of a CWS. Through the initial pilot, a research project at a single utility, EPA
plans to test the following hypotheses to determine whether the components of a CWS, singularly,
collectively, or in some combination, can serve as an effective warning system:
DRAFT-121205
-------
WS System Architecture
Water quality parameters (e.g., pH, chlorine residual, total organic carbon, etc.), in conjunction
with an event detection system can provide early indication of contamination incidents.
Consumer complaints can provide warning of contamination with chemicals that have a
discernable odor or taste in sufficient time to respond in a manner that reduces consequences.
Public health surveillance for indicators of disease in the population can provide early indication
of drinking water contamination, particularly those contaminants that would not be otherwise
detected through utility monitoring and surveillance activities.
Event detection software (i.e., computer-based algorithms) applied to water quality parameters,
consumer complaints, and public health surveillance both singularly and correlatively, can detect
statistical anomalies indicative of possible contamination while minimizing the number of false
alarms that a utility would otherwise have to deal with.
Certain vulnerabilities to contamination can be effectively reduced through the focused
deployment of security monitoring systems, and such a system can help to resolve false alarms.
Integration of these different monitoring and surveillance techniques increases the coverage of
contaminants, reduces the time to initial detection, and improves the overall reliability of the
system
Site characterization and triggered sampling (e.g., grab samples collected in response to a water
quality anomaly, unusual consumer complaints, or an anomaly detected through public health
surveillance) for specific high priority contaminants can provide corroboration of a contamination
incident.
What are the Key Considerations for the WaterSentinel Contamination Warning System
Design?
In the WS-CWS, the design basis can be described in terms of the particular problem that a system is
designed to solve or the function the system is designed to perform. It provides a framework for system
development and a benchmark against which to evaluate the performance of different design options. For
detection systems, the design basis can be described in terms of the incident, or suite of incidents, that a
satisfactory system should detect. The design basis for a drinking water CWS is defined as a series of
contamination scenarios against which specific design options should be evaluated. A contamination
scenario is specified by the location of contaminant introduction, the type of contaminant, and the
amount, concentration, and rate of introduction. In addition to the contamination scenarios that define the
design basis, a CWS design is subject to other requirements and constraints, such as the ability to detect
an event in sufficient time to implement effective response actions. For example, a design option that can
consistently detect a contamination scenario should not be acceptable if detection occurs significantly
after a response is needed.
Developing a design basis for a contamination warning system is challenging because of the large number
of potential contamination scenarios with varying degrees of consequence. The design basis may be
substantially narrowed by initially focusing on those contamination scenarios with the highest
consequences, particularly those with the potential for a high number of fatalities. However, it is not
appropriate to arbitrarily establish a numeric threshold that defines a high-consequence scenario (e.g.,
10,000 fatalities) because this can vary from utility to utility depending on the total population in the
service area, the population density profiles, configuration of the distribution system, and other factors. A
more rational approach is to evaluate and rank the consequences for a large number of potential
contamination scenarios, and use those scenarios with the most significant consequences in the design
basis (i.e., the relative ranking of scenarios is more useful than an absolute threshold). A system
constructed around such a design basis should also detect many lower-consequence scenarios, and, while
some scenarios should go undetected, the number of high-consequence scenarios that are not detected
DRAFT-121205 vi
-------
WS System Architecture
should be minimized. The consequences associated with a particular contamination scenario are largely a
function of the specific contaminant and the location of contaminant introduction.
Another important consideration in the design of a CWS is the timeline associated with a contamination
incident, specifically:
The time during which consequences (exposures, illnesses, fatalities, pipe contamination, etc.) are
experienced in the population,
The time of initial detection, and
The time of response actions.
Analysis of different contamination incident timelines can establish whether or not a given design should
meet an important design requirement - initial detection in a timeframe that allows for the
implementation of response actions that result in a significant reduction in consequences.
The manner in which the integration of multiple monitoring and surveillance strategies, as discussed
above, satisfy the design basis is described below and in further detail in Section 2.0.
Contaminant Coverage
Analysis of contaminant properties and detection techniques clearly demonstrates that no single approach
should provide timely detection for all contaminants of concern; however, the integrated approach
implemented under WS has the potential to provide timely detection of a very high percentage of priority
contaminants. The WS contaminant selection process identified contaminants for consideration in the
WS pilot, which were ultimately grouped into 12 detection classes based on the manner in which they
might be detected through the five WS-CWS monitoring and surveillance components. Table ES-1
provides a summary of the WS-CWS detection classes (USEPA, 2005c).
Table ES-1. WaterSentinel Contamination Warning System Detection Classes
Contaminant
Detection Class
1
2
3
4
5
6
7
8
9
10
11
12
Description
Petroleum products
Pesticides (chlorine reactive)
Inorganic compounds
Metals
Pesticides (chlorine resistant)
Chemical warfare agents
Radionuclides
Bacterial toxins
Plant toxins
Pathogens causing diseases with unique symptoms
Pathogens causing diseases with common symptoms
Persistent chlorinated organic compounds
Spatial Coverage
The monitoring components of the WS-CWS (water quality sensors, sampling and analysis, and enhanced
security monitoring) have intrinsic limitations to the spatial coverage that each can achieve. On the other
hand, surveillance components of the WS-CWS (consumer complaint and public health surveillance) rely
on consumer observations and behavior, and thus provide dense spatial coverage throughout a distribution
system. Thus, integration of both monitoring and surveillance systems in the WS-CWS is necessary to
achieve a high degree of spatial coverage.
DRAFT-121205
vi i
-------
WS System Architecture
Timeliness of Initial Detection
As demonstrated through analysis of contamination incident timelines that considered approximately
100,000 contamination scenarios involving 10 different contaminants in one real drinking water
distribution system, different contaminants are first detected by different monitoring and surveillance
techniques (USEPA, 2005b). By integrating multiple data sources, the time of initial detection is reduced
across all contaminants, and even those that act very rapidly within the exposed population may be
detected in time to implement an effective response. For 6 of the 10 contaminant classes, a strong link
between the public health community and the local water utility is critical to early detection and effective
response to contamination incidents and for 5 of the 10 contaminant classes, public health surveillance
would most likely be used to help establish the credibility of an incident.
Reliability
The multiple monitoring and surveillance techniques used in the WS-CWS extend beyond integration of
multiple water quality data streams to other independent information streams including water quality data,
consumer calls, public health surveillance, security alarms, results from site characterization and sample
analysis. The WS-CWS pilot should provide an unprecedented opportunity to develop the information
necessary to better characterize and quantify the value of integrating information from numerous
monitoring and surveillance activities to improve our ability to reliably detect contamination incidents,
i.e., to minimize the frequency of false alarms. The overall rate of false positive and false negatives for
the integrated data streams should be substantially lower than the rates for any one detection strategy.
These considerations for reliability of the WS-CWS may also be used to quantify 'dual-use' benefits of a
CWS, which are related to system sustainability, another key consideration in the design of the WS-CWS.
Sustainability
The integration of multiple monitoring and surveillance strategies already in use at the utility and public
health department should improve acceptance of the system, and thus long-term sustainability. The CWS
is being designed as a dual-use application that should benefit the utility in day-to-day operations while
also providing the capability to detect intentional or accidental contamination incidents.
Table ES-2 describes the manner in which each of the WS-CWS components addresses each of these
aspects of the WS design basis. Note that some of these benefits cannot be quantified until the WS pilot
is deployed and EPA gains substantial experience; thus the importance of implementing and evaluating
the WS-CWS though a pilot program.
DRAFT-121205 viii
-------
Table ES-2. WS-CWS Components and their Contributions to the Approach for WaterSentinel
WS-CWS
Component
Online
Water
Quality
Monitoring
Sampling
and
Analysis
Enhanced
Security
Monitoring
Consumer
Complaint
Surveillance
Public
Health
Surveillance
Capability
Can indicate the
presence of a
contaminant that
significantly affects one
or more monitored
parameters that serve
as indicators of
contamination.
Can positively identify
the presence of any
contaminant in the suite
of target analytes and
above the MDL.
Can detect an intrusion
that may have provided
the opportunity for
introduction of any
contaminant.
Can indicate the
presence of a
contaminant that
significantly affects one
or more aesthetic
qualities of water.
Can detect the presence
of a symptom or illness
in a population which
may be the result of the
presence of a disease
causing agent. May be
able to identify the
contaminant through
clinical diagnosis/
testing.
Contaminant
Coverage
High detection
potential for
classes 2, 3, 5, 8,
10, and 11;
Moderate detection
potential for
classes 1,4, 7, 9,
and 12.
potential for
classes 1, 2, 3, 4, 7,
and 12; Moderate
detection potential
for classes 5, 6, 8,
9 10 11
Covers all
contaminant
classes.
High detection
potential for
classes 1 and 2;
Moderate detection
potential for
classes 3 and 4.
class6s 2 through
potential varies
Spatial Coverage
Function of
location, number,
and density of
monitoring
stations
Function of
location, number,
and density of
sampling
stations, as well
as sample type
(composite vs.
grab).
Limited to those
elements of
infrastructure for
which physical
security can be
monitored.
Entire service
area for
contaminants
with detectable
organoleptic
characteristics.
Comprehensive
coverage of a
particular city or
county, which
may include all,
or a large portion
of, the service
area.
Timeliness
Function of hydraulic
travel time from the point
of contaminant
introduction to the
sensor, and the
concentration of the
contaminant.
Function of sampling &
analysis frequency and
the total time required to
process the sample and
analyze the results.
Function of the type of
security monitoring
system and the time
required to evaluate a
security breach.
Function of the time
from exposures to
consumer reporting,
complaint
categorization,
assessment and
investigation.
Function of the time
from the initial
exposures, the onset of
symptoms, and the point
at which public health
officials recognize the
incident as a potential
water-borne illness.
Reliability
Rate of false positive /
negative results in this
application is largely
unknown at this time. May
be addressed through event
detection systems and
consequence management.
Function of the reliability of
sampling and analysis
methods (high for
established techniques).
Baseline needed for reliable
interpretation of results.
Can be a reliable means of
especially when these
breaches may involve
contamination, such as in
storage tanks and clear
wells. May be addressed
through consequence
management.
A potentially reliable
indicator for contaminants
with detectable
characteristics if a robust
complaint reporting and
tracking system is in place.
May be a reliable means of
identifying the incidence of
illness in a population, but
communication between
drinking water and public
health officials is not always
quick enough for
appropriate response,
intervention and remedial
actions to take place.
Sustainability
Provides utility with a
better understanding of
water quality variability
throughout distribution
system and provides an
opportunity to optimize
distribution system
operation.
Provides utility with an
opportunity to exercise
sampling and laboratory
protocols and may;
provide information about
previously unknown
contaminants that occur
in the system.
Provides utility with
increased physical
infrastructure protection
and awareness. Reduces
the occurrence of
nuisance tampering.
Provides utility an
opportunity to manage
consumer information
more effectively and can
serve as a tool for
enhanced consumer
confidence.
Provides an opportunity
for collaboration between
utility and local health
depart ment(s).
DRAFT-121205
-------
WS System Architecture
Relating back to the overall objective of WS, Figure ES-1 demonstrates the potential for the WS-CWS to
reduce the impacts of contamination incidents. For 10 contaminant classes, the figure shows the impacts
from a high consequence scenario without the WS-CWS (in blue) and with the WS-CWS (in green). In
each scenario, WS-CWS has the potential to reduce the public health impacts from 6-100%. For classes
8, 10, and 11, represented by one biotoxin and two biological agents, WS-CWS has the potential to
prevent all fatalities assuming availability of sufficient medical resources.
Benefit of CWS: Potential for Reduction in
Fatalities
Without WS-CWS
D Wth WS-CWS
23456789 10 11
Contaminant Class
Figure ES-1. Potential Benefit of CWS in Reducing the Number of Fatalities in 10 Contamination
Scenarios
The initial method of detection and the time period in which response is effective may differ among the
various contaminants. The timeline analysis establishes the timeframes in which utility intervention to
reduce further exposure would be effective as well as the timeframe in which public health intervention
would be effective. It also highlights the importance of the link between public health services and water
utilities, and the importance of rapid utility and public health response strategies.
This design basis forms a framework for system development and a benchmark against which to evaluate
the performance of different design options. This document provides more detail regarding the basis for
including each of the WS monitoring and surveillance components, and describes the general framework
for design and implementation of each component. Initial considerations for evaluation of both the
technical design and the overall program are also discussed.
What is the Approach for Implementation of the Initial WaterSentinel Pilot?
The initial WS pilot should serve as a demonstration project for the conceptual design described in detail
in Section 2.0 of this document. Using this document as an initial framework, EPA anticipates working
with the pilot utility and partner organizations to develop a work plan for implementation of the WS-
CWS. EPA plans to provide support to the WS pilot utility and aims to work closely with the utility to
DRAFT-121205
-------
WS System Architecture
design a program that meets its current and projected needs. In addition, EPA intends to work with the
utility to develop the necessary laboratory capabilities required to support implementation of the WS
program. Once implemented, an evaluation program should assess the effectiveness, costs, and benefits
of the pilot and recommend improvements to the WS-CWS, as well as the sustainability and multiple
benefits provided by implementation of the WS-CWS. The phases of design and implementation of the
initial WS-CWS pilot include the following:
Initial Planning. Includes participation in initial meetings between EPA, the pilot utility, and
local partners (as appropriate) to discuss the objectives, technical approach, and general
implementation strategy.
Assessment. Review of the pilot utility's current practices, procedures, and capabilities for the
technical components of the WS-CWS. This may include an initial request for relevant
information as well as an on-site assessment.
Gap Analysis and Component-Specific Work Plans. Based on the assessments conducted in
the previous phase, EPA technical staff plan to work with the pilot utility to conduct a gap
analysis to determine the appropriate enhancements and modifications to support implementation
of the WS-CWS. From this gap analysis, EPA aims to work with the pilot utility to develop
component-specific work plans for implementation.
Enhancements and Installation. In accordance with the component-specific work plans, EPA
aims to work with the pilot utility to implement enhancements and install the necessary
equipment and systems for implementation.
Baseline Development. Establish a baseline for all components of the WS-CWS, including
online water quality monitoring, sampling and analysis, enhanced security monitoring, consumer
complaint surveillance, and public health surveillance.
Full Deployment. The WS-CWS should be fully operational, information streams should be
integrated, and consequence management plans should be fully implemented.
Evaluation. Evaluation of the WS-CWS should occur at established increments throughout the
pilot.
Refinement. Based on the evaluation(s), WS-CWS components may require refinements or
additional enhancements to ensure proper operation of the system relative to the objectives
established in the component-specific work plans.
Maintenance. Following implementation of the WS-CWS and initial enhancements to the
system, the pilot utility should maintain the CWS. With the advancement of technology and
research, additional cycles of evaluation and refinement should be considered. However, the
frequency at which these evaluations occur can likely be decreased over time.
Until the WS-CWS concept has been demonstrated through evaluation and refinement, the system at the
pilot utility should function as a research project. As the WS-CWS is refined and enhanced, the system
should become protective of public health through the transition from full deployment of the WS-CWS
through maintenance by the pilot utility.
DRAFT-121205 xi
-------
WS System Architecture
Table of Contents
Executive Summary iii
Section 1.0: Introduction 1
1.1 Problem Statement 1
1.1.1 Overview of Contamination Warning Systems 2
1.1.2 WaterSentinel Objectives 3
1.2 Overview of WaterSentinel 4
1.2.1 Monitoring and Surveillance 5
1.2.2 Event Detection 5
1.2.3 Consequence Management 6
1.2.3.1 Credibility Determination 6
1.2.3.2 Response 6
1.2.3.3 Remediation and Recovery 7
1.3 Approach to Implementation 7
1.4 Document Organization 9
Section 2.0 WaterSentinel Contamination Warning System Design Basis 10
2.1 Contaminant Detection Classes 11
2.2 High Impact Contamination Scenarios 13
2.3 Contamination Incident Timeline 15
2.4 Additional Design Considerations 20
2.4.1 System Reliability 20
2.4.2 System Sustainability 22
2.5 Preliminary Cost Analysis for the WaterSentinel Contamination Warning System 23
2.5.1 Online Water Quality Monitoring 24
2.5.2 Sampling and Analysis 25
2.5.3 Enhanced Security Monitoring 25
2.5.4 Consumer Complaint Surveillance 25
2.5.5 Public Health Surveillance 25
2.6 Summary of WaterSentinel CWS Design Basis 26
Section 3.0 Online Water Quality Monitoring 29
3.1 Water Quality Sensors 29
3.1.1 Selection of Water Quality Parameters 29
3.1.2 Selection of Water Quality Sensors 31
3.1.3 Sensor Station Design 35
3.2 Sensor Network Design 36
3.3 Data Management, Analysis, and Interpretation 42
3.3.7 Data Management 42
3.3.2 Analysis and Interpretation 44
3.4 Framework for Evaluation 44
3.4.7 Network Model Confidence 44
3.4.2 CWS Sensor Placement Tool 45
3.4.3 Water Quality Event Detection Systems 45
3.4.4 Sensor Stations 46
DRAFT-121205 xii
-------
WS System Architecture
Section 4.0: Sampling and Analysis 47
4.1 Sampling and Analysis 48
4.1.1 Sampling 48
4.1.2 Analysis 51
4.1.3 Laboratory Support Network 55
4.2 Sampling Circuit Design 57
4.2.1 Baseline Sampling 57
4.2.2 Triggered Sampling 59
4.3 Data Management, Analysis, and Interpretation 60
4.3.1 Data Management 60
4.3.2 Data Analysis and Interpretation 61
4.4 Framework for Evaluation 62
4.4.1 Sampling Locations 63
4.4.2 Assessments 63
4.4.3 Evaluation of Field and Laboratory QC Data 64
4.4.4 Proficiency Testing (PT) Program 65
4.4.5 Tabletop Exercises and Water Contamination Drills 65
Section 5.0: Enhanced Security Monitoring 66
5.1 Integration of Enhanced Security Monitoring into the WS-CWS 66
5.1.1 Other Security System Design Considerations 70
5.1.2 Other Detection Methods 71
5.1.3 Operational Response Actions 72
5.2 Data Management, Analysis, and Interpretation 73
5.2.1 Data Management 73
5.2.2 Data Analysis and Interpretation 74
5.3 Framework for Evaluation 74
Section 6.0: Consumer Complaint Surveillance 76
6.1 Attributes of an Effective Consumer Complaint Surveillance Program (CCSP) 77
6.2 Data Management, Analysis, and Interpretation 80
6.3 Framework for Evaluation 82
6.3.1 Evaluation Tools 82
6.3.2 Data Sets 83
6.3.3 Consumer Confidence Surveys 83
Section 7.0: Public Health Surveillance 84
7.1 Overview of Existing Syndromic Surveillance Systems 85
7.2. Public Health Surveillance and WaterSentinel 87
7.3 Data Management, Analysis, and Interpretation 89
7.4 Framework for Evaluation 90
7.4.7 System Reliability 90
7.4.2 System Sustainability 91
7.4.3 System Evaluation 92
Section 8.0: Information Integration, Management, and Communication 94
8.1 Data Collection and Transmission 94
8.2 Integration and Analysis of Information 96
DRAFT-121205 xiii
-------
WS System Architecture
8.3 Communication of Information 99
Section 9.0: Approach to Evaluation 101
9.1 Technical Evaluation 101
9.1.1 Monitoring and Surveillance 102
9.1.2 Event Detection 102
9.1.3 Credibility Determination 105
9.1.4 Response 105
9.1.5 Remediation and Recovery 106
9.2 Programmatic Evaluation of WaterSentinel 107
9.2.1 EPA Perspective 107
9.2.2 Utility Perspectives 108
9.2.3 Stakeholder Perspectives 109
Section 10.0: References and Resources 111
Appendix A: Acronym List 116
Appendix B: Glossary 119
Appendix C: Overview of Related Projects 127
List of Tables
Table ES-1. WaterSentinel Contamination Warning System Detection Classes vii
Table ES-2. WS-CWS Components and their Contributions to the Approach for WaterSentinel ix
Table 2-1. Ranking Criteria for the Various CWS Components 12
Table 2-2. Detection Potential for Each of the 12 Contaminant Detection Classes 13
Table 2-3. Statistical Summary of Consequence and Detection Times over all Possible Scenarios for
Each Contaminant Detection Class 16
Table 2-4. Timeline Summary for the 90th Percentile Scenarios for each Contaminant Detection
Class 18
Table 2-5. Probability of Sensors Responding 21
Table 2-6. Definition of Cost Categories for WS-CWS Components 23
Table 2-7. Preliminary WS-CWS Component Cost Estimate 24
Table 2-8. WS-CWS Components and their Contributions to the Approach for WaterSentinel 28
Table 4-1. Preservation and Holding Time Table for Radiological, Chemical, and Pathogens 50
Table 7-1. Summary of National Syndromic Surveillance Systems and Tools 86
Table 8-1. Summary of WS Information Streams for Managing Data 94
List of Figures
Figure ES-1. Potential Benefit of CWS in Reducing the Number of Fatalities in 10 Contamination
Scenarios x
Figure 1-1. Overview of WS Concept of Operations 5
Figure 1-2. Overview of the WS Program 7
Figure 2-1. Cumulative Distribution Function of Consequences for all Possible Insertion Points in a
Distribution System using one Specific Contaminant 14
Figure 2-2. Relative Timing of Consequences and Detection 19
Figure 2-3. Comparison of the Posterior Probability of an Event Using Data from Individual and
Multiple Water Quality Sensors 21
Figure 3-1. Contaminant Class Detection by Type of Water Quality Sensor 30
Figure 3-2. Schematic of an Example Water Quality Sensor Station 36
Figure 3-3: Example Water Distribution System with Potential Sensor Locations 40
DRAFT-121205 xiv
-------
WS System Architecture
Figure 3-4. Sensor Network Design Trade-Off Curve 42
Figure 4-1. Overview of WS Sampling Process 49
Figure 4-2. Overview of WS Unknowns Protocol 52
Figure 5-1. Elements of a Physical Security System 67
Figure 5-2. Adversary Task Time versus Security System Time Requirements for an Ineffective
System 69
Figure 5-3. Adversary Task Time versus Security System Time Requirements for an Effective
System 70
Figure 6-1. Components of a Consumer Complaint Surveillance Program 77
Figure 6-2. Consumer Complaint Surveillance Process 80
Figure 6-3. Utility Consumer Complaint Data Flow 81
Figure 7-1. Water Utility - Public Health IT Sophistication Matrix 88
Figure 7-2. WS integration with Public Health 89
Figure 8-1. Data Integration for Event Detection and Credibility Determination 97
DRAFT-121205 xv
-------
WS System Architecture
Section 1.0: Introduction
WaterSentinel (WS) is a program developed by the U.S. Environmental Protection Agency (EPA) in close
partnership with drinking water utilities and other key stakeholders in response to Homeland Security
Presidential Directive 9 (HSPD 9). The program involves designing, deploying, and evaluating a model
contamination warning system (CWS) for drinking water security. A CWS is a system that collects
information from a variety of sources, including monitoring and surveillance programs, in order to detect
contamination events in drinking water early enough to reduce public health or economic consequences.
This document presents an overview of the WS program and the CWS concept, including the design and
development of the system architecture for the WS program, a framework for making design decisions,
and considerations for evaluation of the WS program and the WS-CWS. In addition to guiding the design
of the WS-CWS at the initial pilot utility, the approach described in this document can be used to inform
other stakeholders and utilities interested in implementing a CWS. This is a living document that should
evolve as experience is gained through the implementation of the WS-CWS, and this first version of the
document represents the basis for the design of the first pilot.
1.1 Problem Statement
Through the assessment of vulnerabilities to drinking water systems, water security experts have
identified the distribution system as one of the most vulnerable nodes in a water utility. This finding was
further supported through additional studies and analyses. For example, a Government Accounting
Office (GAO) survey of a panel of nationally recognized water security experts who identified
distribution systems as among the most vulnerable physical components of a drinking water utility due to
the large number of access points, ease of access, and the inability to detect contamination in a timely
manner due to the absence of integrated and reliable monitoring and surveillance systems (GAO-04-29).
Strengthening of key relationships between water utilities and other agencies in terms of preparedness,
detection, and response activities was another important issue highlighted by the GAO study. A report
prepared by EPA for the Homeland Security Council of the White House, and under the direction of other
federal agencies, identified a number of contaminants, which if introduced into a drinking water
distribution system could produce consequences upwards of 10,000 fatalities (USEPA, 2004c). This
same report concluded that in the absence of a CWS, many of these contamination incidents would go
undetected until weeks following the attack when the first cases of disease begin to appear in the
population, at which time it may be difficult or impossible to find even a trace of contamination in the
distribution system.
Contamination of the drinking water distribution system - whether it is accidental or intentional - can
have devastating consequences for public health, critical infrastructure, the economy, and the
environment. Drinking water distribution systems may be accidentally contaminated through cross-
connections with non-potable water, permeation through pipes in low pressure areas, or chemical
reactions or microbial growth within the pipes. Such unintentional events that result in degradation to
distributed water quality may occur with some regularity. Intentional contamination, or even the threat of
contamination can have significant impacts. Drinking water utilities occasionally receive threat or
indications of possible contamination. These contamination threat warnings can be a direct threat or an
unusual observation or discovery that indicates the potential for contamination and initiates actions to
investigate and potentially respond. However, these threat warnings are not standardized and are difficult
to corroborate in the absence of an integrated monitoring and surveillance system and close coordination
with response partners including, but not limited to public health, emergency responders, and law
enforcement.
DRAFT-121205
-------
WS System Architecture
In recognition of the contamination threat and the importance of early detection, the Administration
issued HSPD 9 - Defense of United States Agriculture and Food. This directive was for EPA and other
Agencies, using existing authorities, to build upon and expand current monitoring programs, to:
'develop robust, comprehensive, and fully coordinated surveillance and monitoring systems . . .
for . . . water quality that provide early detection and awareness of disease, pest, or poisonous
agents," and
'develop nationwide laboratory networks for ... water quality that integrate existing Federal and
State laboratory resources, are interconnected, and utilize standardized diagnostic protocols and
procedures.'
By its authority under section 300i-3 of the Safe Drinking Water Act (42 USC section 1434) and to
address the monitoring and surveillance requirements of HSPD 9, EPA intends for WS to build on
existing Agency and utility efforts to enhance the ability to detect and respond to contamination threats
and incidents through the use of a CWS. In June 2004, EPA and the American Water Works Association
(AWWA) established a group of utilities to participate in the Threat Ensemble Vulnerability Assessment
research program, and effort focused on the use of hydraulic modeling tools to better characterize the
consequences of contamination and use that information in the design of a CWS. Further refinement of
the CWS model occurred in close partnership with drinking water utilities and other key stakeholders.
The AWWA established an informal 'Utility Users' Group' in October 2003 as a forum for utilities to
exchange experiences in dealing with water security and contamination issues and discuss approaches for
detection and response. At a subsequent meeting in January 2005, this group focused on issues related
specifically to the design and implementation of a CWS (AWWA, 2005). In addition to this larger effort
organized by AWWA, a group of California water utilities that provided critical input to the development
of EPA's Response Protocol Toolbox (USEPA, 2004a) also convened a series of workshops to focus on
the conceptual design for a CWS and identify considerations for implementation of a CWS that would be
sustainable by meeting security objectives while also providing multiple benefits to routine operations
and system performance. The approach for the design of the WS-CWS described in this document is an
extension of the concepts identified and informed by these and other utility and stakeholder efforts.
1.1.1 Overview of Contamination Warning Systems
A CWS is a proactive approach to managing threat warnings that uses advanced monitoring
technologies/strategies and enhanced surveillance activities to collect, integrate, analyze, and
communicate information to provide a timely warning of potential water contamination incidents and
initiate response actions to minimize public health and economic impacts. Components of the WS-CWS
include the following:
Online water quality monitoring. Online monitors for water quality parameters, such as
chlorine residual, total organic carbon, pH, conductivity, turbidity, etc., should be used to
establish expected levels for these parameters (a 'baseline'). Anomalous changes from the
established water quality baseline should be used as an indicator of potential contamination in the
WS-CWS.
Sampling and analysis. Water samples should be collected at a predetermined frequency and
analyzed to establish a baseline through the use of an 'unknowns' protocol. This 'unknowns'
protocol would target specific, priority contaminants, but may also detect some non-target
analytes if the analytical techniques used in the routine monitoring program are sufficiently
robust and if the analysts are trained and encouraged to investigate tentatively identified
contaminants. In addition, water samples should be collected in response to triggers from water
quality monitors or other information streams to identify the potentially unknown contaminants in
the sample.
DRAFT-121205
-------
WS System Architecture
Enhanced security monitoring. Security breaches, witness accounts, and notifications by
perpetrators, news media, or law enforcement should be monitored and documented through
enhanced security practices. This component of the WS-CWS has the potential to detect a
tampering event in progress, potentially preventing the introduction of a harmful contaminant into
the water.
Consumer complaint surveillance. Consumer complaints regarding unusual taste, odor, or
appearance of the water are often reported to water utilities, which document the reports and
conventionally use them to identify and address water quality problems. Occasionally, water
quality complaints are reported to local agencies other than the water utility, such as 911 call
centers, the health department or a city's general information number. Using an appropriate
methodology that compiles and tracks the information provided by consumers, the utility can
consider these complaints along with data from other CWS components to identify unusual trends
that may be indicative of a contamination incident.
Public health surveillance. Syndromic surveillance conducted by the public health sector,
including information such as over-the-counter sales of medication, as well as reports from
emergency medical service (EMS) logs, 911 call centers, and poison control hotlines may serve
as a warning of a potential drinking water contamination incident. Information from these
sources should be integrated by developing a reliable link between the public health sector and
drinking water utilities.
The key to an effective response to a water contamination threat is minimizing the time between
indication of a contamination incident and implementation of effective response actions. Implementation
of a robust CWS can achieve this objective by providing an earlier indication of a potential contamination
incident than would be possible in the absence of a CWS; thus, the core component of the WS program is
a CWS. A CWS is not merely a collection of monitors and equipment placed throughout a water system
to alert of intrusion or contamination. Fundamentally, it is an exercise in information acquisition and
management. Different information streams should be captured, managed, analyzed, and interpreted in
time to recognize potential contamination incidents in time to respond effectively. While the primary
purpose of a CWS is to detect contamination incidents, accidental or intentional, implementation of a
CWS is expected to result in dual-use benefits for drinking water utilities that should help to ensure
sustainability of the system.
Although many utilities undertake monitoring and surveillance activities, they are not likely to detect a
wide range of possible contamination events. For example, while some utilities currently track consumer
complaint calls, WS proposes to develop a robust spatially based system that, when integrated with data
from public health surveillance, online water quality monitoring, and enhanced security surveillance,
should provide specific, reliable, and timely information for decision makers to implement response
plans. Beyond each individual component of the WS-CWS, WS should integrate information from both
the utility and local public health agency and provide a robust consequence management plan that
involves the appropriate local officials, law enforcement, etc., to ensure that appropriate actions should
occur in response to various triggers/alarms. In the absence of a reliable and sustainable CWS, a utility's
ability to respond to contamination threats and incidents in a timely and appropriate manner is limited.
1.1.2 WaterSentinel Objectives
The overall goal of WS is to design and demonstrate an effective system for timely detection and
appropriate response to drinking water contamination threats and incidents that would have broad
application to the nation's drinking water utilities. The systematic approach to design of the WS-CWS
should reduce the time between indication of potential contamination incidents, evaluation of the possible
DRAFT-121205
-------
WS System Architecture
threat, and implementation of consequence management and response actions. More specifically, EPA's
objectives for the WS program are to design a CWS that:
Provides timely detection of contamination;
Has broad coverage of priority contaminant classes;
Is the most protective of public health using currently available and well-characterized
technologies;
Is sustainable through benefits to the water utility independent of enhanced water security (dual-
use benefits);
Is implementable, cost-effective, and reliable; and
Is ultimately applicable to utilities nationally.
To meet these objectives, EPA intends to test a number of broad hypotheses that are critical to
understanding the efficacy of a CWS. Through the initial pilot, EPA plans to test the following
hypotheses to determine whether the components of a CWS, singularly, collectively, or in some
combination, can serve as an effective warning system:
Water quality parameters (e.g., pH, chlorine residual, total organic carbon, etc.), in conjunction
with an event detection system can provide early indication of contamination incidents
Public health surveillance for indicators of disease in the population can provide early indication
of drinking water contamination
Consumer complaints can provide warning of contamination with chemicals that have a
discernable odor or taste in sufficient time to respond in a manner that reduces consequences.
Event detection software (i.e., computer-based algorithms) applied to water quality parameters,
consumer complaints, and public health surveillance both singularly and correlatively, can detect
statistical anomalies indicative of possible contamination while minimizing the number of false
alarms that a utility would otherwise have to deal with
Certain vulnerabilities to contamination can be effectively reduced through the focused
deployment of security monitoring systems that provide access control and detection capabilities.
Integration of these different monitoring and surveillance techniques increases the coverage of
contaminants, reduces the time to initial detection, and improves the overall reliability of the
system
Site characterization and triggered sampling (e.g., grab samples collected in response to an
anomaly in water quality, consumer complaints, or public health surveillance) for specific high
priority contaminants can provide corroboration of a contamination event
The conceptual approach for the integration of these information streams as part of the WS-CWS is
described in Section 1.2. A detailed discussion of the design basis for the WS system architecture as it
relates to the objectives and hypotheses described above is discussed in Section 2.0.
1.2 Overview of WaterSentinel
The WS concept of operations (ConOps) describes all of the operational activities to be used by a water
utility and public health agency to detect and respond to a contamination incident through the WS-CWS.
The WS ConOps provides the broad context from routine monitoring and surveillance activities to
recovery from an actual contamination incident. Figure 1-1 illustrates the basic ConOps for the WS-
CWS.
DRAFT-121205
-------
WS System Architecture
'\ _ Jlfi
i%&R&m&£m&m.£ Jia gsi <-.m? !?ig! i-P* i-Bi i?S5 H=
p
if
1
1
iff
is
fer
RK
i -
I
HI
If
Even! Detection |.| Credibility Determine lion
/ ^
J HM,.y S^r.Miv |
; I
k .'
1
n
m
i
5G|
: m
/_-_L______.-
: !
: Pi.<,!-!. hi^U, 1
"1 SSr^rtSSi j
~!:^
-«I
r:tl
n
1
il
:i Hi H ^S :^S ^ IP I IP I Ir
H4f>?
":"\t**:tv":"v*:t?:r":" "".?:?.?.? " : ".?*'.?.? " : "':?.*? " : " '??: ' '-
'As -SS'-^As """"""/?.." """ '?!* -::"-:1" I"::r1'-;.i -4'i
j
R.lf n,\ - ».'.,r:.; '..-
JyulifS 1:'<-><- tV---«$
1
iiiji^-EW'iLS-!
S ^»S i HJL SSii; 5
fHIS°H?
'^y^L-v^Sr-- - ->; - :-- =:; - ==. '-:- :- :=. : : : :== =.
ii
Y
I
S
*;
1
i
a
1
S!
Response
f
«««,.,.,.,«««.,.,.,««««.,.,««««.,.,««««,
CH.H<,J,,F.<.....,p01M
P-jtote noaftf- response
...... ..
;
tssss
£1| Remediation and Recovery
=
i
=
-|
z
=:
=
=
tr
=
«,,.,.,««««,,.,.,««««,,.,.,««««,,.,.,««««,,.,.,«^
:
S -imoiina and IT^;VF-S :'
O-,-.r./-a ?
i
=|
^ ^^^r^^^^^^^=^= £=; ^ "; - " " "i" "
^:T 4i;i^:^.ii:iiiifiJs ;::::: :^ . ::::f/: -. :-: " -^j^i E?_^ = - - - E? - ; - ~ : : -
Figure 1-1. Overview of WS Concept of Operations
Sections 1.2.1 through 1.2.3 briefly describe the elements of the WS ConOps and basic operation of the
WS-CWS.
1.2.1 Monitoring and Surveillance
WS-CWS monitoring and surveillance activities rely on an established set of information streams to
detect contamination events. As shown in Figure 1-2, these information streams are primarily managed
by either the drinking water utility or the public health department. In order for WS to be successful and
sustainable, a close relationship between the utility and the local public health department should be
developed and maintained to address exchange of data, coordination, and notification.
1.2.2 Event Detection
The fundamental challenge in relying on a variety of information streams as an indication of a
contamination incident is establishing a means of distinguishing anomalous patterns in these data from an
established baseline. In the WS-CWS model, event detection is a process to analyze signals from
monitoring and surveillance activities to identify anomalies that are indicative of a possible contamination
incident. Event detection algorithms could identify a pattern of unusual water quality, a cluster of unusual
consumer complaints, or unusual symptoms picked up by a public health surveillance program. When
incorporated as part of an event detection system, these algorithms can be used to identify and 'learn
from' changes in data patterns that are indicative of drinking water contamination. In short, the purpose
of the event detection algorithms is to reduce the false positive rate without missing potential events (i.e.,
without incurring false negatives). Additional information on available event detection algorithms,
software, and tools can be found in Overview of Event Detection Systems for WaterSentinel (USEPA,
2005f).
DRAFT-121205
-------
WS System Architecture
1.2.3 Consequence Management
Based on lessons learned from deployment of Bio Watch, an early-warning system designed to detect the
release of biological agents in the air through a comprehensive protocol of monitoring and laboratory
analysis, EPA recognizes that consequence management plans should be in place before any monitoring
and surveillance activities begin. As part of the WS-CWS ConOps, consequence management consists of
a series of actions taken after a potential incident is identified to establish credibility, minimize public
health and economic impacts, and ultimately return to normal operations. The consequence management
guidance developed as part of the WS pilot should build on the concepts and approach described in EPA's
Response Protocol Toolbox (RPTB). The RPTB provides a framework to guide the response to
contamination threats and incidents and establishes the foundation for the primary steps, or phases, for
consequence management as part of the WS-CWS. Sections 1.2.3.1 through 1.2.3.3 describe the general
approach for WS-CWS consequence management. A detailed discussion of EPA's approach for WS
consequence management is discussed in WaterSentinel Consequence Management Strategy (USEPA,
2005i).
1.2.3.1 Credibility Determination
Once monitoring and surveillance activities detect a possible contamination incident, a series of steps
should be taken to determine the credibility of the incident through the process of credibility
determination as discussed in Module 2: Contamination Threat Management Guide of EPA's Response
Protocol Toolbox. Based on this analysis, a decision should be made to return to normal operations or
determine that the incident is 'credible' and implement appropriate response actions. Through the WS-
CWS, the credibility determination process can be enhanced through integration of data from multiple
information streams to corroborate a 'possible' incident and provide additional information to decision-
makers in a timely manner. It is critical that a systematic approach for assessing credibility in response to
contamination threat warnings is used to ensure that all available information is analyzed in a timely and
efficient manner to minimize both false alarms and over-response to a trigger that has not been
determined to be 'credible.'
1.2.3.2 Response
Response actions are initiated upon detection of a 'possible' contamination event and continue through
determination of credibility and confirmation of a contamination threat. As described in Section 1.2.3.1,
an initial trigger indicating 'possible' contamination could come from single or multiple monitoring and
surveillance information streams. Indication of 'possible' contamination should prompt the water utility
to conduct follow up actions such as site characterization, triggered sampling, and analysis for
'unknowns' as part of credibility determination, potentially resulting in notifications to public health and
local response partners and implementation of precautionary response actions to reduce consequences
should the event later be deemed 'credible.' As the information from the initial response actions (such as
site characterization and 'unknowns' analysis and/or additional information from monitoring and
surveillance streams) is collected from or coordinated with the water utility, additional response actions
should be considered and implemented as the credibility of the incident is assessed. This process of
continuous information collection followed by assessment and action should be performed by the water
utility and others from the local, State, or Federal levels of various agencies to respond to the event,
mitigate the consequences, provide internal and external notifications, bring in additional resources for
response and analysis, and manage all related emergency response requirements associated with the
specifics of the event. Rapid response actions should be critical to the success of the CWS. These
response actions should be fully described in the WS Consequence Management Plan prior to
implementation of the CWS.
DRAFT-121205
-------
WS System Architecture
1.2.3.3 Remediation and Recovery
The goal of remediation and recovery is to return the water supply system to service as quickly as
possible while protecting public health and minimizing disruption to normal life (or business continuity).
During the remediation and recovery stage, the immediate urgency of the situation has passed, and the
magnitude of the remedial action requires careful planning and implementation. While rapid recovery of
the system is crucial, it is equally important to follow a systematic process that establishes remedial goals
acceptable to all stakeholders, implements the remedial process in an effective and responsible manner,
and demonstrates that the remedial action was successful.
1.3 Approach to Implementation
In the design of the WS-CWS, EPA aims to partner with drinking water utilities, key water sector
stakeholders, technical experts, representatives from public health departments, law enforcement, State
and other Federal agencies to implement the WS-CWS pilot and evaluate first-generation CWS
components that initially address a representative subset of priority contaminants and improve a utility's
ability to respond to any contamination threat or incident. In addition, the WS-CWS should yield
operational benefits for non-security related water quality issues and enhance collaboration/integration of
water utility and local health department operations. Through a partnership with these organizations,
EPA plans to use the results of the WS pilot to develop a sustainable model for a CWS that can be
implemented by utilities throughout the nation. Figure 1-2 presents an overview of the approach for
design, initial pilot, expansion, and development of national guidance under the WS program.
Phase
Approach
Scope
Design
Specificity
Ctesigri
Conceptual i ---' '
appiscabte
liHiai Pilot
^\
Apply to single Evaluate
pilot utility ',,
\ f-- /
. > Refine <
V / L_~« =*
J and
enhance
m
Hi
H-Tjt] -
Applies to pi!ot utility only
Expansion National Guidance
^ \
Applied b'f I/
multiple Evaluate
±> utilltles ! '> Convert to
f~^_ p" | quittance for
> / 1 s.iy utility
\ 7 Refine ! ^
^ and w
enhance
f
Nic|h tv^^Mj'iim -
Applies to each utility App'ias to rareje of ufcties
Figure 1-2. Overview of the WS Program
Following the enhancements and modifications to the initial WS-CWS design, EPA plans to engage in an
extensive outreach effort to promote the water sector's understanding and adoption of CWSs. This effort
should allow EPA, the utilities, and partner organizations to begin to establish a more protective program
(i.e., a program that could include more contaminants, more cities, more sensors, and overall greater
reliability) once the concept of the WS-CWS has been demonstrated through the initial pilot. As the
adoption and evaluation of CWS expands, the level of sophistication should evolve as well to include
more contaminants, an expanded laboratory network, a higher degree of data integration, and enhanced
detection technologies.
DRAFT-121205
-------
WS System Architecture
Throughout all phases of the WS program, EPA plans to continue to conduct supporting research to
enhance the monitoring and surveillance strategies available to be used in the WS-CWS. Research
priorities include the following:
Evaluation and development of methods for sampling and analysis
Evaluation of water quality monitors and new and emerging technologies for contamination
warning
Evaluation of data collection and transmission techniques
Refinement and enhancement of modeling and data analysis tools
Characterization of contaminant properties and risks to human health, infrastructure, and the
economy
WS program evaluation
The initial WS pilot should serve as a demonstration project for the conceptual design described in detail
in Section 2.0 of this document. Using this document as an initial framework, EPA anticipates working
with the pilot utility and partner organizations to develop a work plan for implementation of the WS-
CWS. EPA plans to provide support to the WS pilot utility and work closely with the utility to design a
program that meets its current and projected needs. In addition, EPA plans to work with the utility to
develop the necessary laboratory capabilities required to support implementation of the WS program.
Once implemented, an evaluation program should assess the effectiveness, costs, and benefits of the pilot
and recommend improvements to the WS-CWS, as well as the sustainability and multiple benefits
provided by the program. The phases of design and implementation of the initial WS-CWS pilot include
the following:
Initial Meetings. Includes participation in initial meetings between EPA, the pilot utility, and
local partners (as appropriate) to discuss the objectives, technical approach, and general
implementation strategy.
Assessment. Review of the pilot utility's current practices, procedures, and capabilities for the
technical components of the WS-CWS. This may include an initial request for relevant
information as well as an on-site assessment.
Gap Analysis and Component-Specific Work Plans. Based on the assessments conducted in
the previous phase, EPA technical staff plan to work with the pilot utility to conduct a gap
analysis to determine the appropriate enhancements and modifications to support implementation
of the WS-CWS. From this gap analysis, EPA anticipates working with the pilot utility to
develop component-specific work plans for implementation.
Enhancements and Installation. In accordance with the component-specific work plans, EPA
plans to work with the pilot utility to implement enhancements and install the necessary
equipment and systems for implementation.
Baseline Development. Establish a baseline for all components of the WS-CWS, including
online water quality monitoring, sampling and analysis, enhanced security monitoring, consumer
complaint surveillance, and public health surveillance.
Full Deployment. The WS-CWS should be fully operational and information streams should be
integrated.
Evaluation. Evaluation of the WS-CWS should occur at established increments throughout the
pilot.
Refinement. Based on the evaluation(s), WS-CWS components may require refinements or
additional enhancements to ensure proper operation of the system relative to the objectives
established in the component-specific work plans.
Maintenance. Following implementation of the WS-CWS and initial enhancements to the
system, the pilot utility should maintain the CWS. With the advancement of technology and
DRAFT-121205
-------
WS System Architecture
research, additional cycles of evaluation and refinement should be considered. However, the
frequency at which these evaluations occur can likely be decreased over time.
1.4 Document Organization
The remaining sections of this document describe the following aspects of WS system architecture:
Section 2.0: WaterSentinel Contamination Warning System Design Basis. This section
provides a detailed description of the WS design basis, CWS components, and cost
considerations.
Section 3.0: Online Water Quality Monitoring. This section describes the rationale for
inclusion of online water quality monitoring as a component in the CWS and presents a
framework for how this component should be implemented as part of the WS pilot.
Section 4.0: Sampling and Analysis. This section describes the rationale for inclusion of
sampling and analysis as a component in the CWS and presents a framework for how this
component should be implemented as part of the WS pilot.
Section 5.0: Enhanced Security Monitoring. This section describes the rationale for inclusion
of enhanced security monitoring as a component in the CWS and presents a framework for how
this component should be implemented as part of the WS pilot.
Section 6.0: Consumer Complaint Surveillance. This section describes the rationale for
inclusion of consumer complaint surveillance as a component in the CWS and presents a
framework for how this component should be implemented as part of the WS pilot.
Section 7.0: Public Health Surveillance. This section describes the rationale for inclusion of
public health surveillance as a component in the CWS and presents a framework for how this
component should be implemented as part of the WS pilot.
Section 8.0: Information Integration and Data Management. This section presents an
overview of the approach to integration of information and management of data from all CWS
components.
Section 9.0: Approach to Evaluation. This section describes how the WS-CWS and overall
pilot program should be evaluated with regard to performance, costs, benefits, and sustainability.
Section 10.0: References and Resources. This section provides a bibliography of the references
cited in this document and provides a brief summary of other documents related to the WS
program.
Appendix A: Acronyms
Appendix B: Glossary
Appendix C: Overview of Related Projects
DRAFT-121205
-------
WS System Architecture
Section 2.0: WaterSentinel Contamination
Warning System Design Basis
System architecture is the conceptual design for the WS-CWS, and describes the monitoring and
surveillance techniques that should be integrated to detect potential drinking water contamination
incidents. This section describes the results of EPA's analyses and technical considerations that lead to
the proposed WS-CWS design basis for the WS-CWS. Factors considered in developing the design basis
include a description of the contaminant threat and identification of high impact contamination scenarios,
the manner by which different classes of contaminants might be detected, the time at which different
detection strategies might provide an initial indication of contamination, and design considerations related
to reliability and sustainability.
In system design, the design basis can be described in terms of the particular problem that a system is
designed to solve or the function the system is designed to perform. It provides a framework for system
development and a benchmark against which to evaluate the performance of different design options. For
detection systems, the design basis can be described in terms of the incident, or suite of incidents, that a
satisfactory system should detect. The design basis for a drinking water CWS is defined as a series of
contamination scenarios against which specific design options should be evaluated. A contamination
scenario is specified by the location of contaminant introduction, the type of contaminant, and the
amount, concentration, and rate of introduction. In addition to the contamination scenarios that define the
design basis, a CWS design is subject to other requirements and constraints, such as the ability to detect
an event in sufficient time to implement effective response actions. For example, a design option that can
consistently detect a contamination scenario should not be acceptable if detection occurs significantly
after a response is needed.
Developing a design basis for a contamination warning system is challenging because of the large number
of potential contamination scenarios with varying degrees of consequence. The design basis may be
substantially narrowed by initially focusing on those contamination scenarios with the highest
consequences, particularly those with the potential for a high number of fatalities. However, it is not
appropriate to arbitrarily establish a numeric threshold that defines a high-consequence scenario (e.g.,
10,000 fatalities) because this can vary from utility to utility depending on the total population in the
service area, the population density profiles, configuration of the distribution system, and other factors. A
more rational approach is to evaluate and rank the consequences for a large number of potential
contamination scenarios, and use those scenarios with the most significant consequences in the design
basis (i.e., the relative ranking of scenarios is more useful than an absolute threshold). A system
constructed around such a design basis should also detect many lower-consequence scenarios, and, while
some scenarios may go undetected, the number of high-consequence scenarios that are not detected may
be minimized. The consequences associated with a particular contamination scenario are largely a
function of the specific contaminant and the location of contaminant introduction. The manner in which
these two parameters are considered in the design basis is discussed in Section 2.1 and 2.2.
Another important consideration in the design of a CWS is the timeline associated with a contamination
incident, specifically:
The time during which consequences (exposures, illnesses, fatalities, pipe contamination, etc.) are
experienced in the population,
The time of initial detection, and
The time of response actions.
DRAFT-121205 10
-------
WS System Architecture
Analysis of different contamination incident timelines can establish whether or not a given design will
meet an important design requirement - initial detection in a timeframe that allows for the
implementation of response actions that result in a significant reduction in consequences. Integration of
the results of the contamination incident timeline analysis into the CWS design is discussed in Section
2.3.
Additional considerations in the design of a CWS include reliability and sustainability, which are
discussed in Section 2.4.
2.1 Contaminant Detection Classes
There are a large number of contaminants that could cause serious harm if introduced into the drinking
water distribution system. Previous prioritization efforts resulted in a list of 80 contaminants that are of
particular concern with respect to intentional water contamination. This 'priority list' was the starting
point for the WS contaminant selection process through which 33 contaminants were identified for
consideration during implementation of the initial WS pilot (USEPA, 2005c). These 33 contaminants,
were grouped into 12 classes based on their potential for detection through each of the following
monitoring and surveillance strategies:
Utility monitoring and surveillance activities:
o Online water quality monitoring for free chlorine residual, TOC, and/or conductivity
o Laboratory analysis for the specific contaminant
o Field testing for the specific contaminant
o Consumer complaint surveillance
Public health surveillance:
o Emergency room (ER) visits, 911 calls, or emergency medical services (EMS) logs
o Clinical diagnosis
o Other forms of public health surveillance including over-the-counter (OTC) sales of
Pharmaceuticals, absenteeism, clinical laboratory orders, etc.
The water quality parameters listed under the first bullet where used in the development of the
contaminant detection classes because they have been shown to be the most reliable indicators of
contamination for the widest number of contaminants, particularly free chlorine residual and total organic
carbon (TOC) concentration (USEPA, 2005h). However, additional water quality parameters may be
used such as pH, redox potential, and turbidity, among others.
The detection potential for each of these monitoring and surveillance techniques was assessed for each of
the 33 contaminants using contaminant-specific information (USEPA, 2005c), and ranked as high,
moderate, or low according to the criteria listed in Table 2-1.
DRAFT-121205 11
-------
WS System Architecture
Table 2-1. Ranking Criteria for the Various CWS Components
Monitoring or Surveillance
Technique
Online Water Quality
Monitoring
Laboratory Analysis
Field Testing
Consumer Complaints
ED Visits, 911 Calls, EMS
Logs
Clinical Diagnosis
Other Forms of Public Health
Surveillance
Detection Potential
High
Change in two or more
parameters
Availability of a validated
lab method
Availability of a validated
field test
Detectable odor in water at
lethal concentrations
Onset of severe symptoms
within one hour
Unique and/or severe
symptoms
Onset of reportable
symptoms
Moderate
Change in only one
parameter
Availability of a non-
validated lab method
Availability of a non-
validated field test
Detectable taste in water
at lethal concentrations
Onset of severe symptoms
within four hours
Symptoms typical of
common ailments
Onset of non-reportable
symptoms
Low
No change in water quality
No method available
No field test available
No odor or taste
Gradual onset of
symptoms
No symptoms readily
evident
No symptoms readily
evident
For changes in water quality, it was generally assumed that a detectable change in chlorine residual, TOC,
or conductivity would occur at contaminant concentrations well below lethal levels for most
contaminants. This assumption is supported by numerous studies that showed substantial change in water
quality at concentrations well below the concentration that would be lethal to half of the population
exposed to that concentration (USEPA, 2005h). For consumer complaints, information about the
organoleptic properties of a contaminant was used to establish detection by this surveillance technique.
While the nature of the odor or taste was well documented, the threshold concentrations typically were
not. However, available threshold data for volatile and semi-volatile chemicals indicates odor thresholds
at the parts per billion or parts per trillion level. Given that many toxic chemicals are lethal at
concentrations in the range of parts per million, it was assumed that contaminants with a documented
taste or odor would be detected at concentrations at or below the lethal concentration.
For the purpose of ranking the detection potential for the 33 baseline contaminants, the three public health
surveillance techniques listed in Table 2-1 were considered collectively. However, the relative time in
which the information is available as well as the reliability and specificity of information received
through public health surveillance varies based on the contaminant, the illness it produces, as well as
other factors. While onset of reportable symptoms is rated as 'high' for other forms of public health
surveillance (e.g., OTC sales, absenteeism, clinical laboratory orders, etc.) in Table 2-1, if this was the
only means of detection through public health surveillance, the contaminant was ranked as having a low
detection potential via that technique due to the decreased reliability, specificity, or timeliness of this
information. For example, even if a contaminant produces reportable symptoms, without corroborating
data from another public health source such as 911 calls or clinical diagnosis, the contaminant was
considered as having a low detection potential because of the delayed timeframe and/or decreased
reliability and specificity of this information.
Groups of contaminants that can be detected by similar monitoring and surveillance strategies were
evident from this analysis, and clustering of the 33 WS baseline contaminants was used to define 12
contaminant detection classes summarized in Table 2-2.
DRAFT-121205
12
-------
WS System Architecture
Table 2-2. Detection Potential for Each of the 12 Contaminant Detection Classes
Source of
Information
Drinking
Water
Utility
Public Health
Monitoring or Surveillance
Technique
Online Water
Quality Monitoring
Laboratory Analysis
Field Testing
Consumer Calls
SS: 911, EMS, or ED Visits
SS: Clinical Diagnosis
SS: Other*
Contaminant Detection Class
1
M
H
H
H
NA
NA
NA
2
H
H
H
H
H
NA
NA
3
H
H
H
M
H
NA
NA
4
M
H
H
M
H
NA
NA
5
H
M
L
H
NA
NA
6
L
M
M
H
NA
NA
7
M
H
M
M
H
NA
8
H
M
M
M
H
NA
9
M
M
M
M
H
NA
10
H
M
M
L
H
NA
11
H
M
M
L
L
M
H
12
M
H
L
L
L
L
L
H: High potential for detection via the specified technique
M: Moderate potential for detection via the specified technique
L: Low potential for detection via the specified technique
NA: Technique not applicable for the listed contaminant class
SS: Syndromic surveillance
*Other types of syndromic surveillance include OTC sales, laboratory orders, etc.
While derived from a list of 33 contaminants, these 12 classes are comprehensive for the 'priority list'
contaminants because the classes are representative of all contaminants on that list (USEPA, 2005c).
Using these contaminant detection classes is particularly useful from a design perspective because they
allow the WS-CWS to be designed from a small number of contaminants that represent the various
detection classes, while still providing coverage of a large number of contaminants, includingand
beyondthose on the 'priority list.' Thus, these contaminant detection classes form a critical element of
the design basis for the WS-CWS.
2.2 High Impact Contamination Scenarios
A drinking water distribution system in even a moderately sized city can consist of thousands of miles of
pipe and tens of thousands of access points. At a minimum, each of these access points, or nodes, can
represent a potential location of contaminant introduction, and in the absence of specific threat
information, one may consider all nodes to be equally likely points of contaminant introduction (Murray
et al., 2004). However, the consequences of an attack can vary significantly at different nodes. Using
hydraulic distribution system, fate and transport, exposure and disease transmission models, the
consequences associated with contamination attacks at each accessible node can be estimated.
To evaluate the consequences associated with different distribution system nodes, attacks were simulated
at all nodes using one contaminant with the contaminant volume, concentration, and injection rate held
constant so that only the insertion point varied. The nodes were then ranked according to the relative
magnitude of the consequences associated with an attack at that node. The results of this analysis are
graphically depicted in Figure 2-1 as a cumulative distribution function (CDF). The CDF shows the
probability (y-axis) that a contamination incident at a particular node would result in consequences at or
below the corresponding value on the x-axis, as shown in Figure 2-1. This particular CDF shows that
there is a relatively low probability (-20%) of a random attack producing consequences impacting more
than 1,000 people. Stated another way, attacks at roughly 80% of the nodes will produce consequences
affecting less than 1,000 people. This is significant given that the most severe attack could result in
DRAFT-121205
13
-------
WS System Architecture
consequences for close to 3,000 people. This information can be used in the design basis to focus efforts
on the relatively small percentage of nodes that produce the highest consequences.
0)
0
500 1,000 1,500 2,000 2,500 3,000
Consequences (# of people)
Figure 2-1. Cumulative Distribution Function of Consequences for all Possible Insertion Points in
a Distribution System using one Specific Contaminant
The consequences shown in Figure 2-1 could represent several different endpoints, such as number of
exposures, number of illnesses, number of fatalities, area of the distribution system contaminated, miles
of pipe contaminated, number of people without potable water, or the overall economic damage resulting
from contamination. Regardless of the metric used to quantify consequences, this approach can be used
to identify the nodes at which contaminant introduction results in the highest consequences for a given
system. However, the relative ranking of nodes would vary as a function of the metric used to quantify
the consequences of the attack; thus the measure used to quantify consequences in this analysis should be
selected thoughtfully.
While the results discussed in this section are specific to one distribution system and one contaminant,
similar results have been obtained for several different systems and all contaminant classes listed in Table
2-2. A consequence assessment, such as the one presented here, can aid in the identification of nodes
where the introduction of a specific contaminant would produce the most significant consequences in a
particular distribution system. While it is desirable to design a system with the potential for detection of
contamination at any node, it is critical that the system be able to detect those incidents with the potential
to yield the highest consequence. Thus, the 'high-consequence' nodes identified through a consequence
assessment are a key element of the design basis for a CWS.
DRAFT-121205
14
-------
WS System Architecture
2.3 Contamination Incident Timeline
A key requirement of the WS-CWS is to provide initial detection in a timeframe that allows for the
implementation of response actions that result in a significant reduction in consequences. This aspect of
the WS-CWS design was evaluated through a timeline analysis, the details of which are provided in
WaterSentinel Contamination Incident Timeline Analysis (USEPA, 2005b). This document describes the
methods and assumptions underlying the timeline analyses, and therefore provides critical insight for
understanding the results from these analyses. In summary, contamination incident timelines were
generated for 10 of the contaminant detection classes listed in Table 2-2 in order to understand how the
contamination events would impact the consumer population over time, the time and method of initial
detection, and the benefit provided by the integrating multiple detection strategies. Using a suite of
modeling tools, contamination scenarios were simulated at each possible point of contaminant
introduction in a specific drinking water distribution system. The conclusions of this analysis were used
to evaluate the WS-CWS design against the requirement for timely detection.
The results of the entire ensemble of timelines (approximately 10,000 scenarios per contaminant, for a
total of 100,000 scenarios) were statistically evaluated for each contaminant class to characterize the
general trends in the propagation of consequences and the time of detection through various monitoring
and surveillance strategies. Table 2-3 presents a statistical summary of the time to the various
consequence and detection events including the median and 25th to 75th percentile range, as calculated
from the complete ensemble of scenarios. Analysis of the time to the first 1% of potential fatalities shows
a very narrow distribution, with the 25th, 50th, and 75th percentiles being equal in most cases. This is due
to the fact that the timing of the first fatalities is largely a function of the contaminant attributes, such as
the time to onset of symptoms. This also explains the range in the median times to the first fatalities
across the different contaminant classes. Fast-acting chemicals, such as those included in classes 2
through 6 result in fatalities shortly after exposure, while the pathogens (classes 10 and 11) have latency
periods of several days to longer than a week. The time to 50% of exposures is much more consistent
across the different contaminant classes, with median values ranging from 5 to 8 hours. Exposures are
largely influenced by system hydraulics and demand patterns, and thus the timing of exposures is less a
function of the contaminant, and more a function of the scale of the incident. The latter point is illustrated
by the large inter-quartile range of times to 50% of potential exposures, which range from 1 to 20 hours.
In general, the larger the consequences, the more time it takes to reach 50% of the potential exposures, for
the simple reason that it takes more time for the contaminated water to reach the larger number of people
that will ultimately be exposed. This is further evidenced by the relatively short times for contaminants
that produced relatively low numbers of exposures over all scenarios (e.g., classes 4, 5, and 9), compared
to those that produced high consequences and generally had longer times.
DRAFT-121205 15
-------
WS System Architecture
Table 2-3. Statistical Summary of Consequence and Detection Times over all Possible Scenarios
for Each Contaminant Detection Class
Class
2
3
4
5
6
7
8
9
10
11
Median1 Time to Consequence or Detection Event (hours)
1% Potential
Fatalities
3 (3 to 3)
8 (8 to 9)
3 (3 to 3)
3 (3 to 3)
3 (3 to 3)
27 (27 to 27)
49 (49 to 50)
19(19to19)
361 (361 to 362)
337 (337 to 337)
50% Potential
Exposures
8(1 to 16)
8 (2 to 18)
5(1 to 8)
6(1 to 10)
7(1 to 14)
6(1 to 12)
8(1 to 19)
5(1 to 10)
8 (1 to 20)
8(1 to 18)
Online WQ
Sensors
8 (5 to 13)
8 (5 to 12)
9 (6 to 15)
9 (6 to 13)
8 (5 to 12)
8 (5 to 12)
7 (5 to 11)
9 (6 to 13)
9 (6 to 15)
8 (5 to 15)
Consumer
Calls
7 (6 to 10)
7 (6 to 10)
7 (5 to 10)
N/A
N/A
N/A
N/A
N/A
N/A
N/A
PHS
911/ED/EMS
7 (7 to 8)
7 (7 to 8)
7 (7 to 7)
7 (7 to 7)
7 (7 to 7)
10(10to 11)
30 (30 to 30)
17(16to19)
N/A
N/A
PHS
Syndromic
N/A
N/A
N/A
N/A
N/A
37 (37 to 38)
59 (59 to 60)
43 (43 to 44)
133 (133 to 134)
207 (205 to 210)
1. The median or 50 percentile is shown in bold, while the 25 to 75 percentile range is shown in parentheses.
For proper interpretation of the statistical analysis, it is important to understand that not all of the
contamination incidents simulated were detected by a given monitoring strategy. Some of the scenarios
resulted in no exposures or fell below the detection threshold and therefore were not detected. For each
contaminant class, online monitoring detected between 33-45% of all class 2-11 scenarios; 911 calls
detected 77-80% of class 2-9 scenarios; customer complaints detected between 64-70% of class 2-4
scenarios; public health surveillance (clinical) detected 50-79% of class 7-11 scenarios; and public health
surveillance (syndromic) detected 51-54% of class 10-11 scenarios. It is important to note that the
scenarios not detected were generally those with few exposures. Furthermore, the time available to detect
and respond to a low consequence scenario is remarkably short as discussed previously. However, it is
important to consider that the WS-CWS is being designed primarily to detect high consequence incidents,
as will be illustrated through specific scenarios, rather than statistical summaries, later in this section.
The times to initial detection reported in Table 2-3 illustrate important trends in detection through the
various monitoring and surveillance strategies. Online monitoring exhibited the largest inter-quartile
range (as a percentage of the median value) of detection times, largely due to the limited number of
monitoring stations that could reasonably be deployed in a distribution system. Scenarios in which a
contaminant is inserted at a location far from the closest monitor should take several hours to detect,
while contamination inserted immediately upstream of the monitor should be detected within minutes.
Furthermore, the time to initial detection through online monitoring is generally independent of the
contaminant class, although some contaminants should be more readily detected at lower concentrations
than others (and some, like class 6, require specialized instrumentation). Additionally, detection through
online monitoring is independent of the number of people exposed - it can detect an incident before
anyone is exposed but can also miss a large incident with numerous exposures if there are no monitoring
stations in the vicinity of the incident.
Contaminants with a discernable taste or odor are likely to be detected through consumer calls (classes 1,
2, 3, and 4). Detection through consumer calls is directly related to exposures, and there are a minimum
number of exposures that should occur before the utility should be alerted to the problem (68 exposures
based on the assumptions of this analysis). For this reason, small-scale incidents should take longer to
detect through consumer calls, or may not be detected at all. However, incidents with even a modest
number of exposures (i.e., more than 100) should be detected very quickly through this surveillance
technique, with the most significant time delay attributable to the time that it takes consumers to call the
utility and the time that it takes the utility to process the calls and recognize a potential problem.
DRAFT-121205
16
-------
WS System Architecture
Chemicals and biotoxins that produce very severe or sudden symptoms (classes 2 through 6) or symptoms
of a unique nature (classes 7 through 9) are likely to be detected through public health surveillance of 911
calls, ER visits, or EMS logs. Similar to consumer complaints, detection through this surveillance
technique is directly related to the number of symptomatic people. Due to the severity of symptoms
associated with most of these contaminants, even a relatively small incident is likely to be detected
through this surveillance method. For fast-acting chemicals, the most significant time delay for detection
through this technique is the time that it takes for public health to recognize the potential problem
followed by the delay in alerting the utility to the potential problem. For other contaminants, such as the
biotoxins, the time lag between exposure and onset of symptoms is the most significant delay in
detection.
Pathogens that produce serious diseases with a gradual onset of symptoms (classes 10 and 11) are likely
to be detected through public health surveillance of clinical cases of a specific disease or surveillance of
general health indicators in the population. As shown in Table 2-3, detection through this form of public
health surveillance occurs much later than detection through online monitoring. Comparison of these
times with the times to 50% of potential exposures clearly indicates that detection through this technique
comes too late to prevent exposures; however, it may provide ample time for medical intervention to
minimize the number of fatalities.
In general, the detection sequences for contaminant classes 1 through 9 can vary from one scenario to
another. For example, in one case it may be public health (PH)-911 - Consumer Calls - Online Monitors.
In another scenario for the same contaminant, it may be Online Monitors - Consumer Calls - PH-911.
This indicates that information from multiple data streams should be available in a similar timeframe for
most of the contaminants in these classes. However, classes for which public health surveillance (clinical
or general) is applicable, there would generally be two detection opportunities: 1) rapid detection through
online monitoring, or possibly 911 calls, in time to limit exposures through utility response actions and 2)
delayed detection through public health surveillance (clinical or syndromic) that comes too late to prevent
exposures, but which could inform medical response actions and minimize the number of fatalities or
severe illnesses.
In addition to the statistical summary presented above, one specific scenario per contaminant was
analyzed in greater detail to illustrate the timing of detection and response actions. The scenario selected
was that resulting in the 90th percentile consequences (i.e., only 10% of the scenarios have consequences
more severe than this example scenario). The timeline summary presented in Table 2-4 lists three critical
points in the timeline for the 90th percentile scenario:
The time that 'possible' water contamination was detected through one of the WS-CWS
components, and the means of initial detection. Following the detection of 'possible'
contamination, it is assumed that the utility should begin collecting additional information in an
effort to establish whether or not the incident is 'credible.'
The time that the contamination incident was determined to be 'credible' and the information that
provided the basis for this determination. Once an incident has been deemed 'credible' it is
assumed that response actions are implemented to prevent further exposures and fatalities.
The consequences that would be prevented, in terms of exposures and fatalities, if effective
response actions were implemented 15 hours after credibility was established.
The time (AT) to each critical event in the timeline is reported relative to the start of contaminant
insertion, which is the time shortly before exposures begin.
DRAFT-121205 17
-------
WS System Architecture
Table 2-4. Timeline Summary for the 901
Class
,th
Percentile Scenarios for each Contaminant Detection
Class
2
3
4
5
6
7
8
9
10
11
'Possible' Contamination
Means of initial detection
Consumer calls
Consumer calls
Online monitoring - cond.
PH surveillance - 91 1 calls
PH surveillance - 91 1 calls
Online monitoring - cond.
Online monitoring -CI2
Online monitoring -TOC
Online monitoring -CI2
Online monitoring - Cb
AT1
4:30
7:00
2:00
7:30
12:30
21:00
6:00
12:00
25:00
3:00
'Credible' Contamination
Basis for Determination
PH surveillance - 91 1 calls
PH surveillance - 91 1 calls
PH surveillance - 911 calls
Online monitoring - TOC
Online monitoring - GC/MS
PH surveillance -911 calls
Site characterization
Site characterization
'Unknowns' analysis - PCR
'Unknowns' analysis - PCR
AT1
5:30
7:30
5:30
13:00
14:00
24:00
9:00
14:30
37:00
14:00
Consequences
Prevented
Exposures
54%
61%
4%
30%
63%
22%
45%
27%
55%
27%
Fatalities
61%
85%
6%
32%
68%
99%
100%
79%
100%
100%
1. AT is the time in hours and minutes (hh:mm) that the incident was deemed 'possible' or 'credible' relative to the start of
contaminant insertion.
These results for the 90th percentile scenarios generally reinforce the trends observed in the overall
statistical analysis. Initial detection and subsequent credibility determination for contaminant classes 1
through 7 should generally occur rapidly through some combination of consumer calls, 911 calls, or
online monitoring, as applicable to the specific contaminant. There is a relatively short time delay
between initial detection and determination that the threat is 'credible' (!/2 to 5!/2 hours) for all of these
contaminants due to comparable times in which information from these various detection strategies
becomes available. The ability to quickly establish credibility in these scenarios provides sufficient time
for response actions to prevent over 50% of potential exposures and fatalities in most cases. The scenario
for contaminant class 4 is an exception in that response actions were implemented too late to prevent a
significant percentage of consequences. Similarly, a relatively small percentage of exposures were
prevented for classes 5 and 7. One reason for this is the relatively low number of exposures compared
with the other scenarios (5 to 10 times fewer exposures), and as discussed previously, low consequence
incidents are more difficult to detect in sufficient time to respond in a manner that limits consequences.
Classes 8 and 9, representing the biotoxins on the WS baseline contaminant list, and classes 10 and 11,
representing the pathogens on the WS baseline list, are all initially detected through online monitoring of
chlorine residual and TOC. These initial detection times vary based on the relative location of the sensor
to the point of contaminant insertion, and over just these 4 scenarios range from 3 to 25 hours. For the
class 8 and 9 contaminants, credibility is established a few hours after initial detection through the use of
field tests implemented during site characterization. For the class 10 and 11 contaminants, credibility can
not be established until the results of polymerase chain reaction (PCR) analysis, part of the 'unknowns'
analysis are available. The field tests available for pathogens are not sufficiently sensitive to detect them
at concentrations of concern in drinking water.
Contaminant classes 2 through 6 represent fast-acting chemicals with a very short time between exposure
and serious health consequences. Thus, for these contaminants the only response actions that would
minimize consequences are those that limit exposures, such as issuance of a 'do not use' notice, several
hours before the majority of potential exposures occur. After this point, there is little that can be done to
minimize the public health consequences. Similarly, rapid response to infrastructure threats, such as
classes 1 and 12, are necessary to minimize the spread of the contaminant, and thus the effort required for
remediation and recovery.
Classes 8 through 11 represent contaminants with a latency period for which medical treatment is
available. There are two opportunities for response for these contaminants. The first occurs if initial
DRAFT-121205
18
-------
WS System Architecture
detection and credibility determination occur quickly enough to limit exposures. The specific scenarios
summarized in Table 2-4 for these contaminant classes follow this model, with initial detection occurring
soon enough to allow for effective response actions to minimize exposures. However, if this first
opportunity is missed, there is a second opportunity for detection through public health surveillance, as
shown in Table 2-3. While this occurs too late to prevent any exposures, it could provide sufficient time
to implement medical response actions to limit the number of serious health consequences and fatalities.
The results of the timeline analysis demonstrate different detection sequences (i.e., the relative time that
the various WS-CWS components provide information about potential contamination) associated with
each contaminant detection class. Figure 2-2 presents a simple graphical summary of the relative timing
of consequences for different contaminant categories and detection by WS-CWS components.
I/)
1
-------
WS System Architecture
Site characterization and 'unknowns' analysis are important for corroborating 'possible'
contamination under many scenarios, but further refinement and validation of these tools are
necessary to be able to use them with confidence.
Public health surveillance of clinical cases and general symptomatic information is a potential
means of detecting a potential public health crisis, but not in sufficient time to limit exposures to
contaminated water resulting from a short-term contamination incident. Furthermore,
communications between public health and the utility need to be optimized to quickly recognize a
potential link to drinking water.
2.4 Additional Design Considerations
The previous subsections have described principal considerations in the development of a design basis for
a contamination warning system, namely high consequence locations of contaminant introduction, high
consequence contaminants and the means by which they might be detected, and the relative time in which
contamination might be detected through different monitoring and surveillance techniques. This section
describes two other important design considerations: reliability and sustainability.
2.4.1 System Reliability
For a CWS, reliability can be considered from at least two perspectives. The first is system operation,
that is, factors such as CWS component downtime and maintenance requirements. The second is system
performance, defined as the ability of the system to provide information that leads decision makers to
successfully infer that contamination has or has not occurred. While both aspects of reliability are
important, the latter is more pertinent to the conceptual design of a system because it relates to the
'information reliability' of a system that is largely based on the acquisition/interpretation/management of
information. Thus, this section will focus on system performance.
System performance can be characterized in terms of the probability of detecting an intentional
contamination incident, which is expected to occur infrequently, relative to the probability of a false
alarm (i.e., a false-positive). For hazardous conditions that occur, the probability that an environmental
indicator (e.g. public health surveillance, water quality monitoring, etc) will identify a contamination
incident can be calculated through the use of Bayesian inference (Hrudey and Rizak, 2004). Given the
low probability of a contamination incident, the probability that any one data stream will identify a
contamination incident will itself be low. However, the information available from the multiple WS-
CWS components can increase the probability of accurately detecting an anomalous condition in the
water system. While the quantitative improvement in probability of correctly identifying contamination
can vary significantly depending on the characteristics of the various information streams, the qualitative
benefit of using multiple data streams should remain. A more complete discussion of this approach and
its implications is found elsewhere (Magnuson, et. al., in preparation).
To illustrate the improvement in system performance achieved by integrating multiple data streams, the
following simple example considers two water quality parameters that should be included in the WS-
CWS design: chlorine and total organic carbon (TOC). For the purpose of this analysis, it is assumed that
a contamination incident should decrease chlorine residual and increase TOC levels. Assuming that the
two sensors are co-located, they measure the same parcel of water, thus eliminating any confounding
factors associated with spatial variability. It is also assumed that the responses from each of the two
sensors are independent of one another in the absence of a water quality change, and that, in the case of a
water quality change, that the response of one sensor is independent of the other. Table 2-5 provides
example event detection characteristics for the chlorine sensor and TOC analyzer. The actual event
detection characteristics would have to be experimentally determined and may vary with the magnitude of
the sensor response.
DRAFT-121205 20
-------
WS System Architecture
Table 2-5. Probability of Sensors Responding
Probability of sensor producing true
positive response
Probability of sensor producing false
positive response
Chlorine Sensor
0.99
0.05
TOO Sensor
0.95
0.01
Figure 2-3 provides an illustration of chlorine and TOC sensor response data with significant deviations
from the baseline. The figure also illustrates the calculated posterior probability of an event, calculated
with one sensor or a combination of both sensors. It was assumed that the event only occurs 1 time in a
1000. When the individual sensors are both detecting a change (t=8), the calculated posterior probability
from the chlorine sensor data and the TOC sensor data is 0.02 and 0.09, respectively. When both sensors
are detecting a change (t=8), the calculated posterior probability increases to 0.65. Another observation
in Figure 2-2 is that when one sensor is responding and the other is not, the posterior probability
calculated from combining the data is in between the probabilities calculated for the individual sensors
(t=3,4). If neither sensor is detecting a change, combining the information decreases the calculated
posterior probability of an event.
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
-0.10
re
&
o
o
Q.
w/ chlorine detector only
w/ TOC detector only
w/ both detectors
10
time
151
1 10-
01
c 54
in
9>
in
c
Ol
m
0
-15
/-v|_l
I 1
10
time
Figure 2-3. Comparison of the Posterior Probability of an Event Using Data from Individual and
Multiple Water Quality Sensors
DRAFT-121205
21
-------
WS System Architecture
While one of the ultimate objectives of Water Sentinel is to better characterize the actual probabilities of
contamination detection through monitoring and surveillance, this example illustrates that integration of
multiple data streams can dramatically improve system performance and overall reliability for detecting
contamination incidents. The multiple monitoring and surveillance techniques used in the WS-CWS
extend beyond integration of multiple water quality data streams to other independent information
streams including water quality data, consumer calls, public health surveillance, security alarms, results
from site characterization and sample analysis. The WS-CWS pilot should provide an unprecedented
opportunity to develop the information necessary to better characterize and quantify the value of
integrating information from numerous monitoring and surveillance activities to improve our ability to
reliably detect contamination incidents, i.e. to minimize the frequency of false alarms.
In addition, the approach described in this section may be beneficial in quantitatively elucidating the
'dual-use' benefits of CWS, which are related to system sustainability, which is more completely
discussed below. Namely, the type of events which could be viewed as 'dual-use' may occur more
frequently that intentional contamination. Using estimates for the occurrence of these non-security
related water quality events, which could be available from water utility operation records, probabilistic
calculations can be performed to inform decisions regarding the implementation of various monitoring
and surveillance strategies with dual-use application. For example, the probabilities of detection could be
compared for different online sensors that respond accurately 85% and 95% of the time, respectively.
Based on the occurrence rate of the 'dual-use' event being considered, it is possible that the calculated
probability of detection of this event by the two sensors is similar. If the first sensor only costs 1/10 of
the second, clear savings may be realized, while maintaining a similar level of 'dual-use' benefit. Such
considerations are important to the long-term sustainability of the system as utilities make decisions
regarding the investment of limited resources in various CWS components. Other considerations related
to sustainability are discussed in the following subsection.
2.4.2 System Sustainability
Sustainability of a CWS considers factors that influence the ability of an entity, such as a drinking water
utility, to operate and maintain the CWS over an extended period of time and in the face of competing
priorities that could siphon resources away from the program. In most cases, the analysis of sustainability
for a CWS should entail a cost-benefit analysis. An order-of-magnitude cost estimate for the
development and implementation of the WS-CWS pilot at one utility in 2006 is presented in Section 2.5.
Benefits of the WS-CWS can be characterized as primary or secondary. Primary benefits should be
related to the early detection of, and response to, a contamination incident, and might be quantified in
terms of consequences avoided due to the implementation of the CWS. The primary benefits of the
system can be estimated through modeling, but should ultimately need to be assessed during evaluation of
the WS pilots.
Additional benefits consider 'dual-use' application of the system, potentially including:
Detection of cross-connections and other distribution system water quality problems.
Enhanced knowledge of distribution system water quality leading to improved operations (e.g.,
more consistent disinfection residual levels, improved corrosion control, early warning about
nitrification episodes, means to evaluate the efficacy of flushing programs, etc.)
Identification of problem valves (closed, partially closed, inoperable).
Improved relationship with public health, including mutual sharing of information and alerts.
Improved coordination with local, state, and federal response organizations.
Consequence management plans applicable to any water quality emergency.
Improved consumer complaint tracking and response.
DRAFT-121205 22
-------
WS System Architecture
Integration of disjointed information resources and systems.
Improved laboratory capability and application of methods developed for WS to routine water
quality monitoring programs.
Established relationships with reference and confirmatory labs.
2.5 Preliminary Cost Analysis for the WaterSentinel Contamination Warning System
An important aspect of sustainability for a CWS at any water utility is the cost of implementation and
maintenance. While the specific costs associated with implementation of the WS-CWS should be based
on the utility-specific system architecture, an order-of-magnitude cost analysis was conducted to provide
an initial assessment of costs associated with various approaches for system design and varying degrees
of utility capability. Preliminary Cost Analysis for WaterSentinel Contamination Warning System
presents a detailed description of the preliminary cost analysis (USEPA, 2005g). One of the objectives of
the WS pilot is to develop more detailed and accurate cost estimates for the deployment and operation of
a CWS.
Costs were estimated for the Fiscal Year 2006 (FY06) WS pilot project and focused specifically on
establishing and/or enhancing those WS-CWS components that were exclusive to the WS program and
that would be incurred to deploy and operate the WS-CWS. Costs for activities that would support the
WS program and other programs, and costs that would be incurred only in the event of a credible incident
were not included. These included Agency programmatic costs; costs related to laboratory network
infrastructure; data management infrastructure; analysis of triggered or response samples; and
consequence management in the event of an incident.
Each WS-CWS component was analyzed, and costs were estimated for 'low,' 'moderate,' and 'high'
categories based on component-specific variations in terms of levels of existing utility capability. Table
2-6 summarizes the definitions of these cost categories for each of the WS-CWS components.
Table 2-6. Definition of Cost Categories for WS-CWS Components
WS-CWS
Component
High-Cost Category
Moderate-Cost Category
Low-Cost Category
Online
Water
Quality
Monitoring
The utility has a hydraulic
and water quality network
model that, through the
application of tracer studies
and water quality monitoring
program, requires significant
modification and refinement
to adequately represent
distribution system
hydraulics prior to designing
a sensor network through
approaches such as TEVA.
The utility has a hydraulic and
water quality network model
that, through the application of
tracer studies and water
quality monitoring program,
has been sufficiently
developed to adequately
represent distribution system
hydraulics prior to designing a
sensor network through
approaches such as TEVA.
The utility has a hydraulic
and water quality network
model that has already
been verified through their
own tracer studies and
water quality monitoring
program ensuring a
network model that
adequately represents
distribution system
hydraulics and is suitable
for designing a sensor
network through
approaches such as TEVA.
Sampling
and
Analysis
Analysis of all baseline
monitoring samples for both
chemicals and pathogens are
performed by a contract
laboratory. Baseline samples
are collected on a monthly
basis.
Analysis of all baseline
monitoring samples for
chemicals performed by utility
laboratory; for analysis of
pathogens a contract
laboratory should be used.
Baseline samples collected on
a monthly basis.
Analysis of all baseline
monitoring samples
performed by utility
laboratory; samples
collected on a monthly
basis.
DRAFT-121205
23
-------
WS System Architecture
ws-cws
Component
High-Cost Category
Moderate-Cost Category
Low-Cost Category
Consumer
Complaint
Surveillance
Manual tracking of consumer
complaints within a utility.
Semi-automated tracking of
consumer complaints within a
utility.
Existing, automated
tracking of consumer
complaints within a utility.
Enhanced
Security
Monitoring
Significant enhancements to
existing physical security
monitoring including
installation of monitoring
equipment at 20 field
locations and 20 water quality
monitoring stations.
Moderate enhancements to
existing physical security
monitoring including
installation of monitoring
equipment at 10 field locations
and 10 water quality
monitoring stations.
Minimal enhancements to
existing physical security
monitoring stations
including installation of
monitoring equipment at 1
field location and 9 water
quality monitoring stations.
Public
Health
Surveillance
No existing syndromic
surveillance at local health
depart ment(s).
Existing syndromic
surveillance at local health
department(s), but there is no
existing mechanism for
integration of water quality
data.
Utility is participating in
National Homeland Security
Research Center (NHSRC)
public health surveillance
research project to
integrate water quality and
public health data.
Table 2-7 summarizes the results of the costing exercise for each WS-CWS component. For all WS-
CWS components, the cost analysis included estimates for preliminary assessments and modifications to
facilitate integration of the component as part of the WS-CWS; costs associated with installations and
enhancements to existing systems, hardware, and operations; costs associated with utility labor and
operation and maintenance; data management and analysis; and estimated EPA support.
Table 2-7. Preliminary WS-CWS Component Cost Estimate
WS-CWS Component
Online water quality monitoring
Sampling and analysis
Consumer complaint surveillance
Enhanced security monitoring
Public health surveillance
High-Cost Category
$ 4,200,000
$ 1,200,000
$ 1,100,000
$ 4,400,000
$ 1,500,000
Moderate-Cost Category
$ 2,500,000
$ 700,000
$ 500,000
$ 2,300,000
$ 800,000
Low-Cost Category
$ 1,400,000
$ 500,000
$ 200,000
$ 700,000
$ 400,000
Additional considerations for the preliminary cost analysis are summarized below for each of the WS-
CWS components.
2.5.1 Online Water Quality Monitoring
As presented in Table 2-6, the state of the utility's hydraulic model was a driving factor in the assessment
of estimated costs associated with online water quality monitoring. The cost analysis included estimated
costs for model refinement and calibration as defined in the cost categories in Table 2-6 as well as costs
associated with conducting tracer studies, and applying TEVA or other approaches to determine
monitoring station locations. Based on preliminary results available from TEVA, an estimate of 30
monitoring stations within the distribution system was used. The actual number of monitoring stations
deployed to achieve a certain level of coverage should vary by utility. Estimated costs for each
monitoring station included the following components:
Capital costs: Include estimated costs associated with the multi-probe sensor, filter apparatus
(for routine or triggered sampling), composite sampler (for routine or triggered sampling),
installation, supervisory control and data acquisition (SCADA) connection, and capital
infrastructure improvements. Capital equipment costs were based on estimates developed for
EPA's Test and Evaluation (T&E) facility for YSI Sonde and Hach Astro TOC (USEPA, 2005J).
DRAFT-121205
24
-------
WS System Architecture
Operation and Maintenance (O&M) costs: Include estimated costs associated with the
maintenance of hardware and software for each monitoring station based on estimates developed
for EPA's T&E facility for YSI Sonde and Hach Astro TOC (USEPA, 2005J).
Labor costs: Include estimated costs associated with reagent preparation, calibration, travel, and
maintenance based on estimates developed for EPA's T&E facility for YSI Sonde and Hach
Astro TOC (USEPA, 2005J).
2.5.2 Sampling and Analysis
As in the estimate for online monitoring, the cost analysis for the sampling and analysis component of the
WS-CWS assumed 30 sampling sites within the distribution system based on preliminary results available
from TEVA. However, the number of sampling sites should vary for each utility implementing a CWS.
Considerations for selection of sampling locations and development of a baseline monitoring program are
discussed in Section 4.2. Analytical costs were based on commercial laboratory estimates for the
contaminants or contaminant classes of concern. For chemical contaminants, it was assumed that up to an
additional two or three contaminants could be identified for little to no additional analytical cost. For
pathogens, select agent costs were based on estimates provided by the Laboratory Response Network
(LRN) for environmental sample analysis by LRN labs performing analyses for commercial customers
during the anthrax attacks in fall 2001. It should be noted that additional investment needed to implement
in-house capability at the utility laboratory is substantial, but utility specific. Laboratory infrastructure
needs would be evaluated in terms of sustainability of the CWS at a given utility and would need to be
consistent with the utility's long-term business case.
For the purpose of this estimate, analytical costs per sampling station included the following:
Pathogens: Six bacteria, four of which are also select agents
Chemicals, radionuclides, and biotoxins: Cyanides (1 contaminants), Arsenic compounds (1
contaminant), Metals (2 contaminants), Fluoride compounds (1 contaminant), Herbicides (1
contaminant), Petroleum products/hydrocarbons (2 contaminants), Organophosphorus compounds
(7 contaminants), Rodenticides (1 contaminant), Carbamates (4 contaminants), PCBs (1),
Radionuclides (3 contaminants), Biotoxins (3 contaminants)
2.5.3 Enhanced Security Monitoring
Many drinking water utilities have implemented or are in the process of implementing physical security
enhancements based on vulnerability assessments conducted as a requirement under the Bioterrorism
Preparedness and Response Act of 2002 (BTACT, 2002). As such, costs associated with implementation
of this component should generally focus on integration of physical security information with other CWS
components and perhaps enhanced security monitoring at a small number of locations.
2.5.4 Consumer Complaint Surveillance
Approaches for recording, tracking, and managing consumer complaints vary from city to city. Upgrades
to consumer complaint surveillance software and data management tools along with establishment of
integrated call centers were driving factors in costs associated with implementation of this component of
the WS-CWS. Additional considerations for implementation of consumer complaint surveillance are
discussed in Section 6.1.
2.5.5 Public Health Surveillance
Integration of public health surveillance as a component of the WS-CWS requires the utility to coordinate
with the local health department(s) and may require the utility to provide support to the health
DRAFT-121205 25
-------
WS System Architecture
department(s) in terms of capital costs, labor, or both. This cost analysis assumed that there will be an
electronic exchange of water quality and health data through a syndromic surveillance system. For the
initial WS pilot, EPA plans to work with the pilot utility and local health department(s) to determine the
most effective means to exchange information given existing systems, existing protocols, and staffing
resources. Section 7.2 discusses alternate options and considerations for implementation of this
component of the WS-CWS.
The results of this preliminary cost analysis suggest the following in terms of design of the WS-CWS:
Leveraging of existing EPA programs (e.g., TEVA) and water security efforts (e.g., enhanced
security monitoring as a result of vulnerability assessments) provides a significant advantage in
terms of both the cost associated with implementation of the WS-CWS at the pilot utility and the
time required to implement the WS-CWS components.
While multiple design options may be considered for implementation of each WS-CWS
component, the cost of implementation and sustainability of these options should be considered in
order to meet the objectives of the WS program as identified in Section 1.
While this costing exercise provides a useful tool for evaluation of design options based on existing utility
capability, it is not meant to be a definitive analysis of costs associated with implementation of the WS-
CWS, but an initial estimate. These costs should be tracked and refined through implementation of the
WS-CWS pilot to assist in the cost-benefit analysis and evaluation of the WS-CWS design and the WS
program.
2.6 Summary of WaterSentinel CWS Design Basis
For the WS-CWS, the design basis is defined as a series of contamination scenarios against which
specific design options should be evaluated. The WS design basis considers possible locations of
contaminant introduction, various contaminant classes, different methods of detection, timing of
detection, reliability, and sustainability. The design of the WS-CWS consists of several monitoring and
surveillance strategies including: water quality monitoring, sampling and analysis, enhanced security
monitoring, consumer complaint surveillance, and public health surveillance. The manner in which this
integrated approach to contaminant warning satisfies the design basis is described by the following:
Contaminant Coverage: Analysis of contaminant properties and detection techniques clearly
demonstrates that no single approach would provide timely detection for all contaminants of
concern; however, the integrated approach implemented under WS has the potential to provide
timely detection of a very high percentage of priority contaminants.
Spatial Coverage: The monitoring components of the WS-CWS (water quality sensors,
sampling and analysis, and enhanced security monitoring) have intrinsic limitations to the spatial
coverage that each can achieve. On the other hand, surveillance components of the WS-CWS
(consumer complaint and public health surveillance) rely on consumer observations and behavior,
and thus provide dense spatial coverage throughout a distribution system. Thus, integration of
both monitoring and surveillance systems in the WS-CWS is necessary to achieve a high degree
of spatial coverage.
Timeliness of Initial Detection: Different contaminants are first detected by different
monitoring and surveillance techniques. Thus, by integrating multiple data sources, the time of
initial detection is reduced across all contaminants, and even those that act very rapidly within the
exposed population may be detected in time to implement an effective response.
Reliability: All monitoring and surveillance techniques should produce false positive and false
negative results, which decreases reliability of detection. However, integration of multiple data
streams can dramatically improve the reliability of the overall system because the overall rate of
DRAFT-121205 26
-------
WS System Architecture
false positive and false negatives for the integrated data streams should be substantially lower
than the rates for any one detection strategy.
Sustainability: The integration of multiple monitoring and surveillance strategies already in use
at the utility and public health department would improve acceptance of the system, and thus
long-term sustainability. The CWS is being designed as a dual-use application that should benefit
the utility in day-to-day operations while also providing the capability to detect intentional or
accidental contamination incidents.
Table 2-8 describes the manner in which each of the WS-CWS components addresses each of these
aspects of the WS design basis. Note that some of these benefits cannot be quantified until the WS pilot
is deployed and EPA gains substantial experience; thus the importance of implementing and evaluating
the WS-CWS through a pilot program.
The design basis presented in this section leads to the multi-pronged approach developed for the WS-
CWS, as summarized in Section 1.0. It also forms a framework for system development and a benchmark
against which to evaluate the performance of different design options. The following sections provide
more detail regarding the basis for including each of the WS monitoring and surveillance components,
and describe the general framework for design and implementation of each component.
DRAFT-121205 27
-------
WS System Architecture
Table 2-8. WS-CWS Components and their Contributions to the Approach for WaterSentinel
WS-CWS
Component
Online Water
Quality
Monitoring
Sampling
and Analysis
Enhanced
Security
Monitoring
Consumer
Complaint
Surveillance
Public
Health
Surveillance
Capability
Can indicate the presence
of a contaminant that
significantly affects one or
more monitored
parameters that serve as
indicators of
contamination.
Can positively identify the
presence of any
contaminant in the suite of
target analytes and above
the MDL.
Can detect an intrusion that
may have provided the
opportunity for
introduction of any
contaminant.
Can indicate the presence
of a contaminant that
significantly affects one or
more aesthetic qualities of
water.
Can detect the presence of
a symptom or illness in a
population which may be
the result of the presence
of a disease causing agent.
May be able to identify the
contaminant through
clinical diagnosis /testing.
Contaminant
Coverage
High detection
potential for
classes 2, 3, 5, 8,
10, and 11;
Moderate detection
potential for
classes 1, 4, 7, 9,
and 12.
High detection
potential for
classes 1, 2, 3, 4, 7,
and 12; Moderate
detection potential
for classes 5, 6, 8,
9,10,11.
Covers all
contaminant
classes.
High detection
potential for
classes 1 and 2;
Moderate detection
potential for
classes 3 and 4.
Covers
contaminant
classes 2 through
1 1 ; detection
potential varies
with type of
surveillance.
Spatial Coverage
Function of
location, number,
and density of
monitoring
stations
Function of
location, number,
and density of
sampling stations,
as well as sample
type (composite
vs. grab).
Limited to those
elements of
infrastructure for
which physical
security can be
monitored.
Entire service area
for contaminants
with detectable
organoleptic
characteristics.
Comprehensive
coverage of a
particular city or
county, which may
include all, or a
large portion of,
the service area.
Timeliness
Function of hydraulic
travel time from the point
of contaminant
introduction to the sensor,
and the concentration of
the contaminant.
Function of sampling &
analysis frequency and
the total time required to
process the sample and
analyze the results.
Function of the type of
security monitoring
system and the time
required to evaluate a
security breach.
Function of the time from
exposures to consumer
reporting, complaint
categorization,
assessment and
investigation.
Function of the time from
the initial exposures, the
onset of symptoms, and
the point at which public
health officials recognize
the incident as a potential
water-borne illness.
Reliability
Rate of false positive /
negative results in this
application is largely
unknown at this time. May be
addressed through event
detection systems and
consequence management.
Function of the reliability of
sampling and analysis
methods (high for established
techniques). Baseline needed
for reliable interpretation of
results.
Can be a reliable means of
identifying an intrusion,
especially when these
breaches may involve
contamination, such as in
storage tanks and clear wells.
May be addressed through
consequence management.
A potentially reliable indicator
for contaminants with
detectable characteristics if a
robust complaint reporting
and tracking system is in
place.
May be a reliable means of
identifying the incidence of
illness in a population, but
communication between
drinking water and public
health officials is not always
quick enough for appropriate
response, intervention and
remedial actions to take place.
Sustainability
Provides utility with a
better understanding of
water quality variability
throughout distribution
system and provides an
opportunity to optimize
distribution system
operation.
Provides utility with an
opportunity to exercise
sampling and laboratory
protocols and may; provide
information about
previously unknown
contaminants that occur in
the system.
Provides utility with
increased physical
infrastructure protection
and awareness. Reduces
the occurrence of nuisance
tampering.
Provides utility an
opportunity to manage
consumer information
more effectively and can
serve as a tool for
enhanced consumer
confidence.
Provides an opportunity for
collaboration between
utility and local health
department(s).
DRAFT-121205
28
-------
WS System Architecture
Section 3.0: Online Water Quality Monitoring
Online water quality monitoring has been used as a tool in the drinking water treatment industry for
objectives such as process control and maintenance of acceptable finished water quality. For example,
turbidity has been used as a process control tool for conventional filtration plants for decades. Chlorine
residual analyzers are used in the treatment plant to ensure that disinfection requirements are met. pH is
monitored to make sure that corrosion control measures are effective. Recently TOC has been used in
many utilities to quantify removal of organic matter through various treatment processes and to optimize
strategies to minimize the formation of organic disinfection byproducts. Given the familiarity of utility
operators with these water quality parameters, and the obvious potential for dual-use application, online
water quality monitoring has been considered as a potential means of detecting contamination incidents in
the distribution system. However, water quality monitoring in the distribution system has been limited to
date, and the application of this tool in the context of a CWS is largely untested.
Nonetheless, online water quality monitoring appears to be one of the more promising approaches for
detecting contamination incidents that is currently available, as demonstrated through research conducted
over the past few years. Thus, online water quality monitoring is included as a component of the WS-
CWS due to its demonstrated potential to rapidly detect contamination through changes in several
commonly used water quality parameters. These changes may result from the aqueous chemistry of the
contaminant (e.g., dissolution of an organic compound may result in an increase in the TOC
concentration) or from reactions with the disinfectant residual (e.g., oxidation of the contaminant
consumes the free chlorine residual). While there are limited empirical data regarding the impact of many
contaminants of concern on conventional water quality parameters, there has been a substantial amount of
research over the past few years demonstrating that many contaminants of concern, including several WS
baseline contaminants, can produce measurable changes in conventional water quality parameters.
Furthermore, many of these contaminants have been shown to impact water quality at concentrations well
below reported lethal dose concentrations. A summary of the results from some of the more
comprehensive water quality studies is presented in Online Water Quality Monitoring as an Indicator of
Drinking Water Contamination (USEPA, 2005h).
Guidance on the design of online contamination warning systems has been provided in ASCE's Interim
Voluntary Guidelines for Designing an Online Contaminant Monitoring System (Pikus, 2004), which
synthesizes many publications on the subject, including: King, et al. (2004), Hargesheimer et al. (2002),
Grayman et al. (2001), and Gullick (2001). This section considers the published body of work on the
subject, but goes beyond these recommendations to develop a detailed and comprehensive approach to the
design of the online monitoring component of the WS-CWS. Specifically, this section describes the basis
for the selection of water quality parameters and sensors, the design of various sensor stations, the
systematic design of the sensor network, the approach to management and analysis of data from an online
water quality monitoring network, and the framework for evaluation of the monitoring network.
3.1 Water Quality Sensors
Sections 3.1.1-3.1.2 provide an overview of consideration for selection of water quality parameters and
selection of water quality sensors in the context of the WS-CWS.
3.1.1 Selection of Water Quality Parameters
As discussed in Section 2.0, one or more conventional water quality parameters have been shown to
change with the presence of representative contaminants from 11 of the 12 contaminant detection classes,
as illustrated in Figure 3-1.
DRAFT-121205 29
-------
WS System Architecture
Figure 3-1. Contaminant Class Detection
by Type of Water Quality Sensor
The assessment of water quality response to the WS Baseline Contaminants indicates that chorine residual
and TOC are potentially the most useful indicators of contamination, with the potential to detect 28 of the
33 WS Baseline Contaminants. In general, the results of these studies illustrate that free chlorine is the
most sensitive indicator of contamination, showing significant changes from baseline values at
concentrations often one to two orders of magnitude below lethal concentrations. Specifically, many
contaminants were detected at concentrations around 1 mg/L, while the corresponding lethal
concentration might range from 10 to 100 mg/L. These studies also indicate that TOC is a particularly
useful parameter for detecting the presence of many organic compounds, with a sensitivity ranging from a
few tenths of a mg/L to more than 1 mg/L, depending on baseline levels and variability. Even at the
upper end of this range, most organic contaminants should trigger a change in TOC concentration at
levels below the lethal concentration.
Other water quality parameters, although not as reliable as indicators of contamination as chlorine and
TOC, may still provide supporting information about potential contamination. Oxidation reduction
potential (ORP) should generally behave similarly to chlorine residual, and can be used to corroborate an
observed change in the chlorine residual. ORP may also serve a more prominent role in systems that use
a chloramine disinfectant residual since certain oxidation reactions can occur without consuming
chloramines. Conductivity and pH are both important to aqueous chemistry and may be useful in
understanding observed changes in other parameters, such as free chlorine residual. Studies have
generally shown that turbidity is an erratic and unreliable primary indictor of contamination; however, as
with pH and conductivity, it may be useful in understanding changes in other measured parameters.
Based on this assessment, the recommended water quality parameters for inclusion in the online
monitoring component of Water Sentinel include chlorine residual and TOC as primary indicators of
contamination, and ORP, pH, conductivity, and turbidity as secondary parameter that may help in the
interpretation of a water quality trigger. In addition, other parameters or sensors may be deployed
depending on the interests of the pilot utility. The core suite of water quality parameters that may be
evaluated under the WaterSentinel pilot are: free chlorine residual, TOC, pH, conductivity, ORP, and
turbidity. The following subsection provides guidance in the selection and design of sensors.
DRAFT-121205 30
-------
WS System Architecture
3.1.2 Selection of Water Quality Sensors
The sensors that can continuously monitor water quality in distribution systems fall into two
configurations:
In-line monitors where the equipment is tapped directly into the water main and monitors water
quality under distribution system pressure.
Online monitors where a slip stream from the water main is continuously analyzed by the
equipment. Presently, online monitors have a longer track record than inline monitors because
online monitors have been in use at source waters and water treatment plants for decades.
The different configurations may consist as single instruments or as suites of instruments including:
Sensors. These consist of water quality specific analysis that utilize membrane or electrode
specific technology and do not require additional reagents. These sensors can be purchased
individually or as a multi-probe sonde.
Analyzers. These equipment types require reagents and analytical process, performed
automatically, to analyze for the specified water quality parameter (e.g., TOC, HACK DPD
chlorine units).
Sensors on a chip. These sensors utilize chip-technology and typically measure a suite of
contaminants on a very small footprint. The more advanced chips are currently undergoing
development and/or testing and do not have the track record of online instruments. These chips
can be used in-line or online.
The sensors used in the WS-CWS should primarily be online sensors and analyzers. However, in-line
sensors and chip technologies may be considered if the products have undergone a sufficient level of
validation and demonstrated robust and reliable operation in a field setting.
Chlorine Residual Sensors and Analyzers
Free or total chlorine residual is monitored by the vast majority of water systems and procedures for
monitoring them are well established. Because many chemical and biological contaminants react with
chlorine, a significant drop in chlorine residual could indicate the presence of contaminants. The biggest
challenge associated with the use of online chlorine measurements as an indicator of contamination is the
identification of an anomaly from a variable chlorine residual baseline.
There are many methods used to measure chlorine: wet chemistry (e.g., N,N-diethyl-/>-phenylenediamine
or DPD), amperometric, and polarographic (with or without membrane), and thus chlorine instruments
come in a variety of configurations, including all listed above. Their performance and maintenance
requirements vary, and some require reagents (King et al., 2004).
Total Organic Carbon (TOC) Analyzers
Water utilities may monitor TOC for a variety of reasons, including regulatory requirements related to
enhanced coagulation and control of disinfection byproducts. As discussed in Section 3.1.1, TOC can be
a valuable element in the online monitoring component of a CWS, particularly for detecting the presence
of organic compounds (such as petrochemicals, solvents, pesticides, and growth media associated with
pathogens). TOC concentrations in distributed water are typically stable and predictable, assuming that
there is no mixing of water from different plants or wells with different TOC concentrations in the
distribution system. This should make it easier to recognize deviations from a stable baseline that could
be indicative of contamination with an organic substance.
Due to the cost and maintenance requirements of TOC instruments, they are not typically used in drinking
water distribution systems. Due to the small market, few vendors offer online TOC analyzers. However,
DRAFT-121205 31
-------
WS System Architecture
there are a small number of reliable, online instruments that should be considered for use in the WS-CWS
pilot. Some require carrier gases (that are supplied from gas cylinders) and frequent replacement of
reagents. However, at least one instrument manufacture offers an analyzer that does not require a carrier
gas and has substantially lower maintenance requirements than conventional TOC instruments.
Conductivity Sensors
Electrical conductivity is a surrogate measure for the amount of total dissolved solids (TDS) present in the
water. Inorganic anions (e.g., chloride, nitrate, sulfate, and phosphate) and cations (e.g., sodium,
magnesium, calcium, iron, and aluminum) typically increase the conductivity of the water. On the other
hand, neutral organic compounds are not good conductors, and tend to reduce conductivity. As was the
case with TOC, conductivity levels in finished water remain fairly stable throughout a distribution
system, assuming that there is no mixing of multiple sources and there are no major corrosion problems.
Sensors designed for finished drinking water are typically mid-range (100 to 2,000 uS/cm at 25 °C).
Conductivity sensors are reliable, accurate, and simple to use and maintain. If the given application site
has problems with coatings, there are non-contacting conductivity sensors that can be used. (King et al.,
2004)
pHSensors
One of the most commonly monitored parameters in a water system is pH. The distribution system pH is
typically controlled at the water treatment plant to reduce corrosion and to comply with the Lead and
Copper Rule, and thus does not show significant variations in a distribution system. Sometimes, however,
biological activity can introduce some variations in the pH of the water. Similarly, degasification (for
example, loss of carbon dioxide at tanks with a free surface) or precipitation of a solid (for example,
calcium carbonate) and other chemical, physical, and biological reactions may cause the pH of a water
sample to change appreciably soon after sample collection (Wagner et al., 2000). Depending on the
alkalinity of the water, pH changes over 0.5 pH unit should alert operators to a potential problem (Burns
et al., 2003).
The electrometric pH measurement method, using a hydrogen-ion electrode, commonly is used for
continuous pH monitoring. A correctly calibrated pH sensor can accurately measure pH to ± 0.1 pH
units; however, the sensor can be scratched, broken, or fouled easily. Because pH sensors are designed as
ion selective electrodes, they typical do not require reagents beyond routine maintenance (e.g.,
replacement of the electrolyte). Detailed instructions for the calibration and measurement of pH are
provided by instrument manufacturers (Wagner et al., 2000).
ORP sensors
Oxidation reduction (or redox) potential (ORP) instruments measure the potential required to transfer
electrons from the oxidants (i.e., reducing agents) to the reductants (i.e., oxidizing agents). This potential
is related to the relative concentrations of the oxidants and reductants in the water. In general, water
without disinfectants is considered 'neutral' relative to its ORP value. However, the addition of most
chemicals or biological contaminants changes the ORP of water. For example, chlorine, either in the
form of free chlorine or hypochlorites, is a potent oxidizing agent (reductant), changing the ORP of water.
As a result, ORP indirectly can measures the chlorine residual in water. However, there is no reliable
conversion from ORP in millivolts to chlorine concentration because pH, temperature, other reductants
and oxidants in water also affect the ORP value. In moving from an oxidizing to a reducing condition,
the ORP typically drops several hundred millivolts. ORP instruments are very sensitive in detecting
small (parts per billion) concentration changes, and thus useful as an alarm surrogate parameter. In alarm
applications, the exact trigger level is usually not critical.
DRAFT-121205 32
-------
WS System Architecture
ORP instruments are typically pH instruments operating in a millivolt mode, with the measuring electrode
being an inert metal such as platinum or gold. Sensors are typically not calibrated, and ORP standard
solutions are used primarily for verification of electrode response rather than calibration. ORP sensors
are simple and require no reagents. However, the electrodes can be damaged by some metals in the water
and can also be covered with inorganic or organic films that affect the values of ORP measured by the
instruments. This can complicate the utility of ORP as a parameter. Routine cleaning and calibration is
required to compensate for electrode degradation, and the electrode should be replaced every year or two
on a regular basis depending on the application (King et al. 2004).
Online experience is necessary at startup to establish the particular operating range and trigger level for an
application. The best practice to determine a trigger level for control is to use a test for chlorine
concentration as a reference.
Turbidity sensors
Turbidity is a measure of a sample's relative clarity, and indirectly a measure of suspended particles.
Turbidity measurements are thus useful because waterborne disease-causing organisms such as bacteria,
viruses, Giardia and Cryptosporidium often attach themselves to particles in water. Because suspended
particulates can protect attached organisms from disinfection, turbidity sometimes can be an important
surrogate parameter of contamination. Furthermore, contaminants that do not dissolve in water or that
react with carbonates in water to form precipitates will increase turbidity. Similarly, it is also possible for
some contaminants to kill and slough off biofilms in the pipes increasing turbidity (King, et al. 2004).
Turbidity instruments transmit a beam of light through a water sample in a cylindrical quartz turbidity
cell, and measure the amount of light scattered at right angles to the beam using a photoelectric sensor.
There are a number of units in which turbidity can be measured, the most common one being
Nephelometric Turbidity Units (NTU). Particle size, concentration of suspended solids as well as
dissolved solids can affect this parameter.
Turbidity instruments of different designs do not yield identical or equivalent results. As a result,
turbidity measured using instruments with different optical designs can differ by factors of two or more
for the same sample, even with identically calibrated instruments. Thus, raw data from different
instruments should not be considered directly interchangeable.
In addition to water characteristics, sensor damage due to biological growth or scratches on the optical
surface of the instrument tends to produce either a negative bias when light beams are blocked or a
positive bias if scratches increase the scatter of the sensor's light beam.
Sensor Selection
Considerations for the selection of online water quality sensors have been the topic of several recently
published reports and studies (ASCE, 2004; King et al, 2004; ISO, 2003; Hargesheimer et al, 2002
manual). Some of these documents present considerations for sensor selection as part of a CWS, whereas
most focus on traditional applications of these devices. In general, considerations for installation and
most instrument characteristics are independent of any specific application of the online sensors.
However, the performance characteristics, as listed below, may be more rigorous for the application of
online sensors in a CWS. General characteristics of online sensors that should be considered in the
selection process are described below.
Physical Characteristics
o Dimensions. Overly large assemblies may not fit in the space available, or may be
troublesome during installation or removal.
DRAFT-121205 33
-------
WS System Architecture
o Weight. If sensor will be inserted into a flowing line, the weight of the assembly should be
considered during installation, maintenance and service.
o Enclosure ratings. Instrument enclosures should, at a minimum, be designed for National
Electrical Manufacturers Association (NEMA) for indoor installation. To the extent possible,
corrosion resistant materials should be utilized for instrument enclosures, plumbing
connections and mounting back planes.
o Connection to water source. Suites of instruments should be clustered to permit use of a
single water input and single drain manifold. It is prudent to have a single sample manifold
connected to a single sample tap so that all instruments are receiving the same sample.
o Power requirements. It is preferable to have sensors that are powered from common voltage
supplies such as 110 VAC or 24 VDC. Brown-out conditions should be considered and
sensors powered by backup batteries should not take excessive power.
o Electrical isolation and connections. Ground loop currents can lead to erroneous readings, so
there should be electrical isolation between the sensors and any electrical devices that receive
their signals (recorders, SCADA systems, programmable logic controllers (PLCs), modems,
remote terminal units (RTUs), etc.) Electrical connections should be water tight and
corrosion resistant.
o Data transmission and storage. A single connection for data transmission is desirable. Both
analog and digital data transmission options are permissible, but preference should be given
to instruments and sites where digital communication is possible. Onsite storage of data
should be minimized.
o Pressure and flow ratings. Nominal line pressure rarely exceeds 100 pound per square inch
(PSI), but pressures to over 200 PSI are not uncommon. Nominal line flow rates would be 3-
5 feet/second, but sensors should be able to survive flow rates of at least 10 feet/second.
o Tolerance to flow and pressure variations. Water hammer can produce pressures far above
200 PSI with resultant damage to inline sensors. Sensor mountings are also at risk to high
pressures, with catastrophic or dangerous results. Sensor packages should include
instrumentation to monitor local pressure conditions. Both mechanical pressure reducers and
barometric loops are frequently used to provide a sample in the proper pressure range.
o Instrument materials that may be affected by oxidants. such as free chlorine. Drinking water
is not highly corrosive but can contain significant amounts of chlorine that may damage some
plastics and metals, such as steel. With chlorine residual a primary indicator in a CWS, and
ORP a secondary indicator, instrument materials should be unaffected by oxidants.
o Instrument materials that may be affected by humidity. While instrument enclosures should
be resistant to water intrusion, instrument materials should also be able to withstand exposure
to ambient humidity during routine maintenance.
Performance Characteristics
o Repeatability. Repeatability should be good if the signals are used for analysis. Compare
candidate sensors to those for process use.
o Accuracy. Required sensor accuracy should be stated before the selection process.
o Drift. Sensors may be mounted in areas where temperature varies considerably. Temperature
drift specifications for zero and span should be known.
o Signal memory. Signal memory defines how long a signal can be generated in time before
repeating the signal
o Warm-uptime. This could be significant when power to the sensors is interrupted. Sensors
that require long periods to return to normal operation should be avoided.
o Supply voltage effects. Often overlooked; sensors should be robust with respect to accuracy
under varying supply voltage.
DRAFT-121205 34
-------
WS System Architecture
o Response time. Fast response time is generally not needed in monitoring applications, but
response times greater than 2 minutes should be avoided.
o Polling frequency. This is the frequency the SCADA or the Control Room will poll remote
instruments for data, and the recommended frequency should be 2 minutes or less.
o Data backup. Individual instruments should have data backup capabilities, in case of a
communication failure, operators can then collect the data from the instrument's data logger
directly.
o Temperature range. A range of 0 to 70 degrees Centigrade should be adequate, although
temperatures would rarely exceed 50 degrees Centigrade at most sites.
o Interference from electrical equipment in the area. Sensors may be mounted in proximity to
pumps and other electrically actuated equipment. The sensors should not be susceptible to
interference from those devices.
Operational Characteristics
o Installation requirements. Typical sites will have limited room and power for installation
activities. Pre-assembly of sensor packages can speed installation, paying close attention to
limitations of the access point to the site (i.e., manholes). Dimensions, weight and the
power/water needs of the sensors should match the conditions at each site.
o Maintenance and calibration requirements. Ideally instruments should require minimal
operator intervention and service. It is reasonable to expect that the instruments should be
looked at once or twice a month, but require active maintenance only monthly.
o Technical skill level required for operation. Instruments will likely be operated by persons
with limited analytical training. Thus, to the greatest extent practical, the instruments should
be plug-and-play with simple menu-driven operation and set-up.
o Compatible cleaning methods and solutions. Sensors should be cleaned during maintenance.
Cleaning methods and solutions should be clearly identified, noting if the cleaning solution
for one type of sensor is not suitable, or incompatible with, any of the other sensors at that
location.
o Availability and quality of reagents. When required, reagents should be available from the
manufacturer, who can assure their quality.
3.1.3 Sensor Station Design
Figure 3-2 illustrates atypical configuration for a sensor station, and includes the typical elements that
are recommended for inclusion in the design of a sensor station for the WS-CWS:
Slip stream line tapped into the distribution system pipe. This line is typically 3/8 inch in
diameter and should be fitted with a valve and regulator to control the flow and pressure into the
instrument. The tap should extend to the center of the distribution system pipe to deliver a
representative stream of water to the sensor. In addition, a backflow prevention device is
recommended to prevent reverse flow through the sensor and back into the distribution system
pipe.
Strainer or filter to remove any particulates that might clog the solenoid valve.
Solenoid valve that is energized when an alarm occurs, or when an operator triggers it, and
collects a sample for further analysis.
Sample container (100 L) to hold the water sample when the solenoid valve is energized.
Smaller lines that first go to the rotometers and then go to the instruments: these could typically
be % in. in diameter. For a suite of instruments, one could go to the TOC analyzer while the
other(s) could go to the remaining instruments (or multi-parameter sensor).
Rotometers to measure the flow to each sensor.
DRAFT-121205 35
-------
WS System Architecture
Uninterrupted Power Supply (UPS). If the sensors are powered by electricity, then a 120 v AC
UPS is recommended, which will provide temporary power to the sensors for 1-2 hours if
electrical power is lost.
Sensors, individual or as a multi-probe sonde. Three tiers of sensor stations are considered: 1) the
full complement of water quality parameters: TOC, chlorine residual, conductivity, pH, ORP, and
turbidity; 2) all of the previously listed water quality parameters except TOC; and 3) online
chlorine analyzers only. Additional parameters may be considered at the discretion of the pilot
utility.
Event detection system. Some sensor platforms include an event detection system that has been
integrated with a suite of sensors which process the data at the site of the sensor station. In other
configurations, multiple sensor stations send data to a single, centralized event detection system.
Discharge lines for the sensor effluent that direct the water to a drain if necessary.
Reagents, gas cylinders, and other consumables as required.
Transmit data to
SCADA System,
or to Control
Room if coming
from an event
detection software
t
Water
Line
Tap Fitting w/Valve
UPS
TOC
Analyzer
Multi
Parameter
Analyzer
Drain
Figure 3-2. Schematic of an Example Water Quality Sensor Station
3.2 Sensor Network Design
Sensor network design is a systematic process for determining the location and number of sensor stations
deployed in a CWS. The design should directly impact two important aspects of system performance: the
time of detection and the spatial coverage of the system. The time detection relative to the start of the
incident is a function of:
1) the travel time between the point of contaminant introduction and the first downstream sensor
station;
2) the delay between sensor measurements (seconds to minutes); assuming the sensors are sensitive
enough to detect a water quality change due to a contaminant, as discussed in Section 3.1.
DRAFT-121205
36
-------
WS System Architecture
3) the time required to transfer the data to the central data acquisition system (seconds to minutes);
and
4) the time necessary to evaluate the data, along with other information, and conclude that the
unusual water quality data is a threat warning.
The most significant time delays are associated with the first and fourth steps, and the former is largely
dependant on the design of the sensor network.
The spatial coverage provided by a sensor network could be viewed from the perspective of geographical
coverage of the service area; however, as discussed in Section 2.1, the majority of scenarios (i.e.,
locations of contaminant insertion) result in relatively low consequences, and only a small fraction of the
scenarios result in the most serious consequences. Thus, maximizing spatial coverage may not provide
optimal protection for consumers. Given this observation, a better approach may be to optimize the
sensor network design to an objective that more closely relates to public health protection. Various
methods have been developed to determine optimal sensor locations, see (Berry et al, 2003, Watson et al,
2004, Ostfeld et al, 2004; Uber et al, 2004) These methods are based on optimizing the sensor network
design for a single objective from the following list:
Minimizing the number of persons exposed to a lethal or infectious dose.
Minimizing the time of detection.
Minimizing the extent of the contamination in the pipe network.
Minimizing the cost of placing sensors.
Maximizing the spatial coverage of each sensor
Of the existing methods, the optimization approach developed by Watson et al. is the most preferable
because it: (1) has been proven to find the exact optimal solution for each objective, (2) is flexible enough
to accommodate any one of the above objectives, and (3) is solvable for even very large distribution
systems. This method is described in more detail below.
Application of this method to date has shown that if a sensor network design is based on minimizing the
health impacts to the population, the design also performs well(though not optimally) in most of the other
objectives. It is also clear that with any reasonable number of sensor stations (i.e., less than 50), there
may be events that may not be detected by a sensor network; however, in general these events should tend
to have small impacts.
Typically utilities have addressed spatial coverage of the sensor networks by using intuitive methods.
When using intuitive methods the selection of sensor locations is primarily based on local site conditions
plus some system wide factors such as proximity to critical customers (e.g. schools), or water mains
serving large number of customers. Although intuitive methods are convenient, they do not place the
sensors in locations that might benefit the utilities most. Academicians have addressed this issue either
by developing optimization/mathematical programming methods (sometimes incorporating
hydraulic/water quality network models) or by using multiple simulations of hydraulic/water quality
network models (as is the case with TEVA).
In the use of the optimization methods, the objectives have typically been:
Minimizing the expected fraction of the population at risk
Minimizing the quantity of contaminated water at a concentration higher than a minimum
hazard level for a given number of sampling sites
Minimizing the detection time for a given number of sensors (or budget)
Minimizing the detection time before a certain quantity of contaminants is consumed
Minimizing the number of sensors (or budget) for a specified time interval of detection
DRAFT-121205 37
-------
WS System Architecture
The purpose of this section is to outline the process that will be used to design the online water quality
sensor network, in particular, the number and locations of water quality sensors. Water quality sensors
should be a critical component of WaterSentinel and by optimizing the number and locations of sensors at
the pilot utility, the overall performance of the CWS can be maximized. The sensor network should be
designed to support the overall goal of WaterSentinel: to detect contamination events in time to reduce the
potential public health and economic consequences.
The EPA has collaborated with researchers at Sandia National Laboratories and the University of
Cincinnati to develop software tools for determining the best locations for sensors throughout distribution
systems. The first step of this work was to formulate the sensor network design problem mathematically
as an optimization problem in which the sensor placement objective is to minimize the time of detection,
the number of people exposed, the spatial extent of contamination, and maximize the number of events
detected (Berry et al, 2003, 2005; Watson et al, 2004). Additional constraints can be added to the
optimization problem; for instance bounds can be set on the overall costs of sensors, the total number of
sensors, or the locations that are suitable for sensor placement. Algorithms and software tools were
developed to solve the optimization problem and are referred to hereafter as Sensor Placement
Optimization Tools.
The proposed approach to sensor network design is largely independent of the specific sensor
technologies that should be used. The approach includes:
1. Identifying potential sensor locations
2. Categorizing costs of locating sensors at those locations
3. Defining the objectives and constraints for sensor location
4. Using an optimization tool to select optimal sensor locations
5. Determining the number of sensors needed
6. Refining the final sensor design
Identifying Potential Sensor Locations
The main requirements for locating sensors at a particular are summarized in (ASCE, 2004) and
summarized below:
Available Utilities (water, electricity, sewer, communication )
o Water: availability of water for use at the monitoring station
o Electricity: power for monitoring equipment and communications, availability of
uninterrupted water supply or battery power if there is an interruption in electricity.
o Drain or Sewer: to dispose of any waste stream generated by the instruments
o Communications: to transmit data via phone lines, wireless, fiber optics, radio, etc.
Physical characteristics and considerations
o Space Availability: to mount the instruments and related equipment
o Mounting Scheme: the feasibility of instruments be mounted on a common backplane
o Accessibility for Maintenance: sensors should be located where access is safe and easy. Sites
requiring confined space entry or other special requirements should be avoided
o Temperature Ratings of Instruments: for both sample temperature and ambient temperature,
should be appropriate for the installation environment.
o Sunlight: Direct sunlight should be avoided as it may cause degradation of many plastic
materials and reagents.
o Humidity: condensing humidity should be avoided.
o Sample Pressure: High and low pressure, frequent pressure fluctuations, or water hammer
may adversely affect instrument performance. Pressure reducers could be used to maintain
the sample in the proper pressure range.
DRAFT-121205 38
-------
WS System Architecture
o Hydraulic Conditions: it is important that that sensor stations be located at locations where
the water in the system is well mixed and thus representative of the water in that section of
the distribution system. If turbulent flow may interfere with sensor performance, this may be
addressed through the design of the sample port and slip stream piping that delivers water to
the sensor (e.g., if entrained air causes problems with sensor performance, a bubble trap can
be installed in the slip stream).
o Physical Security: at the site of sensor station installation to guard against unauthorized
access or tampering. The site should be reasonably secure to prevent tampering with the
instrumentation, introduction of contaminants, falsification of instrument data, and disruption
of the power supply or data communications.
Most drinking water utilities can identify many locations satisfying the above requirements, such as
pumping stations, tanks, valves, or other utility-owned infrastructure. Furthermore, many additional
locations may meet the above requirements, or could be easily and inexpensively adapted for sensor
station locations. Sites owned by other utility services, such as publicly owned treatment works,
collection stations, storage facilities, etc. likely meet all the requirements for locating sensor stations. In
addition, many publicly-owned sites could be easily adapted, such as fire stations, police stations, schools,
city and/or county buildings, etc. By including these sites, the list of potential sites for sensor stations
numbers in the hundreds for the WaterSentinel pilot utility. Finally, most consumer service connections
would also have most of the requirements for sensor placement, with the exception of an existing data
transmission mechanism. There may be legal issues with locating sensors in private homes or businesses;
nevertheless, the benefit of using some of these locations may far outweigh the difficulties. An example
of a water distribution system with potential sensor locations is shown in Figure 3-3.
In addition to the above physical characteristics of potential sites, there will be other considerations that
may constrain sensor station locations, such as the normal variability of the water quality baseline. Even
with regular maintenance and calibration of the sensors, there may be some locations in the distribution
system in which the water quality is so variable that potential contamination incidents cannot be
distinguished from background variability. Therefore, the list of potential sensor locations should be
restricted to locations that are able to maintain relatively stable water quality. For example, locations near
storage tanks may have significant variability in chlorine levels as tanks cycle between draining and
filling. However, such predictable variability might be accounted for in some event detection systems.
Simulations of distribution system chlorine residual levels using hydraulic/water quality models, or
empirical data generated from field studies, can help to identify locations with stable, predictable chlorine
residuals. These points can be assumed to have low variability in other water quality parameters for the
purpose of sensor placement. This process may remove a large number of potential sensor locations, but
for the WaterSentinel pilot utility, at least one-thousand possible sensor locations should remain.
DRAFT-121205 39
-------
WS System Architecture
Legen
d hlJSpltdl'5
1 school
- tanks
* Nodes
pipes
Figure 3-3: Example Water Distribution System with Potential Sensor Locations
Categorizing Sensor Location Costs
Potential sensor locations can be divided into five categories based on the costs of installation. EPA plans
to work with the WaterSentinel pilot utility to determine which locations fall into each category:
Cost category 1: Sites that already have water quality sensors.
Cost category 2: Utility or public-owned sites that meet the site requirements listed above, except
perhaps a data transmission capability which can be inexpensive to add.
Cost category 3: Privately owned sites that meet the site requirements, except perhaps a data
transmission capability which is inexpensive to add. These sites may have an additional cost
associated with gaining access to the space.
Cost category 4: Sites that lack one or more of the following: easy access, sewer, electricity, or
physical security. These sites may be significantly more expensive to adapt for sensors.
Cost category 5: Forbidden sites. Certain locations in a network may not be appropriate sites for
water quality sensors, no matter the cost of placement. In the optimization model, these sites
DRAFT-121205
40
-------
WS System Architecture
should be assigned an infinite cost. It may be difficult for a water utility to enumerate all of the
forbidden (infinite cost) locations, but current sensor placement capabilities allow a utility to
specify some if it wishes.
Defining the objectives and constraints for sensor placement
There are many possible objectives to consider for sensor placement, including the following.
1. Minimizing the public health impacts.
2. Minimizing the time to detection.
3. Minimizing the extent of contamination.
4. Maximizing the number of events detected.
In order to measure these impacts, a set of contamination scenarios should be defined and simulated, and
the resulting objective values should be measured for each potential sensor site. The optimization
methods need to know, for example, the impact of a potential attack at location x, given that the plume
from this attack first encounters a sensor at location y. For public health impacts, this requires simulation
of the fate and transport of a contaminant in the drinking water system, assumptions about the
consumption patterns of the population, estimates of the spatial and temporal distribution of the people
that have been exposed, calculations of the number of people that become ill according to contaminant-
specific dose-response curves, and predictions of the time evolution of health impacts.
The goal of optimization software, for example, may be to select sensor locations which should minimize
the average number of people that become ill from ingestion of the contaminant, considering a large
ensemble of attack scenarios. Additional constraints could be added to this goal; for example, to require
that the worst case population affected is bounded from above by some constant, or that the average
extent of contamination is below some specified number of pipe-feet. The sensor placement software tool
is flexible enough to allow for such considerations and many more. The general plan for the
WaterSentinel pilot is presented below, though it is recognized that there should be significant
interactions with the pilot utility before a final decision is made.
The Sensor Placement Optimization Tool has been described in numerous publications (Berry et al, 2003,
2004, 2005; Watson et al, 2004). The tool can find sensor placement solutions for each of the above
objectives that have been proven to be the exact optimal solutions, (Watson et al, 2004). The tool is
flexible enough to allow for exploring the trade-offs of selecting one objective compared to another. A
future version of the tool may allow for the simultaneous optimization of several objectives. A future
version may also include additional objectives; for example, minimizing the impacts of worst case
attacks, which is profoundly more difficult to solve than the existing objectives.
Sensor Placement Methodology for the WaterSentinel Pilot
The sensor placement process for the WaterSentinel pilot should use an incremental approach, providing
a sequence of sensor layouts, the merits of which can be compared and contrasted. The process should
begin by designing the sensor network under ideal conditions using many simplifying assumptions.
Then, assumptions should be removed one by one in order to make the results more meaningful. At each
iteration, the performance of the given sensor placement should be compared quantitatively and visually
with previous steps in order to understand what has been gained or lost with each assumption. The steps
will include:
Idealized sensor placement
Determining the number of sensors by requiring upper bounds on costs and/or the total number of
sensors (see Figure 3-4)
Constraining the set of potential locations to those with low variability in water quality
Incorporating realistic response delays
DRAFT-121205 41
-------
WS System Architecture
Considering additional high consequence contaminants
Refining the sensor design based on field studies and interactions with pilot utility
Sensor Performance Curve
.EJ2 9°
10
20
30 40 50 60 70
Number of Sensors
80
90
100
Figure 3-4. Sensor Network Design Trade-Off Curve
3.3 Data Management, Analysis, and Interpretation
Considerations for the management, analysis and interpretation of data from online water quality
monitoring are described in this section. Additional information pertaining to the integration of this
information as part of the WS-CWS is discussed in Section 8.0.
3.3.1 Data Management
A data management system should be capable of delivering data from CWS sensors to a data collection
system for analysis, storage, and notification of designated responders. The overall data management
system should include the following elements:
Local Data Logging: Field located water quality instruments each generate a signal which is
directly related to the measured parameter. At each field location, each analyzer's signal should
be stored in a data logger as protection against loss of data elsewhere in the management stream.
Some instruments include built-in data logging capability. For other instruments, a dedicated
data logger should be provided. Typically, numerous signals may be stored by a single data
logger, so one unit should be sufficient for each sensor station location.
Data Concentration: Instrument signals (analog or digital signals) should also be inputted into a
data concentrator, which is a device that collects all local instrument signals and prepares them
DRAFT-121205
42
-------
WS System Architecture
for transmission to a central data management system. A data concentrator may be
programmable logic controller (PLC), a remote terminal unit (RTU) or remote input-output (RIO)
device.
A PLC will typically convert the raw signal from the analyzer to engineering units (mg/1, ppm,
pH, etc.) before passing them on to the transmitting device. An RIO device typically does not
convert the raw signals, so this should be done elsewhere in the data stream. An RTU may
convert the raw data or not, and frequently includes a data logger function, so this device may
serve two purposes in the data management architecture. Analyzers which have the capability to
deliver a digital signal may internally convert the raw signal before it is passed to the data
concentrator.
Data concentrators may perform signal evaluation tasks such as comparison of measured values
against set alarm limits to activate automated samplers, isolate or redirect water flows, or other
purposes.
Data Transmission: Water parameter measurements should be transmitted from the remote
sensing location to a central communications interface at a data warehousing and analysis
location. Common communications methods used include licensed and unlicensed radio, frame
relay, digital subscriber line (DSL), cellular telephone digital data service, and cable television
digital data service. Often several of these will be in use from different remote locations for a
single utility. Each of these transmission methods will require a communications device (radio,
modem, or other similar device) at both the remote instrument location and at the central data
management facility.
Data Processing: At the central data management facility, the measured signals should be
converted to engineering units, if not already done at the data concentrator. The signals are then
delivered to a data warehousing system. This system includes the data storage hardware and
software, and a data storage network which provides interconnection between all data storage and
retrieval computers and interfaces.
The data may initially be received by a dedicated purpose system, such as the utility's SCADA
data historian, but it may then be made available to a special purpose CWS data management
system. The special purpose system can provide services such as broad trend analysis of data
from many remote sensors, incorporation of geographical information system (GIS) data,
comparison of measurements to trends, analysis against known mitigating factors such as planned
maintenance activities, among others. While a special purpose system would be very useful for a
fully operational CWS, it may be overly complex or expensive for some utilities. In that case,
existing information systems, such as a SCADA system and associated data historian, may be
programmed to provide many of the functions of the special purpose system.
While providing the necessary functions of transporting and managing the data, each of the above
described elements introduces vulnerabilities that should be accounted for and minimized.
In order to select the specific data management elements to be used in a CWS, the following should be
considered in selecting data management elements:
Evaluate whether an existing remote monitoring and control system, SCADA, has the capacity to
provide transmission and data handling services.
o Evaluation of existing radio transmission pathways would include not only the available
capacity (available bandwidth) of the radio link, but also whether the radio link can be
established from the CWS field locations.
DRAFT-121205 43
-------
WS System Architecture
o Evaluation of availability of telephone or other communication services at the field location.
o Communication method effectiveness, reliability, and maintainability.
Determine whether the CWS data management system can interface directly with the data
warehouse at the central facility, or whether the CWS data should be routed to the SCADA data
management system, and then made available to the CWS data system.
Requirements for providing maintainability by the utility staff and minimizing cost of ownership.
Vulnerabilities introduced and methods required for minimizing and mitigating those
vulnerabilities.
Use of the SCADA and historical data collection storage and retrieval hardware and software for
CWS data management for a limited capability system.
3.3.2 Analysis and Interpretation
Since water quality sensors are only monitoring for potential indicators of contamination, rather than for
the contaminants themselves, interpretation of the water quality results is necessary to determine whether
or not the water has been potentially contaminated. Thus, the success of the online water quality
monitoring component of the WS-CWS for detecting anomalies that may be indicative of contamination
without generating an unmanageable number of false alarms depends on the performance of the water
quality event detection system. The tools and software currently available for event detection are
discussed in Event Detection for Drinking Water Contamination Warning Systems (USEPA, 2005f). The
reliability of these event detection systems can be further enhanced through integration data streams from
multiple water quality sensors as well as information from system operations and maintenance. A water
quality event detection system has not yet been selected for the WaterSentinel pilot, and it is likely that
several available systems will be evaluated over the course of the pilot, with the process for initial system
selection described in USEPA, 2005f.
Furthermore, ongoing research is developing a database of 'water quality responses' for specific
contaminants (USEPA, 2005h). These profiles should support the analysis of online water quality data
and help to distinguish alarms associated with possible contamination from other anomalies. Once the
'water quality response' for a large number of contaminants have been thoroughly tested and
documented, such information can support the characterization of an incident, credibility determination,
and response decisions.
Following the identification of a water quality anomaly, the next step in data analysis is the integration of
the water quality data with additional CWS data streams. A complete CWS system should include data
analysis and interpretation tools that integrate many data types (online water quality data, field and lab
test data, consumer call information, and public health surveillance data) to improve the overall reliability
and coverage of the system. This higher level of data integration and analysis is discussed in Section 8.0.
3.4 Framework for Evaluation
The evaluation of the online sensor network of the WS-CWS should utilize both laboratory and field-
scale studies. In general, the field-scale studies should consist of tracer tests, hydraulic and water quality
monitoring, and provide the majority of the information used to evaluate the design and performance of
the online sensor network. The following sub-sections describe the portions of the WS-CWS to be
evaluated and provide a brief description of the approach to be utilized.
3.4.1 Network Model Confidence
The sensor placement tools described in Section 3.2 rely on the accuracy of the hydraulic network model
provided by the pilot utility. The ability to evaluate the vulnerability of a distribution system and develop
DRAFT-121205 44
-------
WS System Architecture
adequate sensor locations requires a reasonable representation of the actual dynamics within the
distribution system. These dynamics, in large part, need to adequately describe transport throughout the
distribution system. To that end, there needs to be confidence that the network model represents the
actual behavior within a distribution system. To develop model confidence, tracer tests (using an
inorganic salt measured as conductivity) and hydraulic and water quality monitoring programs should be
developed to collect information regarding the dynamics within a distribution system. The data collected
should be a combination of hydraulic and water quality measures obtained from the utility's SCADA
system, grab sampling program, and continuous monitors placed at remote locations throughout the
distribution system. Bench-scale experiments should be performed to evaluate the water chemistry within
the bulk fluid (e.g., chlorine decay) to establish a baseline of decay for use with a distribution system
network model. These data, coupled with the available distribution system network model, should be
used to evaluate model confidence (Boccelli et al, 2004). Metrics for providing model confidence should
utilize model predictions and observed data to compare residence times of tracer signals, develop
correlation between signals to evaluate path mixing, and compare tracer signal distributions to evaluate
dilution effects. While individually these metrics do not provide an adequate picture of model
confidence, together they indicate the ability of the network model to represent the gross transport and
detailed mixing that occur in a largely interconnected hydraulic network.
In all likelihood, the physical scale of the actual distribution system should prohibit the evaluation of the
entire system simultaneously. Instead, individual sub-regions should be determined that, as a whole,
represent the entire distribution system yet provide a more manageable field-study. Sensors should be
placed throughout the distribution system to provide adequate coverage of the distribution system. This
'coverage' includes providing adequate spatial distribution as well as ensuring the data collected represent
the underlying distribution of, for example, hydraulic residence time and water quality variability. By
providing such coverage, the likelihood of determining the areas of the distribution system that are well or
poorly represented by the network model is increased.
3.4.2 CWS Sensor Placement Tool
The development of an adequate network model provides the first step in improving the utility of the
sensor placement tool discussed in Section 3.2. While there is much that goes into the vulnerability and
risk assessment portion of the sensor placement tool, there is little that can be done to evaluate true
optimality under real-life conditions. Instead, the field evaluations should be focused on establishing
metrics associated with different sensor network designs and evaluating potential trade-offs between
various sensor network designs. The general type of tracer tests (employed above) provide opportunities
to estimate the coverage of different sensor configurations as well as the spread of the tracer signal after
passing a sensor, which is important when evaluating the impacts of response time to a trigger event.
Additionally, smaller-scale tracer tests that simulate a potential intrusion event can be used to specifically
evaluate sensor network configurations for observing specific attack events.
The majority of this work should rely on conductivity sensors for observing the signals, which allow
coverage of the network to be better evaluated. The ability of a suite of water quality sensors to trigger a
response to an event (or non-event) is discussed in the following section.
3.4.3 Water Quality Event Detection Systems
The most important function of an event detection system is to filter out changes in water quality that
normally occur or which have known causes (e.g., changes in chlorine residual resulting from tank
cycling) and signal only those anomalies that are likely to be indicative of possible contamination
incidents. In short, the purpose of the event detection system (EDS) is to reduce the false positive rate
without missing possible contamination incidents.
DRAFT-121205 45
-------
WS System Architecture
In order to characterize available event detection systems, a three step evaluation is proposed. First,
laboratory-scale pipe-loop studies should be performed by T&E and Technology Testing and Evaluation
Program (TTEP) to test available event detection systems for correctly triggering an alarm when a
contaminant is introduced into a water stream representative of the utility's treated water. The second
step of the evaluation should use the water quality data collected from the deployed WS-CWS sensors for
the specific purpose of characterizing the false-positive rates associated with various event detection
systems. This phase of the evaluation should use data from multiple sites and different water quality
baselines to assess the effect of variability on the performance of the event detection system. The third
step in the evaluation should use a few of the deployed WS-CWS sensor stations, with modifications to
allow for the safe introduction of test contaminants. This should provide the opportunity to test the ability
of the event detection system to correctly trigger an alarm under field conditions and thus characterize the
false negative rate of the event detection system. Unlike laboratory-scale experiments, these field
experiments should provide an opportunity to evaluate the ability of the event detection systems, which
should ideally 'learn' on-line, to correctly identify contamination events in the presence of actual
background water quality variability. The modified sensor stations would need to have proper safeties in
place such as backflow prevention and a high level of physical security.
3.4.4 Sensor Stations
The initial selection of the WS-CWS online sensors should be based on evaluations of equipment from
multiple vendors by T&E and TTEP under controlled conditions in pipe loop studies. The field-scale
studies should provide the opportunity to evaluate the equipment under actual field operating conditions.
The evaluation of the equipment itself should focus on the robustness and reliability of the sensors over
time. Some of the metrics that should be considered when evaluating the sensor stations should be
calibration frequency, operation and maintenance, percentage of downtime, ease-of-use (as per utility
personnel), etc. The information from the pipe loop and field studies should be compiled and evaluated to
determine which combination of sensors and ancillary equipment provides the best overall performance,
and to select the components of the semi-permanent CW sensor stations. As new technologies become
available, this equipment should continue to be tested through T&E and TTEP prior to being evaluated in
the field.
Upon deployment, the performance of the semi-permanent CWS sensor stations should be subject to
continual evaluation using the same metrics listed previously as well as the overall costs and benefits of
operating the system. The dual-use application of these sensor stations should also be assessed. For
example monitoring and transmission of water quality data in real-time to the utility SCADA system can
serve the dual purpose of contamination warning and providing information necessary for optimizing
distribution system operations, while saving utility staff hours of sampling and testing time that would
otherwise be required to collect even a fraction of this data. The cost of maintaining online water quality
monitors in the distribution system for water security may thus be offset by the time and cost savings
from manual collection and analysis of water quality samples.
DRAFT-121205 46
-------
WS System Architecture
Section 4.0: Sampling and Analysis
Water utilities have active sampling and analysis programs to support regulatory compliance monitoring.
However, the objectives of compliance or process monitoring are significantly different from those of the
WS-CWS, which relate to the protection of public health from acute hazards. Thus, compliance and
process monitoring generally do not serve a useful function in the context of CWS implementation, with
the possible exception of daily distribution system monitoring for chlorine residual. The utilities'
experience, however, with compliance monitoring may benefit sampling and analysis for WS-CWS
activities. One principle difference between compliance monitoring and WS-CWS requirements is
related to the frequency at which samples are collected and analyzed. The precise frequency is, in turn,
based on the design of the WS-CWS. Table 2-8 summarizes the manner in which sampling and analysis
satisfies the requirements of the design basis in the WS-CWS.
The ability to rapidly detect and identify specific contaminants, or contaminant classes, in drinking water
samples is a critical component of the WS-CWS program. Sampling and analysis of water samples
collected from the distribution system are used in the WS-CWS to detect the presence of specific
contaminants (and related constituents). However, the specific application of these analytical tools should
be considered in the design of the WS-CWS. The following three applications were considered in the
design of the WS-CWS:
Routine, periodic sampling and analysis to provide an indication of contamination
Baseline sampling to establish the background levels of contaminants of concern
Sampling and analysis in response to a trigger generated from other WS-CWS components as part
of the consequence management process
The use of routine sampling and analysis as a means of initial detection of contamination was eliminated
from the design based on several considerations. The results of the contamination incident timeline
analysis demonstrate that routine sampling and analysis does not provide timely detection of the majority
of WS baseline contaminants, and provides information substantially later than the other WS-CWS
components, with the exception of some forms of public health surveillance (USEPA, 2005b).
Furthermore, the results of this analysis indicate that the sampling interval for routine sampling would
need to be in the range of 4 to 48 hours, depending on the contaminant class, to serve as a timely indicator
of contamination. This presents substantial challenges to the sustainability of the program in terms of
staffing, cost, and laboratory capacity. However, sampling and analysis does serve an important role in
the design of the WS-CWS for baseline monitoring and triggered sampling and analysis, as described
below.
The objectives of baseline and triggered sampling and analysis are closely coupled. Due to the low
specificity of other WS-CWS components in terms of their ability to identify specific contaminants or
contaminant classes, there is a critical role triggered sampling and analysis to assist in the credibility
determination process and planning specific response actions based on the threat posed by the specific
contaminant. However, the analytical results from triggered sampling should be evaluated in the context
of background or baseline levels of contaminants to accurately interpret the results and assess credibility.
In order to establish baseline levels for contaminants, it is necessary to conduct baseline monitoring at a
predetermined frequency that takes into account seasonal and system variability. The baseline monitoring
program can also be designed around specific, predefined sampling circuit.
Selection of sites for triggered sampling is situation-specific, and often times more complicated and
uncertain than setting up a baseline sampling circuit. It is likely that there would be multiple sites that
need to be sampled, (e.g., at multiple places downstream or upstream) for a control sample. Furthermore,
DRAFT-121205 47
-------
WS System Architecture
the timing of triggered sampling presents a challenge not faced during baseline sampling. As
demonstrated in the timeline analysis, it is important to minimize the time necessary to collect and
analyze triggered samples. This need for rapid-turn around should be balanced against the need to
identify sample collection sites that would likely represent the water suspected of being contaminated and
to produce reliable analytical results. By contrast, the issue of timeliness for baseline monitoring is much
less significant because these samples are not expected to contain any hazards. One of the more
important objectives of baseline sampling and analysis is accurate measurement of contaminant
concentrations.
The two types of sampling and analysis activities serve distinct roles in WS-CWS and have differing
spatial variability in sampling locations, so separate sampling plans need to be designed for baseline and
triggered sampling, as discussed in Sections 4.2.1 and 4.2.2, respectively. Namely, baseline analysis
should quantitatively determine existing levels of analytes in the distribution system whereas analysis for
triggered sampling should be performed to determine whether the concentrations of contaminants in the
area(s) targeted for monitoring differ significantly from the baseline values. The actual sampling (Section
4.1.1) and analysis (Section 4.1.2) of samples is similar, regardless of the sampling plan, because the
analytes should be the ones measurable by an 'unknowns' protocol as described in Section 4.1.2. Section
4.1 describes the sampling and analysis component of the WS-CWS, while Section 4.2 describes the
sampling circuit design. Data management and analysis is discussed in Section 4.3, while Section 4.4
provides a framework that should be used as a basis for evaluation of the overall WS-CWS sampling and
analysis program.
4.1 Sampling and Analysis
The ability to rapidly detect and identify targetedand otherwater contaminants is a critical component
of the WS-CWS program. Comprehensive sampling procedures and analytical methods are being
developed to support baseline monitoring efforts and to provide rapid detection capabilities for
unidentified radiological, chemical, and biological contaminants in response to a credible contamination
event. This includes an 'unknowns' protocol that provides coverage of the 33 WS baseline contaminants
as well as other contaminants and water quality conditions. The protocol is, in many cases, designed to
detect specific WS baseline contaminants as well as contaminant classes or surrogates. The latter
approach may be used because the direct methods may not be validated for use during the initial stages of
the WS program or because reliable screening methods can provide more timely measurement of a
broader array of contaminants. Sections 4.1.1 to 4.1.3 describe the sampling and analytical elements of
the WS-CWS.
4.1.1 Sampling
In support of WS-CWS objectives, EPA is developing the Sampling Guidance for Unknown
Contaminants in Drinking Water ('unknowns sampling protocol guidance') (USEPA, 2005e). This
document builds on the approach described in Module 3: Site Characterization and Sampling Guide of
EPA's Response Protocol Toolbox and includes a comprehensive suite of procedures for collecting
samples that may be analyzed for radiological, chemical, and biological contaminants in drinking water.
Sampling procedures are described for all WS baseline contaminants and contaminant classes, as well as
other potential contaminants (Figure 4-1).
DRAFT-121205 48
-------
WS System Architecture
and and aciotegiraj 'ie-4 scrc-e-i-rc TED'
Collect qrab samples as
2,2-1.
contain «s
at and in BSL-3
* FteM serswlng tosi*
a? uWistte e 'Jesig?: &
fTTEP) fsaults «nd sana
Figure 4-1. Overview of WS Sampling Process
The unknowns sampling guidance applies equally to both non-emergency sampling (e.g., baseline
monitoring) and sampling conducted in response to a trigger generated by other WS-CWS components.
The specific sampling procedures used for either scenario are identical, but implementation of the
procedures may differ based on the analytical objectives or scope (i.e., number and type of analyses). For
example, a subset of the sampling protocols could be used to support targeted analysis for contaminants
clearly implicated by other sources of information. The entire suite of unknowns sampling procedures
would be used in situations when no information about the nature of the suspected contaminant(s) is
available. The use of these standardized sample collection procedures for baseline monitoring should also
enable sampling teams and the analytical laboratories to practice and prepare for a triggered sampling
event using the same procedures.
Safety Screening. As depicted in Figure 4-1, prior to collection of samples for laboratory analysis, an
initial field safety screen for radiological analysis may be performed using hand-held radiation meters for
alpha, beta, and/or gamma emissions. If abnormally high levels of radioactivity are detected, the site
should be characterized as a radiological hazard, and grab samples would likely be sent to a qualified
laboratory for analysis. If the radiological screen indicates that this class of contaminant is an unlikely
source of concern, the remainder of the sampling procedure should be performed. Note that other safety
screening and site evaluation would likely be conducted as part of site characterization (USEPA, 2004a).
Sample Collection. Samples for chemical and biotoxin analyses should be collected according to specific
analyte and method requirements. It is critical that samples are collected in the appropriate containers, at
the appropriate volumes, and are preserved and/or dechlorinated as specified by the method in order to
obtain reliable analytical results. In some instances, it is desirable to not preserve, dechlorinate, or
DRAFT-121205
49
-------
WS System Architecture
otherwise alter a portion of the sample, in order to perform certain types of analysis, especially if the
sample is later sent to a specialized laboratory for analysis. The container types, required sample
volumes, and required sample preservatives for each contaminant or contaminant class are detailed in the
sampling protocol and summarized in Table 4-1, as presented in the Response Protocol Toolbox
(USEPA, 2004a).
Table 4-1. Preservation and Holding Time Table for Radiological, Chemical, and Pathogens
Contaminant
Class/Type
Radiological
Volatiles
Carbamate
Pesticides
Unknown
organics
(volatile)
Metals/
Elements
Organometallic
compounds
Toxicity
Cyanide
Quarternary
nitrogen
compounds
Semi-volatiles
Unknown
organics
(general)
Unknown
inorganics
Water quality:
Chemistry
Biotoxins
Container Volume
and Type
1 L, Plastic
40 ml, Glass w/
Teflon faced septa
40 ml, Glass w/
Teflon faced septa
40 ml, Glass w/
Teflon faced septa
125 ml, Plastic
(i.e., HPDE)
125 ml, Plastic
(i.e., HPDE)
125 ml, Glass
1 L, Plastic
1 L, Amber PVC or
silanized glass
1 L, Amber w/
Teflon-lined
screw caps
1 L, Amber Glass
1 L, Plastic
1 L, Plastic
1 L, Amber Glass
No. of
Containers
2
5
4
5
2
2
2
2
4
4
4
2
1
2
Dechlorinating
Agent
None
Ascorbic acid
Thiosulfate
None
None
None
Consult
manufacturer's
instructions
Ascorbic acid
Thiosulfate
Sodium sulfite
None
None
None
Consult
manufacturer's
instructions
Preservative
None - mark
samples not
preserved
1:1 HCLtopH<2,
see method
Potassium
dihydrogen citrate
sample pH to -3.8
None - mark
samples not
preserved
Trace metal grade
nitric acid, see
method
Nitric acid to pH <2,
see method
Consult
manufacturer's
instructions
Sodium hydroxide
to pH 12, see
method
Sulfuric acid to pH
2
6M HCI, see
method
None - mark
samples not
preserved
None - mark
samples not
preserved
None - mark
samples not
preserved
Consult
manufacturer's
instructions
Holding Time
6 months
14 days
28 days
7 days
6 months
30 days
Consult
manufacturer's
instructions
14 days
14 Days
7 days to
extraction,
28 days to
analysis
7 days to
extraction,
28 days to
analysis
28 days
Immediate to 14
days
Consult
manufacturer's
instructions
Analytical
Technique
Gross alpha, gross
beta, gamma
isotopes, specific
radionuclides
P&T - GC/MS
P&T - GC/PID/ELCD
HPLC-fluorescence
P&T - GC/MS
ICP-MS
ICP-AES
AA
AA - cold vapor
manual
AA - cold vapor
automater
Rapid toxicity assay
(several vendors)
Titri metric
Spectrophotometric
SPEHPLC-UV
SPE GC/MS
Prep: SPE, SPME,
micro LLE, direct
aqueous injection,
headspace
Analysis: GC/MS,
GC, HPLC, LC-MS
ICP-MS
Conductivity, pH,
alkalinity, hardness,
turbidity
Immunoassays
DRAFT-121205
50
-------
WS System Architecture
Contaminant
Class/Type
Water quality:
Bacteria
Biologicals
Container Volume
and Type
250 ml, Plastic
100 L
concentrated,
Plastic
No. of
Containers
1
5
(20 L
Carboys)
Dechlorinating
Agent
Sodium
Thiosulfate
Thiosulfate
Preservative
None
None
Holding Time
24-30 hrs
TBD
Analytical
Technique
Fecal coliforms,
E co//
PCR
Biological analyses should require direct grab sampling or large volume ultrafiltration sample
concentration, depending on the analytical objectives. Grab sampling (250 mL to 1 L each) should be
used when water samples are suspected to contain high levels of one or several biological contaminants
and/or when high levels of particulates are present that would preclude field concentration. Most drinking
water samples are amenable to concentration in the field or laboratory using simple membrane filtration
procedures prior to analysis (PCR- and culture-based methods) for bacterial or protozoan contaminants,
but this concentration option may not be applicable to viral contaminants. The broad 'screening'
procedure for unidentified pathogens requires that large volume samples (100 to 500 L) be concentrated
in order to obtain the level of sensitivity necessary to determine if pathogens are present in the water at
levels above the baseline (which may be zero). The ultrafiltration approach for concentrating drinking
water samples in the field ensures that most or all potential biological contaminants, including viruses, are
collected for analysis. This sampling approach involves the use of a hollow-fiber ultrafiltration device to
filter large volumes of water (e.g., 100 L) and produce a small volume retentate (e.g., 250 mL). This
retentate is then collected and separated into aliquots for field testing, and laboratory analysis using PCR-
based and/or culture-based analytical methods (see Section 4.4).
The unknowns sampling procedures guidance document should assist utilities in developing proper
sampling procedures for use in routine, baseline, or triggered sampling and should also supplement the
utility's emergency response plan to provide more detailed sampling procedures for drinking water utility
personnel during a possible contamination incident. In addition, the unknowns sampling guidance
document should provide guidance and recommendations for the assembly and training of sampling
teams, preparedness planning, establishing a support network and chain of communication, potential field
or on-site testing and screening procedures, site characterization responsibilities, and development of
information management systems.
4.1.2 Analysis
The ability to screen drinking water samples for the presence of potential contaminants when the nature or
identity of a suspected contaminant(s) is not known is a critical element of the WS-CWS. No single
analytical method is capable of detecting or identifying all potential contaminants, which may include
radiochemical, chemical, or biological agents. The unknowns sampling guidance document discussed
above is complemented by The Protocol for the Analysis of Unknown Contaminants in Drinking Water
(USEPA, 2005f). The unknowns analysis protocol builds on the concepts and analytical approaches
described in Module 4: Analytical Guide of EPA's Response Protocol Toolbox and provides a detailed
analytical approach, including methods for the detection and identification of the WS baseline
contaminants in drinking water. The analytical methods in the unknowns protocol were developed for use
by laboratories to support baseline monitoring activities in preparation for triggered sampling and analysis
performed in response to a 'possible' contamination incident when the identity of the contaminant is
unknown. However, these analytical methods may also be suitable for targeted analysis when specific
contaminants are suspected or known to be present (e.g., such as remediation and recovery efforts). An
overview of the 'unknowns' analysis protocol is presented in Figure 4-2.
DRAFT-121205
51
-------
WS System Architecture
of
DNA viruses. Vac d* VHP
Figure 4-2. Overview of WS Unknowns Protocol
Although this analytical approach is designed to detect WS baseline contaminants or contaminant classes,
as well as other contaminants, not all laboratories should have all of the instrumentation listed in the
protocol available. EPA plans to work with the WS pilot utility to prioritize in-house analytical
capabilities and identify areas for expanding this capability. This in-house capability should be
supplemented and expanded by qualified laboratories to provide comprehensive coverage for unknowns
as well as confirmatory analysis where necessary.
In addition to laboratory capability, laboratory and method performance should be evaluated on an
ongoing basis as discussed in Section 4.4 to ensure that analytical results of known and reliable quality
are generated and transmitted for use with other WS-CWS data streams. The use of these standardized
analytical procedures and guidelines during baseline monitoring activities should exercise the integrated
response capabilities of the analytical laboratory network, sampling teams, and the information
communication network of the WS-CWS and should be instrumental in preparing for a triggered
sampling event. Contaminant-specific (or surrogate agents) proficiency testing (PT) testing may also be
implemented to periodically evaluate laboratory and method performance, along with analytical accuracy.
The 'unknowns' analysis protocol also should provide guidance to the laboratories supporting WS on
preparedness planning (defining capabilities, establishing a chain of communication, information
management, and laboratory network integration), laboratory safety and containment, sample receipt and
safety-screening procedures, and sample referral procedures for when additional laboratory analysis is
DRAFT-121205
52
-------
WS System Architecture
necessary. The 'unknowns' analysis protocol should also assist these laboratories in the development of
practices and standard operating procedures during contamination events. The analyses outlined in the
unknowns analysis protocol, which should be used to support the WS-CWS pilot, are described below.
Radiological analyses. Radiological analyses should be performed only by licensed, specialty
laboratories, and the need for such analyses should be indicated by the field screening for alpha, beta, and
gamma emitters, along with any relevant information gathered during the credibility determination
process. The field screening results should determine the appropriate laboratory to receive radioactive
samples (e.g., high levels of radiation would indicate a radiation hazard, and a qualified radiation
laboratory supporting the WS pilot would receive samples).
Radionuclides should be measured for gross alpha, beta, or gamma radiation using EPA Method 900.0 or
handheld equipment. If the sample is positive for high levels of gross alpha, beta, or gamma radiation and
the laboratory is not equipped to handle radioactive samples, the samples should be sent to an appropriate
laboratory qualified to handle radioactive samples. This laboratory should perform targeted analyses to
identify the specific radionuclide(s) present. If the sample does contain radioactive material, specific
radioisotopes should be determined by the laboratory using EPA 900-series methods or similar acceptable
procedures developed in-house.
Chemical analyses. The analytical approach described in the 'unknowns' analysis protocol integrates
several analytical techniques to screen for a broad range of chemical classes. Depending on the screening
results, these analyses may serve as a 'springboard' for more complete characterization and can be used to
determine which compounds from the method target list are detected. The chemical screen consists of
two elements: (1) application of multiple analytical techniques to screen for a wide range of analytes, and
(2) analytical confirmation of tentative results. The analytical approach and analytical methods described
in the unknowns analysis protocol are designed to accomplish both objectives.
The established analytical techniques, in conjunction with standardized methods for the analysis of
contaminants in water, do not provide complete coverage for all of the WS baseline contaminants. To
address these gaps, exploratory techniques are used, which do not have standardized methods associated
with them. It is important to note that all screens are not prescriptive, and laboratories should have some
flexibility to develop an analytical approach that is consistent with their existing capabilities and
experience. Of the many analytical exploratory techniques available, those used for screening are the
techniques that show the most promise for water analysis, including those with established applications in
other media, but not yet validated for water. The exploratory analytical techniques include not only wet
chemistry and instrumental analysis, but also various types of hand-held equipment and commercially
available test kits. In the unknowns analysis protocol, analysis of contaminants is divided into chemical
classes, such as organic/inorganic, volatile/semivolatile, etc.
Confirmatory analyses should substantiate contaminant identity or quantify unidentified chemicals, and
can provide legally defensible data. Confirmatory analysis may be required in the case of a tentatively
identified chemical. In general, a positive result from a rapid field test or safety screening, performed in
the field or laboratory, should be considered tentative identification and require independent confirmation.
By contrast, chemicals identified through the application of standardized methods typically do not require
independent analytical confirmation because recommended confirmatory steps are often incorporated into
the methods themselves. In some cases, another laboratory with specialized capability may need to
perform the confirmatory analysis. When possible, confirmatory analyses should be performed using
existing standardized methods accepted for analysis of the target analyte in a water matrix.
DRAFT-121205 53
-------
WS System Architecture
Biotoxins analysis. There are hundreds of biotoxins produced by a wide variety of plants and
microorganisms. The two biotoxins included in the WS baseline contaminant list, ricin and botulinum
toxin, are protein toxins and can be detected using immunological (antibody-based) procedures.
Detection of ricin and botulinum toxins using PCR-based methods that target the genes encoding these
proteins are available; however, this is an indirect approach and these assays do not measure the actual
toxins. Several commercial immunoassay formats have been identified for inclusion in the 'unknowns'
analysis protocol. The detection of ricin and botulinum toxin using current immunoassay procedures may
require sample concentration to enable detection of these contaminants at concentrations that would pose
a threat to humans. Sample concentration using ultrafiltration, as summarized above, and described in the
unknowns sampling protocol guidance, is likely to achieve this sensitivity due to the relatively large sizes
of both ricin (65 kilodalton) and botulinum toxin (150 kilodalton) compared to the molecular weight cut-
off (60 kilodalton) of the hollow fiber filtration device. This concentration procedure should be evaluated
as part of WS method development activities. The ability of direct analysis of grab samples to provide
acceptable detection limits should also be evaluated.
The non-protein biotoxins may be considered as organic chemicals, albeit complex in structure, and the
same types of sample preparation and instrumental analysis techniques may be applicable, depending on
the chemical properties of the specific biotoxin. Low molecular weight biotoxins may be treated much
like other organic chemicals and may be analyzed by the same type of analytical techniques (e.g.,
GC/MS). Because most biotoxins tend to be water soluble, LC techniques have been used for the
detection of biotoxins in water. When LC/MS is used, the same precautions may be necessary as those
for other toxic organic chemicals. The analysis of biotoxins is one area where LC/MS has proved
particularly valuable, especially if the molecular weight of the biotoxin precludes its analysis by GC/MS.
Analytical methods for additional biotoxins should be included in the 'unknowns' analysis protocol as
detection capabilities are expanded under the WS-CWS program.
Pathogens. Analytical methods for pathogen detection and identification rely on unique properties of a
specific biological agent or family of agents. Both culture-based and molecular-based (nucleic acid or
protein) detection methods are included in the unknowns analysis protocol to take advantage of the
strengths of each of these analytical techniques. Currently, EPA is working with the US Army's
Edgewood Chemical and Biological Center (ECBC) to evaluate both culture-based and PCR-based
methods for five of the six biological WS baseline contaminants. The methods are currently being
optimized and standardized, and a single laboratory validation study is expected to be conducted in 2006.
EPA is also evaluating availability of commercial PCR-based methods for these same five bacterial
contaminants and may include these methods in the single laboratory validation study being planned for
2006. Molecular (e.g., PCR or reverse transcriptase PCR) based methods provide no information on the
public health significance of the detection of genetic material of a particular pathogen in an environmental
sample.
Because current molecular assays do not determine viability, it is necessary to attempt to determine if
viable organisms are present in a recovered water sample through the use of culture techniques. This
process should be initiated as soon as possible, as many of these organisms may be fragile, and may have
been damaged during the sample collection process. Furthermore, the preparation of samples for
molecular analysis is a destructive process, and culture-based assays should need to be undertaken to
enable the collection and preservation of potentially significant trace numbers of microbial contaminants.
The information from this activity is significant to public health response, for example, testing for
antibiotic and vaccine susceptibility, toxin production, and forensic analysis. Culture-based analytical
methods provide a sensitive means for detecting and enumerating viable bacterial pathogens in water and
are considered the 'gold standard' for water quality monitoring. However, these methods may require
several days (or longer) for growth and subsequent characterization of the target analyte. For some
DRAFT-121205 54
-------
WS System Architecture
pathogens, particularly viruses and protozoa, may be difficult or impossible to 'grow' or replicate in
culture and identification of these agents requires the use of genetic and/or immunological techniques.
Molecular-based analytical methods for pathogen detection and identification, particularly PCR-based
assays, are faster and yield results in hours rather than days, but they do not directly address pathogen
viability or infectivity. However, the speed, sensitivity, and specificity of these assays provide the
potential for rapid detection of biological agents during the evaluation of a 'possible' or 'credible'
contamination threat. However, PCR analyses provide assay results that are often difficult or impossible
to interpret when used in environmental samples. Often closely related organisms or species that have not
been previously identified may be identified by the application of molecular testing techniques to novel
environmental matrices. Every large scale application of these techniques to date has essentially
generated information of this type, and interpretation in the absence of cultured bacteria is problematic at
best, and impossible in many situations. Recent publicly reported examples of the detection of
Francisella tularensis in several cities in the BioWatch program point out problems with sole reliance on
molecular techniques for identification of threat agents in environmental samples (ProMed, 2005).
Within the WS program there is the opportunity to correct these problems by initiating practices that are
designed to maximize the opportunities of using 'gold standard' recovery techniques along with more
rapid molecular assays. This creates the unique opportunity to maximize the potential benefits of this
program in a reasonable and economic manner, while minimizing the potential for high consequence
mistakes that might arise from reliance on a single analytical technique.
4.1.3 Laboratory Support Network
As part of the WS pilot, laboratories should be identified to support the pilot utility for baseline
monitoring as well as triggered sampling and analysis. These laboratories should have the necessary
capability and capacity to support analysis of samples generated through WS-CWS sampling and analysis
activities and may include a combination of state, Regional, federal, and/or commercial laboratories.
Considerations for participation in this laboratory support network include, but are not limited to the
following:
Demonstrated laboratory capability for the analysis of WS baseline contaminants in drinking
water
Demonstrated laboratory capacity to support baseline and/or triggered sampling and analysis
Proximity to the utility laboratory
Certifications and/or accreditations for the analysis of drinking water samples
Membership in existing laboratory networks (e.g., LRN)
EPA recognizes that there should be a sustained investment in the laboratory resources required to
support the WS-CWS. This investment also develops a foundation for a laboratory network to conduct
additional studies to promote the dual-use benefits of the WS-CWS (e.g., microbial analyses for agents
that may cause taste and odor, or corrosion problems). The resources should also be available for
additional environmental studies of other related matrices such as surface and recreational waters as it is
likely that these resources should need a sustained investment to maintain their technical capability for the
WS-CWS, particularly with evolving technological solutions to detection of pathogens.
In response to HSPD 9, and by its authority under section 300i-3 of the Safe Drinking Water Act (42 USC
section 1434), EPA intends to build upon and expand current 'integrated laboratory networks' to provide
analytical support to enhanced monitoring and surveillance activities. The WS-CWS pilot provides EPA
with an opportunity to begin to build a laboratory network for the analysis of drinking water samples that
can be expanded to support other environmental matrices to address the larger capability and capacity
issues resulting from increased homeland security sampling and analysis activities. The laboratory
DRAFT-121205 55
-------
WS System Architecture
support network for the WS-CWS pilot should provide an opportunity to test the concept of the Water
Laboratory Alliance (WLA) in a real-world application. The WLA should be a network of laboratories
with extensive capability for the analysis of water samples for a wide range of potential contaminants.
The WLA should integrate water quality laboratories with the existing LRN, established by Centers for
Disease Control and Prevention (CDC) and with EPA's new environmental LRN (eLRN). To parallel the
organizational structure of the LRN as established by CDC, the WLA should consist of three tiers of
laboratories including the following:
Sentinel laboratories. Sentinel laboratories should be responsible for baseline monitoring
through application of the unknowns analysis protocol at a determined frequency for baseline
monitoring and analysis of samples in response to a trigger, depending on laboratory-specific
capabilities. For triggered samples, Sentinel laboratories should provide preliminary analyses and
should 'rule out' contaminants or refer the sample to a Confirmatory or Reference laboratory for
further analysis, as appropriate. For pathogen monitoring, the Sentinel laboratory should have, at
a minimum, Biosafety Level 2 (BSL-2) capabilities for work involving agents of moderate
potential hazard to personnel and the environment. Although some of the WS baseline pathogen
contaminants can be responsibly analyzed in a BSL-2 laboratory; others, if present in the water
samples, require a more stringent biosafety level and upon detection or other evidence, the
Sentinel laboratory would transfer such samples to the next level laboratory. For certain
pathogens, a Sentinel laboratory may be capable of initiating specific culture protocols, which, if
preliminary molecular indications dictate may be transferred to laboratories with appropriate
biosafety level facilities.
Confirmatory laboratories. Confirmatory laboratories should be responsible for detection and
confirmatory identification of pathogens, toxins, chemical contaminants, and/or radiological
contaminants in referred samples using rapid, advanced technology and specialized methods,
assays, reagents, and support services. Confirmatory labs also are responsible for communication
and coordination with Sentinel labs, including providing training on sample collection,
presumptive analyses, and sample transfer chain of custody procedures. Additionally,
Confirmatory labs may provide guidance and technical assistance to other Confirmatory labs that
encounter difficulties with certain analytical methods. Confirmatory laboratories for pathogen
analysis, which would include the State public health laboratory, would be BioSafety Level 3
(BSL-3). In BSL-3 facilities, all procedures involving the manipulation of infectious materials are
conducted within biological safety cabinets or other physical containment devices, or by
personnel wearing appropriate personal protective clothing and equipment. Moreover, BSL-3
laboratories have special engineering and design features, such as double-door access zone and
sealed penetrations. Additional requirements for confirmatory laboratories, depending on their
specialties should include limited surety capability (i.e., able to handle dilute solutions of
Schedule 1 Chemical Warfare Agents) and/or the ability to analyze radioactive samples. Also,
pathogen confirmatory laboratories should comply with the Select Agent act, and chemical
Confirmatory laboratories for chemical warfare agents should comply with Army Regulation 50-6.
Reference laboratories. Reference laboratories should have highly specialized containment
facilities, specialized analytical capabilities, and specially trained staff. They should be
responsible for definitive identification and/or characterization of chemical structures, pathogens,
along with chimeras and engineered organisms. Reference laboratories should have primary
responsibility for forensics analysis, including attribution, and thus should meet standards of legal
defensibility. However, Sentinel and Confirmatory laboratories should be aware that their results
may be subjected to legal scrutiny as well.
DRAFT-121205 56
-------
WS System Architecture
4.2 Sampling Circuit Design
The WS-CWS sampling circuit design should address the sample collection location, the sample
collection schedule, and the number of samples to collect. Approaches for consideration in the sampling
circuit design for both baseline and triggered sampling are discussed below. In addition, the TEVA tool
and approach for sensor network design discussed in Section 3.0 can also be used to identify sampling
locations as part of the baseline monitoring program. For triggered sampling, the utility's hydraulic
model, if appropriately calibrated, can be used as a tool to identify triggered sample locations under
certain scenarios.
4.2.1 Baseline Sampling
In deploying the sampling and analysis component of the WS-CWS, the initial phase should focus on
establishing the baseline levels of the constituents measured by the 'unknowns' protocol. Establishing the
baseline for each contaminant is critical to distinguish naturally (i.e., those not resulting from intentional
contamination) occurring levels of the contaminant from higher levels observed during triggered sampling
that might be indicative of contamination. For many of the contaminants, the baseline is expected to be
below the minimum detection level, typically because the contaminant should not be present in the
distribution system at any level. Because triggered samples might need to be collected and analyzed at
any location in a distribution system and at any time, the baseline should capture the spatial and temporal
(including seasonal) variability of unknown protocol results. Given this objective, the primary focus of
the baseline sampling plan design is on the selection of sampling locations, sampling schedule, and the
number of samples that capture this variability. Further, it should address the variability in the
measurements made with the unknowns protocol to ensure that the threshold levels established during
baseline sampling are reliable for evaluating levels measured following trigger monitoring.
Sample collection location. In selecting locations at which to collect baseline samples, it should be
important to consider the locations of the on-line monitoring stations, because anomalies detected at these
stations may trigger sampling events in response to 'possible' contamination threats and incidents.
Therefore, it may be important to know the background in their vicinity, i.e., hydraulically related areas.
In addition, selection of candidate sample collection locations for baseline monitoring may be aided by
consideration of distribution system locations identified through existing sampling plans that are used at
the utility. These established sampling location plans include those which are (or would have been) used
for compliance related sampling such as the Total Coliform Rule and the Unregulated Contaminant
Monitoring Rule. Other potential locations include those used for monitoring water treatment plant
operational performance, including disinfection byproduct and lead and copper sampling. These existing
sampling plans should be evaluated to identify locations that represent the full spectrum of water quality
conditions and variability. The following areas reflect the scope of distribution system extremes and
should be considered in designing the sampling circuit:
Areas pre-positioned to be sampling locations in response to triggered events. If a CWS
online monitoring station detects an anomaly that triggers site characterization and sampling, time
should pass before the sample can be collected. Samples should be collected in locations that
bear a spatio-temporal relationship to the sensor station. This may be addressed to some extent
through the use of auto-samplers as a component of the monitoring station, as discussed in
Section 3.1.3. Thus, baseline samples should also be collected from these locations.
Historically high consumer complaint areas. These are localized areas within the distribution
system from which the utility receives consumer complaints associated with poor water quality.
These areas may contain differing levels of some measured contaminants and should be
accounted for when establishing a baseline.
DRAFT-121205 57
-------
WS System Architecture
Locations immediately downstream of pumping stations. Changing of flow and pressure may
cause disturbance within the distribution system mains and alter the water quality downstream of
the pumping station.
Areas of differing water ages. Represents water quality at locations throughout the distribution
system at different distances from the water treatment plant.
Disinfectant booster stations. Addition of a disinfectant (e.g., chlorine) to a distribution system
due to low disinfectant concentration residual values. Areas located upstream and downstream of
the booster station should potentially have differing water qualities.
Water storage tanks. Tanks have differing rates of water turnover, based on supply from the
water treatment plant and consumer demand within the distribution system. These differences
may result in different levels of measured contaminants and other constituents, compared to the
rest of the distribution system.
Cross-connection hazards. A cross-connection is an unprotected actual or potential connection
between a potable water system and a source of contamination (such as wastewater, industrial
fluids, pesticides, etc), where backflow can occur from the source of contamination into the
potable water distribution system. These areas may contain higher levels of some measured
contaminants and other constituents.
Locations of high/low pressure. In particular, areas that are prone to low pressure, or wide
pressure fluctuations, can present an opportunity for infiltration, and thus are candidates for
baseline monitoring sites.
Areas of deteriorating water mains. The age of the pipes, coupled with corrosion and sediment
accumulation over the years, should affect the flow rate and quality of water in distribution
systems.
Areas of stagnation due to low water use. In addition to causing odor problems and pressure
degradation, these areas should likely contain different levels of some measured contaminants
and other constituents in comparison to other areas in the distribution system.
These factors should be used to identify a cost-effective, yet comprehensive, sampling plan that should
capture the distribution system variability as a whole, allowing for an accurate characterization of
baseline levels, that should be necessary to properly interpret the results of triggered sampling and
analysis.
Sampling schedule. Baseline monitoring should establish the levels against which future routine or
triggered monitoring should be compared, so it is important that not only the sampling locations, but the
sampling scheduling reflect the extremes within the distribution system. Temporal variation factors that
should be considered during design of the sampling circuit for baseline sampling include the following:
Seasonal biological changes. Variation in the seasons can produce fluctuations in the
concentrations of waterborne organisms in the source water, which typically increase during
spring runoff and during the warmer months of the year.
Input changes. Storm water or snowmelt can cause runoff from agricultural centers that can
introduce pesticides, organisms, and other substances into the water supply. Heavy rain directly
into surface water reservoirs also can dilute contaminant concentrations to unnaturally low levels.
Source water changes. During high-demand periods, it may be necessary to draw from
groundwater to supplement the surface water supplies. This would lower the concentrations of
biological organisms that are normally introduced to the open surface water supplies, but could
also increase the mineral and chemical concentrations in the system.
Treatment changes. A change in treatment techniques or chemicals used by the public water
system should also affect the levels of parameters measured by the 'unknowns' analysis protocol.
Periodic changes to chemical treatment or the introduction of new chemicals and processes
should change the makeup of the water and should be reflected in the baseline sampling plan.
DRAFT-121205 58
-------
WS System Architecture
Distribution system hydraulics. Many factors can influence the system hydraulics and in turn
impact water quality in the system. Certain times of the day can produce a greater demand over
different legs of the system, flushing them and, thus, providing a different quality of water than
when the water has been stagnant for some time.
To encompass the variations associated with these factors, initial baseline sampling should be performed
over a period of time that should capture this variability, most likely over the course of a year. It is
important to note that the baseline thresholds produced from the initial sampling should not remain static.
Ongoing sampling should be incorporated into the threshold calculations to provide a continually
changing threshold for comparison to levels measured during triggered sampling.
Number of samples. The number of samples that should be collected to establish the baseline level for
each contaminant for the WS pilot is driven by the number of sampling locations, the sampling schedule,
and the measurement variability of the unknowns protocol. The smaller the systemic variability in the
measurement technique, the fewer number of data points should be required to address the last factor.
After the number of samples has been established, the sample load should be compared to laboratory
capacity and field sample collection logistics and limitations, and a workable sample number should be
determined.
4.2.2 Triggered Sampling
Analysis of triggered sample is an important part of the process of credibility determination and
consequence management in WS-CWS (USEPA, 2005i). In general, it should be necessary to develop a
unique sampling plan for triggered monitoring in response to a 'possible' contamination incident. The
purpose of sampling in this case should be to determine whether the results from triggered monitoring
differ significantly from the baseline in the areas of the distribution system suspected of being
contaminated.
The sampling plan for triggered monitoring should not only reflect the need to reliably detect a
contaminant in the distribution system, but for contaminants for which the baseline is greater than zero,
determine whether the level detected is significantly greater than expected. As with baseline monitoring,
the sampling plan for triggered monitoring should address the location of collection, the schedule for
collection, and the number of samples to collect. It is important to remember that each triggered sampling
event should be fundamentally different and should be based on the specifics of the trigger. Therefore,
triggered sampling plans should be rapidly developed for each situation. A generic triggered sampling
plan should be developed to assist in this process. Some considerations regarding the design of this
sampling plan is discussed below:
Sample collection location. If information from other CWS components involved in the trigger is specific
enough to narrow the focus of the monitoring to a specific area, samples may be collected from sites
downstream and upstream of the location of the trigger (e.g., the location of a sensor station that detected
a water quality anomaly). The analytical results of these samples should drive subsequent sampling
efforts. If no additional information is available, samples may be collected from select locations from the
baseline monitoring plan.
Sampling schedule. Unlike baseline monitoring, there is no design for timing of sample collection for
triggered monitoring. Samples should be collected as rapidly as possible after the decision is made to do
so.
Number of samples. Each triggered monitoring sample should to be compared to the individually that
most closely reflects the temporal and spatial conditions of the possible contamination incident, rather
than based on some summary statistic, e.g., the mean of all monitoring samples. Accordingly, the choice
DRAFT-121205 59
-------
WS System Architecture
of the number of samples is not based on achieving an appropriate level of statistical power. Instead, the
number of samples should be sufficient to adequately represent the size and level of variability of the
distribution system area being investigated, which is suspected of being contaminated. This should vary
based on the specifics of the situation, such as how far the contaminant might have spread and pressure
zones or tanks that may define the bounds of the potential spread. Hydraulic models can be valuable tools
in deciding on both the number and the location for collection of triggered samples.
4.3 Data Management, Analysis, and Interpretation
Sections 4.3.1 and 4.3.2 provide considerations for management, analysis, and interpretation of data from
the sampling and analysis component of the WS-CWS.
4.3.1 Data Management
A data management system should be capable of delivering data from the field to the laboratory and the
analytical results from the laboratory along with the field data to a data collection system for analysis,
storage and notification of designated responders and managers. The overall data management system
should include the following elements:
1. Source: Pre-determined sampling locations on a routine basis and analytical results for pre-
determined sampling locations on a routine basis; or sampling locations and analytical results in
response to a trigger.
2. Collection: Sample number, field sample data (i.e., pH, temperature, etc.), any associated field
duplicates, field blanks, or equipment blanks, and analytical results for WS baseline
contaminants.
3. Storage: For sampling, either with a personal digital assistant (PDA) or laptop with sample
collection information from each predetermined location or manually entering sample collection
information from the Chain of Custody and field notes into software data entry system. For
laboratory analysis, analytical results stored in either a Laboratory Information Management
System (LIMS) or an analytical laboratory database. The analytical results would either be taken
directly from the instrument for chemicals or manually entered into the data storage system for
pathogens and radiologicals.
4. Transmission: The field sampling data and the analytical results need to be transmitted to a
central communication interface at a data warehousing and analysis location. Common
communication methods used include manual and automated software data entry systems.
In order to select the specific data management elements to be used in a CWS, the following should be
considered:
1. Evaluate whether the existing means of collecting field sampling data has the capacity to transmit
the data to either the laboratory or the central data management facility.
a. Evaluation of using PDAs in the field
b. Evaluation of using laptops in the field
c. Evaluation of manually entering field data into a data storage system
d. Communication method effectiveness, reliability, and maintainability
2. Evaluate whether the existing means of storing the analytical results data has the capacity to
transmit the analytical data to the central data management facility.
a. Evaluation of the existing laboratory database or existing LIMS
b. Evaluation of manually entering pathogen and radiological data into the existing
laboratory database or existing LIMS
c. Communication method effectiveness, reliability, and maintainability
DRAFT-121205 60
-------
WS System Architecture
3. Determine whether the CWS data management system can interface directly with the data
warehouse at the central facility or if it should be routed to a data management system, and then
made available to the CWS data system.
4. Requirements for providing maintainability by the utility and laboratory staff and minimizing cost
of ownership.
5. Vulnerabilities introduced and methods required for minimizing and mitigating those
vulnerabilities. See Section 8.0 for additional requirements in determining and addressing data
management vulnerabilities.
6. Use of historical data collection storage and retrieval hardware and software for CWS data
management for a limited capability system.
4.3.2 Data Analysis and Interpretation
The question that the data analysis should need to answer in response to the WS-CWS is whether or not a
contamination incident (either accidental or intentional) has occurred. This question should be addressed
through evaluation of the data from all the components of the CWS, including the results of baseline and
triggered sampling and analysis..
The field and analytical baseline data should need to be integrated with online water quality monitoring
data, consumer complaints data, and public health surveillance data. The field and analytical data can be
made available to a special purpose CWS data management system, which might provide services such
as:
broad trend analysis of data from routine baseline monitoring results
analysis of data from triggered sampling results
comparison of field sampling data from multiple sampling locations
comparison of baseline monitoring results from multiple sampling locations
comparison of triggered sampling results to routine monitoring results that correspond to the
temporal and spatial conditions under which the triggered sample was collected
establishing control limits for each field and analytical baseline sampling location
establishing control limits around the variable baseline by taking into account temporal and/or
spatial trends in the data.
Also, by comparing the triggered sampling results from multiple sampling locations with the baseline
monitoring results that have been collected over time, it may be possible to determine what foreign
contaminant(s) have been introduced into the drinking water and the appropriate response actions to take.
When interpreting the data, the data user needs to consider the following information regarding the data
from either baseline monitoring or triggered sampling:
The quality control (QC) results associated with each analytical method and any potential bias
the analytical results
Common interferences for each analytical method and how this could affect the results
False positive and false negative rates of each analytical method
In addition to the circumstances identified above, the following should also be considered when
interpreting the data:
Radiologicals: If either gross alpha/beta or gross gamma levels are detected above background levels and
there is other corroborating evidence to suggest contamination, the sample should be referred to a
Confirmatory laboratory that can handle radioactive samples. The Confirmatory laboratory would
perform targeted analyses to identify the specific radionuclides present in the sample.
DRAFT-121205 61
-------
WS System Architecture
Biologicals: Samples would be sent to a Confirmatory Laboratory for further analysis under one of the
following conditions:
The initial PCR assay gives a reactive result, and the more specific PCR assay gives a reactive
result.
The culture-based result is positive.
The initial PCR assay (single loci) gives a reactive result, there is no presumptive result for one of
the five biologicals, but there is other corroborating evidence.
Chemicals: Samples would be sent to a Confirmatory Laboratory for further analysis under one of the
following conditions:
It is suspected that the sample contains any chemical warfare agent. This sample would be sent to
a Confirmatory laboratory that has surety capability for analysis.
An analyte is tentatively identified but the Sentinel laboratory does not have the analytical
capability to perform a positive identification and quantification.
4.4 Framework for Evaluation
Section 2.4.1 discusses that reliability encompasses two aspectssystem operation and system
performance. System operation requires little discussion in the context of sampling and analysis because
it relates to issues utilities and others are familiar with: equipment maintenance, downtime, availability of
supplies and reagents, etc.
System performance related to sampling and analysis is a more complicated subject. As discussed in
Section 2.5.2, it is related to the ability to characterize and correctly interpret the all data streams, not just
sampling and analysis. A significant part of interpretation is the false positive rate, which is highly
dependent on the precise sampling and analysis method employed. For established methods (such as
EPA drinking water methods), this false positive rate can be low for the concentration ranges potentially
associated with intentional contamination incidents.
The results of baseline sampling should be used to support the proper interpretation of the results from
triggered sampling and analysis. In turn, the overall reliability of the WS-CWS results should depend on
the method that this data stream is integrated with the others (see Section 2.4.1). In this data fusion
process, consideration should be given to the probability of detecting the contaminant through sampling
and analysis. A broad context of the reliability of sampling and analysis in WS-CWS is related to
consequence management activities that should accompany data collection and interpretation.
Baseline sampling and analysis lends itself to dual benefits because the frequency of sampling and the
data obtained may also help improve utility operations and routine water quality. Regardless of the cost,
baseline sampling is vital to the application of the results from triggered sampling and analysis the
credibility determination process. Baseline monitoring provides additional value to utilities and
laboratories involved in implementation of contamination warning systems by providing an opportunity
to practice and improve proficiency in sampling and analysis activities that should be necessary for
response and triggered sampling and analysis.
In assessing the costs of triggered sampling, it is important to remember that the WS-CWS as a whole
would probably not be vital, let alone sustainable, without a well developed and exercised consequence
management plan. Triggered sampling as part of site characterization activities would likely be an
important part of such a plan. Accordingly, the driving issue of sustainability of WS-CWS depends, in
part, on sampling and analysis program that is sustainable.
DRAFT-121205 62
-------
WS System Architecture
The overall goal of the WS sampling and analysis program is to accurately measure what is present in the
drinking water. To evaluate that the sampling and analysis program is accurately measuring what is
present in the drinking water, the evaluation program should consist of the following items:
Field and lab assessments to verify that organizations are implementing key practices required for
the program
Evaluation of QC samples to identify errors in specific components of the sampling and analysis
program
Proficiency testing to evaluate each laboratory's ability to correctly characterize contaminants in
a sample
Tabletop exercises and real-time water contamination drills.
4.4.1 Sampling Locations
The development of an adequate network model is an important tool in the selection of suitable sampling
location, for both baseline and triggered sampling. While there is much that goes into the vulnerability
and risk assessment portion of the various sampling locations, there is little that can be done to evaluate
true optimality. Instead, sampling locations should be evaluated based on the potential trade-offs between
various sampling locations. Through application of the TEVA methodology, the distribution system
model should provide opportunities to estimate the coverage of different sampling locations as well as the
potential spread of a contaminant, which is important when evaluating the impacts of response time to a
triggered event.
Sampling locations should be placed throughout the distribution system to provide adequate coverage of
the distribution system. This 'coverage' includes providing adequate spatial distribution as well as
ensuring the data collected from the various sampling locations represents the entire distribution system.
By providing such coverage, the likelihood of determining the areas of the distribution that are well or
poorly represented by the network model is increased.
4.4.2 Assessments
Sampling Assessments: EPA plans to assess the WS sampling program before the pilot program begins
to determine if the practices in place are sufficient and meet the needs of the WS-CWS and to verify that
the sampling practices are being properly implemented for WS. The assessment should include: (1) a
review of the utility's documentation (e.g., standard operating plans (SOPs), training records), (2)
observations of sampling personnel collecting samples for 'unknowns' analysis to verify that proper
procedures are being followed, and (3) interviews with utility staff, and other potential responders, to
ensure they understand their roles and responsibilities both for baseline monitoring and triggered
sampling and analysis. If the field sampling assessment identifies a number of errors (such as the utility
not having sufficient documentation of their sampling procedures), EPA anticipates conducting additional
training as necessary.
EPA plans to occasionally conduct follow-up assessments of the pilot utility's sampling program for WS.
The frequency of these assessments should be based on need, as indicated by results of the pre-pilot
assessment and results of field QC samples collected during the pilot program. Prior to conducting the
first assessment, EPA aims to develop a field checklist to facilitate the assessments. This checklist should
be revised as needed during the course of the pilot study to streamline or improve the effectiveness of the
assessments.
To ensure the effectiveness of all field assessments, EPA plans to provide the pilot utility with a debrief
immediately following the assessment and with a written report following completion of the assessment.
DRAFT-121205 63
-------
WS System Architecture
The field assessment results should be used to identify and assess areas in which additional attention or
training is needed to sample correctly for either baseline monitoring or triggered sampling for a water
contamination threat or incident.
Lab Assessments. EPA plans to assess the utility laboratory, or other laboratories that are identified as
the Sentinel and Confirmatory laboratories for the WS-CWS, before the sampling and analysis pilot
program begins to determine that the laboratory is following the appropriate techniques for either the
'unknowns' protocol and/or the analytical methods that should be used to analyze samples for the WS-
CWS. EPA anticipates using the existing Drinking Water Laboratory Certification Program to verify that
basic drinking water laboratory qualifications are met for chemical, biological, and radiological analyses
at Sentinel and Confirmatory laboratories and the LRN program to verify that Confirmatory methods
meet the current requirements established for the biological analyses for this program. The Drinking
Water Laboratory Certification program includes routine on-site assessments of participating laboratories
and EPA plans to supplement these existing programs with on-site assessments to evaluate laboratory
facilities, QA practices, personnel qualifications, and performance for specific contaminants and methods
that are required for the WS but not routinely used in either the LRN or traditional Drinking Water
program. For example, Sentinel laboratories may use PCR methods for WS, which may not be covered
under the other programs, and would thus need to be evaluated under WS to ensure proper sample flow
and containment. Also, the ability of Confirmatory laboratories to acceptably process drinking water
samples should be evaluated to ensure reliable data quality for environmental matrices and sample
handling practice for dangerous agents.
EPA plans to develop a suite of laboratory assessment checklists designed to address unique aspects of
the WS program that are not addressed in assessments conducted as part of the LRN or Drinking Water
Certification program. The checklists also should address the laboratory's ability to support both the
baseline and triggered monitoring program. Separate checklists could be developed to address each major
lab activity or analytical method that should be used in the WS program, including application of the
'unknowns' analysis protocol. These checklists should be revised as needed during the course of the pilot
to streamline or improve the effectiveness of the assessments.
EPA may occasionally conduct follow-up assessments of the WS Sentinel laboratories. The frequency of
these assessments should be based on need, as indicated by results of the pre-pilot assessment, results of
proficiency testing (PT) samples, and evaluation of laboratory QC samples collected during the pilot
program. To ensure the effectiveness of all laboratory assessments, EPA plans to provide the audited
laboratories with a debrief immediately following the assessment and with a written report following
completion of the assessment. The laboratory assessment results should be used to identify and assess
areas in which additional attention or training is needed.
4.4.3 Evaluation of Field and Laboratory QC Data
For baseline monitoring, during the WS pilot program, a suite of field and laboratory QC samples should
be employed, such as field and laboratory blanks, field and laboratory duplicates, equipment blanks,
ongoing precision and recovery samples, and matrix spikes. These QC samples provide information
about the precision and bias associated with various components of the sampling and analytical process.
EPA and the utility would review the results of these QC samples to determine if results are acceptable.
Results that are outside of established limits should be examined in the context of other factors to identify
the potential cause and appropriate corrective actions. For example, if results of field duplicates do not
agree within established limits, but results of lab duplicates do agree, EPA may reassess the sampling
team to determine that the samplers are following the utility standard operating plans (SOPs) and identify
any additional training needs. If laboratory QC data suggest a consistent, unexpected bias in a particular
type of analyses, EPA plans to work with the laboratory to identify and correct the source of that bias.
DRAFT-121205 64
-------
WS System Architecture
4.4.4 Proficiency Testing (PT) Program
Any laboratory that is supporting the WS pilot program should be expected to participate in Proficiency
Testing (PT). PT samples provide a means of evaluating a laboratory's performance under controlled
conditions through analysis of unknown samples provided by an external source. Performance is
evaluated against static criteria or against criteria determined using data from all laboratories analyzing
samples during a round of testing. For biological analyses, EPA anticipates working with CDC to identify
the most efficient approach for the distribution of reagents Sentinel and Confirmatory laboratories for the
analysis of pathogens in drinking water samples, including standards and controls necessary for
participation in the PT program. For chemical and radiological analyses, EPA plans to explore the use of
vendors with which EPA has existing contracts/agreements for PT samples.
It is proposed that the PT samples should be distributed on a quarterly basis to the laboratories supporting
the WS-CWS. EPA plans to monitor the results of the PT samples submitted for each laboratory. If there
are serious deviations with the PT results, the laboratories should be notified so that corrective actions can
be initiated and problems can be resolved.
4.4.5 Tabletop Exercises and Water Contamination Drills
Tabletop exercises and drinking water contamination incident simulations should be used to evaluate how
well utility and laboratory staff respond when a trigger from the WS-CWS indicates the need for triggered
sampling and analysis.
The utility and laboratories that are part of the WS pilot should take part in tabletop exercises. The
tabletop exercises should take place before WS-CWS monitoring and surveillance activities are initiated
to verify that the roles and responsibilities of the utility and supporting labs in WS are clearly understood.
It should also establish whether or not the utility and the laboratory are able to respond effectively and
appropriately during a water contamination threat or incident. A 'lessons learned' report should be
developed from the tabletop exercise and put into practice during implementation of the WS-CWS.
The utility and laboratory should also participate in real-time, drinking water contamination incident
simulations to determine how well the utility and laboratories follow the consequence management plan
and credibility determination process during a water contamination threat or incident. Such
contamination incident simulations should take place periodically after the WS-CWS has been
implemented at the utility, and should mimic real-life situations as much as possible. This means a
component of the CWS should indicate that there is a 'possible' contamination incident, the utility should
collect samples, and the laboratory should analyze the samples by their routine analytical methods and the
unknown protocol. EPA expects that most, if not all, of these drills would be planned events, in which
specific circumstances are communicated to the utility and laboratories. If feasible, however, a 'blind'
drill may be conducted to determine if the baseline and/or triggered sampling program is capable of
detecting a change. Such a blind drill could be achieved through the use of off-line CWS components
that still feed information into the WS-CWS data management system (the event would be known to EPA
and the utility manager but not to any other participants). EPA plans to document results from the
drinking water contamination incident simulations in a study report that should be used to identify and
assess areas in which additional attention or training is needed to respond effectively to a water
contamination threat or incident.
DRAFT-121205 65
-------
WS System Architecture
Section 5.0: Enhanced Security Monitoring
Enhanced security monitoring includes the systems, equipment, and procedures required to detect and
respond to security breaches. This includes detection by physical security systems such as alarms and
cameras, witness accounts, and notifications by perpetrators, media and law enforcement as well as
response methods linked to these. A security breach is an unauthorized intrusion into a secured facility
that may be discovered through direct observation, an alarm trigger, or signs of intrusion (e.g., cut locks,
open doors, cut fences). Security breaches are probably the most common threat warnings for a utility,
but in many cases an apparent security breach is actually a false alarm related to routine operation and
maintenance activities. In most cases, actual security breaches are due to criminal activity such as
trespassing, vandalism, and theft rather than attempts to contaminate the water. However, any security
breach should be assessed with respect to the possibility of contamination. Ideally, an enhanced security
monitoring system should be designed so that security breach alarms and notifications related to a
contamination event can be differentiated from other events.
To provide sufficient time for law enforcement to respond and prevent a contamination event, physical
security design features such as fences, locks, and reinforced doors and hatches should be incorporated
into certain facility designs. It may be difficult to physically protect most water facilities from security
breaches that may occur as part of a contamination event, but some facilities can be sufficiently hardened
to prevent intrusion and contamination. However, it may be cost prohibitive to install physical security
features which provide sufficient delay against a sophisticated adversary who possesses the proper tools
and know-how. Therefore, it is often desirable to focus on physical security designs that utilize effective
detection systems that, in turn, should provide reliable warnings and assessments of a security breach. The
security breach may indicate the potential introduction of a contaminant and the appropriate response
actions could begin as soon as possible.
Enhanced security monitoring is one component of the WS-CWS with the potential to detect a
contamination incident before any exposures occur because the alert is linked to an intrusion that occurs
prior to the introduction of a contaminant. Furthermore, this type of monitoring has the means to detect
an intrusion in progress, and thus may provide an opportunity for law enforcement or security personnel
to prevent the introduction of a contaminant and apprehend the perpetrators. Enhanced security
monitoring can also potentially detect an activity that would result in the introduction of any class of
contaminant into a water system.
The consequences of a potential contamination event that is detected by an enhanced security system can,
in some cases, be eliminated or significantly reduced through operational responses that are triggered by
an enhanced security system. These responses may include linking detection alarms to automatically open
and close valves or disable pumps.
5.1 Integration of Enhanced Security Monitoring into the WS-CWS
An effective physical security system should detect a security breach early enough and provide adequate
delay to allow sufficient time for law enforcement to respond and prevent the adversary from completing
their intended act. The consequences of a potential contamination event can in some cases be quickly
eliminated or significantly reduced through an operational response that is triggered by a security alarm or
notification. These operational responses include linking detection alarms to automatically open and close
valves or disable pumps. The time to enact these measures would typically range from a fraction of a
minute to more than an hour if non-automated processes are used and/or large manual valves are required
to be closed.
DRAFT-121205 66
-------
WS System Architecture
This section discusses these concepts in more detail and describes how they should be used to properly
design a physical security system. The elements of a physical security system are depicted in Figure 5-1.
Physical Security System
Detection
Delay
Barriers
Locks
Structural ifHprcvemenl
Response
Response team
Response communications
Figure 5-1. Elements of a Physical Security System
Due to financial constraints and practical considerations, enhanced security monitoring should generally
be limited to selected locations in the utility's system that could serve as a likely location for potential
contamination of the drinking water. Such locations may include water storage facilities, wells, pump
stations, and treatment plants. However, protecting these facilities could be critical depending upon
system configuration and hydraulics because if contamination were to occur at some of these points like
large storage tanks, the consequences could be high. In addition, facilities like storage tanks are usually
highly visible and may be an obvious target for contamination making it important to monitor and protect.
Detection
The first parameter that affects the probability that the adversary should be successful in a security breach
is the likelihood that the intrusion should be discovered. A properly designed detection system includes
the following:
A sensor (equipment or personnel) reacts to its designed initiating event (door opening,
movement, etc) and initiates an alarm.
The information from the sensor should be reported and displayed.
The information is assessed and a determination is made if the alarm is valid.
The system to assess alarms should provide two types of information associated with detection:
whether the alarm is valid or a nuisance alarm, and details about the cause of the alarm (what,
who, where, and how many).
Physical detection systems that should be considered include but are not limited to interior and exterior
intrusion detection systems such as door contact switches, glass break sensors, motion detectors, and
fence mounted sensors. For monitoring and assessing alarms, an evaluation comparing methods,
including monitoring/assessment by utility staff versus using an outside contractor, should be done. If
monitoring is to be done by utility staff, the needs of the monitoring facility including communications,
security, and staffing should be addressed.
The most common method of alarm assessment is through the use of cameras and image display
monitors. Cameras can capture images of the adversary entering the facility and contaminating the system
and may be capable of identifying the quantity and possibly even the type of contaminant, thereby quickly
DRAFT-121205
67
-------
WS System Architecture
moving the event from a 'possible' to a 'credible' contamination scenario. The most effective type of
camera system would be one in which images are transmitted in a manner such that utility or security staff
are quickly alerted to the security breach and the images from the camera are immediately available for
review to help determine if the threat warning is credible. This may be accomplished by complementing
the cameras with interior or exterior sensors like door alarms or motion sensors and by 'freezing' the
image on the monitor, which shows the intrusion event.
The camera system design should define the required level of resolution categorized in one of three ways:
Detection: Assists remote assessment that simply determines the presence of an intruder.
Classification: Allows operator to determine if the alarm is due to a human intruder rather than an
animal or an object.
Identification: Allows operator to identify a specific human intruder.
Other camera options like pan-tilt-zoom capabilities, focal length, color vs. black and white, covert vs.
visible location, height above ground, tamper-proof encasement, and others options should also be
considered during design to ensure the successful implementation of a camera based security system.
Delay
Delay is the function of slowing the adversary on their way to the 'target'. The types of delay that should
be evaluated as part of a physical security system that addresses the contamination threat include: locks,
hardening of doors, windows and walls, barriers for storage tank access ladders, fences, covers for water
storage reservoir vents, and barriers to prevent vehicle access. The physical security system should
incorporate as many 'layers' of protection as possible in order to maximize delay, increase the probability
of detection, and decrease the probability of a successful intrusion. Layers of protection may include a
fence around the site, hardened doors and access hatches on the structures, and access control to the door
where the intended target is located, for example a chemical feed pump or reservoir access hatch.
Delay devices placed 'ahead' of detection are of little or no value. An intruder not faced with detection
has sufficient time to climb a fence or cut a lock without detection. After the penetration has been
detected and response is initiated, any delay provides more time for response, thereby decreasing the
likelihood of intruder success.
Response
Response is the time required by the response team or law enforcement to interrupt an adversarial event.
Response includes both interrupting and stopping the adversaries. The measure of response effectiveness
is the time between receiving a communication of adversarial action and interrupting and stopping it. An
effective security system should be able to detect the adversary early enough so that they do not have time
to carry out their task, are delayed long enough for the response to arrive, and are stopped before the
action is accomplished. However, detecting a contamination incident without being able to stop it does
have value because after an adversary is detected, actions can begin to be taken to mitigate the incident.
For a physical security system to be effective, it should perform the functions of detection, delay, and
response in less time than that required for the adversary to complete the insertion of a contaminant. The
adversary faces certain tasks (which equate to time) to accomplish his objective. In this example, the
adversary should climb a fence, run to the building containing the target equipment, break through the
building door, introduce the contaminant, and leave. This total time is known as the 'adversary task time.'
The physical protection system (PPS) time required starts at first detection and includes the time
DRAFT-121205 68
-------
WS System Architecture
associated with the alarm detection and assessment and the time required for the response force to
intercept the intruder.
Figure 5-2 illustrates an ineffective security system and represents what may occur if the first detection
opportunity were at the target facility itself. In this case, the intruder has completed his tasks before the
response force gets to the scene. The adversary has more time to complete the contamination since he has
already climbed the perimeter area fence before initial detection. Since the first alarm is at the target
facility and the time required for detection and response is greater than the system delay, the security
system is ineffective against the adversary.
Climb
Perimeter
Adversary Afea
Task Start Fence
Run to Target
Through Door
Adrtr«*ry Task
Contaminate Complete
Firet Alarm
Alarm
Assessed
Figure 5-2. Adversary Task Time versus Security System Time Requirements for an Ineffective
System
Given that the detection and response times should be the same regardless of the location of initial
detection, the addition of an intrusion sensor to the perimeter fence allows detection and response to start
earlier. Figure 5-3 represents this system configuration. The two boxes representing 'detect' and
'response' are now shifted to the left because the first detection system is moved 'farther out' from the
target building to the perimeter fence, illustrating that the security system should be effective in
interrupting the same adversary and preventing the contamination act.
However, as discussed previously there is value in detecting a contamination act even if the adversary is
not stopped from completing the act because getting this warning would provide the ability to begin
consequence management efforts.
DRAFT-121205
69
-------
WS System Architecture
Clrsb
^fimgift
Area
T«sk
Run to Target
Through Dour
Task
Complete
Firsrf Alarm
Assss&ed
Figure 5-3. Adversary Task Time versus Security System Time Requirements for an Effective
System
5.1.1 Other Security System Design Considerations
Other issues that should be addressed as part of a physical security system design include the following:
Install access control devices and intrusion detection sensors on doors or hatches to areas that
provide the opportunity for water contamination.
Hardening of doors, hatches, windows, vents, and locks to areas that provide the opportunity for
water contamination.
Communication of intrusion alarms using the utility SCADA system, independent electronic
security system monitored by the utility, or directly to law enforcement agency, depending on the
response procedures which have been developed for potential contamination incidents.
Camera systems should be scalable to allow future system expansion.
Compatibility of potential upgraded monitoring and recording equipment with any existing
cameras.
Definition of the camera system transmission media (fiber cable, coaxial, or twisted pair cabling).
Determine the number, type and location of viewing monitors and the need for a matrix switcher
or multiplexer that allows effective monitoring and recording of images.
Ensure there is sufficient light for cameras - the most common reason for poor image quality is
that light levels are too low or poorly designed. Provide sufficient light to discern individuals at a
distance of 75 feet and to identify a human face at about 33 feet. Avoid backlighting and high
contrast viewing areas.
Use an image compression standard that results in acceptable transmission speed.
Design electrical power supply to be reliable by considering backup power, lightning protection
and redundant systems.
Interconnecting wiring between security system components should be monitored for integrity so
that an abnormal condition such as a wire break or ground fault is indicated.
Establish the number and type of video image recording devices required to store sufficient data.
DRAFT-121205
70
-------
WS System Architecture
If cameras are not feasible for a particular location, install a secondary detection sensor to provide
verification of the primary sensor. For example, a motion detector could be installed to verify the
contact alarm on a door or hatch.
Design physical security systems based on the threat level including the type of contaminants that
may be used as the level of physical protection required and the type of contamination agents may
vary.
Implement policies and procedures to ensure the physical security systems operate as intended.
For additional detailed information regarding the design consideration for physical security systems see
references such as the EPA's Water and Wastewater Security Product Guide (USEPA, 20051) and
AWWA's Interim Voluntary Security Guidance for Water Utilities (AWWA, 2005).
5.1.2 Other Detection Methods
In addition to physical security detection methods, other methods of detection are possible as described at
the beginning of this section. To make these methods more effective and timely the following should be
considered:
Witness accounts: The utility should establish a close working relationship with local law enforcement
since witnesses should likely contact them rather than the utility. It should also be advantageous for a
utility to establish a system to collect as much information as possible from the witness to support the
initial threat evaluation which could be done by using the checklist form available in Module 2 -
Contamination Threat Management Guide of EPA's Response Protocol Toolbox. Both utility staff and
law enforcement should be made aware that timeliness of notifications is critical.
Notifications by perpetrators: Procedures should be established for handling threatening phone calls as
well as the handling and recognition of suspicious letters and packages. A checklist such as the one
available in Module 2 - Contamination Threat Management Guide of EPA's Response Protocol Toolbox
should be used during phone calls to assist staff who might receive a threat in gathering and providing
information to law enforcement. All staff who could potentially receive notifications should be trained in
the use of the checklist and the checklist should be made readily available at all times to staff.
Notifications by media: It would likely be important that utilities establish close relationships with the
members of all forms of the media to emphasize the importance of notifying the utility immediately if a
threat against the water supply is received. An established utility contact should be available to receive
calls from the media at any time. Regular periodic contacts should be made with the media to ensure that
media staff are aware of notification procedures.
Notifications by law enforcement: The utility should establish a procedure for reviewing the available
notification information with law enforcement to assess whether the threat is possible and decide on
appropriate response actions. While law enforcement agents should have the lead in the criminal
investigations, the utility has primary responsibility for evaluating the technical feasibility of the threat
and planning and implementing utility response actions.
For each of the above, proper training of utility staff should be essential for the detection methods to be
effective. Also, contact lists for media and law enforcement should be updated regularly. In addition,
tabletop and full scale exercises as described in Section 5.3 would help improve the methods.
DRAFT-121205 71
-------
WS System Architecture
5.1.3 Operational Response A ctions
The utility consequence management plan should describe actions that can be taken to mitigate the
impacts of a contamination event (USEPA, 2005i). Unlike most of the actions described in this
document, the consequences of a potential contamination event that is detected by an enhanced security
system can in some cases be eliminated or significantly reduced through operational response actions that
are triggered by an enhanced security system. Physical security detection systems with door contact
switches or motion detectors can be designed so that the detection system automatically or manually (with
operator action) triggers operational responses. Examples of operational responses that can be designed
into the system are discussed below.
An alarm triggered by the opening of a hatch, detection of motion, or detection of an intruder by a strain
gage on a ladder at a water storage tank could be tied into the control system to automatically or manually
prevent potentially contaminated water from entering the distribution system. This could be done several
ways depending upon the distribution and storage system configuration, operation, and hydraulics.
Depending upon the design of the storage system, doing this in many cases would not result in an
immediate significant loss of water pressure or flow in the distribution system. Therefore, notification of
the public of a potential contamination event would not be necessary unless the event is later determined
to be credible. Some methods of preventing potentially contaminated stored water are:
Isolating a water storage tank from the distribution system, either manually or automatically,
upon detection of an intrusion alarm. After further investigation, the water storage tank could be
brought back online if no threat of contamination existed. If water contamination was suspected,
then the tank could be drained through the sanitary sewer system after testing to verify that the
water would not adversely impact the sanitary sewer system.
For elevated storage tanks that use a single acting altitude valve to control maximum and
minimum level in the tank, the control system could be configured so that an intrusion alarm
opens the altitude valve to fill the tank to its maximum level. Doing this should usually result in
the distribution system being fed by a pumping station rather than the storage tank because the
hydraulic grade line should have changed. While the tank is being filled, the tank could be
isolated from the system by manually closing the influent and effluent isolation valves. If
possible, the influent and effluent isolation valves for the tank could close automatically upon an
intrusion alarm. This would isolate the tank from the rest of the system and prevent a contaminant
from spreading as described above.
Ground water wells often use pumps to pump water into an adjacent storage tank where booster
pumps are then used to distribute water to consumers. If an intrusion is detected by a storage tank
hatch contact alarm, for example, then the booster pumps and/or well pumps can be automatically
disabled to prevent contaminated water from being distributed. As part of this system, locks
should be installed on pump controls to prevent them from being operated in manual mode by an
intruder.
If a contamination event in the distribution system is suspected, but the entry location of the
contaminant is unknown, the distribution system chlorine residual could be increased in an
attempt to counteract those contaminants that may be inactivated or oxidized by chlorine. In
selecting the chlorine dose, consideration would have to be given to impacts on water taste and
odor and potential toxicity levels. Increasing chlorine levels would likely only be done where the
flow of water to the distribution system could not be stopped using methods described above. If
the contaminant was not known or non-reactive to chlorine, then a 'Do Not Drink' or 'Boil
DRAFT-121205 72
-------
WS System Architecture
Water' warning could be issued by the utility until additional information about the contaminant
and the extent of contamination becomes available.
In addition to linking intrusion detection alarms at distribution system facilities to automatic operational
responses as described above, other types of contamination threat warnings could also trigger similar
operational responses throughout the distribution system and possibly at treatment facilities. For example,
if a witness sees suspicious activities around a specific fire hydrant, the pipes in the vicinity of that
hydrant could be isolated by closing valves. Using a hydraulic model, the extent of contamination could
be estimated and additional isolation could be implemented. Depending upon the contaminant of concern,
the isolated portion of the system may be disinfected with a high dose of chlorine and/or flushed.
Additional adjustments to other components of the distribution system, like booster pumps, water storage
tanks, or the water treatment facility, would likely be required to compensate for the of the area taken out
of service. Details of these response activities should be addressed in the WS consequence management
plan as developed by the pilot utility (USEPA, 2005i).
5.2 Data Management, Analysis, and Interpretation
In order to avoid taking unnecessary response actions and prevent potential threat warnings from going
undetected, it should be essential that effective procedures be established for responding to threat
warnings from enhanced security systems. Utility staff that are responsible for responding to alarms
should be trained in how to identify false alarms, interpret alarm signals, and communicate potential
threat warnings. Section 8.0 provides additional discussion regarding the integration of information
streams and data as part of the WS-CWS.
5.2.1 Data Management
Analog or digital signals from intrusion detection sensors may provide continuous system status (i.e.,
door open/closed) or provide indication that an alarm condition is present (i.e., door contact broken thus
door has been opened). As is the case with process monitoring signals, alarm signals are typically input
into a data concentrator device such as a programmable logic controller (PLC), a remote terminal unit
(RTU), or remote input-output (RIO) device. This device could be the local PLC used for process control
or an independent, security-only PLC. Intrusion alarm signals should be transmitted from the remote
sensing location to a central communications interface at a data warehousing and analysis location. This
location may house the central process control PLC or a central security-only PLC. Data are transmitted
to this location via the most convenient data transmission pathway available (i.e., telephone line, radio
frequency, ethernet, etc.). If desired, data logging of alarms or local indication of alarm condition (i.e.,
audio or visual indication at the facility) may be desired. In addition, intrusion alarms may be transmitted
to the appropriate staff or law enforcement officials through the use of an automatic phone number dialing
system or through the display of camera images on a remote monitor.
If cameras are used, a separate data collection and transmission system is usually required due to the size
of data being handled. One of the significant challenges of designing a cost effective assessment system
that uses cameras is transmission of large amounts of video image data. A number of video compression
methods are available to reduce file sizes but transmitting large video data streams to remote monitoring
locations can be costly even with compression. To reduce costs, some utilities use video recording
systems that are installed at the same remote sites as the cameras are located. This video recording may be
useful in determining the credibility and nature of the contamination event but response would not be as
fast as with a system that can immediately transmit images. There currently are emerging technologies
that have the potential to more cost effectively transmit images to remote monitoring locations or
handheld devices such as cell phones. These technologies include systems that transmit images through
DRAFT-121205 73
-------
WS System Architecture
the SCADA systems that are already installed at many utilities and wireless transmission systems. In
addition, camera systems have another advantage in that there is a potential dual use in that they can in
some cases be used to monitor the system operation.
5.2.2 Data Analysis and Interpretation
The analysis and interpretation of intrusion alarms and images from cameras is often the responsibility of
utility staff. Upon receipt of an intrusion alarm, utility staff can either immediately contact law
enforcement officials about the intrusion, if the system does not do so automatically, or conduct further
analyses to determine if an actual intrusion and/or contamination event has occurred. For safety reasons, it
is preferable for utility staff to not visit the site where the alarm was tripped. Rather staff should, where
possible, remotely view a monitor at a control room or other location that provides access to video
images. Password protected, web-based displays of alarms, camera images, and even process control
information can be made available if set up in advance.
5.3 Framework for Evaluation
The ability to quickly assess a security breach alarm is essential to determining if the breach is indicative
of a credible contamination event. In many cases, an incident as indicated by a security breach alarm
should not immediately signify if there is a contamination event in progress. For many utilities, law
enforcement, supported by water utility staff, should first respond to the facility to determine if the alarm
is a credible indication of a contamination event. In many cases, they may not observe evidence that
contamination has or has not occurred. Further, it may be difficult to determine if the event is a false
alarm triggered inadvertently by utility staffer by another cause such as animals or wind. Therefore,
proper assessment of alarms through enhanced security monitoring using means such as cameras is
essential to quickly and reliably determine if an alarm is a credible and possibly an indication of
contamination.
Many drinking water utilities already utilize some or all aspects of enhanced security monitoring. Basic
access type alarms such as door contact switches and motion detection devices are relatively
commonplace at water utilities. In addition to reducing the risk of potential contamination, security
monitoring systems should deter and prevent other malevolent acts such as vandalism and theft. Access
control systems can be used to limit access to selected facilities to only authorized employees and can
record employee entry/egress to ensure duties such as equipment monitoring are being done. Also,
camera systems can be used to remotely monitor process operations such as equipment performance and
hydraulic levels, providing the utility with multiple benefits.
Several aspects of enhanced security systems can be evaluated to help determine its effectiveness. Since
actual intrusions may be relatively rare, tabletop and full scale simulation exercises dealing with several
threat warning scenarios that test the efficacy of security systems could be developed. The exercises
should involve all those who may be involved in a genuine event including operations and maintenance
staff, treatment and distribution system staff as well as, law enforcement, and contracted security firms.
Full scale exercises could be run in an effort to confirm how distribution system operational responses
such as turning pumps on and off, isolating water storage tanks, or flushing fire hydrants would impact
the system hydraulic grade line and direction of the flow of contaminants. These aspects and potential
methods for evaluation include the following:
Detection, delay, and response effectiveness: Through exercises and simulations, track the
number of security breaches that were or were not detected by the physical security systems
within the time period specified in the system design. The reasons why the breaches were not
DRAFT-121205 74
-------
WS System Architecture
detected in a timely manner should be documented. In the event of an intrusion, document if the
physical barriers provided sufficient delay to allow time for law enforcement to respond. Also,
document law enforcement response times.
Assessment: In the event of an intrusion, document if the assessment system was capable of
assisting the operator in identifying the nature of the threat.
False/nuisance alarms: Through analysis of actual operational history, simulations, and
exercises, determine what percentage of physical security system alarms are false or nuisance, as
well as the reason for the false alarm. Based on this determination, establish a procedure to reduce
false/nuisance alarms and/or identify needed system changes, such as relocating or replacing a
sensor.
Operational response actions: In the event of an intrusion, document if the proper operational
response actions were followed and if they were effective.
Other detection methods: In the event of an intrusion detected through means other than the
physical security system, document the effectiveness of the procedures for detecting a potential
contamination event through witness accounts, and notifications by perpetrators, media and law
enforcement.
A non-intrusive method for evaluating the effectiveness of the enhanced security system is to run a
quantitative analytical computer model simulation on the facility of interest and its physical security
system. Several such models have been developed by Sandia National Laboratories, including EASI
(Estimate of Adversary Sequence Interruption) and SAVI (System Analysis and Vulnerability to
Intrusion), and other commercial products are available. These models are useful tools in providing an
initial determination of the effectiveness of the enhanced security system and identifying the areas where
additional protection is needed. The model analyzes all aspects of physical security design including
detection, delay, and response and improvements may be identified in one or several of these aspects.
DRAFT-121205 75
-------
WS System Architecture
Section 6.0: Consumer Complaint Surveillance
Consumer complaints, if systematically tracked and responded to in a timely and appropriate manner, can
be a valuable component of a CWS. Located throughout a utility's service area from the beginning of the
distribution network to its far reaches, consumers can provide near real-time input regarding changes in
water characteristics, as well as report on suspected tampering to pipelines, hydrants, valves and other
appurtenances. Consumers may detect contaminants with organoleptic characteristics that impart an odor,
taste or visual change to the drinking water. Commercial and industrial consumers may be able to detect
contamination that alters the chemical and physical properties of the water not obvious to the residential
consumer due to anomalies in the products produced from the water, or from impacts on advanced
treatment processes used by a consumer to provide ultra-pure water. Reports of illness, although not
often reported initially through the water utility, can provide important information in confirming water
contamination events detected through public health surveillance as discussed in Section 7.0. Moreover,
historically consumer complaints have played a role in the detection of several drinking water
contamination incidents. For example, a 12 hour diversion of excess hydrofluorosilicic acid from a water
treatment plant to a residential community water supply in Connecticut resulted in the population's
ingestion of fluoride and copper at levels several times greater than normal (Petersen LR et al., 1988).
Consumer complaints reported at the water utility during this event ranged from taste and color to nausea,
vomiting, diarrhea, cramps and skin irritation.
Coverage provided by consumer complaint surveillance includes residential and commercial consumers
throughout the entire distribution system, and thus can provide system-wide coverage for the WS-CWS.
Both the number and geographic distribution of consumer complaints can assist the utility in not only
detecting potential contamination, but the area of contamination as well. However, consumer densities
vary from utility to utility, and also vary within different portions of a utility's service area, possibly
leaving some locations in the service area without this typically spatially continuous component.
The most critical factor when considering consumer complaint surveillance is receiving enough qualified
complaints within a designated period of time for the utility to further investigate. Odor, irritant, and taste
complaints are often received by the utility fairly quickly; within the first few hours after a consumer
notices a problem. However, consumption of water in every portion of the distribution system is not
consistent, and certain portions of the service area may have very few consumers during the late night
hours (e.g., residential areas), or on the weekends (e.g., industrial areas). Consequently, while the time
from a consumer recognizing a problem with their water may be relatively rapid, the time between a
contamination incident and the consumers' use of the water may be lengthy. Timeliness should be a
function of the time between initial exposure, consumer reporting to the water utility and subsequent
categorization, assessment and investigation of the complaint. For contaminant detection classes 1
through 4, consumer reporting and categorizing of the complaint should be fairly quick as they have
particular organoleptic characteristics. Although most microbiological contamination does not affect the
aesthetic characteristics of water they do cause clinical symptoms that might be detected initially by either
consumer complaints or public health surveillance. Usually in cases of microbial contamination or
chemicals that do not affect the aesthetic quality of water, consumer complaint surveillance should
provide secondary detection and corroboration of the contamination event during credibility
determination. Consumer complaints have been effective in prompting investigation or corroboration of
contamination events on numerous occasions. For example, during the cryptosporidiosis outbreak in
Milwaukee, WI and in the fluoride diversion incident in Connecticut there were widespread clinical
symptoms and consumer complaints (Foldy, 2004; Petersen LR et al., 1988).
DRAFT-121205 76
-------
WS System Architecture
6.1 Attributes of an Effective Consumer Complaint Surveillance Program (CCSP)
Consumer complaints, when integrated with other primary utility information streams (online water
quality monitoring, sampling and analysis, public health surveillance, and enhanced security monitoring)
can contribute to the accurate detection of a contamination event. Developing and maintaining a timely
and accurate consumer complaint surveillance program requires a systematic review and enhancements to
the program over time. Educating consumers, as well as communicating with them about the information
they provide, is critical to creating an effective program that minimizes false alarms.
In addition to residential consumers, commercial and industrial consumers may be able to detect more
subtle changes in drinking water quality from observations of abnormal reactions in manufacturing
processes or from upsets in water purification systems used for specialized applications such as the
production of computer chips or pharmaceuticals. Incorporating vital information from the monitoring
and surveillance activities of certain key customers, industries, and institutions (such as hospitals) into the
consumer surveillance program can provide an additional source of valuable data.
Program goals, available resources, and sophistication of data management systems should all be
considered when developing an effective consumer complaint surveillance program. These factors are
different for different utilities and should have an effect on the specific design of a utility's program.
However, all programs need to incorporate the components of an effective consumer complaint
surveillance system, assess their current system, enhance their existing system where appropriate, and
evaluate their system's performance. Figure 6-1 identifies the main components of a consumer
complaint surveillance program and illustrates an adaptive process for continuing to improve the program
overtime.
Consumer
Complaint
Received
Figure 6-1. Components of a Consumer Complaint Surveillance Program
Several key elements are important to ensure that consumer complaint management is translated into an
effective surveillance program. These elements are incorporated into the utility's business processes and
operating procedures, and supported by written policies and procedures by which consumer complaints
are managed. They describe in detail which staff should be involved in consumer complaint
management, including how they should handle the complaint from receipt through resolution and
evaluation. Each of the following elements is important to consider during the integration of a consumer
complaint management system into a WS-CWS.
Complaint receipt and documentation
Categorization and routing
Assessment and investigation
Complaint tracking
DRAFT-121205
77
-------
WS System Architecture
Senior staff oversight
Staff training
Consumer education
Complaint receipt and documentation. Ideally, consumer complaints should be received by the utility's
call center. Specific staff are assigned to document complaints received by telephone, fax, e-mail, U.S.
mail or private carrier, or in person. Calls coming in on the phone system can be identified by Customer
ID based on incoming phone number (or other similar tools). Complaints submitted with payments that
go to a lockbox rather than to the utility should be routed to the person in charge of documenting them at
the call center. A single electronic database should be used to log all complaints and assign a unique
identification number to each complaint. The database should include a standard data entry-screen. The
database may be part of, or integrated with, the utility's customer information system. Off-site daily
back-up of the database is recommended, as is a daily paper printout that could be used should computers
not be available. In addition, a single number for consumers to call with complaints should be identified.
This line should be answered 24/7 but approaches may vary by utility. Consideration should be given to
the call center's capacity and an overflow strategy developed. 'Many cities have emergency phone lines
and call centers available, and these can be requested for use by a water utility during high call volume
periods. These numbers and operators are typically available through a city's emergency manager or
emergency operations center. Staff training should be provided and a shift system designed for dealing
with volumes. Linking to the Utility's Work Order and GIS systems could assist in identifying known
problems or discovering potential problems in need of evaluation. Historic events are also
considerations.
Categorization and routing. Develop a decision tree for call center staff to categorize the type of
complaint and where to route the complaint for assessment and action. The decision tree should permit
staff to easily rule out common causes of complaints, such as entrained air, rusty water, etc. to quickly
identify any complaint that may be indicative of drinking water contamination. Those complaints not
easily ruled out should be sent to a water quality expert who should serve as a single point of contact.
This may be one individual staff member in a small utility or a team of individuals in a larger utility.
Assure that complaints that cover several issues (e.g., overcharge, potholes, and cloudy water) are
dissected for routing to multiple points as appropriate. All complaints (and portions of complex
complaints) that may be associated with contamination (including tampering) should be immediately
routed to the water quality expert who is available 24/7. The single-point of contact promotes clarity in
handling potential contamination problems, facilitates mapping of complaints, allows for more efficient
communication with other CWS components and, with properly assigned authority, instills urgency
throughout the organization.
Assessment and investigation. Complaints should be assessed to determine the possible causes of the
complaint, such as hydrant flushing, change in treatment process, or construction activity. Such
assessment along with inquiries to other components of the CWS can assist in determining the risk
associated with the complaint and the response time necessary. Complaints that cannot be resolved over
the phone, or those which have unknown causes, will likely require an on-site investigation and sampling
for field and laboratory analysis. A site characterization and sampling protocol should be developed to
support investigation of any trigger from the CWS, and this same protocol, possibly with some
modifications, could be used to investigate consumer complaints. The protocol should also include
criteria for determining whether additional field analysis with more advanced equipment such as portable
toxicity analyzers is required and for determining the parameters that should be assessed in the laboratory.
A well thought-out decision tree should also aid utility managers in making decisions about the actions to
take based on the results of field and laboratory analyses. Decision trees can also be developed to help
identify the nature of the problem. For example, the nature of the complaint (characteristic smell or taste)
DRAFT-121205 78
-------
WS System Architecture
might provide insight into the nature of a potential contaminant. Use of EPA's Water Quality/Consumer
Complaint Report Form, included in the Response Protocol Toolbox could further aid the utility in
assessing a potential contamination event.
Complaint tracking. All consumer complaints that may be related to contamination should be tracked
using a database to record geographic and complaint-type data, as well as results of investigation and
related illnesses. Thus, this database should be a subset of a larger database that should track all
complaints. While a GIS linked to the complaint database is preferable, manual tracking on a paper map
is sufficient for smaller systems. Some systems allow complaint data to be managed through their utility
work order system. Whether electronic or manual means are utilized, results should be displayed on
printed maps in a format where the staff can quickly and easily recognize patterns and increasing spatial
extent. Each complaint should be tracked through every step of its assessment, investigation, and
findings/resolution. Links to information generated from other CWS components should be noted in the
database. Periodic reports summarizing complaints by type, location and findings should be submitted to
utility management and kept available for future analysis. For widespread problems, the ability to use
hydraulic/water quality distribution system models can be a useful tool to understand the pattern of
complaints.
Senior staff oversight. The responsibility of the consumer surveillance program should rest with one
individual at the senior staff level of the utility, perhaps the Water Utility Emergency Response Manager.
While not necessarily a line-manager position, the individual should have the authority to direct the
consumer complaint process through all stages for matters that may relate to contamination. This
authority should include related activities in customer service, operations, and the laboratory.
Consideration should be given to having this individual report directly to the chief executive officer
(CEO) or chief operating officer (COO) of the utility. Communications with the call center manager,
water quality manager, and distribution manager, and other agencies that may also receive water quality
complaints, such as the local public health department, should occur.
Staff training. Because consumer complaints may be presented to any member of the utility's staff in an
office or in the field, formally or informally, all staff members should be trained to effectively handle
consumer complaints, find the written procedures, and determine whom to contact with questions.
Training should include the questions to ask consumers to discern water quality problems that may be
associated with contamination. The staff should clearly understand the serious consequences that may be
associated with some consumer complaints and the necessity for quick action. Staff specifically assigned
to key departments related to water contamination and consumer complaints should receive more rigorous
training that includes functional exercises and coordination with public health officials and
representatives of other CWS components. Call center staff would require some specific training. Some
cross training with other functional areas is encouraged.
Consumer education. A truly effective consumer complaint surveillance program should include
educated consumers. Utilities with existing public outreach programs should inform consumers about the
characteristics of their drinking water, connections to hydrants, including meters and backflow prevention
devices, and suspicious persons working on the water distribution system. Telephone numbers to
consumer complaint lines should be published in telephone books, printed on bills, and prominently
appear on websites. Overall, consumers should feel empowered to call when they are concerned about
the quality of their water or tampering of the water system.
Figure 6-2 depicts the consumer complaint surveillance process, beginning with the information noted
and reported by the consumer to the utility and how the utility manages the complaint, monitors the
information it receives, and proceeds once an anomaly is detected.
DRAFT-121205 79
-------
WS System Architecture
Figure 6-2. Consumer Complaint Surveillance Process
The integration and evaluation of consumer data with the utility's other primary data sources results in a
determination of a threat's credibility, which leads to either responding to the contamination event or
returning to routine monitoring. The last step in the process is to ensure the consumer presenting the
complaint is communicated with after their complaint has been assessed. This communication may
happen as part of a call being received and addressed by the call center, or by way of a thank you note
from the utility following the utility's response to a contamination event. It is another way in which the
utility can help to educate its consumers.
6.2 Data Management, Analysis, and Interpretation
The data management system for consumer complaint surveillance can be managed through the utility's
customer information system, work order system or other enterprise information system. Another option,
often used by smaller utilities is to manage complaints with a stand-alone database or even manually, with
logbooks. A utility managing a consumer complaints surveillance program as a component of a CWS
should maintain rigorous recordkeeping, provide seamless tracking, and make information and data
quickly available to staff for evaluation and decision making within the utility.
To gain the most benefit from the consumer complaint surveillance program of a CWS, the information
and data should be integrated with output from other information streams of the CWS, both within the
utility and from outside the utility, such as public health surveillance. In addition to making the data
available to the utility's public health partners, the data should also be made available to other relevant
agencies. An example of the components of a data collection, transmission and integration system for
consumer complaint surveillance are presented in Figure 6-3. Roles (field sample collector, law
enforcement, call center operative, data analyst, etc.) need to be identified and explained in detail.
Primary and backup representatives should also be identified. Data types and formats should be listed,
with references to supplementary documents for further detail as warranted (i.e., explanations of lab
analysis results). Primary and, ideally, backup mechanisms for each data collection, communication, and
transmission point should also need to be identified. Data sharing agreements need to be listed and
should also need to be implemented and executed. In addition, vulnerabilities for each data collection,
transmission, and storage point should need to be identified and addressed. Other data management
design issues that need to be considered include data privacy, sensitivity and security, authorization,
encryption, timeliness, cost, redundancy, and availability. Communication procedures among those
parties involved in specific roles should need to be developed.
DRAFT-121205
80
-------
WS System Architecture
Consumer
Customer call
center
Water quality
and distribution
specialists
Emergency
Responders
System
Collect
Samples
Analyze data
Figure 6-3. Utility Consumer Complaint Data Flow
Overarching information about information integration, management, and communication can be found in
Section 8.0.
In order to determine the specific elements of the utility's data management system for consumer
complaints, the following questions should be explored:
How does the current consumer complaint data management system function?
Is it efficient?
Is it capable of recording and tracking all consumer complaints?
How does it capture multiple information streams?
Can it integrate data?
How does it interface with the utility's other data management systems?
What support does it provide for decision-making?
Is information readily available?
Does it have a geographic component?
Is it able to correlate customers and complaints?
Is storage fireproof?
Where are back-ups of consumer complaint information kept?
Who is responsible for managing the data?
Can the data be accessed 24/7?
Can complaint data be received from and shared with other government agencies?
DRAFT-121205
81
-------
WS System Architecture
6.3 Framework for Evaluation
Most complaints from consumers are subjective and are handled and assessed by individuals not trained
in water quality analysis. Consumers may be reluctant to report a change in the characteristics of their tap
water, or they may not see any significance in the change. Consumers may not know whom to contact to
report a complaint, or they may believe that they cannot sufficiently describe the issue and, therefore,
decide not to complain. Also, the link between consumer symptoms and contaminated water is difficult
to make and may not ever be considered by the consumers or their health care providers. Further, the
efficacy of consumer complaint surveillance in a CWS may be constrained by the lack of processes and
procedures, as well as staff training, at the utility to adequately assess, route, interpret and react to a
complaint. Nonetheless, it is a potentially reliable indicator for contaminants with detectable
characteristics if a robust complaint reporting and tracking system is in place. Additionally, it could be
used by the other components of the CWS as an indicator or to corroborate potential contamination
events.
Over the past several years, the water industry has focused on improving the management of consumer
complaints through the funding of research projects and development of guidance materials. While most
of the investment has been on the management of traditional complaints (e.g., aesthetics, insufficient
pressure, rates/fees and billing), more recent work has concentrated on water quality complaints that
could be related to intentional or unintentional contamination. Properly trained staff, effective complaint
handling procedures, and carefully developed decision tools can not only improve the consumer
surveillance component of the WS-CWS, but also enhance a utility's image among its customers and
other stakeholders. Several cities and special purpose districts have made efforts to consolidate all citizen
concerns to ensure that all complaints are going to the appropriate city agency or department in a timely
manner. One example is New York City's 311 phone number for government information and non-
emergency services. The 311 call center answers calls 24/7, has immediate access to translation services,
uses a state-of-the-art database of city information and services that can be updated in real time and can
quickly adapt in an emergency situation (www.nyc.gov/html/doitt).
The utility's consumer complaint surveillance program can be evaluated through a series of processes and
tools, many of which may already be in use within a utility. This information can begin to assess the
adequacy and efficiency of the program and whether the consumer complaint surveillance system is
operating optimally. If the program is falling short of its goals, or if a component is inadequate and
failing to optimize the information being fed into the system, the utility should have the opportunity to
identify these shortcomings and make the necessary enhancements.
6.3.1 Evaluation Tools
Most customer information systems can produce detailed reports on the average time a caller waits to
speak with a customer service representative, the number of complaints and the type of complaints
recorded. Linked to a GIS, complaints by type can be exported from a customer information system to
generate maps showing areas with chronic, seasonal, or repetitive complaints occur, which may warrant
further investigation in the field.
Periodic review of audio tapes from customer service representatives may also be useful in refining
employee training. In addition, process mapping can assist in evaluating the efficacy of the processes
being employed in the consumer complaint surveillance program.
Assessments of specific or randomly selected complaints can highlight procedures in need of
improvement and program assessments can help to reveal bottlenecks in procedures, reduce response
times and encourage ongoing communication between staff, management, and utility customers.
DRAFT-121205 82
-------
WS System Architecture
6.3.2 Data Sets
Essential to the evaluation of a consumer complaint surveillance program are any correlations between
the number, type, and location of complaints to actual findings by utility personnel or through laboratory
analyses of any anomalies due to operational, maintenance, or construction activities; fire suppression; or
contamination.
While integrated information systems are valuable for all aspects of a consumer complaint surveillance
program and its evaluation, smaller utilities can adequately track and evaluate this aspect of their
programs with basic database or spreadsheet software and manual record keeping.
6.3.3 Consumer Confidence Surveys
Consumer confidence is a good indicator that consumers feel their concerns have been heard and that
their complaints have been addressed appropriately and in a timely fashion. Following up with
consumers who have submitted a complaint is not only good customer service, but an important part of a
thorough consumer complaint surveillance system. Annual or bi-annual surveys should assist the utility
in knowing whether or not its customers feel the consumer complaint surveillance system is working.
Field investigations and consumer follow-up should also be considered in the evaluation. Optimization of
consumer complaint surveillance systems provides other benefits to drinking water utilities beyond the
security objectives of the WS-CWS. Collection and management of information from drinking water
customers can assist in identifying treatment issues and water quality issues throughout the distribution
system.
DRAFT-121205 83
-------
WS System Architecture
Section 7.0: Public Health Surveillance
Close coordination between drinking water utilities and public health departments is a critical component
of the WS-CWS for both detection of contamination events and initiation of response actions. In
addition, the public health sector offers tools, procedures, and lessons learned that can be leveraged to
enhance detection capabilities for a range of contaminants. Furthermore, lessons learned from recent
outbreaks of waterborne disease, including the 1993 outbreak of cryptosporidiosis in Milwaukee, and an
outbreak in Walkerton, Ontario, in May of 2000 caused by E. coll illustrate how the integration of
environmental, health-care, and other types of data can provide earlier warning or more robust validation
of problems than clinical signs and symptoms alone (Foldy, 2004; Hrudey, 2002). As another example, a
retrospective study from two waterborne outbreaks in Saskatchewan and Ontario, Canada comparing
over-the-counter sales with the frequency of emergency room cases of gastrointestinal illness caused by
Cryptosporidium, E. coll O157:H7 and Campylobacter indicated that information from this type of
syndromic surveillance could provide a more timely indication of illness in the population than other,
more traditional, types of surveillance (Edge, 2004).
As defined by CDC, public health surveillance is the ongoing, systematic collection, analysis,
interpretation, and dissemination of data about a health-related event for use in public health action to
reduce morbidity and mortality and to improve health (German, 2001). Although public health
surveillance can be used as a tool in outbreak detection, it also has other applications such as supporting
public health interventions, determining distribution and spread of illness, evaluating prevention and
control measures, and facilitating planning (Buehler, et. al., 2004). Syndromic surveillance is a specific
type of public health surveillance that relies on electronic data such as 911 calls, ER visits, EMS logs,
OTC medication sales, laboratory test orders, workplace or school absenteeism, and other types of data
that may be available in the early stages of an outbreak.
Syndromic surveillance systems seek to use existing health data in real-time to identify changes in
community health status, facilitating notification to those charged with investigation and follow-up of
potential public health crises (Henning, 2004). With the increase in utilization of syndromic surveillance
since September 11, 2001, as a tool for early indication of a bioterrorist attack, many local health
departments are now confronted with the challenge of how to evaluate the effectiveness of these systems
in the absence of a bioterrorism event. Many of these considerations also apply to the evaluation of the
WS-CWS and are discussed in greater detail in Section 6.0.
As a result of the privacy concerns and data sharing restrictions codified by the Health Insurance
Portability and Accountability Act (HIPAA), only summary data may be presented to the water utility
officials by public health. Under the conditions of an aggregate data set agreement, actual date and time
and a full zip code associated with distinguishing health event details can be shared when personal
identifying details such as medical record numbers, names, street addresses, ages and genders are omitted
from the data to be shared. Numerous techniques may be employed by the data providers and public
health systems to summarize and strip protected data elements prior to aggregated transmission to WS-
CWS.
The types of data streams most commonly monitored by public health officials include the following:
Aggregate diagnosis counts by date and geographic area: for a given date and zip code, the
number of patient diagnoses for each relevant condition (i.e., water-borne infection or irritant)
across the public health jurisdiction. This detail would be provided typically by disease-specific
DRAFT-121205 84
-------
WS System Architecture
surveillance activities at the public health department as reported by physician and health care
facilities.
Aggregate lab tests (order counts and/or results) by date and geographic area: for a given
date and zip code, the number of ordered laboratory tests for each relevant condition and, when
possible, the lab results across the public health jurisdiction. This detail would be provided
typically by disease-specific or syndromic surveillance activities at the public health department
as reported by physician and health care facilities or laboratories.
Categorized chief complaint, 911 call, EMS runs, or Poison Control counts by date and
geographic area: for a given date and zip code, the number of categorized complaint
descriptions (i.e., symptoms and possible background for the symptom onset) for each relevant
condition across the public health jurisdiction. This detail would be provided typically by
disease-specific or syndromic surveillance activities at the public health department as reported
by physician and health care facilities, 911 operators, EMS, and/or the Poison Control Center.
Categorized over-the-counter (OTC) medication sale counts by date and geographic area:
for a given date and zip code, the number of product units sold for each relevant condition across
the public health jurisdiction. Information regarding sales/specials of OTC sales is also reported
in order to normalize the data.
Public health surveillance should be able to provide coverage for drinking water utilities within the health
department's jurisdiction. In some cases a single utility may provide drinking water to customers across
multiple health jurisdictions. Adequate communication, coordination, and exchange of data between the
health department, drinking water utilities, laboratories, and healthcare providers is critical in terms of
spatial coverage of this component of the WS-CWS. However, any breakdown in this communication
might sever a portion of a department's area from occurring events. For example, if a health care
provider was unaware they needed to report certain symptoms or conditions; the health department may
be oblivious to these events. Effective networking and strong, clear communication and exchange of
information between all stakeholders should alleviate any spatial coverage issues related to public health
surveillance as a WS-CWS component.
7.1 Overview of Existing Syndromic Surveillance Systems
The type of public health surveillance employed at the local level varies from traditional surveillance
activities to sophisticated syndromic surveillance systems that are customized based on available software
and/or rely on national syndromic surveillance systems to manage local data. New York City, for
example, relies on a customized syndromic surveillance system that collects data from a variety of
different sources. In fact, New York City's first syndromic surveillance systems were established in 1995
to detect outbreaks of waterborne illness by the New York City (NYC) Department of Health and Mental
Hygiene (DOHMH). The program included diarrheal illness at nursing homes, stool submissions at
clinical laboratories, and OTC pharmacy sales. In 1998, daily monitoring of ambulance dispatch calls for
influenza-like illness began; after the 2001 World Trade Center attacks, concern about biologic terrorism
led to the development of surveillance systems to track chief complaints of patients reporting to
emergency departments, OTC and prescription pharmacy sales, and worker absenteeism. These systems
have proved useful for detecting substantial citywide increases in common viral illnesses (e.g., influenza,
norovirus, and rotavirus) and diarrheal illness following the August 2003 blackout. However, though this
system is useful for early detection at the city-wide level, this system has yet to detect localized, such as
specific hospital or institution outbreaks earlier than traditional surveillance. Future plans include
monitoring school health and outpatient clinic visits, augmenting laboratory testing to confirm syndromic
signals, and conducting evaluation studies to identify which of these systems should be continued for the
long term (Heffernan, 2004).
DRAFT-121205 85
-------
WS System Architecture
Although New York City has the capacity to execute a fairly sophisticated public health surveillance
system, other local health departments may be utilizing more basic surveillance measures. As of 2002,
only 25% of local health departments were able to deliver the majority of essential public health services
(Baker, 2002.) The reasons behind these disparities are varied and should provide further challenges for
assessment and implementation of WS-CWS. While impractical to raise the level of all local health
departments' capacities to that of a major city, surveillance standards should be improved to a level where
their integration enriches the capacity of WS-CWS.
There are three primary national syndromic surveillance systems, and a nationally available tool that
could be utilized as part of the WS-CWS for integrated analysis of water quality and public health data.
These systems include the following:
BioSense
Electronic Surveillance System for the Early Notification of Community-Based Epidemics
(ESSENCE)
Real-Time Outbreak and Disease Surveillance (RODS)
National Retail Data Monitor (NRDM)
The national syndromic surveillance systems are summarized in Table 7-1.
Table 7-1. Summary of National Syndromic Surveillance Systems and Tools
Attributes
Developer
Objective
Brief
Description
BioSense
CDC
Enhance nation's capability to
rapidly detect, quantify, and
localize public health
emergencies, by accessing and
analyzing diagnostic and pre-
diagnostic health data.
Internet-accessible secure
system that permits users to
visualize information about
public health trends from early
detection data sources.
BioSense is in the process of
implementing the beginning of
Phase 2 with what is called
Real-Time Clinical Connections
(RTCC) - direct data feeds
from select hospitals. The goal
is to implement feeds from 1 0
hospitals by end of year, but
achieving that goal is becoming
increasingly less likely due to
issues around data use
agreements and specification of
the data stream; no data
streams are active at this point.
ESSENCE
DOD, Johns Hopkins University
Applied Physics Laboratory
(JHU-APL)
Collect and analyze a variety of
data sources for the early
recognition of abnormal
community disease patterns that
could result from natural causes
or terrorist activities.
Data providers de-identify,
encrypt, and post data to a
secure file transfer protocol
(FTP) site at a regular interval
(e.g., daily at midnight or once
every 8 hours). ESSENCE polls
the FTP sites to look for new
entries, which are then ingested,
cleaned, formatted, and
archived in the primary system
archive and applies A secure
website allows for information
transfer to users. Through this
website, users also can view a
map of the geographic
distribution of raw data and data
clusters, view alerts, conduct
queries, and generate summary
reports.
RODS
University of Pittsburgh in
collaboration with Carnegie
Mellon
Computer-based public
health surveillance system
for early detection of
disease outbreaks; the
initial objective of RODS
was to detect large scale
outdoor aerosol releases of
anthrax.
ER and OTC data are
incorporated in the RODS
user interface, the
registration chief complaint
is automatically classified
into one of seven syndrome
categories using Bayesian
classifiers. Data are stored
in a relational database,
aggregated for analysis
using data warehousing
techniques, univariate and
multivariate statistical
detection algorithms are
applied to the data, and
users are alerted when the
algorithms identify
anomalous patterns in the
syndrome counts.
NRDM
University of Pittsburgh in
collaboration with Carnegie
Mellon
Public Health surveillance
tool that utilizes info from
OTC sales for early disease
outbreak identification by
health departments.
Collects data for RODS on
selected OTC health care
products from over 20,000
stores for use by public
health departments free of
charge. Data is transmitted
by a secure FTP link with a
delay of 24 hours or less
into a data warehouse.
These data are merged by
geographic area and
distributed in raw or
analyzed form to users in
46 states, the District of
Columbia, and the CDC.
DRAFT-121205
86
-------
WS System Architecture
Attributes
Data
Sources
Event
Detection
Algorithms
Timeliness
Additional
Information
BioSense
U.S. Department of Defense
(DOD) and Veterans
Administration (VA) medical
treatment facilities, LabCorp,
BioWatch air monitoring
system.
Advanced algorithms for
visualizing and analyzing data
to provide a nationwide, real-
time picture.
Hospital systems data available
in near real-time via PHIN
connection.
httDV/www.cdc.aov/Dhin/comco
nent-initiatives/biosense/
ESSENCE
Military ambulatory visits,
prescription medications, chief-
complaint data from emergency
rooms, patient visits for private
practice groups, OTC sales of
Pharmaceuticals, nurse hotline
calls, school absenteeism
records, data about local
endemic disease, sales
promotions, weather events,
occurrence of high profile events
in the community, 91 1 calls,
poison center calls, requests for
laboratory work
Spatial and spatial-temporal
outbreak-detection algorithms,
forms clusters in time and space
across the region by using zip
codes as the smallest spatial
resolution.
Most of the data are received
within 1 to 3 days of patient visit
httc://www. aeis. fhD.osd.mil/GEI
S/SurveillanceActivities/ESSEN
CE/ESSENCE.asc
RODS
Absenteeism data, sales of
OTC health care products,
and chief complaints from
ERs.
RODS has two detection
algorithms: the Recursive-
Least-Square adaptive filter
and What's Strange About
Recent Events 1 .0.
Hospital data received in
near real-time from clinical
encounters over virtual
private networks and leased
lines using the Health Level
7 (HL7) message protocol.
httcV/rods. health, citt.edu
NRDM
Uses Universal Purchase
Code (UPC) data to collect
info from large national and
regional retail chains.
MapPlot can be used to
detect standard deviations
from normal sales for any
area; Epiplot can be used
for trend analysis.
Retail data is transmitted
and received less than 24
hours after point-of-sale
httcV/rods.health.Ditt.edu/N
RDM. htm
7.2. Public Health Surveillance and WaterSentinel
In the context of the WS-CWS, certain types of agents should be more readily recognized by a public
health surveillance network due to the severity and time of onset of symptoms. Contaminant Detection
Classes 2 to 10 produce symptoms that would elicit ER visits, poison control calls, or 911 calls, generally
within a short time of symptom onset. Ricin contamination, for example, would be first detected by
public health surveillance due to its sudden and severe symptoms onset. These symptoms would prompt
most people to call the poison control center or visit the ER. Consequently, for these contaminants, public
health surveillance would likely identify a contamination incident before the online monitoring, sampling,
or consumer complaints components of CWS.
The timeliness of information from public health surveillance varies in relation to other components of
the WS-CWS, depending on the type of contaminant present, the level of cooperation between health
departments and health care providers, and the technical and staffing support available to a health
department. Timeliness provided by public health surveillance is also contingent on the capacity of health
departments to receive and integrate data from clinical laboratories, health care providers, and syndromic
surveillance sources (OTC sales, EMS logs, etc.). This capability is directly dependent on the health
department's staffing and technical support, as well as the cooperation between the different components.
While a patient may present with symptoms within four hours of exposure, the time by which a health
department is made aware of an event may vary from hours, days, or longer. If a clinical laboratory does
not have electronic reporting capabilities, reporting time should be adversely affected; the data may not be
reported until days or weeks after an event. Additionally, an outbreak may go undetected or unconfirmed
if there is no ability for a health department to analyze the data. It may be the case the health department
has sufficient resources to gather and organize the data on a relatively frequent basis, but may not have
the technical means or time to evaluate the information.
DRAFT-121205
87
-------
WS System Architecture
Utility and public health officials in a jurisdiction deploying WS-CWS should have coordinated,
cooperative two-way communication any time that a waterborne event resulting in health effects is
suspected or detected. This communication should usually include human conversation and may be
augmented by electronic notification as technical sophistication increases within the water utility and/or
public health offices of the area.
Assessment of existing collaboration methods should be initiated in advance of WS-CWS
implementation. Various techniques exist for performing and improving inter-agency collaboration;
minimally, the following questions should be asked as part of the assessment effort:
1. What is the current relationship between the utility and local public health department(s) within
the utility's service area? Is there a high degree of cooperation and collaboration? Are there
established communication protocols, and do they include electronic and 24/7 notification?
2. What types of health data are monitored and what are the sources of these data? What is the
timing associated with collection, transmission, receipt, and analysis of this information? What
are the data-use agreements associated with this information and are these agreements able to be
leveraged to support WS-CWS?
3. What are the current electronic surveillance systems used by the public health departments? Are
there capabilities for electronic notification by the public health surveillance systems of other
entities, such as the water utility and health care providers?
4. What are the current methods for submitting water quality data to the local public health
department(s) within the utility's service area, whether baseline or alert data? What options exist
for WS-CWS to integrate with existing public health systems?
Figure 7-1 provides a high-level characterization of notification and integration schemes based on the
information technology sophistication of water and public health organizations for a given area.
O)
Q.
O
CWS automated detection/alerting triggers
electronic (i.e. messaging, e-mail, pager) alerts
to Water and Public Health officials.
Telephone conversation follows per policy and
procedures.
Data latency from WaterSentinel may be
seconds to hours; latency from public health
may routinely be days.
Full data integration allows alerts to How
automatically. Additional e-mail / pager alerts to
Water and Public Health officials.
Telephone conversation follows per policy and
procedures.
Data latency should be less than 24 hours.
potentially as short as mere seconds. Data
duplication may still be caused by manual
^patient reconciliation.
Detection methods within water utility and
public health offices rely heavily on human
Telephone notification policies and procedures
in place and reviewed regularly.
Data latency may routinely be days in this level
of automation.
PH Surveillance automated detection/alerting
triggers electronic (i.e. messaging, e-mail.
pager) alerts to Water and Public I leallh
officials.
Telephone conversation follows per policy and
procedures.
Data latency from public health may be seconds
to one or two days. Data duplication may be
caused by manual patient reconciliation.
Low
Public Health Sophistication
High
Figure 7-1. Water Utility - Public Health IT Sophistication Matrix
DRAFT-121205
-------
WS System Architecture
Upon consideration of all factors, likely options for implementation include the following:
If less than significant automation exists at both water utility and the associated public health
department(s), policies and procedures for notification and coordination resulting from manual
detection methods should be implemented until a WS-CWS is available.
If significant automation does not exist within the public health department, WS-CWS should
implement an electronic data notification system according to coordinated policies and
procedures with the associated public health department(s) to provide alert notification. This 24/7
automated alerting system would provide for faster information exchange amongst the proper
water utility and public health officials.
Once WS and public health departments are sufficiently automated, the data submission and alert
notification should utilize a shared system with specially designed user screens for the public
health officials and the water utility.
Simultaneously analyzing integrated water quality data and public health surveillance data streams is the
ultimate goal for public health integration with WS-CWS.
Figure 7-2. WS integration with Public Health
7.3 Data Management, Analysis, and Interpretation
Accepting and integrating indicative, aggregate data from public health surveillance activities provides
opportunities for fine-tuning the WS-CWS detection and notification models as a post-event feedback
mechanism. The aggregate public health data should allow predictive models to be verified or adjusted in
order to allow better predictions to be provided by WS in the future; this data also provides additional
feedback on the accuracy of sensors in the affected area. Aggregated diagnoses can provide confirmatory
feedback but are typically available one or more days after the onset of an event. Aggregate lab test
orders and counts of categorized chief complaints can provide more timely indicators of possible health
events if provided on a real-time basis. Aggregate OTC sales data, while readily available from NRDM,
needs significant baselining and analysis to eliminate false signals.
Analysis of the integration between data from the WS-CWS implementation and from public health
should occur as a result of post-event activities. The profile of public health notifications should
optimally include the following, at minimum:
Aggregate indicator counts by severity, date, and sensor
Aggregate alert message type counts transmitted by date and triggering sensor(s)
This profile should be coupled with the WS-CWS data set that includes categorized consumer complaint
counts, sampling and analysis results, and physical security alert or intrusion counts in order to provide a
fully detailed information base from the water utility perspective; the WS data set and the WS algorithms
provide the basis for the notification profile. Analysis at this level of detail coupled with data received
DRAFT-121205 89
-------
WS System Architecture
from public health and correlated by date, message type, and geographic area as indicated by sensor
locations should provide the WS staff, the pilot utility, and WS-CWS, the ability to validate or improve
the configuration and control settings as well as processes and procedures within a given area. Without a
basis for careful and detailed feedback analysis, WS may be unable to improve its usability as a warning
system. More details on data analysis approaches are provided in Section 8.
7.4 Framework for Evaluation
As discussed in Section 2.4.1, system reliability, in the context of system operation and system
performance, and system sustainability provide a means to evaluate the effectiveness of the WS-CWS.
7.4.1 System Reliability
In terms of public health surveillance, system operation is generally reliable. BioWatch operation
reliability, for example, has depended mostly on the stability of the public health information network
(PHIN). Once data definitions, reporting protocols and monitoring systems are in place, they should need
minimal maintenance; computing issues such as server failures would account for the majority of
operation errors. While there may be a human component to system operation error (e.g., worker
absenteeism at a health department,) these events should be minor, especially as more data are transmitted
electronically. Electronic reporting software may require occasional updating and/or staff training.
System performance has the greater affect on public health surveillance, as it is conditional on a system's
ability to discern actual events of concern from those of other consequence. This ability is dependent on
the components of the public health surveillance algorithm; if these components are well refined, then
public health surveillance can aid in the detection and confirmation of outbreaks and become an important
event detection and quality assurance tool. However, if the algorithm is not satisfactory, the information
gathered from public health surveillance can become a liability towards the sensitivity and specificity of
contaminant detection. Three suggested components of a successful surveillance program algorithm are
(Buehler, 2004):
Timely and complete case reports and investigation (data quality)
The ability to recognize differences in data
Continual receipt of new types of data
Data quality, comprised of complete and timely reporting, is one portion of the event detection algorithm.
Expectations of what needs to be reported, as well as accurate definitions of these expectations, are
necessary. The participants in public health surveillance (e.g., health care providers, health departments,
hospital coding) should be speaking in congruent terms for any real understanding of events. For
example, if only vague symptoms are reported, then a relationship between cases might never be
established because a definitive connection is lacking. New York City utilized a standard data format for
ER visits to incorporate computer algorithms in its syndromic surveillance system to ensure data quality
for the purpose of detecting temporal and spatial clusters (Heffernan, 2004.)
The timeliness and accuracy of disease reporting also affects data quality. A lack of accurate disease
reporting can lead to over or underreporting of disease. If numerous data gathering sources are being
utilized (e.g., electronic reporting, paper reporting,) it is possible for one instance of disease or symptom
to be reported two or more times. There is also the potential for duplicate reporting in the event a person
is treated by multiple providers. Over reporting in public health surveillance could artificially inflate
events above that of the alarm threshold, resulting in unnecessary emergency responses Careful
DRAFT-121205 90
-------
WS System Architecture
monitoring of cases, a workable electronic reporting system and effective communication between data
reporting entities can minimize this effect on WS-CWS implementation.
Conversely, certain pathogens may have vulnerability towards underreporting. Pathogens that cause
gastrointestinal illness or vague 'flu-like' symptoms can be underreported or misdiagnosed due to lack of
severity. For example, the CDC estimates that for every reported case of salmonellosis, 38 actually occur
(CDC, 2005.) It has also been suggested that only 8% of those with gastrointestinal illness seek medical
care (Khan, 2001). However, most of the contaminant classes identified for WS-CWS create severe or
unusual symptoms that would increase the incidence of people seeking medical attention and decrease the
incidence of misdiagnosis. Further, deficiency of data can cause QA problems when trying to confirm
contamination events. Being conscious of this issue can ensure safeguards are built into the surveillance
system to account for lack of data.
The ability of public health professionals to recognize differences in data events is also an important part
of the surveillance algorithm. Data gathering is only as useful as the ability to analyze the data; having the
expertise to recognize when an increase of events is, in fact, an outbreak is imperative to the sensitivity
and specificity of a surveillance system. Analysis that does not take into account seasonal inflections, day
of the week changes, and other variables may mistake increases in events as an outbreak, when in fact it is
a normal occurrence. This may elicit false-alarms.
The inability to recognize an increase in events as a contamination event may have more dire
consequences, as preventative and containment measures should not be initiated. Having trained
professionals in positions of recognition is one of the major challenges of public health syndromic
surveillance today. Prior to September 11, 2001, it was estimated that 75% of local public health
administrators never received formal public health training (Gerzoff et al, 1999) While these numbers
may have improved, the CDC still recommends that a 45.3% increase in Epidemiologists is needed to
fully staff terrorism preparedness programs (MMWR, 2004). Recruiting trained persons into these
positions should increase the reliability of public health surveillance by increasing event recognition
abilities.
Receiving new types of data is critical to the new and developing systems of syndromic surveillance.
Integrating OTC sales data, worker absenteeism, or other types of non-traditional data sources can pose a
challenge for health departments. A report on the NMRD found that checking weekend data was still low
among health departments, suggesting surveillance of these data was a challenge (Wagner, 2004.) It is
also necessary to obtain a suitable number of participants in OTC and worker absenteeism surveillance to
ensure a proper sample size. New York City performs worker absenteeism on a company employing
15,000 in multiple locations; not all cities would contain a similar company, and may have difficulty
incorporating worker absenteeism data into their algorithm (Heffernan, 2004). Successful use of these
new data with more traditional surveillance methods should depend on the appropriate utilization of
statistical models and careful consideration of all components.
7.4.2 System Sustainability
Sustainability of a public health surveillance system within WS-CWS should be dependent on the cost of
public health surveillance as well as the maintenance of communications between health departments and
utilities. Cost of surveillance should vary greatly, dependent on the size of the health department, the
resources already available to them, and the depth of surveillance they wish to maintain. New York City's
syndromic surveillance system costs approximately $150,000 annually in maintenance (Heffernan, 2004.)
However, this amount would not be typical of the majority of health departments. Small and even
moderately sized health jurisdictions should be able to sustain a system on far less.
DRAFT-121205 91
-------
WS System Architecture
Sources of funding, particularly for smaller departments, may become an issue in sustainability, as they
are more dependent on local tax levies and grants for support. One study estimates that local taxes provide
43% of health agencies' revenues (Gostin, 2004.) A levy failure could be detrimental to the sustainability
of public health programs, including surveillance activities. Similarly, securing grants and other funding
has been especially difficult in poor, rural minority areas due to a lack of philanthropy and grant writing
effectiveness (Siegel, 2001). Gaining adequate funding may be a challenge for sustainability in some
areas. However, if the problems surrounding funds sustainability can be circumvented in parallel with
maintained cooperation between utilities and public health, then public health surveillance support to WS-
CWS can be sustained.
7.4.3 System Evaluation
The integration of public health surveillance as a component of the WS-CWS should be evaluated jointly
between the public health office and the water utility. Through collaboration and cooperation, the
effectiveness of the communication policies and procedures should be evaluated with a holistic approach
along the lines of the CDC's Framework for Program Evaluation in Public Health (Koplan, 2005):
Engage stakeholders: Consideration of those involved in program operations, of those affected
by program operation, and those who should take decisive action as a result of the evaluation.
Describe the program: Consideration and documentation of the program's need, the program's
expected effects and activities, the program's resources and current stage of development, and the
program's operational context and high-level logic model.
Focus the evaluation design: Concentration on the program's purpose and uses, its user base, the
questions about the program to be addressed in the evaluation, the methods of evaluation to be
utilized, and the agreements in place (or to be established) regarding how the evaluation should
be executed are key.
Gather credible evidence: Identification and assessment of indicators (aspects of the policy /
procedure worth monitoring), sources (from where the evaluation evidence is provided), quality
(appropriateness and integrity of the evaluation evidence and its collection methods), quantity
(amount of evidence collected), and logistics (details around evaluation evidence collection)
should provide significant benefits in the evaluation process. Each organization can utilize
system testing and preparedness drill scenarios to generate data sets for use in pre-implementation
evaluation as well as in post-implementation calibration exercises. Additional evaluation data
may be collected and recorded post-implementation when an event is identified by whatever
means in order to support the calibration exercises and evaluation of system improvements.
Justify conclusions: The methods of analysis, interpretation of the results provided by the
evidence, the judgments based on the analysis and results interpretation, and the
recommendations from those judgments.
Ensure use and share lessons learned: Documenting the evaluation's design and preparation,
recording feedback and follow-up from the evaluation, and dissemination of the knowledge
gained from the evaluation effort.
At a high level, evaluation of what works, what does not work, and what improvements are necessary and
feasible should be performed on regular intervals as part of the policy and procedure review by the joint
working group between the water utility and the public health office(s); the joint working group should
routinely ask, 'How effectively does the defined notification process indicate or validate a health event
occurrence?' This sort of question directly relates to the design basis defined in Section 4.1. Public
health officials can use this process to determine how often a water-sourced event was detected by WS,
and water utility officials can similarly determine how often a detected (or undetected) event is validated
(or indicated) by analyzing aggregate data from the public health department.
DRAFT-121205 92
-------
WS System Architecture
The current capabilities of each office should be well understood by its officials and the joint working
group in order to provide a roadmap and project plan for reaching the next level of desired integration;
policies and procedures should be prepared for update in accordance with proposed improvements in
system integration, whether manual or automated. Leverage of existing or new evaluation tools or
processes and the existing expertise of both organizations are critical to the success of long term
cooperative efforts.
DRAFT-121205 93
-------
WS System Architecture
Section 8.0: Information Integration, Management, and
Communication
A key to the success of the WS-CWS is the effective and timely management of information. Information
management in WS begins with source data collection, but this information passes through a variety of
phases in order to ultimately support decision-making. This section focuses on the collection and
transmission of data, the integration and evaluation of this information, and finally the communication of
this information to the appropriate personnel so that action can be taken.
8.1 Data Collection and Transmission
The data available to WS come in a variety of forms and formats. Each data source should be collected
and transmitted appropriately, generally using means specific to that data source. The guiding design
principle involves using existing data collection and transmission methods as much as possible, and
augmenting these methods when necessary to produce additional or enhanced data flows. These methods
are summarized in the context of WS information management in Table 8-1. Each of these methods has
inherent limitations that should need to be addressed in order to increase the robustness of the system.
Table 8-1. Summary of WS Information Streams for Managing Data
WS-CWS
Component
Online Water
Quality
Monitoring
Sampling and
Analysis
Consumer
Complaint
Surveillance
Public Health
Surveillance
Enhanced
Security
Monitoring
Source
Utility Water
Sensor
Utility Field
Sample
Collector,
Analyzer
Utility Consumer
Local Health
Department
Utility Systems,
Individuals, Law
Enforcement,
Media,
Perpetrators
Collection
Sensor, SCADA
Collector Notes,
Chemical Analysis
of Sample by
Machine
Phone, written,
email, in person
Lab, Observation,
Phone, written,
OTC sales
Security systems,
cameras,
Manual
observation, Phone,
written, email
Local Storage
Intermediate
communications
interfaces
PDA, laptop (chain
of custody data),
Sentinel,
Confirmatory
Laboratory LIMS
Email database,
message pads,
hard copy files
Spreadsheets,
notes, database
silos
Email database,
Local digital video
storage
Transmission
Licensed and unlicensed
radio, frame relay, digital
subscriber line (DSL),
cable television digital data
service, cellular telephone
digital data service
Cell phone, Pager, PDA,
Email Automated
Manual and Automated
Software Data Entry
Systems,
Call / Defect Reporting &
Tracking Software
Telephone, cell phone,
pager, e-mail, electronic
transfers
Digital video transmission,
SCADA, Manual and
Automated Software Data
Entry Systems,
Call Reporting & Tracking
Software
Central
Storage
Data
Warehouse,
SCADA
Laboratory /
Utility DB
(LIMS)
Central
Database,
Audio Tapes
Public health,
OTC databases
Central
Database,
Video Tapes /
Digital Storage
Online water quality monitoring data originate as a signal from a sensor, which is transmitted to a remote
communications interface for processing, including digital signal processing and aggregate and summary
calculations. These data are then transmitted to a central communications interface which further
processes the data for optimized transmission, and finally to a data management module for storage and
analysis. These data provide input to the event detection software for determining whether an anomaly
DRAFT-121205
94
-------
WS System Architecture
trigger has occurred. Transmission can occur via a variety of formats, such as radio, cellular, or cable or
DSL internet service. A SCADA system, if available, can be used for this last step.
Sampling and analysis data originates from a field technician collecting a lab sample. Field record forms
and chain-of-custody forms are filled out manually on site and subsequently entered electronically (PDA,
laptop, desktop). A sentinel and possibly confirmatory laboratory should conduct an analysis of the
sample, resulting in analytical data stored electronically alongside the field data. For chemical analyses,
the data are produced by automated tools that analyze the sample, and the data are saved into a local data
store (spreadsheet style or database, such as a LIMS). For most biological analyses, the data are recorded
manually and entered into a local data store. Ultimately the data are transferred to the central utility
database in electronic fashion for storage.
Consumer complaint data typically originates from phone calls, emails or written mails. Initial call
information is collected by operators using call tracking software. Written mails or emails are often
entered into the same system. This type of system then typically provides reporting, routing, and email
capability so that the call information can be transmitted to the appropriate personnel for further analysis.
Enhanced security monitoring data vary widely in nature and formats are collected through a variety of
manual and automated methods. The data itself may be notifications of incidents such as break-ins, or
may be video feeds from a camera. The providers of this data may be individuals or automated systems.
Examples of individual data providers include witnesses, perpetrators, the media and law enforcement
agencies. These individuals may provide information via a phone call, mail, email or other means.
Examples of automated data providers include alarms, security systems and video surveillance. These
data providers should typically communicate via automated means using distributed control panel
warnings, pagers, remote monitors and other methods. Some data, such as video data, may not be readily
available but should need to be downloaded from the video data storage site on demand.
Public health surveillance data are collected via a variety of sources, such as laboratory data, chief
complaints from hospital visits, 911 calls, poison control calls, and OTC sales. Systems that conduct
public health surveillance are already in place to varying degrees. In addition, much of the data that is
available from public health surveillance systems is available only in aggregated fashion due to HIPAA
regulations developed to protect the privacy of individual patients. Thus, while the true data source for
these data streams is widely varied (phone calls, lab tests, physician notes, OTC sales), the collection and
transmission of these data streams for and to WS should likely be less dynamic. These data streams
should likely be aggregated before transmission. Transmission methods will likely be electronic and
asynchronous, due to the nature of the data processing that is likely to occur. The transmission may be
automated to some degree, dependent on the technological advances of both the source data providers and
the utility. Manual communications (telephone, cell phone, pager, e-mail) should also typically be
present to transmit more urgent communications.
A data management plan should be established in order to address how data should be managed
throughout the CWS. The specifics of each datatype, including source, destination, and collection,
transmission and storage methods (as summarized above), should be presented in detail to illustrate
generally and specifically how data flows through the system. An example general consumer complaint
data flow is presented in Figure 6-3. Roles (field sample collector, law enforcement, call center operative,
data analyst, etc.) should be identified and explained in detail. Primary and backup representatives should
need to be identified. Data types and formats should be listed, with references to supplementary
documents for further detail as warranted (i.e., explanations of lab analysis results). Primary and (ideally)
backup mechanisms for each data collection, communication, and transmission point should need to be
identified. Data sharing agreements should be listed and should need to be implemented and executed.
DRAFT-121205 95
-------
WS System Architecture
Vulnerabilities for each data collection, transmission, and storage point should need to be identified and
addressed. Other data management design issues that need to be considered include data privacy,
sensitivity and security, authorization, encryption, timeliness, cost, redundancy and availability.
A data management plan should be established in order to address how data should be managed
throughout the CWS. The specifics of each datatype, including source, destination, and collection,
transmission, and storage methods (as summarized above), should be presented in detail to illustrate
generally and specifically how data flow through the system.
8.2 Integration and Analysis of Information
Evaluating information in a timely and successful manner is critical to the success of the WS-CWS, and
the ability to make appropriate response decisions in time to reduce consequences. With many different
data sources, data can be evaluated at a number of different levels. This evaluation can occur for each
single data source, and for a combination of data sources. The data sources may be integrated within each
data provider (i.e., utility or local health department) or across data providers (utility and local health
department). This section discusses the possible levels of integration and evaluation, the forms of
integration and evaluation, and the feasibility of automating the processes of integration and evaluation.
As mentioned in previous sections, a variety of data sources exist for each data provider. A local utility
should typically have data collection systems such as SCADA, LIMS, hydraulic distribution system
models, consumer complaint databases, water quality databases, and work order systems.
A local health department should typically have data collection systems such as chief complaints from
hospital visits, EMS records, and calls to 911 or poison control. Over-the-counter sales of pharmaceutical
and other items are also often available, as well as access to more broad surveillance systems such as
RODS, ESSENCE, BioSense and/or the state's National Electronic Disease Surveillance System
(NEDSS). Each of these data streams has different attributes regarding data format, data size, collection
frequency, collection mechanism, storage mechanism, and others.
The first step in enabling the integration and evaluation of this information is identifying the attributes of
each data stream. In the context of the WS-CWS, it may be necessary to evaluate each data source
independently for the purpose of initial event detection. For example, data from the online water quality
sensor network should need to be evaluated independent of other primary data streams in order to identify
water quality anomalies that might indicate a possible contamination incident. This may allow detection
of abnormal levels of contaminations or other triggers in that data source alone. Similarly, call center
data can be evaluated independently for abnormal volumes of calls, common call complaints, and other
events. However, even at this level, some integration and analysis of information are necessary to
produce more reliable triggers and reduce the number of false alarms. For example, the analysis of
unusual water quality data from a single sensor may benefit from data from other nearby sensors,
information from a work order system regarding recent maintenance activity in the vicinity, and
operational data from utility SCADA (e.g., tank and pump operations that could change water quality).
This first level of evaluation is critical because initial detection of an anomaly should likely occur at the
individual WS-CWS component level. Also, the reliability of signals from the individual components
should have a significant impact on the rate of false positives and false negatives for the entire system.
DRAFT-121205 96
-------
WS System Architecture
Generally, there are three levels of data integration involved in the design, presented in Figure 8-1:
First Level: Integration of a primary data stream (e.g., water quality data) with supplemental
information from ancillary data sources and systems (e.g., work order systems), all within the
domain of the data provider. This first level of integration is important for improving the
reliability of event detection and minimizing the number of false alarms. Ideally, this level of
data integration and analysis would be automated as part of an event detection system, and thus
would be part of the actual 'triggering mechanism' for a possible contamination incident.
Second Level: Integration at the data provider level of multiple primary data streams (e.g., water
quality data and consumer complaint information). This level of integration is important for the
initial stages of credibility determination and threat corroboration. From a data management
perspective, the difference between the first and second levels exists on an operational basis - the
first level of integration occurs automatically within the context of an event detection system,
while the second level of integration is a manual process that requires a person to connect the
dots.
Third Level: Integration of information across multiple data providers. This should primarily
occur between the utility and public health, but might also involve law enforcement and other
agencies. This level of integration is critical to establishing the credibility of a potential
contamination incident (or ruling it out).
.Q
3
Q.
Monitoring and
Surveillance
r^~~- ~-==~
Online Monitoring
Sampling and 1
1 Analysis Data J~
Consumer
Complaint Data
Enhanced
Security Data J
r==~ ~==~
Chief Complaints
[_____ Data _____
91 1 calls Data
Poison Control
Data
OTC sales
Data J
S
ev
^
*
ngle-source
;nt detection
Detect
event
Detect
event
Detect
event
Detect
event
Detect
event
Detect
event
Detect
event
Detect
event
Event
Detection
Integrate
data
_ i
Utility
Integrated
Event
Detection
Public health
Integrated
Event
Detection
A
Integrate
data
i
i
Man
corrot
dete
ev
Cross-F
Integrate
Dete
r
ysis
ults
ually
orate
cted
3nt
Credibility Determination
r
Integrate
data
r
rovider
d Event -
ction
i
*
^
Determine
credibility
of detected
event
Integrate
data
Manually
corroborate
detected
event
i
i
i<
^=^
.J Analysis
ultej
Determine
credibility
of detected
event
Integrate
data
Determine
credibility
of detected
event
Possible Future
Capability
Cross-Provi
Q.
ID
Figure 8-1. Data Integration for Event Detection and Credibility Determination
DRAFT-121205
97
-------
WS System Architecture
The first two general levels of data integration are at the data provider level. This could be at the utility,
the local health department, a Sentinel laboratory or other participant in the WS-CWS. Each of these data
providers may contain multiple primary data sources, each of which can be relatively quickly reviewed at
the data provider level, ideally in an automated or semi-automated fashion. For example, a utility
contains in-house data regarding online monitoring, operational data, consumer complaints, sampling and
analysis, and enhanced security monitoring. As part of the event detection process, online water quality
monitoring data should be analyzed in and of themselves, but should also be supplemented by and
integrated with other data streams for corroboration of possible events.
This allows data integration to occur relatively quickly at the utility level, as most of the data are in-house
and available to the appropriate utility personnel. Initial efforts should focus on integrating these data
sources manually - obtaining the data from each utility data source manually, and integrating them
manually so that they may be evaluated in the context of other information from the WS-CWS. It is also
possible to conduct the integration in a semi-automated fashion, so that certain steps of the overall
integration are automated (for example, data retrieval, pre-processing or presentation), but manual
intervention is needed for full integration. Ideally, information integration should be completely
automated at the utility level, so that information from all utility data sources should be collected, pre-
processed and presented to the data evaluator(s) at once. This provides a quicker response capability at
the expense of upfront development cost.
It is useful to evaluate the data-provider integrated set of information separately from each independent
data source. This potentially allows the detection of events that cannot be identified from analyzing a
single data source. Integrated data evaluation can be conducted manually, automatically, or in some
combination thereof. While there are tools and algorithms that can be used to evaluate integrated data, it
should take some time to properly design, train and/or implement these tools as part of the WS-CWS. It
is reasonable to conduct this evaluation manually as a first step, while working towards automated
analysis and evaluation of integrated information. A utility domain expert can manually evaluate multiple
data sources, integrated at the utility level, for anomalous events. It is possible to ease the manual
analysis burden through presentation technologies that present the information in a manner that may more
easily be analyzed. For example, graphical tools can present and overlay time-series data from multiple
sources, and GIS tools can be used to present the information geographically. These tools can enable a
manual data visualization step to assist event detection. Initially, pilot utilities should rely on manual
integration of most data streams, leveraging existing tools, such as GIS platforms, where possible. Fully
automated integration of the various data streams should be a focus of supporting research throughout the
WS pilots.
The third general level of data integration involves integration of data across data providers such as the
utility, local health department, and sentinel laboratories. Generally, this data and information integration
should be used as part of the credibility determination process, to corroborate events that have been
previously detected. However, it is also possible to conduct an integrated analysis of all possible data
sources at once, perhaps enabling the detection of events that would not otherwise be identified. For
example, 911 calls, ER physician data, online water quality monitoring, and OTC sales may be analyzed
in unison. Enabling this integration is difficult because of data privacy concerns and regulations such as
HIPAA. Public health data are typically available only at roll-up levels, such as yearly and by the first
three digits of the zip code. It may be possible to obtain more detailed data (daily, by zip code) through
the use of data-sharing agreements. Integration of this data can occur by data sharing between entities (a
utility sharing data with the local health department, and vice versa), or by data sharing with an
independent entity. Most likely, the initial cross-entity integration efforts should involve automated (e.g.,
daily) delivery of data dumps from local servers to remote servers. Special consideration should be paid
to ensure that data transferred between providers can be interpreted by the destination system. This
DRAFT-121205 98
-------
WS System Architecture
should generally involve the use of agreed-upon data standards to represent the data (typically, using
extensible markup language (XML)), as well as software systems to transform the source data into this
format, and to transform the received data from this format to a format that can be understood by the
destination system.
Evaluation of data streams across data providers follows the integration of these data streams. While it
may be possible to use similar data analysis techniques across data providers as within a single data
provider, the nature of the information should likely require more coordination across the data providers
to provide reasonable results. Recent advances in privacy-preserving data mining may allow the sharing
of more detailed information while still addressing privacy concerns, but it is likely that in many cases,
only aggregated (by location, by time, etc.) data should be available. It should be useful or even
necessary to have domain experts (utility, health department) on hand to properly present and/or analyze
the source data streams. Ultimately, this level of analysis should likely be manual for some time, until
data sharing agreements can be put in place and automated analysis tools can be properly designed, tested
and deployed. Pilot efforts are underway to perform this sort of integrated analysis; these projects are
described in further detail in Overview of Event Detection Systems for WaterSentinel (EPA, 2005k).
8.3 Communication of Information
A CWS should contain many types of information that should need to be communicated. These types of
data include source data, aggregated data, integrated data, results of analysis (manual or automated),
internal recommendations, and public notifications. Each of these types of information has a different
target audience and a variety of possible communication mechanisms. Previous sections have focused on
the transmission of source data from primary data collectors. This section describes the communication
of information - typically, this should be information that has become available after some data analysis
has occurred, during event detection and consequence management phases.
There are many recipients of information in a CWS. These recipients include operators, technicians, data
analysts, decision-makers, action-takers and the general public, including the media. Each recipient has
different information needs with respect to content, format, frequency and timeliness. During the event
detection phase, data analysts would need to communicate with each other and other technicians and
operatives as part of the initial corroboration. This communication should occur both within and across
organizations including the utility, public health agencies, and response agencies. The credibility
determination phase should see similar communication as more data and information is shared at higher
levels of the organizations. This may involve several data analysts across multiple jurisdictions, in order
to provide multiple opinions, ideally reach some sort of consensus, and communicate this information to
decision-makers. The consequence management phase should involve communication of information
between many of the roles listed above; this phase and credibility determination are described in more
detail in WaterSentinel Consequence Management Strategy (USEPA, 2005i).
There are many possible communication mechanisms which may be employed, such as land-line, pager,
cell phone, satellite communications, radio, television, internet, and reverse 911. The type of
communication mechanism is dependent on the information provider, source, recipient, content, format,
timeliness, and other requirements. Many of the communication pathways fall into common use cases,
such as within a data provider (e.g., utility), across data providers or jurisdictions (utility to public health
department, and vice versa), and the public at large. For example, communication between data analysts
and decision-makers during event detection and credibility determination would likely take place via
voice calls, with supplementary data transmitted electronically. Communication during consequence
management should take on many forms, depending on the message and target audience. Emergency
broadcast warnings to the public can use well-established communication mechanisms, such as radio and
DRAFT-121205 99
-------
WS System Architecture
television. This information may also be published on relevant internet web sites. Targeted communities
of households, businesses or other facilities can be reached via systems such as reverse 911, which can
relatively quickly place land-line calls to specific geographical areas.
Redundant communication mechanisms are necessary to reduce the likelihood of communication failures
due to breakdowns. For example, a warning from a public health operative to utility operatives should
take place over multiple communication channels - land line, cell phone, pager and email. Multiple
designated contacts should be identified for key transmissions, as well as someone who is always
available (on call) to receive the transmissions.
A communications plan is needed to encapsulate the use cases, sources, recipients, contents, formats and
mechanisms of communication that are envisioned. This plan should outline the procedures by which
communication can and should take place, both within and across organizations, to ensure that responders
can respond in a timely fashion. The hardware (i.e., cell phones, supporting hardware and software)
necessary for each communication should need to be identified. Special consideration should need to
given to systems that communicate automatically, to ensure that the systems are interoperable across
different jurisdictions, hardware and software systems.
DRAFT-121205 100
-------
WaterSentinel
Section 9.0: Approach to Evaluation
Evaluation is a key step in both the design and implementation of the WS-CWS. As part of the WS-CWS
pilot demonstration project, EPA plans to develop an evaluation plan. Through the development of the
WS system architecture, considerations for the technical and programmatic evaluation of the WS-CWS
pilot were documented and are discussed below. Section 9.1 describes considerations for the approach
for a technical evaluation of the WS-CWS and Section 9.2 describes considerations for the approach for a
programmatic evaluation conducted from the perspectives of EPA, the pilot utility, and key stakeholders.
Independent evaluations may also be valuable. Information documented within this section is not meant
to be exhaustive; rather it should be viewed as an initial framework for development of the WS evaluation
plan. In general, the objectives of this evaluation should be to:
Provide ongoing assessment and feedback into the design of the WS-CWS
Ensure that the WS-CWS is implemented as planned and achieves all of the program's goals
Document enhancements/changes that were made as a result of the pilot demonstration project
Document benefits and costs of the WS program, based on the initial pilot demonstration project
9.1 Technical Evaluation
As illustrated in Figure 1-2, continual evaluation during the WS pilot is a key part of the success of the
project. In this respect, evaluation is not an end unto itself. Rather, the evaluation is intended to
strengthen and support the WS pilot as well as inform future guidance for the implementation of a CWS.
The process for conducting a constructive technical evaluation as part of the WS-CWS pilot includes the
following steps:
1. Determine where WS should achieve results based on the hypotheses identified in Section 1.1.2,
2. Define evaluation criteria for these areas. The nature of these criteria varies with the area
evaluated. For some areas, the criteria are related to the desired goals in the area. For others, the
criteria are related to performance metrics within that area,
3. Identify an evaluation methodology that can be used for each evaluation criteria. Depending on
the nature of the area, this can range from a qualitative analysis of performance toward goals to
using quantitative metric data directly,
4. Develop systems and procedures to collect and evaluate the necessary data, and
5. Utilize these results to help refine and develop the WS model.
As the initial step in this evaluation process, the areas in which WS should achieve results are the
elements of the WS ConOps, illustrated in Figure 1-1. Important considerations for the development of
evaluation criteria for monitoring and surveillance, event detection, credibility determination, response,
and remediation and recovery, as well as proposed methodologies for data collection and evaluation are
discussed in Section 9.1.1 - Section 9.1.5.
Each element of the system should be evaluated using specific criteria; for monitoring and surveillance,
each of the WS-CWS components should also be evaluated (i.e., online monitoring, sampling and
analysis, enhanced security monitoring, consumer complaint surveillance, and public health surveillance).
This is a necessary step in the overall evaluation process because each element has unique attributes that
are important to its functionality. This evaluation may also help to identify and mitigate potential
challenges with the overall system. In addition, evaluation of each of the elements of the WS ConOps
embodies the evaluation of the linkage between the design of the system ('system architecture') and the
functioning of the system ('consequence management'). The discussion below attempts to separate the
elements to the greatest extent possible, however, overlap in the evaluations should exist, resulting in the
need to employ a holistic, yet focused, technique in evaluating the WS ConOps.
DRAFT-121205 101
-------
WS System Architecture
9.1.1 Monitoring and Surveillance
As described in Section 1.0 of this document, the fundamental concept underlying WS is the collection,
management, analysis, and interpretation of different information streams in a timely manner such that
possible contamination incidents can be detected early enough to respond effectively and reduce
consequences.
Specific considerations for the evaluation of each monitoring and surveillance component: online water
quality monitoring, sampling and analysis, enhanced security monitoring, consumer complaint
surveillance, and public health surveillance, are discussed in Sections 3.0- 7.0. Thus, the evaluation of
monitoring and surveillance activities from a systems perspective should focus primarily on the overall
data management, integration, and analysis across all monitoring and surveillance components.
Evaluation Criteria. For information collected from monitoring and surveillance activities conducted by
both the utility and public health, the evaluation should mainly be one of data availability and
management. In this regard, the evaluation should be set in the larger context of data flow throughout all
elements of WS because this aspect of the evaluation goes beyond just the database operability illustrated
in the WS ConOps. Rather, it speaks to all the steps from data collection and transmission, integration
and analysis of information, and finally accurate communication of the information to decision-makers
and other individuals, organizations, and entities involved in monitoring, surveillance, or response related
to a contamination threat or incident.
Evaluation Methodology. This portion of the evaluation may consider the information technology
involved as well as the information sharing culture of the utility and public health community. Evaluation
and iterative improvement regarding the interaction of the databases should likely be a far more
challenging process, because these databases are, in most localities, maintained by different entities.
Thus, the evaluation should focus on the ability of the databases to communicate, both from an
information technology and a policy perspective. The evaluation should improve understanding of:
Methods to improve compatibility of the various monitored by the utility.
Methods to promote the coordinated sharing of information between the utility and public health
officials regarding potential incident triggers.
In any case, the process should be straightforward because the goal is simpleto ensure that the data
flows to its intended user in a timely manner. It is likely that shortcomings in the data flow should be
readily apparent, so evaluation and iterative improvement should likely occur as the pilot progress and
anomalous water quality events, actual or speculative, arise.
9.1.2 Event Detection
Event detection can apply to any of the monitoring and surveillance components of the WS-CWS, albeit
with different levels of sophistication. Due to the varying and sometimes complex nature of the data
streams, computerized algorithms as well as human judgment and interpretation should likely be
necessary for WS event detection. For example, relatively simple 'event detection' procedures for
consumer complaints and enhanced security monitoring could involve simple decision trees, set points, or
SOPs. For more complex data streams such as those from water quality sensors or public health
surveillance system, a sophisticated, computerized algorithm may be a better choice, and perhaps the only
choice. An algorithm is the mathematical operation or statistical technique that is performed on the data
received (e.g., signals sent by water quality sensors) and is incorporated within the event detection
software or tool that interfaces with sensors, other data streams, and other utility software. Event detection
algorithms are applied to the data to filter out the anomalies that normally occur, or which have known
causes, and signal only those events that are likely to be possible contamination incidents.
DRAFT-121205 102
-------
WS System Architecture
Evaluation criteria. Regardless of the method of event detection, there are a number of metrics that can
be used to evaluate how well the particular type of event detection worked. These metrics include the
following:
Interpretability/Integratability of event detection. WS relies on the flow, management, and
integration of data, ultimately resulting in correct interpretation and accurate communication of
information about the potential incident. Thus, a critical criterion for evaluation of an event
detection system is whether the results from this system can be readily interpreted and integrated
with other information available to the users of the information, such as decision makers and
re spenders.
Resource requirements. This measure applies to the costs incurred as a result of time, labor, and
consumables expended during the installation, and operation and maintenance of the event
detection system, and in responding to an event trigger. This metric can also be used to track the
costs associated with the execution of the event trigger protocol to determine whether the
expenditures were commensurate with cause of the trigger (e.g., an event trigger that leads to a
discovery by the utility that a sensor calibration problem is the cause without the implementation
of a drastic response action like a 'do not use' order is indicative of good protocol because the
resources expended were not excessive).
Ability to handle highly variable data. The monitored data streams are influenced by many
factors (e.g., seasonal factors, source water, and treatment variables) and baselines should show
significant change daily, weekly, and seasonally. Event detection system tools and/or algorithms
should have the ability to handle these highly variable data to be effective across time ranges.
Adaptivity. This parameter represents the extent to which the system can learn on its own, as
opposed to having a need to be re-trained over time. Adaptivity is valuable in a system because it
reduces the amount of off-line re-training or adjustment needed.
Sensitivity. In the context of event detection systems, the sensitivity of a test is related to the
proportion of contamination incidents detected by the event detection system relative to all the
contamination incidents that occurred during a given time. In terms of false positives and
negatives:
Sensitivity = (number of true positives) / (number of true positives + number of false
negatives)
Specificity. In the context of event detection systems, the specificity is related to the proportion
of time the system is detected to be without contamination relative to the time the system is free
of contamination (excluding false negatives). In terms of false positives and negatives:
Specificity = (number of true negatives) / (number of true negatives + number of false
positives)
F-measure. The F-measure relates sensitivity and specificity in a single measure of performance.
The F-measure is the harmonic mean of sensitivity and specificity; that is:
F = (2 x sensitivity x specificity)/ (sensitivity + specificity)
Timeliness of data. This refers to the ability to provide meaningful information in a timeframe
appropriate for implementing response actions.
This more detailed evaluation plan for event detection systems has been developed as part of the pre-
selection process to provide a basis for selection of EDSs that should be evaluated as part of the WS-CWS
DRAFT-121205 103
-------
WS System Architecture
pilot. Many of these same aspects also should be considered in the evaluation of other elements and
components of the WS-CWS.
Evaluation methodology. Some event detection systems can be evaluated through model simulations, or
even live simulations (see Section 3.4 for a discussion of evaluation for online water quality sensor
networks). However, due to the large number of event detection systems and evaluation criteria, it is not
efficient to discuss each combination here. Regardless of how they are evaluated, the evaluation
methodology should focus on the type of data, i.e., quantitative or qualitative, that is inherent to each
criterion. For example, the four criteria of interpretability/integratability, the ability to handle highly
variable data, timeliness, and adaptivity should likely produce a quantitative result in terms of their
capability to deliver an answer. However, a more thorough evaluation would involve a qualitative
investigation to elucidate the causes of the success or failure of the system. Only in this manner can any
necessary improvements be made. For example, careful evaluation of actual operations should likely
elucidate correctable bottlenecks impacting timeliness that could be easily overlooked.
Some evaluations should be essentially quantitative, such as those for resource requirements, sensitivity,
and specificity. Of these, resource requirements are the most straightforward because the metrics of
sensitivity and specificity are quantitative and inherently linked. One valuable tool for helping evaluating
the relationship between the two is through the use of receiver operating characteristic (ROC) curves.
The performance and reliability of an event detection system depends on the ability to minimize the
number of erroneous 'detections' of a contamination event (i.e., false positives) while avoiding the
erroneous 'non-detection' of a contamination incident that has actually occurred (i.e., false negative). For
example, for water quality sensors, false negatives are associated with such factors as improper sensor
selection and placement, lack of instrument sensitivity at low contaminant concentrations, interference
caused by background noise, and insufficient data analysis capability. False positives are associated with
oversensitive data streams that generate an indication of contamination when none exists. They can also
be caused by the presence of interferences that mimic actual contaminants within the sensor or by
inappropriate event detection system algorithms.
The generation of ROC curves is a means of visualizing the likelihoods of false negatives and false
positives from an event detection system. These curves are produced by plotting sensitivity versus
specificity. An ideal event detection system would have zero false negatives (i.e., 100% sensitivity) and
zero false positives (i.e., 100% specificity). In reality, such an ideal situation cannot be achieved. For
example, the use of low detection limit sensors would represent a situation where the sensitivity
approaches 100% (i.e., minimal false negatives because the ability to detect has been sharpened), but this
heightened ability to detect increases the likelihood that a non-contaminant would trigger the sensor (i.e.,
a false positive) and as the number of false positives increase, the specificity would drop. Because the
consequences are much greater if an actual event is missed (i.e., a false negative), a certain percentage of
false positives should be acceptable. However, the consequences of a false alarm can be significant,
particularly if they result in substantial response actions. Thus, the false positive rate should be
minimized to the greatest extent possible.
At a conceptual level, the ROC curve shows that the ability to detect contamination incidents and the
level of false alarms are inextricably linked, and have a positive and usually non-linear relationship. The
construction of a ROC curve requires that a set of events exists in a form that can be used to test the event
detection algorithms. The actual ability to construct a ROC curve from WS pilot data may be a
formidable challenge and may not be possible due to the complexities of the variables, in addition to
obtaining enough data from contamination threats and incidents to produce a statistically valid result.
Nonetheless, it is a worthwhile component of the WS evaluation, and even if it should not be as
DRAFT-121205 104
-------
WS System Architecture
productive as hoped, the data gathered to perform the evaluation may themselves point to the optimal
approach for evaluating the event detection element of WS.
9.1.3 Credibility Determination
At first glance, evaluation of the credibility determination element of WS would simply seem to be based
on its success in answering the question 'Does a credible threat exist?' The appeal of a yes-no answer
discounts the complexity of the overall process of managing the evaluation of a threat. The following
simple model describes the credibility determination process in terms of input, evaluation, and output:
Input = all available information relevant to the contamination threat.
Evaluation = systematic evaluation of the collective information to determine whether or not the
water supply could have been contaminated. It is important to consider all available information
as a whole such that any one individual piece of information does not drive the entire decision
process.
Output = conclusions of the threat evaluation (i.e., has something actually happened?).
Credibility determination is a progressive process that is considered in three stages (or decision points);
'possible,' 'credible,' and 'confirmed.' It is also an iterative process in which the credibility of threat is
re-evaluated as additional information becomes available.
Evaluation Criteria._WS does not provide an automatic mechanism for managing a threat; rather, it can
provide additional information to help make a credibility determination. If WS is providing high quality
data streams and these streams can be interpreted optimally, the credibility determination ideally would
be made more quickly and with greater confidence. The evaluation of the credibility determination
process in the context of WS should mainly focus on the application of this process in the presence of
additional information collected from the WS-CWS data streams. It should be the quality of these data
streams, and the ability to quickly and effectively integrate the information from multiple data streams,
that ultimately determine the reliability and performance of the credibility determination element of WS.
Important criteria in this regard include:
Measurement of response times between event trigger and credibility determination.
Measurement of response time between credibility determination and event confirmation.
Evaluation of the efficacy of data integration and the ability of this integration to inform response
decisions, and possibly some assessment of the 'correctness' of the credibility determination.
The latter is important for evaluating the overall 'false positive / false negative rate' of the entire
WS-CWS.
Evaluation Methodology. Effective credibility determination would best be demonstrated through actual
or simulated incidents. These incidents need not include just intentional contamination, but could arise
from unintentional contamination or changes to the water system the may indicate the need for changes in
the water system operations. The later reflects the potential for 'dual use' benefits, such as a previously
undiscovered need to boost chlorine levels in parts of the system during certain times of the month.
Conducting tabletop exercises and drills should also be necessary to exercise the pilot utility's process for
credibility determination. Some evaluation of these drills in terms of how well the credibility
determination element performed should be necessary. The results of these drills should improve the
credibility determination process that takes place for actual incidents.
9.1.4 Response
Response actions within the context of WS are depicted in Figure 1-3 in the areas of operational response,
public health response, and risk communication. Appropriate response actions vary with the stage of the
threat evaluation. The evaluation of the response and the application of 'lessons learned' through the
DRAFT-121205 105
-------
WS System Architecture
evaluation is perhaps of greater importance when response actions are needed during the 'credible' phase
of the threat evaluation, as compared to the 'confirmed' phase. In the former case, it is not necessarily
known with as high a level of confidence if data from WS do indeed indicate the presence of actual
contaminants. Accordingly, response actions should be chosen carefully to avoid unnecessary alerts to
the public when there is no confirmed contamination, which if too frequent, would cause the public to
loose confidence in the CWS and the drinking water utility. Some response to contamination threats is
warranted due to the public health implications of an actual contamination incident. However, a utility
could spend a lot of time and money over-responding to every contamination threat, which would be an
ineffective use of resources. Furthermore, over-response to a contamination threat carries its own adverse
impacts, like a loss of confidence in the CWS by partners (public health agency, etc).
Evaluation Criteria. Although response actions are conceptually different from credibility determination,
the two are operationally linked. Therefore, some of the same evaluation criteria described in Section
9.1.3 should be used. The evaluation of response actions should be largely qualitative and focus on how
appropriate and timely these responses were, given the nature of the incident and the stage of the threat
evaluation. For example, if the incident involved contamination that potentially impacts a substantial
portion of the population, did it provide optimal public health protection? On the other hand, if the
incident reflects a disturbance in plant operations that impacts water quality, but not short-term public
health, did the response convey this condition to the public in a proper means and context so as to not
diminish public confidence? It is difficult to generalize in advance the specifics of the evaluation of this
element of WS, because the specific nature of the incident cannot be predicted.
Evaluation Methodology. As with the evaluation of 'credibility determination', the evaluation of
'response actions' should likely be based on a mixture of actual or simulated incidents, along with table
top exercises and drills. The evaluation of response actions taken during the credible phase should occur
in the context of the processes and procedures identified in the consequence management plan. The
results of the evaluation, particularly of responses during the pilot program, should likely serve to inform
local government about response actions in the context of water contamination threats and incidents.
9.1.5 Remediation and Recovery
Even in the absence of a confirmed contamination event, activities viewed broadly as 'remediation and
recovery', may be required following the more possible situations of highly credible false alarms,
unintentional contamination, or 'upsets' to the water system which indicate the need for changes in the
water system operations. Regardless of what necessitates the remediation activity, the remediation
process involves a sequence of activities including: system characterization; selection of remedy options;
provision of an alternate drinking water supply during remediation activities; and monitoring to
demonstrate that the system may be returned to normal operation. The goal of remediation and recovery
is to return the water supply system to service as quickly as possible, while protecting public health and
minimizing disruption to normal life (or business continuity).
Evaluation Criteria. Evaluation should be based on whether the situation involved intentional
contamination, unintentional contamination, water system upsets, etc. The remediation and recovery
approach, in terms of the involvement of law enforcement in particular, should be different for intentional
versus unintentional contamination. For any case, relevant criteria such as the efficacy of the
decontamination technologies, the quality of the sampling and analysis, and the process by which the
water system is deemed fit to return to normal operation should all be evaluated. Another criterion
common to all remediation and recovery approaches is the rate of the activities. While rapid recovery of
the system is crucial, the evaluation should take into account that it is equally important to follow a
systematic process that establishes remedial goals acceptable to all stakeholders, implements the remedial
process in an effective and responsible manner, and demonstrates that the remedial action was successful.
DRAFT-121205 106
-------
WS System Architecture
Evaluation Methodology. Like the evaluation methodology discussed in the two previous subsections,
the evaluation of this element should likely be based on a mixture of actual situations, along with table
top exercises and drills. The evaluation methodology should vary based on the model for remediation
employed. For instance, if intentional contamination is involved, it may be useful to perform the
evaluation in light of the probable model for remediation and recovery, which may resemble a Superfund
remedial response program, although a contaminated water system probably would not be classified a
Superfund site, per se. However, some of the same principles involved in evaluation Superfund site
clean-ups may apply here. Part of the evaluation of this phase, if necessary, should involve looking for
the ability to apply the Superfund modelor other existing, remedial plansto the remediation of water
systems.
On the other extreme, WS may detect confirmed incidents that primarily affect system operation and not
public health. Often, techniques to implement required changes in system operation are known, so the
remediation model largely resembles routine maintenance or system optimization. Accordingly, the
evaluation should center on the appropriateness and timeliness of the remedial action, in the context of the
role that WS played in it. For instance, if WS data streams suggest a confirmed, inadvertent cross-
connection, the remediation and recovery actions should be correlated with the data streams. This is not
really an evaluation of the WS data streams themselves; rather, it is an exploration of how they
contributed to the remediation process. For instance, did they help pinpoint the source of the cross-
connection, saving utility resources in manually locating it?
Given the impossibilitydue to resource, response, and technology limitationsof fulfilling the ideal
performance goal of zero exposures, effectiveness of the WS-CWS should be defined along more
pragmatic objectives. EPA's contamination incident timeline analyses, for example, use endpoints of 1
percent fatalities and 50 percent of exposures to evaluate the timeliness of its simulated CWS. A key
output is determining the percentage decrease in exposures and fatalities that occurs as a result of the
warning system. EPA intends that these endpoints serve only to guide the Agency as it seeks to evaluate,
in the design phase, the components of a contamination warning system. Ultimately, while EPA expects
to provide general guidance to utilities and communities in the design and implementation of effective
CWSs, the process of balancing the resources versus the performance goals of the system should rest with
the utility and the community. Additional considerations as part of the programmatic evaluation of WS
are discussed in Section 9.2
9.2 Programmatic Evaluation of WaterSentinel
Section 1.1.2 briefly summarizes the overall objectives of the WS program. The objectives can actually
be thought of as collectively representing the needs of the many parties who would participate in the WS
pilot program as well as those involved in the promotion of CWS implementation beyond the pilot stage.
The evaluation discussed below is based on the projected perspectives of several of these parties. There
are other interested groups that may also have different perspectives, and the ones selected below may
sufficiently encompass many other groups.
9.2.1 EPA Perspective
Like the technical evaluation discussed in the section above, the evaluation of the success of WS at the
program level is an iterative process, allowing for continual improvement of the program. Thus, the
programmatic evaluation should not occur just at the end of the pilot period. Rather, the WS pilot should
need to be systematically evaluated in terms of results, accomplishments, limitations, sustainability, cost-
benefits, and other such metrics that may become apparent during the course of the pilot. Evaluations
should be in the context of the WS implementation and operational paradigm, but should also include
DRAFT-121205 107
-------
WS System Architecture
comparative evaluations with other programs related to water quality monitoring, water security, public
health protection, syndromic surveillance, and critical infrastructure protection.
This evaluation should include an evaluation of each individual element of WS in addition to a holistic
evaluation of the overall program that is more than just a sum of the evaluation of the individual elements.
This evaluation should accordingly consist of the following:
A Pre-implementation Evaluation Plan that describes the framework for the overall evaluation of
WS, including its ultimate goals, measures of success, limitations, and capabilities from technical,
programmatic, and policy perspectives.
Midway through the pilot, evaluate the WS against several benchmarks, such as: 1) to what extent
progress has been made on the types of issues revealed in the Pre-implementation Evaluation
Plan; 2) emerging issues not identified in the WS Pre-implementation Evaluation Plan; and 3)
progress and developments in these or other appropriate areas relative to WS evaluation.
A comprehensive quantitative and qualitative analysis of and development of recommendations
about the overall system design and implementation, using both the issues identified in the WS
Pre-implementation Evaluation Plan as well as any other developments that have occurred during
the pilot period, in any relevant technical, programmatic, or policy area.
This holistic evaluation is mainly related to the goal of WS stated in Section 1.1.2, namely an
investigation through a pilot demonstration project of the CWS concept, i.e., the timely warning of
potential water contamination incidents through enhanced and integrated monitoring and surveillance.
However, the ultimate goal of WS pilot project is develop a CWS model that can be adopted and
implemented by drinking water utilities of all sizes and with varying characteristics. Accordingly, EPA is
inherently interested in the evaluation of the program from the perspectives of utilities and other
stakeholders, which relate to the six specific objectives bulleted in Section 1.1.2. These six objectives
encompass a number of perspective-related issues, as discussed below.
9.2.2 Utility Perspectives
Much information should be generated from the WS program that may impact the ability of the pilot
utility to continue operation of the WS-CWS, and for other utilities to implement their own CWS, as
alluded to in Figure 1-2. The evaluation of the WS program from the utility perspective should focus on
issues of key importance to drinking water utilities operating WS or any other monitoring and
surveillance program, for that matter. These issues would include the following:
Management. This portion of the evaluation should focus on the management structures and
priorities that affect the ability of a utility to operate a CWS. This may range from availability of
human resources to the ability to commit funding for monitoring and surveillance programs
outside of the realm of compliance monitoring. The challenge is managing competing priorities,
e.g., regulatory compliance, infrastructure replacement and upgrades, consumer rate base, etc.
Effective management in this regard may involve a site specific analysis of the compatibility of
WS with other requirements and programs (e.g., can some WS monitoring also meet certain
regulatory monitoring requirements?). This gets to the heart of the desire to make the WS-CWS a
dual-use application, with benefits that extend well beyond just security.
Analytical capability/capacity. Section 4.4 focuses on the evaluation of the WS Sampling and
Analysis Program from a technical standpoint. Utilities may also be very interested in an
evaluation of how implementation of a CWS impacts their overall analytical capability and
requirements, both in the laboratory and in the field, especially with regards to their regulatory
compliance programs. In this respect, it is appropriate to note the following: historically, many
utilities have made the business decision to contract out much of their analytical work, not due to
technical challenges, but rather due to the overall cost-effectiveness of this approach. This has
DRAFT-121205 108
-------
WS System Architecture
important ramifications for the sustainability of the WS program which relies on rapid turn
around of laboratory analysis for contaminants that are not of routine interest to most utilities and
which are infrequently sampled.
Government and community relations. This relates to the ability of the utility to interact local
government and community partners to support a CWS program. Effective interaction with the
public as a partner and as a customer is another important consideration in the evaluation of the
WS-CWS in terms of government and community relations. This evaluation may be of additional
value to the utility as operation, maintenance, and sustainability of a comprehensive WS-CWS is
dependent on organizations and programs outside of the control and management of the utility.
Local government and community partners should remain engaged through public health
surveillance, consequence management, training, and evaluation to support an active and
protective CWS.
Operability/sustainability. Evaluation of the operability/sustainability of the WS program from
the management perspective may greatly aid other utilities with implementing WS. This
evaluation should contain specific cost-benefit results, enabling to the utility to implement a
sustainable CWS that meets their anticipated needs. This could also be characterized as the
manner in which a WS-CWS fits into the business model of the modern public (or private) utility.
Scalability. The evaluation of WS at the pilot city is, in some respects, site specific. Many
aspects of this specificity are discussed above. An additional aspect is the ability to scale the
CWS model to different sized utilities and communities. This evaluation should be largely
qualitative, but should broadly address scalability in terms of factors such as population served,
geographically region served, hydraulic regions present, along with the many other factors that
tend to make water systems unique.
9.2.3 Stakeholder Perspectives
There are a number of stakeholders deeply interested in and potentially affected by the WS program. A
few are included below, along with the type of, and possible measures for, evaluation perhaps appropriate
for each:
Drinking water consumers. Aside from the interaction of water utilities with the public, the
public perspective of a CWS program monitoring the drinking water in their community should
be evaluated. Examples of factors that may be evaluated include public confidence in the
government to protect them and benefits/problems reported by consumers as a part of the
program. It should be useful to evaluate the means of effectively conveying the potential costs
and benefits of the program to the public such that they can make an informed decision regarding
their willingness to support it.
Emergency Responders and other Local Government Services. The emergency response
community, and others within the local community, would likely be impacted the WS program.
Accordingly, an evaluation of the impact of the WS program on their activities should be very
useful as other water systems implement a CWS. A key aspect of this evaluation should entail a
thorough understanding of the maintenance of emergency response and governmental services
upon which WS relies, such as the ability to perform response activities related to consequence
management.
Public Health community. WS represents anew level of complexity for the public health
community and operations in terms of coordination with drinking water utilities. An important
aspect of this is the maintenance of public health syndromic surveillance programs, which are
part of the WS-CWS. In addition, coordination with the public health community in terms of
response also should be evaluated.
Drinking Water Trade Associations. These groups actively represent the various interests of
drinking water utilities at the local, state, and national levels with respect to the development and
DRAFT-121205 109
-------
WS System Architecture
implementation of drinking water programs. A thoughtful evaluation of the program with their
needs in mind should prove critical for the longer-term viability of the WS-CWS program.
Water Quality Researchers and Technology Developers. These groups are interested in
aspects of the WS program, mainly to focus their efforts on needs in their respective areas. The
effective evaluation of the program in terms of the science and technology supporting the
program should help meet their research and business objectives.
DRAFT-121205 110
-------
WaterSentinel
Section 10.0: References and Resources
Note: In addition to references cited throughout the document, this section also will include a summary of
other documents available based on the list of system architecture documents developed for the WS Team
Retreat. Documents will be listed as 'limited distribution' where appropriate.
American Society of Civil Engineers. Interim Voluntary Guidelines for Designing an Online Contaminant
Monitoring System. US. EPA Cooperative Research and Development Agreement, X-83128301-0,
December 9 2004.
AWWA. Utility-based Analytical Methods to Ensure Public Water Supply Security. Denver, CO:
AWWA, 2003.
AWWA. Manual Ml9, Emergency Planning for Water Utilities, 1996 Edition.
AWWA. Contamination Warning Systems for Water: An Approach for Providing Actionable Information
to Decision-Makers. Denver, CO: AWWA, 2005.
Baker, Edward J Jr and Koplan, Jeffrey. Strengthening the Nation's Public Health Infrastructure:
Historic Challenge, Unprecedented Opportunity; It rakes a system that is competent to handle routine
public health situations to handle the emergencies. Health Affairs. November, 2002.
Berry, J., Hart, W.E., and Phillips, C.A., "Locating sensors in municipal water networks," Proceedings of
the ASCE/EWRI Congress, 2003.
Berry, J., Fleischer, L., Hart, W.E., Phillips, C.A., and Watson, J.P. 2005, "Sensor Placement in
Municipal Water Networks," J. Water Resources Planning and Management, 131 (3): 237-243 (2005).
Boccelli, D. L., Shang, F., Uber, J. G. and Wang, J.. "Tracer Studies and Water Quality Monitoring for
Evaluating Network Model Confidence." 4th International Conference on Watershed Management and
Urban Water Supply, Shenzhen, China. 2004.
Bravata, DM., McDonald, KM., Smith, WM., Rydzak, C., Szeto, H., Bucridge, DL., Haberland, C.,
Owens, DK. Systematic review: Surveillance systems for early detection of bioterrorism- related diseases,
Ann INtern Med, 140(11), 910-22, June 2004.
BTACT, 2002, Public Health Security and Bioterrorism Preparedness and Response Act of 2002, PL
107-188. January 23, 2002. http://www.fda.gov/oc/bioterrorism/PL 107-188.pdf
Buehler, et. al. Framework for evaluating public health surveillance systems for early detection of
outbreaks: recommendations from the CDC Working Group. MMWR Recomm.Rep. 53.RR-5, 1-11,
2004.
Burns, N., Budd, G. and Horsley M. Water Quality Monitoring in Response to Heightened Security (M7),
Proceedings of the 2003 AWWA Water Quality Technology Conference, November 2-6, 2003,
Philadelphia, PA, 2003.
CDC, Division of Bacterial and Mycotic Diseases. Disease Information FAQ, January 2005.
http://www.cdc.gov/ncidod/dbmd/diseaseinfo/foodborneinfections g.htm#howdiagnosed
DRAFT-121205 111
-------
WS System Architecture
CDC. Public Health Information Network. http://www.cdc.gov/PHIN. 2005.
Clayton K., Allgeier, S., Apanian, D., Murray, R., Pulz, P., 2005 Syndromic Surveillance Conference
Advancing Disease Surveillance Seattle, WA, "Role of Public Health Syndromic Surveillance in
Contamination Warning Systems for Drinking Water". 2005.
Edge, V.L., et al. Syndromic surveillance of gastrointestinal illness using pharmacy over-the-counter
sales. A retrospective study of waterborne outbreaks in Saskatchewan and Ontario. Can. J. Public Health
95.6,: 446-50, 2004.
Foldy, S. Linking Better Surveillance to Better Outcomes, MMWR Morb Mortal. Wkly Rep, 53(Suppl),
12-17, 2004.
GAO-04-29 Drinking Water: Experts' Vies on How Federal Funding Can Best Be Spent to Improve
Security, United States General Accounting Office Report to the Committee on Environment and Public
Works, U.S. Senate, October 2003.
German, R.R., Updated Guidelines for Evaluating Public Health Surveillance Systems Recommendations
from the Guidelines Working Group Guidelines Working Group, CDC MMWR; 50: RR-13, 2001.
Gerzoff, R., Brown, C., and Baker, E. Full-Time Employees of US Local Health Departments, 1992-
1993. Journal of Public Health Management Practice. 5(3): 1-9, 1999
Gostin, Lawrence et al. The Future of The Public's Health: Vision, Values and Strategies. Health Affairs,
July 2004.
Grayman, Walter M., et al., Design of Early Warning and Predictive Source-Water Monitoring Systems.
AWWA Research Foundation and American Water Works Association. 2001.
Gullick, Richard W., Monitoring Systems for Early Warning of Source Water Contamination. Voorhees,
NJ: American Water Works Service Company, Inc. December 31, 2001.
Hall, J. et al., Evaluation of Water Quality Sensors as Devices to Warn of Intentional Contamination in
Water Distribution Systems. 2005.
Hargesheimer, Erika, et al, eds. Online Monitoring for Drinking Water Utilities. AWWA Research
Foundation and American Water Works Association. 2002.
Haught, Roy C., et al. Evaluation of Water Monitoring Instrumentation at EPA's Water Awareness
Technology Evaluation Research Security Center. Proceedings of the American Water Works Association
WQTC Conference. American Water Works Association. 2003.
Heffernan, R., et al. New York City Syndromic surveillance systems. MMWR. Morb. Mortal. Wkly. Rep.
53 Suppl (2004): 23-27.
Henning, K. J. What is Syndromic surveillance? MMWR Morb. Mortal. Wkly. Rep. 53 Suppl, 5-11, 2004.
Hrudey, S.E., Rizak, S., Discussion: Rapid Analytical Techniques for Drinking Water Security
Investigations, Journal of American Water Works Association, Volume 96, Number 9, 2004.
DRAFT-121205 112
-------
WS System Architecture
Hrudey, S.E., Huck P.M., Payment, P., Gillham, R.W., and Hrudey, E. J., Walkerton: Lessons learned in
comparison with waterborne outbreaks in the developed world J. Environ. Eng. Sci./Rev. gen. sci. env.
1(6), 397-407, 2002.
International Standard ISO 15839. Water quality On-line sensors/analysing equipment for water
Specifications and performance tests. 2003.
Khan AS, Swerdlow DL, Juranek DD. Precautions against biological and chemical terrorism directed at
food and water supplies. Public Health Rep, 116:314, 2001.
King, K., Engelhardt, T., Ogan, K., "5.2 Candidate Instruments and Observables" in Interim Voluntary
Guidelines for Designing an Online Contaminant Monitoring System, 2004.
Koplan, J.P., Milstein, R.L, Wetterhall, S.F. Framework for Program Evaluation in Public Health.
September 17, 1999/48(RR11); 1-40. http://www.cdc.gov/mmwr/preview/mmwrhtml/rr481 lal .htm.
October 25, 2005.
Magnuson, et. al "Interpreting multiple data streams in a system for monitoring and surveillance for
drinking water contamination", in preparation.
MMWR: Brief Report: Terrorism and Emergency Preparedness in State and Territorial Public Health
Departments United States, 2004 MMWRMorb. Mortal. Wkly. Rep., 54(18); 459-460, 2004.
Murray, R., Janke, R., Uber, J., Published in the Proceedings of the ASCE/EWRI Congress, Salt Lake
City, UT. The Threat Ensemble Vulnerability Assessment (TEVA) Program for Drinking Water
Distribution System Security. 2004.
Murray, R., Janke, R., Uber, J., Herrmann, J., Allgeier, S., The EPA's TEVA Research Program for
Designing and Evaluating Early Warning System Architectures. 2005.
Murray, R., Uber, J., Berry, J., Hart, W., Watson, J.P., Presentation for the Water Quality Technology
Conference, "Sensor Network Design for Contamination Warning Systems". 2005.
Neter, J., Wassermand, W., & Kutner, M. H. Applied linear statistical models, Fourth Edition: Irwin,
Chicago. 1996.
Ostfeld, A. and Salomons, E., "Optimal Layout of Early Warning Detection Stations for Water
Distribution System Security," Journal of Water Resources Planning and Mangement, 130 (5), pp 377-
385.2004.
Petersen, LR; Denis D, et al, Community Health Effects of a Municipal Water Supply Hyperfluoridation
Accident, AJPH, 78, 6, 711-713, 1988.
Pikus, Interim Voluntary Guidelines for Designing an Online Contaminant Monitoring System, 2004.
ProMed, "Tularemia, Air Sensor Detection - USA (District of Columbia)" 2005.
htto://www.promedmail.org/pls/promed/f?p=2400:1202:17534429829067225889::NO::F2400 P1202 C
HECK DISPLAY.F2400 PI202 PUB MAIL ID:X.3Q577
Reingold, A., Biosecurity and Bioterrorism: Biodefense Strategy, Practice, and Science If Syndromic
Surveillance Is the Answer, What Is the Question? Vol. 1, No. 2, 77-81, Jun 2003.
DRAFT-121205 113
-------
WS System Architecture
Siegel, Bruce et al. Addressing Health Disparities In Community Settings. The Robert Wood Johnson
Foundation. 2001.
Sosin, DM. Draft framework for evaluating syndromic surveillance systems, J Urban Health, 80(2 Suppl
1), 18-13, 2003.
States, Stanley, Jessica Newberry, Jennifer Wichterman, John Kuchta, Michele Scheuring, and Leonard
Casson. "Rapid Analytical Techniques for Drinking Water Security Investigations." Journal AWWA.
Vol. 98, No. 1, 52-65, January 2004.
Uber, J., Janke, R., Murray, R., and Meyer, P., 2004, "A Greedy Heuristics Model for Locating Water
Quality Sensors in a Water Distribution System," Proceedings of the ASCE/EWRI Congress, 2004.
USEPA. Exposure Factors Handbook 1997.
httD://cfbub.epa.gov/ncea/cfm/recordisplav.cfm?deid=12464&CFID=4899461&CFTOKEN=36365412
USEPA. National Center for Environmental Assessment. Volatilization Rates from Water to Indoor Air
Phase II. 2000.
USEPA. Response Protocol Toolbox: Planning for and Responding to Drinking Water Contamination
Threats and Incidents Response Guidelines Interim Final. 2004a.
USEPA. Roadmap to an Alliance: Integration of Drinking Water Laboratories into the Laboratory
Response Network (LRN). 2004b.
USEPA. Drinking Water Security Assessment and Identification ofCountermeasures, 2004c.
CLASSIFIED. For Official Use Only.
USEPA. Water Contaminant Information Tool (WCIT) Population Plan, 2005a.
USEPA. WaterSentinel Contamination Incident Timeline Analysis, 2005b. SENSITIVE. For Official Use
Only.
USEPA. WaterSentinel Contaminant Selection, 2005c. SENSITIVE. For Official Use Only.
USEPA. WaterSentinel Contaminant Fact Sheets, 2005d. SENSITIVE. For Official Use Only.
USEPA. Sampling Guidance for Unknown Contaminants in Drinking Water. 2005e.
USEPA. Protocol for the Analysis of Unknown Contaminants in Drinking Water. 2005f
USEPA. Preliminary Cost Analysis for WaterSentinel Contamination Warning System. 2005g. For
Official Use Only.
USEPA. Online Water Quality Monitoring as an Indicator of Drinking Water Contamination, 2005h. For
Official Use Only.
USEPA. WaterSentinel Consequence Management Strategy, 2005i. For Official Use Only.
USEPA, Cost Estimates for Monitoring Networks with YSI Sonde and Hach Astro TOC, 2005J. For
Official Use Only.
DRAFT-121205 114
-------
WS System Architecture
USEPA, Overview of Event Detection Systems for Water Sentinel, 2005k. For Official Use Only.
USEPA, Water and Wastewater Security Product Guide. 20051.
http: //cfpub .epa. gov/safe water/watersecurity/guide/
Wagner, R.J., Mattraw, H.C., Ritz, G.F., Smith, B.A.. Guidelines and Standard Procedures for
Continuous Water-Quality Monitors: Site Selection, Field Operation, Calibration, Record Computation,
and Reporting. U.S.G.S. Water-Resources Investigations Report 00-4252, Reston, 2000.
Wagner, et al. National Retail Data Monitor for Public Health Surveillance MMWR. 53(Suppl),40-42,
2004.
Watson, Jean-Paul, et al, "A Multiple-Objective Analysis of Sensor Placement Optimization in Water
Networks", Proceedings of the ASCE/EWRI Congress, June 2004.
DRAFT-121205 115
-------
WS System Architecture
Appendix A: Acronym List
API application program interface
ASCE American Society of Civil Engineers
AWWA American Water Works Association
BSL biological safety level
CCSP Consumer Complaint Surveillance Program
CDC Centers for Disease Control and Prevention
CDF cumulative distribution function
CEO chief executive officer
CIS customer information system
Cl Chlorine residual
CMP Consequence Management Plan
ConOps concept of operations
COO chief operating officer
CWS contamination warning system
DHS Department of Homeland Security
DOD Department of Defense
DPD a testing reagent
DQO data quality objectives
DSL Digital Subscriber Line
DSS distribution system simulator
DSS Decision Support System
EAST Estimate of Adversary Sequence Interruption
ebXML electronic business extensible markup language
EC electrical conductivity
ECBC Edgewood Chemical Biological Center
EDS Event detection system
EDXL Emergency Data Exchange Language
eLRN Environmental Laboratory Response Network
EMS emergency medical services
EPA U.S. Environmental Protection Agency
ER emergency room
ESSENCE Electronic Surveillance System for the Early Notification of
Community-Based Epidemics
FDA U.S. Food and Drug Administration
FERN Food Emergency Response Network
FTP file transfer protocol
FY fiscal year
GAO Government Accounting Office
GC/MS gas chromatography/mass spectrometry
DRAFT-121205
116
-------
WS System Architecture
GIS
HIPAA
HL
HSC
HSPD
JBAIDS
JHU-APL
LC/MS
LIMS
LRN
MCL
MDL
NEDSS
NEMA
NEMI-CBR
NHSRC
NRDM
NTU
NYC
NYC DOHMH
O&M
ORP
OTC
PCR
PDA
PE
PH
PHIN
PHS
PLC
PPS
PSI
PT
QA
QC
RHIO
RIO
ROC
RODS
RPTB
RSD
RTCC
Geographical information systems
Health Insurance Portability and Accountability Act
health level
Homeland Security Council
Homeland Security Presidential Directive
Joint Biological Agent Identification and Diagnostic System
John's Hopkins University Applied Physics Laboratory
liquid chromatography/mass spectrometry
laboratory information management systems
Laboratory Response Network
maximum contaminant level
method detection limit
National Electronic Disease Surveillance System
National Electrical Manufacturers Association
National Environmental Methods Index - Chemical, Biological,
Radiological Methods
National Homeland Security Research Center
National Retail Data Monitor
Nephelometric Turbidity Units
New York City
New York City Department of Health and Mental Hygiene
operations and maintenance
oxygen reduction potential
over-the-counter
polymerase chain reaction
personal digital assistant
performance evaluation
Public Health
Public Health Information Network
public health surveillance
Programmable Logic Controllers
physical protection system
pound per square inch
proficiency testing
quality assurance
quality control
Regional Health Information Network
remote input-output
receiver operating characteristic
Real-time Outbreak and Disease Surveillance
Response Protocol Toolbox
relative standard deviation
Real Time Clinical Connections
DRAFT-121205
117
-------
WS System Architecture
RTU remote terminal unit
SAM Standardized Analytical Methods for Use During Homeland Security
Events
SAVI System Analysis and Vulnerability to Intrusion
SCADA supervisory control and data acquisition
SOP standard operating plan
SRMD Standards and Risk Management Division
SS syndromic surveillance
T&E EPA's Test and Evaluation
TCR Total Coliform Rule
TDS total dissolved solids
TEVA Threat Ensemble Vulnerability Assessment
TOC total organic carbon
TTEP Technology Testing and Evaluation Program
UPC Universal Purchase Code
UPS Uninterrupted Power Supply
USDA U.S. Department of Agriculture
VA Veterans Administration
WATERS Water Awareness Technology Evaluation Research and Security
WCIT Water Contaminant Information Tool
WLA Water Laboratory Alliance
WQ water quality
WS WaterSentinel
XML extensible markup language
DRAFT-121205
118
-------
WaterSentinel
Appendix B: Glossary
Working Draft for Discussion
Agency. A division of government with a specific function, or a non-governmental organization (e.g.,
private contractor, business, etc.) that offers a particular kind of assistance. In the incident command
system, agencies are defined as jurisdictional (having statutory responsibility for incident mitigation) or
assisting and/or cooperating (providing resources and/or assistance).
Analytical Approach. A plan describing the specific analyses that are performed on the samples
collected in the event of a water contamination threat. The analytical approach is based on the specific
information available about a contamination threat.
Analytical Confirmation. The process of determining an analyte in a defensible manner.
Automation. Ability of the monitoring/field technology, analytical method, or surveillance system to
provide notification with limited analysis or interaction.
Availability. Identifies whether the technology, method, or surveillance system is available for
implementation in the pilot or requires additional research and/or validation.
Baseline. Background levels of specific contaminants; normal ranges for water quality parameters;
incidence of disease, consumer complaints, security breaches, or reports of information. Depending on the
source of information, baseline can be site or system specific and may have a seasonality component.
Bias. A systematic or persistent distortion of a measurement process that results in a measurement
different than the sample's true value.
Concept of Operations (ConOps). Identifies routine, day-to-day operations for maintaining the
WaterSentinel contamination warning system at a water utility and public health agency to detect and
respond to a contamination event. The ConOps provides the broad context from routine monitoring and
surveillance activities to recovery from an event.
Confirmed. In the context of the threat evaluation process, a water contamination incident is 'confirmed'
if the information collected over the course of the threat evaluation provides definitive evidence that the
water has been contaminated.
Confirmatory Stage. The third stage of the threat evaluation process from the point at which the threat is
deemed 'credible' through the determination that a contamination incident either has or has not occurred.
Consequence. The adverse outcome resulting from a drinking water contamination incident. In the
context of the threat management process, the consequence considers both the number of individuals
potentially affected as well as the severity of the health effect experienced upon exposure.
Consequence Management Plan. Provides a decision-making framework that governs when, how,
what, and who will be involved in making decisions in response to contamination threat warnings to
minimize the response timeline and implement operational or public health response actions
appropriately.
DRAFT-121205 119
-------
WS System Architecture
Consequence Management. Refers to the process and procedures for implementing response actions
that are initiated upon detection of a 'possible' contamination event and continues through determining if
the threat is credible and confirming the contamination threat. An initial trigger indicating possible
contamination could come from single or multiple monitoring and surveillance information streams.
Indication of possible contamination will prompt the water utility to conduct follow up actions such as
site characterization, triggered sampling, analysis for unknowns, notifications, and precautionary actions
to reduce consequences should the event be later determined credible or confirmed. As the information
from the initial response actions and/or additional detection information is collected from or coordinated
with the water utility, additional response actions will be considered and implemented as the event is
assessed for credibility. This process of continuous information collection followed by assessment and
action will be performed by the water utility and others from the local to State to Federal levels of various
agencies to respond to the event, mitigate the consequences, provide internal and external notifications,
bring in additional resources for response and analysis, and manage all related emergency response
requirements associated with the specifics of the event.
Contaminant Classes. WS contaminants can be categorized into 12 categories based on their ability to be
detected by routine sampling, online monitoring, consumer complaints, and public health surveillance.
Contamination Warning System (CWS). Active deployment and use of monitoring
technologies/strategies and enhanced surveillance activities to collect, integrate, analyze, and
communicate information to provide a timely warning of potential water contamination incidents and
initiate response actions to minimize public health impacts.
Consumer Complaint Surveillance. Consumer complaints regarding unusual taste, odor, or appearance
of the water are often reported to and recorded by water utilities which conventionally use them to
identify and address water quality problems. Using an appropriate methodology, WS could track and
analyze these complaints to look for unusual trends that may be indicative of a contamination incident.
Credible. In the context of the threat evaluation process, a water contamination threat is characterized as
'credible' if information collected during the threat evaluation process corroborates information from the
threat warning.
Credible Stage. The second stage of the threat management process from the point at which the threat is
deemed 'possible' through the determination as to whether or not the threat is 'credible'.
Credibility Determination. Detected events will be considered 'possible' indications of contamination
and will be validated through the process of credibility determination. Based on this analysis a decision
will be made to return to normal operations or move to the credible stage and implement consequence
management and response actions. It is critical that the systematic approach for assessing credibility in
response to contamination threat warnings ensures that all available information is analyzed in a timely
and efficient manner to minimize both false alarms and over-response to a trigger that has not been
determined to be credible.
Data Management. Manages, analyzes, and interprets different data streams in a timely manner to
recognize potential contamination incidents in time to respond effectively.
Design Basis. The range of conditions and events taken explicitly into account in the design of a facility,
according to established criteria, such that the facility can withstand them without exceeding authorized
limits by the planned operation of safety systems.
DRAFT-121205 120
-------
WS System Architecture
Detection Time. The time for water contaminated with detectable concentrations to reach each 'sensor'
in the network (i.e., an opportunity for a detection event). Travel times to each 'sensor' will be predicted
by TEVA as a time series.
Distribution System. A network of pipes that distribute potable water to customers' plumbing systems.
Dual Use. Application of contamination warning system components to routine operations.
Emergency Operations Center. A pre-designated facility established by an agency or jurisdiction to
coordinate the overall agency or jurisdictional response and support to an emergency.
Emergency Response Plan. A document that describes the actions that a drinking water utility would
take in response to various emergencies, disasters, and other unexpected incidents.
Enhanced Security Monitoring. Security breaches, witness accounts, and notifications by perpetrators,
news media, or law enforcement can be monitored through enhanced security practices.
Event Detection. Event detection is defined as a signal from monitoring and surveillance activities that
is indicative of a possible contamination incident. This signal could be a pattern of unusual water quality,
a cluster of unusual consumer complaints, or unusual symptoms picked up by a public health surveillance
program. Event detection algorithms are applied to the data to filter out the anomalies that normally
occur, or which have known causes, and signal only those events that are likely to be possible
contamination incidents. In short, the purpose of the event detection algorithms is to reduce the false
positive rate without missing potential events.
False Positive. (1) Rate at which a contamination warning system incorrectly indicates a contamination
incident. (2) Rate at which the technology, analytical method, or surveillance system detects a
contaminant, class of contaminants, or change from the baseline when the contaminant or contaminants
are not present.
False Negative. (1) Rate at which a contamination warning system fails to detect a contamination
incident. (2) Rate at which the technology, analytical method, or surveillance system does not detect a
contaminant, class of contaminant, or change from the baseline when the contaminant or contaminants are
present.
Field Safety Screening. Screening performed to detect any environmental hazards (i.e., in the air and on
surfaces) that might pose a threat to the site characterization team. Monitoring for radioactivity as the
team approaches the site is an example of field safety screening.
Health Care Provider. Any individual or organization involved in the care of patients. Health care
providers include physicians and hospitals.
Homeland Security Presidential Directive 7 (HSPD 7). HSPD 7 - Critical Infrastructure Identification,
Prioritization, and Protection - designated EPA and other agencies as the sector-specific agencies for
critical infrastructure areas. EPA was designated as the agency responsible for protection activities for the
Nation's drinking water and wastewater infrastructure. A key component of this responsibility is the
hardening of drinking water and wastewater system infrastructure to address vulnerabilities.
Homeland Security Presidential Directive 9 (HSPD 9). HSPD 9 is the directive that charges EPA and
other agencies, using existing authorities, to build upon and expand current monitoring and surveillance
DRAFT-121205 121
-------
WS System Architecture
programs to develop robust, comprehensive and fully coordinated surveillance and monitoring systems to
provide early detection and awareness of water contamination. In order to support the monitoring and
response to an incident, HSPD-9 also directs EPA to develop nationwide laboratory networks that
integrate existing federal and state laboratory resources.
ID50. The dose that results in infection in 50% of the population exposed to that dose.
Incident. A confirmed occurrence that requires response actions to prevent or minimize loss of life or
damage to property and/or natural resources. A drinking water contamination incident occurs when the
presence of a harmful contaminant has been confirmed.
Incident Command System. A standardized on-scene emergency management concept specifically
designed to allow its user(s) to adopt an integrated organizational structure appropriate for the complexity
and demands of single or multiple incidents, without being hindered by jurisdictional boundaries.
LD50. The dose that results in death in 50% of the population exposed to that dose.
Laboratory Information Management Systems (LIMS). Sophisticated software packages that track
and analyze laboratory information. Interfacing with laboratory instruments and personnel at the front
end and databases at the back end, LIMS provide information management at the integrated laboratory
level.
Laboratory Response Network (LRN). The LRN is charged with the task of maintaining an integrated
network of state and local public health, federal, military, and international laboratories that can respond
to bioterrorism, chemical terrorism and other public health emergencies.
Latency Period. The period of time that elapses between exposure of an individual to a causative agent
and the appearance of signs or symptoms of disease.
Monitoring and Surveillance. Element of the WS-CWS to provide a standardized set of information
streams to detect contamination events.
Online Water Quality Monitoring. Sensors located within the treatment and distribution system can
potentially detect an identifiable change from an established water quality baseline, such as chlorine
residual, pH, conductivity, turbidity, etc., and serve as an indicator of potential contamination in the WS-
CWS.
Possible. In the context of the threat evaluation process, a water contamination threat is characterized as
'possible' if the circumstances of the threat warning appear to have provided an opportunity for
contamination.
Possible Stage. The first stage of the threat management process from the point at which the threat
warning is received through the determination as to whether or not the threat is 'possible'.
Precision. The degree to which a set of measurements obtained under similar conditions conform to
themselves. Precision is usually expressed as standard deviation, variance, or range, in either absolute or
relative terms.
Public Health. The health and well being of an entire population or community. Public health does not
specifically address the health of individuals.
DRAFT-121205 122
-------
WS System Architecture
Public Health Surveillance. Ongoing, systematic collection, analysis, and interpretation of health-
related data essential to the planning, implementation, and evaluation of public health practice (Sosin,
2003). Syndromic surveillance by the public health sector as well as reports from emergency medical
service (EMS) runs, 911 call centers and poison control hotlines might serve as a warning of a potential
drinking water contamination incident if there is a reliable link between the public health sector and
drinking water utilities.
Quality Assurance. An integrated system of management activities involving planning, implementation,
documentation, assessment, reporting, and quality improvement to ensure that a process, item, or service
is of the type and quality needed and expected by the client.
Quality Control. The overall system of technical activities that measures the attributes and performance
of a process, item, or service against defined standards to verify that they meet the stated requirements
established by the client; operational techniques and activities that are used to fulfill requirements for
quality.
Rapid Field Testing. Analysis of water during site characterization using rapid field water testing
technology in an attempt to tentatively identify contaminants or unusual water quality.
Reliability. For a contamination warning system (CWS), reliability can be considered from at least two
perspectives: system operation and system performance. System operation refers to factors such as CWS
component downtime and maintenance requirements. System performance is defined as the ability of the
system to provide information that leads decision makers to successfully infer that contamination has or
has not occurred.
Remediation and Recovery. The goal of remediation and recovery is to return the water supply system
to service as quickly as possible while protecting public health and minimizing disruption to normal life
(or business continuity). During the remediation and recovery stage, the immediate urgency of the
situation has passed, and the magnitude of the remedial action requires careful planning and
implementation. While rapid recovery of the system is crucial, it is equally important to follow a
systematic process that establishes remedial goals acceptable to all stakeholders, implements the remedial
process in an effective and responsible manner, and demonstrates that the remedial action was successful.
Response Decisions. Part of the threat management process in which decisions are made regarding
appropriate response actions that consider: 1) the conclusions of the threat evaluation, 2) the
consequences of the suspected contamination incident, and 3) the impacts of the response actions on
drinking water customers and the utility.
Response Guidelines. A manual designed to be used during the response to a water contamination
threat. Response Guidelines should be easy to use and contain forms, flow charts, and simple instructions
to support staff in the field or decision officials in the Emergency Operations Center during management
of a crisis.
Response Protocol Toolbox (RPTB). These modules provide a framework to guide the response to
contamination threats and incidents and establishes the foundation for the primary steps, or phases, for
consequence management as part of the WS-CWS
DRAFT-121205 123
-------
WS System Architecture
Response Time. The time to decide on an appropriate response action and mobilize resources to
implement that action once an event is determined to be credible (as defined in EPA's Response Protocol
Toolbox).
Robustness. The ability of an instrument to sustain performance under field conditions (e.g., a research-
grade instrument may have excellent precision and bias specifications, but have poor robustness, and
would be unsuitable for deployment).
Routine Sampling. Water samples can be collected at a predetermined frequency to establish a baseline
or in response to a trigger and subsequently analyzed by the application of a robust unknowns protocol to
establish a baseline and serve as preparedness and training for response to a possible contamination
incident. This unknowns protocol would provide coverage for specific, priority contaminants, but may
also detect some non-target analytes if the analytical techniques used in the routine monitoring program
are sufficiently robust and if the analysts are trained and encouraged to investigate tentatively identified
contaminants.
SCADA Systems. SCADA stands for Supervisory Control And Data Acquisition. It is not a full control
system, but rather focuses on the supervisory level. It is a software package that is positioned on top of
the hardware to which it is interfaced, in general via Programmable Logic Controllers (PLCs), or other
commercial hardware modules.
Security Breach. An unauthorized intrusion into a secured facility that may be discovered through direct
observation, an alarm trigger, or signs of intrusion (e.g., cut locks, open doors, cut fences). A security
breach is a type of threat warning.
Security Surveillance. Ongoing, continual monitoring and investigation of security breaches, witness
accounts, notifications by perpetrators, news media, or law enforcement.
Site Characterization. The process of collecting information from an investigation site in order to
support the evaluation of a drinking water contamination threat. Site characterization activities include
the site investigation, field safety screening, rapid field testing of the water, and sample collection.
Stakeholders. WaterSentinel stakeholders include water utilities, laboratories, states, emergency
responders, public health officials, law enforcement, Federal agencies, technical experts, among others.
Surrogate. Utilizing general water quality parameters such as temperature, residual chlorine, pH,
turbidity, etc. as an indication of a contamination event.
Surveillance Systems. Systems that collect and analyze morbidity, mortality, and other relevant data and
facilitate the timely dissemination of results to appropriate decision makers (Bravata, et al., 2004).
Sustainability. Sustainability of a contamination warning system (CWS) considers factors that influence
the ability of an entity, such as a drinking water utility, to operate and maintain the CWS over an
extended period of time and in the face of competing priorities that could siphon resources away from the
program. In most cases, the analysis of Sustainability for a CWS will entail a cost-benefit analysis.
System Architecture. WaterSentinel system architecture provides a framework for developing a
contamination warning system (CWS) in support of the WS program. The WS system architecture will
define the conceptual approach for the WaterSentinel contamination warning system (WS-CWS) and
DRAFT-121205 124
-------
WS System Architecture
document the most effective combination of CWS components to yield a sustainable program that can be
adopted and implemented by drinking water utilities.
Technology Testing and Evaluation Panel (TTEP). Office of Research and Development program for
analysis of technologies that could be candidates for deployment in a contamination warning system.
Through TTEP, EPA will continue to evaluate existing detection and sensor equipment, as well as data
management integration software, among others, to determine which technologies would have application
for WaterSentinel.
Threat. An indication that a harmful incident, such as contamination of the drinking water supply, may
have occurred. The threat may be direct, such as a verbal or written threat, or circumstantial, such as a
security breach or unusual water quality.
Threat Ensemble Vulnerability Assessment (TEVA). An NHSRC research program that is a central
element in the design of the WS-CWS. TEVA uses an ensemble approach to sensor placement by
simulating contaminant insertion at all accessible nodes within a distribution system. The sensor
placement algorithm tries to minimize the overall public health impacts across all scenarios, which favors
detection of attacks that occur at nodes that produce the greatest impact.
Threat Evaluation. Part of the threat management process in which all available and relevant
information about the threat is evaluated to determine if the threat is 'possible' or 'credible', or if a
contamination incident has been 'confirmed.' This is an iterative process in which the threat evaluation is
revised as additional information becomes available. The conclusions from the threat evaluation are
considered when making response decisions.
Threat Management. The process of evaluating a contamination threat and making decisions about
appropriate response actions. The threat management process includes the parallel activities of the threat
evaluation and making response decisions. The threat management process is considered in three stages:
'possible', 'credible', and 'confirmatory.' The severity of the threat and the magnitude of the response
decisions escalate as a threat progresses through these stages.
Threat Warning. An unusual occurrence, observation, or discovery that indicates a potential
contamination incident and initiates actions to address this concern.
Timeline Analysis. Contamination incident timelines illustrate the time over which consequences
resulting from a drinking water contamination incident would develop in a population, and the time at
which various detection and intervention strategies might be effective, thus providing a rational basis for
the WS-CWS design.
Vulnerability Assessment. A systematic process for evaluating the susceptibility of critical facilities to
potential threats and identifying corrective actions that can reduce or mitigate the risk of serious
consequences associated with these threats.
Water Contamination Incident. A situation in which a contaminant has been successfully introduced
into the system. A water contamination incident may or may not be preceded by a water contamination
threat.
Water Contamination Threat. A situation in which the introduction of a contaminant into the water
system is threatened, claimed, or suggested by evidence. Compare water contamination threat with water
DRAFT-121205 125
-------
WS System Architecture
contamination incident Note that tampering with a water system is a crime under the Safe Drinking
Water Act as amended by the Bioterrorism Act.
Water Laboratory Alliance (WLA). A network of laboratories with extensive capability for the
analysis of water samples for a wide range of potential contaminants. It is proposed that the WLA
integrate existing water quality labs with the existing Laboratory Response Network, established by CDC
to support analysis of potential biothreat agents.
WaterSentinel. WaterSentinel is a robust, comprehensive monitoring and surveillance program that
integrates elements of a contamination warning system (CWS) to inform response decisions and minimize
public health and economic impacts.
Witness Account. A threat warning may come from an individual who directly witnesses suspicious
activity, such as trespassing, breaking and entering, or some other form of tampering. The witness could
be a utility employee, law enforcement officer, citizen, etc.
DRAFT-121205 126
-------
WaterSentinel
Appendix C: Overview of Related Projects
EPA anticipates that WS would build on and integrate water security activities and programs developed
by EPA's Water Security Division and National Homeland Security Research Center (NHSRC) to
enhance the design and implementation of the WS-CWS at a pilot utility. Key EPA programs and
projects that plan to be leveraged to support the WS program are described below.
In addition to these efforts, EPA is working closely with stakeholders and partner organizations to
identify and participate in projects related to elements of CWS design and implementation. Information
from these efforts has been and continues to be considered throughout the various phases of the WS
program. Examples of related CWS efforts include the following:
AWWA Utility Users Group
California Utilities Contamination Warning System Workgroup
California Space Authority Water Monitoring Project
AWWA Consumer Complaint Management
Water Quality Monitoring and Event Detection Project (Charleston, SC)
Wireless Underwater Telemetry System for Surface Water Quality Monitoring (Water Telemetry
work)
NJ American/Rutgers/USGS Consortium
Region III Security Project 'Drinking Water Distribution System Early Warning Monitoring
System for the District of Columbia'
Hydra Remote Monitoring System
Department of Homeland Security (DHS) Study of the Municipal Water System
Threat Ensemble Vulnerability Assessment (TEVA) Research Program
NHSRC's Threat Ensemble Vulnerability Assessment (TEVA) research program has been a central
element in the design of the WS-CWS, particularly the online monitoring aspects of the system. TEVA is
a suite of software tools for water security that can be used to assess the consequences of contamination
events in distribution systems, design online monitoring networks, and evaluate mitigation strategies.
TEVA uses an ensemble approach to sensor placement by simulating contaminant introduction at all
accessible nodes within a modeled distribution system. The sensor placement algorithm uses an
optimization routine to minimize the overall public health impacts across all scenarios, which favors
detection of attacks that occur at nodes that produce the greatest impact (Murray, et. al, 2004).
TEVA's computational framework integrates an extended period simulation hydraulic model, exposure
models, fate and transport models, disease transmission models, and numerous detection models. The
program's approach to distribution system modeling for the purpose of sensor placement is ideally suited
to the development of a CWS. In developing the general system architecture for WS, EPA used TEVA to
simulate the consequences of a large number of different contamination scenarios, which were
subsequently analyzed to evaluate the timing of detection and response, through various CWS strategies.
These timelines were used to assess which strategies potentially provided the greatest opportunity for
intervention in a drinking water contamination incident. In addition, the cost-benefit algorithm in TEVA
would be used during design and implementation of the WS-CWS at the WS pilot utility to maximize the
benefit in terms of increased protection per unit cost (Murray, et. al., 2005).
The TEVA Program has partnered with AWWA's Water Utility Users Group in order to ground truth the
software tools on real drinking water distribution systems. When the WS project evolves to promote the
design and implementation of CWSs at other utilities, the TEVA tools can be applied to the distribution
DRAFT-121205 127
-------
WS System Architecture
system of each utility to establish the number and location of sensor and sampling locations. EPA
expects that the experience gained from TEVA would also aid in the development of utility-specific
contamination incident timelines to better understand the consequences of a drinking water contamination
incident over time in a population, and the time at which various detection and intervention strategies
might be effective, thus providing a rational basis for the WS-CWS design.
Directly in support of WS design and implementation and as Phase 2 of the TEVA research program,
NHSRC is designing a comprehensive field study to assess the utility-specific system architecture design
for the CWS, and verify performance. EPA anticipates that the field study would be implemented at the
participating pilot drinking water utility to support the evaluation and implementation of the WS-CWS
components at the pilot utility, including measures of online monitoring system performance, consumer
complaint assumptions, and the effectiveness of the sampling and analysis protocols for routine and
triggered monitoring.
Design of the field studies would be reviewed by subject matter experts before implementation, and the
results of the studies documented and evaluated thoroughly to address any need for refinements of the
utility-specific system architecture design or the implementation of the various components.
Public Health Syndromic Surveillance Pilots
Design and implementation of the WS program's public health surveillance component is aided by pilot
programs in this area being led by EPA's NHSRC. The Center currently is conducting a multi-year,
multi-phase demonstration project to integrate water quality and consumer complaint data from drinking
water utilities with municipal syndromic surveillance data (Clayton, et. al., 2005). The goal of the pilots
is to assess the value of these additional data streams in detecting accidental and intentional drinking
water contamination events, which directly supports this aspect of the WS program, as well. EPA is
conducting this project by working with pilot cities that already use nationally recognized public health
syndromic surveillance systems, including Real-time Outbreak and Disease Surveillance (RODS) and
Electronic Surveillance System for the Early Notification of Community-based Epidemics (ESSENCE).
EPA anticipates that the public health syndromic surveillance demonstration project would consist of the
following elements:
Development of system interfaces to display integrated data for use by local water utilities and
local public health officials to review and evaluate signals
Sharing of anomaly alerts between a water utility and all public health agencies within the
utility's service area
Integration of near real-time data from the drinking water utility potentially including: water
quality data, security incidents, and consumer complaints into existing public health surveillance
systems
Evaluation and analysis of results, as well as guidance for future integration efforts
EPA expects that results of these demonstration projects would inform the integration of water quality
and syndromic surveillance data during the WS pilot. Participants in these projects should also inform
aspects of data analysis, credibility determination, and consequence management as they relate to public
health. Their analysis should help define and validate detection and response assumptions related to
coordination and integration with the drinking water utility and local public health department.
DRAFT-121205 128
-------
WS System Architecture
Technology Testing and Evaluation Program (TTEP)
The EPA NHSRC's Technology Testing and Evaluation Program (TTEP) aims to provide independent
assessments of water-security related technologies considered for use in WS. The program tests and
reports on the performance of technologies for use by the water industry, including technologies
specifically related to water security. In support of WS, it is anticipated that TTEP would develop a
standardized approach for the evaluation of commercially available event detection software and would
evaluate promising software for inclusion in the WS pilot and further evaluation. TTEP would also
conduct additional water quality sensor evaluations for technologies that could be integrated into future
iterations of the WS-CWS. Additional studies evaluating the impact of various contaminants on the
target water quality parameters would also be conducted through TTEP. Continued evaluation of other
monitoring technologies and field test equipment should support various WS components and site
characterization activities. EPA anticipates that the results of these studies would be available on TTEP's
website and can be used by utilities to select the technologies appropriate to each individual system.
Water Quality Sensor Studies
This is a program led by NHSRC to evaluate the sensitivity of water quality sensors and the potential use
of water quality parameters to indicate the presence of a contaminant in water is being leveraged to
support the selection of sensors and parameters for WS. EPA initiated the program in 2003 to investigate
online sensors that monitor for standard drinking water parameters that could be used to trigger a
contamination event within a drinking water distribution system and support CWS approaches being
considered by water utilities. Research is conducted under this program on a pilot-scale system using a
recirculating, pipe-loop distribution system simulator (DSS). The simulator is located at the Water
Awareness Technology Evaluation Research and Security (WATERS) Laboratory within EPA's Test and
Evaluation (T&E) Facility in Cincinnati, Ohio.
Water quality parameters evaluated through this program include pH, free chlorine residual,
oxidation/reduction potential (ORP), dissolved oxygen, specific conductance, turbidity, total organic
carbon (TOC), chloride, ammonia, and nitrate. Based on initial research, free and total chlorine residual,
TOC, ORP, specific conductance, and chloride were consistently able to indicate a change in water
quality due to injections of various contaminants into the pipe-loop (Hall, et al, 2005). In support of the
WS program, additional research should be conducted to evaluate the effects of WS baseline
contaminants on water quality parameters (USEPA, 2005h).
Sampling Guidance
EPA's WSD is developing detailed guidance for the sampling of chemical, radiological, or biological
agents in drinking water to address all activities associated with sampling for chemical, radiological, and
biological agents in drinking water, including sample collection and sample handling for samples
collected in response to a trigger from one or a combination of CWS information streams (sample
guidance). The guidance would describe the steps for sampling all potential WS baseline contaminants
and contaminant classes.
The sampling guidance is aimed at individuals directly involved in collecting samples for analysis. The
methods contained in the guidance would apply to both non-emergency uses, such as routine monitoring,
determining background concentrations of WS baseline contaminants; as well as emergency response
related to a possible breach of security. The guidance may supplement a utility's emergency response
plan to provide more detailed sampling procedures for drinking water utility personnel during a possible
contamination event.
DRAFT-121205 129
-------
WS System Architecture
Protocol for Analysis of Samples that Contain Unknown Contaminants (Unknowns
Protocol)
EPA's WSD is developing a protocol for the analysis of samples that contain unknown contaminants that
can help any water utility in their incident response activities, as well as the WS pilot utility, if a potential
contamination incident is investigated (USEPA, 2005f). EPA anticipates that the 'unknowns protocol'
would present staged analyses for chemical, radiological, and/or biological agents in drinking water to
help narrow down, and ultimately identify, the contaminant(s) in the sample. The protocol can also be
used to develop laboratory practices and standard operating procedures during contamination events.
The unknowns protocol is aimed at the individuals directly involved in analyzing the samples, and would
provide a common-sense management approach to laboratory activities that need to be considered during
investigation of a drinking water contamination incident. The protocol for analyzing water samples
suspected of containing an unknown contaminant should be robust enough to detect WS baseline
contaminants and contaminant classes as well as other contaminants that could be introduced into the
drinking water system through intentional or unintentional means.
Although the analytical protocol described should be able to detect WS baseline contaminants and
contaminant classes as well as other contaminants, EPA has recognized that not all laboratories would
have all of the instrumentation listed in the protocol. EPA anticipates that guidance would be provided on
how to prioritize in-house analytical capabilities and determine what steps can be taken in the event of a
contamination incident.
Water Laboratory Alliance
EPA's WSD worked with technical experts and stakeholders in the drinking water community to design a
conceptual framework for the Water Laboratory Alliance, a laboratory network that would support the
WS-CWS and provide surge capacity in the event of a water contamination threat or incident (USEPA,
2004b). EPA plans to integrate the Water Laboratory Alliance with the existing Laboratory Response
Network (LRN) to supplement the network's existing clinical analysis capability with environmental
analysis capability, including presumptive analyses by utility laboratories. EPA anticipates that the
Water Laboratory Alliance would also be integrated with EPA's eLRN to leverage aspects of their toxic
industrial chemical and chemical warfare capability and laboratories. In addition to working closely with
existing laboratory programs, as part of the Water Laboratory Alliance program development, EPA plans
to develop a protocol for availability and access to standards and reagents, implementation of
performance evaluation (PE)/proficiency testing (PT), and training programs and laboratory drills.
Method Development and Validation for Pathogens
In response to the need for monitoring and detection of pathogens and based on the WS contaminant
selection process, the EPA WSD is standardizing and validating culture-based and molecular methods for
five pathogens (Bacillus anthracis, Burkholderia pseudomallei, Francisella tularensis, Salmonella typhi,
and Vibrio cholerae) and is working with NHSRC, the U.S. Army's Edgewood Chemical and Biological
Center (ECBC), and CDC to leverage new methods being evaluated at ECBC and current techniques used
by CDC. EPA WSD is already standardizing the culture-based methods concurrent with an on-going
evaluation of method detection limits for these same methods at ECBC. Independent validation studies
would be conducted on the standardized methods to verify the procedures' performance at laboratories
representative of those that would use the methods during the WS pilot and in overall water security
support.
DRAFT-121205 130
-------
WS System Architecture
In addition to ongoing assessments of the suitability of the molecular methods for these five pathogens
being developed at ECBC, EPA WSD is also evaluating commercially available molecular method
options. Evaluation criteria include performance, ease-of-use, and ability to implement for use as part of
the WS pilot. After the assessments are complete, WSD aims to move forward with method
standardization and validation, similar to the process already underway for the culture-based methods.
Method Development and Validation for Chemicals and Biotoxins
EPA's WSD is developing methods for chemicals and biotoxins for which no validated drinking water
methods currently exist. EPA anticipates that these methods would respond to the need for monitoring
and detection of hazardous contaminants that would result in high mortality rates or have the potential for
major public health impacts. As part of this program, EPA WSD plans to expand the list of contaminants
already validated for existing drinking water methods to include contaminants of concern identified by
WSD, thereby simply expanding proven analytical techniques currently used by environmental
laboratories to include WS baseline contaminants where these are not already included in the scope of the
method.
EPA anticipates that single laboratory validation and lowest concentration minimum reporting limit
studies for these methods would begin after the study plans are finalized.
Water Contamination Information Tool
The Water Contaminant Information Tool (WCIT) is a secure, online database under development to
provide information on contaminants of concern for water security. As a planning tool, WCIT can be
used to help create and update vulnerability assessments, emergency response plans, and site-specific
response guidelines. As a response tool, WCIT can be used to provide real-time data on water
contaminants to help first responders (including utilities) make better decisions. In addition, EPA
anticipates that WCIT would help determine what information about priority contaminants is missing,
which would direct future research efforts. Contaminant specific information contained in WCIT can be
used in support of the WS pilot implementation in areas related to monitoring and surveillance, laboratory
analysis, and consequence management. For example, a utility could use WCIT as a tool to identify tastes
and odors associated with a particular contaminant. Also, if a utility receives a notification from public
health related to certain symptoms, WCIT could be used as a tool to assist in the credibility determination
process to identify contaminants associated with these symptoms based on exposure to contaminated
drinking water.
NEMI-CBR
The National Environmental Methods Index-Chemical. Biological, Radiological (NEMI-CBR) developed
jointly by the EPA Standards and Risk Management Division (SRMD) and the U.S. Geological Survey is
an additional source of information on analytical methods that could be used to support WS sampling and
analysis activities. NEMI-CBR is a web-based database for locating, evaluating, comparing, and
retrieving analytical methods for chemical, biological, and radiological-related contaminants in water.
NEMI-CBR includes methods for both screening and confirmation and provides multiple methods for the
same analyte, where applicable. The companion Expert System, CBR Advisor, is linked to NEMI-CBR
and provides advice (based on EPA's Response Protocol Toolbox) for evaluating threats, safely collecting
samples, and selecting the best method for a given situation, even when the contaminant identity is
unknown. NEMI-CBR is designed to be used as a planning and training tool by laboratories in
preparation for an intentional or accidental contamination event from chemical, biological, and
radiochemical agents.
DRAFT-121205 131
-------
WS System Architecture
SAM
EPA's Standard Analytical Methods for Use During Homeland Security Events (SAM) compendium also
provides yet another resource for analytical methods and considerations for implementation that could be
leverage to support WS sampling and analysis activities. SAM provides pre-selected methods (validated
and non-validated) that could be used in a terrorism event in which multiple laboratories would be
involved. SAM specifies one method per contaminant/matrix combination to enable sample loads to be
shared among laboratories while maintaining data comparability and to simplify the task of outsourcing
analytical support to the commercial laboratory sector. The single-method approach also would improve
the data validation efficiency. SAM includes only confirmatory methods for analytes in a number of
matrices (solid, oily-solid, aqueous/liquid, drinking water, air, surface, and dust) and is intended to be
used when the agency responsible for managing response to an incident determines that multiple
laboratories are needed for sample analysis during an event. The current version of the SAM document
can be found at: http://www.epa.gov/ordnhsrc/pubs/reportSAM092905.pdf
DRAFT-121205 132
------- |