vEPA
United
Environmental Protection
Ag*ncy
Overview of Event Detection Systems for
WaterSentinel
Draft, Version 1.0
December 12, 2005
-------
-------
U.S. Environmental Protection Agency
Water Security Division
Ariel Rios Building, Mail Code 4601M
1200 Pennsylvania Avenue, N. W.
Washington, DC 20460
EPA817-D-05-001
-------
-------
WS Event Detection Systems
Disclaimer
The Water Security Division, of the Office of Ground Water and Drinking Water, has reviewed and
approved this draft document for publication. This document does not impose legally binding
requirements on any party. The word "should" as used in this Guide is intended solely to recommend or
suggest and does not connote a requirement. Neither the United States Government nor any of its
employees, contractors, or their employees make any warranty, expressed or implied, or assumes any
legal liability or responsibility for any third party's use of or the results of such use of any information,
apparatus, product, or process discussed in this report, or represents that its use by such party would not
infringe on privately owned rights. Mention of trade names or commercial products does not constitute
endorsement or recommendation for use.
Questions concerning this document or its application should be addressed to:
Irwin Silverstein, Ph.D., P.E.
U.S. EPA - New England, Region I
1 Congress Street
Boston, MA 02114
(617)918-1632
Silverstein.Irwin@epamail.epa.gov
DRAFT-121205
-------
WS Event Detection Systems
Acknowledgements
The Water Security Division would like to recognize the following organizations and individuals for their
support in the preparation of Overview of Event Detection Systems for WaterSentinel:
Steve Allgeier, Office of Water - Water Security Division
Regan Murray, Office of Research and Development - National Homeland Security Research
Center
Sean McKenna, Sandia National Laboratories
Doron Shalvi, Computer Sciences Corporation (CSC)
DRAFT-121205
-------
WS Event Detection Systems
Executive Summary
An effective contamination warning system (CWS) should be able to identify deviations from established
baselines and system anomalies in a timely manner by integrating and analyzing information from online
water quality monitoring, sampling and analysis, enhanced security monitoring, consumer complaints,
and public health surveillance. The WaterSentinel (WS) approach for a CWS involves the active
deployment and use of monitoring technologies/strategies that are not contaminant-specific and enhanced
surveillance activities to collect, integrate, analyze, and communicate information from a variety of
sources. These information streams are utilized to provide a timely warning of potential water
contamination incidents and initiate response actions to minimize public health and economic impacts.
As part of an effective CWS, a method for event detection should be developed such that it can quickly
recognize true contamination incidents while minimizing the occurrence of false positives. An effective
event detection system, coupled with the other components of the WS-CWS, should contribute to the
CWS goal of reliable and timely detection of contamination to protect public health and water system
infrastructure.
The event detection system component of the CWS includes methods to distinguish potential
contamination incidents from the normal background variability of the baseline. Inability to identify true
incidents could result in false negatives for the WS-CWS, with potentially detrimental impacts on the
health and security of the public. On the other hand, the consequences of a false alarm can also be
significant and consideration should be taken to minimize false positives. Hence, the event detection
system should employ software that can 'train itself on the normal variation in daily, weekly, and
seasonal water quality patterns so that it can reliably recognize an anomaly that is truly indicative of a
contamination incident. Algorithms, utilizing discrimination, clustering, or statistical techniques, are
being used to identify anomalous patterns in the fields of public health and cyber security. Laboratory
studies have determined that the use of algorithms for event detection in the environmental field is
feasible. Currently, several field projects are underway using a combination of algorithmic approaches to
identify anomalies in water quality patterns. Other organizations have developed software for evaluating
water quality with predetermined threshold points acting as alarm triggers, as opposed to triggers based
on sophisticated algorithms. Regardless of the degree of sophistication of an event detection system,
human judgment and interpretation should always be an element of the credibility determination and
decision-making process.
Present findings suggest that methods are available for testing and evaluating the event detection
component of the WS-CWS. However, these results are merely preliminary and should be utilized with
caution. Further testing of event detection within the context of a working utility distribution system is
needed to validate its usefulness within an integrated system, such as the WS-CWS. Long-term testing
should better ascertain the effectiveness, ability to be replicated, and sustainability of an event detection
system and the entire WS-CWS system architecture.
DRAFT-121205 in
-------
WS Event Detection Systems
Table of Contents
Executive Summary iii
Section 1.0: Introduction 1
1.1 Recognizing Anomalies 1
1.2 Event Detection and Consequence Management 2
1.3 Objectives 3
1.4 Document Organization 3
Section 2.0: Overview 5
Section 3.0: Event Detection and Water Quality Anomalies: Proof of Concept 7
3.1 Categories of Event Detection Algorithms 7
3.2 Demonstrating the Concept of Applying Event Detection Software to Water Quality Data 8
3.2.1 Study at EPA's Test and Evaluation (T&E) Facility 8
3.2.2 Decision Support Software Project (Discrimination and Clustering Algorithms) 9
3.2.3 Algorithm Development Project (Statistical Algorithms) 9
Section 4.0: Evaluating the Effectiveness of Event Detection Systems 11
4.1 Evaluation Criteria 11
4.2 Receiver Operating Characteristic (ROC) Curves 14
4.3 Technology and Testing Evaluation Program (TTEP) 16
Section 5.0: Projects Currently Using Event Detection 17
5.1 Decision Support System for Water Distribution System Monitoring for Homeland Security ...17
5.2 Hydra Remote Monitoring System (RMS): A Case Study on Beta Test Sites and Results 18
5.3 Water Quality Change Detection 19
5.4 RODS Project 20
5.5 The Electronic Surveillance System for the Early Notification of Community-based
Epidemics (ESSENCE) Project 20
5.6 California Space Authority Water Monitoring Project 21
5.7 Data Processing and Analysis for Online Distribution System Monitoring 21
DRAFT-121205 iv
-------
WS Event Detection Systems
Section 6.0: Other Water Quality Analysis Activities 23
6.1 Additional Projects 23
6.2 Commercial-Off-The-Shelf (COTS) Products 24
6.2.1 MIKE NET-SCADA 25
6.2.2 PureSense 25
6.2.3 AQUIS 25
6.2.4 Psynapse Technologies 26
6.2.5 Clarion 26
6.2.6 Sensicore 26
6.2.7 Bristol Babcock 26
Section 7.0: Summary and Preliminary Conclusions 27
Section 8.0: References 29
Appendix A: Acronym List 33
List of Figures and Tables
Figure 1-1. Overview of WS Concept of Operations 3
Figure 4-1. EDS Evaluation Process 11
Figure 4-2. Comparative ROC Curves 15
Figure 5-1. Water Quality Profile of an Alarm Event 19
Table 6-1. Other Water Quality Monitoring Projects 23
DRAFT-121205
-------
WS Event Detection Systems
Section 1.0: Introduction
In response to Homeland Security Presidential Directive 9 (HSPD 9) and by its authority under section
300i-3 of the Safe Drinking Water Act (42 USC section 1434), the U.S. Environmental Protection
Agency (EPA), in collaboration with other agencies, plans to build upon and expand current monitoring
programs to expand current surveillance and monitoring systems that provide early detection of water
contamination. Ideally, these systems would identify the presence of contamination prior to human
exposures that would result in public health impacts. Currently available technologies for deployment in
distribution systems are not sufficiently advanced to detect specific contaminants and provide such a
timely alert. In addition, a system architecture that relies on the detection of specific contaminants faces a
number of challenges for success because it is unlikely that the technologies would be able to provide
contaminant-specific detection for all potential contaminants and the costs associated with deployment of
multiple contaminant-specific technologies at a number of locations throughout complex distribution
systems would be overwhelming. Also, water utility personnel would be severely burdened by the
calibration, operation, and maintenance needs of what would likely be highly sophisticated technologies.
To meet the charge of HSPD 9, the WaterSentinel (WS) program is a demonstration project whereby
EPA, in partnership with a pilot utility and laboratories, would design, deploy and evaluate a model
contamination warning system (CWS). The WS approach involves the active deployment and use of
monitoring technologies/strategies that are not contaminant specific and enhanced surveillance activities
to collect, integrate, analyze, and communicate information from a variety of sources to provide a timely
warning of potential water contamination incidents and initiate response actions to minimize public health
and economic impacts. More information about WS program design is available in WaterSentinel System
Architecture (USEPA, 2005a).
1.1 Recognizing Anomalies
The different data streams for decision-making envisioned for the WS-CWS include sensors for
conventional water quality parameters (e.g., pH, residual chlorine, total organic carbon, conductivity,
etc.), public health surveillance information, consumer complaints tracking, routine or triggered sampling
and analysis data, and enhanced security monitoring. The fundamental challenge to the reliance on a
variety of information streams as an indication of a contamination incident is a means of distinguishing
anomalous patterns in these data from background signals. Statistical formulas and mathematical models
that analyze data are called algorithms. While not widely deployed for analyzing water quality data and
consumer complaint clusters, algorithms incorporated into event detection software also can be used to
identify and 'learn from' changes in data patterns that are indicative of the introduction of a contaminant
into the source water or the distribution system of a water utility, or at least indicative of a significant
change from the current nature of water quality. More information about how water quality parameters
change in response to the intentional or accidental introduction of contaminants into a water utility's
distribution system is available in WaterSentinel Online Water Quality as an Indicator of Drinking Water
Contamination (USEPA, 2005b).
Algorithms currently are being used to recognize anomalies in such fields as public health (Watson, W.,
et al., 2005), cyber security (Jackson, 2003), and for jet engine maintenance (Shroder, 2005). Algorithms
can also be used to optimize the number and locations of contaminant sensors in a network (Berry, et al,
2005; Uber, et al., 2004; Ostfeld and Salomons, 2004). However, as pointed out by McKenna, et al.,
DRAFT-121205
-------
WS Event Detection Systems
2005, the majority of the work in this area has assumed contaminant-specific sensors with perfect
detection characteristics.
In the public health field, detection of an anomaly that is two to three standard deviations from the
baseline observation is typically the threshold for sending an alert to the appropriate public health
representatives. For example, the American Association of Poison Control Centers receives poison
incidence monitoring data in real-time that are first evaluated by algorithms at hourly intervals. If
anomalies in the data reflecting an increased number of calls, unusually high reporting of a specific
substance, and clusters of unusual health effects are cited, the results are sent to subject matter experts to
make a final determination whether the data outliers are of potential public health significance. In
addition, the application of event detection algorithms in the public health field is being expanded to the
field of water security. In the case of the Real-time Outbreak and Disease Surveillance (RODS) project
described below, anomalies regarding emergency room visits, sales of over-the-counter (OTC) drugs, and
hospital admission rates can be indicative of the effect of exposure to a waterborne contaminant. This
type of event detection based on syndromic surveillance is being combined with event detection for water
quality with the expectation that better communication and information sharing between the public health
community and the water utility will shorten the recognition time necessary for taking appropriate
response actions. In addition, the Electronic Surveillance System for the Early Notification of
Community-Based Epidemics (ESSENCE) project described below is a biosurveillance system being
developed to evaluate data from pharmacies, hospitals, clinics, etc., with the intent to integrate a water
quality data stream into an overall event detection system (EDS).
1.2 Event Detection and Consequence Management
Event detection is the trigger that sets in motion the process of consequence management that includes
credibility determination, response actions, and recovery. The relationship between event detection and
consequence management in the WS concept of operations is described in WaterSentinel System
Architecture (USEPA, 2005a) and illustrated in Figure 1-1 below. While event detection initiates the
credibility determination process, it is the analysis of the information associated with this trigger that
determines the set of response actions the utility should take. These actions can include sending a site
characterization team into the field and requesting information from law enforcement and/or public health
agencies.
While the EDS can indicate a possible contamination threat, it is only part of the credibility determination
process and it alone cannot be the sole decision-making tool for initiating response actions that are
warranted to protect public health. In addition, the human element is integral to the credibility
determination and subsequent consequence management steps. However, it is possible to develop a tool
to support officials in decision-making by guiding them through the initial stages of the evaluation
process and aiding in the synthesis of information necessary to make timely and appropriate response
decisions. Although ultimately the decision should always rely on human judgment and the evaluation of
incomplete information, this decision tool can be a great aid in the process, and might substantially reduce
the time to make critical response decisions. Therefore, event detection software that utilizes algorithms
developed to evaluate water quality data are an important component of a successful CWS.
DRAFT-121205
-------
WS Event Detection Systems
Monitoring an
Online monitoring
Sampling & analysis
Operational data
Consumer complaint
Enhanced security
| OTC sales
Chief complaints
| 911 calls
Poison control
| Other
Credibility Determination
Response
Remediation and Recovery
Credible or
confirmed
Rule out - Return to
routine monitoring
and surveillance
I
I sJ 1=14=111" W it=
ry Actions
Response
Figure 1-1. Overview of WS Concept of Operations
1.3 Objectives
The objectives of this document are as follows:
Describe the concept of event detection algorithms to identify water quality anomalies
Describe tools and approaches to evaluate the effectiveness of event detection
Cite current projects and currently available event detection products being used to evaluate water
quality data
Cite other projects involved with water quality monitoring
Discuss how the WS pilot is an appropriate exercise for building upon the preliminary results of
current event detection projects
1.4 Document Organization
The remaining sections of this document describe the following aspects of the WS EDS:
Section 2.0: Overview. This section presents an overview of the functions of EDSs.
Section 3.0: Event Detection and Water Quality Anomalies: Proof of Concept. This section
describes the different algorithm categories and ways to evaluate how they perform, to support
the online water quality monitoring component of WS.
Section 4.0: Evaluating the Effectiveness of Event Detection Systems. This section outlines an
approach that can be used for evaluating the effectiveness of an EDS.
DRAFT-121205
-------
WS Event Detection Systems
Section 5.0: Projects Currently Using Event Detection. This section describes projects
involving event detection in the water sector.
Section 6.0: Other Water Quality Analysis Activities. This section provides examples of
projects conducted by utilities, commercial vendors, and research organizations to improve water
monitoring capabilities and data analysis and a description of companies that offer off-the-shelf
tools for event detection and decision support.
Section 7.0: Summary and Preliminary Conclusions. This section provides a summary of the
document and some preliminary conclusions.
Section 8.0: References. This section provides a bibliography of the references cited in this
document.
Acronyms
A complete glossary of terms related to event detection and the WS program is available in
WaterSentinel System Architecture (USEPA, 2005a).
DRAFT-121205
-------
WS Event Detection Systems
Section 2.0: Overview
The most important function of the event detection software is to filter out the anomalies or changes in
water quality patterns that normally occur (e.g., changes in water turbidity caused by the surge effect of
activation of a booster pump in the water distribution system), or which have known causes, and signal
only those anomalies that are likely to be indicative of possible contamination incidents. In short, the
purpose of the event detection software is to reduce the false positive rate without missing potential
events. One approach to identifying an anomaly of concern employs a baseline estimator to quickly
distinguish false warnings from regular fluctuations in water quality data upon system start-up (Cook, J.,
et al., 2005). Overtime, the event detection software's programmed ability to become familiar with
signal patterns that are associated with 'normal' operations (i.e., actuation of a booster pump every day at
3:00 PM) will result in a continued drop off in the number of false warnings. This concept is analogous
to computer network security event detection software products that also define a baseline of normal
activity and report any events that do not neatly fall within a predefined cluster. For example, a baseline
profile of the network is created where all routine events are grouped into clusters representing normal
activity. The software 'trains' itself to recognize any patterns outside of those clusters based on what
different network users do, the resources they typically request, the types of files they transfer, etc. (Cook,
J., et al., 2005). In another example of event detection software, a neural network developed by
Intelligent Automation Corporation is used to interpret and learn from electrochemical signals produced
by fish exposed to source water, so that anomalous patterns can be recognized that are indicative of a
contamination event (Intelligent Automation Corporation, 2005).
The degree to which water quality data should be collected before a baseline is well established will
depend on the inherent variations in the distribution system. Although these variations can occur daily,
weekly, and seasonally, the process by which the event detection software 'learns' how to identify
anomalies can become functional early on (e.g., one event detection software vendor, discussed below in
Section 6.6, indicated that it would take two to three months for its software to complete an initial training
period using a utility's water quality data). As additional water quality data are received by the event
detection software, its ability to distinguish anomalies unrelated to a contamination incident (i.e., false
positives) from true incidents will improve. Although a period of up to one year may be necessary to
characterize certain seasonal trends, the EDS and online water quality monitoring system should become
more reliable long before this full level of variability is characterized (i.e., as the system learns the normal
degree of variability). Therefore, the pilot study should have a functioning EDS in its early stages and
like all components of the CWS, improvement is anticipated as experience is gained in data interpretation
and integration.
This document will describe various approaches to event detection, and characteristics of these systems
that impact performance. To demonstrate the extent to which event detection is currently being employed
in the field, a number of ongoing projects at water utilities, both civilian and military, will be presented.
These projects involve utilities that are field testing event detection algorithms for changes in water
quality and studies where the use of syndromic public health surveillance models that also rely on
algorithms are being evaluated. The document will also include a description of other utility activities
being planned which also incorporate event detection. The document will conclude with a discussion of
how current projects involving event detection have performed and how the WS program can build upon
this knowledge base during the pilot study, recognizing that all potential contamination events may not be
detected by surrogate water quality measures.
DRAFT-121205
-------
WS Event Detection Systems
Section 3.0: Event Detection and Water Quality Anomalies:
Proof of Concept
While event detection algorithms have been used for a number of years to identify anomalies in a variety
of other fields, they have limited use in the environmental field; however, a number of published and
ongoing studies indicate a growing interest in EDSs for environmental applications, including the
detection of anomalies in water quality data. Research investigations are under way with the EPA, the
military, water utilities, and vendors of event detection softwares where the algorithms are being used to
evaluate patterns in water quality data to identify contamination. The following sections describe the
different algorithm categories and ways to evaluate how they perform. Subsequent sections will present
projects that illustrate how this concept is being used in laboratory and field studies.
3.1 Categories of Event Detection Algorithms
Although many event detection algorithms exist, they generally fall into one of the following categories:
discrimination, clustering, or statistical techniques.
Discrimination involves the development of models to separate samples into two or more discrete classes
which are known in advance, using, for example, rules, decision trees, Artificial Neural Networks
(ANNs), Bayesian Networks, and Support Vector Machines. For example, some types of ANNs are
iteratively 'trained' to learn how to map input patterns to output classes. The iterative training process
involves teaching (supervised learning) the ANN so that it can adjust its parameters according to whether
its suggested output class(es) was correct or incorrect. Practical examples include training and using an
ANN to input the data from a sensor (following some preprocessing), and select one of several possible
output classes representing contaminants (e.g., strychnine, gasoline, ricin, none, etc.) which have been
defined in advance. While many other supervised learning approaches exist, the general idea in
discrimination is that given an input data pattern, a model is being trained to select one or more classes
that have been defined in advance. After being trained, the model can be used as-is for operational
scenarios, or can be further trained on-the-fly when presented with new training data. Therefore, it is an
adaptive model.
Clustering involves the development of models to organize the data into clusters which are not known in
advance, typically by looking at the similarity of the data according to selected measures based on
multiple measured variables (i.e., multivariate analysis). This type of learning is often called
unsupervised learning, since the target outputs are not known in advance, and there is no set of output
classes to train to as a target. As an example, this approach may be used to organize online water quality
monitoring sensor data into categories that were previously unknown. Such categories, indicated by
clusters of one or more cells in a map grid, represent clusters of input pattern data similar in nature.
Further examination is typically required to identify the input characteristics that led to the clustering; this
examination may lead to clusters of data representing, for example, normal water quality, low levels of
specific conductance, high levels of a toxin, presence of an unknown contaminant, and many others,
including clusters for which the signature similarity of the cluster is not readily apparent. In this fashion,
new, potentially actionable information may be obtained. This type of knowledge discovery may be more
useful in a retrospective analysis than a prospective one, allowing the system to first identify the set of
DRAFT-121205
-------
WS Event Detection Systems
possible clusters, and then train a separate component of the system to discriminate new patterns into the
clusters, using the discrimination techniques previously mentioned.
Statistical techniques involve the use of probabilities and sufficient population sample sizes to establish
normal and abnormal conditions in a system, and to make predictions of future events based on the
current state of the system being observed. A statistical model may be as simple as one that determines
the mean of a single observed parameter, such as chloride measurement from a sensor, and flags any
readings that fall outside of a certain number of standard deviations from the mean (assuming normal
distribution). More commonly, statistical control charts may be used to monitor one variable over time,
such as emergency room visits with respiratory or gastrointestinal complaints, triggering a response when
certain signals (i.e., step, spike, exponential) are detected in the data (Shmueli, G., 2005). In another
example, prior probabilities are used to compute future probabilities to determine whether spatially-
organized counts of OTC drug sales are significantly higher than the expected baseline, triggering an alert
(Neill, D.B. and Moore, A.W., 2005).
3.2 Demonstrating the Concept of Applying Event Detection Software to Water Quality
Data
The previous section described the different categories of event detection algorithms. The following
studies provide specific examples where event detection algorithms are being used to evaluate changes in
water quality patterns. The first two studies present results based on laboratory experiments. All the
contaminant data presented in the first study are for concentrations well below the literature LD50 values
(WaterSentinel Contaminant Fact Sheets, USEPA, 2005c). For the second study, sodium cyanide,
sodium fluoroacetate, and aldicarb were tested at concentrations below the lethal dose (which is based on
the assumption that a person weighing 70 kg consumes 2 liters of water per day) and the range of testing
for sodium arsenate encompassed the lethal dose for this fourth chemical (Inchem, 2005, CDC, 2005, and
Wikipedia, 2005). Therefore, these concentrations were 'recognized' during the test at concentrations
that would be protective of acute human health exposure. Research is underway by EPA's National
Homeland Security Research Center (NHSRC) to determine the health risks associated with consuming
sub-acute concentrations of WS priority contaminants. As that information becomes available it will
likely be used in the design of future studies. The third study indicates how algorithm development is
associated with the collection of baseline water quality data at a number of water utilities.
3.2.1 Study at EPA's Test and Evaluation (T&E) Facility
At EPA's T&E Facility in Cincinnati, Ohio, a laboratory study was undertaken to determine whether it
was feasible to monitor water quality sensor data for changes that were indicative of the injection of a
contaminant, and whether event detection software algorithms were capable of 'recognizing' such an
anomaly (more information about the change in these water quality parameters observed during this study
is available in WaterSentinel Online Water Quality as an Indicator of Drinking Water Contamination,
USEPA, 2005b). Water quality data were obtained before, during, and after the introduction of
contaminants (i.e., secondary effluent from a wastewater treatment plant, potassium ferricyanide, arsenic
trioxide, nicotine, aldicarb, a malathion-containing insecticide, a glyphosate-containing herbicide, and E.
coli K-12 strain with growth media), ranging in concentrations from about 6 milligrams per liter (mg/L)
to about 2 grams per liter, into the facility's recirculating pipe loop. Sensor data were continuously
collected and electronically archived to establish stable baseline conditions and to record sensor responses
to the contaminants injected.
DRAFT-121205
-------
WS Event Detection Systems
The types of change in sensor data were dramatic and consistent enough for several parameters, (e.g.,
specific conductance, total organic carbon (TOC), total/free chlorine, chloride, and oxidation reduction
potential (ORP)), to detect the injection of 11 of the 13 contaminants. After the pipe loop experiments
were conducted, the sensor data were provided to the Hach Company of Loveland, Colorado for
evaluation by its event detection software called the Event Monitor. The Event Monitor contains a library
representing the characterization of over 80 contaminants of concern through laboratory scale 'beaker'
experiments. By comparing water quality signature patterns to this library, the Event Monitor was able to
identify most of the contaminants introduced into the pipe loop. A report summarizing these experiments
is anticipated by mid November 2005 (Kroll, D. and King, K, 2005). A follow-up to this laboratory study
has been initiated using chemical and biological warfare agents (Hall, J. et al., 2005).
These experiments first demonstrated the feasibility in detecting chemical and gross biological
contamination based on a noticeable change in commonly used water quality sensors. Next an event
detection algorithm using discrimination was able to interpret the sensor data to not only recognize that an
anomaly had occurred but 'identify' what caused the anomaly. These results illustrate on a laboratory
scale how water quality sensors and event detection algorithms can work together to provide timely
warning of a contamination incident.
3.2.2 Decision Support Software Project (Discrimination and Clustering Algorithms)
In 2005, another laboratory study was initiated to investigate the capability of water quality monitoring
and event detection softwares for 'recognizing' that a contaminant had been introduced (see Section 5.1
for more detail). The first portion of this project involved the development of a prototype Decision
Support Software (i.e., event detection software) by using a laboratory flow loop to collect data about
water chemistry response using combinations of different commercially available, field grade water
quality sensors. During these experiments, four chemical threat substances (i.e., sodium arsenate, sodium
cyanide, sodium fluoroacetate, and aldicarb) were introduced at concentrations ranging from 15 to 100
mg/L (sodium arsenate), 3 to 10 mg/L (sodium fluoroacetate), 0.5 to 10 mg/L (sodium cyanide), and 1 to
10 mg/L (aldicarb). The event detection software first organized existing data into clusters and then
compared new data to the clusters to discriminate whether they fell within a normal or anomalous pattern
in real time. The laboratory results were encouraging as the event detection software was able to
discriminate normal water chemistry from the patterns signifying the introduction of contaminants (Byer
and Carlson, 2005).
Like the T&E experiments, this study demonstrated the feasibility in detecting chemical contamination
based on a noticeable change in commonly used water quality sensors. Similarly, event detection
algorithms, in this case using a combination of discrimination and clustering, were able to interpret the
sensor data and recognize that contamination had been introduced. A field testing phase of the study
began in September 2005 and can provide an opportunity to observe how well water quality sensors and
event detection algorithms work together in a distribution system. This phase, inclusive of a simulated
attack using modulated concentrations of fluoride, will more closely represent a test of the WS concept of
operations and therefore provide useful feedback in preparation for implementation of the WS pilot study.
3.2.3 Algorithm Development Project (Statistical Algorithms)
In a third study (see Section 5.3), actual water quality data (e.g., ORP, pH, conductivity, temperature,
dissolved oxygen (DO), and chlorine) collected at a southwestern water utility and a consortium of
northeast utilities, as well as from a simulated data set, are being used to develop event detection
algorithms. The use of a simulated data set allows for the insertion of 'events' so that the rate of false
positives and false negatives associated with the detection algorithms can be evaluated. This approach
DRAFT-121205
-------
WS Event Detection Systems
has the advantage over the real data set, which does not have any 'events' to detect. The transient
characteristic of measured water quality data in the distribution system necessitates algorithms that can be
updated in real time. The goal of the study is to develop a statistical model that first recognizes temporal
trends in key water quality variables associated with diurnal and seasonal cycles, the influence of physical
operations in the distribution system, electronic drift in water quality sensors or by dynamic
chemical/biological processes within the distribution system, and then confidently recognizes a change
caused by abnormal operating conditions (i.e., the introduction of a contaminant) (Srinivasan, S., et al.,
2005).
The preliminary results for this study illustrated how a statistical relationship between data collected
during one time period can be used to predict data that will be collected during future time periods. The
study also indicated that the event detection algorithm could be 'tuned' so that that during periods of
increased water security concern, the algorithm could operate at a higher probability of detection. In
addition, this study is using simulated data to test algorithm performance. Like the previous two studies,
the feasibility of using water quality data and event detection algorithms to detect contamination is being
demonstrated. Obtaining information on algorithm tuning provides an opportunity to evaluate how this
strategy might be used during the pilot to increase the probability of detection, with the corresponding
realization that during this time period, more false positives would occur. Finally, the use of simulated
data has value to the WS-CWS because it is an approach that can be used during the pilot study to test the
performance of the EDS.
In summary, the three studies described above show that all three types of event detection algorithms
(e.g., discrimination, clustering, and statistical) are being used in the environmental field to identify
anomalies in water quality data. In the studies involving discrimination and clustering algorithms, event
detection softwares successfully determined that contamination had been introduced into a laboratory
control loop based on the signals generated by standard water quality sensors, as envisioned for the WS-
CWS. Although proving this concept in the field will require further study, there are a number of current
projects that are investigating the combination of water quality monitoring and EDS as a means of
providing an early trigger to identify contamination in a distribution system. Many of these projects are
in the early stages where baseline data are being collected to 'train' the event detection software so that it
will be able to recognize anomalies and identify those associated with contamination incidents. A further
discussion of these projects is provided in Section 5.0.
DRAFT-121205 10
-------
WS Event Detection Systems
Section 4.0: Evaluating the Effectiveness of Event Detection Systems
Section 3.0 demonstrated that the application of event detection now includes the environmental field, in
addition to the other fields like public health and cyber security. While several examples were cited
where an event detection software product could fulfill the requirement for event detection in the overall
WS concept of operations, a process is needed in the pre-implementation phase of this program to
determine what event detection software product(s) should be used at the pilot utility. This section
outlines an approach that can be used for evaluating the effectiveness of an EDS. First, general criteria
are presented for evaluating event detection algorithms and tools, primarily based on information that can
be provided by event detection software vendors currently engaged in relevant projects. An initial
qualitative evaluation can be made based on this information. Next, a detailed discussion is presented
regarding the use of receiver operating characteristic (ROC) curves, which includes some of these criteria,
to relate tool performance to the occurrence of false positive and false negative responses. Simulated
data, inclusive of baseline profiles and 'contamination incidents', can be provided to vendors so they can
generate ROC curves and enable a more quantitative evaluation. Then, a more rigorous option, which can
build upon the previous steps, is presented involving third party validation of available event detection
products by means of EPA's Technology Testing and Evaluation Program (TTEP). This process is
depicted in Figure 4-1, including a final evaluation step that the pilot study would provide.
Simulated DaLa
ifi Data
V»nd» lnfiL(t
,
₯
jVeodac Down'
^ ,. . _ j SeltffiJ fJ Initial Ouantilatitfe
Qualitative Stop L __J| ~4
r- 5~ ~yi 5| p
S&bcEiDM for
IIEE h
"T
StilteCl'tKi fc
TTEP -Third Partv «"l N
Validation ~ T
Pilot Study - Final
Evaluation Step
Figure 4-1. EDS Evaluation Process
4.1 Evaluation Criteria
In the context of an EDS, algorithms and tools are distinguished from one another as follows. The
algorithm is the mathematical operation or statistical technique that is performed on the data for the
purpose of detecting anomalies (e.g., unusual trends in water quality) and is incorporated within the event
detection software or tool that interfaces with sensors, other data streams, and other utility software. The
following are some measures that can be used to evaluate the effectiveness of EDS tools and/or
algorithms.
Sensitivity: The sensitivity of a test is the proportion of those cases having a positive test result of all
positive cases (e.g., the proportion of people diagnosed with a waterborne disease relative to the total
number of people with the disease) tested; that is:
Sensitivity = (# true positives) /(# true positives + # false negatives)
In other words, the proportion of contamination incidents detected by the event detection software relative
to all the contamination incidents that occurred over a given period of time.
DRAFT-121205
11
-------
WS Event Detection Systems
Specificity: The specificity is equal to the proportion of true negatives of all the negative samples (e.g.,
the proportion of people diagnosed to be free of a waterborne disease relative to the total number of
people without the disease) tested, that is:
Specificity = (# true negatives) / (# true negatives + # false positives)
In other words, the proportion of time the system is detected to be without contamination relative to the
time the system is free of contamination (excluding false negatives). Specificity is also defined as 1.0
minus the false positive fraction (i.e., number of false positives divided by the sum of false positives and
true negatives).
F-measure: The F-measure can be used as a single measure of performance of the test. The F-measure is
the harmonic mean of sensitivity and specificity; that is:
F = (2 x sensitivity x specificity) / (sensitivity + specificity)
A perfect system, with no false positive or false negative results, would have an F-measure equal to 1.0.
An F-measure between 0.7 and 0.8 would be considered a 'good' value.
Time to Detect: This measures the delay between the time the event detection software first starts
receiving information about the contamination incident and the time that the system recognizes the
anomalous data as indicative of contamination. The response time can be measured between the
introduction of a surrogate contaminant or use of simulated data and an alert given by the event detection
software. The 'time to detect' for an EDS is primarily a function of the calculation interval and the nature
of the algorithm itself. It is also related to the F-measure (i.e., as the time to detect is decreased, the
sensitivity and specificity may degrade through the amount of data necessary to distinguish a true 'event'
from normal fluctuation in baseline data. For example, if an EDS signals an 'event' immediately when a
water quality parameter rises or falls, it may provide rapid detection, but at the cost of diminished
sensitivity and specificity. On the other hand, if an EDS needs ten minutes of data representing a
significant change from the baseline to signal an 'event', the time to detection goes down, but the
sensitivity and specificity should improve. Further discussion of the many other factors in a CWS that
affect the time to detect, is provided in WaterSentinel Contamination Incident Timeline Analysis (USEPA,
2005d).
Ability to handle highly variable data: Water quality data are influenced by many factors (e.g., seasonal
factors, source water, and treatment variables) and concentration baselines should show significant
change daily, weekly, and seasonally. Event detection software should have the ability to handle these
highly variable data to be effective over these various time scales, and should have the ability to relate
predictable water quality changes to known causes.
Adaptivity: Can the system learn on its own, or does it need to be re-trained over time? Adaptivity is
valuable in a system because it reduces the amount of off-line re-training or adjustment needed.
Resource requirements: This measure applies to the costs incurred as a result of time, labor, and
consumables expended during the installation, and operation and maintenance of the event detection
software, and in responding to an event trigger. This metric can also be used to track the costs associated
with the execution of the event trigger protocol to determine whether the expenditures were
commensurate with cause of the trigger (e.g., an event trigger that leads to a discovery by the utility that a
sensor calibration problem is the cause without the implementation of a drastic response action like a 'do
not use' order is indicative of good protocol because the resources expended were not excessive).
DRAFT-121205 12
-------
WS Event Detection Systems
Contaminant coverage: This measure applies to the range of contaminants or contaminant groups that
should be detected by the EDS. Note that contaminant coverage is also strongly dependant upon the
specific water quality parameters monitored. For the purpose of evaluating the EDS, it is assumed that
the sensors have the theoretical capability to detect the contaminant of interest.
The following aspects should be considered in addition to the measures discussed above when developing
and executing an evaluation/testing protocol for event detection tools and algorithms. These aspects
apply more in the context of how well the event detection tools and algorithms should perform when
integrated with the other components of the CWS:
Calibration: Does the tool require calibration runs of various contaminants in a water matrix in order to
identify or interpret events? In other words, is it sufficient for the tool to establish a baseline specific to
the pilot utility so that deviations from the baseline can be evaluated effectively by the event detection
software, or must training of the tool include the use of contaminants or surrogates in the pilot utility's
drinking water? Additionally, does the tool have an inherent ability to recognize a contaminant by
comparing water quality patterns to a pre-determined event library?
Compatibility: Is the tool compatible with existing data systems in use at the utility?
Cost: What is the cost of the tool? Is it targeted at enterprise-level (10K - 100K+), mid-range (1 - 10K),
or open-source / free? What are the calibration costs, especially if live agents are required?
Customization: Typically, there should need to be some customization of whatever tool is selected. This
is especially true for commercial-off-the-shelf (COTS) systems. How is this customization performed?
What kinds of skills (business analyst, database analyst, software developer) are needed to perform this
customization? What components (user interface, business rules, data processing) need to be customized?
Functionality: What is the ease of entering knowledge into the tool, or of the tool learning the knowledge
on its own? What kind of user interface is provided in order to use the tool, for tool-builders, knowledge
entry, end-users and other roles? Does the tool provide a 'user interface' builder so that an interface can
easily be built or modified for end-users?
Necessary Inputs: Does it require certain water quality sensors as inputs? Does it incorporate other data
streams as inputs (e.g., pressure data, OTC drug sales)? Can the algorithm work if one or more data
streams are temporarily unavailable (i.e., a sensor goes off-line for one or two days)?
Neutrality: Most tools are product neutral - they should run standalone, and against most or many
popular databases and other data stores. However, some event detection software products are developed
to be compatible with only one product line, e.g., there are water quality event detection software
products that should function only with the vendor's water quality sensors.
Performance Verification: Has the software been tested with simulated or actual baseline data to assess
its performance with respect to such parameters as false negative and false positive rates (e.g., verification
under TTEP)?
Scalability: Will the tool easily incorporate the addition of new sensors at a later point in time? Will the
addition of another sensor negate the existing knowledge base of the tool? Will calibration need to be
redone with the addition of a new sensor?
DRAFT-121205 13
-------
WS Event Detection Systems
Sustainability: What are the ongoing maintenance costs of the tool? How are upgrades deployed? How
do upgrades affect existing customized components?
Transparency: Is the algorithm well defined? Does it have a basis in existing theory and/or strong ties to
algorithms used in other fields? Does it rely on proprietary advances that will not be available for
review?
4.2 Receiver Operating Characteristic (ROC) Curves
The performance and reliability of an EDS (i.e., data collection and interpretation) depend on its ability to
minimize the number of erroneous 'detections' of an event that is not a contamination incident (i.e., false
positives) while avoiding the erroneous 'non-detection' of a contamination incident that has actually
occurred (i.e., false negative). False negatives are associated with improper sensor selection and
placement, lack of instrument sensitivity at low contaminant concentrations, interference caused by
background noise, and insufficient data analysis capability. False positives are associated with
oversensitive detectors that generate an indication of contamination when none exists. They can also be
caused by the presence of benign substances that mimic the interaction between a target contaminant and
a sensor or by inappropriate event detection software algorithms. The use of ROC analysis, first
developed in the 1950s as a byproduct of research involving the filtering radio signals noise, is an
important tool for determining how well the EDS should perform. Too many false positives can result in
complacency during an actual contamination incident, while the occurrence of an incident that is not
detected (i.e., a false negative) can have serious public health and public confidence ramifications because
an incident that the system was designed to detect was missed.
The generation of ROC curves is a means of determining the likelihoods of false negatives and false
positives from an EDS. These curves are produced by plotting sensitivity versus specificity. An ideal
EDS would have zero false negatives (i.e., 100% sensitivity) and zero false positives (i.e., 100%
specificity). In reality, such an ideal situation cannot be achieved. For example, the use of low detection
limit sensors would represent a situation where the sensitivity approaches 100% (i.e., minimal false
negatives because the ability to detect has been sharpened), but this heightened ability to detect increases
the likelihood that a detected anomaly that is not related to a contamination incident would trigger an
event detection software alert (i.e., a false positive) and as the number of false positives increase, the
specificity would drop. Because the consequences are much greater if an actual event is missed (i.e., a
false negative), a certain percentage of false positives should be acceptable. However, the consequences
of a false alarm can be significant, particularly if they result in substantial response actions, thus the false
positive rate should be minimized to the most extent possible.
Ranges of sensor performance with regard to sensitivity (false negatives) and specificity (false positives)
are illustrated by the ROC curve in the Figure 4-2.
DRAFT-121205 14
-------
WS Event Detection Systems
100^
Excel Jen
Good
Wonhless
i
eo go TOO
Specificity
Figure 4-2. Comparative ROC Curves (Source: IVD Technology, 2005)
Another way to interpret the ROC curve is to consider that a perfect event detection algorithm for all
detection threshold values would have a zero false alarm rate ('0' value on the x-axis corresponding to a
no false positives and therefore a specificity value of 100%) and a 100% probability of detection (100
value on y-axis corresponding to no false negatives): the top left corner of the curve. For non-perfect
algorithms, "the change in the threshold value at which measured values that deviate from predicted
values are considered to be an 'event' define a continuous relationship between the rate of false positives
and the probability of detection" (McKenna, et al., 2005). At relatively high detection limits, the rate of
false alarms will be low and so will the probability of detecting an 'event.' As the detection limits are
lower, more 'events' are detected and the number of false positives increases as indicated by top-left
curve. A poorer performing algorithm is represented by the central curve, while the 45-degree line
represents an algorithm that is no better than guessing (i.e., equal chance to be right or wrong) about the
occurrence of an event. At a conceptual level, the ROC curve shows that the ability to detect events and
the level of false alarms are inextricably linked, and have a positive and usually non-linear relationship.
The construction of a ROC curve requires that a set of events exists in a form that can be used to test the
event detection algorithms.
DRAFT-121205
15
-------
WS Event Detection Systems
4.3 Technology and Testing Evaluation Program (TTEP)
The EPA's Office of Research and Development's (ORD) and NHSRC established the TTEP to conduct
third-party performance evaluations of commercially available homeland security technologies. These
evaluations incorporate the guidance of stakeholders from the water sector and other federal agencies, as
well as a high degree of quality assurance oversight. Included among water security technologies are
water quality sensors, field test technologies, as well as software for distribution system modeling/design
and event detection. TTEP is in the process of reviewing event detection products (refer to next section)
from a number of vendors to solicit their participation and is developing protocols that will be used to
evaluate how effectively anomalies can be identified from variations in water quality data. The testing
protocol is currently in development and may include actual field data as well as simulated data. It is
anticipated that the initiation of testing and evaluation of potential event detection products for use in the
WS pilot study should begin in the last quarter of 2005 and be completed by the end of the first quarter of
2006.
DRAFT-121205 16
-------
WS Event Detection Systems
Section 5.0: Projects Currently Using Event Detection
There are many current projects involving event detection in the water sector. These projects typically
involve collaboration between a utility and governmental, research, and commercial entities. Several of
these projects are listed below.
5.1 Decision Support System for Water Distribution System Monitoring for Homeland
Security
Utility: South Carolina Commission of Public Works
Government: U.S. Air Force
Research: Colorado State University, American Water Works Association Research Foundation
Commercial: Advanced Data Mining International, LLC
Approach: Discrimination and Clustering
Status: Prototype, Proof-of-concept
This study was initiated at Colorado State University using event detection software developed by
Advanced Data Mining, LLC. Also involved in this project are the Charleston, South Carolina
Commission of Public Works, the U.S. Air Force, and the American Waterworks Association Research
Foundation (AwwaRF, Project No. 3086). The concept behind this project involves the use of
'intelligent' software or a decision support system (DSS) and conventional water quality monitoring to
determine whether a contaminant has been injected into a distribution system. The main objective of this
project is to prove the concept that such a combination of sensors and software can automatically learn
and remember the baseline pattern of 'normal' water quality characteristics (i.e., based on previously seen
data), detect deviations from this 'normal' pattern, based on new data, that could indicate contaminant
introduction, generate an alarm, and advise operators on follow-on actions. The approach is very similar
to the concept used by other software products described below. It uses neural networks and multi-
dimensional vectors to define normal and outlier behaviors (i.e., discrimination and clustering algorithms,
as described in Section 3.2.2).
As discussed above, a combination of clustering followed by discrimination algorithms was able to
distinguish pre-contaminant injection water quality sensor patterns from post-contaminant injection
patterns for four chemical threat substances, in laboratory pipe loop tests. These results were presented at
the AWWA Water Security Congress in Oklahoma City, Oklahoma, in April 2005 (Cook, et al, 2005).
The combination of DSS and water quality sensors now are being tested in the Charleston distribution
system. The main water quality parameters of interest include conductivity, pH, and UV-254 as a
surrogate for TOC. Baseline water quality data are being collected that include fluoride concentrations as
a prerequisite for evaluating how the EDS would respond to a simulated attack using modulated
concentrations of fluoride (Cook, 2005).
DRAFT-121205 17
-------
WS Event Detection Systems
5.2 Hydra Remote Monitoring System (RMS): A Case Study on Beta Test Sites and
Results
Utility: Greer, South Carolina water utility
Government: Savannah River National Laboratory
Research: Savannah River National Laboratory
Commercial: PDA Technologies, Inc., Hydra RMS
Approach: Statistical, Adaptive
Status: In development, beta testing
In Greer, South Carolina, PDA Technologies, Inc. is working with the water utility to manage water
quality data as well as to assist the utility in improving its security (e.g., biometric login and data
encryption are part of the system being provided for Greer by PDA). The components of the data
management system include the use of a real-time hydraulic model (e.g., EPANET or PipelinNet),
collection of standard water quality parameter data using multi-probe sensors from various vendors (e.g.,
Hach Pipe Sonde 7 in 1) at 5 locations in the distribution system, geographic information system (GIS)
data, and event detection software (Hydra RMS) that interprets the data streams with an adaptive
monitoring algorithm. The manufacturer claims that public health surveillance data and consumer
complaint information that would be tied to GIS, are examples of additional data streams that can be
interpreted by the event detection algorithm, especially with regard to where an anomaly may have
occurred. The Hydra RMS is capable of two-way communication with the sensors over secure fiber
optics and includes trend analysis tools as it continuously compares real-time water quality data against
benchmark water profiles. An alarm notification can be sent via cell phone, pager, or email.
Beta testing of the EDS began in Greer in November 2003. Water quality data are received from 2 Hach
Pipe Sondes and are analyzed every 15 seconds. The purpose of the beta testing is to prove the concept
that by continuously collecting and comparing water quality data, any deviation from an established
baseline can be recognized by an event detection tool. The preliminary results of this study were
presented to the Metropolitan Washington Council of Governments in early 2005. The EDS has been
able to identify a range of water quality values that represent a baseline profile. The establishment of this
baseline has enabled the EDS to recognize as an 'alarm event' an incident in May 2004 where a
mislabeled concentration of sodium hydroxide was used during treatment (refer to Figure 5-1). This
result is an example of the type of dual benefit that can be attained by such a system (Yang, 2005).
The beta testing is still continuing as more baseline data are being collected and adaptive algorithms are
being further developed. Future plans include a cooperative research effort with Savannah River National
Laboratory to develop improved water quality parameter sensors that would enable the event detection
software to recognize patterns that would identify different contaminant groups. There also are plans to
conduct surrogate agent exercising of the event detection capability (Page, 2005).
DRAFT-121205 18
-------
WS Event Detection Systems
Beta Test: Greer, South Carolina
Profile Alarm
P1-Chem Profile Ntetch 5*2/2004 08,25
Figure 5-1 Water Quality Profile of an Alarm Event (Image was reproduced with permission from
PDA Technologies, Inc.)
5.3 Water Quality Change Detection
Government: USEPA ORD, U.S. Geological Survey (USGS), U.S. Dept. of Interior
Research: Sandia National Laboratories
Approach: Statistical
Status: In development
This project (see Section 3.2.3) is being conducted by Sandia National Laboratories (SNL), EPA's
NHSRC, and the USGS). Actual and simulated data are being analyzed by SNL to develop statistically-
based event detection algorithms. The use of a simulated data set allows for the insertion of'events' so
that the ROC of the detection algorithms can be evaluated. To date, statistical and data summaries have
been processed in detail for development of event detection algorithms (e.g., hourly measurements from
500 to 625 days) for five monitoring stations at the southwest utility for total chlorine, pH, temperature,
and specific conductance. Similarly, DO, pH, specific conductance, temperature, and turbidity data
collected from October 2003 to September 2004 at three stations in the northeast utility are being studied.
One of the three stations is source water, which presents a matrix and set of contamination concerns that
differ from the distribution system, and the other two locations are treated, chlorinated water (McKenna,
et al., 2005).
Among the preliminary results for this study are the development of two event detection algorithms using
measured (i.e., chlorine, temperature, pH, and conductivity) and simulated water quality data and
evaluation of the change detection algorithms using ROC curves for simulated data sets. ROC curves
DRAFT-121205
19
-------
WS Event Detection Systems
were generated for each algorithm for each of the water quality parameters and for a fused data set
consisting of all four water quality parameters. Improved ROC curves (i.e., a higher probability of
detecting an 'event' with a corresponding lower false alarm rate) resulted from the fusing of data, the type
of data input more analogous to the WS concept of using multiple water quality sensors. The study also
indicated that the event detection algorithm could be 'tuned' such that during periods of increased water
security concern, the algorithm could operate at a higher probability of detection with the corresponding
realization that during this time period, more false positives would occur. Additional investigation is
needed to determine how well the simulated data set corresponds to an actual contamination event that
could occur in the distribution system (McKenna et al., 2005).
5.4 RODS Project
Utility: Cincinnati, Pittsburgh
Government: NHSRC
Research: University of Pittsburgh, Carnegie Mellon University
Commercial: Hach
Approach: Discrimination and Clustering
Status: Operational, Primarily Syndromic Health Data, OTC Sales
The RODS System is maintained by the RODS Laboratory in Pittsburgh. RODS is a collaborative effort
among researchers at the University of Pittsburgh and the Auton Laboratory in Carnegie Mellon
University's School of Computer Science. Drs. Wagner, Tsui, and Espino founded the RODS Laboratory
in 1999 to investigate methods for real-time detection and assessment of disease outbreaks. Current
research interests of the faculty include algorithm development, assessment of novel types of surveillance
data, natural language processing and analyses of detectability.
Public health surveillance pilots are being conducted by NHSRC to test the concept of adding near real-
time water quality parameters as a data stream into the RODS program. The demonstration project
retrospectively evaluates historical water quality data/operating conditions with public health surveillance
indicators, tests algorithms that correlate public health and water quality data, and shares anomalies with
water utilities in addition to public health officials. Two utilities, Greater Cincinnati Water Works
(GCWW) and the Pittsburgh Water and Sewer Authority (PWSA), are currently participating in
NHSRC's RODS study.
The PWSA has submitted historical data to the RODS Laboratory and is currently working with RODS to
determine the best way to transfer real-time water quality data to RODS. Data proposed to be transferred
are being collected using a Hach Expanded Event Monitor panel and probes, which measure pressure,
temperature, conductivity, chlorine, ORP, DO, and turbidity. The GCWW is currently working with the
Cincinnati RODS Steering Committee to identify the best way to share anomaly notifications. Issues
being negotiated include patient confidentiality and the broad range of GCWW's service area, which
includes numerous health jurisdictions within two states.
5.5 The Electronic Surveillance System for the Early Notification of Community-based
Epidemics (ESSENCE) Project
Utility: Milwaukee, Washington Suburban Sanitary Commission (Montgomery County, MD),
Albuquerque, NM
Government: Walter Reed Army Institute of Research, Defense Threat Reduction Agency
(DTRA), NHSRC (evaluating), Department of Defense (evaluating)
DRAFT-121205 20
-------
WS Event Detection Systems
Research: Applied Physics Laboratory
Commercial:
Approach: Statistical, Unspecified Artificial Intelligence Algorithms
Status: Operational, Primarily Syndromic Health Data, OTC sales
The Johns Hopkins University Applied Physics Laboratory (JHU/APL) developed the Electronic
Surveillance System for the Early Notification of Community-based Epidemics (ESSENCE). ESSENCE
collects and analyzes a variety of data sources for the early recognition of abnormal community disease
patterns that could result from natural causes or terrorist activities. Similar to the RODS pilot, NHSRC is
also conducting a demonstration project with ESSENCE to evaluate the feasibility and predictive value of
analyzing water quality data with the health indicator data already available in ESSENCE.
NHSRC is currently negotiating a sole source contract with JHU/APL and pursuing ESSENCE Data
Sharing Agreements with Milwaukee Water Works and the Washington Sanitary Sewer Commission
(Maryland).
The objectives of both the RODS and ESSENCE projects are consistent with the event detection
component of the WS concept of operations shown in Figure 1-1. As indicated in the figure, not only are
event detection algorithms being used for water quality and public health data streams, but
communication between utility and the public health sector is inherent in the project design. This feature
underscores the importance of utility-public health interaction for determining how to best address any
identified anomalies as the credibility determination phase of consequence management is triggered. For
more information on RODS and ESSENCE, refer to WaterSentinel System Architecture (USEPA, 2005a).
5.6 California Space Authority Water Monitoring Project
Utilities: Contra Costa and Southern California Metropolitan Water District
Commercial: Frontier Technologies, Inc.
Approach: Statistical
Status: Planning Stage
This project is being overseen by the California Space Authority (a nonprofit corporation representing the
commercial, civil, and national defense/homeland security interests of California's diverse space
enterprise community) to investigate the remote transmission of water quality monitoring data using
satellite technology that include standard water quality parameter data (chlorine residual, conductivity,
TOC, pH, and ammonia). The intent is to evaluate the data using an approach analogous to the use of
event detection algorithms developed by Frontier Technologies, Inc. to test jet engine performance. This
project is still in the initial planning stages.
5.7 Data Processing and Analysis for Online Distribution System Monitoring
Utility (U.S.): Philadelphia and Oklahoma City
Utility (Australia): Melbourne and Newcastle
Research: AwwaRF and the Commonwealth Scientific and Industrial Research Organization
(CSIRO)
Approach: Discrimination, Clustering, and Statistical
Status: Baseline Data Collection and Initial Algorithm Development
DRAFT-121205 21
-------
WS Event Detection Systems
Water quality data are being collected on daily, weekly, and seasonal time scales from U.S. and
Australian utilities. Among the data being evaluated are pH, ORP, temperature, residual chlorine,
conductivity, pressure, flow, and turbidity. The intent of the project is to examine data processing
methods that can be used to identify anomalous patterns in online monitoring data associated with
specific contamination incidents. Among the data processing methodologies being used are time series
analysis and spectral analysis, state-space models, clustering and discrimination, and statistical control
charts. The design of this project is consistent with the WS approach (AwwaRF, 2005).
DRAFT-121205 22
-------
WS Event Detection Systems
Section 6.0: Other Water Quality Analysis Activities
There are many other activities in which utilities, commercial vendors, and research organizations are
working to improve water monitoring capabilities and data analysis. In some cases the data are being
evaluated with regard to predetermined set points instead of a more sophisticated approach using event
detection algorithms. In other cases, data are being transmitted to commercial entities for evaluation and
it is unclear whether set points or algorithms are being used. In many cases, the intent of the
organizations is to eventually have the capability to evaluate water quality data with an event detection
tool. Examples representative of these types of projects are summarized in Table 6-1. Also provided
below is a description of companies that offer off-the-shelf tools for event detection and decision support.
6.1 Additional Projects
Table 6-1. Other Water Quality Monitoring Projects
Utility/Location
Research
Commercial
Project Description/Status
Langley Air Force
Base/VA
No Research
Organizations
Involved
Hach
The U.S. Air Force has a Cooperative Research And
Development Agreement (CRADA) with Hach Company
that involves data collection at Langley Air Force Base.
Two event detection stations have been installed and the
data are sent to Hach. If funding becomes available, the
Air Force would like to conduct pilot tests at 3 of their
bases for the collection of water quality data and the use
of event detection algorithms, possibly using off the shelf
technology from two manufacturers in a parallel testing
format (e.g., Hach and PureSense).
Pinellas County/FL
University of
South Florida
Constellation
Technologies
Pinellas County is working on a project with Dr. Daniel
Lim of the University of South Florida and Constellation
Technologies to develop an on-line system to detect
multiple potential agents simultaneously. An initial pilot
field test is anticipated during 2005. This project does not
include an investigation of software to handle data output
from the system at this point in time.
DRAFT-121205
23
-------
WS Event Detection Systems
Utility/Location
Research
Commercial
Project Description/Status
Indianapolis Water
Company/IN
No Research
Organizations
Involved
Clarion
Sensing
Systems
Clarion Sensing Systems' (Indianapolis, IN) Sentinal is
a remote computing platform consisting of a hardware and
software system for distributed data that features logical
data processing at the monitoring sites and compatibility
with various forms of wireless and wired data
transmission. The system integrates sensor data into a
single display that presents information through the
internet, a local area network, or a local terminal. The
data are presented in a web page format with analytical
and historical data storage capability. Each monitoring
site has its own Internet Provider address and serves its
own web page to allow for specific site monitoring and
remote configuration of the water quality profile of the site.
The Sentinal system can be integrated into an existing
system such as the supervisory control and data
acquisition (SCADA), and its software is compatible with
spiral development approaches, since new sensor
technologies can be integrated into the system.
Clarion claims to have experience within its software
group to develop self-learning analytical software
appropriate for event detection and evaluation. This
group has developed self-learning software for monitoring
water quality in distribution systems and chemical
pollution in river water. Clarion is currently receiving
water quality parameter data in real time from the
Indianapolis Water Company, which is operated by Veolia
Water. The Indianapolis system is complex with over 60
sources of ground and surface water (Harmless, 2005).
Contra Costa/CA
No Research
Organizations
Involved
PureSense
This water retailer has a current project with PureSense
(see more information on PureSense below) involving the
collection of water quality data and the wireless
transmission of the data via an l-node to PureSense for
their evaluation (Fowler, 2005).
Copenhagen/Denmark
No Research
Organizations
Involved
Unknown
An emerging technology project in Denmark is integrating
information from existing data sources for a water
distribution network in a utility. In the pilot project in
Copenhagen, the data from real-time sensors are
provided to a SCADA system. The sensor information is
stored in a database allowing for analysis (the algorithm
approach not known). Automated checks of the system
compare against baseline measurement. Data are
validated with standard modules which will flag potentially
suspect or corrupt data. The project started in 2001 and
is expected to be completed in 2005 (USEPA, 2005e).
6.2 Commercial-Off-The-Shelf (COTS) Products
Many companies offer COTS tools for event detection and decision support. Two such examples
described above are the Hydra RMS made by PDA Technologies, Inc. (refer to Section 5.2) and the Event
Monitor by Hach Company (refer to Sections 3.2.1 and 5.4). Frontier technologies also manufactures
EDSs that are used to evaluate jet engine performance, and are being planned for use in a water quality
monitoring project (refer to Section 5.6). There are other companies that offer tools or toolsets focused
on event detection as applied to water systems; some of these companies and tools are listed below
(USEPA, 2005e).
DRAFT-121205
24
-------
WS Event Detection Systems
6.2.1 MIKE NET-SCADA
MIKE NET-SCADA unites EPA modeling software and SCADA systems in an effort to optimize system
performance to recognize and respond to alarm conditions. The system's online module performs real-
time comparisons of the measured and calculated data, automatic data pre-processing for the off-line
module, and pressure/flow calculations at any point of the system. The model results are stored back into
the SCADA database and the online viewer is used to display detailed model results. In addition, the
online module features automatic data validation procedures, in which all measurements are automatically
checked and validated with standard modules. These modules will tag questionable data and, if possible,
fill in gaps in the time series. This ensures that only validated data will be transferred and used as
boundary conditions in the strategic model, decreasing the potential for false alarms (USEPA, 2005e).
MIKE NET-SCADA's off-line module models IF-THEN scenarios, models system breakdowns, and
predicts system behavior using demand and control rules prediction. It uses Microsoft Access to store
and maintain model alternatives. Coupling the online and off-line results of MIKE NET-SCADA allows
the operator to quickly detect abnormalities and help analyze ways in which the abnormality can be
remedied or its impact minimized.
Analytical Technology, Inc.'s, (Collegeville, PA) Series C15 Water Quality Monitoring system allows the
user to choose those parameters for which monitoring is desired and to integrate those components into a
monitoring package suitable for continuous monitoring, alarming, and data collection. System
components are currently available for free chlorine, combined chlorine (for chloramine treated systems),
dissolved ozone, pH, ORP, conductivity, and temperature. In addition, DO and turbidity modules should
be added to the system in the future (USEPA, 2005).
6.2.2 PureSense
EPA has a CRADA with another event detection software vendor (i.e., PureSense) to test its ability to
mine data and detect an anomalous event. PureSense is developing a software suite that can be used
across a range of water sensors from various manufacturers. The software allows data mining and an
'early warning system' approach to detect anomalous events, but the algorithms are proprietary PureSense
offers services to utilities whereby the water quality data are transmitted to their Iowa facility for
evaluation. The PureSense System includes the following four components to enable data transmission
and analysis: 1) the iNode is a remote data communication device that uses cellular and Wi-Fi services
to collect monitoring data and send commands to remote sensors, 2) the iWatch is an internet data
management system that enables the integration of disparate data sets, including data from remote online
sensors, 3) the iServe performs automated analysis of real-time data, and 4) the AlertNet provides
automated alerts.
In November 2004, the T&E facility provided PureSense with data similar to what was provided to Hach.
A preliminary review of PureSense's performance in evaluating these data was conducted in 2005 and a
report is anticipated in November 2005. A prototype event detection software is anticipated in the first
quarter of 2006. Puresense is also involved with arranging field trials in collaboration with a number of
US water utilities.
6.2.3 AQUIS
AQUIS is a water network management system designed for both on- and off-line, real-time monitoring.
The software, produced by Seven Technologies of Denmark, is used to create models to efficiently
manage water resources. The models allow utility managers to minimize the impact of operational
DRAFT-121205 25
-------
WS Event Detection Systems
disruptions in order to maintain continuity and quality of service. The software also allows managers to
explore strategies for responding to emergencies, including the introduction of contaminants and
increased demands placed on the system by extensive fire-fighting or other surge demands. AQUIS is
currently in use in 1,500 cities across the world.
AQUIS offers a Contingency Management Software Package that has five modules designed to establish
a point of entry for contaminants, determine a method for limiting the spread of the contaminant, and
determine methods to mitigate any harmful effects. The modules include a model manager for GIS data
management and a hydraulic module for modeling throughout the distribution system. A water-quality
module tracks the chemical composition of the water throughout the system, and a diagnostic module
identifies the source of contaminants. Finally, a flushing module facilitates cleaning the distribution
network.
6.2.4 Psynapse Technologies
Psynapse has developed the Checkmate Intrusion Protection System for the Technical Support Working
Group (TSWG) within the U.S. Department of Defense. This product recognizes when non-typical
network activity is a genuine threat. It is intended as a cyber security product by combining computer and
behavioral science to conduct real-time assessment of each visitor to a network. Once determining that
the behavior of the visitor indicates an attempted security breach, access is terminated. Despite the
intended use of this product, the company claims that this neural net technology can be used to determine
indications in any data streams that suggest a non-normal anomalous event.
6.2.5 Clarion
Clarion Sensing Systems' Sentinal is a remote computing platform consisting of a hardware and
software system for distributed data that features logical data processing at the monitoring sites and
compatibility with various forms of wireless and wired data transmission. The application of this product
to the Indianapolis, Indiana water utility is described above in Section 6.1.
6.2.6 Sens/core
Sensicore, Inc., Ann Arbor, Michigan, manufactures software that can respond to a simple water quality
threshold value and then sound an alarm.
6.2.7 Bristol Babcock
The Briston Babcock Company, Watertown, Connecticut, supplies water quality monitoring systems that
can integrate into SCADA systems. These systems can provide online continuous monitoring for changes
in water quality parameters and also provide other security information to utility operators (Elf, 2005).
DRAFT-121205 26
-------
WS Event Detection Systems
Section 7.0: Summary and Preliminary Conclusions
The fundamental concept underlying the WS-CWS is the gathering, managing, analyzing, and
interpreting of different information streams in a timely manner to recognize potential contamination
incidents early enough to respond effectively. Event detection plays an important role in filtering out the
anomalies that normally occur, or which have known causes, and signaling only those events that are
likely to be possible contamination incidents. While event detection algorithms have a greater history of
use in the public health and cyber security fields, as compared to the environmental field, a number of
completed and ongoing laboratory and field research projects exists that involve the use of event detection
tools and algorithms to evaluate data obtained from online water quality monitoring in distribution
systems.
Preliminary results from the EPA T&E facility studies indicated that a variety of water quality sensors
showed noticeable responses to a number of organic and inorganic chemicals, as well as gross biological
contamination. A consistent pattern of sensor change was observed with the most consistent change
associated with specific conductance, TOC, total/free chlorine, chloride, and ORP. In addition, the SNL
study has demonstrated that statistical algorithms are capable of identifying 'events' associated with
variations of the same types of water quality parameters monitored at the T&E facility. Similarly, the
laboratory testing at Colorado State University demonstrated that clustering/classification algorithm-
based decision support software could successfully characterize water chemistry based on sensor data and
detect anomalies in real time when contaminants of potential concern were introduced into a closed loop
system.
Studying the performance of EDSs has progressed from the laboratory to the field. The projects in
Charleston and Greer, South Carolina continue to collect water quality data to establish a baseline profile
and as cited above, an 'event' was detected in the Greer distribution system caused by the improper use of
a water treatment chemical. Once sufficient baseline data have been collected in Charleston, the EDS will
be evaluated on its response to a simulated attack using modulated fluoride concentrations. Baseline data
collection and algorithm development are also continuing at the U.S. and Australian utilities participating
in the AwwaRF-CSIRO project. Furthermore, the RODS and ESSENCE projects are examining the
feasibility and predictive value of analyzing water quality data and health indicator data, whereby the
water and public health sectors will communicate with one another to best inform decision-making when
anomalies are identified, as envisioned in the WS concept of operations.
Although the reported results of the above projects are encouraging, they are preliminary at best. Also,
while these projects are comprised of portions of the system architecture envisioned for the WS-CWS,
none includes the entire set of data streams that comprise WS (i.e., water quality monitoring, consumer
complaint surveillance, security monitoring, periodic water quality sampling and analysis, and public
health surveillance). Long-term testing within an actual water utility's distribution system is needed to
verify how well event detection can perform both as an individual component of the WS-CWS system
architecture, and as part of a system of integrated data streams. The WS pilot program presents an
opportunity to conduct this type of testing and obtain knowledge that over time can be used to refine a
CWS adoptable by other utilities. Therefore, a field demonstration project that includes all the WS-CWS
components is needed to determine whether the system architecture can be replicated at other water
utilities, whether false positive rates are kept to a minimum without sacrificing the ability to detect a real
event, and whether this approach can be sustainable nationwide. It is anticipated that the capabilities of
EDSs will play a critical role in determining the success of these pilots.
DRAFT-121205 27
-------
WS Event Detection Systems
DRAFT-121205 28
-------
WS Event Detection Systems
Section 8.0: References
AwwaRF, A Summary of AWWA Research Foundation Projects, November, 2004/2005.
Berry, Jonathan W., Hart, W.E., Phillips, C.A., Uber, J.G., and Watson, P.W., "Validation and
Assessment of Integer Programming Sensor Placement Models," World Water & Environmental
Resources Congress, 2005.
Byer, D. and Carlson, K.H., "Real-Time Detection of Intentional Chemical Contamination in the
Distribution System," AWWA Journal, June 2005.
CDC website, 2005: http://www.cdc.gov/niosh/idlh/62748.html
Cook, J., Roehl, E., Daamen, R., Carlson, K., and Byer, D., "Decision Support System for Water
Distribution System Monitoring for Homeland Security," American Water Works Association - Water
Security Congress, 2005.
Hall, J., Zaffiro, A., Marx, R., Kefauver, P., Krishnan, R., Haught, R., and Herrmann, J., "Parameters for
Rapid Contaminant Detection in a Water Distribution System," American Waterworks Association -
Water Security Congress, 2005.
IPCS Inchem, "Chemical Safety Information from Intergovernmental Organizations - Arsenic" 2005:
http://www.inchem.Org/documents/pims/chemical/pimgQ42.htmtfDivisionTitle:7.2.1. l%20%20Adults
Intelligent Automation Corporation website, 2005:
http://www.iac-online.com/Products/product_detail.asp?product_id=71.
IVD Technology, 2005: http://www.devicelink.com/ivdt/archive/05/03/002.html
Jackson, G., Checkmate Intrusion Protection System: Evolution or Revolution. Pysnapse Technologies.
2003: www.psynapsetech.com.
Kroll, D. and King, K.,, "Operational Validation of an On-line System for Enhancing Water Security in
the Distribution System," American Waterworks Association - Water Security Congress, 2005.
McKenna, S., Wilson, M., Cruz, V., Madueuke, N., and Srinivasan, S., "Status Report: Task 4, Sandia
National Laboratories - EPA NHSRC Inter-Agency Agreement, FY'2005," 2005a.
McKenna, S., Hart, D., and Yarrington, L., "Impact of Sensor Detection Limits on Protecting Water
Distribution Systems from Contamination Events," for submission to the Journal of Water Resources
Planning and Management, 2005b.
Neil, D.B. and Moore, A. W., 2005, Methods for Detecting Spatial and Spatio-Temporal Clusters. In
Wagner, et al., eds., Handbook of Biosurveillance, 2005.
Ostfeld, A. and Salomons, E., "Optimal Layout of Early Warning Detection Stations for Water
Distribution System Security," Journal of Water Resources Planning and Management, 130(5), pp. 377-
385, 2004.
Personal communication with Robert Fowler, Contra Costa Water Utility, July 5, 2005.
DRAFT-121205 29
-------
WS Event Detection Systems
Personal communication with Martin Harmless, Clarion Sensing Systems, July 29, 2005.
Personal communication with Tom Elf, Bristol Babcock Company, August 2, 2005.
Personal communication with Dan Page, PDA Technologies, Inc., August 9, 2005.
Personal communication with Ron Shroder, Frontier Technologies, Inc., September 1, 2005.
Personal communication with John Cook, November 1, 2005.
Personal communication with Dr. Jeffrey Yang, LY International, November 4, 2005.
Srinivasan, S., Madueke, N., McKenna, S., "Literature Review of Approaches for Determining
Variability in Background Water Quality Parameters and Change Detection", 2005.
Shmueli, G., "Wavelet-Based Monitoring in Modern Biosurveillance," Working Paper, Smith School of
Business, University of Maryland, 2005.
Uber, J., Janke, R., Murray, R., and Meyer, P., "A Greedy Heuristics Model for Locating Water Quality
Sensors in a Water Distribution System," Proceedings of the ASCE/EWRI Congress, 2004.
USEPA. WaterSentinel System Architecture, 2005a. For Official Use Only.
USEPA. Online Water Quality Monitoring as an Indicator of Drinking Water Contamination, 2005b. For
Official Use Only.
USEPA. WaterSentinel Contaminant Fact Sheets, 2005c. SENSITIVE. For Official Use Only.
USEPA. WaterSentinel Contamination Incident Timeline Analysis, 2005d. SENSITIVE. For Official Use
Only.
U.S. EPA, Technologies and Techniques for Early Warning Systems to Evaluate and Monitor Drinking
Water Quality: A State-of-the-Art Review, 2005e.
Watson, W., Litovitz, T., Klein-Schwartz, W., Rodgers, G., Youniss, J., Reid, N., Rouse, W., Rembert,
R., and Borys, D., Annual Report of the American Association of Poison Control Centers Toxic Exposure
Surveillance System, Toxicology, 2003.
Wikipedia, "Cyanide" 2005: http://en.wikipedia.org/wiki/Cyanide
DRAFT-121205 30
-------
WS Event Detection Systems
Appendix A: Acronym List
ANN
AwwaRF
COTS
CRADA
CSIRO
CWS
DO
DSS
DTRA
EDS
EPA
ESSENCE
GCWW
GIS
HSPD 9
JHU/APL
NHSRC
ORP
ORD
OTC
PWSA
RMS
ROC
RODS
SCADA
SNL
T&E
TOC
TSWG
TTEP
USGS
WS-CWS
WSD
Artificial Neural Networks
American Water Works Association Research Foundation
commercial-off-the-shelf
Cooperative Research And Development Agreement
Commonwealth Scientific and Industrial Research Organization
contamination warning system
dissolved oxygen
Decision Support System
Defense Threat Reduction Agency
Event Detection System
U.S. Environmental Protection Agency
Early Notification of Community-Based Epidemics
Greater Cincinnati Water Works
geographic information system
Homeland Security Presidential Directive No. 9
Johns Hopkins University Applied Physics Laboratory
National Homeland Security Research Center
oxidation reduction potential
Office of Research and Development
over -the-counter
Pittsburgh Water and Sewer Authority
Remote Monitoring System
receiver operator characteristic
Real-time Outbreak and Disease Surveillance
Supervisory Control and Data Acquisition
Sandia National Laboratories
Test and Evaluation
total organic carbon
Technical Support Working Group
Technology Testing and Evaluation Program
U.S. Geological Survey
WaterSentinel
WaterSentinel Contamination Warning System
Water Security Division
DRAFT-121205
33
------- |