AWWA's Water Utilities Users Group Meeting Status and Future Plans EPA’s Threat Ensemble Vulnerability Assessment TEVA Research Program American


A WW A's WATER UTILITIES USERS GROUP

MEETING

Status and Future Plans:

EPA's Threat Ensemble Vulnerability Assessment
(TEVA) Research Program

American Waterworks Association & Environmental Protection
Agency, National Homeland Security Research Center

September 13, 2006
Omi-Shoreham Hotel - Washington, D. C.

-------
A WWA Utility Users Group Meeting Notes

SUMMARY

Purpose:

The American Water Works Association (AWWA) and the United States Environmental
Protection Agency (US EPA) held a Threat Ensemble Vulnerability Assessment
(TEVA)/Water Utility Users Group Meeting at the Omni Shoreham Hotel in Washington,
DC on September 13, 2006, the day following the AWWA Water Security Congress.
These notes summarize the meeting presentations and discussions. The purpose of the
meeting was to:

•	Gain feedback from the User Group Utilities

•	Update Utilities on progress of the TEVA Research Team

•	Outline new initiatives planned for the TEVA Research Team

•	Continue to work collaboratively to advance Water Security

The top priorities identified by the Water Utility Users Group at the August, 2005 TEVA
Utilities and Users Group meetings were the need for: (1) field verification of the TEVA
models, (2) development of response tools, methodologies, and guidance, and (3)
development of improved capabilities for data analysis and integration. Specific
recommendations included the following:

¦	Field Experience

o Field validate TEVA consequence and sensor placement predictive
models

¦	Risk Management

o Develop response tools, methodologies, and guidance

¦	Data Analysis and Integration

o Understand water quality monitoring and integrate with other data

streams to develop an improved model of the water distribution system to
improve daily operations
o Automate the evaluation of water quality data along with other data

streams for operators to use in real time to make decisions
o Design methods to better use existing databases, e.g., customer
complaint and line break data

¦	Improvement of TEVA-CWS Tools - Assumptions

o Incorporate more realistic sensor performance characteristics into the

TEVA-CWS tools,
o Enable sensor network design for worst case objectives as well as

average impacts,
o Improve health impacts assessment methodology
o Re-evaluate feasibility of contaminant attack scenarios and develop
prescreening methods to limit the size of threat ensembles.

¦	Provide Water Utility Support

o Continue to work collaboratively with TEVA Utilities to perform

contaminant consequence assessments and design CWS.
o Assist in the design of field studies and/or the analysis of existing and
future water quality data sets.

2

-------
A WWA Utility Users Group Meeting Notes

In 2005, water utilities recommended that EPA perfect the TEVA software tools and
develop broad water utility guidance to improve water security rather than emphasize
software tool distribution. Several water utilities expressed security concerns about
releasing the tools into the public domain due to their usefulness in designing an attack.
Utilities emphasized the need for EPA to focus on developing definitive guidance that will
improve their ability to respond to contamination events, such as developing response
strategies. Also, WaterlSAC was recommended for distribution of sensitive products in
order to more quickly get useful information disseminated to the water community.

Examples of the TEVA Research Team's communication and outreach efforts
throughout include Contamination Warning System (CWS) Workshops, papers and
presentations, EWRI Special Journal issue on Water Security, and other activities that
are in process, pending, or planned. One of the pending projects is a WaterlSAC paper
that is in the final phase of internal approval. EPA committed to working on utility-
specific projects over the next year.

2006 TEVA Meeting Focus Track Discussions:

As a result of continued collaboration with the users group, the TEVA Research
Program has worked to address the priority issues identified from the August,
2005 meeting. The purpose of this meeting was to share some of these results
and advances with the users group and gather feedback. This discussion was
organized on three focus tracks that include an EPA briefing on the status of specific
research, open discussion with the water utilities to obtain feedback on the research and
tools being developed and consideration of options for software tool distribution and
possible use by the water utility community. The three focus track discussions
highlighted and discussed:

(1) Development and distribution of the TEVA-CWS software

(2) Development of contaminant event detection system tools, and

(3) Development of contaminant response tools and methodologies.

Consistent with the Users Group recommendation to improve TEVA software and
assumptions, an integrated TEVA consequence assessment and sensor placement
optimization tool, named TEVA-CWS, was demonstrated. The need to develop generic
guidelines for sensor placement was recognized as an important need, but more difficult
problem. The TEVA Research Team committed to investigating and hopefully
developing general guidance for consequence assessment and sensor placement
design for distribution systems. .

The TEVA-CWS is able to optimize sensor placement at a specific utility for numerous
competing objectives, consider perfect and imperfect sensors, and minimize impacts
based both on the mean and the maximum impacts. Sensor designs can be developed
to minimize time of detection, public health impacts, extent of contamination, or the
quantity of contaminated water. See focus track discussion No. 1, which
demonstrated the TEVA-CWS software.

EPA's efforts to develop Event Detection Systems (EDS) represented an
accomplishment in the area of Data Analysis and Integration. The EDS algorithms
distinguish between normal variations in water quality and changes in water quality
triggered by the presence of contaminants. The EDS are designed to be connected to

-------
A WWA Utility Users Group Meeting Notes

the SCADA system and, eventually, to a real-time model. The TEVA Research
Program's prototype EDS (Canary) was presented and discussed. The EDS tool will be
piloted at the WS utility in 2007. See focus track discussion No. 2, which
demonstrated the TEVA Research Program's EDS and provided an update on the
WaterSentinei field studies.

A moderated discussion on EPA's planned improvements to distribution system
modeling tools was provided during lunch. This discussion highlighted EPA's efforts to
develop new TEVA-CWS tools which could be used to reduce risks. The features,
status, and expected release dates of future TEVA Research Program software tools
was provided. To build a stronger foundation for research, it was noted that attention
needs to be given to improving network models and quantifying their accuracy. See
discussion on planned improvements to distribution system modeling.

The TEVA Research Program's efforts to support the water community and develop
tools which can aid Risk Management were highlighted by discussion of EPANET-RTX
and the demonstration of the contaminant source tracking tool. EPANET-RTX (an
extension to EPANET to perform real-time hydraulic modeling), a prototype algorithm for
contaminant source tracking or identification, and a TEVA utility's response options case
study were provided as examples of the progress being made in the development of
real-time tools and response measures. Currently available commercial software tools
provide near real-time modeling capabilities via their connection to SCADA systems.
EPANET-RTX will be a prototype extension to EPANET enabling real-time modeling to
support utility operations and other water security response tools. Maintaining EPANET
and using it as a foundation for tool development is critical to EPA's efforts to work with
utilities. Additionally, a status update was provided on EPA's efforts to perform field
studies to support the WaterSentinei pilot in Cincinnati. See focus track discussion
no. 3, which highlighted EPA's efforts to develop response tools, along with the
lunch discussion.

Next Steps:

The water utility users group requested that EPA identify the TEVA Research Program's
goals and provide a schedule for achieving such goals. EPA indicated that they would
develop a matrix of research goals along with a projected schedule to accomplish such
goals and provide the information to the Water Utility Users Group.

AWWA requested utility user group comments on the tentative plans for AWWA to
facilitate a broader discussion between utilities and the manufacturers. One of the topic
discussions will be determining what the manufacturers can address versus what EPA
can address. Utility participants suggested developing a framework for such a meeting
along with an agenda to stimulate more discussion. AWWA indicated that the meeting
with the manufacturers would be broader than just the TEVA research. AWWA noted
that the meeting with the manufacturers is being considered for 2007.

EPA will continue to work with the individual TEVA utilities on consequence assessment
and sensor placement.

-------
A WWA Utility Users Group Meeting Notes

Purpose

The American Water Works Association (AWWA) and the United States Environmental
Protection Agency (US EPA) held a Threat Ensemble Vulnerability Assessment
(TEVA)/Users Group Utilities Meeting at the Omni Shoreham Hotel in Washington, DC
on September 13, 2006, the day following the AWWA Water Security Congress. These
notes summarize the meeting presentations and discussions. The purpose of the
meeting was to:

•	Gain feedback from the User Group Utilities

•	Update Utilities on progress of the TEVA Research Team

•	Outline new initiatives planned for the TEVA Research Team

•	Continue to work collaboratively to advance Water Security

The top priorities identified by the Water Utility Users Group at the August, 2005 TEVA
Utilities and Water Utility Users Group meetings were the need for: (1) field verification
of the TEVA models, (2) development of response tools, methodologies, and guidance,
and (3) development of improved capabilities for data analysis and integration. Specific
recommendations included the following:

¦	Field Experience

o Field validate TEVA consequence and sensor placement predictive
models

¦	Risk Management

o Develop response tools, methodologies, and guidance

¦	Data Analysis and Integration

o Understand water quality monitoring and integrate with other data

streams to develop an improved model of the water distribution system to
improve daily operations
o Automate the evaluation of water quality data along with other data

streams for operators to use in real time to make decisions
o Design methods to better use existing databases, e.g., customer
complaint and line break data

¦	Improvement of TEVA-CWS Tools - Assumptions

o Incorporate realistic sensor performance characteristics into the TEVA-
CWS tools.

o Enable sensor network design for objectives other than protection against

average impacts (e.g., minimize worst case),
o Improve health impacts assessment methodology
o Validate contaminant attack scenarios and develop prescreening
methods to limit the size of threat ensembles.

¦	Provide Water Utility Support

o Continue to work and support the TEVA utilities
o Coordinate with utilities prior to additional modeling/simulation analyses
o Analyze existing and future water quality data sets

In 2005, water utilities recommended that EPA perfect the TEVA software tools and
develop broad water utility guidance rather than emphasis on tool distribution. Several
water utilities expressed security concerns about releasing the tools into the public
domain due to their usefulness in designing an attack. Utilities emphasized the need for

5

-------
A WWA Utility Users Group Meeting Notes

EPA to focus on developing concrete measures that will improve their ability to respond
to contamination events, such as developing response strategies. Also, WaterlSAC was
recommended for distribution of sensitive products in order to more quickly get useful
information disseminated to the water community.

Overview of Progress since August, 2005 TEVA Meeting

EPA committed to working on utility-specific support projects in 2006-2007. Areas of
progress in the TEVA program were highlighted. Improvements in the TEVA software,
improvements and modifications to utility models, and development of a software
repository with an error tracking database, were named as examples of the areas that
progress has been made. Extensive improvements have been made in the area of
quality assurance to better support the credibility of the TEVA results.

An integrated TEVA consequence assessment and sensor placement optimization tool,
named TEVA-CWS, was demonstrated. Producing generic sensor placement guidelines
is a more difficult problem, but the TEVA Research Team is working to generalize
information learned on vulnerabilities and sensor monitoring station placement to more
fully support the water community.

The Sensor Placement Optimization Tool (SPOT), developed at Sandia National
Laboratories and now incorporated into the TEVA-CWS, is able to optimize sensor
placement for numerous competing objectives, consider perfect and imperfect sensors,
and minimize impacts based both on the mean and the maximum impacts. Sensor
designs can be developed to minimize public health impacts, time of detection, extent of
contamination, or the quantity of contaminated water.

Event Detection Systems (EDS) algorithms were developed to distinguish between
normal variations in water quality and changes in water quality triggered by the presence
of contaminants. The EDS are currently being designed to be connected to a SCADA
system but, eventually, to a real-time model, and will be demonstrated at the
WaterSentinel pilot in 2007.

EPANET-RTX, a prototype algorithm for contaminant source tracking, and a case study
to examine feasible response options given a contaminant release in a distribution
system were cited as examples of the progress being made in the development of real-
time tools and response measures. Currently available commercial software tools offer
near real-time modeling capabilities via their connection to SCADA systems. EPANET-
RTX will be a prototype extension to EPANET enabling real-time modeling to support
utility operations. EPA noted that maintaining EPANET and continuing to use it as a
foundation for tool development is critical to their work with utilities.

Examples of the TEVA Research Team's communication and outreach efforts include
Contamination Warning System (CWS) Workshops, papers and presentations, EWRI
Special Journal issue on Water Security, and other activities that are in process,
pending, or planned. One of the pending projects is a WaterlSAC paper that is through
the last phases of internal approval.

6

-------
A WWA Utility Users Group Meeting Notes

Focus Track Discussion #1: Path Forward for TEVA-CWS

The discussion focused on TEVA-CWS software. The participants discussed what they
want to see in software applications and what issues there are with TEVA software and
its use. TEVA Software consists of utility specific contaminant consequence
assessment and sensor placement design; a contaminant event detection tool, also
known as Canary; contaminant source tracking; and, prototype EPANET enhancements,
e.g., multi-species extensions, distributed processor extensions, and real-time
extensions.

A demonstration of TEVA-CWS using a hypothetical contaminant and a TEVA utility
model was provided. TEVA-CWS outputs predict health impacts from contaminant
releases across the network. TEVA-CWS displays health impacts results in tables,
charts, and graphs, and uses the impacts data to determine optimal sensor placement
locations in order to minimize such impacts.

EPA requested that utilities consider the following questions when reviewing the TEVA-
CWS software:

•	What types of software would utilities likely use, if any?

•	Who at the utility would use TEVA software?

•	How should results be displayed, given security concerns?

As the WaterSentinel program expands to add more pilot projects in the next year, EPA
intends to release a version of the TEVA-CWS software to one or two consultants. In 1-
2 years, EPA may release the software to utilities. The purpose of this focus track
discussion was to obtain utility feedback on the three questions above in order to
determine the priority that EPA should place on the development of a utility version of
the TEVA-CWS software.

TEVA-CWS generates health impacts data given a user-specified ensemble of threat
scenarios (contaminant attacks) to optimize sensor placement. The software outputs
maps, charts, and data tables. The delay in response to contamination event is the most
critical parameter influencing the magnitude of public health impacts. If the response
delay time is too long, the contamination warning system does not provide much benefit,
therefore utilities must strive to minimize response time.

The software allows costs associated with sensor placement to be considered when
determining potential sensor locations. The software can be used to model sensor
placement based on several objectives, including minimizing mean mass of contaminant
consumed, cost, population exposed, and fatalities. The model can minimize public
health impacts to the users of the distribution system given some number of sensor
monitoring stations. Sensor monitoring stations are located at model junctions.

The tool can use census data or estimates of model node population based on water
demand using a per capita water usage rate. For use of census data, users need to
input the population per node. The parameters that can be specified to estimate health
impacts include:

•	Ingestion quantity (e.g., two liters of water per day)

7

-------
A WWA Utility Users Group Meeting Notes

• LD50 value for the contaminant

• Body mass (e.g., 70 kg) for chemical contaminants

• Latency time of the contaminant

• Fatality time of the contaminant

• Fatality rate of the contaminant

It was recommended that the tool include a glossary of contaminants that would
automatically populate the parameters for health data. EPA considered such a glossary
and decided not to include it due to security concerns. Some contaminant information
can be found in the Water Contaminant Information Tool (WCIT), which is available from
EPA/Water Security Division to subscribers.

Given that the population in an inner city can expand during the day, it was recognized
that changes in demand complicated the prediction of health impacts. For example,
some cities and suburbs, such as the Washington, DC area, are served by two different
water systems.

The TEVA-CWS also includes parameters regarding the duration, amount, and location
of the contaminant injection. One, several, or all locations can be specified. Multiple,
simultaneous contaminant release locations can also be specified. The injection points
can also be constrained by pipe diameter. Nodes associated with certain pipes need to
be presorted and then uploaded to the tool from a separate file. For instance, one could
prescreen model nodes for those connected only to pipes greater than 12-inches. All
junctions and all nodes can be selected. The "all-nodes" category includes tanks as well
as junctions.

A map is output to show the relative effect (public health impacts) of injecting a certain
amount of chemical at each node. The ability to show fatalities by injection location is
considered sensitive and may not be available on a tool developed and distributed for
wider use.

The tool allows very large numbers of threat scenarios to be considered in the design of
sensor monitoring locations. For security concerns, some aspects of the tool could be
encapsulated. A web-based tool could also be developed so that utilities would "log in"
to run contaminant consequence assessment and sensor placement design analyses.

The TEVA-CWS software is used to run large numbers of modeling simulations. The
High Performance Computing cluster located at EPA uses 38 compute machines and 72
processors. Running 8,000 scenarios with a few sensor placement designs could take
now as little as 45 minutes as opposed to a single computer which could take 24 hours.
EPA may be able to provide a large cluster of processors in a secure area and allow
utilities a secure connection to the software. EPA could then send unclassified
information back to the utility. Because the tool could provide the recipe for an attack,
EPA and the utilities would need to be able to control access to the software.

Water utility reactions, comments, and questions to the TEVA-CWS software
demonstration and possible distribution of the software to the wider water community
included:

¦ Tool would be useful, but may be too complicated for a novice user.

-------
A WWA Utility Users Group Meeting Notes

¦ If the user inputs the data incorrectly, without a full understanding of the input
parameters, the results could be wrong or misleading.

¦ Utilities will likely require adequate training and continued partnership with EPA
to properly use the TEVA-CWS tool.

¦ Many utility representatives recommended that EPA keep the tool and run it for
the utilities.

¦ Concerns were raised regarding security and liability and accountability. If, for
example, the tool indicates that 12 sensors are needed to effectively cover the
system and the utility can only afford to install 8 sensors, will the utility be liable.

¦ Although the variety of options in the TEVA-CWS tool were helpful, were all
necessary. Could the majority of options be predefined?

¦ EPA should evaluate options for simplifying the tool while still providing sufficient
flexibility.

¦ Could the TEVA-CWS software be used to optimize the location of mobile
sensors?

¦ Could risk maps (areas of the network that are more vulnerable) be created
ahead of time versus using the tool after a potential event is detected?

¦ Development of sampling plans alone may not be sufficient to justify utility
ownership and use of the tool.

¦ Tool would probably best benefit utilities as an in-house tool if it assisted with
response efforts.

¦ Some utilities did not see the benefit in committing time to learn software that has
limited use. Could the tool support other utility needs, such as response planning
and design of other non-security sampling and monitoring programs?

¦ Many utilities preferred a collaborative approach between themselves and EPA
to develop CWS.

¦ Security concerns were expressed with the use of consultants operating the
TEVA-CWS software and preferred to have EPA continue operating the software.

¦ A utility indicated a preference for using the software themselves, because
system knowledge and expertise.

EPA indicated the software was adaptable to change as the technology of sensors
advances. Event detection algorithms were adaptable to new detection devices. EPA
acknowledged that it will require a significant expenditure of funds to create a utility
version of TEVA-CWS. The security of information exchange and the technical aspects
of the tool would need to be further developed prior to a utility wide release.

Has the TEVA-CWS software been compared with other intuitive approaches based on
expert knowledge of distribution systems? EPA indicated that studies have shown that
optimization methods clearly outperform expert-based approaches. TEVA-CWS uses
the only method that theoretically always gets the best solution. It was noted that the
model still uses a number of assumptions, so there will always be uncertainty in the
predictions.

An informal survey of utility use of PipelineNet found that not many utilities are using it.
It should be noted that PipelineNet is primarily a GIS-based EPANET, so the survey is
probably also an indication of water utility's lack of EPANET use, as opposed to
WaterCAD or H20MAP.

Adding response capabilities to TEVA-CWS would probably not increase the in-house
demand. Most small systems do not perform their own modeling, and large utilities are

-------
A WWA Utility Users Group Meeting Notes

using commercial software models instead of PipelineNet. How many utilities perform
hydraulic and water quality modeling? A relatively small number of systems nationwide
perform modeling, and only a limited number of them would consider moving forward
with security related software. Distribution system modeling is still relatively new.

The preference of the water utility users group is to have the expertise and capability
remain within EPA because EPA personnel have the expertise required to operate the
software properly and EPA use would reduce security concerns.

EPA requested feedback on whether to devote resources to develop a TEVA-CWS tool
for wide release to the water community. EPA will plan upcoming efforts on what is
learned at this and future Water Utility User Group meetings. Resources within the
TEVA Research Program are limited, therefore, keeping the software within the agency
will limit the pool of utilities that can use and benefit from the software tool. A consultant
version of the TEVA-CWS software tool is being developed to support WaterSentinel
Program. This version will mirror research version and, therefore, its distribution will be
tightly controlled.

Focus Track Discussion #2: TEVA Event Detection Software

EPA indicated that the motivation for researching EDS included the:

• Unavailability of robust, in-situ, real-time, contaminant specific sensors

• Knowledge that water quality parameters have been shown to change
significantly in the presence of contaminants

• Knowledge that water quality also varies considerably in distribution systems,
both spatially and temporally.

Event/change/anomaly detection algorithms are being developed to distinguish between
normal variations in water quality and changes in water quality triggered by the presence
of contaminants. Thus, sensors used to detect chlorine, pH, total organic carbon (TOC),
and specific conductance can also be used indirectly to detect contaminants via
mathematical data analysis tools that compare variances in the data streams. Real
world water quality data are noisy, have daily and seasonal cycles, contain drift, and get
recalibrated, so it is not obvious to the human eye whether changes are "normal" or not.
Three EDS algorithms were highlighted:

• Time increments, or first differences, predict the water quality for the next time
step and indicates when there is a large difference between the predicted value
and the actual value. It is the simplest of the three algorithms and only requires
the previous measurement to be kept in the time history.

• Linear filter method filters a time window of previously observed data to predict
the signal at the next time step.

• Multivariate classification is a pattern recognition algorithm that uses a five-
dimensional space to characterize data points. The algorithm identifies data
points that are not within a specified distance from a recognized cluster.

EDS create an alert based on the number of standard deviations between the predicted
water quality and the actual water quality. Working with Sandia National Laboratories,

-------
A WWA Utility Users Group Meeting Notes

the TEVA Team has developed CANARY, an EDS tool. The tool has been tested on
data from 2 utilities and data from EPA's Test and Evaluation Facility.

EPA explained how the EDS algorithms differ from set points. The algorithms find
significant changes in water quality, while set values find extremes in water quality. A
sudden change in water quality that is within the set points would only set off an alarm
with the algorithm method. On the other hand, a gradual change to a sustained level
above a high set point, or below a low set point, would only set off an alarm with the set
point method. Utilities may decide to use both methods simultaneously.

The TEVA Research Team is actively trying to deploy EDS and determine whether they
work.

WaterSentinel will deploy the Canary EDS and other commercially available EDS and
compare performance. EPA is evaluating how well the EDS work, whether they are
reliable enough on which to base consequence management decisions, how many false
positives will be generated, and how alarms should be managed. A ROC curve is used
to evaluate the algorithms. The curve compares the false alarm rate to the probability of
detection. In other words, it compares sensitivity and specificity. The ideal tool would
have no false alarms and perfect event detection. The ROC curve allows the tools to be
evaluated by the area under the curve. Increased area means increased reliability of the
algorithm.

More data is required to evaluate the algorithms. Data from field sensors is available to
measure false positives, but data from real contamination events is needed to measure
true positives. EPA is proposing to simulate more than 100,000 events using EPANET-
MSX to evaluate the EDS. EPA presented a diagram illustrating the major components
of the CWS Evaluation Framework. It was noted that increased variability in the chlorine
level makes it harder to detect events using the algorithm. EPA suggested some
questions for discussion after demonstration of the Canary EDS software:

• How do utilities plan to use water quality data from sensor platforms?

• Are there dual uses for an EDS? Could it be used to detect accidental events?

• What information would utilities need to select a particular EDS?

• Is it preferable for an EDS to detect only the beginning of the event or the entire
duration of the event?

• What is an acceptable false positive rate? What does the false positive rate
need to be to ensure that the utility will respond to every alarm?

To highlight the importance of false positives, an example of the airport screening
scenario was cited as an example by one of the utility participants. Scanning 10,000
people would likely identify 200 false events because the accuracy of the metal
detectors is 98 percent. Online monitors have sensitivities of approximately 85 percent,
so the occurrence of false positives would be much greater. For instance, if a utility has
60,000 data points in a week, there could be 2,000 false events. EPA noted that the
detection of events uses the sensor plus the algorithm, so the accuracy of the sensor
alone cannot be used to compute the statistical probability of false positives. In addition,
utilities will use EDS as only one aspect of event detection. Water quality monitoring,
consumer complaints, and public health surveys will all contribute information. The
multiple data streams will reduce uncertainty. The water quality sensors and EDS will

-------
A WWA Utility Users Group Meeting Notes

also evaluate a series of data over time, not a single data point as is the case with
airport screening. Multiple parameters (e.g., chlorine, pH, temperature, and specific
conductance) can also be screened simultaneously. Events detected by changes in
multiple parameters may increase reliability. EDS will not be used independently and
site characterization will be important.

Could EDS be used to look at day-to-day operations? The EDS might be good for
quality control in addition to security. Could these tools be used to detect cross
connections and backflows? The AWWA Emerging Issues Committee is considering
using EDS tools for nitrification control, surface water treatment, bacteriological control,
and corrosion detection. It may be possible to develop a more refined and sophisticated
tool that could use average chlorine levels for trending and detection of abnormal
events. EDS could also be used to monitor turnover in storage and optimize operation.

The Canary EDS algorithms were demonstrated. The algorithms distinguish events from
background water quality variations. The tool measures the confidence in event
detection. More confidence is attributed to events detected for a three hour time step as
opposed to a one hour time step. Three types of algorithms are used to detect events.
Some algorithms detect events that others do not. The tool allows different sensors to
be mapped. The tool needs to train on 1,000 hours of data. The multivariate
classification algorithm has the least false positives, and the time increment algorithm is
the least robust. The time increment system has the most false positives, but returns
back to normal very quickly.

Canary can read in information in quasi-real-time and can update every two minutes.
The TEVA Research Team is working on developing a library function so that the system
can remember more data, including daily and seasonal patterns. The tool does not have
access to hydraulic models, so it does not incorporate flow. The tool should be self-
learning and can use data as far back as desired. In addition to modeling with the three
algorithms, the same algorithm with multiple thresholds and windows can also be
compared. The best way to combine the algorithms to eliminate false alarms will be a
future research activity. There is also a way to give more weight to certain algorithms.

Where would the Canary EDS tool reside? In the SCADA infrastructure? It was
indicated that Canary, as yet, does not directly talk to SCADA, but efforts were underway
to link Canary to the SCADA system of the first WaterSentinel Pilot. The tool would be
most beneficial if it alerted the operator sitting in the control room. One utility indicated
that they used the Hach Event Detection Monitor. Not enough data is available to
determine if events should be investigated. There was a desire for more information
about what is happening in the algorithm. Is it a barrier to the technology if the algorithm
is contained in a "black box?" It is only a problem until it is certified that the tool works.
The group discussed the Homeland Security certification and noted that it is not a form
of technology verification. The tool needs to be incorporated with the SCADA system. If
the tool resided at a sensor monitoring site it would be a significant drawback. The need
for manufacturers to develop "plug and play" sensors that could work with all event
monitors was identified.

The predictive capabilities of the Hach system to detect a contamination event should be
compared with the TEVA algorithms. It was suggested that the Technology Testing and
Evaluation Program (TTEP) perform an evaluation. Participants were not sure if Hach
would agree to have their system tested. Discussion followed as to how to pull out the

-------
A WWA Utility Users Group Meeting Notes

algorithm to test it with simulated events. Utilities using the Hach system offered to give
the TEVA Research Team the system to conduct tests. Hach used a similar method to
test their system initially. Many participants were enthusiastic about having the system
evaluated with the simulated events. It was not certain whether TTEP would be in the
position to conduct the evaluations, but the TEVA Research Team may tackle the
evaluation themselves.

Planned Improvements to Distribution System Modeling

A moderated lunch discussion on EPA's planned improvements to distribution system
modeling was provided. It was noted that there are some water distribution system
problems where models are a necessary part of the solution. However, there are also
many reasons why models and modeling methodologies (algorithms) fall short.

Model calibration and model verification are both important. Calibration is the
adjustment of model parameter values to make predictions match selected observations.
There are time-varying and static model parameters. The time-varying parameters are
more difficult to calibrate. Verification is using the calibrated model to make predictions,
and comparing them with observations, where observations are independent of those
used for calibration and represent a different set of environmental conditions or stresses
(e.g., a different season or a different operating condition). An example of a utility that
had a main break and used a model to predict how the system would respond was
provided. The actual response of the system was totally different than the model
predicted. Calibration leads to verification. If a system is recalibrated, it must be verified
again.

The difference between uncertainty and variability was discussed. Parameter
uncertainty is randomness caused by data limitations and can be reduced through data
collection. Variability is randomness caused by external environmental fluctuations and
cannot be reduced. However, variability can be understood and modeled to reduce
uncertainty.

It was noted that many water security issues cannot be evaluated solely in the lab or the
field including consequence assessment, CWS analysis and design, and rapid response
decision. Models are necessary, but currently limited by lack of data and understanding
of parameter uncertainty and variability. Parameter uncertainty and variability are a new
research focus. Models are also limited by inability to model realistic water quality
reactions; inability to automatically adapt models in real-time, based on system
operation, hydraulics, and water quality SCADA data; and, inefficient computational
procedures.

To build a stronger foundation for research, attention needs to be given to improving
network models and quantifying their accuracy. A significant data set is needed to
quantify how well a hydraulic model compares to real measurements. Some systems
have used tracers to validate models. They determine validity by comparing the three
parameters: tank levels, chlorine residuals, and pressures. A participant said that his
experience is that when the model does not match the system, the system often needs
to be fixed, not the model. For example, the model can be used to find a closed valve
that should be open.

-------
A WWA Utility Users Group Meeting Notes

The features, status, and expected release dates of future TEVA Research Program
software tools was provided. Specific software tools identified included EPANET-MSX
(Multiple Species Extensions), EPANET-DPX (Distributed Processor Extensions),
EPANET-MCX (Monte Carlo Extensions), and EPANET-RTX (Real-Time Extensions).

The development principles for the EPANET extensions were identified:

• Collaboration between USEPA/NHSRC and USEPA/WSWRD

• Team-based approach

• Commitment to open source

• Cooperation with software developers (e.g., licensing with collaborative
requirements).

The utility user group agreed that it is important for them to have a network hydraulic and
water quality model that they have confidence in, and that real-time modeling is
valuable. Participants discussed the sensitivity of the information available in the
models.

Update: Field Study Support and Evaluation of CWS

EPA presented an update on the field studies for the WaterSentinel Pilot. The
objectives of the presentation were to discuss the role of field studies and data collection
in implementing and evaluating a CWS in conjunction with the first Sentinel pilot. The
pilot has installed six contamination warning system (CWS) stations based on a 30-
station sensor design determined by SPOT and simulated data associated with varying
water quality. The 30 stations were classified by simulated chlorine concentration
average and standard deviation, which is used to measure the variability of the chlorine
levels. The six stations chosen include two with high average chlorine and low standard
deviation, two with low average chlorine and high standard deviation, one with medium
average chlorine and high standard deviation, and one with medium average chlorine
and low standard deviation. Three systems are installed, and three will be installed in
the near future. The locations with a high average chlorine and high variability are
located close to the tank, and those with low average chlorine and high variability are
influenced by the tank. Ultimately 20-30 stations will be deployed.

It is difficult to evaluate event detection algorithms with actual contaminants and realistic
water quality variability. The overall EDS evaluation will be performed under simulated
conditions, which require adequate data and statistical representation. Long-term
chlorine, conductivity, pH, total organic carbons, and turbidity data are being collected
from treatment plant effluents. After collecting data, the TEVA Research Team will
develop statistical representations and conduct separate studies to develop kinetic
models that will simulate events on top of the simulated water quality variability. Models
will simulate trade-offs between true-positives and false-positives. The effort will provide
for comprehensive evaluation of EDS and improve sensor placement. The effort should
show how well the algorithms can work based on where sensors are placed in the
system.

The objectives of the Sentinel Pilot field studies are to perform tracer tests and water
quality monitoring, provide model confidence and improve network models, and assist in
evaluation of various Sentinel and TEVA computational tools. The system will be
studied as a series of individual sub-regions. Field sensors will be located, based on

-------
A WWA Utility Users Group Meeting Notes

CWS design from the TEVA Research Program, to provide adequate coverage of the
distribution system.

Tracers will be pumped into a selected pipe in multiple pulses for at least 24 hours. The
distribution system will then be monitored until the tracer is flushed out. The research
team will try to determine why actual observations differ from the model predictions.

A smaller scale study will focus on skeletonization, which impacts consequence
assessments and may impact sensor design. Increased skeletonization increases the
impact of an event by aggregating population at nodes and reducing transportation time.
Smaller-scale tracer tests will be performed to evaluate the impacts of model
skeletonization on hydraulic and transport dynamics.

EPA showed a diagram of an example small-scale tracer study area. Two flow monitors
allow measurement of boundary conditions, Automatic Meter Readers at individual
houses measure the demand at each house hourly, and conductivity sensors measure
the tracer signal. The small-scale study is planned for the Spring, 2007. EPA is also
talking to systems doing their own field studies.

A participant asked if these studies would be used to validate models or recalibrate
them. EPA indicated that the studies would help validate the models. Whether the
models need to be recalibrated may depend upon the results.

Focus Track Discussion #3: Responding to Contamination
Events

EPA indicated that response is the area that has received the least attention thus far
from the TEVA Research Team, although EPA recognizes it is a high priority of the utility
users group. The presentation provided a status update on EPA's efforts to develop
response tools. The follow on discussion focused on tools that could be developed to
help utilities to decide what actions to take after an event is detected. EPA asked what
computational tools would be helpful in a response mode. It was noted that the first step
must be to determine if it is a false alarm or a real event and the percentage of the
population that could be affected.

EPA suggested that the utility needs to answer the following questions when responding
to a contamination event:

•	Where did the contamination originate?

•	Where should confirmatory samples be taken?

•	Where is the contamination plume now? Where will it be?

•	What part of the population is at risk?

•	Can the contamination be isolated? Does it need to be flushed from the system?

Example outputs of an event reconstruction software program were demonstrated. The
contaminant source tracking program can determine where the contamination originated
based on hydraulics and positive and negative sensor readings. The negative sensor
readings help to eliminate potential sources. The model would need to be supplemented
with grab sample data and confirmed with police and public reports.

15

-------
A WWA Utility Users Group Meeting Notes

It was noted that there may be an optimal way to place the sensors so that utilities can
deduce the source more easily. However, the sensor placement formulations are
difficult to solve. Comments and questions from utility participants included:

¦	Models showing forward progress of the contaminant in the distribution system
are needed.

¦	Could the model be used in advance to have preplanned sampling protocols for
detection at each sensor site?

¦	How long it takes to run the model? Experience has shown that by the time the
site was sampled, the contaminant was past sampling point.

The TEVA Research Team members indicated that the running of the contaminant
source tracking model is relatively simple and fast.

It is critical to determine if there was an event, so the system needs to be integrated into
the SCADA system. Ideally, analysis would be done all the time, and backtrack
simulations would be performed continuously. The information would be available to
the utility in the case of an alarm, but would not always need to be displayed.

The potential functions of the real-time model include event detection, source tracking,
forecasting, hydraulic control (isolation, flushing, and decontamination), determination of
sampling locations, and identification of at-risk populations on maps. The capability for
the water program to have measurements and predictions in real-time increases the
potential benefits.

EPA indicated that a computational tool (prototype was demonstrated) is now available
to help identify the contaminant source. Utilities questioned if they would have enough
time to do anything about it. EPA discussed a planned case study with a TEVA utility
which will examine the feasibility and logistical requirements for closing valves and
flushing the distribution system to address a water contamination event. Even if the
model does not allow the utility to detect and respond fast enough, it is in the utility's
best interest to be as prepared as possible. There is a time lag between when water
reaches a group of service connections, and when the water is taken into homes and
circulated in the pipes and appliances and used by its occupants. In other words, there
may be more time available for effective response than predicted by models.

There was some concern as to the potential use of point-of-use devices and their effect
on the assessment of public health consequences. The general sense of the group was
that it would be inappropriate to assume that point-of-use devices are in place to
mitigate public health consequences.

It was noted that there is only a small period of time when the utility will be operating
without public knowledge of an event. Current programs are aiming for an eight-hour
response time. Bacteriological test guidance require another test after a positive result,
so by the time the public notice is issued, the issue is resolved. The group discussed
how to reduce response time and what is practical. Most participants agreed that the
response time will not be less than eight hours. The group also discussed the
consequences of issuing a false alert. False alerts will cause consumers to take true
alerts less seriously. Pre-educating the public regarding the testing and the possibility
of false positives is critical.

16

-------
A WWA Utility Users Group Meeting Notes

It was suggested that federal guidance be developed on the process for issuing do-not-
use and do-not-drink notices. It was suggested that the Response Protocol Toolbox
include such guidance. The group noted that there were still questions about how much
water will be consumed once the situation is over.

Progress in these areas would be helpful to the utilities. Tools need to be easy to use in
order for people to feel comfortable using them in an emergency. A tool needs to be
used almost daily or be very similar to systems that are used daily. The difficulties with
processing so much information were recognized.

Group Discussion on Utility Perspectives on Implementing a
Contamination Warning System

It was agreed that the utility user group should play a role in the prioritization of research
needs for contamination warning systems. Could more information be provided about
the potential research goals for the TEVA Research Program, including schedule
information? EPA indicated that a near-term, mid-term, and long-term list could be
provided. EPA indicated that they would develop a matrix of research goals along with a
projected schedule to accomplish such goals.

AWWA asked if the utility user group would like to have a broader discussion with the
manufacturers in a neutral setting in Denver. One of the topic discussions will be
determining what the manufactures can address versus what EPA can address. The
need for "plug and play" systems was emphasized.

It was recommended that a set of objectives be developed for a meeting between
manufacturers and utilities. AWWA indicated that the meeting with the manufacturers
would be broader than just the TEVA research.

EPA indicated that it may be useful to communicate with commercial software
companies to better integrate EPA's research tools into the commercial programs used
by water utilities. EPA noted that some software companies are requesting that EPA
share their tools. It was suggested that a draft agenda for the proposed meeting with the
manufactures be developed by the User Group Utilities. The manufacturers should
know that only a small subset of utilities will be present and that heavy sales pitches will
not be well-received. The utility user group will meet, possibly for a half-day, to refresh
everyone on the pertinent topics prior to the meeting with the manufacturers. AWWA
committed to laying out the ground rules for the meeting and be a neutral facilitator.

Utility user group requested that a vision for the interaction with the manufacturers be
developed. The manufacturers and utility user group should discuss what technologies
are available now. A participant asked if there was a standardization committee. A
current AwwaRF project is developing guidance for interoperability.

Efforts at a TEVA Utility were highlighted as a case study example for designing and
implementing a CWS. A year ago, the TEVA utility, using internal funds, embarked on
efforts to design an online monitoring system. They used PipelineNET (EPANET in a
GIS environment) and TEVA to look at potential sites for locating sensor monitoring
stations. A utility led team was formed to identify the criteria for designing the CWS.
TEVA analyses were conducted by EPA. The utility used the TEVA results to select
water security monitoring sites and PipelineNET was used for water quality monitoring

-------
A WWA Utility Users Group Meeting Notes

sites. One site was identified by both systems. It was determined that changing one of
the optimal sites to one that was more convenient did not reduce the benefit significantly.
The utility will pilot the plan in the first one or two years and then install permanent
equipment. It was highlighted as a good cooperative effort between EPA, the utility, and
its consultants. EPA indicated the project was also efficient for them because the utility
and their consultant provided a working and calibrated EPANET model.

It was suggested that EPA develop a template of the needs/specifications that would
better facilitate the TEVA Research Team to more easily support water utilities to
perform consequence assessments and sensor placement designs. EPA agreed to
provide the utilities with input about how to make the process more efficient.

AWWA noted that the meeting with the manufacturers may take place in five to six
months, after the holiday season.

AWWA and EPA thanked everyone for participating in the discussions. EPA thanked
everyone for their participation in the TEVA Research Program.

Next Steps:

EPA will continue to work with the individual TEVA utilities on consequence assessment
and sensor placement.

-------