&EPA
United States
Environmental Protection
Agency
Water Security Initiative: System Evaluation of the
Cincinnati Contamination Warning System Pilot
Monitoring and Surveillance
Water Quality Monitoring
Enhanced Security Monitoring
Customer Complaint Surveillance
Public Health Surveillance
Possible Contamination
Consequence Management
Sampling and Analysis
Response
Office of Water (MC-140)
EPA-817-R-14-001A
April 2014
-------
-------
Disclaimer
The Water Security Division of the Office of Ground Water and Drinking Water has reviewed and
approved this document for publication. This document does not impose legally binding requirements on
any party. The findings in this report are intended solely to recommend or suggest and do not imply any
requirements. Neither the U.S. Government nor any of its employees, contractors or their employees
makes any warranty, expressed or implied, or assumes any legal liability or responsibility for any third
party's use of, or the results of such use of, any information, apparatus, product or process discussed in
this report, or represents that its use by such party would not infringe on privately owned rights. Mention
of trade names or commercial products does not constitute endorsement or recommendation for use.
Questions concerning this document should be addressed to:
Steve Allgeier
U.S. EPA Water Security Division
26 West Martin Luther King Drive
Mail Code 140
Cincinnati, OH 45268
(513)569-7131
-------
Acknowledgments
The Water Security Division of the Office of Ground Water and Drinking Water would like to recognize
the following individuals and organizations for their assistance, contributions and review during the
development of this document.
Gary Burlingame, Philadelphia Water Department
Yakir Hasit, CH2M Hill
Robert Janke, USEPA, ORD, NHSRC
Daniel R. Quintanar, City of Tucson Water Department
Connie Schreppel, Mohawk Valley Water Authority
Stanley States, Pittsburgh Water and Sewer Authority
Jeff Swertfeger, Greater Cincinnati Water Works
-------
Executive Summary
The goal of the US Environmental Protection Agency's (EPA) Water Security Initiative (WSI) is to
design and demonstrate an effective monitoring system for timely detection and response to drinking
water contamination threats and incidents. A contamination warning system (CWS) integrates
information from multiple monitoring and surveillance components to alert a water utility to possible
contamination and guides response actions through consequence management.
System design objectives for an effective CWS are: operational reliability, spatial coverage, contaminant
coverage, alert occurrence, timeliness of detection and response, and sustainability. Metrics were defined
for each of these design objectives to provide a basis for the technical evaluation of the Cincinnati CWS.
Evaluation techniques used to quantitatively or qualitatively evaluate each of the metrics include analysis
of empirical data from routine operations, drills and exercises, modeling and simulations, forums, and a
benefit-cost analysis. This report describes the analysis of data collected from the Cincinnati CWS during
the evaluation period from January 2008 through June 2010.
The major outputs from the evaluation of the Cincinnati pilot include:
1. Cincinnati Pilot System Status, which describes the post-implementation status of the Cincinnati
CWS following the installation of all monitoring and surveillance components.
2. Component Evaluations, which includes analysis of performance metrics for each component of
the Cincinnati CWS.
3. System Evaluation, which integrates the results of the component evaluations, modeling and
simulations, and a benefit-cost analysis.
The reports that present the results from the evaluation of the system and each of its six components are
available in an Adobe portfolio, Water Security Initiative: Comprehensive Evaluation of the Cincinnati
Contamination Warning System Pilot (USEPA 2014a).
Contamination Warning System Design
A multi-component design was adopted to meet the CWS design objectives. Such a system integrates
information from multiple monitoring and surveillance tools common to the drinking water industry and
public health sector that collectively provide timely and comprehensive detection capabilities. The
monitoring and surveillance components of the Cincinnati CWS are:
Water Quality Monitoring (WQM) comprises 15 stations located throughout the distribution
system that measure chlorine residual, pH, total organic carbon, conductivity, turbidity and
temperature. Data from each monitoring station is transmitted in real time over a communication
network to an operations and control center where the data is continuously analyzed for
anomalies by an automated event detection system.
Enhanced Security Monitoring (ESM) includes the equipment and procedures that detect and
respond to security breaches at critical distribution system facilities that provide access to
finished water. Security equipment such as cameras, motion activated lighting, door contact
alarms, ladder and window alarms, area motion sensors, and access hatch contact switches
generate alerts when key facilities are breached.
Customer Complaint Surveillance (CCS) enhances the collection, and automates the analysis,
of calls from customers reporting water quality concerns, which may be indicative of a water
-------
quality issue in the distribution system. Work orders and interactive voice response menu
selections are monitored by an automated event detection system.
Public Health Surveillance (PHS) involves the analysis of health-related data to identify disease
events that may stem from drinking water contamination. Public health data analyzed in the
Cincinnati CWS include 911 calls, emergency medical service data, Drug and Poison Information
Center calls and hospital admission reports.
If any of these four monitoring and surveillance components detects an anomaly, an alert is generated and
investigated according to documented procedures. If contamination is considered Possible at the
conclusion of that investigation, Consequence Management procedures are initiated in an attempt to
determine whether contamination is Credible. Additionally, procedures under the Sampling and
Analysis (S&A) component guide the field investigation, sample collection and laboratory analysis for
chemicals, radionuclides, pathogens and biotoxins through a laboratory network. Positive laboratory
results are generally sufficient to Confirm a contamination incident.
Methodology
Several methods were used to evaluate the performance of the Cincinnati CWS. Data was tracked over
time to illustrate the change in performance as the CWS evolved during the evaluation period. Statistical
methods were also used to summarize large volumes of data collected over the evaluation period. Data
was also evaluated and summarized for each reporting period over the evaluation period. In this
evaluation, the term reporting period is used to refer to one month of data that begins on the 16th of the
indicated month and ends on the 15th of the following month. Thus, the January 2008 reporting period
refers to the data collected between January 16, 2008, and February 15, 2008. Additionally, 19 drills and
two full-scale exercises designed around mock contamination incidents were used to practice and evaluate
the full range of procedures, from initial detection through response.
Because there were no contamination incidents during the evaluation period, there is no empirical data to
fully evaluate the detection capabilities of the Cincinnati CWS. To fill this gap, a computer model of the
Cincinnati CWS was developed and challenged with a large ensemble of simulated contamination
incidents in a simulation study. An ensemble of 2,015 contamination scenarios representing a broad
range of contaminants and injection locations throughout the distribution system was used to evaluate the
effectiveness of the CWS in minimizing public health and utility infrastructure consequences. The
simulations were also used in a benefit-cost analysis, which compares the monetized value of costs and
benefits and calculates the net present value of the CWS. Costs include implementation costs and routine
operation and maintenance labor and expenses over a 20-year lifecycle for the CWS. Benefits included
reduction in consequences (illness, fatalities and infrastructure contamination) and dual-use benefits to
routine system operation.
Design Objective: Operational Reliability
For a CWS to consistently detect extremely rare contamination incidents, it must achieve a high degree of
operational reliability, which is defined as the availability and production of data of acceptable quality
and quantity for reliable event detection. Operational reliability of the Cincinnati CWS was evaluated
through data completeness and availability.
Data completeness was 95% for the CWS over the entire evaluation period. Issues with WQM equipment
during the early stages of deployment contributed significantly to lost data. ESM, CCS and PHS
IV
-------
regularly had months of 100% data completeness, with only intermittent periods of data loss. After the
components were optimized by June 2009, data completeness for the CWS regularly exceeded 95%.
Average availability for the individual CWS components during the evaluation period ranged between
73% and >99%, as shown in Table ES-1. The single greatest contributor to downtime was issues related
to the WQM event detection system (CANARY), which was particularly significant during the early
portion of the evaluation period. As problems with CANARY were resolved, the availability of the
WQM component, and the entire CWS, increased. Had CANARY been fully operational during the
evaluation period, the WQM component would have been available for 89% of the time, rather than the
73% availability observed during the evaluation period.
Table ES-1. CWS Component Availability
Component
WQM
ESM
CCS
PHS
Availability
73% (89%)1
97%
>99%
90%
The value in parenthesis (89%) represents WQM
component availability when downtime caused by
the CANARY event detection system is excluded.
The CCS component had the highest availability at >99%, followed closely by ESM at 97%. The PHS
tools deployed specifically for this project, 911 and emergency medical service surveillance, were
available 90% of the time; however, the PHS tools that were in place prior to the pilot were mature
systems that were available >99% of the time.
Availability of the entire CWS was evaluated in terms of percentage of time when one, two, three or four
components were concurrently available. Overall, downtime of multiple components was rare. Three of
the four components were available >99% of the time, and all four surveillance components were
available 78% of the time. The longest periods of multi-component downtime were 26 hours for two
components and 8 hours for three components, which were well below the average residence time of
contaminated water in the distribution system during simulated contamination scenarios (5.3 days). This
indicates that even with multiple components unavailable for a period, it is still likely that a significant
contamination incident will be detected by the CWS. For more information, see Section 4.0.
Design Objective: Spatial Coverage
The Cincinnati CWS monitoring and surveillance components were selected and designed to provide
redundant coverage throughout the distribution system in order to maximize the potential of the system to
detect contamination regardless of injection location. Through a multi-component design, the Cincinnati
CWS achieved broad spatial coverage of the study area, which includes the most populous region of the
Greater Cincinnati Water Works (GCWW) service area, with approximately 760,000 customers and
covering 294 square miles. Area coverage ranged from 72% for WQM to 100% for CCS, PHS and S&A.
Population coverage was greater than area coverage, ranging from 84% for WQM to 100% for PHS and
S&A.
Results from the simulation study were evaluated to determine the number of contamination scenarios
originating from each of the 94 pito zones that were detected by the CWS. (A pito zone is a small region
of the distribution system, ranging from 0.3 to 15 square miles, in which water quality and pressure are
fairly constant.) This analysis showed that 100% of the scenarios originating from 51 pito zones and
-------
94.1% of scenarios originating from another 38 pito zones were detected by the CWS. The 44 scenarios
that were not detected were spread across 43 pito zones, indicating that there is no spatial trend to
undetected scenarios. The primary reason that these 44 scenarios were not detected is that they produced
low consequences, which generate weak signals and thus are difficult to detect regardless of where the
injection occurs. In summary, all regions of the distribution system were effectively covered by the
CWS. For more information, see Section 5.0.
Design Objective: Contaminant Coverage
The design of the Cincinnati CWS ensured the system had robust detection capabilities for a variety of
contaminants, including nuisance chemicals, toxic chemicals and biological agents. Seventeen
contaminants were selected to represent a wide range of contamination threats, and during simulation
studies all were found to be detectable by at least one monitoring and surveillance component at a
concentration equal to or less than the critical concentrations necessary to cause significant public health
or infrastructure consequences.
Table ES-2 presents the ratio of critical concentration to detection threshold for each contaminant across
the components. A ratio of 1.0 or greater indicates that the component can detect the contaminant at or
below the critical concentration. Conversely, ratios less than 1.0 indicate that the component would not
detect the contaminant until the concentration exceeds the critical concentration that would result in
adverse public health or infrastructure consequences.
Table ES-2. Ratio of Critical Concentration to Detection Threshold
Contaminant
Nuisance Chemical 1
Nuisance Chemical 2
Toxic Chemical 1
Toxic Chemical 2
Toxic Chemical 3
Toxic Chemical 4
Toxic Chemical 5
Toxic Chemical 6
Toxic Chemical 7
Toxic Chemical 8
Biological Agent 1
Biological Agent 2
Biological Agent 3
Biological Agent 4
Biological Agent 5
Biological Agent 6
Biological Agent 7
WQM
4.76
33.3
225
463
185
104
57.6
352
1.97
0.0333
265
1,310
2.40
3.57
7.87
9.70
0.582
CCS
20.0
-
5.86
50.5
22.8
4.03
-
-
-
-
88.2
-
-
-
-
-
-
PHS
-
-
458
3,640
1,640
290
668
850
950
300
4,500
3,940
2.40 x 104
4.54
10.0
1.74
1.64
S&A
2.00 x 104
2.00 x 104
1,470
3.39 x 104
3.69 x 10s
5.80 x 104
6,680
4.08 x 104
57.0
6.60 x 107
2.25 x 104
4.93 x 105
24.0
90.7
20.0
5.79 x 104
3.30 x 105
Results from the simulation study demonstrate that the Cincinnati CWS was able to detect 98% of
simulated contamination incidents from an ensemble of 2,015 scenarios involving 17 contaminants and
injection locations throughout the entire distribution system. These results demonstrate the value of a
multi-component CWS, in which the detection capabilities of the monitoring and surveillance
components are complementary and provide broad contaminant coverage. The majority of the 44
VI
-------
scenarios that were undetected involved a contaminant that does not cause acute health effects and is
detectable by only a single component. Small, isolated contamination incidents that produced limited
consequences were more difficult to detect than incidents producing widespread consequences. For more
information, see Section 6.0.
Design Objective: Alert Occurrence
One of the goals of the Cincinnati CWS design is to minimize the number of invalid alerts without
compromising the ability of the system to detect real water quality anomalies or public health incidents.
Valid alerts are valuable in that they provide early warning of unusual water quality conditions in the
distribution system. However, too many invalid alerts can divert personnel from other duties and may
ultimately lead to the perception that the CWS is unreliable and therefore unsustainable. The alert rates
for all four monitoring and surveillance components decreased during the transition from the optimization
phase to the real-time monitoring phase, as is evident from the average number of alerts per reporting
period for each of these phases shown in Table ES-3.
Table ES-3. Invalid Alerts per Reporting Period During Optimization
and Real-time Monitoring
Component
WQM
ESM
CCS
PHS
System
Average Number of Invalid Alerts per Reporting Period
Optimization
33
82
17
25
152
Real-time Monitoring
17
23
14
15
69
Invalid alerts occurred frequently, with more than 150 alerts during most reporting periods in the first
year of operation. However, once the system was optimized by improving the quality of the underlying
data (i.e., through improved maintenance of equipment) and updating event detection system
configurations to reflect normal variability in the data, the number of invalid alerts was substantially
reduced to 69 per reporting period. While most alerts were determined to be invalid, the CWS did detect
84 valid alerts, with more than half caused by unusual system operating conditions or public health events
(unrelated to drinking water).
The Cincinnati CWS was designed to include a variety of surveillance tools to increase contaminant
coverage as well as the reliability of the system for utility managers that need to decide whether or not
contamination may be Possible. Through this multi-component design, weaknesses in the detection
capabilities of one component are offset by the strengths of another. Furthermore, co-occurring alerts
from multiple components can increase the utility manager's confidence that the alerts are valid and
indicative of a potential water quality issue. The results of the simulation study demonstrate that alert
clusters are common for simulated contamination incidents. Specifically, alert clusters occurred in 86%
of simulation scenarios detected by the CWS, with three or more components alerting in 50% of the
simulated contamination scenarios. In contrast, alert clusters were rare in the empirical data (which did
not include any contamination incidents). In fact, a cluster of three component alerts occurred only once
in the empirical data, and consisted of invalid alerts. The prevalence of valid alert clusters in the
simulated data and the paucity of valid alert clusters in the empirical data would suggest that valid alert
clusters involving alerts from multiple components are likely the result of a real water quality issue in the
distribution system. For more information, see Section 7.0.
VII
-------
Design Objective: Timeliness of Detection and Response
For a CWS to have the maximum potential to reduce consequences of a contamination incident, it must
detect the incident early enough to allow sufficient time to implement response actions under the
consequence management plan.
Given that there were no real contamination incidents during the evaluation period, simulated
contamination scenarios were used to evaluate this design objective. Results from the simulation study
show median detection times less than 7 hours for WQM, CCS and PHS, while ESM typically detected
the incident before the start of contaminant injection. During the investigation, the median time for
Possible determination was 5.5 hours, just under 6.5 hours for Credible determination, and just under 9.5
hours for Confirmed determination. CCS alerts were almost always generated shortly after the first
exposure to contaminated water. While PHS alerts are also driven by exposures, the results showed more
variability in the time of PHS alerts due to the delay between exposure and symptom onset. The timing
of WQM alerts was strongly dependent on the hydraulic travel time from the injection location to the
WQM station. When multiple components detect a simulated contamination incident, threat level
escalation and implementation of response actions occurred much more quickly compared to scenarios in
which just one component detects contamination.
For simulated contamination scenarios that produced a significant number of fatalities, the response
stemming from detection by the CWS facilitated a large reduction in the number of fatalities when
compared to the same scenario without CWS detection and response capabilities in place. Figures ES-1
and ES-2 show representative scenarios for a biological agent and toxic chemical, respectively, and depict
key timeline metrics and primary consequences.
VIM
-------
1,800
1,600
1,400
1,200
| 1,000
ro 800
Day 1 08:42
Possible Time
Without CWS
Day 1 20:14
Credible Time
00:00
Day 1
Contaminant
Injection
Day 1 08:01
WQM Alert
Day 1 21:27
lonfirmed Time
Day 2 03:46
PH Response
Day 2 18:54
Lab Results
With CWS
\
00:00
Day 4
\
00:00
Day8
Day 1 20:14
PHS-AC Alert
Day 1 20:14
Public Notification
Day 1 11:47
WQ Parameters
Results
Figure ES-1. Timeline and Consequences fora Contamination Scenario Involving Biological
Agent 4
The CWS reduced the number of fatalities by 99% in the scenario involving Biological Agent 4. This
reduction in consequences was largely attributable to the public notification being issued early in the
response process, which dramatically reduced the number of individuals exposed to the contaminant.
Additionally, prophylactic treatment provided as part of the public health response prevented a large
number of potential fatalities.
IX
-------
2,000
1,800
1,600
1,400
« 1,200
0)
I 1,000
Without CWS
06:42
Confirmed Time
05:19
Credible Time
With CWS
800
03:51
600 1 Possible Time
400 -
200 -
0
00:00
Day 1
Contaminant
Injection
01:27
PH Response
03:06
PHS-AC Alert
06:31
PHS-911 Alert
12:09
Lab Results
00:00
Day 2
06:02
WQ Parameters
Results
05:51
Public Notification
04:11
First Op Change
04:47
WQM Alert
Figure ES-2. Timeline and Consequences for a Contamination Scenario Involving Toxic
Chemical 6
The CWS resulted in a 55% reduction in fatalities in the scenario involving Toxic Chemical 6. This
reduction in fatalities is primarily due to the public notification, which sharply reduced the number of
exposures after individuals complied with the public notification. A typical scenario involving a toxic
chemical unfolds quickly, with PHS alerts occurring early in the scenario due to the rapid onset and
progression of symptoms, which differs from typical scenarios for biological agents. For more
information, see Section 8.0.
Design Objective: Sustainability
A key design objective for the CWS is to develop a sustainable system that provides an acceptable
benefit-cost trade-off. The full cost of the CWS is comprised of three broad categories: initial
deployment costs, lifecycle operations & maintenance (O&M) expenses and equipment renewal and
replacement costs. There is also a small cost offset due to the salvage value of equipment at the end of
the lifecycle. The breakdown of these costs for the Cincinnati CWS over a 20-year lifecycle is shown in
Table ES-4.
Table ES-4. Total Lifecycle Cost of the Cincinnati CWS
Cost Element
Deployment Costs
Lifecycle O&M Costs
Renewal and Replacement Costs
Total Cost
$11,936,000
$4,598,000
$2,569,000
-------
Cost Element
Salvage Value
Lifecycle Cost
Total Cost
($127,000)
$18,976,000
A benefit-cost analysis was performed to evaluate whether the monetized benefits of a CWS were greater
than the total lifecycle cost of the Cincinnati Pilot. Thirty scenarios, three scenarios each for ten
contaminants, were evaluated during the benefit-cost analysis. Table ES-5 shows the benefits, in
millions of dollars, for the scenario with median consequences for each contaminant. The monetized
benefits exceeded the total lifecycle cost of the CWS for 23 (77%) of the scenarios and was more than
100 times the cost of the CWS in 19 (63%) of the scenarios. The primary driver of monetized benefits for
most scenarios was the reduction in public health consequences of water contamination.
Table ES-5. Benefits Attributable to the Cincinnati CWS due to
the Reduction in Consequences from a Contamination Incident
Contaminant ID
Nuisance Chemical 1
Toxic Chemical 1
Toxic Chemical 5
Toxic Chemical 6
Toxic Chemical 7
Toxic Chemical 8
Biological Agents
Biological Agent 4
Biological Agents
Biological Agent 6
Total Value
$6 million
$462 million
$72 million
$2,605 million
$252 million
$252 million
$145,027 million
$9,789 million
$30,097 million
$14 million
Despite demonstrating significant monetized benefits in this analysis, the probability of water
contamination is very low. Thus, the business case for deploying a CWS depends largely on dual-use
benefits realized through the Cincinnati CWS. For example, GCWW was able to utilize WQM sensors to
optimize chlorine residuals throughout the distribution system, reducing the overall chlorine dose and
associated costs. Several non-monetizable benefits were realized across multiple CWS components
including the ability to detect a wide range of distribution system water quality issues. Additionally, the
Cincinnati CWS demonstrated benefits to business practices, such as improved communication and
coordination within the utility and its external partners. Overall, the investment in the CWS improved the
response posture of GCWW and the local partners for "all hazards."
Management and personnel from GCWW and local partners demonstrated a strong willingness to
maintain the CWS beyond the pilot. This was demonstrated in the high rate of alert investigations
(greater than 90%) after the CWS was optimized. Furthermore, active participation in drills and exercises
indicated a willingness to adopt the CWS components and procedures. Finally, GCWW is considering
upgrading the WQM component and continues to engage local partners through the Public Health Users
Group. For more information, see Section 9.0.
Summary and Conclusions
Evaluation of the Cincinnati pilot produced a comprehensive assessment of the multi-component CWS
design deployed under WSI. Through layers of redundancy built into the CWS and each of its
components, the system achieved a high degree of operational reliability during the evaluation period.
XI
-------
The multi-component Cincinnati CWS achieved comprehensive contaminant and spatial coverage
through the implementation of a variety of data streams and monitoring points throughout GCWW's
distribution system. Analysis of simulation study results showed a 98% detection rate for 2,015 simulated
contamination scenarios, which emphasizes the value of a multi-component CWS, in which the detection
capabilities of the monitoring and surveillance components are complementary and provide broad
contaminant coverage. For the contaminant coverage and timeliness of detection capabilities, weaknesses
in the capabilities of one component are offset by the strengths of another. Moreover, simulation study
results emphasized that timely detection and threat level determination lead to quicker implementation of
response actions and a significant reduction in consequences.
The overall success of a CWS depends not only on reliable data, but also requires the commitment of
utility personnel and external partners who are aware of the possible causes of changes in observed water
quality data, customer complaints, or trends in public health data. In Cincinnati, this was accomplished
and demonstrated by a strong commitment of utility personnel and local partners to maintain the CWS.
The overarching goal of the CWS - to improve situational awareness such that potential water quality
issues in the distribution system can be quickly detected and proactively addressed - was achieved during
the Cincinnati pilot through deployment of a multi-component monitoring and surveillance system
combined with "all-hazards" response planning.
XII
-------
Table of Contents
SECTION 1.0: INTRODUCTION 1
1.1 CONTAMINATION WARNING SYSTEM DESIGN OBJECTIVES 1
1.2 EVALUATION OBJECTIVES 2
1.3 ORGANIZATION OF THIS REPORT 2
SECTION 2.0: OVERVIEW OF THE CINCINNATI CWS 3
2.1 ENHANCED SECURITY MONITORING 3
2.2 WATER QUALITY MONITORING 4
2.3 CUSTOMER COMPLAINT SURVEILLANCE 4
2.4 PUBLIC HEALTH SURVEILLANCE 5
2.5 CONSEQUENCE MANAGEMENT 6
2.6 SAMPLING AND ANALYSIS 7
2.7 CWS EVALUATION TIMELINE 8
SECTION 3.0: METHODOLOGY 10
3.1 ANALYSIS OF EMPIRICAL DATA FROM ROUTINE OPERATIONS 10
3.2 DRILLS AND EXERCISES 10
3.3 SIMULATION STUDY 12
3.4 BENEFIT-COST ANALYSIS 15
3.5 FORUMS 16
3.6 LITERATURE AND RESEARCH 17
SECTION 4.0: OPERATIONAL RELIABILITY 18
4.1 DATA COMPLETENESS 18
4.2 AVAILABILITY 20
4.3 SUMMARY 24
SECTION 5.0: SPATIAL COVERAGE 26
5.1 AREA AND POPULATION COVERAGE 26
5.2 SUMMARY 28
SECTION 6.0: CONTAMINANT COVERAGE 29
6.1 CONTAMINANT DETECTION THRESHOLD 29
6.2 CONTAMINATION SCENARIO COVERAGE 31
6.3 SUMMARY 32
SECTION 7.0: ALERT OCCURRENCE 34
7.1 INVALID ALERT OCCURRENCE 34
7.2 VALID ALERT OCCURRENCE 38
7.3 ALERT CO-OCCURRENCE 44
7.4 SUMMARY 51
SECTION 8.0: TIMELINESS OF DETECTION AND RESPONSE 52
8.1 DETECTION TIME 52
8.2 RESPONSE TIME 61
8.3 CONSEQUENCE REDUCTION 71
8.4 SUMMARY 81
SECTION 9.0: SUSTAIN ABILITY 83
9.1 NET PRESENT VALUE 83
9.2 DUAL-USE BENEFITS 88
XIII
-------
9.3 WILLINGNESS TO MAINTAIN THE CWS 90
9.4 SUMMARY 91
SECTION 10.0: SUMMARY AND CONCLUSIONS 93
10.1 HIGHLIGHTS OF ANALYSIS 93
10.2 LIMITATIONS OF THE ANALYSIS 94
10.3 POTENTIAL APPLICATIONS OF THE CINCINNATI CWS 95
SECTION 11.0: REFERENCES 96
SECTION 12.0: ABBREVIATIONS 98
SECTION 13.0: GLOSSARY 99
APPENDIX A: CINCINNATI CONTAMINATION WARNING SYSTEM MODEL 106
A.I OVERVIEW OF CONTAMINATION WARNING SYSTEM MODEL ARCHITECTURE 106
A.2 EPANET TOOLKIT 107
A.3 HEALTH IMPACTS AND HUMAN BEHAVIOR 109
A.4 ENHANCED SECURITY MONITORING 115
A.5 WATER QUALITY MONITORING 118
A.6 CUSTOMER COMPLAINT SURVEILLANCE 122
A.7 PUBLIC HEALTH SURVEILLANCE 125
A.8 CONSEQUENCE MANAGEMENT 130
APPENDIX B: BENEFIT-COST ANALYSIS METHODOLOGY 136
B.I INTRODUCTION 136
B.2 IDENTIFICATION OF MONETIZABLE COSTS AND BENEFITS 136
B.2.1 Identification ofMonetizable Costs 136
B.2.2 Identification ofMonetizable Benefits 137
B.3 FINANCIAL ANALYSIS 138
B.3.1 Overview of Present Value Calculations 138
B.3.2 Monetization of Costs 139
B.3.3 Monetization of Dual-use Benefits 139
B.3.4 Monetization of Benefits during a Contamination Incident. 139
XIV
-------
List of Figures
Figure 2-1. CWS Architecture 3
Figure 2-2. Timeline for Optimization and Real-time Monitoring of the Cincinnati CWS 8
Figure 4-1. System-wide Data Completeness per Monthly Reporting Period 19
Figure 4-2. Component Downtime for Each Monitoring and Surveillance Component 21
Figure 4-3. Downtime Attributed to Sub-components for Each Monitoring and Surveillance Component 22
Figure 6-1. Scenarios Detected by Contaminant 32
Figure 7-1. Invalid Alerts by Component per Reporting Period (Normalized for Downtime) 35
Figure 7-2. Causes of Invalid Alerts for the CWS and each Component 36
Figure 7-3. Detection Percentage of Simulation Scenarios by Component 39
Figure 7-4. Simulation Scenarios First Detected by Each Component 41
Figure 7-5. Causes of Valid Alerts for the System and each Component 42
Figure 7-6. Change in Finished Water Quality Resulting from a Treatment Plant Change 43
Figure 7-7. WQM Alert Caused by the Treatment Plant Change Shown in Figure 7-6 44
Figure 7-8(a). Multiple Component Detections of Nuisance Chemicals 46
Figure 7-8(b). Multiple Component Detections of Contaminants with Taste or Odor 47
Figure 7-8(c). Multiple Component Detections of Contaminants with Rapid Symptom Onset 48
Figure 7-8(d). Multiple Component Detections of Contaminants with Delayed Symptom Onset 48
Figure 7-9. Distribution of Time Delays between Consecutive Alerts from the Simulation Study 49
Figure 8-1. Timeline for a Typical Contamination Scenario with Nuisance Chemical 1 53
Figure 8-2. Timeline for a Typical Contamination Scenario with Toxic Chemical 1 54
Figure 8-3. Timeline for a Typical Contamination Scenario with Toxic Chemical 5 55
Figure 8-4. Timeline for a Typical Contamination Scenario with Biological Agent 3 56
Figure 8-5. Timeline for a Typical Contamination Scenario with Biological Agent 4 56
Figure 8-6. Timeliness of Monitoring and Surveillance Component Alerts and Sampling and Analysis Results for
Distribution System Attack Scenarios 57
Figure 8-7. Timeliness of CCS Alerts by Contaminant 58
Figure 8-8. Timeliness of PHS Alerts for Contaminants with Rapid Symptom Onset (including those with a taste or
odor) 59
Figure 8-9. Timeliness of PHS Alerts for Contaminants with Delayed Symptom Onset 60
Figure 8-10. Timeliness of WQM Alerts by Station 61
Figure 8-11. Timeline for Full Scale Exercise 2 63
Figure 8-12. Timeline for Full Scale Exercise 3 64
Figure 8-14. Timeliness of Threat Level Determination and Responses for Nuisance Chemicals 66
Figure 8-15. Timeliness of Threat Level Determination and Responses for Contaminants with Taste or Odor 67
Figure 8-16. Timeliness of Threat Level Determination and Responses for Contaminants with Rapid Symptom
Onset 68
xv
-------
Figure 8-17. Timeliness of Threat Level Determination and Responses for Contaminants with Delayed Symptom
Onset 69
Figure 8-18. Timeliness of Threat Level Determination and Responses for Contaminants Detectable by 1, 2 or 3
Components 70
Figure 8-19. Timeline and Consequences for a Contamination Scenario Involving Biological Agent 4 73
Figure 8-20. Timeline and Consequences for a Contamination Scenario Involving Toxic Chemical 6 74
Figure 8-21. Timeline and Consequences for a Contamination Scenario Involving Toxic Chemical 4 75
Figure 8-22. Reduction in Fatalities Attributable to the CWS 77
Figure 8-23. Reduction in Illnesses Attributable to the CWS 78
Figure 8-24. Reduction in Healthcare Burden Attributable to the CWS 79
Figure 8-25. Reduction in Miles of Pipe Contaminated Attributable to the CWS 81
Figure 9-1. Benefit-Cost Analysis of the Cincinnati CWS 87
Figure 9-2. Percentage of CWS Alerts Investigated and the Number of Alerts Received 90
Figure A-l. CWS Model Architecture 107
Figure A-2. EPANET Model 108
Figure A-3. Health Impacts and Human Behavior Model 110
Figure A-4. Enhanced Security Monitoring Model 116
Figure A-5. Water Quality Monitoring Model 120
Figure A-6. Customer Complaint Surveillance Model 124
Figure A-7. Public Health Surveillance Model 128
Figure A-8. Consequence Management Model 131
XVI
-------
List of Tables
Table 3-1. Drills and Exercises Performed during the Pilot Evaluation Period 10
Table 3-2. Theoretical Detection Capabilities of the CWS Relative to the Contaminants Modeled in the Simulation
Study 12
Table 3-3. Summary of Scenario Variables Considered in the Simulation Study 14
Table 3-4. Summary of Primary and Secondary Consequences for Contaminants Evaluated in the Benefit-Cost
Analysis 16
Table 4-1. Data Completeness per Component 18
Table 4-2. Availability Criteria for Each Monitoring and Surveillance Component 20
Table 4-3. CWS Component Availability 23
Table 4-4. Concurrent CWS Component Availability 24
Table 5-1. Area and Population Coverage 27
Table 5-2. Number of Pito Zones with Corresponding Detection Percentages 28
Table 6-1. Ratio of Critical Concentration to Detection Threshold 30
Table 7-1. Number of Total Alerts and Normalized Alerts for the CWS and each Component 36
Table 7-2. Invalid Alerts per Reporting Period During Optimization and Real-time Monitoring 37
Table 7-3. Simulated Contamination Scenarios Detected by each Component 38
Table 7-4. Co-occurrence of WQM, CCS and PHS Alerts for Simulated Contamination Scenarios 49
Table 7-5. Co-occurrence of WQM, CCS and PHS Alerts Observed in the Emperical Data 50
Table 8-1. Number of Scenarios with Injections at High and Low Demand Periods 58
Table 8-2. Primary and Secondary Consequence Types for Each Contaminant 71
Table 9-1. Cost Elements used in the Calculation of Total Lifecycle Cost of the Cincinnati CWS 84
Table 9-2. Benefits Attributable to the Cincinnati CWS due to the Reduction in Consequences from a Contamination
Incident, in Millions of Dollars 85
Table 9-3. Dual-use Benefits of the Cincinnati CWS 89
Table A-l. EPANET Inputs 108
Table A-2. EPANET Outputs 108
Table A-3. HI/HB Model Inputs 112
Table A-4. HI/HB Model Parameters 112
Table A-5. HI/HB Model Outputs 114
Table A-6. ESM Model Inputs 116
Table A-7. ESM Model Parameters 117
Table A-8. ESM Model Outputs 117
Table A-9. WQM Model Inputs 120
Table A-10. WQM Model Parameters 121
Table A-ll. WQM Model Outputs 121
Table A-12. CCS Model Inputs 124
XVII
-------
Table A-13. CCS Model Parameters 125
Table A-14. CCS Model Outputs 125
Table A-15. PHS Model Inputs 128
Table A-16. PHS Model Parameters 129
Table A-17. PHS Model Outputs 130
Table A-18. CM Model Inputs 133
Table A-19. CM Model Parameters 134
Table A-20. CM Model Outputs 134
Table B-l. Cost Elements for Implementation and Operation of the Cincinnati CWS 137
Table B-2. Monetizable Benefits Attributable to the Cincinnati CWS due to the Reduction in Consequences from a
Contamination Incident 138
Table B-3. Cincinnati CWS Present Value Assumptions 138
Table B-4. CWS Component Useful Life 139
Table B-5. Contaminant-specific Medical Treatment Cost per Illness 140
Table B-6. Contaminant-specific Remediation Methods and Significant Cost Factors 141
XVIII
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 1.0: Introduction
The purpose of this document is to describe the evaluation of the Cincinnati contamination warning
system (CWS) pilot, the first such pilot deployed under the United States Environmental Protection
Agency's (EPA) Water Security Initiative (WSI). The following subsections of the introduction present
the CWS design objectives, the overall objectives of the evaluation and the organization of this report.
1.1 Contamination Warning System Design Objectives
The Cincinnati CWS was designed to meet six overarching objectives, which are described in detail in
WaterSentinel System Architecture (USEPA, 2005) and are presented briefly below:
Operational Reliability. The objective of this aspect of CWS design is to achieve a sufficiently
high degree of system availability such that the probability of missing a contamination incident
becomes exceedingly low. This design objective is met through redundancies built into the CWS
and each of its components. Metrics evaluated under this design objective include: data
completeness and availability.
Spatial Coverage. The objective of this aspect of CWS design is to monitor the entire
population served by the drinking water utility. This design objective depends on the location
and density of monitoring points in the distribution system and the hydraulic connectivity of each
monitoring point to downstream regions and populations. Metrics evaluated under this design
objective include: area coverage and population coverage.
Contaminant Coverage. The objective of this aspect of CWS design is to provide detection
capabilities for all priority contaminants. This design objective is further defined by binning the
priority contaminants into 12 classes according to the means by which they might be detected
(USEPA, 2005). Use of these detection classes to inform design provides more comprehensive
coverage of contaminants of concern than would be achieved by designing the CWS around a
handful of specific contaminants. Contaminant coverage is largely determined by the specific
data streams analyzed by each monitoring and surveillance component. Metrics evaluated under
this design objective include: contaminant detection threshold and contamination scenario
coverage.
Alert Occurrence. The objective of this aspect of CWS design is to minimize the rate of invalid
alerts (alerts unrelated to drinking water contamination or other unusual water quality conditions)
while maintaining the ability of the system to detect real incidents. This design objective depends
on the quality of the underlying data as well as the event detection systems that analyze that data
for anomalies. Metrics evaluated under this design objective include: invalid alert occurrence,
valid alert occurrence and alert co-occurrence.
Timeliness of Detection and Response. The objective of this aspect of CWS design is to
provide initial detection of a contamination incident in atimeframe that allows for the
implementation of response actions that result in significant consequences reduction. Metrics
evaluated under this design objective include: detection time, response time and consequence
reduction.
Sustainability. The objective of this aspect of CWS design is to provide benefits to the utility
and partner organizations while minimizing the costs. This can be achieved by leveraging
existing systems and resources that can readily be integrated into the design of the CWS.
Furthermore, a design that results in dual-use applications that benefit the utility's day-to-day
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
operations while also providing the capability to detect intentional or accidental contamination
incidents, will also improve sustainability. Metrics evaluated under this design objective include:
net present value, dual-use benefits and willingness to maintain the CWS.
1.2 Evaluation Objectives
The purpose of WSI was to pilot and evaluate a drinking water CWS. The lack of established design
standards for the relatively new CWS concept precluded an evaluation of system performance in absolute
terms. Instead, the Cincinnati CWS was evaluated to characterize how well it met the design objectives
described above. Several sources of information were used to conduct this evaluation, including data
collected during routine operation, drills and exercises, and computer simulations.
Evaluation of the Cincinnati CWS pilot was performed at both the system and component level. This
report presents results from the evaluation of the integrated CWS. Six additional reports, which are listed
in Section 11, present the results from the detailed evaluation of each of the primary CWS components.
Both the system and the components were evaluated against the design objectives using the same general
metrics. However, the system evaluation considers the performance of the integrated CWS and
characterizes metrics that are applicable only to the system as a whole, such as the potential reduction in
consequences of a contamination incident. Furthermore, the CWS evaluation report does not present a
detailed analysis of the performance of individual components, which can be found in the component
evaluation reports.
1.3 Organization of this Report
This document contains the following sections:
Section 2: Overview of the Cincinnati CWS. This section provides a brief overview of each
component of the Cincinnati CWS and presents a summary of significant milestones and
modifications made to the CWS during the evaluation period of the pilot.
Section3: Methodology. This section describes the data sources and techniques used to
evaluate the Cincinnati CWS.
Sections 4 through 9: Evaluation of CWS Performance relative to the Design Objectives.
Each of these sections addresses one of the design objectives listed in Section 1.1. Each section
introduces the metrics that will be used to evaluate the CWS relative to that design objective.
Each of these metrics is discussed in a dedicated subsection that defines the metric, provides an
overview of the evaluation method, and presents the results.
Section 10: Summary and Conclusions. This section provides an overall summary of
Cincinnati CWS performance and discusses limitations and applications of the results.
Section 11: References. This section lists all sources and documents cited in this report.
Section 12: Abbreviations. This section defines all abbreviations used in this report.
Section 13: Glossary. This section provides definitions for terms used in this report.
Appendix A: Cincinnati Contamination Warning System Model. This appendix describes
the Cincinnati CWS model used in the simulation study as well as the design of the study itself.
Appendix B: Benefit-Cost Analysis Methodology. This appendix describes the methodology
and assumptions used to evaluate the net present value of the Cincinnati CWS.
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 2.0: Overview of the Cincinnati CWS
The overall architecture of the Cincinnati CWS is presented in Figure 2-1, which shows two operational
paradigms: 1) monitoring and surveillance and 2) response. Monitoring and surveillance consists of the
following four components: enhanced security monitoring, water quality monitoring, customer complaint
surveillance and public health surveillance.
Monitoring and Surveillance
Response
Enhanced Security ^ /Customer Complaint/
Monitoring JI Surveillance J
/ WaterQuality / / Public Health /
I Monitoring j \ Surveillance )
i Event Detection
Initial Alert
Investigation
// Can \,
contamination be
"\ruled out?/
! Yes
t
No
Take corrective action if necessary;
continue surveillance
Consequence
Management
Credibility Determination Process to
confirm or rule out contamination
Check alerts from surveillance
components
Assess outside data sources
Sampling and
Analysis
/Field Investigation to support
credibility determination
Site characterization
Field and lab analyses
Response Actions protect
public health during the
investigation process and may
include:
Isolation
Flushing
Public alerts/notifications
Remediation and Recovery
restores a system to normal
operations and may include:
System characterization
Remedial action
Post-remediation activities
Figure 2-1. CWS Architecture
The purpose of routine monitoring and surveillance is to detect unusual water quality conditions that may
be indicative of possible drinking water contamination. The monitoring and surveillance components are
not designed to detect specific contaminants, but rather changes from baseline conditions that warrant
further investigation (or in the case of enhanced security monitoring, detect unauthorized access to a
drinking water distribution system facility). If the conclusion from the initial investigation is that the alert
is valid, CWS operations transition to response. During response, the investigation of the Possible
contamination incident continues under consequence management in an attempt to determine whether or
not contamination is Credible. Additionally, procedures under sampling and analysis are used in an
attempt to confirm the incident and identify the contaminant. Consequence management also guides
response actions that are intended to protect utility infrastructure and the public from potentially
contaminated drinking water while the investigation proceeds.
The six components that make up the Cincinnati CWS are described in more detail in the following
subsections.
2.1 Enhanced Security Monitoring
Enhanced security monitoring (ESM) is one of the four monitoring and surveillance components of a
CWS. This component includes the systems, equipment and procedures for detecting and responding to
security breaches at distribution system facilities such as pump stations, elevated storage tanks and
reservoirs that are vulnerable to contamination. At GCWW, ESM capabilities were installed at 12
distribution system facilities. ESM data streams consist of 59 pieces of physical security equipment such
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
as motion sensors, video cameras and magnetic proximity switches, which are monitored using associated
response procedures. ESM alerts are transmitted over the utility's Supervisory Control and Data
Acquisition (SCADA) network to the control facility at the main treatment plant where they are displayed
on a SCADA interface.
The status of the ESM component is monitored 24/7, 365 days a year via the SCADA user interface. A
physical security breach that generates an alert initiates an investigation to determine whether or not the
intrusion presented an opportunity to contaminate drinking water. The investigation includes a review of
video clips (if available) and a physical site inspection to verify an intrusion. If video or on-site evidence
corroborates the security breach and potential access to the drinking water, contamination is considered
Possible. A detailed evaluation of the ESM component of the Cincinnati CWS can be found in Water
Security Initiative: Evaluation of the Enhanced Security Monitoring Component of the Cincinnati
Contamination Warning System Pilot (USEPA, 2014b).
2.2 Water Quality Monitoring
Water quality monitoring (WQM) is one of the four monitoring and surveillance components of a CWS.
This component consists of a network of monitoring stations located throughout a drinking water
distribution system, a data management system, and procedures for responding to alerts. At GCWW,
there are 15 WQM stations with sensors representing a total of 82 data streams installed throughout the
distribution system, as well as a monitoring station at each of the two treatment plants. The data from the
two treatment plant monitoring stations are not used to detect contamination, but instead to facilitate the
investigation and validation of alerts produced by any of the 15 monitoring stations in the distribution
system. Specifically, data from the two monitoring stations located at the treatment plants provides a
benchmark for water quality in the distribution system. The parameters monitored include free chlorine
residual, specific conductivity, oxidation reduction potential, pH, temperature, total organic carbon and
turbidity.
Data from the remote WQM stations are polled every two minutes and transmitted via digital cellular to a
centralized SCADA system. The SCADA system collects and displays the data from all monitoring
stations in real time. Simultaneously, the data is transmitted to an event detection system, which is an
algorithm that continually analyzes water quality data, along with metadata such as sensor alerts and data
quality flags, to monitor for changes in water quality triggered by abnormal conditions. If abnormal
conditions are detected, a visual and audible alert is generated.
The status of the WQM component is monitored 24/7, 365 days a year via the SCADA user interface.
Unusual water quality that generates a WQM alert initiates an investigation to determine the cause of the
alert. The investigation considers plausible causes, and may include a site inspection to verify that the
equipment is functioning properly. If all reasonable explanations and likely benign causes are ruled out,
contamination is considered Possible. A detailed evaluation of the WQM component of the Cincinnati
CWS can be found in Water Security Initiative: Evaluation of the Water Quality Monitoring Component
of the Cincinnati Contamination Warning System Pilot (USEPA, 2014c).
2.3 Customer Complaint Surveillance
Customer complaint surveillance (CCS) is one of the four monitoring and surveillance components of a
CWS. This component involves monitoring customer complaints about water quality to identify
degradation of distributed water quality, potentially including contamination. Customers may detect
contaminants with characteristics that impart an odor, taste, or visual change to the drinking water or that
result in instantaneous symptoms such as a mild dermal irritation.
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Three independent data streams are monitored: Interactive Voice Response (FVR), work request and work
order systems. The IVR allows customers to self-select whether they are calling about a water quality
concern by pushing the corresponding number on the phone (i.e., push 5 for a water quality concern). A
customer service representative may generate a work request if, after interviewing the caller, (s)he
believes that additional investigation of the complaint is warranted by a water quality specialist. Upon
review, the specialist may convert the work request into a work order to initiate the requested follow-up
action. A custom event detection system analyzes each data stream independently in real time. If the
event detection system detects an anomaly, automated email alerts are generated. While the work request
data stream was monitored and maintained through January 2009, GCWW disabled alert generation for
this data stream as part of the transition to real-time analysis. After a year of receiving alerts, GCWW
deemed the data stream to be redundant with the work order data stream. The work request data stream is
included only in analyses that cover the optimization period up until the transition to real-time analysis.
The status of the CCS component is monitored by call center personnel during normal business hours and
by a dispatcher during off hours. When a CCS alert is generated, the complaints are first examined to
determine whether they are spatially clustered and/or have similar complaint descriptions. If so, the
investigator reviews other information such as distribution system work and operations for possible
benign causes of the alert. If the investigators conclude that the calls are unrelated, the investigation into
possible contamination is closed, and regular procedures for customer complaint follow-up are
implemented. If the complaints are clustered and there is no benign explanation for the complaints,
contamination is considered Possible. A detailed evaluation of the CCS component of the Cincinnati
CWS can be found in Water Security Initiative: Evaluation of the Customer Complaint Surveillance
Component of the Cincinnati Contamination Warning System Pilot (USEPA, 2014d).
2.4 Public Health Surveillance
Public health surveillance (PHS) is one of the four monitoring and surveillance components of a CWS.
This component involves monitoring health seeking behaviors in an effort to detect the early signs of a
public health incident in a community. Most of the priority contaminants considered under WSI can
cause serious health effects to individuals exposed to a sufficiently high dose. Presumably, some of the
symptomatic individuals would seek healthcare, and in sufficient numbers, these health seeking behaviors
can produce a PHS alert.
In the Cincinnati CWS pilot, the following public health surveillance data streams are monitored: 911
calls, emergency medical service logs, Cincinnati Drug and Poison Information Center (DPIC) calls,
emergency department (ED) visits (including both hospital and urgent care facilities) and reporting from
astute clinicians. This diverse set of data streams has the potential to detect contaminants that produce
rapid onset of symptoms following exposure, as well as those with delayed symptom onset. However, for
this evaluation only the PHS systems that were installed for the pilot are included in analyses.
Operational reliability was evaluated for only the 911 and EMS data streams, while alert occurrence was
evaluated for the 911, EMS and DPIC data streams.
Each of the data streams listed above has a unique monitoring and notification strategy. Alerts from the
911 or emergency medical service data streams are automatically emailed to members of a PHS User
Group, which includes members from the county and city health departments, DPIC, GCWW and law
enforcement. Distribution of alerts to this diverse group facilitates information sharing from a variety of
sources during the alert investigation. The DPIC call center is staffed 24/7, 365 days a year by trained
personnel. ED data is monitored by EpiCenter, an automated syndromic surveillance tool that can send
alert notifications to personnel at the county and city health departments. Reporting from astute clinicians
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
is an informal surveillance method that relies upon the observations of healthcare providers to alert public
health officials when they observe unusual symptoms or diseases in their patients.
Once an alert is received, it is reviewed by personnel from the health department or DPIC. If the reviewer
believes that the alert is possibly related to drinking water contamination, they implement a process called
the "communicator," which is an autodialing system used to send out a message to all members of the
PHS User Group. Typically, a call is convened to review and evaluate the alert. If the PHS User Group
concludes that the alert is a valid indicator of a public health incident, and if the causative agent could
have been delivered via the drinking water supply, then contamination is considered Possible. A detailed
evaluation of the PHS component of the Cincinnati CWS can be found in Water Security Initiative:
Evaluation of the Public Health Surveillance Component of the Cincinnati Contamination Warning
System Pilot (USEPA, 2014e).
2.5 Consequence Management
Consequence management (CM) is one of the two response components of a CWS. This component
includes the plans and procedures that govern the investigation of and response to a Possible, Credible, or
Confirmed contamination incident. These procedures are meant to minimize response and recovery
timelines through a pre-planned, coordinated effort. Investigative and response actions initiated upon
determination of a possible contamination incident are used to establish credibility, minimize public
health and economic consequences and ultimately return the utility to normal operations.
The procedures that govern consequence management are documented in a Consequence Management
Plan (CMP). The CMP consists of a series of decision trees that guide the investigation to determine if
the contamination incident is Credible or Confirmed and the implementation of response actions to
minimize consequences.
The threat level determination process in CM involves the collection of additional information related to
the Possible contamination incident from a variety of sources, including all monitoring and surveillance
components as well as sampling and analysis activities, which are discussed in Section 2.6. If there is
sufficient information to corroborate the initial alert(s), contamination is considered Credible. If sampling
and analysis activities identify a contaminant in a sample from the distribution system or if there is a
preponderance of evidence from a variety of sources, contamination is Confirmed. A determination that
contamination is Credible or Confirmed may result in elevated response actions.
Response actions taken during consequence management are intended to minimize public health
consequences and contamination of utility infrastructure. A range of response actions is available to the
utility, and the level of response action generally correlates with the threat level (i.e., more aggressive
response actions will be considered when the contamination incident is considered Credible compared to
Possible). While the response actions are situation-specific, potential response actions to contamination
can be grouped into three categories: operational response, public notification and public health response.
Operational response typically involves the manipulation of distribution system control points (e.g.,
pumps, valves, tanks, etc.) to either limit the spread of potentially contaminated water or to purge it from
the system. In general, operational responses are considered as early as the time when contamination is
considered Possible. However, the impact of a specific operational response on utility operations,
customers and the environment must be considered in the context of the threat level.
Public notification involves direct communication to the public and often includes instructions regarding
use restrictions (e.g., Do not drink or Do not use). The intent of public notification is to limit exposure to
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
potentially contaminated water. Because issuance of a use restriction has a serious impact on the public,
the utility would implement such an action only if contamination were considered Credible or Confirmed.
Unlike the previous two response actions, public health response is not implemented by the utility, but
instead by public health officials in response to a developing public health crisis. In fact, public health
response may be implemented independent from the utility and any potential connection to contaminated
water. Public health response might include mobilization of additional medical resources and issuance of
prophylaxis. In extreme cases, public health or government officials may recommend temporary
evacuations to remove the public from the source of exposure.
A detailed evaluation of the CM component of the Cincinnati CWS can be found in Water Security
Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination
Warning System Pilot (USEPA, 2014f).
2.6 Sampling and Analysis
Sampling and analysis (S&A) is one of the two response components of a CWS. This component is a
support function under CM that provides information to the threat level determination process and
decisions regarding response, remediation, and recovery actions. S&A includes the capabilities,
equipment and procedures for conducting site characterization (SC) and laboratory analysis (LA) during
the investigation of a contamination incident. SC and LA are the two primary processes undertaken when
S&A is activated in response to a possible water contamination incident.
SC involves the collection of information from a location in the distribution system to support the threat
level determination process. SC activities include site approach and observation, field safety screening,
rapid field testing of drinking water at the site and collection of samples for laboratory analysis. The
location of a SC will be situation-specific; however, SC teams are often dispatched to the location of an
alert from one of the four monitoring and surveillance components. SC activities performed at the
location of an ESM alert are unique in that the investigation could show signs of tampering at a utility
facility that would inform the threat level determination process. For all other sites, information to inform
the threat level determination process may be limited to the results from field testing of water. Depending
on the perceived hazards at the site, SC may be performed by either trained utility personnel or by
Hazmat responders.
Samples collected from the field during SC are delivered to laboratories for further analysis. This
involves transport of samples from the field to one or more laboratories using chain of custody
procedures, laboratory and method mobilization, sample analysis, quality control (QC) procedures and
reporting of the results. LA is pre-planned to identify laboratories and methods prior to an incident in
order to streamline the process and reduce the time between sample collection and reporting. Because the
identity of a potential contaminant is often unknown during a suspected, but unconfirmed, contamination
incident, GCWW identified a baseline suite of analytes that would be included in any laboratory
investigation into a contamination incident. Analytes outside of this baseline suite would be analyzed
only if evidence was available to implicate a potential contaminant outside of the baseline suite.
A detailed evaluation of the S&A component of the Cincinnati CWS can be found in Water Security
Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning
System Pilot (USEPA, 2014g).
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
2.7 CWS Evaluation Timeline
The Cincinnati CWS was fully deployed and operational by the end of 2007 and a detailed description of
the CWS at this point in the project can be found in Water Security Initiative: Cincinnati Pilot Post-
Implementation System Status (USEPA, 2008). Figure 2-2 shows the significant activities that occurred
during the pilot from January 2008 through June 2010. Two phases of the pilot occurred during this
period, the optimization phase and the real-time monitoring phase.
Jan-09
CCS and ESM
Real-time Monitoring
Jun-09
Transfer to
Utility Ownership
Jun-09
PHS and WQM
Real-time Monitoring
Jan-08
;
Jun-10
Design &
Implementation
Complete
Jan-08
V
Optimization
Jan-08 - Jan-09
Windstorm
Sep-08
Real-time Monitoring
Jan-09 -Jun-10
End of Data
Collection
Jun-10
Figure 2-2. Timeline for Optimization and Real-time Monitoring of the Cincinnati CWS
The optimization phase lasted from January 2008 through January 2009, and represents a period during
the pilot when the components were fully operational but not deemed ready for real-time monitoring.
During the optimization phase, components generated data, which was analyzed to demonstrate
performance relative to several key metrics such as availability, alert rates, data accuracy and
completeness and the level of effort required to maintain the component. Findings from this ongoing
evaluation were used to modify the system in an effort to improve performance. During this phase, there
was one full-scale exercise (FSE) and several smaller drills and exercises, which are not shown in this
figure but are described in Section 3.2. These drills and exercises were used to assess implementation of
procedures by utility personnel and local partners, and the findings were used to optimize procedures.
There was also a major incident during this period, a windstorm that interrupted the power supply
throughout the city, which occurred in September 2008. This event had a significant impact on CWS
performance, as discussed later.
January 2009 marked the start of the transition to real-time monitoring, the period during which CWS
alerts were immediately investigated when they occurred. During this period, procedures were
implemented to ensure that alerts were acknowledged and investigated 24/7. CCS and ESM were the first
two components to begin real-time monitoring in January 2009. The transition of WQM to real-time
monitoring occurred in stages from January to June 2009 when the transition was completed. PHS also
completed the transition to real-time monitoring during June 2009. Coincidentally, the formal agreement
between EPA and the City of Cincinnati ended in June 2009, at which time ownership of the Cincinnati
CWS was completely transferred over to GCWW and its partners. Through June 2009, EPA and its
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
contractors provided support for operation and maintenance of the CWS, but following this transfer of
ownership, GCWW assumed these responsibilities. The period between June 2009 and June 2010
provided 13 months of data collection during real-time monitoring, which is indicative of expected
performance for a stable, optimized CWS. The final FSE was conducted during this period in October
2009 and provided an opportunity to evaluate personnel implementation of the fully tested and optimized
procedures developed during the pilot.
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 3.0: Methodology
This evaluation includes data on the performance, operation and sustainability of the Cincinnati CWS
from January 16, 2008 to June 15, 2010. The following section describes six evaluation techniques and
data sources that were used to fully evaluate the performance of the Cincinnati CWS against the design
objectives described in Section 1.1: empirical data from routine operations, results from drills and
exercises, results from computer simulations of the Cincinnati CWS, results from a benefit-cost analysis,
findings from forums such as lessons learned workshops and information from literature and research.
3.1 Analysis of Empirical Data from Routine Operations
The preferred method for evaluating the performance of the Cincinnati CWS was through the analysis of
empirical data collected during the evaluation period. Empirical data was analyzed over time to illustrate
the change in performance as the CWS evolved during the evaluation period. Statistical methods were
also used to summarize large volumes of data collected over either the entire or various segments of the
evaluation period. Data was also evaluated and summarized for each reporting period over the evaluation
period. In this evaluation, the term reporting period is used to refer to one month of data that spans from
the 16th of the indicated month to the 15th of the following month. Thus, the January 2008 reporting
period refers to the data collected between January 16, 2008 and February 15, 2008.
One of the primary sources of empirical data used in the evaluation was the investigation checklists that
were completed for CWS alerts and which documented information such as alert time, location and cause.
Other sources of empirical data used in the evaluation include: O&M logs, labor reporting records and
other databases used to manage data from the CWS.
3.2 Drills and Exercises
During the evaluation period, no Possible contamination incidents were detected, and thus some
procedures were not utilized during routine operations. Drills and exercises, designed around mock
contamination incidents were used to practice and evaluate the full range of procedures, from initial
detection through response. Drills and exercises also provided an opportunity to identify procedures
requiring modification to achieve the desired outcome in an effective and efficient manner. All of the
drills and exercises that were designed to test and evaluate the Cincinnati pilot were compliant with
Homeland Security Exercise and Evaluation Program guidelines. Findings from drills and exercises were
used to evaluate several aspects of CWS performance, such as timeliness of decisions and response
actions. Nineteen drills and two FSEs were conducted over the course of the evaluation period. Note that
there was one FSE (FSE 1) that was conducted prior to the evaluation period, and thus is not included in
this analysis. Table 3-1 provides the date and a brief description of each drill or exercise and indicates
which components were included.
Table 3-1. Drills and Exercises Performed during the Pilot Evaluation Period
Drill
S&A Drill 1
ESM Drill 1
WQM Drill 1
S&A Drill 2
Date
05/07/08
06/26/08
07/14/08
07/15/08
Description
Evaluated incident response procedures, along with related CM activities.
Evaluated interactions among local law enforcement and GCWW Security
and Distribution Division personnel in response to an ESM alert.
Evaluated response to an initial alert caused by changes in chlorine and
conductivity, followed by an alert caused by a change in total organic
carbon (TOC).
Provided GCWW SC team members and Cincinnati Fire Department
Hazardous Material (HazMat) responders with an opportunity to cross-
train on SC procedures.
10
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Drill
CCS Drill 1
PHS Drill 1
FSE2
WQM Drill 2
ESM Drill 2
S&A Drill 3
PHS Tabletop
Exercise
S&A Drill 4
WQM Drill 3
CCS Drill 2
ESM Drill 3
PHS Drill 2
CCS / S&A Drill
FSE3
ESM Drill 4
CCS Drill 3
S&A Bioterrorism
(BT) Agent Drill
Date
08/19/08
08/22/08
10/01/08
02/25/09
03/11/09
03/31/09
04/22/09
04/23/09
04/29/09
04/29/09
04/30/09
07/28/09
09/16/09
10/21/09
04/13/10
04/15/10
05/10/10
Description
Evaluated alert recognition and investigation procedures through various
alert notification methods where simulated customer complaints produced
both IVR and work request alerts.
Evaluated the alert investigation procedures and the interactions between
local public health partners and the GCWW Water Utility Emergency
Response Manager (WUERM).
Provided an opportunity for the utility and local response partner agency
to exercise procedures related to the detection of and response to a
drinking water contamination incident. This exercise involved WQM,
CCS, PHS, S&A and CM.
Evaluated changes made to the component alert investigation procedures
based on results from the first WQM drill and FSE 2.
Evaluated interactions among local law enforcement and GCWW Security
and Distribution Division personnel in response to an ESM alert.
Evaluated procedures that guide interactions between GCWW and
laboratories as well as the receipt and interpretation of laboratory results.
Evaluated the ability of GCWW and local public health partners to
determine if a simulated PHS alert was due to drinking water
contamination.
Evaluated the implementation of revised SC and sample collection
procedures.
Evaluated implementation of alert investigation procedures during non-
business hours by personnel that had not previously participated in a drill.
Evaluated alert recognition and investigation procedures through various
alert notification methods where simulated customer complaints produced
both IVR and work request alerts.
Evaluated interactions among local law enforcement and GCWW Security
and Distribution Division personnel in response to an ESM alert.
Provided local public health partners and the GCWW WUERM the
opportunity to practice PHS alert investigation procedures.
Evaluated the alert recognition and investigative procedures associated
with the CCS component and implementation of the SC procedures as
they relate to field deployment and investigation following a CCS alert.
Evaluated utility and local response partner agency protocols, including
implementation of the Incident Command System, external notifications,
resource coordination, media relations and the execution of field
investigation procedures. This exercise involved CCS, PHS, S&A and
CM.
Evaluated interactions among local law enforcement and GCWW Security
and Distribution Division personnel in response to a witness account of an
intrusion.
Evaluated implementation of alert investigation procedures during non-
business hours.
Practiced SC and partner laboratory capabilities, including internal
notification procedures to prepare to receive and analyze samples using
the Laboratory Response Network BT Agent Screening Protocol.
11
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
3.3 Simulation Study
Evaluation of certain design objectives relies on the occurrence of contamination incidents with known
and varied characteristics (Davis and Janke, 2011; Davis, Janke and Magnuson, 2013). Because there
were no contamination incidents during the evaluation period, there is no empirical data to fully evaluate
the detection capabilities of the Cincinnati CWS. To fill this gap, a computer model of the Cincinnati
CWS was developed and challenged with a large ensemble of simulated contamination incidents in a
simulation study. A detailed description of the Cincinnati CWS model is provided in Appendix A of this
report. This section describes the design of the simulation study that generated the data used in the
evaluation.
To perform a robust evaluation of system performance, the simulation study was designed to challenge
the Cincinnati CWS model with a wide range of contamination scenarios. The attributes that defined a
contamination scenario in this study include: contaminant, injection location, start time of the injection,
and mass injection rate. Each attribute is further described below.
Contaminant
A broad range of contaminant types producing a range of symptoms was selected for the simulation study
in order to characterize the detection capabilities of the monitoring and surveillance components of a
CWS (http://www.epa.gov/wcit). For the purpose of the simulation study, a representative set of 17
contaminants was selected from the comprehensive contaminant list that formed the basis for CWS
design. These contaminants are grouped into the following broad categories (the number in parentheses
indicates the number of contaminants from that category that were simulated during the study):
Nuisance Chemicals (2). These chemicals have a relatively low toxicity and thus generally do
not pose an immediate threat to public health, but can make the drinking water supply unusable
(e.g., dyes and malodorants).
Toxic Chemicals (8). These chemicals are highly toxic and pose an acute risk to public health at
relatively low concentrations (e.g., pesticides).
Biological Agents (7). These materials are derived from biological sources and pose an acute
risk to public health at relatively low concentrations (e.g., bacterial pathogens).
These 17 contaminants also presented a range of detection challenges. Table 3-2 lists the 17
contaminants, indicating which of the monitoring and surveillance components have the potential to
detect each. This assessment of detection potential was based largely on literature review and research, as
discussed in Section 3.6, and was used to parameterize the Cincinnati CWS model. ESM is not shown in
the table because the detection capabilities of this component are contaminant neutral.
Table 3-2. Theoretical Detection Capabilities of the CWS Relative
to the Contaminants Modeled in the Simulation Study
Contaminant1
Nuisance Chemical 1
Nuisance Chemical 2
Toxic Chemical 1
Toxic Chemical 2
Toxic Chemical 3
Toxic Chemical 4
Toxic Chemical 5
Toxic Chemical 6
WQM
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
CCS2
Yes
No
Yes
Yes
Yes
Yes
No
No
PHS
No4
No4
Yes
Yes
Yes
Yes
Yes
Yes
S&A3
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
12
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Contaminant1
Toxic Chemical 7
Toxic Chemical 8
Biological Agent 1
Biological Agent 2
Biological Agent 3
Biological Agent 4
Biological Agent 5
Biological Agent 6
Biological Agent 7
WQM
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
CCS2
No
No
Yes
No
No
No
No
No
No
PHS
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
S&A3
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
The 17 contaminants modeled in the simulation study were assigned generic IDs for security purposes.
2 Detection by CCS is possible only for contaminants that change the aesthetic character of the water in a manner
that can be detected via the human senses.
3 S&A is not considered an early detection component, but is included in this table to show the detection capabilities
of this response component relative to the three monitoring and surveillance components.
4 Based on design of simulation model, early detection via PHS does not apply to nuisance chemicals. For Nuisance
Chemical 1, customers would detect odor and not consume a sufficient volume of water to produce adverse health
effects. For Nuisance Chemical 2, concentrations are sufficient to produce only long-term, chronic health effects,
which are not considered in this model.
Injection Location
The location where the contaminant is injected into the system has a direct and dramatic impact on
consequences. It plays a role in defining the flow path of the contaminant through the system as well as
to the downstream users who could be exposed to harmful concentrations of the contaminant. Given that
there is no prior knowledge of the location of an intentional contamination incident, it was assumed that
all distribution system model nodes are potential attack locations, with the exclusion of nodes with zero
demand and nodes at terminal points in the distribution system. Applying these criteria to the GCWW
distribution system model resulted in 5,799 potential injection locations. Two types of contaminant
injection locations were simulated: sites at GCWW facilities and sites at distribution system nodes.
Scenarios in which the injection was simulated at a distribution system node are referred to as distribution
system attack scenarios, while those in which the injection was simulated at a utility facility are referred
to as facility attack scenarios.
Injection Start Time
The start time of the injection will also impact the magnitude and distribution of consequences due to
diurnal variations in flow patterns and water demand. In this study, two injection start times were
selected based on the maximum and minimum total modeled demands across the entire distribution
system. The 9:00 a.m. start time was selected to represent an injection commencing during a period of
the day when the total demand across the system was large and sustained for a significant duration. The
12:00 a.m. start time was selected to represent an injection during the sustained low demand period that is
characteristic of early morning.
Mass Injection Rate
The mass injection rate is directly proportional to the contaminant concentration and thus has a direct
impact on consequences. The contamination concentration in the pipe also depends on the flow rate at the
injection location, which varies widely throughout the system. For this reason, three mass injection rates
were selected for each combination of contaminant and injection location. The duration of contaminant
injection was calculated from the mass injection rate and the total mass of contaminant, which was
estimated from a detailed analysis of the availability of each contaminant. However, injection duration
13
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
was bounded between a minimum of 60 minutes and a maximum of 24 hours in order to represent
reasonable scenario conditions. The mass injection rate remained constant over the injection duration.
A summary of the scenario variables considered in this study are summarized in Table 3-3, and resulted
in set of 591,498 scenarios (34,794 unique scenarios per contaminant). These scenarios were screened
using the Threat Ensemble Vulnerability Assessment tool, a software application that is highly efficient at
executing large ensembles (http://www.epa.gov/nhsrc/toolsandapps.html).
Table 3-3. Summary of Scenario Variables Considered in the Simulation Study
Scenario Variable
Contaminant Type
Injection Location
Injection Time
Injection Rate
Range of Values
Seventeen chemicals and biological agents representing a variety of detection
challenges and a range of public health consequences or infrastructure contamination.
5,799 nodes, representing all feasible injection locations in the distribution system
model. Injections occur at facilities (facility attack scenarios) or in the distribution
system (distribution system attack scenarios).
Two times, representing periods of high and low water demand and different
distribution system operating conditions.
Three rates, with a minimum of 60 minutes and a maximum of 24 hours.
The results from the Threat Ensemble Vulnerability Assessment tool were screened to identify an
ensemble of scenarios for detailed evaluation in the simulation study. The criteria for identifying
scenarios for inclusion in this ensemble are:
The scenario that produces the largest consequences for each contaminant in each of 94 pito
zones in the GCWW distribution system was included in the ensemble. A pito zone is a small
region of the distribution system in which water quality and pressure are fairly constant. Pito
zones range in size from 0.29 to 15 square miles, with an average area of 3.1 square miles.
One custom designed scenario for each contaminant injected at each of the 25 utility distribution
system facilities was included in the ensemble.
The screening criteria yielded 2,023 scenarios or 119 scenarios per contaminant. Eight of these scenarios
did not produce any significant consequences and were thus removed, resulting in an ensemble of 2,015
scenarios used in the simulation study.
This ensemble achieved the main goal of the simulation study, which was to challenge the Cincinnati
CWS model with a diverse set of simulated contamination incidents, which:
Represent a broad range of contaminants,
Include injection locations throughout the distribution system, and
Include scenarios that are optimized to maximize consequences.
The results from the simulation study were used to evaluate a number of performance metrics presented in
this report. Several analyses were performed on two ensemble subsets: facility attack scenarios and
distribution system attack scenarios. The facility attack scenarios subset consists of the 425 scenarios
injected at one of the 25 facility nodes in the distribution system model. The distribution system attack
scenarios subset consists of the remaining 1,590 scenarios injected at distribution nodes in the distribution
system model. If a scenario subset is not specified, the results are from an analysis performed on the
entire ensemble of 2,015 scenarios.
14
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
3.4 Benefit-Cost Analysis
To evaluate the sustainability of the Cincinnati CWS in a quantitative fashion, a benefit-cost analysis was
conducted to compare the monetized value of costs and benefits and calculate the net present value of the
CWS. Cost and benefits that cannot be monetized were evaluated qualitatively. The results of the
benefit-cost analysis were used to assess the sustainability of the Cincinnati CWS.
The total cost of the Cincinnati CWS over an assumed 20-year lifecycle was determined by summing
implementation costs, annual O&M costs, renewal and replacement costs, and the salvage value of major
pieces of equipment at the end of the lifecycle. Implementation costs included labor and other
expenditures (equipment, supplies and purchased services) for installing the system components.
Implementation costs were summarized in Water Security Initiative: Cincinnati Pilot Post-
Implementation System Status (USEPA, 2008), which was used as a primary data source for this analysis.
Annual O&M costs include labor and other expenditures (supplies and purchased services) necessary to
operate and maintain the system and investigate alerts. O&M costs were obtained from procurement
records, maintenance logs, investigation checklists and training records. Procurement records provided
the cost of supplies, repairs and replacement parts, while maintenance logs tracked the staff time spent
maintaining the system. To account for the maintenance of documents, the cost incurred to update
documented procedures following drills and exercises conducted during the evaluation phase of the pilot
was used to estimate the annualized cost. Investigation checklists and training records tracked the staff
hours spent on investigating alerts and training, respectively. The total O&M costs were annualized by
calculating the sum of labor and other expenditures incurred over the course of a year.
Renewal and replacement costs are based on the cost of replacing major pieces of equipment at the end of
their useful life. In general, the useful life of each item was estimated using field experience,
manufacturer-provided data, and input from subject matter experts. Equipment was assumed to be
replaced at the end of its useful life over the 20-year lifecycle of the Cincinnati CWS. The salvage value
is based on the estimated value of each major piece of equipment at the end of the 20-year lifecycle.
Salvage value was estimated using straight line depreciation for all equipment with an initial value greater
than approximately $1,000.
The benefits of a CWS were considered in two broad categories, consequence reduction and routine
operations. Benefits related to consequence reduction include reductions in the following consequences
of contamination: fatalities, cost of medical treatment, cost of distribution system remediation, cost of
alternate water supply during remediation, lost water and wastewater utility revenues and lost wages and
business revenue. Benefits related to routine operations include any value derived from operation of a
CWS that are not related to detection of and response to contamination, such as the identification of
unusual water quality incidents, information used to support regulatory compliance, increased public
confidence in the water supply and improved coordination with external partners. Many operational
benefits are difficult to monetize in a reliable manner and were thus evaluated qualitatively.
Benefits from consequence reduction were estimated using the results of the simulation study described in
Section 3.3. Due to the level of effort required to estimate the consequences of a contamination incident,
only 30 of the 2,015 scenarios evaluated in the simulation study were considered in the benefit-cost
analysis. However, the 30 scenarios selected for this study represent a wide range of conditions and
potential consequences. Table 3-4 presents the 10 contaminants evaluated as well as their primary and
secondary consequence. The selected contaminants included five toxic chemicals, four biological agents
and one nuisance chemical. Three scenarios were selected for each contaminant based on the ranking of
primary consequences produced by the scenario without the CWS in operation. Scenarios were selected
15
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
with consequences near the 25th, 50th and 75th percentile ranking of the distribution of consequences for
all scenarios in which that contaminant was used.
Table 3-4. Summary of Primary and Secondary Consequences for Contaminants
Evaluated in the Benefit-Cost Analysis
Contaminant
Toxic Chemical 1
Toxic Chemical 5
Toxic Chemical 6
Toxic Chemical 7
Toxic Chemical 8
Biological Agent 3
Biological Agent 4
Biological Agent 5
Biological Agent 6
Nuisance Chemical 1
Primary Consequence
Fatalities
Fatalities
Fatalities
Fatalities
Fatalities
Fatalities
Fatalities
Fatalities
Fatalities
Miles of Pipe Contaminated
Secondary Consequence
Illnesses
Illnesses
Illnesses
Illnesses
Illnesses
Illnesses
Illnesses
Illnesses
Illnesses
Not Applicable
3.5 Forums
Qualitative information about the design and operation of the Cincinnati CWS was obtained through a
variety of forums. These sessions provided an opportunity for front line personnel, supervisors, senior
managers and representatives from partner organizations with an opportunity to provide feedback on the
Cincinnati CWS in areas such as the value of various enhancements, potential system improvements, and
long-term plans for the CWS. Three types of forums were conducted over the evaluation period: routine
component review meetings, lessons learned workshops and exit interviews.
Routine Component Review Meetings. Routine meetings, held at a frequency of once per
month to once per quarter, were held for each component (WQM, ESM, CCS, PHS, CM and
S&A). During these meetings, recent performance data was reviewed and potential component
modifications were discussed. These review meetings were particularly important during the
optimization phase, during which recent performance data provided feedback to the team
regarding the efficacy of component modifications.
Lessons Learned Workshops. Within a few months of the transition from optimization to real-
time monitoring, a workshop was held for each component to capture lessons learned from the
Cincinnati CWS pilot and to elicit feedback regarding how these lessons learned could be
incorporated into guidance and tools. Utility personnel provided a detailed assessment of the
strengths and weaknesses of the tools, equipment and procedures used in the CWS over the
course of the pilot.
Exit Interviews. Exit interviews were held for four of the components (WQM, ESM, CCS and
PHS) at the end of the evaluation period in June 2010. The purpose of these interviews was to
gather perspectives from GCWW and its partners regarding performance and operation of the
Cincinnati CWS after one year of full ownership with real-time monitoring. These exit
interviews also provided an opportunity to discuss GCWW's plans for the CWS into the future.
Exit interviews were not conducted for CM or S&A because the last major activity for these
components was FSE 3 in October 2009, and the after action report from that exercise served as
the closeout for those two components.
16
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
3.6 Literature and Research
Some aspects of the evaluation required information from open source literature and the results of
empirical research. One of the more important aspects of system performance that was evaluated using
literature and research is the ability of the monitoring and surveillance components to detect specific
contaminants or classes of contaminants. Open source literature was reviewed and assessed by subject
matter experts to estimate minimum detection limits for CCS, PHS and S&A. Information gathered
through literature review was also used to parameterize some modules of the Cincinnati CWS model,
such as health effects (e.g., symptom thresholds, lethal doses, etc.), drinking water usage patterns, and
health seeking behaviors. Additional description of information sources used to parameterize the
Cincinnati CWS model can be found in Appendix A.
Empirical data derived from bench-scale studies were used to quantify the response of the water quality
parameters monitored by the WQM component to specific contaminants over a range of concentrations.
The bench-scale studies were performed on finished water from the two treatment plants operated by
GCWW. Fresh aliquots of finished water from each treatment plant were collected and incrementally
dosed with the contaminant under evaluation. The change in each water quality parameter was plotted as
a function of concentration for each contaminant, and empirical relationships were developed that could
be used to estimate the minimum contaminant concentration that would produce a measureable change in
one of the monitored water quality parameters (Hall, et al., 2007).
17
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 4.0: Operational Reliability
For a CWS to consistently detect extremely rare contamination incidents, it must achieve a high degree of
operational reliability, meaning that it is available and producing data of sufficient quality and quantity
for reliable event detection. To evaluate how well the CWS met this design objective, the following two
metrics were evaluated: data completeness and availability. The following subsections define each
metric, describe how they were evaluated, and present the results.
4.1 Data Completeness
Definition: The number of usable data hours generated by the CWS or one of its components expressed
as a percentage of the total number of potential data hours. For data to be considered usable, it must have
been collected and of acceptable accuracy and quality. Potential data hours are calculated as the number
of hours in a defined time period multiplied by the number of data streams under consideration (e.g., a
component with 3 data streams has 3 potential data hours in 1 hour).
Analysis Methodology: This metric was evaluated using empirical data collected from the pilot over the
evaluation period. Periods of missing or unusable data were determined from an analysis of the data
collected from each component over the evaluation period. Causes of lost data were also documented and
include QC failures, data transmission errors and faulty equipment, among others. Data completeness
was evaluated for the system as well as each of the four monitoring and surveillance components. Note
that for the PHS component, data completeness was tracked only for the 911 and emergency medical
service data streams. EpiCenter and DPIC were operational prior to implementation of the CWS, and
monitoring data completeness for these systems fell outside of the scope of the pilot. For CCS, the work
request data stream was included for all analyses conducted on data from the optimization period and
excluded for all analyses conducted on data from the real-time monitoring period after it was discontinued
in January 2009. WQM data streams include 82 sensors at 15 stations in the distribution system. ESM
data streams consist of 59 pieces of physical security equipment such as motion sensors, video cameras
and contact alarms installed at 12 critical facilities.
Results: The number of data streams, potential data hours per 30-day period, potential and actual data
hours over the evaluation period, and percentage of data completeness for each component and the entire
system is reported in Table 4-1. Overall, the CWS had 146 data streams providing 105,120 potential data
hours in a 30-day period. Data completeness for the system was 95% over the evaluation period, while
data completeness ranged from 93% to 99% for individual components. Data completeness for the
system is largely influenced by ESM and WQM, which collectively have 141 data streams compared with
three for CCS and two for PHS. Redundancy built into the CWS, such as multiple data streams per
component, provides a sustained ability to detect potential contamination or unusual water quality
conditions even when system data completeness falls below 100%.
Table 4-1. Data Completeness per Component
Component
WQM
ESM
CCS
PHS
System
Number of
Data Streams
82
59
3
2
146
Potential Data
Hours per 30
Days
59,040
42,480
2,160
1,440
105,120
Potential Data
Hours During
Evaluation
Period
1.77x1 0s
1.25x1 0s
6.35 x104
4.23 x104
3.14 x106
Actual Data
Hours During
Evaluation
Period
1.65x1 0s
1.23x1 0s
6.28 x104
4.03 x104
2.99 x106
Data Completeness
During Evaluation
Period
93%
98%
99%
95%
95%
18
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Figure 4-1 demonstrates the percentage of system data completeness for each monthly reporting period
(percentage of usable data hours relative to total potential data hours). The monthly reporting period with
the lowest data completeness occurred during the September 2008 reporting period, and was due to a
system-wide power outage lasting for several days caused by the Hurricane Ike windstorm. Aside from
the September 2008 result, there is a gradual improvement in data completeness as the CWS was
optimized leading up to the transition to real-time monitoring beginning in 2009. However, after all
components transitioned to real-time monitoring by June 2009, there was a decline in data completeness
due to continued challenges with some of the equipment, mostly attributable to the WQM component as
discussed below.
100%
CCS, ESM Real-
time Monitoring
95%
90%
D.
O
o
re
IS
Q
85%
80%
75%
70%
WQM, PHS Real-
time Monitoring
Start Date of Monthly Reporting Period
Figure 4-1. System-wide Data Completeness per Monthly Reporting Period
During the early stages of the evaluation period (between the January 2008 and February 2009 reporting
periods), data completeness for the WQM component was continuously in the 80% to 95% range, but
never reached 100%. The two major causes of incomplete data for the WQM component were equipment
maintenance and calibration activities. Additionally, there was a significant system-wide outage due to
the Hurricane Ike windstorm in September 2008, in which data was temporarily unavailable from all 15
WQM stations, resulting in a large number of lost data hours. Other system-wide outages occurred when
maintenance activities required the Water Security SCADA system to be taken off-line temporarily.
While there were several instances of system-wide outages that affected data completeness for all 15 of
the WQM stations, localized outages affecting only one station at a time were more common.
During the latter stages of the evaluation period (between the February 2009 and May 2010 reporting
periods), data completeness for the WQM component ranged between 82% and 99%, with downtime
attributed to recurring equipment maintenance issues and sensors being taken off-line due to prolonged
equipment malfunction. Continued fluctuation in percentage of data completeness for the WQM
component (even following the transition to real-time monitoring in June 2009) demonstrates the
challenge of keeping complex equipment in proper working order, even after the initial start-up issues had
been resolved.
19
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
The ESM and CCS components consistently provided 100% data completeness during most monthly
reporting periods, although there was some lost data for these two components. The ESM component
experienced some data loss due to the windstorm in September 2008 and problems with digital cellular
communication at some remote locations. The CCS component has three data streams, so when one is
down, a significant reduction in percent data completeness can result. In early November 2008, no FVR
data was available for nearly three weeks due to an unnoticed data communication issue. This resulted in
the lowest percentage of data completeness (74%) for CCS during any monthly reporting period.
During each monthly reporting period, data completeness for the PHS component was regularly in the
86% to 100% range, achieving 100% in all but two reporting periods after PHS transitioned to real-time
monitoring. Overall, the lowest value for PHS data completeness (81%) occurred during the April 2010
reporting period, due to a five day delay in data transmission.
4.2 Availability
Definition: The percentage of time that each component of the CWS is operational and maintains the
ability to detect contamination incidents. Periods in which a component is not available are termed
downtime events.
Analysis Methodology: This metric was evaluated using empirical data collected from the pilot over the
evaluation period, which documented all downtime events lasting longer than one hour. A downtime
event for the component occurs when any one of the sub-components fails to meet the availability criteria
shown in Table 4-2. The total downtime for each component over a monthly reporting period was
determined by adding the durations of the individual downtime events occurring within that period. The
hours that each component was available was calculated as the difference between the total number of
hours and the hours of downtime in a reporting period. Total downtime for each sub-component (i.e.,
data collection equipment, event detection system and alert notification) was calculated for each
component and for the system to examine the underlying cause of downtime events over the evaluation
period. Availability for the PHS component was tracked only for the 911 and emergency medical service
data streams. EpiCenter and DPIC were operational prior to implementation of the CWS, and monitoring
the availability of these systems fell outside of the scope of the pilot. However, local partners
participating in the Cincinnati pilot reported that both systems have historically demonstrated a high
degree of operational reliability.
Table 4-2. Availability Criteria for Each Monitoring and Surveillance Component
Component
WQM
ESM
CCS
PHS-911
Sub-component Availability Criteria
Data Collection/Equipment
At least 12 of the 15 monitoring
stations are transmitting either
chlorine or TOC data
At least 75% of intrusion
detection devices are
transmitting signals to SCADA
Either the IVR, work request, or
the work order system is
operational and providing data to
event detection
Operational data is transmitted
from the 91 1 server to event
detection
Event Detection System
The CANARY event
detection system is
operational
Not applicable1
Event detection system is
operational
Event detection system is
operational
Alert Notification
CANARY and SCADA
systems are operational
SCADA system is operational
Email server is operational
Email server is operational
20
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Component
PHS-
Emergency
Medical
Services
Sub-component Availability Criteria
Data Collection/Equipment
Upload of emergency medical
services run data to the database
server and an operational data
connection from the database to
event detection
Event Detection System
Event detection system is
operational
Alert Notification
Email server is operational
Enhanced security monitoring does not have event detection separate from the data provided by the intrusion
detection devices.
The criteria listed in Table 4-2 define availability of individual components in terms of their sub-
components. Similarly, the availability of the entire CWS is defined in terms of the primary monitoring
and surveillance components: WQM, ESM, CCS and PHS. The CWS is fully available if all four
monitoring and surveillance components are concurrently available, it is partially available when between
one and three of the components are concurrently available, and it is unavailable when all four of the
components are concurrently unavailable. Thus, an analysis of co-occurring downtime events among the
components was performed to characterize overall system availability.
Results: Figure 4-2 shows the total downtime for each component (bars with color coding) and the total
potential hours of operation for each monthly reporting period (gray line across the top of the figure).
This figure shows that the hours of downtime varied widely by reporting period and component.
However, it is apparent that WQM generally experienced the most downtime and CCS the least. The
specific causes of component downtime are discussed below.
3
O
O
O
Start Date of Monthly Reporting Period
Figure 4-2. Component Downtime for Each Monitoring and Surveillance Component
Figure 4-3 demonstrates the distribution of downtime by the sub-components listed in Table 4-2 for the
system and each of the four monitoring and surveillance components over the evaluation period. The
large pie chart on the left includes a summation of total downtime hours, distributed among three
common sub-components. There was a total of 6,786 hours of system downtime, of which event
detection system downtime was the largest contributor (60% of overall downtime hours). This was
predominately the result of downtime of the CANARY event detection system used by the WQM
component, as discussed in more detail below. Data collection and alert notification downtime occurred
less frequently at 29% and 11%, respectively.
21
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
The four smaller pie charts in Figure 4-3 depict the underlying causes of component downtime over the
evaluation period. The pie chart representing downtime hours for the WQM component demonstrates
that the vast majority of downtime hours, 3,710 hours (92%), were due to event detection system
downtime. The CANARY event detection system was still under development during the evaluation
period and frequent, planned updates and maintenance required it to be taken off-line. Furthermore, once
CANARY was taken off-line for even a short period, it remained non-functional for one to three days as it
collected the data to create a baseline, which was necessary to perform event detection. This resulted in
additional downtime hours until it returned to full operational status. The remaining 8% of downtime was
the result of outages of equipment at water monitoring stations, most often sensors or communication
systems.
System
n = 6,786 (hrs)
Event Detection System
4,097 (hrs)
60%
PHS
n = 2,005 (hrs)
Data Collection
Event Detection
System
Alert Notification
324 (hrs)
WQM
n = 4,034 (hrs)
Data Collection
Event Detection
System
233 (hrs)
36%
ESM
n = 656 (hrs)
Data Collection
Alert Notification
CCS
n=91 (hrs)
Data Collection
Event Detection
System
Figure 4-3. Downtime Attributed to Sub-components for Each Monitoring and Surveillance
Component
The pie chart in Figure 4-3 representing the PHS component shows a total of 2,005 downtime hours over
the evaluation period, which is the second highest number of downtime hours after the WQM component.
In contrast to the WQM component, the event detection systems implemented for the PHS component
were fully developed software products at the time of implementation. Thus, there were not nearly as
many downtime hours involving the event detection system sub-component. The majority of downtime
hours (1,400 hours or 70%), were due to problems with the data collection sub-component, specifically
network instability concurrent with database unavailability and one instance of an extended period of data
communication failure.
22
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
As described in Table 4-2, only two sub-components are applicable to ESM, data collection and alert
notification. The corresponding pie chart in Figure 4-3 shows that ESM experienced 656 downtime hours
attributable to alert notification (423 hours or 64%) and data collection (233 hours or 36%). The
predominant cause of alert notification downtime was the result of planned maintenance and updates to
the SCADA system. Data collection downtime occurred due to intermittent issues with the digital
cellular communications at remote ESM stations.
CCS proved to be the most reliable component with only 91 downtime hours over the evaluation period.
This downtime was attributable to issues with either the event detection system (84.5 hours or 93%) data
collection (6.5 hours or 7%). This was the result of two events, one required the event detection system to
be shut down for a weekend and another that resulted from a power outage that interrupted all CCS data
streams.
Table 4-3 summarizes the availability of each component over the evaluation period, which was
calculated as a percentage of the total hours in which the components were operational according to the
availability criteria described in Table 4-2. During the evaluation period CCS had the highest availability
(>99%), as the only major event, which resulted in downtime was an unexpected problem with the event
detection system, which required the system to be taken off-line for a weekend. ESM also experienced
minimal downtime events, which resulted in 97% availability. PHS experienced more issues, particularly
with the data collection sub-component, resulting in an availability of 90% over the evaluation period.
WQM had the lowest availability (73%), which was largely attributable to issues with the CANARY
event detection system, as discussed previously.
Following the conclusion of the evaluation period, the CANARY event detection system was updated to
allow the buffer to be filled with recent water quality data captured by the monitoring stations and stored
in a database. As a result of this enhancement, the event detection system could immediately recover
from a downtime event as long as the data collection sub-component remains operational. To estimate
the potential availability of the WQM component if the new version of CANARY had been in operation
during the entire evaluation period, WQM availability was recalculated excluding CANARY downtime
events. In Table 4-3, the value in parenthesis (89%) represents WQM component availability when
downtime caused by the CANARY event detection system was excluded, which is comparable to the
availability of PHS.
Table 4-3. CWS Component Availability
Component
WQM
ESM
CCS
PHS
Percent Availability
73% (89%)1
97%
>99%
90%
The value in parenthesis (89%) represents WQM component
availability when downtime caused by the CANARY event
detection system was excluded.
The inherent redundancy built into a multi-component CWS allows it to maintain protection when
individual components of the system are temporarily unavailable. However, the loss of any component
can impact other design objectives such as contaminant coverage, spatial coverage and timeliness of
detection. Availability of the entire CWS was evaluated in terms of the availability of its four monitoring
and surveillance components.
23
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
The percentage of time when one, two, three or four components were concurrently available is
represented in Table 4-4. The CWS was fully available 78% of the time (i.e., all four monitoring and
surveillance components were concurrently available). The concurrence of multi-component downtime
events was infrequent, and at least three of the four components were available more than 99% of the time
(i.e., system was partially available for 99% of the evaluation period). The CWS was never completely
unavailable during the evaluation period.
The high degree of system availability is largely due to the resilient design of the CWS and its
components. The CCS and PHS components utilize stable computer algorithms and analyze highly
reliable data streams. ESM uses rugged equipment with a demonstrated capability to operate
continuously for extended periods. WQM is more prone to downtime events due to the complexity of the
monitoring equipment; however, this is compensated by the fact that the component monitors many
independent data streams that provide resiliency. Finally, the system as a whole is equipped with back-up
power supply, servers, and other equipment to provide an additional level of reliability.
Table 4-4. Concurrent CWS Component Availability
Number of Components
Available
1
2
3
4
Percent Availability1
100%
99.9%
99.3%
78.0%
Percentages are calculated relative to 21,160
hours in the evaluation period.
The only event during the evaluation period that had the ability to create a significant period of multi-
component downtime was the Hurricane Ike windstorm, a rare event that left 90% of GCWW's service
area without electricity for several days and exhausted uninterruptible power supply built into the system.
Even with the temporary loss of some electronic equipment due to the power failure, the CWS provided
monitoring and response capabilities that were used to mitigate the impacts of the windstorm on utility
operations.
The longest period that two and three components were simultaneously unavailable was 26 and 8 hours,
respectively. To evaluate whether these period represent significant gaps in detection capabilities, these
values were compared with simulation results. The distribution system model was used to estimate the
amount of time that contaminated water remains in the distribution system for the 30 simulated
contamination scenarios used in the benefit-cost analysis described in Section 3.4. The average time that
contaminated water remains in the distribution system for these scenarios was determined to be 5.3 days,
which is substantially longer than the 26 hours that two components were concurrently unavailable.
Thus, while these periods of multi-component downtime may impact the timeliness of detection, it is
unlikely that the overall ability of the CWS to detect contamination would be impacted.
4.3 Summary
As a result of the layers of redundancy designed into the multi-component CWS, the system demonstrated
sustained, continuous detection capabilities, even when components or sub-components were temporarily
unavailable. Overall, the optimized CWS demonstrated a high degree of operational reliability and a
sustained ability to detect contamination incidents.
24
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Data completeness was 95% for the CWS over the entire evaluation period. Issues with WQM equipment
during the early stages of deployment contributed significantly to lost data, while ESM, CCS and PHS
had intermittent periods of data loss. However, after the systems were optimized by June 2009, data
completeness for the system regularly exceeded 95%.
On average, the individual CWS components were available between 73% and >99% of the time. The
single greatest contributor to downtime was the WQM event detection system (CANARY), especially
during the early portion of the evaluation period. After issues with CANARY were resolved, the
availability of the WQM component and the entire CWS increased. Had CANARY been fully
operational during the entire evaluation period, the WQM component would have been available for 89%
of the time, rather than the 73% availability observed during the evaluation period. The CCS component
had the highest availability at >99%, followed closely by ESM at 97%. The PHS tools deployed
specifically for this project, 911 and emergency medical service surveillance, were available 90% of the
time, whereas the PHS tools that were in place prior to the pilot were mature systems that were available
>99%ofthetime.
Overall, downtime of multiple components was rare. Three of the four components were available >99%
of the time. The longest periods of multi-component downtime was 26 hours for two components and 8
hours for three components, which was well below estimates of the time that contaminated water would
remain in the system, and thus potentially be detected. This indicates that even with multiple components
unavailable for a period, it is still likely that a significant contamination incident will be detected by the
CWS.
25
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 5.0: Spatial Coverage
Given the large number of access points in a drinking water distribution system and the uncertainty
regarding where a contaminant might be injected, the Cincinnati CWS was designed to cover the entire
distribution system. Specifically, monitoring and surveillance components were selected and designed to
provide redundant coverage throughout the distribution system. To evaluate how well the CWS met this
design objective, the following metric was evaluated: area and population coverage. The following
subsections define this metric, describe how it was evaluated, and present the results.
5.1 Area and Population Coverage
Definition: Area coverage is defined as the percentage of the distribution system area that is monitored
and protected by the integrated CWS. A portion of the distribution system is monitored by the CWS if a
contaminant injected in that area would flow past a point where it could be detected by any of the
monitoring and surveillance components. An area is protected by the CWS if it is downstream of a point
of potential detection. Population coverage is defined as the number of people that reside within the area
covered by the CWS, expressed as a percentage of the total population in the study area.
Analysis Methodology: Area coverage was determined for each of the four monitoring and surveillance
components using a variety of techniques as described below:
The area covered by the ESM component was evaluated using the CWS simulation model to
estimate the area and population that would be impacted by an uninterrupted contaminant
injection at ESM locations.
The area covered by the WQM component was evaluated using GCWW's distribution system
model to simulate contaminant injections throughout the distribution system and determine which
WQM locations observed a potentially detectable concentration.
The area covered by the CCS component was determined from the design of the component and
an analysis of availability of telephone service in the GCWW service area.
The area covered by the PHS component was determined from the design of the component (e.g.,
regions of the distribution system monitored by the various surveillance tools) and was further
evaluated using the location data from PHS alerts that occurred during the evaluation period.
The area covered by the S&A component was determined from the design of the component.
The results from these component assessments of spatial coverage were used to characterize the area
covered by the integrated CWS, including an analysis of portions of the distribution system covered by
multiple components.
In addition, the results of the simulation study were used to evaluate detection of simulated contamination
scenarios originating from locations throughout the distribution system. This analysis provides another
perspective on area coverage by demonstrating the ability of the system to detect contamination from
geographically distributed sources.
The results from the analysis of area coverage were converted to population estimates using 2000 census
block data. Specifically, the population that lives in the area covered by the CWS is considered to be
covered by the system. Interpolation was used in cases where a census block was only partially in the
area covered by the CWS. Note that while data from the 2010 census was available at the time this report
26
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
was developed, data from the 2000 census was used because the distribution system model used for these
analyses better represents that time period.
These analyses are limited to the portion of GCWW's retail area in the City of Cincinnati and surrounding
communities in Hamilton County, a region that covers 294 square miles and serves approximately
760,000 people. GCWW sells treated water to large wholesale customers outside of Cincinnati, but these
areas were not considered in this analysis. While this analysis was constrained to the study region, it is
worth noting that some components of the CWS provide coverage for customers outside of the study
region. For example, CCS and the DPIC and EpiCenter PHS data streams extend through most of
GCWW's retail and wholesale areas.
Results: Table 5-1 shows the area and population coverage for each component as well as the entire
system. WQM covered 72% (244 square miles) of the study area and 84% of the population with only 15
monitoring locations. ESM is designed to monitor for intrusions that may lead to contamination at a
limited number of sites in the distribution system. But even with this limited number of sites, the results
from simulated contamination incidents show that the ESM sites supplied water to 96% of the study area
and 99% of the population within the study area. Thus, while the area monitored by ESM is small, the
area protected by ESM encompasses most of the study area. CCS is potentially capable of monitoring the
entire study area if a customer is able to contact the utility. Empirical data suggests that 96% of GCWW
customers have access to a telephone, and this result was used as an estimate of the population coverage
for CCS. The 911 and emergency medical service data streams cover the portion of the study area in the
boundaries of the City of Cincinnati, while DPIC and EpiCenter cover the entire study area and associated
population. Theoretical coverage for S&A was also 100%, because samples can be collected at any
location within the distribution system in response to an alert. The results of this analysis demonstrate
that all components of the CWS provide robust spatial coverage throughout the distribution system.
Table 5-1. Area and Population Coverage
Component
WQM
ESM
CCS
PHS
S&A
System
Population Coverage
84%
99%
96%
100%
100%
100%
Area Coverage
244 square miles
24 sites
294 square miles
294 square miles
294 square miles
294 square miles
In addition to the analysis of theoretical spatial coverage presented above, the results from the simulation
study were analyzed to estimate the ability of the Cincinnati CWS to detect simulated contamination
incidents originating from locations throughout the distribution system. This analysis was based on 94
pito zones, which are areas of relatively constant water quality and pressure within the distribution
system. During the study, contaminant injection was simulated for the 17 contaminants listed in Table 3-
2 in each of the 94 pito zones, resulting in 1,590 scenarios (eight scenarios in which no consequences
were generated were eliminated during the study design).
Table 5-2 summarizes the detection percentages by the pito zone in which contaminant injection
occurred. The first column shows the detection percentage and the second column displays the number of
pito zones, which had the corresponding detection percentage. For example, 100% of all injections
occurring in 51 pito zones were detected by the CWS. The remaining three columns show the total,
median, and range of populations in the group of pito zones with the corresponding detection percentage.
Considering the population range, it is apparent that the two groups of pito zones with the highest
27
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
detection percentages (89 of 94 pito zones) include some of the most populous areas in the distribution
system. This group of 89 pito zones with detection percentages greater than 94% contains 91.6% of the
population.
Table 5-2. Number of Pito Zones with Corresponding Detection Percentages
Detection
Percentage
100%
94.1%
93.8%
93.3%
88.2%
No. of
Pito Zones
51
38
3
1
1
Population in Pito Zones
Total
429,955 (56.6%)
265,314(35%)
34,674 (4.6%)
1,085(0.1%)
16,440(2.2%)
Median
7,701
5,841
13,412
1,085
16,440
Range
1,561 to 24,362
504 to 31, 055
6,725 to 14,537
1,085 to 1,085
16,440 to 16,440
The CWS successfully detected 1,546 out of the 1,590 scenarios, for an overall detection percentage of
97%. The 44 scenarios that went undetected by the CWS involved injections spread over 43 pito zones
indicating that there was no specific area of the distribution system that was poorly covered by the CWS.
Thirty-eight of the undetected scenarios involved Nuisance Chemical 2, which can be detected only by
one water quality parameter monitored by the WQM component. Six of the undetected scenarios
involved Biological Agent 6 and Biological Agent 7. While these two contaminants can be detected by
both WQM and PHS, contaminated water did not flow through any of the WQM stations for the six
undetected scenarios. Furthermore, the scenarios produced relatively few health consequences, making
them difficult to detect through PHS. Thus, all 44 of the scenarios that were not detected produced
relatively few consequences.
5.2 Summary
Through a multi-component design, the Cincinnati CWS achieved broad spatial coverage of the study
area, which includes the most populous region of GCWW's service area and covers 294 square miles.
Area coverage ranged from 82% for WQM to 100% for CCS, PHS and S&A. Population coverage was
greater than area coverage, ranging from 84% for WQM to 100% for PHS and S&A.
Results from the simulation study were evaluated to determine the number of contamination scenarios
originating from each of the 94 pito zone that were detected by the CWS. This analysis showed that
100% of the scenarios originating from 51 pito zones and 94.1% of scenarios originating from another 38
pito zones were detected by the CWS. The 44 scenarios (3% of the total number of scenarios) that were
not detected by the CWS were spread across 43 pito zones, indicating that there is no specific area of the
distribution system that is not effectively covered by the CWS. The undetected scenarios produced few
consequences relative to the rest of the ensemble, which is the primary reason they were difficult to
detect.
28
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 6.0: Contaminant Coverage
Given the large number of potentially harmful drinking water contaminants and the uncertainty regarding
which contaminant might be involved during a specific incident, the Cincinnati CWS was designed to
provide redundant coverage for a broad range of contaminants. To evaluate how well the Cincinnati
CWS met this design objective, the following two metrics were evaluated: contaminant detection
threshold and contaminant scenario coverage. The following subsections define each metric, describe
how it was evaluated and present the results.
6.1 Contaminant Detection Threshold
Definition: The lowest concentration of a specific contaminant that can be reliably detected by the CWS.
Analysis Methodology: Contaminant detection threshold was assessed for each of the 17 contaminants
evaluated under the simulation study. This metric could not be directly assessed from the Cincinnati
CWS pilot due to the lack of contamination incidents. Instead, contaminant detection thresholds were
estimated based on the design of the components in conjunction with results from research and studies
available in the open literature. The following methods were used to estimate the detection thresholds for
individual components:
ESM was designed to detect intrusions that could provide access to distributed drinking water
rather than indicators of contamination, thus detection thresholds do not apply to this component.
For WQM, the detection threshold was estimated from bench-scale laboratory studies that
measured the change in water quality parameter values at known contaminant concentrations.
For some of the biological agents, the detection threshold was estimated for the co-contaminant
that would be necessary to maintain viability of the biological agent in chlorinated water.
For CCS, the detection threshold was estimated from taste and odor threshold reported in the
literature. These odor and taste thresholds represent a sample of the population, and the actual
detection thresholds for an individual can vary widely from these estimates. Furthermore, some
individuals may not be able to detect the contaminant at any concentration through the senses.
For PHS, the detection threshold was estimated as the minimum contaminant concentration that
could produce a dose that would result in acute symptoms in the exposed population. Similar to
taste and odor thresholds, the dose resulting in onset of acute symptoms varies widely for
individuals.
For S&A, the detection threshold was estimated as the minimum reporting limit for the analytical
method used in the Cincinnati CWS pilot.
Detection thresholds were assessed relative to contaminant-specific critical concentrations that are based
on adverse consequences to the exposed population or utility infrastructure. The critical concentration
provides a useful benchmark against which to assess whether the detection threshold is low enough to
detect contamination that could result in substantial public health or infrastructure consequences. Each
contaminant was grouped into the categories described in Section 3.3, which determined how the critical
concentration was determined:
Nuisance Chemical. The critical concentration for nuisance chemicals was selected at levels that
would make the water unacceptable to customers (e.g., concentrations that result in objectionable
aesthetic characteristics).
29
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Toxic Chemical. For chemical contaminants that are lethal to individuals exposed to a high
dose, the critical concentration was based on the mass of contaminant that a 70 kg adult would
need to consume in one liter of water to have a 10% probability of dying (LDi0).
Biological Agent. For biological agents that are infectious or lethal to individuals exposed to a
high dose, the critical concentration was based on the mass of contaminant that a 70 kg adult
would need to consume in one liter of water to have a 10% probability of dying (LD10).
To determine whether the detection threshold was sufficient to detect water contaminated at
concentrations equal to or greater than the critical concentration, the ratio of the critical concentration to
detection threshold was calculated.
Results: Table 6-1 presents the ratio of critical concentration to detection threshold for each contaminant
across the components. A ratio of 1.0 or greater indicates that the component can detect the contaminant
at or below the critical concentration. Large ratios demonstrate the contaminants that can be detected at
concentrations significantly lower than the critical concentration. Conversely, ratios less than 1.0 indicate
that the component would not detect the contaminant until the concentration has exceeded the critical
concentration that would result in adverse public health or infrastructure consequences.
Table 6-1. Ratio of Critical Concentration to Detection Threshold
Contaminant
Nuisance Chemical 1
Nuisance Chemical 2
Toxic Chemical 1
Toxic Chemical 2
Toxic Chemical 3
Toxic Chemical 4
Toxic Chemical 5
Toxic Chemical 6
Toxic Chemical 7
Toxic Chemical 8
Biological Agent 1
Biological Agent 2
Biological Agents
Biological Agent 4
Biological Agents
Biological Agent 6
Biological Agent 7
WQM
4.76
33.3
225
463
185
104
57.6
352
1.97
0.0333
265
1,310
2.40
3.57
7.87
9.70
0.582
CCS
20.0
-
5.86
50.5
22.8
4.03
-
-
-
-
88.2
-
-
-
-
-
-
PHS
-
-
458
3,640
1,640
290
668
850
950
300
4,500
3,940
2.40 x 104
4.54
10.0
1.74
1.64
S&A
2.00 x 104
2.00 x 104
1,470
3.39 x 104
3.69 x 10s
5.80 x 104
6,680
4.08 x 104
57.0
6.60 x 107
2.25 x 104
4.93 x 105
24.0
90.7
20.0
5.79 x 104
3.30 x 105
This table shows that WQM can theoretically detect all 17 contaminants, and 15 of them at concentrations
below the critical concentration. The ratio of critical concentration to detection threshold is below 1.0 for
Toxic Chemical 8 and Biological Agent 7, which indicates that adverse health consequences might occur
prior to detection. Note that for some of the biological agents, a co-contaminant is needed to maintain the
viability of the agent. These co-contaminants produce a significant change in water quality, and thus are
responsible for the detection capabilities of WQM for these biological agents. If a co-contaminant were
not at concentrations high enough to be detected by WQM, then the biological agent would be
inactivated, and thus of no concern to public health. CCS can theoretically detect six of the 17
30
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
contaminants at ratios greater than 1.0. PHS can theoretically detect 15 of the 17 at ratios greater than
1.0, but it would not be expected to pick up the nuisance chemicals as they are not toxic.
Comparing across the three monitoring and surveillance components (WQM, CCS and PHS), the ratios
are generally larger for PHS in comparison to WQM and CCS, which indicates that PHS can detect
concentrations of most contaminants well below the critical concentration. This is because most
contaminants produce symptoms in exposed individuals at concentrations much lower than the LD10,
which in turn would generate a PHS alert.
All of the contaminants are theoretically detectable by S&A with very high ratios for most of the
contaminants, indicating the ability of this component to detect contaminant concentrations several orders
of magnitude lower than the critical concentrations. This result indicates that as long as sampling is
initiated soon after the initial detection occurs at a location hydraulically connected to the location of an
alert, it is likely that the contaminant will be present in the sample at detectable concentrations.
As indicated in Table 6-1, five of the 17 contaminants (Toxic Chemicals 1 through 4 and Biological
Agent 1) are detectable by all three of the monitoring and surveillance components (WQM, CCS, and
PHS), providing redundant detection capabilities. Furthermore, the ratio of critical concentration to
detection threshold is above 1.0 for these five contaminants across all three components, demonstrating
reliable detection capabilities.
Eleven contaminants are detectible by two of the monitoring and surveillance components (either WQM
and CCS or WQM and PHS), with detection ratios above 1.0 for all except Toxic Chemical 8 and
Biological Agent 7 for WQM. Only one contaminant (Nuisance Chemical 2) is theoretically detectable
by just one monitoring and surveillance component, WQM. However, the detection threshold for this
contaminant is 33 times lower than the critical concentration. Thus, if Nuisance Chemical 2 is in the area
covered by WQM at concentrations above the critical concentration, it will likely be detected.
6.2 Contamination Scenario Coverage
Definition: The number or percentage of simulated contamination scenarios that generate an alert from
at least one component.
Analysis Methodology: This metric could not be directly assessed from the Cincinnati CWS pilot due to
the lack of contamination incidents during the evaluation period. Instead, the results from the simulation
study were used to characterize contamination scenario coverage. As discussed in the previous section,
each of the 17 contaminants simulated in this study is detectable by at least one component; thus, all
2,015 scenarios evaluated in the study are potentially detectable by the CWS. Results were aggregated by
contaminant type and by detection status to determine the relative detection rates for different
contaminants.
Results: The results of the simulation study demonstrate that the CWS successfully detected 1,971 out of
the 2,015 scenarios, a 98% detection rate. Figure 6-1 shows the counts and percentages of scenarios
detected by the CWS for each of the 17 contaminants in the simulation study.
31
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
^
o
1
3
Q
Total Detection: (1,971/2,015)
Figure 6-1. Scenarios Detected by Contaminant
The 44 scenarios that went undetected by the CWS involved injections spread over 43 pito zones,
indicating that they were not spatially clustered. Of the undetected scenarios, 86% (38) involved
Nuisance Chemical 2, which can be initially detected only by WQM, a component with spatial coverage
limited to 244 square miles of the 294 square miles in the study area. Given the near perfect detection of
the other 16 contaminants, the underperformance of the CWS for Nuisance Chemical 2 demonstrates the
value of multiple surveillance components. The remaining 14% (6) of the undetected scenarios involved
Biological Agents 6 and 7, which are theoretically detectable by both WQM and PHS. Overall, WQM
did not detect many of the scenarios involving these two biological agents (less than 11%); however, PHS
was able to detect most scenarios that were not detected by WQM. Therefore, the six scenarios involving
Biological Agents 6 and 7 that were not detected by either WQM or PHS can be explained by three
factors: no appreciable change in water quality conditions, which prevented detection by WQM
(applicable to all six undetected scenarios), the absence of any significant health consequences (applicable
to five of the undetected scenarios), and one instance where significant symptomatic cases and deaths
would have occurred, but were spread out in time making detection by PHS difficult.
6.3 Summary
The design of the Cincinnati CWS ensures the system has robust detection capabilities for a variety of
contaminants, including nuisance chemicals, toxic chemicals, and biological agents. All 17 contaminants
evaluated in the simulation study were theoretically detectable by at least one monitoring and surveillance
component (WQM, CCS or PHS) at a concentration equal to or less than the critical concentrations
necessary to cause significant public health consequences (lethal to 10% of the exposed population) or
infrastructure consequences (would require distribution system remediation). Eleven contaminants were
detectable by two components, and five were detectable by all three components. ESM was not
considered in this analysis because its detection capabilities are independent of contaminant type. With
32
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
respect to response capabilities, S&A can detect all 17 contaminants at detection limits well below the
critical concentration.
The Cincinnati CWS detected 98% of simulated contamination incidents from an ensemble of 2,015
scenarios involving 17 contaminants and injected at locations throughout the entire distribution system,
including locations at utility facilities. The majority of the 44 scenarios that went undetected involved a
contaminant that does not cause acute health effects and is detectable by only a single component. This
result emphasizes the value of a multi-component CWS, in which the detection capabilities of the
monitoring and surveillance components are complementary and provide broad contaminant coverage.
As expected, it was observed that small, isolated contamination incidents that produced limited
consequences were more difficult to detect than incidents producing widespread consequences.
33
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 7.0: Alert Occurrence
Ideally a CWS would generate an alert only when a contamination incident, public health incident, or
other significant water quality anomaly is occurring. However, invalid alerts do occur. The goal of this
design objective is to minimize the number of invalid alerts without compromising the ability of the
system to detect real water quality anomalies or public health incidents. In the Cincinnati CWS, alert
occurrence was optimized by improving the quality of the underlying data analyzed by each component
and optimizing the event detection system configuration used by the components. To evaluate how well
the CWS met this design objective, the following three metrics were evaluated: invalid alert occurrence,
valid alert occurrence and alert co-occurrence. The following subsections define each metric, describe
how it was evaluated and present the results.
7.1 Invalid Alert Occurrence
Definition: The occurrence of alerts with a cause other than a verified water quality anomaly,
contamination incident or public health incident.
Analysis Methodology: During the evaluation period of the Cincinnati CWS pilot, alerts were generated
in real time for each of the four monitoring and surveillance components (WQM, ESM, CCS and PHS),
and recorded in a data management system. Personnel from GCWW or one of the public health partners
reviewed the alerts to determine a probable cause and designate the alert as valid or invalid. In most
cases, the results of the alert investigations were recorded on a checklist and ultimately uploaded to a
database for further analysis. These data were used to characterize the rate and cause of invalid alerts
generated by each component over the evaluation period.
In the case of WQM, alert occurrence was evaluated using the "reprocessed alerts," which were generated
by running the water quality data generated by the sensors during the evaluation period through an
updated (bug-free) version of CANARY.
The total number of invalid alerts is equal to the number of total alerts minus the number of valid alerts
observed during the evaluation period. Invalid alerts were categorized by one of the four general causes
described below:
Equipment faults. Alerts caused by equipment that is not functioning properly. Equipment
faults can directly generate invalid alerts (e.g., an improperly configured motion sensor), or can
produce inaccurate data that subsequently generates invalid alerts (e.g., malfunctioning water
quality sensors).
Procedural errors. Alerts caused by deviations from standard operating procedures, such as
miscoding data or propping open alarmed doors at secure utility locations. Invalid alerts resulting
from procedural errors can be reduced with additional staff training.
Background variability. Alerts caused by typical variations in a data stream monitored by a
component of the CWS. Invalid alerts due to background variability can be minimized but not
eliminated entirely.
Other. Alerts due to a cause other than equipment faults, procedural errors, or background
variability. The actual cause of an alert categorized as "other" may be known or unknown.
Results: Each of the components experienced periods of downtime during the evaluation period. In
particular, the WQM and PHS components experienced significant downtime early in the evaluation
period. This results in an artificially low alert rate because no alerts occurred during periods of
34
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
downtime. To correct for downtime and allow for an equivalent, cross-component comparison, alert rates
for all components were normalized for downtime. Normalization was achieved by dividing the number
of alerts for that period by the percent availability of the component during the reporting period (i.e., if a
component availability was 100%, no adjustment is made to the alert rate). These normalized alert rates
are shown in Figure 7-1.
300 n
250
200
150
100
Start Date of Monthly Reporting Period
Figure 7-1. Invalid Alerts by Component per Reporting Period (Normalized for Downtime)
During the first year of the evaluation period, alert occurrence was typically above 150 per reporting
period. While the vast majority of alerts were determined to be invalid, there were 12 valid WQM alerts
and 7 valid CCS alerts during the first year, although none were due to water contamination. The number
of CWS alerts gradually decreased as the evaluation period progressed, largely due to successful efforts to
optimize performance.
The majority of invalid alerts were due to equipment problems. The first step in reducing the number of
invalid alerts was to improve the performance of equipment and improve the quality (i.e., accuracy and
precision) of the underlying data. Next, procedural errors that generated invalid alerts were reduced
through staff training and practice with the alert investigation procedures. After equipment and
procedural issues had been largely resolved, the next step in the optimization process was to adjust the
configuration of automated event detection systems (e.g., alerting thresholds, precision settings, etc.).
These adjustments could be made only after sufficient baseline data had been collected, which provided
information about the variability that could be expected under normal conditions. The event detection
systems were configured to maintain detection capabilities without producing an excess number of invalid
alerts. This optimization process was largely completed by the December 2008 reporting period, and the
resulting reduction in invalid alert occurrence is evident in Figure 7-1.
The number of sources monitored by a component can impact the number of alerts, and in general, the
more sources that are monitored, the greater the number of alerts that are generated. Table 7-1 shows the
number of sources monitored by each component along with the total number of invalid alerts that
occurred during the evaluation period. To account for the effect of the number of sources on alert
occurrence, the number of alerts was divided by the number of sources to normalize the alert rate. For
CCS, the number of sources is equivalent to the number of data streams monitored by the component.
35
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
For WQM and ESM, the number of sources is defined by the number of monitoring locations, 15 and 12,
respectively. While ESM and WQM had the highest number of total alerts over the evaluation period,
they had the lowest normalized alert rates.
Table 7-1. Number of Total Alerts and Normalized Alerts for the CWS and each Component
Component
WQM
ESM
CCS
PHS
System
Number of Sources
Monitored by the
Component
15
12
3
3
33
Total Number of Invalid
Alerts During the
Evaluation Period
770
1,579
466
602
3,417
Number of Invalid Alerts
Normalized by Number of
Sources
51
132
155
201
104
Throughout the evaluation period, the causes of invalid alerts were tracked for each component. All but
1% of invalid alerts were caused by procedural issues, equipment problems, or background variability as
represented in Figure 7-2.
System
n = 3,417
Background Variability
1386
41%
PHS
n = 602
Background Variability
WQM
n = 770
Procedural
Equipment
Background Variability
ESM
n = 1,579
Procedural
Equipment
Other
CCS
n = 466
Procedural
Background Variability
Other
Figure 7-2. Causes of Invalid Alerts for the CWS and each Component
36
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
The reprocessed WQM data produced fewer alerts than the real-time alerts, because many alerts in the
real-time dataset were caused by problems with CANARY (USEPA, 2014b). In the reprocessed dataset,
equipment issues caused by system-wide and station outages were the most common causes of invalid
alerts. Additionally, some water quality sensors suffered from chronic performance issues, which
contributed to invalid alerts until the sensors were taken off-line. Background variability was the second
highest cause of invalid alerts. Procedural issues were not a significant contributor to invalid alerts for
WQM.
Equipment issues, most commonly associated with communications equipment, were the most significant
cause of invalid ESM alerts. Interference with the radio transmissions used to communicate ESM alerts
to the GCWW SCADA system caused ESM alerts and was the greatest contributor to this category of
invalid alerts. After the issue was resolved, the number of equipment/communication-related alerts was
reduced. Another change to the component during the same period was the switch from ladder motion
sensors to physical hatch barriers at all indoor elevated storage tank locations. There were also a few
invalid alerts caused by motion sensors exposed to the elements (such as wind causing outdoor motion
sensors to trigger), even though no intruders were attempting to enter these sites. Many procedural
invalid alerts were caused by employees not informing security when entering an ESM site and by doors
being propped open when contractors were working on-site.
Most invalid CCS alerts are attributed to background variability caused by normal variation in call
volume. During the April 2008 reporting period, alert thresholds were increased to reduce the number of
alerts caused by normal fluctuations. Following these changes, the average number of CCS alerts per
reporting period was reduced to a level deemed acceptable by utility personnel responsible for performing
CCS alert investigations.
For the PHS component, all invalid alerts for the 911, EMS, and DPIC data streams were attributed to
background variability in call volume. Equipment associated with the component proved reliable, and as
a result no equipment-related alerts occurred during the evaluation period.
The invalid alert rates for all four monitoring and surveillance components decreased from the
optimization phase to the real-time monitoring phase. Table 7-2 demonstrates this trend through the
average number of invalid alerts per reporting period during each of these phases.
Table 7-2. Invalid Alerts per Reporting Period During Optimization
and Real-time Monitoring
Component
WQM
ESM
CCS
PHS
System
Average Number of Invalid Alerts per Reporting Period
Optimization
33
82
17
25
152
Real-time Monitoring
17
23
14
15
69
The ESM component produced the most alerts during each of these phases, but also showed the greatest
percentage reduction in invalid alert rates during the transition from the optimization to the real-time
monitoring phase. This resulted in comparable alert rates among the four components during real-time
monitoring. This reduction in alerts was achieved through an intensive effort to correct problems with
equipment, IT systems, and O&M procedures during the first year of operation. Following system
37
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
optimization, the time and resources required to maintain this level of alert occurrence was substantially
lower compared with that required to optimize the system.
7.2 Valid Alert Occurrence
Definition: The occurrence of alerts caused by verified water quality anomalies, contamination incidents,
or public health incidents.
Analysis Methodology: Occurrence of valid alerts was evaluated using two different data sources:
simulation study results and empirical data.
As described in Section 3.3, the simulation study challenged a computer model of the Cincinnati CWS
with simulated contamination incidents. Furthermore, the baseline data used for each component in the
simulation study was screened to ensure that it would not generate invalid alerts, which would have
confounded interpretation of the study results. Thus, when a component generated an alert during the
simulation study, it was by design a valid alert. The results from this study were evaluated to determine
the frequency at which each of the CWS components generated a valid alert during a contamination
scenario, and the frequency that each component generated the first alert during a scenario. Many of the
results presented in this section are based on analyses that were limited to either distribution system attack
scenarios or facility attack scenarios. If a specific scenario type or subset is not specified, the results are
from an analysis performed on the entire ensemble of scenarios. This distinction is important because
ESM can detect only facility attack scenarios, but will typically be the first to detect this type of scenario.
The empirical data was collected from the Cincinnati CWS pilot during the evaluation period, as
described in Section 7.1. While valid alerts were rarer than invalid alerts, three of the four monitoring
and surveillance components did generate a few valid alerts, which were evaluated under this metric.
Results: A summary of the number of scenarios theoretically detectable and the number of scenarios
actually detected during the simulation study is presented in Table 7-3. Whether a scenario is
theoretically detectable or not depends on the contaminant used in the scenario. The assumptions
regarding which contaminants are detectable by each component is summarized in Table 3-2. Table 7-3
shows the number of scenarios that were detected and those that were first detected by each component
for the full ensemble and for the distribution system attack scenarios only. The detection results for each
component are discussed below.
Table 7-3. Simulated Contamination Scenarios Detected by each Component
Component
Scenarios Theoretically Detectable
Scenarios Actually Detected (full ensemble)
First Detected (full ensemble)
Scenarios Detected (distribution system
attack scenarios only)
First Detected (distribution system attack
scenarios only)
ESM
425
425
425
-
-
WQM
2,015
649
257
458
257
CCS
714
693
547
564
547
PHS
1,777
1,706
742
1,396
742
SC
2,015
1,666
-
1271
-
LA
2,015
1,729
-
1,307
-
Detection of Simulated Contamination Scenarios by Component
Figure 7-3 shows the detection percentage for each component, evaluated relative to the number of
scenarios each component could theoretically detect, as shown in Table 7-3, but also relative to the total
number of scenarios in the ensemble (2,015). The former analysis provides an indication of how well the
component performed relative to its intrinsic detection capabilities. Ideally, a component would detect
38
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
100% of the scenarios that are theoretically detectable. The detection percentage calculated relative to the
full ensemble provides an indication of component performance relative to a broad spectrum of
contamination threats; however, this metric is heavily influenced by the design of the scenarios included
in the ensemble. For example, if the ensemble included more scenarios using contaminants that have
aesthetic characteristics, then CCS would have had a higher detection rate relative to the full ensemble.
Theoretically Detectable*
HFull Ensemble (2,015)
ESM
WQM
CCS PHS
Component
LA
Figure 7-3. Detection Percentage of Simulation Scenarios by Component
* Total count varies by component; see Table 7-3 for details.
All facility attack scenarios (425) were detected by ESM yielding a detection rate of 100%. At the
ensemble level, ESM had a detection rate of 21%, reflecting its ability to detect injections only at
facilities with security monitoring. CCS can detect only the six contaminants in the ensemble that change
the aesthetic character of the water, but detected 97% (693) of the 714 scenarios that were theoretically
detectable by CCS. All of the scenarios that were theoretically detectable but not actually detected by
CCS were facility attack scenarios in which early detection by ESM, followed by isolation of the
contaminated facility, resulted in few exposed customers and thus few calls to the utility. At the
ensemble level, CCS detected 34% (693) of the 2,015 scenarios. PHS detected 96% (1,706) of the 1,777
scenarios that could have been theoretically detected (all scenarios involving 15 of the 17 contaminants
that produce health consequences). At the ensemble level, PHS detected 85% (1,706) of the 2,015
scenarios. The theoretically detectable scenarios that PHS failed to detect involved contaminants that
pose a risk to public health primarily through inhalation. For these contaminants, only a single exposure
event in the morning was modeled (7:00 a.m. showering event). The limited opportunity for exposures
resulted in fewer symptomatic individuals and a larger temporal spread in clusters of related cases, which
presented a challenge for detection by PHS.
All scenarios were theoretically detectable by WQM; however, the contaminant concentration must be
sufficiently high at one of the 15 monitoring location to change water quality in a manner that causes the
39
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
event detection system to generate an alert. WQM detected 649 of the 2,015 scenarios, yielding a
detection rate of 32%, the lowest among the four components. While the 15 monitoring stations are
strategically located throughout the distribution system to maximize spatial coverage, several scenarios
used injection locations that produced a contaminant spread that did not reach a monitoring station at a
detectable concentration. In the simulation study, it was observed that all scenarios with injections in 28
of the 94 pito zones (30%) were not detected by WQM, indicating that contaminated water did not reach
any of the monitoring stations at detectable concentrations. Injections at these 28 pito zones include 592
scenarios over all 17 contaminants, indicating that failure to detect by WQM can be attributed to system
hydraulics and location of the monitoring stations rather than contaminant properties. While the detection
rate is low for WQM, it did successfully detect almost all of the scenarios that would have produced
significant consequences. Many scenarios that were missed by WQM would have produced only a few
illnesses and no fatalities.
SC can theoretically detect all contamination scenarios at concentrations high enough to produce a
positive result from either a rapid field test or a water quality parameter test. SC detected 1,666 of the
2,015 scenarios, a detection rate of 83% relative to the entire ensemble. The LA component can also
theoretically detect all contamination scenarios; however, the sample must be sent to the proper
laboratory and have a sufficiently high concentration to produce a positive result using LA methods. LA
detected 86% (1,729) of the 2,015 scenarios. For the majority of contaminants, samples were collected in
sufficient time to capture a detectable concentration. However, the conditions of some scenarios resulted
in delayed sample collection, and by the time samples were collected, the contaminated water had passed
through the distribution system and samples collected did not have a contaminant concentration above the
detection threshold.
First Detection of Simulated Contamination Scenarios by Component
A majority of contamination scenarios are theoretically detectable by multiple components. The
component that generates the first alert in a scenario is referred to as "the first component to detect." The
first component to detect a scenario has a significant influence on how the scenario unfolds including
progression and timeliness of threat level determination, timeliness of response actions and ultimately the
reduction in potential consequences. Hence, the data were analyzed to determine the number and
percentage of scenarios that were first detected by each component relative to the total number of
scenarios detected by that component. This analysis was performed on both the full ensemble and just the
distribution system attack scenarios. The results are displayed in Figure 7-4.
40
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
100%
90%
Full Ensemble (2,015)
HDistribution Attack
Scenarios(1,590)
ESM
WQM CCS
Component
PHS
Figure 7-4. Simulation Scenarios First Detected by Each Component
The analysis of the distribution system attack scenarios, excluding the facility attack scenarios, was
conducted because ESM is always the first to detect an attack at a utility facility with security monitoring,
as evident in Figure 7-4. The analysis of distribution system attack scenarios provided a useful
comparison of initial detection by the three remaining components, WQM, CCS and PHS. In all
distribution system attack scenarios that were detected by CCS, it was the first component to detect 97%
of the time (547 out of 564 scenarios). WQM and PHS were the first component to detect in a little more
than half of the distribution system attack scenarios that they detected. However, with significantly
higher number of detections by PHS (1,396) compared to WQM (458), more scenarios were detected first
by PHS (742) than by WQM (257).
Valid Alerts Observed in the Empirical Data
Analysis of empirical data generated during the evaluation period demonstrates that valid alerts were rarer
than invalid alerts. However, WQM, CCS and PHS did generate a total of 84 valid alerts over the course
of the evaluation period. Figure 7-5 shows the causes of valid alerts for the entire system as well as for
each of these three monitoring and surveillance components. Seven categories of valid alerts were
identified:
Main Break. A confirmed break in a water distribution system pipe.
Distribution System Work. A planned activity in the distribution system such as flushing and
pipe repair or replacement.
Treatment Plant Change. An adjustment in chemical feed or a unit process at a drinking water
treatment facility.
Verified, Non-Standard System Operation. Atypical system operations, such as an unusual
change in system pumping or valving, resulting in unusual water quality patterns.
41
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Public Health Incident. An occurrence of disease, illness or injury within a population that is a
deviation from the disease baseline in the population.
Confirmed Water Contamination. A confirmed change in water quality that is the result of the
introduction of a contaminant into the distribution system. There were no confirmed water
contamination alerts during the evaluation period.
Other. A verified alert that could not be definitively attributed to one of the above causes. For
CCS, an example of a valid alert categorized as "other" was an alert generated by a backlog of
complaints due to customers calling the utility after a long holiday weekend during which the call
center was closed. For WQM, an example of a valid alert categorized as "other" was an alert
caused by a real TOC spike for which a cause could not be identified.
Main Break
Distribution System V\tork
Treatment PlantChange
Verified Non-Standard System Operation
PublicHealth Incident
Other
n = 49
WQM
Main Break
Di stri bution System Work
Treatment Plant Change
Verified Non-Standard
System Operation
Other
n = 12
CCS
Distribution System Work
Treatment Plant Change
Other
n = 23
PHS
PublicHealth Incident
Figure 7-5. Causes of Valid Alerts for the System and each Component
WQM generated 49 valid alerts during the evaluation period. The most common cause of valid WQM
alerts was verified non-standard system operations (47%), such as changes in pumping or valving that
produced a detectable change in water quality. The next common causes were distribution work (27%)
and treatment plant changes (14%). An example of a treatment plant change occurred on February 9,
2010 at 10:40 p.m., when unusual finished water quality was observed leaving GCWW's primary
treatment plant. Most notably, chlorine residual levels increased from 0.77 to 1.95 mg/L with a
corresponding decrease in pH from 8.42 to 8.17. The duration of this unusual water quality incident was
approximately 1.5 hours, and the abrupt change in finished water quality can be seen in Figure 7-6.
42
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
2.0
2/8/10
7.9
2/12/10
Figure 7-6. Change in Finished Water Quality Resulting from a Treatment Plant Change
This slug of water propagated out into the distribution system, and although the slug was attenuated, it
was observed at six of the WQM stations deployed throughout the distribution system. Figure 7-7 shows
the attenuated signal at one of these monitoring stations, which generated an alert on February 10, 2010 at
6:20 a.m., or 7.7 hours after the slug of water left the treatment plant. The investigator was able to
quickly identify the source of this unusual water quality by reviewing recent data for finished water
quality leaving the treatment plant.
43
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Event Detection System Alert System CL2
System pH
O)
o
c
Z 1.0
O
G
o
2/8/10
2/9/10
2/10/10
Date
2/11/10
-- 8.0
7.9
2/12/10
Figure 7-7. WQM Alert Caused by the Treatment Plant Change Shown in Figure 7-6
There were 12 valid CCS alerts that were primarily the result of two events. In January 2009, there were
five CCS alerts that were ultimately determined to be related to distribution system work being performed
in the area where calls originated. In August 2008, the CCS component generated five valid alerts due to
elevated chlorine levels in the distribution system, which could be correlated to data produced by the
WQM component to provide further corroboration of the change in water quality.
PHS generated 23 valid alerts, all attributable to public health events that were unrelated to water quality.
These were typical seasonal health events, such as an increase in respiratory distress at the onset of the
allergy season. In another example, a series of alerts were received in October 2009, which public health
officials determined were related to an increase in H1N1 influenza cases in Cincinnati.
While the number of valid alerts was only a small percentage of the total number of alerts generated over
the evaluation period, these examples demonstrate the ability of the monitoring and surveillance
components of the Cincinnati CWS to detect unusual conditions in the system that are related either to
water quality or to public health. These detection capabilities support the contaminant detection function
of the CWS as well as routine monitoring of distribution system and public health conditions.
7.3 Alert Co-occurrence
Definition: Alerts from multiple components that occur within a specified period. Alerts that meet this
criterion define a cluster. If the alerts are valid and share a common cause, the cluster is considered valid.
Analysis Methodology: Co-occurrence of alerts was evaluated using two different data sources:
simulation study results and empirical data.
44
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
As described in Section 7.2, all alerts generated during the simulation study are valid by design.
Furthermore, all alerts generated during a specific contamination scenario are caused by the same
contamination scenario, and thus constitute a valid cluster. This large dataset of valid alert clusters was
divided into groups of similar contaminants, which were evaluated to determine the combinations of
components that detected simulated contamination incidents. The time delays between consecutive alerts
in a cluster were also evaluated.
Alerts were tracked throughout the Cincinnati CWS pilot evaluation period, as described in Section 7.1.
The dataset of WQM, CCS and PHS alerts was used to identify alert clusters from multiple components,
and each cluster was categorized as valid or invalid based on observations from the alert investigations.
Comparison between these observed alert clusters and the simulation study results demonstrate the
frequency with which the predicted alert patterns derived from the simulation study results occur in the
empirical data from the pilot.
Results: This analysis is intended to provide insight regarding the combinations of alerting components
that are indicative of an actual contamination incident. For this analysis, the number of scenarios in the
ensemble of distribution system attack scenarios that were detected by all relevant combinations of two or
more components was determined. The components considered in the permutations include: CCS, PHS,
WQM, SC and LA. This analysis was performed for the following four groups of contaminants:
Nuisance chemicals: Nuisance chemicals 1 and 2. These two contaminants do not produce
acute health consequences at concentrations modeled in the study and are thus undetectable by
PHS.
Contaminants with taste or odor: Toxic Chemicals 1 through 4; Biological Agent 1. These
five contaminants change the aesthetic character of the water (taste, odor, color, or dermal
irritation) and are detectable by CCS. Note that Nuisance Chemical 1 also has a taste and odor
but was placed exclusively in the "nuisance chemicals" group for this analysis.
Contaminants with rapid symptom onset: Toxic Chemicals 5 through 7; Biological Agents 2
and 3. These five contaminants produce symptoms in exposed individuals between 10 minutes
and 4 hours after the time of exposure. Note that Toxic Chemicals 1 through 4 and Biological
Agent 1 also lead to a rapid onset of symptoms but were placed exclusively in the "contaminants
with taste and odor" group for this analysis.
Contaminants with delayed symptom onset: Toxic Chemical 8; Biological Agents 4 through
7. These five contaminants produce symptoms in exposed individuals between 1 day and 2
weeks after the time of exposure. None of the contaminants in this group are detectable by CCS.
Additionally, note that all contaminants are detectable by WQM and S&A. Figures 7-8(a) through 7-
8(d) show the combinations of alerting components that are indicative of a simulated contamination
incident for each of the four categories listed above.
45
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Figure 7-8(a) indicates that none of the scenarios involving nuisance chemicals were detected by PHS,
which was expected given that these contaminants do not produce illness. CCS successfully detected all
scenarios involving Nuisance Chemical 1, but none involving Nuisance Chemical 2, which was expected
given that it does not alter the aesthetic character of the water. CCS and WQM both generated alerts in
67% of scenarios for Nuisance Chemical 1, with CCS being the first to generate alerts in 77% of those
scenarios. WQM was involved in the detection of 79% of scenarios involving nuisance chemicals that
were detected by multiple components and one scenario in which it was the only component to detect
contamination. S&A successfully detected all but one of the scenarios that involved nuisance chemicals,
providing important corroborating information to establish the credibility of the incident.
o
Si
9)
Q
I/)
0
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Figure 7-8(b) shows that CCS detected all contaminants that impart a taste or odor to the water, as
expected. In all but three scenarios, CCS was the first to detect, indicating its importance as an early
warning system for contaminants with aesthetic characteristics. These contaminants were also detected
by PHS and S&A and in that order, with one exception in which PHS alerted before CCS. In 15% of the
scenarios, WQM provided a fourth method of detection, and always alerted before PHS for this
contaminant group. In general, this group of contaminants that impart a taste or odor to the water can be
reliably detected by multiple components.
cjnn
a
"5 400
£
9)
Q
(A
Q ^nn
nj
c
0
o
W onn
s
1
<- A r\r\
3
z.
r\
CCS + WQM +
PHS + S&A (15%)
WQM + CCS + PHS + CCS +
PHS+S&A(100%) S&A(100%)
CCS+PHS +
S&A (85%)
WQM PHS CCS
Component
Figure 7-8(b). Multiple Component Detections of Contaminants with Taste or Odor
Figure 7-8(c) shows that initial detection by PHS followed by S&A was the most common detection
pattern for contaminants that produce rapid onset of symptoms. In 60% of scenarios in which the PHS
and WQM both detected contamination, PHS detected contamination before WQM. Because of the rapid
onset of symptoms, enough cases are generated to quickly trigger a PHS alert. This demonstrates that
PHS is valuable as an early warning component of the CWS for contaminants with rapid symptom onset.
500
o
£
9)
Q
I/)
0
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Figure 7-8(c). Multiple Component Detections of Contaminants with Rapid Symptom Onset
Figure 7-8(d) shows that all scenarios involving contaminants with delayed symptom onset were detected
by PHS. Initial detection occurred via either PHS or WQM, and WQM was the first to detect 94% of the
scenarios involving both components. S&A detected 48% of scenarios involving contaminants with
delayed symptom onset. It was more common for S&A to successfully detect the contaminant when the
scenario was also detected by WQM because the WQM alert also triggers automated sample collection at
the site and time of the alert, thus preserving an aliquot of water that likely has a contaminant
concentration above the detection threshold for S&A (i.e., laboratory analysis or field testing). On the
other hand, PHS alerts occur only when enough symptomatic individuals seek healthcare. In the case of
contaminants with delayed symptom onset, the PHS alert is often delayed until after the contaminated
water has largely left the distribution system, making it challenging to collect a sample with a
concentration above the detection threshold for S&A. While WQM detected only 32% of the scenarios
with delayed symptom onset, timely response to these alerts is critical because WQM may be the only
component to provide early detection of contaminants with delayed symptom onset.
WQM
PHS
Component
CCS
Figure 7-8(d). Multiple Component Detections of Contaminants with Delayed Symptom Onset
The simulation study results were also used to evaluate the time between sequential component alerts.
For example, the time difference between the start of the first and second alerts is calculated for each
scenario and referred to as the second alert delay. This analysis considered the first alert from WQM,
each CCS subcomponent (with two subcomponents) and each PHS subcomponent (with five
subcomponents). For example, if a CCS IVR alert occurred at 30 minutes, a PHS-911 alert at 45 minutes,
and a PHS-ED alert at 120 minutes, the second alert delay is 15 minutes (45 minutes - 30 minutes) and
the third alert delay is 75 minutes (120 minutes - 45 minutes). Figure 7-9 presents the median alert
delays, as calculated over all distribution system attack scenarios.
48
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Contaminant Type
Nuisance Chemical
Taste or Odor
Rapid Symptom Onset
Delayed Symptom Onset
C
D
ft O
0 D
A 0
D 0 A
2nd Alert Delay
3rd Alert Delay
A 4th Alert Delay
» 5th Alert Delay
*
) 500 1,000 1,500 2,000 2,500
Median Delay Between Alerts (minutes)
Figure 7-9. Distribution of Time Delays between Consecutive Alerts from the Simulation Study
In general, the median delay time for successive alerts showed an increasing trend, where the second alert
delay < third alert delay < fourth alert delay < fifth alert delay. The median time delay of the distribution
system attack scenario ensemble for the fifth alert onwards was greater than 18 hours. Alerts occurring
this late, after four other alerts have already occurred, are unlikely to expedite threat level determinations
or responses.
Table 7-4 shows the occurrence of alert clusters for the different sequences of components listed in the
first column. The Alert Cluster Order reflects the sequence and combination of alerts (e.g., CCS/PHS
indicates a CCS alert followed by a PHS alert, while PHS/CCS indicates that the PHS alert occurred
before the CCS alert). The analysis of alert clusters in simulated contamination scenarios was performed
independently on toxic chemicals and biological agents, as defined in Section 3.3, due to the significantly
different detection patterns for these two broad contaminant groups. For each of these two groups, the
number of scenarios in which the indicated alert cluster order occurred is shown in the column titled "#
Alert Clusters." The average delay columns show the average number of minutes between the alert start
times for the components listed in the first column. For example, in the CCS/PHS row, the average delay
is calculated as the difference between the start of the first CCS alert and the first PHS alert. For rows
with three components in the alert cluster order, Delay 1 is the difference between the alert start times for
the two components listed first, and Delay 2 is the difference between the alert start times for the second
and third components. For example, in the row with alert cluster order CCS/PHSAVQM, Delay 1
represents the average time between a CCS and PHS alert and Delay 2 represents the average time
between a PHS and WQM alert. The most frequently occurring alert pattern for toxic chemicals was
CCS/PHS, with a count of 317 alert clusters and an average delay of 97 minutes. The most frequently
occurring alert pattern for biological agents was WQM/PHS, with a count of 169 alert clusters and an
average delay of 2,933 minutes.
Table 7-4. Co-occurrence of WQM, CCS and PHS Alerts for Simulated Contamination Scenarios
Alert Cluster
Order1
CCS/PHS
PHS/CCS
PHS/WQM
Toxic Chemicals
# Alert
Clusters
317
1
49
Average Delay
(minutes)
Delay 1 = 97
Delay 1 = 181
Delay 1 = 458
Biological Agents
# Alert
Clusters
81
0
34
Average Delay
(minutes)
Delay 1 = 74
-
Delay 1 = 472
49
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Alert Cluster
Order1
WQM/PHS
CCS/PHS/WQM
CCS/WQM/PHS
WQM/CCS/PHS
Toxic Chemicals
# Alert
Clusters
16
48
8
2
Average Delay
(minutes)
Delay 1 = 99
Delay 1 = 77
Delay 2 = 342
Delay 1 = 59
Delay 2 = 28
Delay 1 = 49
Delay 2 = 51
Biological Agents
# Alert
Clusters
169
9
4
0
Average Delay
(minutes)
Delay 1 =2,933
Delay 1 = 75
Delay 2 = 412
Delay 1 = 27
Delay 2 = 46
-
The alert cluster patterns observed in the simulation study results (shown in Table 7-4) were used to
characterize alert clusters observed in the empirical data from the Cincinnati CWS pilot in order to
identify how often the patterns seen in the simulation study results were observed in the empirical data.
Table 7-5 shows these results using a table structure similar to that used in Table 7-4. To perform this
analysis, the delay window listed in Table 7-5 was created based on one standard deviation above and
below the average delay reported in Table 7-4 for each alert cluster order, and for both toxic chemicals
and biological agents. The real-time monitoring alerts were then searched to identify clusters that match
those observed in the simulation study results and have alert start times that fall within the delay window.
For example, for toxic chemicals, all CCS alerts were searched to identify if a PHS alert started within 50
to 134 minutes of the start of a CCS alert, matching the pattern observed in the simulation data. Alert
clusters that met these criteria were included in the counts reported in Table 7-5.
Table 7-5. Co-occurrence of WQM, CCS and PHS Alerts Observed in the Emperical Data
Alert Cluster
Order
CCS/PHS
PHS/CCS
PHS/WQM
WQM/PHS
CCS/PHS/WQM
CCS/WQM/PHS
WQM/CCS/PHS
Toxic Chemicals
# Alert
Clusters
5
6
51
10
1
0
0
Delay Window
(minutes)
Delay 1 =50 to 134
Delay 1 = 135 to 227
Delay 1 =75 to 841
Delay 1 =30 to 168
Delay 1 =36 to 108
Delay 2 = 48 to 636
Delay 1 =26 to 91
Delay 2 = 2 to 53
Delay 1 = 10 to 88
Delay 2 = 47 to 55
Biological Agents
# Alert
Clusters
3
-
52
412
1
0
-
Delay Window (minutes)
Delay 1 = 61 to 85
-
Delay 1 = 67 to 877
Delay 1 =508 to 5, 358
Delay 1 =67.5 to 83
Delay 2 = 198 to 626
Delay 1 = 8 to 47
Delay 2 = 27 to 63
-
Table 7-5 shows the frequency of the alert clusters occurring in the real-time alerts. The most frequently
occurring alert pattern for toxic chemicals was PHS/WQM, with a count of 51 alert clusters and a delay
window of 75 to 841 minutes. The most frequently occurring alert pattern for biological agents was
WQM/PHS, with a count of 412 alert clusters and a delay window of 508 to 5,358 minutes. In both
cases, the component alerting order with the maximum occurrence also had the largest delay window.
Thus, the number of alert clusters associated with the indicated order appears to be a function of the size
of the alert window more than an intrinsic characteristic of the system.
The combination of PHS/WQM for both toxic chemicals and biological agents appears with similar
frequency in both simulated and empirical data sets. The alert cluster order observed most frequently in
50
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
the empirical data, WQM/PHS with a long delay window, was also observed frequently in the simulation
study results. As discussed above, the frequency of occurrence of this pattern is due, at least in part, to
the large delay window. However, this large delay window has a basis in reality because in the case of a
biological agent with delayed symptom onset, it is possible to have a WQM alert occur within a few hours
of the start of a contamination incident, while the PHS alert can be delayed for several days or weeks.
Thus, even though these alert patterns can appear randomly in an operational CWS, it is important to
investigate potential causal relationships between WQM and PHS alerts that occur within a temporally
and spatially meaningful cluster. In the Cincinnati CWS, this concept was incorporated into PHS alert
investigation procedures that require investigators to review data and alerts from other components within
a 2-week period preceding the PHS alert.
Other alert cluster patterns that were observed in the simulation study results were not observed in the
empirical data. For example, the most frequent combination for the simulated results (CCS/PHS) did not
occur with nearly the same frequency in the empirical data. While only one instance of three components
alerting occurred in the empirical data for each contaminant group (CCS/PHS/WQM), this alert cluster
order was the third most frequent for toxic chemicals in the simulation study.
Overall, the co-occurrence of temporally related alerts from all three monitoring and surveillance
components (WQM, CCS and PHS) was extremely rare in the empirical data. Additionally, during real-
time monitoring, there was never an occurrence of an alert cluster consisting of multiple, valid alerts from
different monitoring and surveillance components. These results indicate that a temporally and spatially
significant alert cluster consisting of valid alerts from multiple components is likely related to a real water
quality issue, and thus should be thoroughly investigated.
7.4 Summary
The occurrence of valid and invalid alerts has a significant impact on the benefits and sustainability of the
CWS. Benefits of a CWS are realized through detection of unusual water quality conditions that are of
interest to the utility. On the other hand, too many invalid alerts can divert personnel from other duties
and may ultimately be perceived as an indication that the CWS is unsustainable. Although invalid alerts
initially occurred frequently, with more than 150 invalid alerts during most reporting periods in the first
year of operation, once the system was optimized by improving the quality of the underlying data and
updating event detection system configurations to reflect normal variability, the number of invalid alerts
was reduced to about 69 per reporting period. While most alerts were determined to be invalid, the CWS
did detect 84 valid alerts, with more than half of the valid alerts caused by non-standard system operations
and public health events that were unrelated to drinking water.
The Cincinnati CWS was designed to include a variety of surveillance tools to increase contaminant
coverage as well as the reliability of the system for utility managers that need to decide whether or not
contamination may be Possible. Through this multi-component design, weaknesses in the detection
capabilities of one component are offset by the strengths of another. In particular, co-occurring alerts
from multiple components can increase the utility manager's confidence that the alerts are valid and
indicative of a potential water quality issue. Different contaminant types such as nuisance chemicals or
those with rapid or delayed symptom onset trigger different combinations of component alerts and the
timing of those alerts occur in predictable patterns. The co-occurrence of two alerting components,
especially the combination of PHS and WQM, was frequent in both simulated and empirical data. While
any combination of three components alerting was observed only once in the empirical data, alert clusters
involving three or more components was common in the simulation study results. This would suggest
that valid alert clusters involving alerts from multiple components are probably the result of a real water
quality issue in the distribution system.
51
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 8.0: Timeliness of Detection and Response
For a CWS to have the maximum potential to reduce consequences of a contamination incident, it must
detect the incident early enough to allow sufficient time to implement effective response actions. The
timeliness of detection is heavily dependent upon the design of the individual monitoring and surveillance
components. The timeliness of response is primarily governed by consequence management. However,
the overall timeline of a contamination incident is largely influenced by the details of the scenario, most
notably the injection location, contaminant mass, and contaminant injection rate, which will determine the
hydraulic travel time and spread of the contaminant through the distribution system. The specific
contaminant used in the incident will determine which monitoring and surveillance components have the
potential to generate alerts as well as the manner in which consequences unfold.
This design objective was evaluated through analysis of detection and response times measured during
routine operations, drills and exercises, and simulations. However, to evaluate whether the timing of
detection and response actions was sufficient, the reduction in consequences for simulated contamination
scenarios, attributable to deployment and operation of the CWS, was assessed. Thus, this section will
present results for the reduction in consequences in addition to an analysis of detection and response
times.
To evaluate how well the CWS met this design objective, the following three metrics were evaluated:
detection time, response time, and consequence reduction. The following subsections define each metric,
describe how it was evaluated and present the results.
8.1 Detection Time
Definition: The time between the initial presence of abnormal water quality in the distribution system
(e.g., injection of a contaminant) and the start of a component alert. The delays that occur between these
two events vary by component, but generally result from the following:
Hydraulic travel time between the injection location and a customer or a sensor,
Time to generate data (e.g., a security alert, a reading from a water quality sensor, a call from
customer with a water quality complaint, a health seeking behavior from a symptomatic
individual), and
Time to analyze the data and generate an alert, which in the case of WQM, CCS and PHS relies
on an automated event detection system.
Analysis Methodology: Results from the simulation study were used to calculate the detection time for
each contamination scenario as the difference between the start of a component alert and the start of
contaminant injection. The resulting detection times were analyzed by component and contaminant. The
latter is an important stratification of the results because the contaminant properties can impact which
component detects the contamination incident as well as the relative timing of alerts. Many of the results
presented in this section are based on analyses that were limited to either distribution system attack
scenarios or facility attack scenarios. If a specific scenario type or subset is not specified, the results are
from an analysis performed on the entire ensemble.
Results: Five sets of results are presented for the analysis of detection times. First, timelines for five
representative simulated contamination scenarios are presented to illustrate how typical contamination
scenarios unfold. Next, the results of a statistical analysis of initial alert times by component are
52
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
presented. The remaining three subsections present the results of further analysis of the differences in the
times of initial alerts for each of the following components: CCS, PHS and WQM.
Illustrative Contamination Scenario Timelines
The five scenarios presented in this section were selected to demonstrate the variability in the sequence
and timing of alerts and response actions, which is largely driven by differences in the scenario variables
such as contaminant type, injection location and injection start time (12:00 a.m. or 9:00 a.m.).
The timeline for a typical contamination scenario with Nuisance Chemical 1 is shown in Figure 8-1. The
injection occurred at a distribution system node during the morning, a period of high water demand. The
first CCS alert occurred 2.5 hours after injection and led to a Possible determination within the next half
hour. CCS was the first component to detect in 85% of the distribution system attack scenarios involving
this contaminant, and the scenario represented in Figure 8-1 reflects this tendency. WQM was the first to
detect this contaminant in 15% of the distribution system attack scenarios. In scenarios where WQM was
the first to detect, the injection was at midnight and detection by CCS was delayed until the first exposure
event in the morning, approximately seven hours later.
Possible determination was followed by an operational (Op) response to limit the spread of the
contaminant 15 minutes later. Water quality (WQ) field testing occurred approximately 2.5 hours after
the Possible determination and the results from rapid field testing (RFT) elevated the threat level to
Credible less than six hours after injection, and public health (PH) response occurred at the same time.
The first WQM alert occurred approximately eight hours after the start of the injection and provided
information sufficient to elevate the threat level to Confirmed. Public notification was issued
approximately nine hours after the start of the injection. While preparation of the public notice began at
the time contamination was determined to be Possible, issuance of the notice was delayed until
contamination was Confirmed because there were no adverse health impacts during this scenario. An
assumption of the model is that public notification will be issued before contamination is Confirmed only
if there is clear a risk to public health, which is based on observations from drills and exercises.
08:16
WQM Alert
T
00:00
Day 1
Contaminant
Injection
03:12
First Op Chan
02:31
CCS-IVR Alert
(
RF1
ge
05:30
WQ Parameter
Results
^^^^^^^1 i j
02:57 0
Possible Time Cred
D5:47
" Results
05:50
PH
Response
<
F
i
i
5:50
ble Time Cor
08:48
3ublic Notification
10:47
Lab Results
T
12:00
08:48 DaV 1
firmed Time
Figure 8-1. Timeline for a Typical Contamination Scenario with Nuisance Chemical 1
The timeline for a typical contamination scenario with Toxic Chemical 1 is shown in Figure 8-2. The
injection occurred at a distribution system node at midnight. The first CCS alert occurred 6 hours and 40
minutes after injection, which corresponds to the first opportunity for exposure in the model design. CCS
was the first component to detect all distribution system attack scenarios that involve Toxic Chemical 1.
This is true even for scenarios with injections at midnight where detection by CCS is delayed until the
53
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
first exposure event occurs approximately 7 hours after the start of the injection. Thus, the example
scenario shown in Figure 8-2 is considered representative of most scenarios involving Toxic Chemical 1.
The first CCS alert led to a Possible determination 27 minutes after the alert, and operational response
actions were implemented 7 minutes later. The public health response occurred within half an hour of the
Possible determination. A PHS-DPIC alert occurred next, 7 hours and 40 minutes after the start of the
injection, and provided sufficient information to elevate the threat level to Credible. The first WQM alert
occurred 8 hours and 17 minutes after injection and provided sufficient information to Confirm
contamination. Public notification was issued approximately 9 hours after the start of the injection, and
exactly 2 hours after the Possible determination, which is the assumed time needed to prepare the notice.
The field testing and laboratory analysis results were available approximately 10 and 17 hours after the
injection, respectively. While these results were not available in time to inform the response actions
simulated in the model, definitive identification of the contaminant through laboratory analysis would
certainly inform later stages of the response to an actual contamination incident.
00:00
Day 1
Contaminant
Injection
07:40
PHS-DPIC Aler
07:31
PH
Response
07:13
First Op Change
06:39
CCS-IVR Alert
L
t
I
r
07:06
Possible Time
07:40
Credible Time
c
08:17
WQM Alert
Put
09:06
Dlic Notification
09:32
WQ Parameter
Results
09-49 17:19
RFT Results Lab Results
ui r
12:00 00:00
Day 1 Day 2
08:23
anfirmed Time
Figure 8-2. Timeline for a Typical Contamination Scenario with Toxic Chemical 1
The timeline for a typical contamination scenario with Toxic Chemical 5, which lacks a taste or odor, is
shown in Figure 8-3. The injection occurred at a distribution system node in the morning. The public
health response occurred 1.5 hours after injection, but before the first alert. The rapidity of the public
health response was due to an unusually high number of cases seen in the emergency department and
clear indications of the causative agent based on observed symptoms, which prompted officials to
mobilize public health resources to care for the injured even though the source of exposure had not yet
been determined. The first alert occurred just 4 minutes after the public health response and was from the
PHS-DPIC component. This alert triggered a teleconference among public health partners, including the
drinking water utility, where it was determined that contaminated water was a possible source of the
exposures. This prompted the utility to implement operational response actions 20 minutes later. While
another PHS alert (from the Astute Clinician (AC) data stream) occurred about 1 hour after the Possible
determination, there was still no direct evidence linking the exposures to contaminated drinking water.
However, utility SC teams were sent to locations of suspected exposure, where the results of water quality
parameter testing indicated a potential problem with the water, which was sufficient evidence to consider
water contamination to be Credible about 2.5 hours after the Possible determination was made. The
determination that water contamination was Credible, combined with the number of reported illnesses,
54
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
was sufficient for the utility to issue public notification at this point. The first WQM alert occurred 5
hours after the start of the injection. The preponderance of evidence from all of these signals was
sufficient to Confirm contamination 5.5 hours after the start of the injection, even though confirmatory
laboratory results would not be available until 5 hours later.
Comparing this scenario with the one shown in Figure 8-2 shows the impact of the injection start time on
the scenario timeline. In Figure 8-2, the injection begins at midnight, and the first exposures and
subsequent alerts are delayed for several hours. Figure 8-3 shows that exposures and alerts occur soon
after the start of an injection in the morning. However, contamination was determined to be Possible with
half an hour of the first alert in both cases. This reflects the efficient, streamlined alert investigation
procedures developed for the Cincinnati CWS.
02:08
First Op Change
04:16
Public Notification
Day 1
Contaminant
Injection
01:33
PHS-DPIC Alert
01:29
PH 02:45
Response PHS-AC Alert
j h r
f
10 01 :48
Possible Time
04:16
WQ Parameter
Results
05:01
WQM Alert
n
L
05:34
10:23
Lab Results
r
12:00
Day 1
Confirmed Time
04:16
Credible Time
Figure 8-3. Timeline for a Typical Contamination Scenario with Toxic Chemical 5
The timeline for a typical contamination scenario with Biological Agent 3, which lacks a taste or odor, is
shown in Figure 8-4. The injection occurred at a distribution system node at midnight. The first alert
was generated by the Astute Clinician data stream of the PHS component, which occurred 9 hours and 45
minutes after the start of the injection. This alert triggered a teleconference between the utility and public
health partners, which resulted in a Possible determination 45 minutes later, and implementation of an
operational response 20 minutes after that. The first WQM alert occurred approximately 12 hours after
the start of the injection, and once the initial investigation of this alert was completed 45 minutes later,
contamination was deemed Credible. Similar to the example for Toxic Chemical 1, the combination of
Credible contamination and a large number of illnesses was sufficient for the utility to issue a public
notification. The results of the laboratory analysis were available 25 hours after the start of the injection,
and were sufficient to confirm contamination when they were reported to the WUERM one hour later.
Additional PHS alerts occurred after contamination had been Confirmed and thus were inconsequential to
the investigation and response.
55
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
10:50
First Op Change
11:55
WQM Alert
09:45
PHS-AC Alert
12:54
WQ Parameter
Results
12:57
Public Notification
01:01
Lab Results
02:01
PH
Response
?*
00:00
Day 1
Contaminant
Injection
10:30
Possible Time
12:57
Credible Time
00:00
Day 2
02:01
Confirmed Time
00:00
Day 3
Figure 8-4. Timeline for a Typical Contamination Scenario with Biological Agent 3
The timeline for a typical contamination scenario with Biological Agent 4, which lacks a taste or odor and
has a delayed symptom onset, is shown in Figure 8-5. The injection occurred at a distribution system
node in the morning. The first WQM alert occurred eight hours after injection, and once the initial
investigation of this alert was completed 42 minutes later, contamination was determined to be Possible.
Results of WQ parameter testing were available approximately three hours after the Possible
determination, but were insufficient to establish that contamination was Credible. It was not until a PHS
alert from the Astute Clinician data stream occurred 20 hours after the injection that there was sufficient
evidence to establish that contamination was Credible and issue public notification. Further investigation
of the PHS alert and discussions between the utility and public health partners provided enough evidence
to Confirm contamination a little more than one hour later, even though the identity of the contaminant
was still unknown at that time. Public health response was delayed until there was sufficient information
about the probable identity of the contaminant approximately seven hours after contamination was
Confirmed. Results of laboratory analysis were not available until late on the second day of the scenario.
20:14
PHS-AC Alert
08:01
WQM Alert
_r '
11:47
WQ Parameter
Results
n F
L *
20:14
Public Notification
03:46
PH
Response
'I 1 ~l
II J I
00:00 08:42 20:14 00:00
Day 1 Possible Time Credible Time Day 2
Contaminant
Injection 21 '27
Confirmed Time
18:54
Lab Results
"I
T
00:00
Day3
Figure 8-5. Timeline for a Typical Contamination Scenario with Biological Agent 4
The timelines for biological agents with delayed symptom onset are significantly longer than those for
nuisance and toxic chemicals. These biological agents cannot be detected by CCS, which leaves only
WQM and PHS to provide initial detection of distribution system attack scenarios. WQM can detect
56
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
these agents within hours of the injection, but if WQM fails to detect the contamination scenario, the first
PHS alert can be delayed by several days.
Timeliness of Initial Alerts by Component
Figure 8-6 shows a box-and-whisker plot of the timeliness of alerts from the monitoring and surveillance
components and analytical results from the investigative components plotted for distribution system
attack scenarios. This figure shows the statistical distribution of alert times for each component for the
subset of scenarios that was detected by that component (the number of scenarios detected is shown to the
right of the plot). The median CCS alert occurs much earlier than PHS or WQM alerts, which is
consistent with CCS being the first component to detect in 97% of the scenarios that are detectable by
CCS, as shown in Figure 7-3. This is anticipated as contaminants that have a perceptible taste, smell or
color are detected quickly by customers at fairly low concentrations. This prompts a percentage of them
to call the utility, consequently triggering CCS alerts. The call threshold to trigger a CCS alert in the
Cincinnati CWS is relatively low, and thus the component could detect a contamination incident after just
a few calls.
Time of CCS Alert
Time of WQM Alert
Time of PHS Alert
Time of SC Results
Time of LA Results
n= 564
-i n = 458
n= 1,271
n= 1,396
n= 1,545
1,000 2,000 3,000 4,000 5,000
Time Since Contaminant Injection (minutes)
6,000
Figure 8-6. Timeliness of Monitoring and Surveillance Component Alerts and Sampling and
Analysis Results for Distribution System Attack Scenarios
The PHS alert times show high variability over the entire ensemble. However, when the PHS alert times
are analyzed by contaminant, much less variability in initial alert times is observed, indicating that the
contaminant-specific delays in onset of symptoms contribute significantly to the variability in PHS alert
time. On the other hand, the variability in the time of the initial WQM alert is primarily driven by the
hydraulic travel time between the injection location and the WQM station. The timing of results from the
investigative components, SC and LA, is largely driven by the time to reach a Possible determination,
which is a precursor to initiating these activities. The distribution in the time of LA results is further
expanded by contaminant-specific properties, such as the time to deliver the sample to a lab that can
analyze for the specific contaminant and the method analysis time.
57
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
To illustrate the impact of contaminant-specific properties on detection times, the time of initial alerts for
CCS, PHS and WQM is analyzed by contaminant for the distribution system attack scenarios in the
following subsections.
Timeliness of Alerts by Contaminant for CCS
Figure 8-7 shows the timeliness of CCS alerts by contaminant for distribution system attack scenarios
involving contaminants with a taste or odor. CCS alerts for Nuisance Chemical 1, Toxic Chemical 1 and
Toxic Chemical 4 showed significant variability while those for Toxic Chemical 2, Toxic Chemical 3 and
Biological Agent 1 had a much smaller distribution. This result can be attributed to the different
percentage of injection times during high and low demand periods for each contaminant rather than to
contaminant properties.
Biological Agent 1
Nuisance Chemical 1
ToxicChemical 3
ToxicChemical 1
ToxicChemical 2
ToxicChemical 4
C
th
h-| |
*=
I
h-| |
*
h
^
HI
) 50
H
H
100 150 200 250 300 350 400 450
TimeSince Contaminant Injection (minutes)
Figure 8-7. Timeliness of CCS Alerts by Contaminant
As seen in Table 8-1, for Nuisance Chemical 1, Toxic Chemical 1, and Toxic Chemical 4, one half to one
third of the injections occurred at low demand (12:00 a.m.) whereas more than 91% scenarios for Toxic
Chemical 2, Toxic Chemical 3, and Biological Agent 1 had injections at high demand (9:00 a.m.).
Injections at 12:00 a.m. result in a delay of several hours between the start of the scenario and the first
exposure, which yields a delay in the first CCS alert. Thus, a subset of scenarios that are predominately
morning injections (i.e., Toxic Chemical 2, Toxic Chemical 3, and Biological Agent 1) will have a much
narrower distribution of alert times compared with those that have a more equivalent mix of injections at
high and low demand (9:00 a.m. and 12:00 a.m., respectively). The significant delay in CCS alert times
in scenarios with injections at low demand (12:00 a.m.) is an artifact of the model design in which no
customer is exposed until the morning, several hours after the start of the injection. While this modeling
assumption was considered reasonable because water demand in the GCWW distribution system is
substantially lower at midnight than it is in the morning, it is possible that there would be enough calls
shortly following an injection at midnight to trigger a CCS alert.
Table 8-1. Number of Scenarios with Injections at High and Low Demand Periods
Contaminant ID
High Demand
(9:00 a.m.)
Injection
Low Demand
(12:00 a.m.)
Injection
Total
% High Demand
Injections
58
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Contaminant ID
Toxic Chemical 2
Biological Agent 1
Toxic Chemical 3
Nuisance Chemical 1
Toxic Chemical 1
Toxic Chemical 4
High Demand
(9:00 a.m.)
Injection
93
90
86
63
61
48
Low Demand
(12:00 a.m.)
Injection
1
4
8
31
33
46
Total
94
94
94
94
94
94
% High Demand
Injections
98.9%
95.7%
91.5%
67.0%
64.9%
51.1%
Timeliness of Alerts by Contaminant for PHS
Figure 8-8 shows the timeliness of PHS alerts plotted for distribution system attack scenarios involving
injection of contaminants with rapid onset of symptoms, including those that have a taste or odor. The
time between an exposure and onset of low-level symptoms is also shown in Figure 8-8 by the "X"
symbol. Those contaminants with longer symptom delays generally have later PHS alert times. A strong
correlation (r = 0.84) was observed between the contaminant-specific symptom onset delays and the
median PHS alert times for these contaminants. Toxic Chemical 6 was an exception to this correlation
where the median PHS alert was later than projected based its symptom onset delay. This was the result
of a large number of low demand (12:00 a.m.) injection scenarios in the ensemble for Toxic Chemical 6,
which resulted in a seven hour delay before the first consumption event and subsequent symptoms and
health seeking behaviors necessary to trigger a PHS alert.
ToxicChemical7
ToxicChemical 5
ToxicChemical 1
Biological Agent 1
ToxicChemical 3
ToxicChemical 4
ToxicChemical 2
Biological Agents
Biological Agent2
ToxicChemical 6
v i m
/\ i M i *
x iin i
HI H
X iT-l
X m 1
X
-i
H 1
1 1 1
X H
X
H
X
*] i
r~ i
x Low Sym ptom Onset
Delay
i
1
1
0 100 200 300 400 500 600 700
Time Since Contaminant Injection (minutes)
Figure 8-8. Timeliness of PHS Alerts for Contaminants with Rapid Symptom Onset (including
those with a taste or odor)
PHS alerts for Biological Agent 1, Toxic Chemical 2 and Biological Agent 2 exhibited the least
variability. This can be attributed to the distribution of injection times in ensembles for each contaminant,
where these three contaminants had more than 95% of the injections during high demand (9:00 a.m.).
59
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
The remaining seven contaminants had simulations with a more diverse blend of injections at both low
and high demands. These contaminants with a mix of injection times show more variability because there
is a significant difference between the times of injection and first consumption (and consequently health
seeking behavior and PHS alerts) for injections at low and high demand.
Figure 8-9 shows the timeliness of PHS alerts plotted for distribution system attack scenarios involving
injection of contaminants with delayed symptom onset. The time between an exposure and onset of low-
level symptoms is also shown on this figure (X). Similar to the results for contaminants with delayed
symptom onset, those contaminants with longer symptom delays generally have later PHS alert time. A
strong correlation (r = 0.95) was observed between the symptom onset delay and the median PHS alerts
times. This is particularly evident when the results in Figures 8-8 and 8-9 are compared, noting the
different scales on the x-axis. Biological Agents 6 and 7 have longer median alert times and a larger
distribution of alert times compared with Toxic Chemical 8, even though all three have identical symptom
onset delays. The reason for this is that exposure to Biological Agents 6 and 7 occurs by inhalation, and
there is only one inhalation exposure event per day (during showering at 7:00 a.m.). This artifact of the
model will result in both later alerts and a broader distribution of alert times.
Biological Agent4
ToxicChemical 8
Biological Agent?
Biological Agent6
Biological Agents
X
t-
*
X
X
I
nz
,
\ i
x Low Symptom Onset
Delay
~|
J
,
X
I
0 1,000 2,000 3,000 4,000 5,000 6,000 7,000
TimeSince Contaminant Injection (minutes)
Figure 8-9. Timeliness of PHS Alerts for Contaminants with Delayed Symptom Onset
Timeliness of Alerts by Contaminant for WQM
The WQM alerts showed no discernible trend when plotted by contaminant. This can be attributed to the
fact that all contaminants evaluated in this study are theoretically detectable by WQM and most produce a
detectable change in water quality at concentrations significantly lower than those that would cause acute
health effects (Allgeier, et al, 2010), as shown in Table 6-1. Modeling results indicate that the timing of
WQM alerts is driven by the hydraulic travel time between the injection location and the WQM stations.
Figure 8-10 shows the number of WQM alerts and distribution of WQM alert times generated during the
distribution system attack scenarios for each of the 15 WQM stations. As can be seen from the counts in
60
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
this figure, six WQM stations were responsible for 79% of the alerts: F, L, K, B, M and C. However, the
six monitoring stations (O, E, H, I, N and G) consistently generated the earliest alert times. These six
monitoring stations are located closely downstream of major pump stations and in fairly populous areas.
O
E
H
1
N
I G
2 C
il 1 n=13
d i n=27
H n=17
E-i n=30
in li n=49
ifln n=23
({EH 1
n=98
5 K i-n 1 i I n=130
O
§ M
F
L
B
1
A
hO=l
j i i_i
n^_^_i^_^_i i
i n=107
n=151
n=146
i
u I l-| i n=27
I
. n K
0 2,000 4,000 6,000 8,000 10,000
TimeSince Contaminant Injection (minutes)
Figure 8-10. Timeliness of WQM Alerts by Station
The median total time to alert ranged from 1.6 hours (Station O) to 43.8 hours (Station A). The three
stations with the longest alert delays (Stations A, B and D) were also the stations with the lowest
percentage of alerts produced with respect to the number of potential alerts (the number of scenarios in
which the WQM station witnessed a detectable contaminant concentration). These stations experience
high water quality variability that can mask water quality anomalies. Thus, CANARY was configured to
require a longer period of unusual water quality before an alert is generated. This is intended to reduce
the number of invalid alerts received, though it also increases the time to detect when a true water quality
anomaly is present.
Each component has different factors that drive the timeliness of the alerts. CCS alerts occur quickly
after the first opportunity for consumption due to the low thresholds of the event detection system. The
timing of PHS alerts is impacted by the symptom onset delay, and thus the characteristics of the
contaminants. The timing of WQM alerts are driven by hydraulic travel time to the WQM station from
the injection site. Taken independently, these factors show that each component has the ability to detect
some types of scenarios more quickly than others. However, when all of these components are integrated
into a CWS, the resulting system has the potential to detect a wide variety of scenarios early enough to
provide time for effective response and consequence mitigation.
8.2 Response Time
Definition: The time between detection of a contamination incident and implementation of various
investigative and response actions, including: threat level determination, operational response, public
notification, and public health response.
61
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Analysis Methodology: Response time was evaluated using two different data sources: results from
drills and exercises and simulation study results.
The results from 21 drills and exercises conducted during the evaluation period, described in Section 3.2,
were used to estimate various metrics on the response timeline. These drills and exercises provided
evaluators with the opportunity to observe and characterize the time required to implement various
response actions in real time under conditions of a simulated contamination incident. The timeliness
metrics evaluated during drills and exercises can be broadly grouped into two categories:
1. Threat level determination process, which includes the time to Possible, Credible and Confirmed
determinations, as well as the time for results from SC and LA
2. Response actions, which include operational response, public health response and public
notification
Some artificialities are introduced during drills and exercises because participants are aware of the
activity, which can result in more aggressive response actions than might be observed in the early stages
of a real-world alert investigation. Additionally, the results from drills and exercises are limited by the
conditions of the specific contamination incident developed for the drill or exercise. However, these
results provide a useful benchmark for the response time metrics. Furthermore, the results from the
simulation study, described in Section 3.3, provide an expanded set of contamination scenarios from
which to evaluate response timeliness metrics.
Results: The results from drills and exercises were one of the data sources used to parameterize the CWS
model used in the simulation study. Thus, the results from these simulations should provide a reasonable
estimate of response times for a variety of contamination scenarios. For illustrative purposes, the
timelines generated during two full-scale exercises are described below.
FSE 2 was conducted on October 1 and 2, 2008, with the objective of exercising protocols for
investigating and responding to a Possible drinking water contamination incident. The FSE was based on
a scenario involving the intentional injection of a large quantity of a biological agent into the distribution
system. The exercise was initiated with a WQM alert.
Figure 8-11 shows significant events along the timeline for FSE 2. Following the initial WQM alert,
additional WQM alerts were initiated two hours later, followed by the first CCS alert 30 minutes after
that. The combination of WQM and CCS alerts prompted the utility to conclude that contamination was
Possible, and to subsequently deploy the SC team. Additional customer calls were sufficient to establish
that contamination was Credible 3 hours and 45 minutes after the start of the exercise. Field sampling
results from site characterization were available 2 hours and 40 minutes after the site characterization
team deployed. A PHS alert was initiated 6.5 hours after the start of the exercise, and was instrumental in
the decision to issue a public notification two hours later.
FSE 2 occurred early in the evaluation period, before the investigation and response procedures for the
Cincinnati CWS had been streamlined. Observations from this exercise led to many revisions and
refinements to the CWS procedures, including updates to roles and responsibilities for responders and
streamlining of communication protocols, such as the development of the PHS communicator protocol.
This led to improved response times in later drills and exercises.
62
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
00:00
WQM Alert
02:10
First Operational Response
02:00
Additional WQM
Alerts
05:30
Field Sampling Results
02:30
CCS Alert
02:50
SC Team Deployed
06:30
PHS Alert
08:30
Public Notification
00:00
09
30
02:45
Possible Determination
03:45
Credible Determination
09:30
Confirmed Determination
Figure 8-11. Timeline for Full Scale Exercise 2
FSE 3 was conducted on October 21 and 22, 2009, to provide GCWW's Incident Command System
second-in-command personnel and local response partner agencies with the opportunity to exercise
response procedures. The FSE was based on a scenario involving the intentional injection of a large
quantity of a toxic chemical into the distribution system. The exercise was initiated with a CCS alert.
Figure 8-12 shows significant events along the timeline for FSE 3. The initial CCS alert was generated
by the IVR data stream, and a second CCS alert, generated by the work orderdata stream, occurred 30
minutes later. A review of the underlying calls associated with these alerts showed that they were all
from the same neighborhood, which led to the conclusion that contamination was Possible 44 minutes
after the start of the exercise. The first operational response was implemented 16 minutes after the
Possible determination and the SC team was deployed 40 minutes later. Rapid field test results from SC
were available three hours after the SC team was deployed, and were sufficient to establish that
contamination was Credible 15 minutes later. Contamination was Confirmed based on the preponderance
of evidence just 15 minutes after the Credible determination, and before the PHS alert was initiated. This
prompted the utility to issue a public notification 5 hours and 15 minutes after the start of the exercise.
63
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
00:00
CCS Alert
00:30
CCS Alert
01:00
First Operational Response
03:40
Rapid Field Test Results
01:15
CCS Alert
01:40
SC Team Deployed
02:45
CCS Alert
04:20
PHS Alert
05:15
Public Notification
00:00
03:55
Credible Determination
08:00
00:44
Possible Determination
04:10
Confirmed Determination
Figure 8-12. Timeline for Full Scale Exercise 3
Comparison of the timelines for FSE 2 and FSE 3 shows a marked improvement in response times in FSE
3. During FSE 3, the cluster of calls that triggered the CCS alert was sufficient to conclude that
contamination was Possible, while in FSE 2 that determination was delayed for several hours.
Additionally, operational responses were implemented sooner during FSE 3 than they were during FSE 2.
The critical decision to issue a public notification was made three hours and fifteen minutes sooner during
FSE 3, which would have a dramatic impact on limiting further exposures. The improved performance
observed during FSE 3 was a result of acting on the lessons learned from FSE 2 and an increased
confidence in implementing procedures and decision-making that resulted from drills held during the year
between the two FSEs.
As noted previously, these two examples, as well as the other 19 drills and exercises conducted over the
evaluation period, represent a limited number of contamination scenarios. Furthermore, the performance
of the personnel involved in implementation of investigative and response procedures improved over the
course of the pilot. Thus, the dataset of timeline metrics derived from these drills and exercises is limited.
To address this limitation, the timeline metrics from the simulation study were analyzed for response
times. Figure 8-13 shows the timeliness of threat level determination and response actions for all
simulated distribution system attack scenarios. The median time that each of the three threat levels was
reached occurred sequentially, as expected: Possible determination at 330 minutes, followed by Credible
at 385 minutes, followed by Confirmed at 562 minutes. Overall, the time at which operational response
was implemented (a median of 320 minutes) corresponded closely to the time of Possible determination.
This outcome is related to the model assumption that once a CWS alert is validated, the utility would
begin implementing operational response actions that do not impact customers in an effort to limit the
spread of potentially contaminated water. This simplifying assumption is consistent with utility decisions
and actions that were demonstrated during drills and exercises held later in the evaluation period, such as
FSE 3.
The median time of public notification (458 minutes) is between the median times of Credible and
Confirmed determinations. This is consistent with utility response actions during FSEs, in which public
notification was issued only after contamination was deemed Credible, but often before contamination
was Confirmed. The median public health response was 477 minutes, and this response action is driven
primarily by public health information derived from cases at hospital emergency departments.
64
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Possible Determination
Credible Determination
Confirmed Determination -
Operational Response
Public Health Response
Public Notification
,|
1
, I
' 1
H
.1 1 1
' 1 1
i n 1
i n 1
i n - 1
315
313
315
i n - 1 515
i n - 1 °53
n "1 /ftQ
0 1,000 2,000 3,000 4,000
Time Since Contaminant Injection (minutes)
Figure 8-13. Timeliness of Threat Level Determination and Responses
The ensemble was broken down by the contaminant groups described in Section 7.3 (nuisance chemicals,
contaminants with taste or odor, contaminants with rapid symptom onset and contaminants with delayed
symptom onset) and the corresponding timeliness plots are shown in Figures 8-14 through 8-17.
Grouping contaminants in that manner led to a reduction in variability of the timeliness metrics indicating
that these metrics were a function of contaminant properties.
65
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Figure 8-14 shows the threat level determination and response timeline metrics for the subset of
distribution system attack scenarios that involve injection of nuisance chemicals. The median times for
the three threat levels occurred sequentially, as expected: Possible determination at 341 minutes, followed
by Credible at 417 minutes, followed by Confirmed at 611 minutes. These threat level determination
times were similar to those observed for the complete set of distribution system attack scenarios.
However, the median time for operational response (91 minutes) occurred earlier than the median
Possible determination for nuisance chemicals. This is a result of the large number of nuisance chemical
scenarios that were detected by WQM, and the fact that operational responses can be implemented in
response to a verified WQM alert before a Possible determination is made. For nuisance chemicals, the
median times for Confirmed determination and public notification were identical (611 minutes). This is
due to the fact that no health consequences are involved in scenarios involving the nuisance chemicals,
which eliminates the triggers that could prompt public notification before contamination is Confirmed.
Possible Determination
Credible Determination
Confirmed Determination
Operational Response i
Public Notification
0 200 400 600 800 1,000 1,200 1,400 1,600 1,800
Time Since Contaminant Injection (minutes)
Figure 8-14. Timeliness of Threat Level Determination and Responses for Nuisance Chemicals
66
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Figure 8-15 shows the threat level determination and response timeline metrics for the subset of
distribution system attack scenarios that involve injection of contaminants with taste or odor. Of the four
groups of contaminants, those with taste and odor have the shortest times for threat level progression and
response actions. This is primarily due to rapid detection of these contaminants through CCS, or PHS in
the case of chemicals that are also toxic. The median times for the threat level determination are 59
minutes for Possible, 117 minutes for Credible, and 182 minutes for Confirmed. The median time for
operational response was 71 minutes, similar to the median time for Possible determination. In general,
CCS alerts will not result in an operational response until a Possible determination is made. The median
time for public notification was 184 minutes, which is very close to the median time for Confirming
contamination. However, the time of public notification in scenarios that are rapidly detected is driven
primarily by the two hour period required to prepare the notice rather than by the time to determine that
contamination is Credible. The median time for public health response was 96 minutes.
Possible Determination
Credible Determination
Confirmed Determination
Operational Response
Public Health Response
Public Notification
^Eh
i
1.
1
HI
1
I
I
1
i n - 470
. - AC
I
I
n - 470
n=470
0
n - 470
n - 470
0 100 200 300 400 500 600
Time Since Contaminant Injection (minutes)
Figure 8-15. Timeliness of Threat Level Determination and Responses for Contaminants with
Taste or Odor
67
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Figure 8-16 shows the threat level determination and response timeline metrics for the subset of
distribution system attack scenarios that involve injection of contaminants with rapid symptom onset.
The median times for the threat level determination are 300 minutes for Possible, 448 minutes for
Credible and 679 minutes for Confirmed, which are slightly longer than those for the full set of
distribution system attack scenarios. The distribution of times for operational response is comparable to
that for Possible determination. The time distribution of public notification corresponds closely with
Credible determination, and the median times were exactly the same at 448 minutes. The reason public
notification is issued around the same time as contamination is deemed Credible for this class of
contaminants is that the rapid symptom onset quickly generates a large number of cases, which increases
the urgency to issue the notification. The median public health response time was 447 minutes, similar to
the time of Credible determination but with more variability that is driven by differences in the difficulty
of quickly identifying a causative agent for this group of contaminants. Specifically, for contaminants
with unique and uncommon symptoms, a small number of cases can be sufficient to draw a tentative
conclusion about the causative agent. However, contaminants that produce symptoms that are similar to
those caused by common illnesses require more cases to make a tentative identification.
Possible Determination
Credible Determination
Confirmed Determination
Operational Response
Public Health Response
Public Notification
200 400 600 800 1,000 1,200 1,400
Time Since Contaminant Injection (minutes)
Figure 8-16. Timeliness of Threat Level Determination and Responses for Contaminants with
Rapid Symptom Onset
68
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Figure 8-17 shows the threat level determination and response timeline metrics for the subset of
distribution system attack scenarios that involve injection of contaminants with delayed symptom onset.
The median times for the threat level determination are 2,013 minutes for Possible, 1,644 minutes for
Credible and 1,704 minutes for Confirmed. These times are 2.5 to 6.7 times longer than the times for
contaminants with rapid symptom onset, which is driven by the long delay between exposure and
symptom onset and the fact that none of these contaminants can be detected by CCS. Furthermore, the
median time for Possible determination is longer than that for Credible or Confirmed determination due to
the fact that less than half of the scenarios that reached Possible went on to reach Credible or Confirmed.
The time distribution of operational response corresponds closely to the time of Possible determination
with a median time for operational response of 1,922 minutes. In this group of contaminants, the
necessary condition to implement an operational response is reached close to the time of Possible
determination, which requires notification of the WUERM in addition to sufficient confidence that
contamination is possible. The time distribution of public notification corresponded closely with Credible
determination where the median time for public notification was 1,900 minutes, which was driven by
either cases of illness in the public or widespread absence of a chlorine residual in the distribution system
(as determined by WQM alerts or the results of chlorine residual testing in the distribution system as part
ofSC).
Possible Determination
Credible Determination
Confirmed Determination
Operational Response
Public Health Response
Public Notification
i
i 1 | [H n = 226
H^~]H n = 224
i
H
i 1 |H n=226
. n - A^fi
i n
= 349
n=455
0 2,000 4,000 6,000 8,000
Time Since Contaminant Injection (minutes)
Figure 8-17. Timeliness of Threat Level Determination and Responses for Contaminants with
Delayed Symptom Onset
69
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Figure 8-18 shows the median value of each timeline metric for all distribution system attack scenarios
that were detectable by one, two or three monitoring and surveillance components (CCS, PHS and
WQM). The figure shows a modest improvement in threat level determination and response times for
scenarios that can be detected by two components compared with those detectable by only one
component. A significantly larger incremental improvement in threat level determination and response
times was observed for scenarios that were detectable by three components. The largest improvement in
the threat level determination timeline occurred for the Credible and Confirmed determinations. This is
consistent with results from drills and exercises that indicate that information from multiple components
is necessary to conclude that contamination is Credible or Confirmed. The timing of operational response
is closely coupled with the time of Possible determination, and both of these timeline metrics improved
by approximately nine hours for scenarios detectable by three components compared with those
detectable by only one component. The median time for public notification was reduced from 21 hours
for scenarios detectable by only one component to around three hours for scenarios detectable by three
components. Public health response did not occur for scenarios that were detectable by only one
component but had a median time of approximately one and a half hours for scenarios detectable by three
components.
O)
IB
o
1
£ 1
.2
ro
Q.
ro
O
"ra
o
o
01
.c
c
Ol
o
Q.
E 3 -
o
O
5
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
that provides indicators of contamination through multiple independent data sources. This increased
confidence yields a more efficient and timely threat level determination process, which drives
implementation of response actions that can dramatically reduce consequences. The reduction in
consequences achieved through timely response actions is discussed in the next section.
8.3 Consequence Reduction
Definition: Consequences of a contamination incident that are avoided due to deployment and operation
ofaCWS.
Analysis Methodology: Consequence reduction was evaluated using the results from the full ensemble
of the simulation study. Three types of consequences are generated during simulations: illnesses,
fatalities and miles of pipe contaminated. Table 8-2 shows which consequences apply to each
contaminant and indicates the primary and secondary consequence categories for each.
Table 8-2. Primary and Secondary Consequence Types for Each Contaminant
Contaminant ID
Nuisance Chemical 1
Nuisance Chemical 2
Toxic Chemical 1
Toxic Chemical 2
Toxic Chemical 3
Toxic Chemical 4
Toxic Chemical 5
Toxic Chemical 6
Toxic Chemical 7
Toxic Chemical 8
Biological Agent 1
Biological Agent 2
Biological Agent 3
Biological Agent 4
Biological Agents
Biological Agent 6
Biological Agent 7
Consequence Type
Illnesses
n/a
n/a
S
P
P
S
S
S
S
S
P
S
S
S
S
S
S
Fatalities
n/a
n/a
P
S
S
P
P
P
P
P
S
P
P
P
P
P
P
Miles of Pipe
P
P
S
S
S
S
S
S
S
S
S
S
S
S
S
S
S
P - primary consequence, S - secondary consequence, n/a - no consequences in this category
Consequences were simulated for a baseline case without a CWS and for the case in which the full
Cincinnati CWS was in operation. The difference in consequences for these two cases represents the
reduction in consequences that is attributable to the Cincinnati CWS. For the case without a CWS,
individuals could still seek healthcare in response to their symptoms, and limited public health response
actions were assumed to occur that would mitigate consequences to some degree. For the case with the
CWS, consequences were further reduced through the following actions:
Improved public health response. Information from the CWS can result in an earlier and more
effective public health response, particularly if information from the CWS provides clues
regarding the identity of the causative agent.
71
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Operational response. The utility can alter the flow of water in the distribution system, which
can limit the spread of the contaminant, minimizing the number of individuals exposed to harmful
levels of the contaminant. However, operational responses can have unintended consequences,
such as maintaining a higher concentration of a contaminant in one area of the system, which
could expose individuals to more harmful concentrations of the contaminant.
Public notification. The utility, in cooperation with the local public health department, can issue
a notice to the public advising them to not use the drinking water, which will further limit
exposures to the contaminated water. Furthermore, the public notification could include guidance
to symptomatic individuals seeking effective healthcare.
Results: Figure 8-19 is a timeline and consequence graphic for a distribution system attack scenario
involving Biological Agent 4, which is representative of the timeline demonstrated by many of the
scenarios involving biological agents. It shows the key timeline metrics relative to the start of the
injection and the time series of fatalities for the case with and without the Cincinnati CWS in operation.
In this scenario, the injection occurred at high demand (9:00 a.m.), and WQM was the first component to
alert about eight hours after injection. A Possible determination was reached 41 minutes later. At this
point, an SC team was deployed and water quality field testing results were available at 3 hours and 5
minutes after Possible.
At 20 hours and 14 minutes after injection, a PHS Astute Clinician alert occurred, which elevated the
threat level to Credible and public notification was immediately issued. In all scenarios in the simulation
study, after the first PHS alert occurs, the communicator protocol is invoked, which initiates a
teleconference between the water utility and key representatives from public health agencies, which
represents the current operational strategy for the PHS component. In the scenario depicted in Figure 8-
19, the communicator teleconference contributed to raising the threat level to Confirmed 1 hour and 13
minutes after Credible determination. The public health response was initiated 27 hours and 46 minutes
after the injection. Laboratory results for the water sample were available 34 hours and 12 minutes after
Possible determination. While these results were available after the Confirmed determination, analytical
confirmation of the contaminant's identity would inform the later stages of response to an actual
contamination incident.
The time series of fatalities forthis simulated contamination scenario is also shown in Figure 8-19 with
and without the CWS in operation. The time delays for onset and progression of symptoms and fatalities
for this contaminant are in the range of 12 hours to 6 days, and in this scenario the first fatalities occurred
about 39 hours after injection and continued rising for six more days. In the absence of a CWS, there
would have been more than 1,600 fatalities as a result of this contamination incident. With the CWS in
operation, the number of fatalities was reduced to 20, close to a 99% reduction. This reduction in
consequences was largely attributable to the public notification being issued early in the response process,
which dramatically reduced the number of individuals exposed to the contaminant. Additionally,
prophylactic treatment provided as part of the public health response prevented a large number of
potential fatalities.
72
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
1,800
1,600
1,400
1,200
1,000
800 -
600 -
400 -
200 -
0
Day 1 08:42
Possible Time
Without CWS
Day 1 20:14
Credible Time
Day 1 21:27
Confirmed Time
Day 2 03:46
PH Response
Day 2 18:54
Lab Results
00:00
Day 1
Contaminant
Injection
Day 1 08:01
WQM Alert
With CWS
t
t
00:00
Day 4
00:00
Day 8
Day 1 20:14
PHS-AC Alert
Day 1 20:14
Public Notification
Day 1 11:47
WQ Parameters
Results
Figure 8-19. Timeline and Consequences for a Contamination Scenario Involving Biological Agent
4
Figure 8-20 is a timeline and consequence graphic for a distribution system attack scenario involving
Toxic Chemical 6, which is representative of the timeline demonstrated by many of the scenarios
involving toxic chemicals without a taste or odor. It shows the key timeline metrics relative to the start of
the injection and the time series of fatalities for the case with and without the Cincinnati CWS in
operation. In this scenario, the public health response occurred 1 hour and 27 minutes after injection but
before the first CWS alert due to an unusually high number of cases seen in the emergency department,
demonstrating that health departments will respond to an emerging health crisis even before the source of
the exposure is known. The PHS Astute Clinician alert occurred 3 hours and 6 minutes after the start of
the injection. Possible determination was reached 45 minutes later, following a teleconference between
the utility and public health agencies, during which it was concluded that contaminated water could be a
source of the exposure. The utility implemented an operational response 20 minutes after the Possible
determination in an attempt to contain the contaminated water. The first WQM alert occurred 4 hours and
47 minutes after the start of the injection, and the threat level was elevated to Credible in the next 32
minutes. The public notification was issued 32 minutes after the Credible determination. Subsequently, a
PHS-911 alert occurred and water quality field testing was conducted, which raised the threat level to
Confirmed 1 hour and 20 minutes later. Laboratory results for the water sample were available five hours
later. While these results were available after the Confirmed determination, analytical confirmation of the
contaminant's identity would inform the later stages of response to an actual contamination incident.
The time series of fatalities for this simulated contamination scenario is also shown in Figure 8-20 with
and without the CWS in operation. This contaminant has a short delay between exposure and onset of
symptoms and rapid illness progression, which contributed to fatalities occurring early in the scenario,
73
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
reaching about 750 in the first 10 hours. Moreover, the profiles for number of fatalities match closely for
run types with and without the CWS in operation due to the time required to gather sufficient information
during the investigation to conclude that contamination is Credible and subsequently issue public
notification. At the time when a majority of individuals were in compliance with the public notification
(approximately four hours after issuance of public notification), the number of fatalities stopped rising for
the case in which the CWS was in operation. For the case without the CWS, the number of fatalities
continued to rise until reaching about 1,750 approximately 36 hours after the start of the injection. In this
example, the CWS would have reduced fatalities by about 1,000, which is more than a 55% reduction.
This reduction in fatalities is primarily due to public notification, which sharply reduced the number of
exposures once most individual began to comply with the notification. In comparison to the scenario
described above for Biological Agent 4, a typical scenario involving a toxic chemical unfolds more
quickly, with several alerts occurring early in the scenario due to the rapid onset and progression of
symptoms.
Without CWS
With CWS
00:00
Day 1
Contaminant
Injection
01:27
PH Response
03:06
PHS-AC Alert
t
06:31
PHS-911 Alert
12:09
Lab Results
00:00
Day 2
06:02
WQ Parameters
Results
05:51
Public Notification
04:11
First Op Change
04:47
WQM Alert
Figure 8-20. Timeline and Consequences for a Contamination Scenario Involving Toxic Chemical
6
Figure 8-21 is a timeline and consequence graphic for a distribution system attack scenario involving
Toxic Chemical 4, which is representative of the timeline demonstrated by many of the scenarios
involving toxic chemicals with a taste or odor. It shows the key timeline metrics relative to the start of
the injection and the time series of fatalities for the case with and without the Cincinnati CWS in
operation. In this scenario, CCS was the first component to detect with FVR and work order (WO) alerts
occurring at 6 hours and 43 minutes and 7 hours and 16 minutes after the injection, respectively.
Contamination was considered Possible 26 minutes after the first CCS alert and was followed by an
operational response 10 minutes later. A PHS-DPIC alert occurred 8 hours and 28 minutes after the start
74
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
of the injection, which immediately raised the threat level to Credible. The PHS-DPIC alert was followed
by a PHS-911 alert three minutes later, which raised the threat level to Confirmed in the next 12 minutes.
Public notification was issued 26 minutes after Confirmed determination, with the delay due to the two
hours required to prepare a public notification following the determination that contamination is Possible.
Public health response occurred 21 minutes after the public notification. The SC and LA results were
available 2 hours and 27 minutes and 8 hours and 34 minutes after the Possible determination,
respectively. While these results were available after the Confirmed determination, analytical
confirmation of the contaminant's identity would inform the later stages of response to an actual
contamination incident.
The time series of fatalities for this simulated contamination scenario is also shown in Figure 8-21, with
and without the CWS in operation. The profiles for fatalities with and without the CWS in operation
match closely for the first 13 hours when the fatalities reached about 475. The fatalities with the CWS
reached 550 at about 16 hours after injection and then stopped rising, while the fatalities in the absence of
a CWS kept increasing until they reached about 850 at about 38 hours. In this scenario, the CWS would
have reduced fatalities by about 300, more than a 35% reduction. As with the previous scenario, the
exposures avoided due to public notification were the main driver for the reduction in consequences.
Without CWS
00:00
Day 1
Contaminant
Injection
06:43
CCS-IVR Alert
07:16
CCS-WO Alert
07:19
First Op Change
15:43
Lab Results
00:00
Day 2
09:36
WQ Parameters
Results
08:31
PHS-911 Alert
09:09
Public Notification
09:30
PH Response
Figure 8-21. Timeline and Consequences for a Contamination Scenario Involving Toxic Chemical
4
The three timelines shown are intended to provide insight into how a contamination scenario unfolds,
demonstrating timeline metrics and the primary consequences (fatalities) for three representative
contaminants.
75
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
In the remainder of this section, a statistical summary of results from the entire ensemble of simulated
contamination scenarios, excluding 100 scenarios in which there was no operational response or public
notification, will be presented to illustrate the reduction in health-related consequences that might be
realized under a wide range of scenarios. The 100 scenarios in which operational response or public
notification were not implemented were excluded because without these response actions, there is no
reduction in consequences attributable to utility response actions. The statistical summary of the
reduction in consequences (i.e., fatalities, illnesses, healthcare burden and miles of pipe contaminated)
attributable to the CWS is presented for individual contaminants in Figures 8-22 through 8-25. The
figures show the 10th, 25th, 50th, 75th and 90th percentiles for the reduction in consequence in the form of
box and whisker plots. Figures 8-22 through 8-24 are each broken up into three plots, each plot grouping
contaminants with a similar range in consequence reduction, and are arranged in decreasing order of
median consequence reduction. Note that in Figure 8-24, a reduction in the number of individuals
receiving healthcare is beneficial as it indicates that fewer individuals were exposed to the contaminant
(either through operational response or compliance with a public health notification) and, therefore, were
not in need of medical treatment.
76
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
in
9)
o
I
Figure 8-22. Reduction in Fatalities Attributable to the CWS
77
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
250,000
200,000
= 150,000
c
~ 100,000
o
3
HI
* 50,000
J
30,000
25,000
« 20,000
15,000
T3
HI
o:
10,000
5,000
-100
Figure 8-23. Reduction in Illnesses Attributable to the CWS
78
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
o>
u
4,500
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
led to high fatalities and consequently a greater opportunity for the CWS to reduce those fatalities.
Further, fatalities from exposure to Biological Agent 3 can be reduced through prophylactic treatment and
proper medical care, both of which are generally implemented sooner due to information provided
through the CWS. Although not as dramatic, the reduction in fatalities for Biological Agent 5 was also
large, with median and 90th percentile reductions around 4,000 and 15,500 respectively. The reductions
in fatalities for Toxic Chemicals 4 and 6 were in the hundreds for the majority of scenarios, and suggest
that contaminants with rapid symptom onset and progression result in similar numbers of fatalities with or
without a CWS in operation. In these cases, exposure to a lethal dose could result in rapid onset of
symptoms and death sooner than it would be possible to seek effective medical treatment. Scenarios
involving the ten contaminants shown in the bottom plot of Figure 8-22 had reductions in fatalities
generally less than 100; however, these contaminants tend to generate low numbers of fatalities even in
the baseline case, thus there is not much opportunity to further reduce fatalities in scenarios involving
these contaminants.
Figure 8-25 shows box and whisker plots for reductions of miles of pipe contaminated. The reduction in
miles of pipe contaminated represents areas of the distribution system that avoided contamination due to
operational responses implemented by the utility. Not only would this reduction in contaminant spread
reduce the number of potential exposures, it would also reduce the amount of pipe material (and number
of buildings and homes) that would need to be remediated. The figure shows no discernible trend with
contaminant type, which was expected given that operational responses are limited to a finite number of
control points in the system and depend on the area suspected of being contaminated and not the identity
of the contaminant. The largest reduction in consequences occurred for facility attack scenarios for which
the spread of contamination was reduced drastically due to quick detection by ESM and the operational
response that followed, which limited the spread of contaminated water. Consequence reduction was
negative for a few scenarios, indicating the CWS consequence was greater than the baseline case.
Incidents of negative consequence reduction are primarily due to a trade-off in the type of consequences
that are reduced by operational response actions. For example, operational responses may be
implemented that intend to limit the spread of contaminated water, which leads to a higher contaminant
concentration in the impacted area. This can result in more individuals in the contaminated area being
exposed to a lethal dose of the contaminant, rather than a diluted concentration that would have occurred
if no operational changes were made. However, in a vast majority of scenarios across all contaminants,
the reduction in miles of pipe contaminated was positive.
80
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
900
o
o
9-
b.
5
800
600
400
200
o
-200
X
Figure 8-25. Reduction in Miles of Pipe Contaminated Attributable to the CWS
8.4 Summary
Empirical data collected from the Cincinnati CWS pilot, including results from 21 drills and exercises,
were used to parameterize the Cincinnati CWS model, which was used to simulate a wide variety of
contamination scenarios with varying contaminants, injection locations, and injection times. The results
of the simulation study were used to quantify the timeliness and effectiveness of utility actions
implemented in response to more than 2,000 simulated contamination scenarios. WQM, CCS and PHS
all had median detection times of less than seven hours and the median time for Possible determination
for all distribution system attack scenarios was 5.5 hours and just under 9.5 hours for Confirmed
determination. Variability in the timeliness of PHS alerts was primarily driven by contaminant
characteristics, particularly the delay between exposure and symptom onset. The timing of CCS alerts
was driven by the injection time, with longer delays occurring for injections during low demand periods
(12:00 a.m.). The timing of WQM alerts was found to be independent of contaminant type, but strongly
81
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
dependent on the hydraulic travel time to the WQM station. Finally, it was observed that when multiple
components detect contamination, threat level escalation and implementation of response actions occurred
more quickly compared to scenarios where just one component detects contamination.
For contamination scenarios that produced a significant number of fatalities, the response stemming from
detection by the CWS resulted in a large reduction in the number of fatalities when compared to the same
scenario without CWS detection and response capabilities in place. Biological Agent 3 had the most
significant reduction in fatalities with a median reduction in fatalities of 20,076 and a maximum of
231,448. Conversely, there were not significant reductions in health-related consequences observed for
contaminants that did not result in significant health-related consequences in the baseline case.
82
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 9.0: Sustainability
A key design objective for the CWS is to develop a sustainable system that provides an acceptable
benefit-cost trade-off. The full cost of the CWS consists of two broad categories, capital expenditures to
deploy the system and O&M expenses to keep the system functioning for its lifespan. Together they
comprise the lifecycle cost of the system that can be compared with the benefits received over the lifespan
of the system to determine if it is sustainable. The primary benefit of a CWS is the potential reduction in
consequences in the event of a contamination incident; however, such a benefit may be rarely, if ever,
realized. Thus, dual-use benefits to a utility, which are unrelated to detecting and responding to
contamination incidents, will be an important driver for the Sustainability of the system. Ultimately, the
Sustainability of the system can be demonstrated through utility and partner organization compliance with
the protocols and procedures necessary to operate and maintain the CWS. To evaluate how well the CWS
met this design objective, the following three metrics were evaluated: net present value, dual-use benefits,
and willingness to maintain the CWS. The following subsections define each metric, describe how it was
evaluated, and present the results.
9.1 Net Present Value
Definition: The difference between the present value of benefits and the present value of costs.
Analysis Methodology: A financial analysis was performed to quantify in dollars the lifecycle cost of
the CWS and to similarly monetize the benefits that could be quantified in dollars using reasonable
assumptions. Because costs occur over the assumed 20-year lifecycle of the system, all monetary values
were adjusted to a common base year (2007) using a fixed rate of inflation and summed to represent the
present value (PV) of the costs. The PV of the benefits was calculated using a common base year (2007)
dollars. The benefit-cost analysis used these normalized costs to compute the net present value (NPV) of
the Cincinnati CWS by subtracting the PV of the costs from the PV of the benefits.
The costs for deploying and operating the Cincinnati CWS were thoroughly documented and considered
fixed in the benefit-cost analysis. However, there is more uncertainty regarding the benefits that would be
accrued under the lifecycle of the Cincinnati CWS. Thus, the benefit-cost analysis was performed under
two conditions: 1) assuming that a significant contamination incident occurred and 2) in the absence of a
contamination incident. Given that no contamination incidents occurred during the evaluation period of
the pilot, the Cincinnati CWS model, described in Appendix A, was used to estimate the consequences of
contamination incidents with and without a CWS in place, with the difference in consequences under
these two conditions providing an estimate of the reduction in consequences attributable to the CWS. The
monetary value of this reduction in consequences was determined using a variety of assumptions about
the costs to public health, water and wastewater utilities, and businesses served by the drinking water
utility. These assumptions are described in detail in Appendix B. For the condition in which no
contamination occurred, the benefit-cost analysis considered only those dual-use benefits that could be
reliably monetized.
Results: This section first presents the total cost of the Cincinnati CWS. Next, the net present value is
presented for the condition under which a significant contamination incident occurred, comparing the
benefits of consequence reduction with the lifecycle cost of the Cincinnati CWS. Finally, the net present
value is presented for the condition under which there is no contamination incident.
Table 9-1 presents the total cost of the Cincinnati CWS and each of its components broken out into the
following categories: deployment, O&M, renewal and replacement, and salvage value. Deployment costs
capture all labor costs for EPA, utility, and local partner personnel, as well as other direct charges for
83
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
equipment, consumables and purchased services necessary to design and install the system. It also
includes the costs (and savings) as a result of modifications to the system during the first year of
operation. Lifecycle O&M costs represent the present value of all of the costs that GCWW and local
partners incur each year to operate and maintain the CWS. Renewal and replacement costs represent all
costs associated with replacing major pieces of equipment during the 20-year lifespan of the Cincinnati
CWS based on the equipments' standard life expectancies. Finally, the salvage value is the estimated
residual value of the system components after 20 years of operation. Appendix B describes the data
sources and financial assumptions used to calculate the values presented in this table.
Table 9-1. Cost Elements used in the Calculation of Total Lifecycle Cost of the Cincinnati CWS
Cost
Element
Deployment
Costs
Lifecycle
O&M Costs
Renewal and
Replacement
Costs
Salvage
Value
Lifecycle
Cost
WQM
$4,229,000
$2,515,000
$1,556,000
($97,000)
$8,203,000
S&A
$2,544,000
$643,000
$260,000
($11,000)
$3,436,000
ESM
$1,389,000
$568,000
$257,000
($19,000)
$2,195,000
CM
$1,431,000
$548,000
$23,000
-
$2,001,000
PHS
$1,306,000
$241,000
$242,000
-
$1,788,000
CCS
$1,038,000
$84,000
$231,000
-
$1,353,000
Total
$11,936,000
$4,598,000
$2,569,000
($127,000)
$18,976,000
Note: Any discrepancies in totals by element or component are a result of rounding.
The total lifecycle cost of the Cincinnati CWS is approximately $19 million. As anticipated, deployment
cost ($11.9 million) is the largest element of the overall cost. The initial cost of equipment and contractor
services accounted for the majority of the deployment costs, but these costs also include the effort
required to optimize the system in the year following system deployment. The costs to operate and
maintain ($4.6 million) and replace equipment ($2.6 million) over a 20-year lifespan constitute 37% of
the overall cost of the system and are important expenditures to consider when deciding to deploy a CWS.
The salvage value provides a small offset (less than 1%) to the total cost of the CWS.
Analysis of cost by component shows that WQM was the most expensive component, accounting for 43%
of the total CWS costs. The deployment costs account for the majority (52%) of the total cost of the
WQM component, which is expected given that this is an equipment intensive component. The O&M
costs over the lifespan of the WQM component were also significant at 31% of the lifecycle cost for
WQM and 55% of the total O&M costs for the CWS. The least expensive component is CCS because it
was able to leverage existing call center software and capabilities at the utility, and therefore did not
require large expenditures for new equipment. The lifecycle cost for CM was greater than that of two of
the monitoring and surveillance components, PHS and CCS, even though CM required minimal
equipment. The deployment cost for CM was driven by the large number of individuals from a variety of
organizations who committed a great deal of time to development of Consequence Management Plan and
other associated documentation. O&M costs are also significant for CM and include the expense of
regularly exercising procedures, conducting drills and regularly updating documents.
The nature of the Cincinnati pilot is such that the cost of deploying this CWS are likely higher than those
that would be incurred for a utility deploying a similar system. One reason for this is that the Cincinnati
pilot was the first comprehensive CWS deployed, and the lack of previous experience to draw from
resulted in additional costs during system design. This first pilot was both a demonstration and a research
project; therefore, some aspects of the project were implemented to collect information about design
84
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
alternatives. Furthermore, the research aspects of the project resulted in a substantial effort to document
and evaluate system performance. Finally, this pilot was directly implemented by EPA and its contractors
in collaboration with GCWW and local partner organizations. This implementation approach introduced
substantial overhead costs that would not be incurred by a utility implementing a similar project
independently. For these reasons, the costs of the Cincinnati CWS should not be directly extrapolated to
projects at other utilities.
The benefits of the Cincinnati pilot were evaluated under the condition of a significant contamination
incident occurring during the lifecycle of the Cincinnati CWS. To perform this analysis, 30 simulated
contamination incidents were selected for a detailed cost analysis. Three scenarios were identified for
each of 10 contaminants, which were selected to represent the range of water distribution system
contamination threats. The contaminants include nuisance chemicals that do not cause acute health
consequences, moderately toxic chemicals, and highly potent biological agents. The three scenarios
evaluated for each contaminant were selected from a set of 119 scenarios to represent reductions in
consequences at the 25th, 50th and 75th percentiles. More information about the scenarios selected for the
benefit-cost analysis can be found in Section 3.4.
The reduction in consequences generated by the Cincinnati CWS model were used in conjunction with
the methodology described in Appendix B to develop estimates of the monetary value of the reduction in
consequences attributable to the CWS. For each of the 30 scenarios, Table 9-2 presents the reduction in
public health costs, lost revenue, distribution system remediation costs and total costs. Due to the
selection of scenarios with widely differing consequence reductions, the associated cost savings also vary
widely, from 0 to 414 billion dollars. With the exception of the three scenarios for Nuisance Chemical 1
and the 75th percentile scenario for Toxic Chemical 8, the reduction in the cost to public health was the
most significant benefit. The results for Nuisance Chemical 1 are expected given that it does not cause
acute health effects. For most scenarios, the value of the reduction in lost business revenue and
remediation costs were within one order of magnitude of each another, with the notable exception of the
75th percentile scenario for Toxic Chemical 8, which had the largest reduction in remediation costs among
the 30 scenarios. This is due to the expensive remediation techniques required to safely remove Toxic
Chemical 8 from a contaminated distribution system.
Table 9-2. Benefits Attributable to the Cincinnati CWS due to the Reduction in Consequences from
a Contamination Incident, in Millions of Dollars
Contaminant ID
Nuisance Chemical 1
CWS Model Analysis
25 Percentile
50 Percentile
75 Percentile
Public Health
$0
$0
$0
Revenue
$0
$3
$15
Remediation
$0
$4
$38
Total
$0
$6
$52
Toxic Chemical 1
25 Percentile
50 Percentile
75 Percentile
$1
$456
$932
$4
$4
$25
$1
$1
$6
$6
$462
$963
Toxic Chemical 5
25 Percentile
50 Percentile
75 Percentile
$0.0
$72
$425
$10
$0
$0
$4
$0
$0
$14
$72
$425
Toxic Chemical 6
25 Percentile
50 Percentile
$222
$2,556
$11
$43
$1
$6
$225
$2,605
85
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Contaminant ID
CWS Model Analysis
75 Percentile
Public Health
$5,736
Revenue
$27
Remediation
$7
Total
$5,770
Toxic Chemical 7
25 Percentile
50 Percentile
75 Percentile
$1
$252
$891
$18
$0
$0
$2
$0
$0
$21
$252
$891
Toxic Chemical 8
25 Percentile
50 Percentile
75 Percentile
$0
$252
$1,466
$0
$0
$106
$0
$0
$3,390
$0
$252
$4,962
Biological Agent 3
25 Percentile
50 Percentile
75 Percentile
$42,391
$145,008
$413,678
$2
$17
$26
$0.3
$1
$2
$42,393
$145,027
$413,706
Biological Agent 4
25 Percentile
50 Percentile
75 Percentile
$1,411
$9,785
$16,659
$0
$4
$17
$0
$1
$1
$1,411
$9,789
$16,677
Biological Agents
25 Percentile
50 Percentile
75 Percentile
$7,743
$30,093
$66,322
$0
$18
$36
$0
$3
$5
$7,743
$30,115
$66,364
Biological Agent 6
25 Percentile
50 Percentile
75 Percentile
$0
$14
$114
$0
$0
$0
$0
$0
$0
$0
$14
$114
Note: Zero values in this table are actual values, not a result of rounding.
Figure 9-1 shows the total monetary value of the reduction in consequences, as reported in Table 9-2, for
each of the ten contaminants. The bottom, middle, and top of each box corresponds to the 25th, 50th and
75th percentile scenarios for each contaminant, respectively. The ten box plots are divided among three
charts to allow the y-axis (i.e., monetary value of the benefit) to be scaled appropriately for each group of
contaminants. The chart on the top shows the five contaminants for which the lowest value was realized,
while the chart at the bottom right shows the two contaminants for which the greatest value was realized.
As a point of reference, the total lifecycle cost of the Cincinnati CWS is shown in Figure 9-1 as a red line.
86
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
$1,000
_ $900
2 $800
7 $700
I $600
9)
CO
"S
| $200
> $100
$500
$400
$300
S.
i/>
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
utility operations. Benefits must be monetized to be included in the benefit-cost analysis. While many
dual-use benefits were identified over the course of the evaluation period, only one could reliably be
monetized. This monetized benefit was a reduction in chlorine utilization that was realized through
information provided through the WQM component, which allowed utility operators to more accurately
dose chlorine to meet distribution system residual targets. The resulting cost savings was estimated to be
$4,410 per year. Assuming a steady cost for chlorine (i.e., its price increasing at the same rate as
inflation), the PV of the benefit over 20 years is $88,200. Subtracting this benefit from the PV of the
lifecycle cost of the CWS ($19 million) results in a large negative NPV, illustrating that in the absence of
a contamination incident, a strict financial analysis is insufficient to make a business case for the
Cincinnati CWS. However, there are significant dual-use benefits that cannot be monetized, which must
be considered when evaluating the sustainability of the CWS.
9.2 Dual-Use Benefits
Definition: Subjective valuation of benefits that are not the primary reason for the system's deployment.
Analysis Methodology: Information collected from forums such as routine component review meetings,
lessons learned workshops, and interviews were used to identify dual-use applications of the Cincinnati
CWS. Section 3.5 provides a summary description of these data collection forums.
Results: The Cincinnati CWS resulted in benefits to GCWW's routine operations that go beyond the
detection of contamination incidents. Table 9-3 shows the dual-use benefits of the Cincinnati CWS
identified by GCWW and partner organization personnel over the evaluation period of the pilot. None of
these benefits could be quantified in a way that could be translated into a cost savings, and therefore
required qualitative judgment regarding the value of the benefit provided. The benefits identified for the
system and its components were grouped into the seven broad categories shown in Table 9-3. While non-
monetizable, these benefits provide significant value to the utility and partner organizations, and thus to
the customers served by the utility.
The total lifecycle cost of the Cincinnati CWS expressed as an annual cost is $1.28 million. Considering
that GCWW supplies water to approximately 1.1 million people, the cost of the CWS is just higher than
$1 per person per year. Given the significant value that the dual-use benefits of the CWS provides to the
utility and its customers, it seems that a cost of $1 per customer per year could be justified.
88
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Table 9-3. Dual-use Benefits of the Cincinnati CWS
Benefit
Ability to detect and respond to a wide
range of distribution system water quality
issues
Improved knowledge of distribution
system water quality
Information to support activities related to
regulatory compliance
Potential cost savings in operation and
capital improvement
Improved coordination and
communication within the utility and with
external partner organizations
Improved relationship among public health
agencies
Increased public confidence in the water
supply
"All-hazards" preparedness
Description
A CWS allows the utility to quickly identify, diagnose and
respond to undesirable water quality conditions resulting from
operations or other activities not initiated by the utility, such as
hydrant flushing by the fire department, thereby minimizing the
impact on the customer.
A CWS provides the utility with nearly continuous information
that can be used to develop an improved understanding of the
water quality throughout the distribution system as it varies by
time and location.
A CWS supports compliance with drinking water regulations by
providing spatial and temporal data about water quality, which
enables a prompt response to developing water quality issues
before they become compliance issues. Additionally, it
provides information that can be used to assess the impact of
potential future regulations on utility operations.
The monitoring components of a CWS provide the utility with
data that can be used to modify operations for more efficient
use of chemicals and power resulting in cost savings.
Additionally, the data can be used to evaluate potential capital
improvement projects intended to improve distribution system
water quality and operations.
Implementation of a CWS requires active participation from
many divisions within the utility and from external partner
agencies, such as public health agencies, police, fire
(including HazMat), etc., which improves coordination and
communication during both routine activities as well as
emergency situations.
Public health agencies that participated in the CWS improved
relationships not only with GCWW but also with each other.
The CWS demonstrates to the public the utility's efforts to
provide a consistent, high quality product, thereby indicating its
commitment to public health, resulting in improving the public
confidence in the quality of their drinking water.
The CWS monitoring components, response infrastructure,
and experience gained during drills and exercises can be
utilized by the utility and its partner agencies to more efficiently
and effectively monitor and respond to any emergency, such
as natural disasters, public health emergencies, non-water
related terrorist attacks, etc.
WQM
X
X
X
X
X
X
X
S&A
X
X
X
X
X
X
ESM
X
X
X
PHS
X
X
X
X
X
CCS
X
X
X
X
X
X
CM
X
X
X
X
X
89
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
9.3 Willingness to Maintain the CWS
Definition: Behaviors that demonstrate the willingness and intent of an organization to maintain the
CWS.
Analysis Methodology: The percentage of alerts that were investigated was tracked and used as a
measure of the willingness of persons and organizations to monitor and maintain the CWS. Additionally,
participation in drills, exercises, and other forums was tracked, which was used as a measure of the
willingness of persons and organizations to participate in the CWS.
Results: Figure 9-2 shows the percentage of alerts investigated relative to the number of alerts that
occurred during each monthly reporting period over the entire evaluation period. Prior to June 2009,
GCWW and local partner personnel were not expected to fully investigate all alerts because the rate of
invalid alerts was deemed too high and the CWS was still being optimized to reduce the rate of invalid
alerts. For this reason, the alert investigation rate was low during the first 14 months of the evaluation
period; however, it gradually increased as the pilot transitioned to the real-time monitoring phase of the
evaluation period.
I
§7 no/ .
c
Q
S? \C? \<^ \C? \<^ \<^ ^ \*^ \^ \*^ \*^ \*^
. h$y. ^. i^N. K$y. ^. K$y. h$y.h$y. i^y K$y K$y ^.^.^^.NS5,^,^,^^,^ ^ ^ ^^v^^^^
Start Date of Monthly Reporting Period
Figure 9-2. Percentage of CWS Alerts Investigated and the Number of Alerts Received
The alert investigation rate exceeded 90% during four of the last five reporting periods. The trend of
increasing alert investigation rates correlates with improvement in the quality of the underlying data that
generates alerts, thus reducing the number of invalid alerts as shown by the red diamonds in Figure 9-1.
As discussed in Section 7.1, the optimization efforts that had the most significant impact on alert rates
include improved performance of equipment and adjustment of alert thresholds to more accurately reflect
normal system variability. Additionally, as more users became proficient with alert investigation
procedures through exercises and training, the time and effort required to investigate alerts decreased,
resulting in increased alert investigation rates. GCWW also reports that the number of alerts and the time
90
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
required to investigate each alert has continued to decrease even three years after the end of the evaluation
period.
Drills, exercises, component meetings, and lessons learned workshops were conducted routinely. These
events were used to evaluate key aspects of component and system performance for particular activities or
scenarios that could not be characterized via routine operations. Over the course of the evaluation period,
21 drills and exercises were conducted that collectively covered every component of the CWS.
Participation in these drills and exercises by GCWW personnel and local partners was 100% for most
activities. The continual participation in exercises indicates acceptance of the tools and procedures
associated with the CWS.
Component meetings, including the PHS Users Group meetings, were initially conducted on a weekly to
monthly basis early in the component development process. Once implementation was complete, the
meeting schedule was adjusted to monthly or quarterly. Lessons learned workshops provided an open
forum to gain feedback on the performance, operation and sustainability of components during the
evaluation period. Personnel expressed specific feedback regarding the strengths and weakness of each
surveillance tool in the context of their effectiveness in identifying possible contamination incidents. In
general, GCWW and local partner personnel exhibited a high degree of participation and interaction in
these forums, demonstrating a high degree of commitment to the project.
More than three years after completion of the WSI pilot, GCWW and local partners continue to operate
and maintain the CWS. Engagement with external partners has also continued, as evidenced by
continuation of PHS Users Group meetings twice a year and plans to conduct another full-scale exercise.
Furthermore, GCWW has plans to upgrade the WQM component by standardizing the instruments at the
existing WQM stations to a suite of instruments considered most valuable and sustainable. They are also
considering the addition of more WQM locations and continue to pilot new sensor technologies.
9.4 Summary
A benefit-cost analysis was performed to evaluate whether the monetized benefits of a CWS were greater
than the total lifecycle cost of the Cincinnati pilot ($18,976,000). If a contamination event occurs, the
consequences can be significant, justifying the investment in a CWS. Thirty scenarios were evaluated
under the benefit-cost analysis, of which 73% had monetized benefits that exceeded the total lifecycle
cost of the CWS. Of the scenarios in which the benefits exceeded the CWS cost, half realized benefits
that were valued at more than lOOx the cost of the CWS. The primary driver of monetized benefits for
most scenarios was the reduction in public health consequences of water contamination. While these
results make a compelling business case for deployment of a CWS, the probability of contamination is
unknown, but presumably very low.
Thus, the business case for deploying a CWS may rely on the value of dual-use benefits that were realized
through the Cincinnati CWS. For example, GCWW was able to utilize WQM sensors to optimize
chlorine residuals throughout the distribution system, reducing the overall chlorine dose and associated
costs. Several non-monetizable benefits were realized across multiple CWS components including the
ability to detect a wide range of distribution system water quality issues. Additionally, the Cincinnati
CWS demonstrated benefits to business practices, such as improved communication and coordination
within the utility and with its external partners. Overall, the investment in the CWS improved the
response posture of GCWW and the local partners for "all-hazards," which was demonstrated in
GCWW's response to the consequences of Hurricane Ike.
Management and personnel from GCWW and local partners demonstrated a strong willingness to
maintain the CWS beyond the pilot. This was demonstrated in the high rate of alert investigations,
91
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
greater than 90%, after the CWS was optimized. Furthermore, active participation in drills and exercises
indicated a willingness of utility and response partner personnel to adopt the CWS components and
procedures. Finally, the utility is considering upgrading the WQM component and continues to engage
local partners through the Public Health Users Group.
92
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 10.0: Summary and Conclusions
The evaluation of the Cincinnati CWS involved analysis of empirical data, observations from drills and
exercises, results from modeling and simulations, qualitative observations gleaned from participants
during forums, and a benefit-cost analysis. A set of performance metrics was defined for each of six
design objectives, and results were presented showing how well the Cincinnati CWS performed relative
to each metric. Highlights, limitations, and considerations for interpretation of this analysis are presented
in this section.
10.1 Highlights of Analysis
Evaluation of the Cincinnati pilot produced a comprehensive assessment of the multi-component CWS
design deployed under WSI. Through layers of redundancy built into the CWS and each of its
components, the system achieved a high degree of operational reliability during the two-year evaluation
period, with 95% data completeness and at least three of the four monitoring and surveillance components
available >99% of the time. There was minimal multi-component downtime with the longest period
involving two components concurrently down for 26 hours, and three components concurrently down for
eight hours. While these periods of multi-component downtime may impact the timeliness of detection,
they would not likely impact the overall ability of the CWS to detect a contamination incident, given that
a detailed analysis of 30 simulated contamination scenarios in the simulation study showed that on
average, contaminated water would remain in the distribution system at detectable levels for 5.3 days.
The multi-component Cincinnati CWS achieved comprehensive contaminant and spatial coverage by
monitoring a variety of data streams and locations throughout GCWW's distribution system. Results
from a simulation study demonstrated a 98% detection rate for 2,015 simulated contamination scenarios
involving a broad range of contaminant types (i.e., nuisance chemicals, toxic chemicals, and biological
agents). The majority of the 44 scenarios that were not detected by the CWS involved a contaminant that
does not cause acute health effects and is detectable by only a single component. This result emphasizes
the value of a multi-component CWS, in which the detection capabilities of the monitoring and
surveillance components are complementary and provide broad contaminant coverage. For example,
while WQM covers only 72% of the distribution system area, it provides reliable detection capability for
a wide range of chemical and biological agents. In comparison, CCS covers 100% of the distribution
system area, but is able to detect only contaminants that cause a discernible taste or odor in water. Thus,
the capabilities and limitations of the components balance out to provide a robust monitoring and
surveillance system with broad spatial and contaminant coverage.
Results from the simulation study demonstrate that multiple components would generate alerts that are
spatially and temporally related during a contamination incident. Co-occurring alerts from multiple
components can increase a utility manager's confidence that the alerts are valid and indicative of a
potential water quality issue. Different contaminant types such as nuisance chemicals or those with rapid
or delayed symptom onset trigger different combinations of component alerts, and the timing of those
alerts occur in predictable patterns. The co-occurrence of two alerting components, especially the
combination of PHS and WQM, was frequent in both simulated and empirical data. A combination of
three components alerting was observed only once in the empirical data; however, alert clusters involving
three or more components was common in the simulation study results. This would suggest that valid
alert clusters involving alerts from multiple components are probably the result of a real water quality
issue in the distribution system.
During real-time operation, most alerts were determined to be invalid; however, the CWS did detect 84
valid alerts involving main breaks, minor treatment process upsets, non-standard system operations or
93
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
public health incidents that were unrelated to drinking water. Although invalid alerts initially occurred
frequently, with more than 150 alerts during most reporting periods in the first year of operation, once the
system was optimized by improving the quality of the underlying data and updating event detection
system configurations to reflect normal variability, the number of invalid alerts was reduced to about 69
per reporting period.
A multi-component CWS also increases the timeliness of detection and response during a possible
contamination incident. Results from the simulation study show median detection times less than seven
hours for WQM, CCS and PHS, while ESM typically detected the incident before the start of contaminant
injection. During the investigation, the median time for Possible determination was 5.5 hours and just
under 9.5 hours for Confirmed determination. It was observed that when multiple components detect
contamination, threat level escalation and implementation of response actions occurred more quickly
when compared to scenarios in which just one component detects contamination. Timely detection and
threat level determination lead to quicker implementation of response actions and a significant reduction
in consequences.
10.2 Limitations of the Analysis
The fact that the CWS deployed in Cincinnati was the first of its kind has several implications for the
evaluation presented in this report. Important considerations included:
This was a pilot project and thus a variety of equipment, instrumentation and software
applications were relatively novel when implemented. Some of the equipment that was installed
proved unreliable and required an unsustainable level of effort to maintain. For some
components, significant trial and error was necessary to achieve acceptable performance.
Improved products are now available. In many cases, the Cincinnati pilot was the first real-time
installation of hardware and software products for this specific application. Thus, many issues
were encountered and resolved, and these improvements have been incorporated into many
commercially available products. In addition, the increased awareness of the CWS application
has motivated vendors to make their products more effective and reliable to implement.
The planning and implementation approach, in which EPA took the lead role, was inefficient.
Utility-led planning could potentially alleviate various pitfalls observed during implementation of
the Cincinnati CWS and better leverage existing systems.
While an extensive amount of data from a variety of sources was available for evaluation of the
Cincinnati pilot, there were some limitations of the analysis. Data completeness for the evaluation was
relatively high, but there were some gaps in data collection. Specifically, some water quality data was
lost during periods in which the data communications system was down. Also, there were some instances
in which alert investigation checklists were incomplete or missing.
As noted earlier, no known contamination incidents occurred during the evaluation period of the
Cincinnati pilot. Thus, it was necessary to use results from computer simulations of contamination
incidents to evaluate certain performance metrics. While these simulations were very detailed and the
supporting models were parameterized using data from real-world observations, the model is still only an
approximation. Thus, the results of the simulation study should be considered only in the context of the
design and assumptions intrinsic to the study.
94
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
10.3 Potential Applications of the Cincinnati CWS
The Cincinnati CWS was tailored to the capabilities and structure of GCWW and its local partners;
therefore, extrapolation to other utilities should be performed carefully. However, the Cincinnati CWS
revealed numerous applications and lessons that can be applicable to other CWSs.
The CWS design and implementation approach used in Cincinnati is just one of many possibilities.
Based on the results presented here, and capabilities of other cities, it may be possible to refine elements
of the design to reduce deployment costs while still achieving the utility's specific objectives. The results
of the evaluation presented here may aid other utilities seeking to improve existing capabilities or add
additional functionality as part of an effective CWS. Many utilities have existing capabilities that can be
leveraged to build an effective CWS at a much lower cost than was incurred for the Cincinnati CWS.
At the start of the pilot, there was concern that the monitoring and surveillance components would
generate too many alerts and that eventually these alerts would be largely ignored. In the early stages of
the pilot, this was the case. However, once the system had been optimized to reduce the occurrence of
invalid alerts, investigation rates approached 100%, indicating that the alert rate was acceptable to
personnel responsible for monitoring the system. Furthermore, some staff members observed that the
data and alerts generated by the system provided a deeper understanding of the impact of system
operations on distribution system water quality. In addition, water quality anomalies and public health
incidents not caused by contamination have been detected. This demonstrates that real-time monitoring
and surveillance can provide valuable information for day-to-day operations.
Analysis of the simulation study results emphasizes the value of a multi-component CWS, in which the
detection capabilities of the monitoring and surveillance components are complementary and provide
broad contaminant coverage. With respect to the CWS design objectives (i.e., spatial coverage,
contaminant coverage and timeliness of detection), limitation in the capabilities of one component are
offset by the strengths of another. Additionally, co-occurring alerts from multiple components can
increase the utility's confidence that the alerts are valid and indicative of a potential water quality issue.
The Cincinnati CWS demonstrated benefits to business practices, such as improved communication and
coordination within the utility and with its external partners. Improved communication strategies and
documented response procedures developed for the Cincinnati CWS are widely applicable to a variety of
situations. In particular, the ability to respond to a wide variety of hazards, including extreme weather
events such as the Hurricane Ike windstorm, is enhanced by CWS capabilities. Furthermore, these
procedures are highly portable and can be adapted to meet the specific needs of a variety of applications.
Given that improved communication and response protocols are relatively inexpensive to implement, they
should be considered as one cost-effective strategy for improving any utility's monitoring and response
capabilities. The overall success of a CWS depends not only on reliable data, but also requires the
commitment of utility personnel and external partners who are aware of the possible causes of changes in
observed water quality data, customer complaints or trends in public health data. Periodic drills and
exercises can be an effective means of maintaining this commitment and knowledge.
The overarching goal of the CWS is to improve situational awareness such that potential water quality
issues in the distribution system can be quickly detected and proactively addressed. The Cincinnati pilot
demonstrated that this can be achieved through a multi-component monitoring and surveillance system
combined with "all-hazards" response planning.
95
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 11.0: References
Allgeier, S.C., Hall, J., Rahman, M., and Coates, W. 2010. "Selection of Water Quality Sensors for a
Drinking Water Contamination Warning System," Proceedings of2010AWWA Water Quality
Technology Conference, Savannah, GA.
Bertakis, Klea D., et al. 2000. "Gender differences in the utilization of health care services." Journal of
Family Practice, 49.2: 147-152.
Davis, M.J. and Janke, R. 2009. "Development of a Probabilistic Timing Model for the Ingestion of Tap
Water." Journal oj'Water Resources Planning and Management, 135(5), 397-405.
Davis, M.J. and Janke, R. 2011. "Patterns in Potential Impacts Associated with Contamination Events in
Water Distribution Systems." Journal of Water Resources Planning and Management, 137(1),
1-9.
Davis, M. J., Janke, R., and Magnuson, M. L. 2013. "A Framework for Estimating the Adverse Health
Effects of Contamination Events in Water Distribution Systems and its Application." Risk
Analysis, doi: 10.1111/risa.l2107
Hall, J., et al. 2007. "Online Water Quality Parameters as Indicators of Distribution System
Contamination." JournalAWWA. Vol. 99, Issue 1: 66-77'.
Schappert, Susan M., and C. W. Burt. 2006. "Ambulatory care visits to physician offices, hospital
outpatient departments, and emergency departments: United States, 2001-02." Vital and Health
Statistics. Series 13, Data from the National Health Survey 159:1.
U.S. Environmental Protection Agency. 1997. Exposure Factors Handbook. EPA 600-P-95-002F.
U.S. Environmental Protection Agency. 2005. WaterSentinel System Architecture, EPA 817-D-05-003.
U.S. Environmental Protection Agency. 2007. Dermal Exposure Assessment: A Summary of EPA
Approaches, EPA 600-R-07-040F.
U.S. Environmental Protection Agency. 2008. Cincinnati Pilot Post-Implementation System Status, EPA
817-R-08-004.
U.S. Environmental Protection Agency. 2014a. Water Security Initiative: Comprehensive Evaluation of
the Cincinnati Contamination Warning System Pilot EPA 817-R-14-001.
U.S. Environmental Protection Agency. 2014b. Water Security Initiative: Evaluation of the Enhanced
Security Monitoring Component of the Cincinnati Contamination Warning System Pilot, EPA
817-R-13-001C.
U.S. Environmental Protection Agency. 2014c. Water Security Initiative: Evaluation of the Water
Quality Monitoring Component of the Cincinnati Contamination Warning System Pilot, EPA
817-R-13-001B.
96
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
U.S. Environmental Protection Agency. 2014d. Water Security Initiative: Evaluation of the Customer
Complaint Surveillance Component of the Cincinnati Contamination Warning System Pilot, EPA
817-R-13-001D.
U.S. Environmental Protection Agency. 2014e. Water Security Initiative: Evaluation of the Public
Health Surveillance Component of the Cincinnati Contamination Warning System Pilot, EPA
817-R-13-001E.
U.S. Environmental Protection Agency. 2014f. Water Security Initiative: Evaluation of the Water
Consequence Management Component of the Cincinnati Contamination Warning System Pilot,
EPA817-R-13-001F.
U.S. Environmental Protection Agency. 2014g. Water Security Initiative: Evaluation of the Sampling
and Analysis Component of the Cincinnati Contamination Warning System Pilot, EPA 817-R-13-
001G.
U.S. Environmental Protection Agency. EPANET.
U.S. Environmental Protection Agency. Water Contaminant Information Tool.
97
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 12.0: Abbreviations
AWWA American Water Works Association
CCS Customer Complaint Surveillance
CI Confidence Index
CL2 Free Chlorine Residual
CM Consequence Management
COND Conductivity
CPI Consumer Price Index
CWS Contamination Warning System
DPIC Drug and Poison Information Center
ED Emergency Department
EPA Environmental Protection Agency
EMS Emergency Medical Service
ESM Enhanced Security Monitoring
FSE Full-Scale Exercise
GCWW Greater Cincinnati Water Works
HazMat Hazardous Materials
IT Information Technology
IVR Interactive Voice Response
LA Laboratory Analysis
NPV Net Present Value
O&M Operations and Maintenance
ORP Oxidation Reduction Potential
PHS Public Health Surveillance
PV Present Value
QC Quality Control
RFT Rapid Field Test
S&A Sampling and Analysis
SC Site Characterization
SCADA Supervisory Control and Data Acquisition
TLD Threat Level Determination
TOC Total Organic Carbon
WO Work Order
WQM Water Quality Monitoring
WQ&T Water Quality and Treatment
WSI Water Security Initiative
WUERM Water Utility Emergency Response Manager
98
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Section 13.0: Glossary
Alert. Information from a monitoring and surveillance component indicating anomalous conditions that
warrant further investigation to determine if the alert is valid.
Alert investigation. A systematic, documented process for determining whether or not an alert is valid,
and identifying the cause of the alert. If an alert cause cannot be identified, contamination is Possible.
Anomaly. Deviations from an established baseline. For example, a water quality anomaly is a deviation
from typical water quality patterns observed over an extended period.
Baseline. Normal conditions that result from typical system operation. The baseline includes predictable
fluctuations in measured parameters that result from known changes to the system. For example, a water
quality baseline includes the effects of draining and filling tanks, pump operation, and seasonal changes
in water demand, all of which may alter water quality in a somewhat predictable fashion.
Benefit. An outcome associated with the implementation and operation of a contamination warning
system that promotes the welfare of the utility and the community it serves. Benefits can be derived from
a reduction in the consequences of a contamination incident and from routine operations.
Benefit-cost analysis. An evaluation of the benefits and costs of a project or program, such as a
contamination warning system, to assess whether the investment is justifiable considering both financial
and qualitative factors.
Biotoxins. Toxic chemicals derived from biological materials that pose an acute risk to public health at
relatively low concentrations.
Box-and-whisker plot. A graphical representation of nonparametric statistics for a dataset. The bottom
and top whiskers represent the 10th and 90th percentiles of the ranked data, respectively. The bottom and
top of the box represent the 25th and 75th percentiles of the ranked data, respectively. The line inside the
box represents the 50th percentile, or median, of the ranked data. Note that some data sets may have the
same values for the percentiles presented in box-and-whisker plots, in which case some lines will not be
visible.
Bulk concentration (of contaminant). The concentration of a contaminant solution that is injected into
the distribution system during a contamination scenario.
Bulk volume (of contaminant). The total volume of a contaminant solution that is injected into the
distribution system during a contamination scenario.
Confidence index. In the Cincinnati contamination warning system model, a quantitative indicator of the
reliability of the data used in the threat level determination process. The confidence index is calculated
for each of the four monitoring & surveillance component, site characterization and laboratory analysis.
Confirmed. In the context of the threat level determination process, contamination is Confirmed when
the analysis of all available information from the contamination warning system has provided definitive,
or nearly definitive, evidence of the presence of a specific contaminant or class of contaminant in the
distribution system. While positive results from laboratory analysis of a sample collected from the
distribution system can be a basis for confirming contamination, a preponderance of evidence, without the
benefit of laboratory results, can lead to this same determination.
99
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Consequence management. Actions taken to plan for and respond to Possible contamination incidents.
This includes the threat level determination process, which uses information from all monitoring and
surveillance components as well as sampling and analysis to determine if contamination is Credible or
Confirmed. Response actions, including operational changes, public notification, and public health
response, are implemented to minimize public health and economic consequences, and ultimately return
the utility to normal operations.
Consequence management plan. Documentation that provides a decision-making framework to guide
investigative and response activities implemented in response to a possible contamination incident.
Contamination incident. The introduction of a contaminant in the distribution system with the potential
to cause harm to the utility or the community served by the utility. A contamination incident may be
intentional or accidental.
Contamination scenario. Within the context of the simulation study, parameters that define a specific
contamination incident, including: injection location, injection rate, injection duration, time the injection
is initiated and the contaminant that is injected.
Contamination warning system. An integrated system of monitoring and surveillance components
designed to detect contamination in a drinking water distribution system. The system relies on integration
of information from these monitoring and surveillance activities along with timely investigative and
response actions during consequence management to minimize the consequences of a contamination
incident.
Costs, implementation. Installed cost of equipment, IT components, and subsystems necessary to
deploy an operational system. Implementation costs include labor and other expenditures (equipment,
supplies, and purchased services).
Cost, lifecycle. The total cost of a system, component, or equipment over its useful or practical life.
Lifecycle cost includes the cost of implementation, operation & maintenance and renewal & replacement.
Costs, operation & maintenance. Expenses incurred to sustain operation of a system at an acceptable
level of performance. Operational and maintenance costs are reported on an annual basis, and include
labor and other expenditures (supplies and purchased services).
Costs, renewal & replacement. Costs associated with refurbishing or replacing major pieces of
equipment (e.g., water quality sensors, laboratory instruments, IT hardware) that reach the end of their
useful life before the end of the contamination warning system lifecycle.
Coverage, contaminant. Specific contaminants that can potentially be detected by each monitoring and
surveillance component, as well as sampling & analysis, of a contamination warning system.
Coverage, spatial. The areas within the distribution system that are monitored by, or protected by, each
monitoring and surveillance component of a contamination warning system.
Credible. In the context of the threat level determination process, a water contamination threat is
characterized as Credible if information collected during the investigation of Possible contamination
corroborates information from the validated contamination warning system alert.
100
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Data completeness. The amount of data that can be used to support system or component operations,
expressed as a percentage of all data generated by the system or component. Data may be lost due to
quality control failures, data transmission errors, and faulty equipment among other causes.
Distribution system attack scenarios. A simulated contamination incident in which the injection
occurred at a distribution system node (and not at a utility facility). For every contaminant, one
distribution system attack node was selected as an injection location for each of the 94 pito zones to
ensure that the spatial extent of the distribution system was represented.
Distribution system model. A mathematical representation of a drinking water distribution system,
including pipes, junctions, valves, pumps, tanks, reservoirs, etc. The model characterizes flow and
pressure of water through the system. Distribution system models may include a water quality model that
can predict the fate and transport of a material throughout the distribution system.
Dual-use benefit. A positive application of a piece of equipment, procedure, or capability that was
deployed as part of the contamination warning system, in the normal operations of the utility.
Ensemble. The comprehensive set of contamination scenarios evaluated during the simulation study.
Event detection system. A system designed specifically to detect anomalies from the various monitoring
and surveillance components of a contamination warning system. An event detection system may take a
variety of forms, ranging from a complex set of computer algorithms to a simple set of heuristics that are
manually implemented.
Evaluation period. The period from January 16, 2008 to June 15, 2010 during which data was actively
collected for the evaluation of the Cincinnati contamination warning system pilot.
Facility attack scenarios. A simulated contamination incident in which the injection occurred at a utility
facility (e.g., a distribution system storage tank or a pump station). The injection node set for the facility
attack nodes included all GCWW facilities in the retail portion of the distribution system.
Flow rate. The volume of water moving past a fixed location per unit time.
Hydraulic connectivity. Points or areas within a distribution system that are on a common flow path.
Incident timeline. All significant activities that occur during a contamination incident, beginning with
the start of contaminant injection. Elements of the incident timeline include: time for detection, time for
alert investigation, time for threat level determination and time to implement response actions.
Injection duration. The cumulative time over which the bulk volume of a contaminant is injected into
the distribution system at a specific location for a given scenario within the simulation study.
Injection location. The specific node in the distribution system model where the bulk contaminant is
injected into the distribution system for a given scenario within the simulation study.
Injection rate. The mass flow rate at which the bulk volume of a contaminant is injected into the
distribution system at a specific location for a given scenario within the simulation study, in units of
mg/min or organisms/min.
101
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Interactive voice response. An automated call management system that transfers utility customer calls
to designated customer service representatives based on customer selected issues such as billing or water
quality problems.
Invalid alert. An alert from a monitoring and surveillance component that is not due to an anomaly and
is not associated with an incident or condition of interest to the utility.
Investigative component. Site characterization and laboratory analysis activities implemented as part of
the threat level determination process for the purpose of determining if contamination is Credible or
Confirmed, and for identifying the contaminant.
Metric. A standard or statistic for measuring or quantifying the performance of the contamination
warning system or its components.
Model. A mathematical representation of a physical system.
Model parameters. Fixed values in a model that define important aspects of the physical system.
Module. A sub-component of a model that typically represents a specific function of the real-world
system being modeled.
Monetizable. A cost or benefit whose monetary value can be reliably estimated from the available
information.
Monitoring & surveillance component. Element of a contamination warning system used to detect
unusual water quality conditions, including possible contamination incidents. The four monitoring &
surveillance components of a contamination warning system include: 1) water quality monitoring, 2)
enhanced security monitoring, 3) customer complaint surveillance and 4) public health surveillance.
Net present value. The difference between the present value of benefits and costs, normalized to a
common year.
Node. A mathematical representation of a junction between two or more distribution system pipes, or a
terminal point in a pipe in a water distribution system model. Water may be withdrawn from the system
at nodes, representing a portion of the system demand.
Nuisance chemicals. Chemical contaminants with a relatively low toxicity, which therefore generally do
not pose an immediate threat to public health. However, contamination with these chemicals can make
the drinking water supply unusable.
Optimization phase. Period in the contamination warning system deployment timeline between the
completion of system installation and real-time monitoring. During this phase, the system is operational
but not expected to produce actionable alerts. Instead, this phase provides an opportunity to learn the
system and optimize performance (e.g., fix or replace malfunctioning equipment, eliminate software bugs,
test procedures and reduce occurrence of invalid alerts).
Pathogens. Microorganisms that cause infections and subsequent illness and mortality in the exposed
population.
102
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Pito zone. An area of the Greater Cincinnati Waterworks distribution system in which water quality and
pressure are fairly constant. There are 94 pito zones in the Greater Cincinnati Water Works distribution
system model, ranging in area from 0.3 to 15 square miles.
Possible. In the context of the threat level determination process, a water contamination threat is
characterized as Possible if the cause of a validated contamination warning system alert is unknown.
Priority contaminant. A contaminant that has been identified by EPA as a monitoring target under the
Water Security Initiative. Priority contaminants may be initially detected through one of the monitoring
and surveillance components and Confirmed through laboratory analysis of samples collected during the
investigation of a Possible contamination incident.
Public health incident. An occurrence of disease, illness or injury within a population that is a deviation
from the disease baseline in the population.
Public health response. Actions taken by public health agencies and their partners to mitigate the
adverse effects of a public health incident, regardless of the cause of the incident. Potential response
actions include: administering prophylaxis, mobilizing additional healthcare resources, providing
treatment guidelines to healthcare providers and providing information to the public.
Public notification. A publicly released statement that includes a directive to utility customers, such as
boil-water before use, do-not-drink the water, or do-not-use the water. The notification is prepared by the
water utility and health department, and provided to media outlets to broadcast to the public when the
safety of drinking water has been compromised.
Radiochemicals. Chemicals that emit alpha, beta, and/or gamma particles at a rate that could pose a
threat to public health.
Real-time monitoring phase. Period in the contamination warning system deployment timeline
following the optimization phase. During this phase, the system is fully operational and utility personnel
and partners respond to alerts in real-time and in full accordance with alert investigation procedures.
Optimization of the system still occurs as part of a continuous improvement process; however, the system
is no longer considered to be developmental.
Remediation and recovery. The stage of a contamination incident following Confirmed contamination,
which involves the implementation of system decontamination and return to service.
Risk communication. Communication activities within an organization and with external parties that
address the consequences and outcome of an incident.
Routine operation. The day-to-day monitoring and surveillance activities of the contamination warning
system that are guided by the component response procedures. To the extent possible, routine operation
of the contamination warning system is integrated into the routine operations of the drinking water utility.
Salvage value. Estimated value of assets at the end of the useful life of the system.
Scenario subset. A group of scenarios that represent a portion of the full ensemble. Typically, scenario
subsets will be defined by specific values or ranges of values for scenario parameters.
Security breach. An unauthorized intrusion into a secured facility that may be discovered through direct
observation, an alert or signs of intrusion (e.g., cut locks, open doors, cut fences).
103
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Simulation study. A study designed to systematically characterize the detection capabilities of the
Cincinnati contamination warning system. In this study, a computer model of the Cincinnati
contamination warning system is challenged with an ensemble of 2,015 simulated contamination
scenarios. The output from these simulations provides estimates of the consequences resulting from each
contamination scenario, including fatalities, illnesses, and extent of distribution system contamination.
Consequences are estimated under two cases, with and without the contamination warning system in
operation. The difference provides an estimate of the reduction in consequences.
Site characterization. The process of collecting information from an investigation site to support the
investigation of a contamination incident during consequence management.
Threat level. The results of the threat level determination process, indicating whether contamination is
Possible, Credible or Confirmed.
Threat level determination process. A systematic process in which all relevant information available
from a contamination warning system is evaluated to determine whether the threat level is Possible,
Credible or Confirmed. This is an iterative process in which the threat level is revised as additional
information becomes available. The conclusions from the threat evaluation process are considered during
consequence management when making response decisions.
Threat level index. In the Cincinnati contamination warning system model, a quantitative indicator of
the threat level associated with a specific contamination scenario. The threat level index is calculated by
the Cincinnati contamination warning system model by summing the confidence indices from all
component models. A value greater than or equal to 1.0 represents Possible contamination, greater than
or equal to 2.0 represents Credible contamination, and greater than or equal to 3.0 represents Confirmed
contamination.
Time for Confirmed determination. A portion of the incident timeline that begins with the
determination that contamination is Credible and ends with contamination either being Confirmed or
ruled out. This includes the time required to perform lab analyses, collect additional information, and
analyze the collective information to determine if the preponderance of evidence confirms contamination.
Time for contaminant detection. A portion of the incident timeline that begins with the start of
contamination injection and ends with the generation and recognition of an alert. The time for
contaminant detection may be subdivided for specific components to capture important elements of this
portion of the incident timeline (e.g., sample processing time, data transmission time, event detection
time, etc.).
Time for Credible determination. A portion of the incident timeline that begins with the recognition of
a Possible contamination incident and ends with a determination regarding whether contamination is
Credible. This includes the time required to perform multi-component investigation and data integration,
implement field investigations (such as site characterization and sampling), and collect additional
information to support the investigation.
Time for initial alert investigation. A portion of the incident timeline that begins with the recognition
of an alert and ends with a determination regarding whether or not contamination is Possible.
Time-step. In the Cincinnati contamination warning system model, a set interval of time (i.e., every 15
minutes) at which the model performs calculations, reads inputs or generates outputs.
104
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Toxic chemicals. Highly toxic chemicals that pose an acute risk to public health at relatively low doses.
Valid alert. Alerts due to water contamination, verified water quality anomalies (e.g., a change in water
quality caused by work in the distribution system), or public health incidents.
Water Utility Emergency Response Manager. A role within the Cincinnati contamination warning
system filled by a mid-level manager from the GCWW. Responsibilities of this position include:
receiving notification of validated alerts, verifying that a valid alert indicates Possible contamination,
coordinating the threat level determination process, integrating information across the different
monitoring and surveillance components, and activating the consequence management plan.
Work Order. An internal record documenting the requirement for and execution of a utility-lead activity
in the distribution system. For GCWW, water quality work orders, which are monitored by CCS, require
the collection and testing of water samples from the location of a customer complaint.
105
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Appendix A: Cincinnati Contamination Warning System
Model
A.1 Overview of Contamination Warning System Model Architecture
To perform the simulation study, it was necessary to develop a detailed computer model of the Cincinnati
Contamination Warning System (CWS). This model is comprised of sub-models representing the
individual component of the CWS. The component models describe the data processing, decision logic,
and sequencing steps that represent the activities executed by the corresponding component. Each
component model consists of blocks referred to as modules. Modules represent a logical grouping of
steps or a key function within the component model. Each module is parameterized using a variety of
data sources as described in this appendix and operates on a set of inputs in order to produce a set of
outputs that serve as inputs to a subsequent module.
To understand how the model functions, it is important to distinguish between model parameters, inputs
and outputs:
Parameters. Fixed values in the model that define important aspects of each component to
accurately represent the physical system. Example parameters include the physical locations of
monitoring stations and times necessary to complete various steps of the investigation process.
Inputs. Values that will change during the course of the simulation study. An input may vary
with respect to time, location or scenario. For example, an input is a contaminant concentration
profile, which consists of concentrations at a particular location as a function of time for a
specific scenario.
Outputs. The results generated from a module or a model during the simulation. Some outputs
are generated only once per scenario, while others are generated during multiple time-steps over
the course of the scenario. Example model outputs include alerts generated by components or
response actions implemented during consequence management.
The overall model architecture is depicted in Figure A-l. It includes a software application that models
hydraulic and water quality conditions in a distribution system (EPANET), a Health Impacts and Human
Behavior (HI/HB) model that simulates health consequences and human behaviors in response to various
symptoms, models of the primary CWS monitoring and surveillance components (Enhanced Security
Monitoring (ESM), Water Quality Monitoring (WQM), Customer Complaint Surveillance (CCS), and
Public Health Surveillance (PHS), and a model of the Consequence Management (CM) process. The
interconnecting lines depict how information flows among the models (e.g., outputs of each of the four
monitoring and surveillance models serve as inputs to the CM model). The following sections describe
each of the primary elements of the CWS model in greater detail.
106
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Contamination
^ Scenarios J
EPANET
Concentration
Profiles
Contaminant
Properties
ESM Model
Exposure
Assumptions
Health Impacts and
Human Behavior Model
Scenario
Consequences
Consumer
Actions
WQM Model
1
r
CCS Model
i
PHS Model
Component
Outputs
Data table
Component Model
Database
Blue - Generated DB
Yellow- Input DB
Consequence
Management
Model
Scenario
Modifications
Supplied Software
Figure A-1. CWS Model Architecture
This model operates in discrete time-steps (i.e., a set time interval) rather than continuously. While the
inputs that govern operation of the CWS are changing constantly, it is impractical for the model to keep
up with all the necessary calculations at every instant, and such data resolution does not produce more
accurate predictions from the model. A practical solution to this challenge is to define a time-step at
which point the most recent input values are used to calculate a new set of output values. For example,
the concentration profile of a contaminant at a given point in the distribution system changes
continuously, but usually in very small increments over very short time intervals. So instead of using
every second of the contaminant concentration profile, it is approximated at a 15 minute time-step. Use
of discrete time-steps reduces the quantity of data generated by the model by almost three orders of
magnitude and drastically reduces run time without a significant loss in accuracy.
A.2 EPANET Toolkit
EPANET is a common hydraulic and water quality modeling application widely used in the water
industry to simulate hydraulics and water quality through the distribution system
(http://www.epa.gov/nrmrl/wswrd/dw/epanet.htmltfcontent). EPANET is used in conjunction with a
distribution system model that represents the arrangement of pipes, pumping facilities and storage
facilities in a utility's distribution system. In the simulation study, EPANET along with the Greater
Cincinnati Water Works (GCWW) distribution system model was used to produce contaminant
concentration profiles at every node in the GCWW distribution system model.
107
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
The EPANET Toolkit is a dynamic link library and is utilized in the simulation model for automation of
parameter adjustments, automation of hydraulic and quality simulation, and customized extraction of
concentration profiles. Instead of needing to mine output data after the EPANET model finishes, the
Toolkit allows the extraction of concentration data needed for the ESM, WQM and HI/HB models. This
ability reduces file size and optimizes the entire process of extracting EPANET outputs.
A conceptual configuration of the EPANET model is shown in Figure A-2. To generate the contaminant
concentration profiles, EPANET used the inputs that define the contamination scenario, which are listed
in the input database icon shown in the figure and described in detail in Table A-l.
^^^^^ ^^^^-
~~^~~ ^^"
Scenarios
1 . Injection Flow Rate
2. Injection Duration
3. Injection Start Time
4 Iniectinn Nnrie
EPANET with
1 I4'l'4 Pk' 4 'U. *'
utility Distribution
System Model
C^^^^^^^
~~ ____ ____ - "
Concentration
^^ Profiles ^^
Database
Blue - Generated DB
Yellow-Input DB
Supplied Software
Figure A-2. EPANET Model
Table A-1. EPANET Inputs
Input
Injection Node
Injection Time
Injection Rate
Injection Duration
Description
The location of contaminant injection. It was assumed that all distribution system nodes in
the system are potential injection locations with the exception of terminal points in the
system and nodes that have no demand for water (i.e., they are not access points in the
system).
The time at which a contaminant is injected into the distribution system at the injection
node. Two injection times were selected: one representing a period of high demand and
low pumping (9:00 a.m.), and the other representing a period of low demand and high
pumping (12:00 a.m.).
The mass flow rate of a contaminant being added to the distribution system at the injection
location. Injection rates were selected to achieve a target concentration in the system that
would result in harmful consequences (e.g., adverse public health or infrastructure
contamination). Injection rates were calculated based on three typical flow rates in
distribution pipes of various sizes.
The continuous length of time the contaminant is injected into the distribution system at the
injection node. The duration is calculated from the mass injection rate and the total mass of
contaminant injected into the distribution system.
For each 15-minute time-step, concentrations at each node are recorded and stored as the model outputs
shown in Table A-2.
Table A-2. EPANET Outputs
Output
Node ID
Contaminant
Concentration
Description
Unique identifier for each node in the distribution system model.
The concentration of a contaminant (mg/L or organisms/L) at each
time.
node as a function of
108
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Output
Time of
Concentration
Description
The time at which a recorded contaminant concentration occurred at a specific node. Note
that concentrations are recorded only at the times of exposure events, as described in
Section A.3.
The concentration profiles generated by EPANET were used in the HI/HB model to determine the dose
received by individuals exposed to contaminated water. The concentration profiles were also used as
inputs to the WQM model to determine when and where WQM alerts are generated.
A.3 Health Impacts and Human Behavior
The HI/HB model was designed to simulate the health effects in the population served by the distribution
system resulting from exposure to contaminated drinking water. The HI/HB model also simulates actions
of individuals who either detect a problem with the drinking water or experience symptoms after being
exposed to a harmful contaminant. This task is accomplished by tracking the health effects, actions and
outcomes of each individual modeled in the simulation. The individual behaviors tracked in this model
provide inputs for two of the contamination warning system components: calls to the utility provide
inputs to the CCS model and health seeking behaviors provide inputs to the PHS model. Furthermore, the
cumulative outputs from this model determine the overall public health consequences (illnesses, fatalities
and healthcare burden) of each scenario.
Figure A-3 shows the relationships among the several modules and queues (shown as green rectangles
and parallelograms) that comprise the HI/HB model, along with the outputs generated by the model (blue
parallelograms). The HI/HB model first uses the output from EPANET to execute the exposure module,
which determines the contaminant dose at each node in the distribution system model during each time-
step of the simulation.
109
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Concentration
Profiles
Exposure Module
(Populates Dose Table)
Database
Blue - Generated DB
Yellow- Input DB
Sensory Impacts Module
(Determines if a consumer
detects a contaminant via the
senses)
I
Chemical Disease
Module
(Determines the symptoms and
disease progression for
individuals exposed to a
chemical)
Pathogen Disease
Module
(Determines the symptoms and
disease progression for
individuals exposed to a
pathogen)
Case Table
1. Health seeking
behavior
2. Health outcome
Consumer Action Module
(Determines the health-seeking behaviors taken
by each individual and populates the queues
shown below)
GCWW Queue
DPIC Queue
Dr. Queue
911 Queue
ED Queue
EMS Queue
Figure A-3. Health Impacts and Human Behavior Model
The Exposure module calculates the cumulative dose for exposed individuals, which is then used by the
Health Impact modules to determine the health impacts experienced by each exposed individual. In
addition to the output from EPANET, the Exposure module also uses the time of public notification,
which is outputted by the consequence management model. This time is used to determine the time when
a compliant individual stops using the water and thus experiences no further increase in their cumulative
dose. Three pathways are possible through these modules, depending on whether the contaminant is
detectable by individuals through a taste or odor, and whether the contaminant is a chemical or a
pathogen. Each individual who either detects the contaminant or receives a dose sufficient to produce
health effects becomes a "case" in the Case Table. The disease progression timeline and ultimate
outcome of each individual are tracked in the Case Table.
The HI/HB model includes three health impacts modules: Sensory Impact, Chemical Disease and
Pathogen Disease. The Sensory Impact module is used to determine whether or not an exposed individual
detects the contaminant through the senses, based on the concentration in the water at the time of the
exposure event and a detection threshold for the specific contaminant. The two disease modules include
the logic and parameters to determine the disease progression timeline, expression of symptoms, and
ultimate health outcome based on the cumulative dose received.
110
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
The various symptoms and sensory perceptions experienced by each affected individual are inputs to the
Consumer Action module, which determines the actions taken by each individual in response to their
condition. Available actions include: doing nothing, calling GCWW, calling the Drug and Poison
Information Center (DPIC), calling 911, visiting a primary healthcare doctor, visiting the emergency
department (ED) or administering self-treatment (not shown in the figure). Individuals may also receive
prophylactic treatment for some contaminants once the contaminant identity is known. The specific
actions selected by an individual are based on the contaminant type, symptom level, and demographic
characteristics of each individual. Data from literature reviews was used to estimate the percentage of
symptomatic individuals in each demographic group that would pursue each of the available options at
each stage of disease progression (Bertakis et al., 2000; Schappert and Bert, 2006).
For each time-step, individuals are processed up to the number that can be handled by the available
capacity of the queues at that time. With one exception, the queues operate on a first in first out basis.
The ED queue includes a triage function such that individuals with more serious symptoms are
automatically moved ahead of individuals with less serious symptoms. When an individual is processed,
the queue and treatment information is added to their record in the Case Table, and then one of the
following occurs:
Each individual is automatically moved to a new queue due to a referral or queue logic. For
example, a call to 911 always results in an Emergency Medical Service (EMS) response, which
represents the policy of the City of Cincinnati Fire Department. Therefore, when someone is
processed in the 911 queue, they automatically move to the EMS queue.
Individuals will take another action if they wait too long in their current queue (e.g., they will
drive themselves to the ED if an EMS unit has not arrived in a specified amount of time), or, if
their symptoms worsen while waiting in one queue, they may be switched to another queue (e.g.,
if a person waiting for prophylactic treatment becomes symptomatic, they leave the prophylaxis
queue and enter the ED queue).
The individual is done taking action and has been processed through the appropriate queues. All
cases eventually arrive at this point, where logic within the queue processing routine is used to
determine if the individual received effective medical treatment.
All queues are defined by two ceilings on capacity: one representing normal, non-emergency conditions,
and the other representing mobilization of additional resources during response to a recognized public
health crisis. The capacity ceiling of some queues is fixed over a 24/7 period, while other ceilings vary
with the time of day and day of the week. Each queue also has an associated mean processing time that
quantifies the length of time that a particular resource is committed to a specific individual. Finally, some
queues have a maximum wait tolerance that defines the length of time that an individual will remain in
the queue before they exit the queue and pursue another option.
The model executes all the routines depicted in Figure A-3 to initially populate the Case Table. The
model does not operate on each time-step over the duration of the simulation, as this would result in
unnecessary computing overhead. Instead, the model determines when the next action takes place and
produces outputs only during those time-steps. This approach reduces the model run time significantly.
The primary inputs to the HI/HB model, shown in Table A-3, are the contaminant concentration profiles
at each distribution system model node, attributes of each node (e.g., population at the node), and
attributes of the contaminant used in the scenario (e.g., health effects parameters). This input data is
passed to the Exposure module, which includes parameters that define exposure events, such as the time
of consumption or showering events and the volume of water used during each event. The Exposure
111
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
module calculates the cumulative dose for exposed individuals, which is then used by the Health Impacts
modules to determine the health effects experienced by each exposed individual.
A pre-defined sequence of exposure events is used by this model, which establishes central tendencies in
the timing of consumption and showering. The central tendency in the timing of the consumption events,
as well as the volume consumed, is based on surveys of drinking water usage in the United States (Davis
and Janke, 2008; USEPA, 2007). Based on the results of these studies, the model uses five consumption
events per day, roughly corresponding to three meals and two breaks between the meals. In addition,
children and adults are assumed to take one shower per day in the morning hours (USEPA, 1997), during
which there is the potential for inhalation of aerosolized water droplets containing the contaminant. In the
model, it was assumed that infants do not take showers. While one time of day is selected for each of
these exposure events, these times represent a central tendency in consumption behavior. The actual time
at which individuals consume water (or take a shower) is governed by a normal distribution around the
central tendency.
Table A-3. HI/HB Model Inputs
Input
Contaminant ID
Contaminant
Concentration
Time of Exposure
Description
Sanitized identifier for the contaminant used in the scenario. Linked to the appropriate
contaminant attributes described in Table A-4.
The concentration of a contaminant (mg/L or organisms/L) at each node as a function of
time. These values are outputs from EPANET.
The time that an individual is exposed to drinking water during consumption or showering
events. The timing of the consumption events is based on surveys of drinking water usage
in the United States (Davis and Janke, 2008; USEPA, 2007). Based on the results of these
studies, the model uses five consumption events per day (07:00, 09:30, 12:00, 15:00,
18:00), roughly corresponding to three meals and two breaks between the meals. The
model uses one showering event per day at 07:00.
Parameters for the HI/HB Model are listed in Table A-4. Key parameters include detection threshold
concentrations, as well as the cumulative dose that would produce mild, moderate and severe symptoms
in an exposed individual. There are also model parameters that govern the probability of infection in the
case of a pathogen, or fatality in the case of a toxic chemical or biotoxin, as a function of cumulative dose
received.
Once the threshold is surpassed for a specific age group at a specific node, all individuals at that node
who are still using the water will experience the symptoms associated with that threshold. Each
individual will likely take action based on the symptoms they experience, as determined by the Consumer
Action module, and these actions can vary for each individual assigned to that node.
Table A-4. HI/HB Model Parameters
Parameter
Node ID
Population
Description
Unique identifier for each node in the distribution system.
The number of consumers at each node in the distribution system, including the distribution
of this population among the following five demographic groups: children (younger than 18
years, further subdivided into infants younger than 1 year and older children); adult
females (18 to 65 years); adult males (18 to 65 years); senior females (older than 65 years)
and senior males (older than 65 years). Population was calculated by Threat Ensemble
Vulnerability Assessment using nodal demands from the GCWW distribution system
model.
112
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Parameter
Consumption Volume
Detection Threshold
and Probability
Threshold Doses for
Symptoms
Symptom Onset
Delays
Threshold Doses for
Fatality
Fatality Onset Delay
Probability of Fatality
with Medical
Treatment
Probability of Fatality
if Untreated
Queue Capacity
Processing
Times
Treatment Window
Effective Treatments
Threshold Number of
Cases for a
Differential Diagnosis
Description
The volume consumed by an individual over a 24-hour period, specifically defined for
infants, children, and adults. The total volume consumed by an infant in the model is 0.30
L/d, by a child is 0.595 L/d and by an adult is 1 .41 L/d distributed over five ingestion
events. The volume inhaled by a child or an adult during the showering event is 0.00006
L/d. The model assumes no exposure due to showering for infants. The volume
consumed during a consumption event is based on surveys of drinking water usage in the
United States (Davis and Janke, 2008; USEPA, 2007).
The concentration of a specific contaminant, above which it can be detected through the
senses (taste, odor, or irritation), and the probability that an individual exposed to a
contaminant at concentration at or above the threshold will detect it. The threshold and
probabilities for detecting a contaminant are based on data reported in the literature for
each specific contaminant. In cases where such data was not available, it was assumed
that between 90 and 100% of the population would detect the contaminant.
The dose of each contaminant that produces mild, moderate, and severe symptoms. All
thresholds are contaminant-specific attributes that determine the type and severity of
symptoms based on the cumulative dose received. Values were derived from expert
judgment of medical specialists and toxicologists, October 13, 2009.
The time delay between exposure to a contaminant above a threshold dose and the onset
of symptoms. Specific onset delays are defined for each contaminant and each symptom
level: mild, moderate, and severe. Values were derived from expert judgment of medical
specialists and toxicologists, October 13, 2009.
Points distributed along the dose response curve that relate the cumulative dose received
by an individual to the probability of death. Values were derived from an extensive review
of contaminant databases and peer reviewed literature.
The time delay between exposure to a lethal dose of a contaminant and death. Values
were derived from an extensive review of contaminant databases and peer reviewed
literature. The dose is cumulative over the simulation duration, but the model does include
metabolic degradation of contaminants post-exposure.
The probability that an individual exposed to a lethal dose of a contaminant will die after
receiving effective medical treatment. Values were derived from expert judgment of
medical specialists and toxicologists, October 13, 2009.
The probability that an individual exposed to a lethal dose of a contaminant will die at the
end of the disease (after the fatality onset delay) in the absence of medical intervention.
Values were derived from an extensive review of contaminant databases and peer
reviewed literature.
A time series for each queue showing the maximum number of individuals that can be
processed simultaneously based on the available resources at the current time-step (e.g.,
available operators, open hospital beds, etc.). Two capacity ceilings are defined for each
queue: one reflecting normal, non-emergency conditions; and the other reflecting
mobilization of additional resources in response to an emerging public health crisis. The
capacity ceiling parameters were provided by each department for normal business hours,
non-business hours, and emergency conditions.
The time it takes to process an individual in a queue (e.g., call processing time, EMS
transport time, etc.). Along with the queue capacities, this determines when resources are
available to process additional individuals. The processing time for an individual to
complete each queue was provided by each department.
The amount of time, relative to the onset of disease, within which treatment must be
received in order to be effective. The size of the treatment window is specific to each
contaminant. Values were derived from expert judgment of medical specialists and
toxicologists, October 13, 2009.
An indicator regarding which of the following treatment alternatives may prove effective
following exposure to a specific contaminant: self-treatment, treatment by a primary care
physician, treatment by an EMS technician, and treatment by an ED physician. The
parameter also indicates whether or not the treatment alternative is limited due to a finite
resource. Values were derived from expert judgment of medical specialists and
toxicologists, October 13, 2009.
For each contaminant and associated symptom level, the number of individuals
experiencing those symptoms that must be seen in the ED to cause public health officials
to tentatively identify the causative agent. Values were derived from expert judgment of
medical specialists and toxicologists, October 13, 2009.
113
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Parameter
Description
Threshold Number of
Cases for Public
Health Response
For each contaminant, the total number of individuals that must been seen in the ED
before public health officials recognize an emerging public health crisis. Values were
derived from expert judgment of medical specialists and toxicologists, October 13, 2009.
Health Seeking
Behaviors
The probability that an individual will pursue each of the following health seeking
behaviors: do nothing, self treat, call GCWW, call DPIC, call 911, visit a doctor, or visit the
ED. Unique probabilities are assigned to each combination of contaminant, symptom level,
and demographic group. The probabilities change after public notification has been issued
to better align healthcare choices with effective treatments. Information obtained from
literature reviews was used to estimate the percentage of symptomatic individuals in each
demographic group that would pursue each of the available options based on the specific
symptoms they experience (Bertakis et al., 2000; Schappert and Bert, 2006).
The HI/HB model executes the Exposure module, the appropriate Health Impact module, and the
Consumer Action module to populate the Case Table. The Consumer Action module generates
information about the action taken by each individual and saves this information to the appropriate record
in the Case Table. If an individual receives effective medical treatment, the time of treatment is recorded.
If the individual does receive effective medical treatment in a defined window of opportunity relative to
the time of disease onset, that individual's probability of dying is reduced, and their outcome is
determined using this lower mortality rate. The efficacy of various treatment options, as well as the
window of opportunity for treatment, is dependent upon the contaminant and associated disease.
The Case Table records the outputs shown in Table A-5, including: the timeline of the health impacts,
actions taken by each individual, and outcome of each individual. The fully populated Case Table
provides information used as inputs to the other models shown in Figure A-l.
Table A-5. HI/HB Model Outputs
Output
Case ID
Location ID
Time of Stop Use
Compliance
Time of Detection
Time of Infection
Time of Symptoms
Time of Health
Seeking Behavior
Time of Medical
Treatment
Time of Death
Differential Diagnosis
Confidence
Times Public Health
Response is
Activated
Description
A unique identifier for each individual exposed to contaminated water during a
contamination scenario.
The specific distribution system model node that the individual is assigned to for all
exposure events (i.e., home location).
The date and time that an individual stops using contaminated water due to compliance
with a "do not use" notice from the water utility. A distribution of times ranging from 30 min
to 10 hours after issuance is assumed for compliance with a "do not use" notice, and the
model assumes that approximately 10% of the population will never comply. Also,
individuals will immediately stop using water if they detect the contaminant through the
senses.
The date and time that an individual detects contaminated water via the senses.
The date and time that an individual is either infected with a pathogen or becomes
symptomatic due to chemical exposure.
The date and time that an infected individual experiences each discrete level of symptoms:
mild, moderate, and severe.
The date and time that an individual takes various actions in response to their current
condition. Times are recorded for each health-seeking behavior option that an individual
takes over the course of the scenario.
The date and time that an individual receives health care that effectively treats their
condition. If effective medical treatment is received in time, the individual's prognosis
improves.
The date and time that an individual dies due to exposure to the contaminated water. This
field is blank if the individual recovers or never receives a fatal dose of contaminant.
A time series of values between 0 and 2 indicating the confidence of public health officials
in the identity of the contaminant responsible for illnesses observed in the ED. Confidence
in the identity of the contaminant can range from nil (0) to absolute certainty (2).
The date and time when each effective public health response (e.g., instructions provided
to health partners, mobilization of additional hospital personnel, etc.) has been
implemented. These times establish when expanded queue capacities and improved
medical referrals will be in effect.
114
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Other key outputs from the HI/HB model include the time that additional resources are available to
healthcare providers for the purpose of treating patients. Specifically, the output from the Public Health
Response module will determine when the expanded queue capacities are in effect.
A.4 Enhanced Security Monitoring
The ESM model was designed to simulate the systems, equipment, and procedures that detect and
respond to security breaches at distribution system facilities (e.g., pump stations, storage tanks, etc.) that
are vulnerable to contamination. For each distribution system facility considered, the model uses site-
specific information about the path between an assumed point of entry and an assumed point of
contaminant introduction, the steps required to introduce the contaminant, and the path of egress from the
facility following the completion of the attack. The model assumes that all attacks would use a pump to
inject the contaminant, and that the attacker would leave as soon as they started pumping the contaminant
into the distribution system.
The model also simulates the physical security alerts and monitoring systems for the facility that have
been breached in order to generate the alert that would be displayed to security personnel. Following alert
recognition, the model simulates the alert investigation process based on the procedures used by GCWW
personnel.
Figure A-4 provides an overview of the ESM model showing the relationships among the three modules,
shown as green rectangles, which comprise the model: ESM Intrusion module, ESM Alert Generation
module and ESM Alert Investigation module. The inputs to and outputs from each module of the ESM
model are shown as blue parallelograms.
The first module that operates is the Intrusion module. Here, location-specific attack and retreat times are
used for each ESM location, based on site-specific factors such as intrusion entry points, access points to
drinking water, and feasible injection volumes. Attack and retreat times serve as inputs to the Alert
Generation module, which accounts for processing time for monitoring devices, as well as alert and video
transmission times.
The ESM alerts are transmitted to the Alert Investigation module, which simulates the actions that would
be taken by GCWW personnel in response to ESM alerts. This module is based on alert investigation
procedures developed for the ESM component and timeline metrics characterized during drills, exercises,
and routine operation of the CWS. These metrics account for the time required to recognize the alert and
perform a variety of investigative functions, including a review of video clips, if available, and on-site
inspection of the ESM site. The outputs from this module include the time of key notifications, the time
investigators arrive on site, the time that contaminant injection is interrupted and the ESM confidence
index.
The ESM confidence index is an overall indicator of the reliability of the information from the ESM
component, considering all available data from all ESM alerts and the ongoing investigation. The value
of the ESM confidence index will change over time as the investigation progresses. An ESM alert could
result in Possible, Credible, or Confirmed contamination without additional information from another
component. Under the model assumptions, an ESM alert can result in a Possible determination if there is
no employee call-back following a brief waiting period after the alert is received. ESM can result in a
Credible determination under either of the following conditions: 1) observation of signs of tampering
during site investigation, and 2) conclusive video evidence of an intrusion and pumping equipment at a
utility facility. Finally, ESM can provide sufficient information to Confirm contamination if GCWW
responders observed an ongoing injection during a site inspection.
115
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
1. Injection location
2. Facility attributes
ESM Intrusion Module
(Execution of a contamination attack)
1. Time of intrusion
2. Time of contaminant injection
3. Time of intruder retreat
ESM Alert Generation Module
(Generates intrusion alarms and video from facility that is the location
of the injection)
1. Time of alert
2. Location of alert
3. Time video received
ESM Alert Investigation Module
(Procedures used by GCWWpersonnel to investigate an ESM alert)
Data table
Module
1. Time that the WUERM and supervisor are notified
2. Time that police are notified
3. Time for investigators to arrive on site
4. Time injection is interrupted (if applicable)
5. ESM confidence index
Figure A-4. Enhanced Security Monitoring Model
The primary inputs to the ESM model are shown in Table A-6 and include: the intrusion location,
injection start time, and injection duration. The ESM model will be activated only if the injection
location (node) occurs at one of the ESM sites.
Table A-6. ESM Model Inputs
Input
Location of Intrusion
Scenario Start Time
Duration of the
Injection
Description
The node at which the contaminant is introduced under the conditions of the scenario. The
ESM model will be activated only if the attack node is associated with a utility facility with
ESM capabilities.
The date and time that the scenario starts, at which point the perpetrators start introducing
the contaminant into the distribution system from the location of intrusion.
The total time that the equipment is actively injecting the contaminant into the distribution
system at the location of intrusion. The injection duration was determined using EPANET
to ensure that the contaminant is spread through the distribution system at potentially
harmful concentrations.
116
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Parameters for the ESM model are listed in Table A-7. Key parameters include attack and retreat times,
as well as the response times for GCWW security personnel and law enforcement personnel to travel to
the location of the ESM alert.
Table A-7. ESM Model Parameters
Parameter
Attack and Retreat
Times
Video Surveillance
Alert Transmission
Time
Employee Call in Wait
Time
Video Clip
Transmission Time
Time to Contact Local
Law Enforcement
Time to Contact
GCWW Security
Law Enforcement
Response Time
GCWW Security
Response Time
Site Investigation
Time
Description
Specific attack and retreat times were derived for each ESM site. The time to attack and
retreat was based on the specific layout of each facility and the specific location in the
facility from which the injection would occur. Times to execute various actions were
provided by Sandia National Labs, the American Waterworks Association (AWWA), and
estimates from security experts.
The location-specific information regarding the presence/absence of video equipment at
the ESM site. This information is based on the physical design of the ESM component of
the Cincinnati CWS.
Time to transmit an alert intrusion signal from the remote programming logic controller to
the SCADA user interface at the control center (five seconds). Times were directly
measured from the physical system.
The time for an operator to wait for an employee to call in after entering a remote GCWW
facility. Time is documented in GCWW procedures.
Time to transmit a video clip from the remote facility to the SCADA user interface at the
control center (three minutes). Times were directly measured from the physical system.
The time it takes for the utility control center operator to dial 91 1 and inform the 911
dispatcher of the intrusion event. The values for this parameter were obtained from drills
and exercises performed during the evaluation period of the Cincinnati pilot.
The time it takes for the utility control center operator to dial GCWW Security and inform
the guard of the intrusion event. The values for this parameter were obtained from drills
and exercise performed during the evaluation period of the Cincinnati pilot.
The time it takes for local police to reach the location of the ESM alert. The values for this
parameter were obtained from average law enforcement response times.
The time it takes for GCWW Security to reach the location of the ESM alert. The values for
this parameter were obtained from online mapping software.
The site-specific time it takes for the Plant Supervisor to conduct an investigation at the
location of the ESM alert. The values for this parameter were obtained from drills and
exercises performed during the evaluation period of the Cincinnati pilot.
The primary outputs from the ESM model are shown in Table A-8, and include the time the alert is
received, the time at which various responders arrive on site, and the time at which the site investigation
is completed. If responders arrive on site in time to interrupt the injection, that time is also outputted by
the model. Another important output, which is an input to the downstream Consequence Management
model, is the ESM confidence index.
Table A-8. ESM Model Outputs
Output
Time of Alert
Time of WUERM
Notification
Time of Law
Enforcement
Response
Time of GCWW
Security Response
Time of Site
Investigation
Description
The date and time when the ESM equipment detects intrusion into a utility facility,
generates an alert, and transmits that alert to the SCADA user interface.
The date and time when the utility control center operator or Plant Supervisor contacts
WUERM after completing the ESM investigation.
the
The date and time when local law enforcement arrives on-site.
The date and time when GCWW Security arrives on-site.
The date and time when the Plant Supervisor completes their investigation of the site of
the suspected intrusion.
117
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Output
Time the Injection is
Interrupted
ESM Confidence
Index
Description
The date and time when the injection is stopped by investigators from GCWW, if the time
that responders arrive at the injection location occurs prior to the time the injection would
be completed.
A time series of values from the ESM component indicating the reliability of information
available from the investigation of the ESM alert and the degree of confidence in the
suspicion that the drinking water has been contaminated.
A.5 Water Quality Monitoring
The WQM model was designed to simulate the network of monitoring stations throughout the GCWW
drinking water distribution system and the associated investigative procedures designed to detect unusual
water quality conditions. The network consists of fifteen WQM stations in the distribution system, which
monitor for the following parameters: free chlorine (CL2), specific conductivity (COND), oxidation
reduction potential (ORP), pH and total organic carbon (TOC).
Data generated by the network of monitoring stations is transmitted to an operations center where it is
continuously analyzed for potential anomalies by an event detection system. When an anomaly is
detected, an alert is generated and displayed on a user interface. GCWW personnel follow a standardized
alert investigation procedure to determine the cause of the alert. If the alert cannot be attributed to a
benign or known cause, contamination is considered Possible.
Figure A-5 provides an overview of the WQM model, showing the relationships among the modules and
software applications that comprise the model. The three modules, shown as green rectangles, which
constitute the WQM model include: Contaminant Profile Simulator module, WQM Alert Processing
module, and WQM Alert Investigation module. Additionally, the WQM model incorporates the
CANARY software application, shown as a pink rectangle, which is the event detection system used at
the Cincinnati pilot. The inputs to and outputs from each module of the WQM model are shown as blue
parallelograms.
The Contaminant Profile Simulator uses contaminant-specific correlation factors to simulate the change in
water quality due to the presence of a specific contaminant. The inputs to this module include the
baseline water quality data from the GCWW pilot as well as the simulated contaminant concentrations
produced by EPANET. By applying the correlation factors to the contaminant concentration profiles, the
Contaminant Profile Simulator generates a time series of changes in each water quality parameter. These
changes are superimposed on GCWW baseline water quality data to generate water quality parameter
values that reflect the impact of the contaminant concentration at each monitoring location.
The Contaminant Profile Simulator provides the input water quality parameter dataset analyzed by
CANARY, which uses a linear filter algorithm to search for anomalies. Specifically, this algorithm uses
historic water quality data to predict water quality at the next time step. Differences between the current
water quality value and the predicted value are recorded and compared to a threshold value. Additionally,
the differences across all sensors can be joined to create a combined difference value. If the threshold
value is exceeded, CANARY generates an alert.
Once an alert is generated, the WQM Alert Processing module simulates the time delay between detection
of the anomaly at the monitoring station and display of the alert on the SCADA user interface. The delay
is due to data transmission and event detection processing time, and while on the order of only a few
minutes, it is still accounted for.
118
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
The alert investigation process begins when the Water Quality and Treatment (WQ&T) Chemist is
notified of the alert. The investigation includes a decision point where the investigation of the alert may
be terminated unless a priority water quality parameter, specifically TOC or CL2, is included in the
trigger parameters for the alert. The prioritization of trigger parameters and relative changes that warrant
further investigation are based on data collected from WQM alert investigations during the Cincinnati
pilot. The investigation continues with a simultaneous review of water quality data (by the WQ&T
Chemist) and distribution system operations (by the Operator). After completion of these reviews,
distribution system work orders are reviewed to determine whether recent or ongoing work in the system
could have caused the water quality that generated the alert.
Once the review of water quality data, operational data, and distribution system work orders has been
completed, three actions are implemented: 1) the Water Utility Emergency Response Manager (WUERM)
is notified of the WQM alert, 2) a remote sample is collected at the WQM station that produced the alert,
and 3) all downstream WQM stations are set to automatically collect a sample after any subsequent alert
is generated. If no cause for the alert has been identified thus far, the WQM station that generated the
alert is inspected by a technician. The model accounts for the time required for the technician to gather
equipment and drive to the location of the WQM station. The model further assumes that the technician
verifies that all instrumentation is functioning properly, and thus faulty readings are ruled out as a
potential cause of the alert. The outputs from this module include the time when results from the
investigation are reported to the WUERM, the time of sample collection, and the WQM confidence index.
The WQM confidence index is an overall indicator of the reliability of the information from the WQM
component and the degree of confidence in the suspicion that the drinking water has been contaminated.
The value of the WQM confidence index will change over time as the investigation progresses. Under the
model assumptions, the following conditions can lead to a determination that contamination is Possible:
1. Completion of the investigation of a single WQM alert for which no benign cause is identified,
2. During the investigation of the first WQM alert, a second WQM alert occurs at a WQM station
that is hydraulically connected to the first, or
3. During the investigation of one WQM alert, the WUERM discovers that other component(s) have
detected potential indicators of contamination that are consistent with the information from the
WQM alert.
The WQM component can also produce information sufficient to establish that contamination is Credible
if two hydraulically connected WQM stations alert due to low chlorine residuals and the site inspection
has been completed for the first alert.
119
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Concentration
Profiles
Contaminant
Reaction Factors
Water Quality
Database
1. Monitoring location
2. Time-series contaminant
concentrations
Contaminant
Profile Simulator
Module
1. Monitoring location
2. Baseline WQ data
Contamination event
superimposed on
baseline WQ data
WQM Alert Processing
Module
(Processes the alert and
transmits it to SCADA)
CANARY
(Location-specific
configurations)
i
r
1. Probability of an
anomaly
2. WQ alert parameters
Data table
Module
Database
Blue - Generated DB
Yellow - Input DB
Supplied Software
1. Time of alert
2. Location of alert
3. WQ parameters
WQM Alert
Investigation Module
(Procedures used by GCWW
personnel to investigate a WQM
alert)
1. WUERM notification time
2. Time of sample collection
3. WQM confidence index
Figure A-5. Water Quality Monitoring Model
The primary inputs to the WQM model are shown in Table A-9 and include: the contaminant
identification, contaminant concentration profiles, and baseline water quality data. This information,
along with key parameters described below, is used to create a dataset for each monitoring station that
simulates the change in water quality resulting from contamination superimposed on baseline water
quality data.
Table A-9. WQM Model Inputs
Input
Contaminant ID
Contaminant
Concentrations
Baseline Water
Quality Data
Description
Sanitized identifier for the contaminant used in the scenario. Linked to the appropriate
contaminant attributes described in Table A-4.
For each scenario, the concentration of the contaminant (mg/L or organisms/L) at each
WQM location as a function of time. These values are outputs from EPANET.
The baseline data captures CL2 (mg/L), TOC (mg/L), pH, ORP (mV) and COND (uS/cm)
values that were reported for each time-step during the time period being modeled in the
simulation study for each monitoring location. The same baseline water quality data is
used for all scenarios. This baseline data was obtained from WQM instrumentation during
the evaluation period of the Cincinnati pilot.
120
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Parameters for the WQM model are listed in Table A-10. Key parameters include alert processing time,
time for initial investigation, the time to conduct a WQM site investigation, and the contaminant-specific
reaction factors.
Table A-10. WQM Model Parameters
Parameter
Polling Interval
Alert Processing Time
Time for Initial
Investigation
Time to Conduct
WQM Site
Investigation
Contaminant-Specific
Reaction Factors
Description
Established time between WQM data collection events by the SCADA system (2 minutes).
Time for data transmission and processing time between CANARY and SCADA. The
values for these parameters were obtained from direct measurements of SCADA system
performance.
Time required for GCWW staff to review water quality data and distribution system
operations during investigation of a WQM alert (first alert only). In the case of subsequent
alerts, an abbreviated investigation is performed that evaluates the connectivity among
alerting stations. The values for this parameter were obtained from drills and exercises
performed during the evaluation period of the Cincinnati pilot.
For each monitoring station, the time needed to perform the complete set of instrument
checks and rapid field tests on water collected at the WQM station. Note that the WQ&T
Technician keeps in constant contact with the WUERM, so results are reported to the
WUERM immediately. The values for this parameter were obtained from drills and
exercises performed during the evaluation period of the Cincinnati pilot.
For the given contaminant, empirical factors relating the concentration of the contaminant
to a subsequent change in the value of the following water quality parameters: CL2, TOC,
pH, ORP, and COND. These correlation factors were derived from the results of bench-
scale contaminant spiking studies (Hall, et al., 2007).
The primary outputs from the WQM model are shown in Table A-ll, and include the time the alert is
received, WUERM notification time, field safety screening results, water quality testing results, and the
WQM confidence index.
Table A-11. WQM Model Outputs
Output
WQM Location ID
WQM Alert Start Time
WUERM Notification
Time
WQM Sample
Collection Time
Field Safety
Screening Results
Water Quality Testing
Results
WQM Confidence
Index
Description
Unique identifier for each WQM location that produces an alert during the simulated
scenario.
The date and time at which each unique WQM alert is first displayed on the SCADA user
interface.
The date and time at which the WUERM is notified of each unique WQM alert.
The date and time when a water sample was collected (using a remote controlled sampling
system) from an alerting WQM station.
The date, time, and results ("normal" or "abnormal") for all field safety screening performed
during site investigations within the WQM component. Based on scenario assumptions,
the results of field safety screening will be "normal" at locations other than the site of
contaminant injection.
The date, time, and results ("normal" or "abnormal") for all water quality testing performed
during site investigations in response to a WQM alert. The design of the model assumes
that the results of water quality testing demonstrate that the monitoring station is
performing correctly.
A time series of values from the WQM component indicating the reliability of information
available from the investigation of the WQM alert and the degree of confidence in the
suspicion that the drinking water has been contaminated.
121
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
A.6 Customer Complaint Surveillance
The CCS model was designed to simulate the systems and procedures used by GCWW to detect
contamination through calls from customers reporting aesthetic changes to the quality of their drinking
water. Customers may detect contaminants with characteristics that impart an odor, taste, or visual
change to the drinking water, or that result in instantaneous yet minor symptoms, such as a mild irritation.
In the CCS model, all customers in the GCWW service area have the potential to detect contaminants that
change the aesthetic characteristics of the drinking water. Customers exposed to water contaminated at
concentrations above the contaminant-specific detection threshold may detect the contaminant, and may
call the utility. Calls to the utility are tracked through an interactive voice response system (IVR), which
includes a menu option specific to water quality issues.
Figure A-6 provides an overview of the CCS model showing the relationships among the three modules,
shown as green rectangles, that constitute the model: a Work Order (WO) Generation module, CCS Event
Detection module, and CCS Alert Investigation module. The inputs to and outputs from each module of
the CCS model are shown as blue parallelograms.
Calls to the utility reporting water quality problems are generated by the HI/HB model, as described in
Section A.3, and are one of the primary inputs to the CCS model. The first module that operates is the
WO Generation module. In this module, work orders are created in response to customer calls reporting
water quality concerns. The model assumes that each call reporting a water quality issue is converted into
a new WO, which is consistent with GCWW's procedures. The customer calls that are tracked through
the IVR and WOs are inputs to the Event Detection module.
The Event Detection module simulates the event detection systems used in the CCS component of the
Cincinnati CWS, which analyzes both the IVR and WO data streams using the following three
algorithms:
One Day, Weekday Scan. Monitors current data and evaluates it against recent historic data. If
the number of IVR selections or WOs in the previous 24 hours equals or surpasses the threshold,
an alert is generated. Does not operate between 12:00 a.m. Saturday morning through 11:59 p.m.
Sunday night.
One Day, Weekend Scan. The same as the One Day, Weekday Scan, but applies only to the
hours between 12:00 a.m. Saturday morning through 11:59 p.m. Sunday night.
Two Day Scan. Monitors current data and evaluates it against recent historic data. If the number
of IVR selections or WOs in the previous 48 hours equals or surpasses the threshold, an alert is
generated.
The one day scans have a reset function such that if an alert is generated, the algorithm begins counting
from zero again starting at the time of alert. Any data contributing to previous alerts cannot contribute to
the count triggering subsequent alerts, even if it falls in the 24 hour period. Thus, many one day alerts
could result from a surge of IVR selections or WOs. The two day scan is continuous and will not alert
until the number of calls or WOs in the previous 48 hours falls below the threshold before surpassing it
again. Thus, a surge of IVR selections or WOs would likely result in only one alert from the two day
scan, as the algorithm would remain above the threshold during the event. The alerts generated by the
Event Detection module serve as the inputs to the CCS Alert Investigation module.
The Alert Investigation module simulates GCWW's procedures for investigating a CCS alert, which
includes an assessment of the underlying complaints for clustering and similar problem descriptions as
122
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
well as possible benign explanations for the alerts such as distribution work or operational changes.
Additionally, recent water quality data in the area of the complaints is reviewed, which is simulated in the
model by checking the WQM component confidence index. The investigation process follows one of two
paths depending on whether the alert is from the IVR or the WO data stream. However, the investigations
of both types of alerts are eventually turned over to the WQ&T Chemist, who makes the determination
whether contamination is Possible. The outputs from this module include the time when results from the
investigation are reported to the WUERM and the CCS confidence index.
The CCS confidence index is an overall indicator of the reliability of the information from the CCS
component, considering all information available from the alert investigation at any given time. The
value of the CCS confidence index will change over time as the investigation progresses and as more
alerts are generated. Under the model assumptions, a fully investigated CCS alert will result in a Possible
determination. Subsequent CCS alerts are not fully investigated but incrementally increase the CCS
confidence index up to a maximum of value of 1.5. However, information from another monitoring and
surveillance or investigative component is necessary to elevate the threat level to Credible.
123
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
1. Time of call to GCWW
2. Location of call
Work Order Generation Module
(Converts calls to GCWW work orders)
1. Time of work order generation
2. Location of work order
CCS Event Detection Module
(Counting algorithm identifies potential call clusters)
1. Type of alert
2. Time of alert
3. Location of alert
CCS Alert Investigation Module
(Procedures used by GCWW personnel to
investigate a CCS alert)
Data table
Module
1. WUERM notification time
2. CCS confidence index
Figure A-6. Customer Complaint Surveillance Model
The primary inputs to the CCS model are shown in Table A-12 and include the date and time of utility
calls generated by the HI/HB model, the time and date of WOs, the number of customers in the call queue
waiting to talk to a customer service representative, and the WQM confidence index from the WQM
model described in Section A.5.
Table A-12. CCS Model Inputs
Input
Utility Call Time
Work Order
Generation Time
Description
The date and time a customer who has detected the contaminant calls the utility. These
values are created in the HI/HB model and reside in the Case Table.
The date and time that a WO is generated by the WO Generation module of the CCS
model.
124
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Input
Customer Service
Representative Call
Queue
WQM Confidence
Index
Description
The number of callers waiting to talk to a customer service representative at each time-
step. This value is the number of individuals waiting in the GCWW queue from the HI/HB
model.
Generated by the WQM model, is used as a proxy for water quality data in the area of a
CCS alert. It is used during the investigation of a CCS alert to determine if degraded water
quality is spatially correlated with customer complaints.
Parameters for the CCS model are listed in Table A-13. Key parameters include event detection system
scan times and thresholds as well as the time to investigate a CCS alert.
Table A-13. CCS Model Parameters
Parameter
WQ&T Chemist
Capacity
CCS Event Detection
System Scan Time
and Thresholds
CCS Event Detection
System Alert
Transmission Time
and Data Processing
Delay
Call Queue Threshold
Time to Investigate a
CCS Alert
Description
The number of WQ&T Chemists handling customer complaints during business and non-
business hours. The values for this parameter are based on GCWW staffing practices.
The size of the window (i.e., one day scan and two day scan) utilized to monitor current
data and evaluate it against recent historical data to generate CCS alerts. If the number of
utility calls or work orders in the scan window equals the threshold (four water quality calls
or three WOs), an alert will be generated. The values for the thresholds were obtained from
the CCS event detection system configuration file utilized by the GCWW CCS in June
2010.
The CCS event detection system processes data at least 2 minutes old and runs every
minute. Thus, alert generation is typically delayed 3 minutes from the last event in the
window. The values for this parameter were obtained from the CCS event detection system
configuration file utilized by the GCWW CCS in June 201 0.
The number of calls waiting to be answered by customer service representatives that is
necessary to cause the utility to increase suspicion that there may be a problem with the
water quality. This value was provided by GCWW and was confirmed in one of the drills
held during the evaluation period Cincinnati pilot.
The time necessary to query customer service representatives (business hours) or
dispatchers (non-business hours), evaluate the calls for clustering, check for active or
recent distribution work in the area of the calls, and review recent water quality data in the
area of the calls. The values for this parameter were obtained from drills and exercises
performed during the evaluation period of the Cincinnati pilot.
The outputs from the CCS model include the alert start time, WUERM notification time, and CCS
confidence index for alerts, and are described in Table A-14.
Table A-14. CCS Model Outputs
Output
CCS Alert Start Time
CCS Alert Location
WUERM Notification
Time
CCS Confidence
Index
Description
The date and time the CCS alert is generated.
The pito zone(s) that contain the calls that caused the alert.
The date and time at which the WUERM is notified of each unique CCS alert.
A time series of values from the CCS component indicating the reliability of information
available from the investigation of the CCS alert, and the degree of confidence in the
suspicion that the drinking water has been contaminated.
A.7 Public Health Surveillance
The PHS model was designed to simulate new and existing syndromic surveillance systems and
procedures used by Cincinnati area public health partners to detect unusual clusters of illness and disease.
The component operates by analyzing health seeking behaviors and identifying unusual trends that may
be an early indicator of an emerging outbreak.
125
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
PHS monitors a number of data streams, and there is a unique mechanism for event detection for each
data stream that involves automated analysis of the data or standard procedures to identify anomalies or
deviations from the base state of disease/illness within the population. If an alert is generated through one
of these systems, the local health partners work collaboratively with GCWW utility personnel to conduct
an investigation to determine whether or not the public health alert is related to contaminated drinking
water. A communicator protocol was implemented as a part of the alert investigation process to facilitate
discussions among GCWW and the public health partners regarding the possibility of water
contamination. During the discussions, representatives from each partner organization provide real-time
updates to further the investigation process.
Figure A-7 provides an overview of the PHS model showing the relationships among the three modules,
shown as green rectangles, which comprise the model: PHS Pre-processing module, PHS Event Detection
module and PHS Alert Investigation module. The inputs to and outputs from each module of the PHS
model are shown as blue parallelograms.
The first module to operate in the PHS model is the Pre-processing module. The inputs to this module are
located in the case table generated by the HI/HB model and include the time and action associated with
the health seeking behaviors taken by each exposed individual. The Pre-processing module converts
these actions into the format required by the event detection systems used to analyze the various data
streams: 911, EMS, DPIC, ED, and primary care physicians. In general, all data streams capture the
following information: case ID, location, symptom category, and the date and time that information from
the case entered the data stream,
The Event Detection module uses the outputs from the Pre-processing module to search for unusual
clusters of health seeking behavior. Each of the data streams uses a unique algorithm for event detection,
and the model was parameterized with the event detection system configurations used in the Cincinnati
pilot:
911: SaTScan analyzes 911 calls generated by the HI/HB model against a 21-day baseline
dataset of 911 calls generated during a portion of the evaluation period of the Cincinnati pilot.
911 calls will always generate an EMS response, so customers that call 911 will always be placed
in the EMS queue.
EMS: The Early Aberration Reporting System analyzes EMS run records generated by the
HI/HB model against a 21-day baseline dataset of EMS runs generated during a portion of the
evaluation period of the Cincinnati pilot. Each case uses the zip code of the location of the run
and the associated syndrome.
DPIC: Two unique surveillance methods are approximated in the DPIC module: volume-based
statistical surveillance and human surveillance. Both of these simple algorithms are based on
fixed thresholds for the number of DPIC calls generated within the HI/HB model.
ED: An algorithm analyzes ED visits generated within the HI/HB model. The algorithm
generates alerts when pre-established thresholds, determined for each syndrome category, are
exceeded.
Primary care physician: An algorithm analyzes primary care physician visits generated within
the HI/HB model. The algorithm generates alerts when pre-established thresholds are exceeded,
which are assigned on a contaminant-specific basis.
The PHS alerts generated by the Event Detection module provide the inputs to the Alert Investigation
module, which simulates the investigation process implemented by GCWW and its public health partners.
126
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
This module is based on alert investigation procedures developed for routine operation of the PHS
component of the Cincinnati CWS. This module accounts for the time required to recognize the alert and
perform a variety of investigative functions, including activating the communicator protocol, holding a
call to discuss the PHS alert investigation, and contacting frontline healthcare providers for more
information about individual cases. The outputs from this module include the time when results from the
investigation are reported to the WUERM and the PHS confidence index.
The PHS confidence index is an overall indicator of the reliability of the information from the PHS
component, considering all available data from all PHS alerts and the ongoing investigation. The value of
the PHS confidence index will change overtime as the investigation progresses. Under the model
assumptions, PHS can result in a Possible determination in one of two ways: 1) completion of a single
PHS alert in which water contamination cannot be ruled out as a potential cause, or 2) receipt of multiple
PHS alerts that increase the confidence index to 1.0 and an indication of potential contamination from
another component. While the PHS confidence index can increase above the threshold for Credible (2.0),
information from another component is necessary for a PHS alert to be considered a Credible indicator of
drinking water contamination. This additional information can come from another monitoring and
surveillance component (WQM or CCS) or an investigative component (site characterization (SC) or
laboratory analysis (LA) and is necessary to draw a potential connection between the PHS alert and the
drinking water.
127
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
1. Location of case
2. Health seeking behavior taken by individual
3. Time health seeking behavior was taken
Baseline PHS Data
(for 911 &EMS)
PHS Pre-processing Module
(Pre-processing of healthcare actions stored in the Case Table
converts input data to format required for event detection)
911 and EMS case
data superimposed
on baseline data
PHS Event Detection Module
(Five independent PHS tools that monitor: 1) 911 calls, 2) EMS runs,
3) DPIC calls, 4) ED visits, and 5) visits to a primary care physician)
1. Type of alert
2. Time of alert
3. Location of alert
PHS Alert Investigation Module
(Procedures used by GCWW personnel and local public health
partners to investigate a PHS alert)
Data table
Module
Database
1. WUERM notification time
2. PHS confidence index
Figure A-7. Public Health Surveillance Model
The primary inputs to the PHS model are shown in Table A-15 and include location, symptom level, and
the times of individual health seeking behaviors. These are the primary outputs generated from the
HI/HB model and recorded in the case table.
Table A-15. PHS Model Inputs
Input
Case ID
Location ID
Symptom Level
911 Call Time
Emergency Medical
Service Run Time
Description
A unique identifier for each individual exposed to contaminated water during a
contamination scenario. Case IDs are assigned in the HI/HB model and reside in the Case
Table.
The specific distribution system model node that the individual is assigned to for all
exposure events (i.e., home location).
Contaminant-specific category for symptoms experienced by an exposed individual. Values
were derived from an extensive review of contaminant databases and peer reviewed
literature.
The date and time the customer calls 91 1 in response to their current symptoms.
The date and time an EMS unit responds to and treats a symptomatic individual.
128
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Input
DPIC Call Time
ED Visit Time
Primary Care
Physician Visit Time
Description
The date and time the customer calls DPIC to seek medical advice in response to their
current symptoms.
The date and time the customer is admitted to a hospital ED to seek medical assistance in
response to their current symptoms.
The date and time the customer visits a primary care physician to seek medical assistance
in response to their current symptoms.
Parameters for the PHS model are listed in Table A-16. Key parameters include event detection system
alert parameter, the time to investigate a PHS alert, and the time to activate the communicator protocol.
Table A-16. PHS Model Parameters
Parameter
Description
Analysis Frequency
A parameter used in PHS event detection systems that determines how frequently data is
analyzed for anomalies. This scheduled time varies for the different surveillance tools of
the PHS model. The values for this parameter replicate the configuration of the PHS event
detection systems used in the Cincinnati CWS.
911 Alert Parameters
The parameters for determining the conditions that must be met to generate a 911 alert.
These include the minimum p-value (.025) for an alert cluster and the number of 911 cases
in the cluster (17 calls) for the cluster to be considered anomalous. The values for these
parameters replicate the configuration of the 911 event detection system used in the
Cincinnati CWS.
Emergency Medical
Service Alert
Parameters
The parameters for determining the conditions that must be met to generate an EMS alert.
These include the number of minutes of historical data used to establish the baseline
(10,080 minutes) and the ratio of EMS runs to zip codes in the alert (1.5). The values for
these parameters replicate the configuration of the EMS event detection system used in
the Cincinnati CWS.
Epicenter Alert
Parameters
Unlike the 911 and EMS event detection systems, Epicenter could not be used directly in
the PHS model. The behavior of Epicenter is replicated by applying syndrome-specific
thresholds for daily number of simulated cases to generate the alert. The thresholds were
determined through an analysis of historical emergency department data and established
at four standard deviations above the mean daily totals for the syndrome.
Primary Care
Physician and
Emergency
Department Physician
Disease Reporting
Alert Parameters
Threshold for the number of visits to primary care physicians or ED physicians above
which the health department is notified about the unusual frequency of patients expressing
similar symptoms. Reporting thresholds are contaminant-specific. The values for this
parameter are based on consultation with DPIC subject matter experts.
DPIC Statistical
Surveillance Alert
Parameters
The parameters for determining the conditions that must be met to generate a DPIC
statistical surveillance alert. These include the analysis window (24 hours) and the
threshold for calls to DPIC from within the same zip code (four calls) to trigger an alert.
The values for these parameters replicate the configuration of the DPIC's statistical
surveillance event detection system.
DPIC Human
Surveillance Alert
Parameters
The parameters for determining the conditions that must be met to generate a DPIC
human surveillance alert. This simple algorithm assumes that DPIC will suspect water
contamination if two calls to DPIC originate from the same node within 4 hours of each
other. The values for these parameters were determined through consultation with DPIC
subject matter experts.
Time to Investigate
Alert
The time to investigate a PHS alert and identify if the underlying cases are clustered and
potentially due to a common exposure route. The value of this parameter varies based on
the type of alert and was obtained from drills and exercises performed during the
evaluation period of the Cincinnati pilot.
Time to Activate
Communicator
The time (minutes) after local health partners complete their preliminary investigation,
initiate the communicator protocol, and convene a conference call to discuss the alert. The
value for this parameter was obtained from drills and exercises performed during the
evaluation period of the Cincinnati pilot.
Time for
Communicator
Discussion
The time for GCWW and local public health partners to discuss active PHS alert(s) and
determine whether water contamination is Possible. The value for this parameter was
obtained from drills and exercises performed during the evaluation period of the Cincinnati
pilot.
129
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
The outputs from the PHS model include the alert start time, WUERM notification time, and PHS
confidence index for alerts shown in Table A-17.
Table A-17. PHS Model Outputs
Output
PHS Alert Start Time
PHS Alert Location
WUERM Notification
Time
PHS Confidence
Index
Description
The date and time the PHS alert is generated and the appropriate public health agency is
notified.
Pito zone containing the initial underlying case(s) that generated the alert.
The date and time at which the WUERM is notified of each unique PHS alert.
A time series of values from the PHS component indicating the reliability of information
available from the investigation of the PHS alert, and the degree of confidence in the
suspicion that the drinking water has been contaminated.
A.8 Consequence Management
The CM model was designed to simulate the actions taken to investigate and respond to Possible water
contamination incidents in the distribution system. These actions are meant to minimize response and
recovery timelines through a pre-planned, coordinated effort. Investigative and response actions are
implemented to establish credibility, minimize public health and economic consequences, and ultimately
return the utility to normal operations. The model is largely based on the series of decision trees
documented in the GCWW Consequence Management Plan that guide the threat level determination
process and various response actions.
Figure A-8 shows the relationships among the modules that constitute the CM model, shown as green
rectangles, along with the outputs generated by the model, shown as blue parallelograms. The CM model
consists of five modules: Threat Level Determination (TLD) module, Site Characterization module,
Laboratory Analysis module, Public Notification module, and the Operational Response module.
Data generated by the monitoring and surveillance components are the primary inputs to the CM model,
and include: WUERM notification, component confidence indices, alarm types and alarm locations.
Location information is expressed in terms of pito zones, which are specific pressure zones identified
within the GCWW distribution system.
130
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Laboratory Analysis Module
(Procedures and methods used to
process and analyze samples and
report results)
Site Characterization Module
(Procedures conducted by the Site
Characterization Team, including:
field safety screening, rapid field
testing, sampling, and sample
transport to labs)
1. Component Cl
2. SC results
3. LA results
Threat Level Determination
(TLD) Module
(Simulates the investigative function of
the WUERM using component outputs
to determine when contamination is
Possible, Credible, or Confirmed)
Time of Possible,
Credible, and Confirmed
determinations
Public Notification Module
(Utility notification of the public
regarding use restrictions, such
as "do not use")
Operational Response Module
(Changes to system operations
made by the utility to reduce the
spread of contamination)
Data table
Module
1. Time of operational response
2. Time of public notification
Figure A-8. Consequence Management Model
The TLD module steps through each time-step of the simulation and considers data that would be
available at that time to establish if/when contamination is deemed Possible, Credible or Confirmed.
module simulates the investigative functions of the WUERM using information generated by the
This
131
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
monitoring and surveillance component models (ESM, CCS, WQM and PHS), SC, and LA to determine
the level of confidence in the possibility that a contamination incident has occurred. These TLD outputs
are the primary drivers for implementation of response actions such as a public notification and
operational response.
The SC module is initiated in response to Possible contamination as determined through the investigation
of an alert from one or more of the monitoring and surveillance components. Once the threat level
reaches Possible, one SC team will begin mobilization and will deploy to the location of the first
validated alert. This module includes SC team mobilization, travel time, deployment, site approach, field
safety screening, sample retrieval, rapid field testing, sampling for laboratory analyses, and transport to
GCWW for disposition of samples to method laboratories. This is a critical step in the investigation
process and involves collecting information from an investigation site to support the threat level
determination process. SC activities start with performing a site hazard assessment when approaching a
suspected contamination site(s) and ends when water samples are collected and sent to a laboratory for
analysis.
The LA module is based on procedures and methods GCWW and partner laboratories use to process and
analyze samples collected during the investigation of a Possible contamination incident. LA includes
mobilization of laboratories to prepare for sample receipt and analysis, sample delivery to laboratories,
sample analysis, data review, and reporting analytical results to the WUERM. There are two types of labs
that may be mobilized in the LA module: auto and triggered laboratories. Any samples collected over the
course of the investigation are always sent to all auto laboratories, where they collectively analyze the
samples for an established suite of contaminants. Triggered laboratories are used only in situations where
there is evidence to suggest that a potential contaminant is outside of the established suite of contaminants
analyzed by the auto laboratories. Typically, this evidence will come from one of the monitoring and
surveillance components (CCS, WQM, or CCS), the differential diagnosis generated by the HI/HB model
(see Table A-5), or SC results.
Each of the monitoring and surveillance components, along with SC and LA, produce a time series of
confidence indices, a numeric indicator of the strength of the signal from the component that
contamination has occurred, which are used by the TLD module to determine the overall threat level. The
confidence indices (CIs) for each component at each time-step are monitored and summed by the TLD
module to represent the overall threat level index (TLI):
TLI = C!ESM + C!WQM + CIccs + Clpns + CIsc + C!LA
The threat level determination process classifies a contamination threat to be Possible when one of the
following two conditions are met: 1) the investigation of an alarm from a single component is completed
and uncovers no benign explanation for the alarm, or 2) information from an ongoing alarm investigation
is supplemented by information from additional component alarms that are related spatially and
temporally. The threat level reaches Possible when:
The TLI > 1.0 and
The WUERM has been notified and informed of the results of the investigation, which results in
the continuation of the investigation of the Possible contamination incident.
In general, contamination is determined to be Credible when indicators of contamination from two or
more independent components is related temporally and/or spatially; however, in some cases, information
from only WQM or only ESM may be sufficient to establish that contamination is Credible. The threat
level reaches credible when:
The incident has met the criteria for Possible as described above and
132
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
TLI > 2.0 and
In the case of a PHS alarm, at least one of the other data streams has a confidence index > 0.0.
Contamination is Confirmed when definitive laboratory analysis results are available, or when a
preponderance of evidence indicates that contaminated drinking water poses a direct threat to public
health. As described previously in Section A.4, an ESM alert can lead to a Confirmed contamination if
the site investigation catches the contaminant injection in progress. The threat level reaches Confirmed
when:
The incident has met the criteria for Credible as described above and
TLI > 3.0
The threat level is an input to the Operational Response and Public Notification modules. These modules
generate the time, location, and type of response actions, which are ultimately used to revise parameters
within EPANET and the HI/HB model to determine revised consequences of each scenario with the CWS
in place. The consequences from the baseline condition (without CWS in place) can then be compared
with those from the CWS condition to determine the reduction in consequences attributable to
deployment of the CWS.
Once contamination is deemed Possible, the Operational Response module is executed. Operational
responses are changes to system operations that attempt to minimize the spread of contaminated water by
physically or hydraulically isolating portions of the distribution system.
Public notification is the series of notifications the utility, either by itself or in concert with public health
partners, makes to the public regarding use of drinking water. The Public Notification module is designed
to simulate the activities that lead up to issuance of a "do not use" notice to the public, which is intended
to prevent future exposures to the contaminated water. Preparation of a public notification begins once
contamination is deemed Possible and can be issued as soon as contamination is deemed Credible. Logic
in the HI/HB model determines if and when each individual complies with the use restriction (see the
parameter "time of stop use compliance" in Table A-5).
The primary inputs to the CM model, shown in Table A-18, are the confidence indices, alert times, and
the WUERM notification times. These inputs come from the component models and are used to
determine the threat level and the contaminant identification.
Table A-18. CM Model Inputs
Input
Component Type(s)
Alert Location ID
Alert Start Time
WUERM Notification
Time
Sample ID
Sampling Location
Component
Confidence Indices
Description
Indicator of the component that produced the alert. Options: ESM, WQM, CCS, and PHS.
The node associated with the location of the component alert. The values for this
parameter are generated by the component models.
The date and time assigned to the component alert by the component models.
The date and time at which the WUERM is notified about a component alert. The values
for this parameter are generated by the component models.
A unique identifier for each sample collected during a scenario, which can be traced back
to a sample location and collection time. The values for this parameter are generated by
the component in the model where the sample was collected.
The node from which samples were collected.
A time series of values from each component indicating the reliability of information
available from the investigation of the component alert and the degree of confidence in the
suspicion that the drinking water has been contaminated.
133
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Parameters for the CM model are listed in Table A-19. Key parameters include thresholds for threat level
determination, time to mobilize and deploy the SC team, time for laboratory analysis of samples, time to
evaluate and implement operational responses, and time to prepare a public notification.
Table A-19. CM Model Parameters
Parameter
Threshold for Threat
Level Determination
SC Team Mobilization
Time
Time to Deploy the
SC Team
Time for Laboratory
Mobilization
Time for Sample
Receipt, Disposition,
and Delivery to
Contract Labs
Time to Analyze
Samples and Perform
QC Data Review
Reporting Limits
Time to Prepare
Public Notification
Time to Implement
Operational
Response
Description
The minimum threat level index for Possible, Credible, and Confirmed determinations to be
declared. The values for this parameter were obtained from drills and exercises performed
during the evaluation period of the Cincinnati pilot.
The time it takes the SC team to mobilize, which begins with the time the WUERM directs
the SC team to deploy and ends with the time they leave for the site. Different values for
this parameter are established for normal business hours and non-business hours. The
values for this parameter were obtained from drills and exercises performed during the
evaluation period of the Cincinnati pilot.
The time it takes the SC team to deploy their equipment and prepare for the site approach
once they arrive at the investigation site. The values for this parameter were obtained from
drills and exercises performed during the evaluation period of the Cincinnati pilot.
The time it takes the laboratories to prepare to analyze samples from the time they are
notified by GCWW. The value for this parameter varies by method, laboratory, and
whether initial notification of the labs occurs during business or non-business hours. The
values for this parameter were obtained from utility and external lab procedures.
The time required for GCWW to receive and inventory samples collected in the field,
prepare chain of custody forms, deliver samples to in-house chemists, package samples
for shipment, and deliver samples to external labs. For laboratory analyses that are part of
the baseline suite, drive time was provided by GCWW or documented during drills
performed during the evaluation period of the Cincinnati pilot. For laboratory analyses that
are performed by triggered labs, drive time estimated based on the location of laboratory
that was assumed to be used.
The time required to analyze samples, review QC information, and prepare the results for
reporting. This is a laboratory-specific parameter and was estimated based on laboratory
method analysis time requirements.
The minimum reporting limit (i.e., concentration) for each contaminant simulated in the
study. The values for this parameter were obtained from actionable concentrations for
each contaminant provided by GCWW.
The time necessary to prepare and distribute a public notification through broadcast media
(e.g., television, radio, text notifications, etc.) such that it is available for public viewing. The
values for this parameter were obtained from drills and exercises performed during the
evaluation period of the Cincinnati pilot.
The time necessary to evaluate operational response options and select, plan, and
implement the operational response action that best protects the public and utility
infrastructure from exposure to contaminated water. Operational response actions
modeled include isolation of tanks and reservoirs and manipulation of pumps and valves.
The values for this parameter were obtained from drills and exercises performed during the
evaluation period of the Cincinnati pilot and discussions with utility operators.
The primary outputs from the CM model are shown in Table A-20, and include the threat level index, the
zone of impact, the time and action for an operational response, and the time of public notification. The
zone of impact is an output from the TLD module that identifies all pito zones that may be contaminated
at a given time-step based on information generated by the various components.
Table A-20. CM Model Outputs
Output
Threat Level Index
Description
A numeric indicator of the threat level associated with a potential contamination incident at
each time-step, which relates to the Threat Level as described below.
134
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Output
Threat Level
Contaminant
Identification
Confidence Index
Zone of Impact
Field Safety
Screening Results
Rapid Field Test
Results
Time Samples
Delivered to
Laboratories
Laboratory Results
Operational Change
Time of Public
Notification
Description
A discrete indicator of the level of confidence in the assertion that the distribution system
has been contaminated. There are three threat levels: Possible, Credible, or Confirmed.
The Threat Level is determined from the value of the Threat Level Index and other criteria,
such as notification of the WUERM.
A numeric indicator of the confidence of utility officials in the identity of the contaminant,
which is based on information from all components (including SC and LA). Note that this
parameter considers input from local public health partners, as modeled by the Differential
Diagnosis module generated by the HI/HB model.
A running list of all the pito zones that could be contaminated according to the information
available to the WUERM, updated at each time-step.
The date, time, and results ("normal" or "abnormal") for all field safety screening performed
during SC activities. Based on scenario assumptions, the results of field safety screening
will be normal at locations other than the site of contaminant injection.
The date, time, and results ("normal" or "abnormal") for all water quality testing performed
during SC activities. The results from rapid field testing are based on the contaminant
concentration at the time and node from which the sample was collected for field testing. If
the contaminant concentration is above a specified minimum detection level fora given
test, the results are reported as abnormal.
The date and time when samples are delivered to the specified laboratory for sample
disposition and analysis.
The date, time, and results ("normal" or "abnormal") for a specific laboratory analysis. The
results from laboratory analysis are based on the contaminant concentration at the time
and node from which the sample was collected for laboratory analysis. If the contaminant
concentration is above a specified minimum detection level for a given analysis, the results
are reported as abnormal.
The date and time at which the operational change is implemented. The operational
change is translated to a specific EPANET operational rule that is changed to model the
response.
The time that a "do not use" notice is issued to the public.
135
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Appendix B: Benefit-Cost Analysis Methodology
B.1 Introduction
To evaluate the sustainability of the Cincinnati Contamination Warning System (CWS) in a quantitative
fashion, a benefit-cost analysis was conducted. This type of financial analysis compares the benefits and
costs to determine which value is larger. Therefore, it requires a complete and transparent accounting of
the actual costs of deploying the Cincinnati CWS and estimated benefits derived from operation of the
system expressed in comparable terms, which for this analysis are U.S. dollars in 2007 (the year the
system was deployed). The benefits and costs were assessed over a 20-year lifecycle. The basis for
selecting a 20-year lifecycle is the heavy reliance of the system on sensor technology, information
technology (IT) systems, and human processes that will likely be obsolete and thus need to be replaced or
updated within 20 years.
The remainder of this document presents the systematic benefit-cost analysis methodology in the
following two subsections:
Identification of Monetizable Costs and Benefits. This section describes the approach and
sources used to determine the overall costs and benefits associated with the deployment and
operation of the Cincinnati CWS.
Financial Analysis. This section presents the methodology used to compare the benefits and
costs. It also presents the methodology used to determine the lifecycle costs of the CWS, and to
estimate the monetary value of benefits associated with identifying and responding to a
contamination incident.
B.2 Identification of Monetizable Costs and Benefits
To conduct the benefit-cost analysis, a comprehensive list of costs and benefits was compiled from which
only those that could be monetized were considered. The costs of deploying and operating the Cincinnati
CWS were readily monetized as the cost data had been tracked during implementation of the system.
However, benefits of the CWS were more challenging to identify and, with one exception, depended on
the reduction in consequences resulting from early detection and response to a contamination incident
using CWS capabilities, which was monetized using assumptions discussed in Section B.3.
B.2.7 Identification of Monetizable Costs
The costs considered in the benefit-cost analysis included all costs associated with the implementation
and operation of the CWS during the 20-year lifecycle; however, they did not include those associated
with pre-existing resources or operations, even if those capabilities were leveraged for the CWS. The
main source of the cost data used in performing this analysis was the Water Security Initiative: Cincinnati
Pilot Post-Implementation System Status report (USEPA, 2008). This report describes the configuration
and cost of the pilot CWS as it was deployed in Cincinnati, Ohio as of December 2007. The total cost of
the Cincinnati CWS over an assumed 20-year lifecycle was determined by summing all costs associated
with implementation and operation of the CWS. The major cost elements for the Cincinnati CWS are
described in Table B-l.
136
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Table B-1. Cost Elements for Implementation and Operation of the
Cincinnati CWS
Cost Element
Description
Deployment Costs
The total costs for designing and implementing the CWS. Deployment costs
include all US Environmental Protection Agency (EPA), extramural, Greater
Cincinnati Water Works (GCWW) and local partners' labor costs as well as other
direct charges for equipment, consumables, and purchased services.
Modification Costs
The cost of modifications to the Cincinnati CWS after system implementation was
completed in January 2009 through the end of the evaluation period in June 2010.
The costs were tracked by EPA and GCWW over the evaluation period of the
pilot, and include all appropriated equipment and labor costs associated with the
modification.
Equipment Renewal and
Replacement Costs
The costs associated with replacing equipment during the 20-year lifecycle of the
Cincinnati CWS. The costs were identified for major pieces of equipment,
generally with a replacement value of $500 or more. The useful life of the
equipment was estimated using field experience with the equipment,
manufacturer-provided data, and the recommendations of subject matter experts.
Salvage Value
The salvage value is the estimated value of the system components at the end of
the 20-year lifecycle of the Cincinnati CWS. The salvage value was estimated
using straight line depreciation for all equipment with an initial value greater than
approximately $1,000 and represents a credit against the system costs in the
benefit-cost analysis. The useful life of the equipment was estimated from
experience with equipment at the Cincinnati pilot along with professional
judgment.
Operation and Maintenance
(O&M) Costs
The costs incurred to operate and maintain the CWS over the 20-year lifecycle of
the Cincinnati CWS. The O&M costs represent all EPA, extramural, GCWW, and
local partners' labor costs as well as other direct charges for consumables and
purchased services for maintaining the CWS. Additionally, the annual cost to
maintain and update CWS documentation was extrapolated from the costs
incurred to update documents following drills and exercises conducted during the
pilot evaluation period using an assumed frequency of future drills and exercises
over the 20-year lifecycle of the CWS.
B. 2.2 Identification of Monetizable Benefits
The benefits considered in the benefit-cost analysis include all improvement in GCWW's capability to
detect and respond to unusual water quality conditions realized through operation of the CWS. Benefits
were grouped into one of two broad categories: those resulting from the reduction in consequences from a
contamination incident and those related to day-to-day utility operations (dual-use benefits).
Information about dual-use was obtained directly from GCWW and local partner staff through a variety
of forums, including routine component review meetings, lessons learned workshops, and exit interviews
as described in Section 3.5. These forums provided an opportunity for front line personnel, supervisors,
senior managers and representatives from partner organizations with an opportunity to provide feedback
on the Cincinnati CWS in areas such as the value of various enhancements implemented during the pilot,
application of the CWS to activities other than contaminant detection, and long-term plans for the CWS.
With only one exception, there was insufficient information to monetize dual-use benefits. The one dual-
use benefit that could be monetized was the reduction in chlorine applied to maintain the target
disinfectant residual throughout the distribution system, which was realized through the chlorine residual
data generated by the Water Quality Monitoring (WQM) component.
The other category of benefits considered in the benefit-cost analysis is the reduction in the consequences
of a contamination incident due to early detection and response realized through operation of the CWS.
Because there were no contamination incidents in the GCWW distribution system over the course of the
pilot, the Cincinnati CWS model, described in Appendix A, was used to estimate the consequences of
137
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
simulated contamination incidents both with and without the CWS in operation. The difference in
consequences under these two conditions was calculated to determine the benefits due to consequences
avoided that were attributable to the CWS.
The Cincinnati CWS model allows for the evaluation of various consequences that could result from
intentional contamination. Table B-2 describes three types of benefits, expressed in terms of reduced
consequences, attributable to early detection and response to a contamination incident through the CWS.
These benefits were monetized using various assumptions described in Section B.3.
Table B-2. Monetizable Benefits Attributable to the Cincinnati CWS
due to the Reduction in Consequences from a Contamination
Incident
Benefit in terms of Consequence Reduction
Public health
Revenue
Remediation
Description
The reduction in fatalities, number of people requiring
medical treatment, and lost leisure time attributable to early
detection and response through operation of the CWS.
The reduction in lost water revenue, lost wastewater
revenue, and lost business revenue attributable to early
detection and response through operation of the CWS.
The reduction in distribution system remediation cost and
the cost of an alternate water supply attributable to early
detection and response through operation of the CWS.
B.3 Financial Analysis
The financial analysis required the benefits and costs identified to be expressed in terms of the value of a
dollar in a common reference year. Because the Cincinnati CWS was substantially complete in 2007, it
was decided to express the present value (PV) of all costs and benefits in 2007 dollars, allowing for an
unbiased comparison.
B.3.1 Overview of Present Value Calculations
With the costs and benefits occurring over the 20-year lifecycle of the CWS, their values required
adjustment to reflect the time-value of money (i.e., one dollar in the future is worth less than one dollar
today due to the investment potential of that dollar). As discussed earlier, 2007 was selected as the
reference year for all PV calculations because most of the deployment costs were incurred in that year.
All costs incurred prior to 2007 were adjusted to 2007 dollars by using the change in the Consumer Price
Index (CPI) between 2007 and the year that the cost was incurred. Future costs were adjusted to 2007
dollars by using a 2.1% annual discount rate. The general assumptions used to calculate the PV of the
CWS costs and benefits are presented in Table B-3.
Table B-3. Cincinnati CWS Present Value Assumptions
Description
Term of Analysis
Present Value
Discount Rate for PV
Cost Basis
20 years
2007 dollars (no inflation)
2.1%
Source
Subject matter expert judgment regarding the
useful life of the CWS
Year in which most of the implementation costs of
the Cincinnati CWS were incurred
Office of Management and Budget, 2010
138
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
B.3.2 Monetization of Costs
The lifecycle cost analysis for the Cincinnati CWS was performed using the costs incurred to design,
deploy, operate and maintain the Cincinnati pilot CWS and includes the items described in Section B.2.1.
The modification costs were combined with the deployment costs for the purpose of the lifecycle cost
analysis. The lifecycle cost was determined by calculating the PV of the annualized O&M costs, the
periodic renewal/replacement costs, the salvage value of the equipment, and combining these annualized
costs with the deployment and modification costs. As indicated above, the monetized costs and benefits
were adjusted to 2007 dollars using the change in the Consumer Price Index between 2007 and the year
that the cost was incurred.
While a 20-year lifecycle was assumed for the entire CWS, individual pieces of equipment and
subsystems would need to be replaced or updated more frequently; thus, costs to update the CWS would
occur over the entire 20-year lifecycle rather than as one lump sum at the end of that period. The useful
life assumptions used for major pieces of CWS equipment are presented in Table B-4.
Table B-4. CWS Component Useful Life
CWS Equipment
Water Quality Sensors
ESM Contact Alarms
Security Lighting
Fixed Cameras
Video System
Laboratory Instruments
Field Instruments
Information Technology (IT)
Systems
Documentation
Useful Life (years)
3 to 7
7
15
7
5
10
7
5
2 to 7
Cost Assumptions
$3,700 to $24,950, per sensor
$260, per contact alarm
$311, per lighting fixture
$1,037, per camera
$1 1 ,000 for the entire system
$585 to $56,122, per instrument
$645 to $7,750, per instrument
$35,822 for all CWS-specific IT systems
$7,280 for all documents, if updated in-house
B.3.3 Monetization of Dual-use Benefits
The only dual-use benefit of the Cincinnati CWS that was monetizable was a reduction in chlorine usage
resulting from the utilization of chlorine data from the WQM component. To calculate the cost savings,
the utility provided six months of chlorine dose data, indicating the changes in the dosages that occurred
as a result of chlorine sensor data (10,011 Ibs), and their cost for chlorine ($470 per ton.) The cost saving
from this period was doubled to represent an annual value, and it was assumed to represent a typical year.
B.3.4 Monetization of Benefits during a Contamination Incident
The general approach and key assumptions used to estimate the monetary value of the consequence
avoided due the operation of the CWS included:
Fatalities. The fatalities cost was calculated by multiplying the number of lives lost by a unit
value per life estimated to be $7.1 million.
Medical Treatment. The medical treatment cost was determined by multiplying the number of
individuals who received medical treatment by the respective treatment costs. Treatment costs
were estimated from the Healthcare Cost and Utilization Project using either the International
Classification of Diseases, ninth revision code or the clinical classification category for treatment
of the specific contaminant and the estimated length of hospital stay required for that
contaminant. These figures include the cost of hospitalization, medicine, supportive care, and
139
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
prophylactic treatment but not the cost of treatment for chronic illness resulting from the
exposure. The contaminant-specific medical treatment costs are reported in Table B-5.
Lost Leisure Time. The lost leisure time due to illness was determined by multiplying the
number of individuals who experienced illness by half of the average hourly wage rate in
Hamilton County ($11.34 per hour), times 16 hours per day for the remediation period. Note that
lost wages are included under Lost Business Revenue.
Alternate Water Supply. It was assumed that a temporary alternate water supply would be
provided using bottled water. The cost of bottled water was multiplied by the gallons consumed
per person per day (1.28 gallons), multiplied by the number people affected, multiplied by the
duration of the outage.
Water System Remediation. The remediation cost was determined by calculating the cost of all
labor, equipment, and treatment chemicals needed to treat the contaminated water prior to
disposal, and remediate distribution system pipes and storage tanks. The remediation process was
considered from planning through demobilization. An overview of the assumed remediation
strategy for each contaminant considered in the benefit-cost analysis is presented in Table B-6.
Lost Drinking Water Revenue. The water revenue lost was determined by calculating the
demand at the affected nodes for the duration of the remediation period and multiplying it by the
average water service revenue of $2.53 per 1,000 gallons.
Lost Wastewater Revenue. The wastewater revenue lost was calculated by prorating the average
daily revenue for the utility ($423,871) by the percentage of the service area affected and
multiplying by the duration of the remediation period in days.
Lost Business Revenue. The business revenue lost was estimated by assuming that all businesses
in zip codes affected by the contamination incident would be shut down for the duration of the
remediation period. U.S. 2000 Census data reports the yearly revenue generated per zip code,
which is converted into a value for daily revenue generation by dividing by 365. The daily
revenue value was then multiplied by the number of days in the remediation period. For zip
codes that were partially contaminated, the daily revenue value was proportionally adjusted by
the percentage of the zip code affected.
Table B-5. Contaminant-specific Medical Treatment Cost per Illness
Contaminant
Toxic Chemical 1
Toxic Chemical 5
Toxic Chemical 6
Toxic Chemical 7
Toxic Chemical 8
Biological Agent 3
Biological Agent 4
Biological Agent 5
Biological Agent 6
Nuisance Chemical 1
Value (2007 dollars)
$9,098
$11,728
$11,728
$11,728
$11,896
$55,663
$7,426
$7,665
$9,240
$0
Assumed Medical Treatment1
3.2 days of supportive therapy
3.8 days of supportive therapy and agent-specific medication
3.8 days of supportive therapy and agent-specific medication
3.8 days of supportive therapy and agent-specific medication
3.8 days of supportive therapy
11 days of supportive therapy
4.8 days of supportive therapy and agent-specific medication
5.2 days of supportive therapy and agent-specific medication
5.2 days of supportive therapy and agent-specific medication
Not applicable. Acute illnesses do not occur from exposures to
doses assumed in this study.
Supportive therapy includes any form of treatment intended to relieve symptoms or help the patient live that does not directly
address the causative agent for the illness. Supportive therapy may include administration of intravenous fluids and mechanically
assisted breathing.
140
-------
Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot
Table B-6. Contaminant-specific Remediation Methods and Significant Cost Factors
Remediation Method
Application of chlorine with contact
time of two hours followed by flushing
and discharging to combined sewer
Application of chlorine with contact
time of two hours under acidic
conditions followed by flushing and
discharging to combined sewer
Application of chlorine with contact
time of two hours under alkaline
conditions followed by flushing and
discharging to combined sewer
Application of dispersant with contact
time of two hours followed by flushing
and discharging to combined sewer
Application of acidified water with a
contact time of two hours followed by
flushing to reverse osmosis treatment
unit
Significant Cost Factors
Chemical feeders ($10,000 each)
Sodium Hypochlorite ($0.91/gal)
Chemical feeders ($10,000 each)
Sodium Hypochlorite ($0.91/gal)
Hydrochloric Acid ($1.57/gal)
Chemical feeders ($10,000 each)
Sodium Hypochlorite ($0.91/gal)
Sodium Hydroxide ($1.71/gal)
Chemical feeders ($10,000 each)
Dispersant ($1.91/lb)
Chemical feeders ($10,000 each)
Hydrochloric Acid ($1.57/gal)
Reverse Osmosis Treatment Units ($500,000 ea)
Concentrate Disposal ($20/gal)
Contaminants
Toxic Chemical 5
Toxic Chemical 6
Biological Agent 3
Biological Agent 4
Biological Agent 5
Biological Agent 6
Toxic Chemical 7
Toxic Chemical 1
Nuisance Chemical 1
Toxic Chemical 8
141
------- |