&EPA
United States
Environmental Protection
Agency
Water Security Initiative: Evaluation of the
Consequence Management Component of the
Cincinnati Contamination Warning System Pilot
Monitoring and Surveillance
Water Quality Monitoring
Enhanced Security Monitoring
Customer Complaint Surveillance
ublic Health Surveillance
Possible Contamination
Consequence Management
Sampling and Analys
Response
Office of Water (MC-140)
EPA-817-R-14-001F
April 2014
-------
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
The Water Security Division of the Office of Ground Water and Drinking Water has reviewed and
approved this document for publication. This document does not impose legally binding requirements on
any party. The findings in this report are intended solely to recommend or suggest and do not imply any
requirements. Neither the U.S. Government nor any of its employees, contractors or their employees
make any warranty, expressed or implied, or assumes any legal liability or responsibility for any third
party's use of or the results of such use of any information, apparatus, product or process discussed in this
report, or represents that its use by such party would not infringe on privately owned rights. Mention of
trade names or commercial products does not constitute endorsement or recommendation for use.
Questions concerning this document should be addressed to:
Jeff Pencil
U.S. EPA Water Security Division
1200 Pennsylvania Ave, NW
Mail Code 4608T
Washington, DC 20460
(202)564-0818
or
Steve Allgeier
U.S. EPA Water Security Division
26 West Martin Luther King Drive
Mail Code 140
Cincinnati, OH 45268
(513)569-7131
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Acknowledgements
The Water Security Division of the Office of Ground Water and Drinking Water would like to recognize
the following individuals and organizations for their assistance, contributions, and review during the
development of this document.
• Yeongho Lee, Greater Cincinnati Water Works
• Jeff Swertfeger, Greater Cincinnati Water Works
• Mike Tyree, Greater Cincinnati Water Works
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Executive Summary
The goal of the Water Security Initiative (WSI) is to design and demonstrate an effective multi-
component warning system for timely detection and response to drinking water contamination threats and
incidents. A contamination warning system (CWS) integrates information from multiple monitoring and
surveillance components to alert the water utility to possible contamination and uses a consequence
management plan (CMP) to guide response actions.
System design objectives for an effective CWS are: spatial coverage, contaminant coverage, alert
occurrence, timeliness of detection and response, operational reliability, and sustainability. Metrics for
the consequence management (CM) component were defined relative to the system metrics common to all
components in the CWS, but the component metric definitions provide an additional level of detail
relevant to the CM component. Evaluation techniques used to quantitatively or qualitatively evaluate
each of the metrics include analysis of empirical data from routine operations, drills and exercises,
modeling and simulations, forums, and an analysis of lifecycle costs (see Section 3.0). This report
describes the evaluation of data collected from the CM component from the period of January 2008 -
June 2010.
The major outputs from the evaluation of the Cincinnati pilot include:
1. Cincinnati Pilot System Status, which describes the post-implementation status of the Cincinnati
pilot following the installation of all monitoring and surveillance components.
2. Component Evaluations, which include analysis of performance metrics for each component of
the Cincinnati pilot.
3. System-Level Performance Summary, which integrates the results of component evaluations, the
simulation study and results of a benefit-cost analysis.
The reports that present the results from the evaluation of the system and each of its six components are
available in an Adobe portfolio, Water Security Initiative: Comprehensive Evaluation of the Cincinnati
Contamination Warning System Pilot (USEPA 2014).
Consequence Management Component Design
The CM component was designed to provide guidance and equipment that would facilitate the Cincinnati
pilot with the management of possible drinking water contamination incidents as detected by one or more
of the CWS monitoring and surveillance components. Thus, the CM component was comprised primarily
of incident response plans and communication equipment that would minimize response times, and
therefore, minimize deleterious effects from contamination. The three CM design elements included: 1)
incident response plans; 2) response partner network; and 3) communication equipment (see Section 2.0).
Of the six design objectives identified for the CWS (contaminant coverage, spatial coverage, timeliness of
detection and response, operational reliability, alert occurrence and sustainability), only two (timeliness of
detection and response and sustainability) are directly applicable to the CM component.
A summary of the results used to evaluate whether the CM component met each of the design objectives
relevant to this component is provided below.
Methodology
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Several methods were used to evaluate CM performance. Data was tracked over time to illustrate the
change in performance as the component evolved during the evaluation period. Statistical methods were
also used to summarize large volumes of data collected over either the entire or various segments of the
evaluation period. Data was also evaluated and summarized for each reporting period over the evaluation
period. In this evaluation, the term reporting period is used to refer to one month of data that spans from
the 16th of the indicated month to the 15th of the following month. Thus, the January 2008 reporting
period refers to the data collected between January 16th 2008 and February 15th 2008. Additionally, five
drills and three full-scale exercises designed around mock contamination incidents were used to practice
and evaluate the full range of procedures, from initial detection through response.
Because there were no contamination incidents during the evaluation period, there is no empirical data to
fully evaluate the detection capabilities of the component. To fill this gap, a computer model of the
Cincinnati CWS was developed and challenged with a large ensemble of simulated contamination
incidents in a simulation study. An ensemble of 2,015 contamination scenarios representing a broad range
of contaminants and injection locations throughout the distribution system was used to evaluate the
effectiveness of the CWS in minimizing public health and utility infrastructure consequences. The
simulations were also used for a benefit-cost analysis, which compares the monetized value of costs and
benefits and calculates the net present value of the CWS. Costs include implementation costs and routine
operation and maintenance labor and expenses, which were assumed over a 20 year lifecycle of the CWS.
Benefits included reduction in consequences (illness, fatalities and infrastructure damage) and dual-use
benefits from routine operations.
Design Objective: Timeliness of Response
For CM, timeliness of response refers to the time it took the Cincinnati pilot to verify, characterize, and
respond to a contamination incident as detected by one or more of the CWS monitoring and surveillance
components. Factors that impact this objective include: time to notify response partners; time for
deploying field personnel and equipment; time for assessing hazard levels; time for collecting and
screening drinking water samples; time to identify and implement operational responses; time to identify
and implement public health response; time to determine threat levels; time to implement public
notification, and time to restore the system to normal operations (see Sections 4.0, 5.0 and 6.0). Site
characterization (SC) activities, which describe the field response, investigation and sampling procedures,
are described in Water Security Initiative: Evaluation of the Sampling and Analysis Component of the
Cincinnati Contamination Warning System Pilot (USEPA, 2013a).
One key metric for CM, which is implemented after an incident has been determined to be Possible, was
the time for threat level determination. This included the amount of time required for investigative
actions leading to both Credible and Confirmed determination (see Section 4.0). For a simulated
contamination incident exercise, a hierarchy of investigation information types evolved, which seemed to
accelerate the progression through threat level determinations. Two primary examples from the exercises
included:
1. Number of alerts, system connectivity, positive rapid filed tests and signs of intrusion accelerated
the declaration of Credible contamination incident and
2. Signs of intrusion and health impacts accelerated the declaration of Confirmed (assumed
contamination) incidents and did not necessarily depend on positive laboratory analysis.
The simulation study was utilized to determine the impact of various metrics on threat level
determination. The analyses resulted in a better understanding of the role that the type of contaminant,
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
the number of components detecting the incident, and the number of alerts play in threat level
determination. Three primary examples of the findings include:
1. Toxic chemical contaminants are identified much quicker than biological agents, resulting in
earlier declaration of Credible and Confirmed determination;
2. The number of components detecting the contaminant resulted in an accelerated declaration of
Credible and Confirmed determination; and
3. The number of alerts reduced the amount of time required for Credible determination declaration.
A second key metric was the average time for identifying appropriate response actions (e.g. operational
responses and public notification) during a simulated contamination incident. The specific threat levels in
the simulated contamination exercises varied, but these times were clearly influenced by the
circumstances presented by the exercise scenario. Operational responses were initially driven by what
actions the utility could implement quickly to isolate or slow contamination without impacting service to
customers. As the incident progressed, new investigation evidence was used to revise response actions or
implement new response actions, as necessary.
The time to develop and implement public notification was consistent throughout the exercises, with an
average time of 169 minutes from direction to prepare to release. Given the variability of exercise scopes,
and the accompanying revision of the crisis communication plan, it was not possible to make statistical
inferences concerning the improvement.
Figure ES-1 is a timeline progression (in hours) showing major response action milestones for the
Cincinnati Full Scale Exercise (FSE) 2, the largest of the three full scale exercises that were conducted
during the pilot. Field-verified timeline information from drills and full scale exercises was used to build
the CM portion of the simulation model. For more information on this topic, see the relevant subsections
regarding Timeliness of Response for each CM component.
00:00 02:10 02:30 05:00
WQM WQM Alerts
Alert #1 #2
02:00
Operations Checked
Director Notified
r i
CCS Alert Expanded 05:30
Isolation Site
08:40
Public 09:30
Notification Crisis
Response Characterization Response Communication
Action Activities
02:40
WQM Alerts
#4
03:00
Initial Isolation
Response Action
r^ •*
V 1
Action Press Conference
06:30
V ~\
PHS Alert
r ^
r i
10:30
Flushing
Response
Action
r "^
r
.,,..,,,,,,
01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00
00:00 11:30
Figure ES-1. Timeline Progression for Response Actions During Full Scale Exercise 2
Design Objective: Sustainability
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Sustainability is a key objective in the design of a CWS and each of its components, which for the
purpose of this evaluation is defined in terms of the cost-benefit trade-off (see Section 7.0). Costs are
estimated over the life cycle of the system to provide an estimate of the total cost of ownership and
include the implementation costs, enhancement costs, operation and maintenance (O&M) costs, renewal
and replacement costs, and the salvage value. The benefits derived from the system are defined in terms
of primary and dual-use benefits. Metrics that were evaluated under this design objective include: costs,
benefits, and compliance. The costs used in the calculation of lifecycle costs for the CM component are
presented in Table ES-1. These costs were tracked as empirical data during the design and
implementation phase of project design, and were analyzed through a benefit-cost analysis of the
Cincinnati pilot (see Section 7.0). It is important to note that the Cincinnati CWS was a pilot research
project, and as such incurred higher costs than would be expected for a typical large utility
installation.
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Table ES-1. Cost Elements used in the Calculation of Lifecycle Cost
Parameter
Implementation Costs
Annual O&M Costs
Renewal and Replacement Costs
Salvage Value1
Value
$1,430,627
$33,948
$22,624
-
Calculated using major pieces of equipment.
To calculate the total lifecycle cost of the CM component, all costs and monetized benefits were adjusted
to 2007 dollars using the change in the Consumer Price Index between 2007 and the year that the cost or
benefit was realized. Subsequently, the implementation costs, renewal and replacement costs, and annual
O&M costs were combined to determine the total lifecycle cost:
CM Total Lifecvcle Cost: $2,000,828*
*Actual costs from the above table were adjusted to 2007 dollars to calculate the total 20 year life cycle cost.
A similar CM component implementation at another utility should be less expensive when compared to
the Cincinnati pilot as it could benefit from lessons learned and would not incur research-related costs.
Benefits were measured by identifying applications of the CWS to any other purpose other than detection
of intentional and unintentional drinking water contamination incidents. Information was collected from
forums, lessons learned workshops and interviews. Key benefits that were identified included: stronger
interagency relationships with response partners, strengthened incident command structure and increased
preparedness of utility management and staff to respond to "all-hazards" type incidents.
Compliance with protocols and procedures necessary to operate and maintain the CWS is the ultimate
measure of the sustainability of the CWS, including the CM component. Compliance was evaluated
through documentation of qualitative data during drills and exercises and during forums with the pilot,
including lessons learned workshops. This was demonstrated through 100% utility participation in full
scale exercises where utility personnel were able to obtain a better understanding of CM procedures
through response to simulated water contamination incidents. For more information on this topic, see the
relevant subsections regarding Sustainability for each CM component.
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Table of Contents
LIST OF FIGURES 10
LIST OF TABLES 11
SECTION 1.0: INTRODUCTION 1
1.1 CWS DESIGN OBJECTIVES 1
1.2 ROLE OF CM IN THE CINCINNATI CWS 2
1.3 OBJECTIVES 3
1.4 DOCUMENT ORGANIZATION 3
SECTION 2.0: OVERVIEW OF THE CM COMPONENT 4
2.1 INCIDENT RESPONSE PLANS 4
2.2 RESPONSE PARTNER NETWORK 5
2.3 COMMUNICATION EQUIPMENT 6
2.4 CM COMPONENT ROLES AND RESPONSIBILITIES 6
2.5 SUMMARY OF SIGNIFICANT CM COMPONENT MODIFICATIONS 7
2.6 TIMELINE OF CM DEVELOPMENT PHASES AND EVALUATION -RELATED ACTIVITIES 9
SECTION 3.0: METHODOLOGY 10
3.1 TRAINING AND EXERCISE PROGRAM 10
3.1.1 Discussion-Based Exercises 10
3.1.2 Operations-Based Exercises 12
3.2 SIMULATION STUDY 14
3.3 FEEDBACK FORUMS 17
3.4 ANALYSIS OF LIFECYCLE COSTS 17
SECTION 4.0: PERFORMANCE OF INCIDENT RESPONSE PLANS 19
4.1 DESIGN OBJECTIVE: TIMELINESS OF RESPONSE 19
4.1.1 Credibility Determination 20
4.1.2 Response Actions 29
4.1.3 Remediation and Recovery 40
4.1.4 Summary 41
SECTION 5.0: PERFORMANCE OF INTEGRATION OF THE RESPONSE PARTNER NETWORK 44
5.1 DESIGN OBJECTIVE: TIMELINESS OF RESPONSE 44
5.1.1 Understanding Response Partner Roles and Responsibilities 44
5.1.2 Integration of Response Partners 45
5.1.3 Nature of Response Partner Support 46
5.1.4 Summary 47
SECTION 6.0: PERFORMANCE OF COMMUNICATION EQUIPMENT 48
6.1 DESIGN OBJECTIVE: TIMELINESS OF RESPONSE 48
6.1.1 Use of 800 MHz Radios 48
6.1.2 Summary 48
SECTION 7.0: SUSTAINABILITY 49
7.1 COSTS 49
7.2 BENEFITS 52
7.3 SUMMARY 53
SECTION 8.0: SUMMARY AND CONCLUSIONS 55
8.1 DESIGN OBJECTIVE: TIMELINESS OF RESPONSE 55
8.1.1 Incident Response Plans 55
8.1.2 Integration of Response Partner Network 57
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
8.1.3 Communication Equipment 57
8.2 DESIGN OBJECTIVE: SUSTAINABILITY 57
SECTION 9.0: REFERENCES 63
SECTION 10.0: ABBREVIATIONS 64
SECTION 11.0: GLOSSARY 66
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
List of Figures
FIGURE 2-1. TIMELINE OF CM COMPONENT ACTIVITIES 9
FIGURE 4-1. GENERIC CM TIMELINE 20
FIGURE 4-2. COMPARISON OF FSE TIME FROM POSSIBLE TO CREDIBLE DETERMINATION 21
FIGURE 4-3. TIME FROM POSSIBLE TO CREDIBLE DETERMINATION BY CONTAMINANT TYPE 22
FIGURE 4-4. TIME FROM POSSIBLE TO CREDIBLE DETERMINATION BY NUMBER OF COMPONENTS DETECTING 23
FIGURE 4-5. TIME FROM POSSIBLE TO CREDIBLE DETERMINATION BY NUMBER OF ALERTS RECEIVED 24
FIGURE 4-6. COMPARISON OF FSE TIME FROM POSSIBLE TO CONFIRMED DETERMINATION 26
FIGURE 4-7. TIME FROM POSSIBLE TO CONFIRMED DETERMINATION BY CONTAMINANT TYPE 27
FIGURE 4-8. TIME FROM POSSIBLE TO CONFIRMED DETERMINATION BY NUMBER OF COMPONENTS DETECTING 28
FIGURE 4-9. TIME FROM POSSIBLE TO CONFIRMED DETERMINATION BY NUMBER OF ALERTS RECEIVED 29
FIGURE 4-10. TIME FROM POSSIBLE CONTAMINATION TO PUBLIC HEALTH RESPONSE BY CONTAMINANT TYPE 31
FIGURE 4-11. TIME FROM POSSIBLE TO PUBLIC HEALTH RESPONSE BY NUMBER OF COMPONENTS DETECTING 32
FIGURE 4-12. TIME FROM POSSIBLE TO PUBLIC HEALTH RESPONSE BY NUMBER OF ALERTS RECEIVED 32
FIGURE 4-13. TIME FROM POSSIBLE TO OPERATIONAL RESPONSE BY CONTAMINANT TYPE 35
FIGURE 4-14. TIME FROM POSSIBLE TO OPERATIONAL RESPONSE BY NUMBER OF COMPONENTS DETECTING 36
FIGURE 4-15. TIME FROM POSSIBLE TO OPERATIONAL RESPONSE BY NUMBER OF ALERTS RECEIVED 37
FIGURE 4-16. TIME FROM POSSIBLE TO PUBLIC NOTIFICATION BY CONTAMINANT TYPE 38
FIGURE 4-17. TIME FROM POSSIBLE TO PUBLIC NOTIFICATION BY NUMBER OF COMPONENTS DETECTING 39
FIGURE 4-18. TIME FROM POSSIBLE TO PUBLIC NOTIFICATION BY NUMBER OF ALERTS RECEIVED 40
FIGURE 7-1. O&M LABOR HOURS PER REPORTING PERIOD 51
10
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
List of Tables
TABLE 2-1. CONSEQUENCE MANAGEMENT DESIGN ELEMENTS 4
TABLE 2-2. CM ROLES AND RESPONSIBILITIES - CINCINNATI LOCAL RESPONSE PARTNER AGENCIES 5
TABLE 2-3. CM ROLES AND RESPONSIBILITIES - UTILITY ICS PERSONNEL 6
TABLE 2-4. SIGNIFICANT CM COMPONENT MODIFICATIONS 7
TABLE 3-1. DISCUSSION-BASED EXERCISES 10
TABLE 3-2. CM OPERATIONS-BASED EXERCISES 12
TABLE 4-1. SITE CHARACTERIZATION METRIC CATEGORIES AND REPORT LOCATION 19
TABLE 4-2. EXERCISE TIMES FOR CREDIBLE DETERMINATION 21
TABLE 4-3. NUMBER OF ALERTS RECEIVED - TIMES FOR CREDIBLE DETERMINATION 24
TABLE 4-4. EXERCISE TIMES FROM CREDIBLE TO CONFIRMED DETERMINATION 26
TABLE 4-5. NUMBER OF ALERTS RECEIVED-TIMES FOR CONFIRMED DETERMINATION 29
TABLE 4-6. TIME TO NOTIFY RESPONSE PARTNERS 30
TABLE 4-7. NUMBER OF ALERTS RECEIVED - TIME TO PUBLIC HEALTH RESPONSE 33
TABLE 4-8. TIME TO IDENTIFY OPERATIONAL RESPONSE ACTIONS 34
TABLE 4-9. NUMBER OF ALERTS RECEIVED - TIME TO OPERATIONAL RESPONSE 37
TABLE 4-10. TIME TO PREPARE AND ISSUE PUBLIC NOTIFICATION 38
TABLE 4-11. NUMBER OF ALERTS RECEIVED - TIME TO PUBLIC NOTIFICATION 40
TABLE 5-1. RESPONSE PARTNER RECOMMENDATIONS BY EXERCISE 45
TABLE 5-2. RESPONSE PARTNER EXERCISE PARTICIPATION 45
TABLE 5-3. NATURE OF RESPONSE PARTNER SUPPORT 47
TABLE 7-1. COST ELEMENTS USED IN THE CALCULATION OF LIFECYCLE COST 49
TABLE 7-2. IMPLEMENTATION COSTS' 50
TABLE7-3. ANNUALO&MCOSTS 50
TABLE 7-4. EQUIPMENT COSTS 52
11
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Section 1.0: Introduction
The purpose of this document is to describe the evaluation of the consequence management (CM)
component of the Cincinnati pilot, the first such pilot deployed under the U.S. Environmental Protection
Agency's (EPA) Water Security Initiative (WSI). This evaluation was implemented by examining the
performance of the CM component relative to the design objectives established for the contamination
warning system (CWS).
1.1 CWS Design Objectives
The Cincinnati CWS was designed to meet six overarching objectives, which are described in detail in
WaterSentinel System Architecture (USEPA, 2005) and are presented briefly below:
• Spatial Coverage. The objective for spatial coverage is to monitor the entire population served
by the drinking water utility. It depends on the location and density of monitoring points in the
distribution system and the hydraulic connectivity of each monitoring location to downstream
regions and populations.
• Contaminant Coverage. The objective for contaminant coverage is to provide detection
capabilities for all EPA priority contaminants. This design objective is further defined by binning
the priority contaminants into 12 classes according to the means by which they might be detected
(USEPA, 2005). Use of these detection classes to inform design provides more comprehensive
coverage of contaminants of concern than would be achieved by designing the system around a
handful of specific contaminants. Contaminant coverage depends on the specific data streams
analyzed by each monitoring and surveillance component, as well as the specific attributes of
each component.
• Alert Occurrence. The objective of this aspect of system design is to minimize the rate of
invalid alerts (alerts unrelated to contamination or other anomalous conditions) while maintaining
the ability of the system to detect real incidents. It depends on the quality of the underlying data
as well as the event detection systems that continuously analyze that data for anomalies.
• Timeliness of Detection and Response. The objective of this aspect of system design is to
provide initial detection of a contamination incident in a timeframe that allows for the
implementation of response actions that result in significant consequences reduction. For
monitoring and surveillance components, this design objective addresses only the detection of an
anomaly and initial investigation of the subsequent alert. The CM component of the system is
evaluated with respect to timeliness of response, or the time it takes to complete investigative
actions and implement response actions once a contamination has been determined to be Possible.
• Operational Reliability. The objective for operational reliability is to achieve a sufficiently high
degree of system availability, data completeness and data accuracy such that the probability of
missing a contamination incident becomes exceedingly low. Operational reliability depends on
the redundancies built into the CWS and each of its components.
• Sustainability. The objective of this aspect of system design is to develop a CWS that provides
benefits to the utility and partner organizations while minimizing the costs. This can be achieved
through leveraging existing systems and resources that can readily be integrated into the design of
the CWS. Furthermore, a design that results in dual-use applications that benefit the utility in
day-to-day operations, while also providing the capability to detect intentional or accidental
contamination incidents, will improve Sustainability.
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
The design objectives provide a basis for evaluating each component, in this case CM, as well as the
entire integrated CWS. Because the deployment of a drinking water CWS is a new concept, design
standards or benchmarks are unavailable. Thus, it is necessary to evaluate the performance of the pilot
CWS in Cincinnati against the design objectives relative to the baseline state of the utility prior to CWS
deployment.
1.2 Role of CM in the Cincinnati CWS
CM is a key component of the Cincinnati CWS and consists of actions taken to plan for and respond to
possible contamination incidents. These actions are meant to minimize response and recovery timelines
through a pre-planned, coordinated effort. Investigative and response actions initiated upon determination
of a Possible contamination incident are used to establish credibility, minimize public health and
economic impacts and ultimately return the utility to normal operations. CM is designed to guide the
utility through this process of determining whether a Possible contamination incident is Credible and can
be Confirmed. CM also assists the utility in working with local partners, communicating with the public,
and determining appropriate response actions.
Figure 1-1 provides an overview of the surveillance and response activities which comprise the
Cincinnati CWS. The CWS is divided into surveillance components (water quality monitoring, public
health surveillance, customer complaint surveillance, and enhanced security monitoring) and response
components (CM and sampling and analysis). As illustrated, the surveillance components help to detect
water quality anomalies and potential causes, while the response components aid in the implementation of
response actions that minimize adverse impacts and ultimately assist in returning the system to normal
operation. Consequence management plans (CMP), as a part of the CM component, focus on integrating
common elements of existing utility plans, such as the emergency response and communication plans,
while also coordinating utility actions with those of its local, state and federal partners.
Monitoring and Surveillance
Response
Enhanced Security ^ ^Customer Complaint^
Monitoring JI Surveillance )
Water Quality
Monitoring
Public Health
Surveillance
Event Detection
Initial Alert
Investigation
Can
No
! Yes
V
Take corrective action if necessary;
continue surveillance
Consequence
Management
Sampling and
Analysis
/Credibility Determination Process to \ /Field Investigation to support
confirm or rule out contamination credibility determination
Check alerts from surveillance
components
Assess outside data sources
Site characterization
Field and lab analyses
Response Actions protect
public health during the
investigation process and may
include:
• Isolation
• Flushing
• Public alerts/notifications
Remediation and Recovery
restores a system to normal
operations and may include:
• System characterization
• Remedial action
• Post-remediation activities
Figure 1-1. CWS Surveillance and Response Overview
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
1.3 Objectives
The overall objective of the CM component evaluation is to demonstrate how the component functioned
as part of the Cincinnati CWS (i.e., how effectively the component achieved the two design objectives
described in Section 1.1). This evaluation will describe how well the incident response plans, response
partner network, and communication equipment functioned when responding to possible water
contamination incidents during exercises. Data gathered from planned exercises and the simulation study
yielded sufficient data for the evaluation.
1.4 Document Organization
This report will present the assessment of data collected to evaluate the performance of the CM
component for each of the CM design elements of the Cincinnati CWS. The report is organized as
follows:
• Section 2: Overview of the CM Component. This section describes the composition and
purposes of the CM component of the CWS.
• Section 3: Methodology. This section describes the various sources of information and methods
used to evaluate the CM component.
• Sections 4 through 6: Performance of the CM Design Elements. These sections each focus
on the performance of each of the individual CM component design elements evaluated as a part
of the Cincinnati pilot (incident response plans, response partner network, and communication
equipment). In each section, a description of the performance metric is presented, as well as an
evaluation of how well the overall design objectives were met.
• Section 7: Sustainability. This section presents the labor hours expended in developing and
refining the CM component used in the Cincinnati pilot, including the labor hours expended for
implementation and O&M costs of the CM component. There is also a discussion of dual-use
benefits gained by the Cincinnati pilot through implementation of the CM component during non-
contamination incidents.
• Section 8: Summary and Conclusions. This section summarizes key lessons learned from the
evaluation and includes a description of the cost-benefit of the CM component.
• Section 9: References. This section contains references for all documents cited in this report.
• Section 10: Abbreviations. This section lists all acronyms approved for use in the CM
component evaluation.
• Section 11: Glossary. This section defines terms used throughout the CM component
evaluation.
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Section 2.0: Overview of the CM Component
The following section provides an overview of the Cincinnati pilot CM component including the three
major design elements, which are summarized in Table 2-1. First, incident response plans included the
Cincinnati Pilot Consequence Management Plan, which was developed through modification and
incorporation of existing emergency response plans, partnerships and procedures to address the unique
challenges associated with response to a drinking water contamination incident. The goal of the
component design was first to identify existing Cincinnati pilot response plans, identify and remedy gaps
in response roles and responsibilities, and then develop integrated CM protocols. Incident response plans
also included supporting documents such as the Crisis Communications Plan (CCP) and the Confirmatory
Sampling Field Decision Guide (CSFDG).
Second, a network of response partner agencies was developed that would support implementation of the
guidance contained in the integrated incident response plans. Finally, communication equipment
consisting of eight hand-held 800 MHz radios was procured for the utility. These radios were to be used
by the utility during field response actions associated with contamination incident investigation. Prior to
the CWS pilot, utility field crews utilized cell phones for communication. Procurement of the 800 MHz
radios provided the field crew secure and guaranteed communication to the water utility emergency
response manager (WUERM) and local response partners such as Hazardous Materials (HazMat) team
(see Sections 2.3 and 6.0).
Table 2-1. Consequence Management Design Elements
Design Element
Incident Response
Plans
Response Partner
Network
Communication
Equipment
Description
This included the following:
• CMP to guide the utility through actions taken upon notification of a Possible
contamination incident,
• CCP to guide the utility and partners on when/how to make notifications, define
the message, work with the media and develop a delivery system, and
• CSFDG to guide operational responses, including isolation and distribution
system sampling.
This included a network of local, state and federal response partner agencies that
had various response roles during a contamination incident.
This included equipment (e.g., inter/intranet sharing sites, reverse phone information
systems, 800 MHz radios) required by the Cincinnati pilot to effectively respond to
contamination incidents.
2.1 Incident Response Plans
Three different incident response plans were developed during the course of the Cincinnati pilot study as
detailed below.
• Consequence Management Plan. The Cincinnati Pilot Consequence Management Plan was
developed to serve as a preparedness and response guide in the event of a Possible water
contamination incident. The process of determining whether an incident is Possible was
documented in the utility Cincinnati Pilot Operational Strategy. The Possible contamination
evaluation process is conducted in response to an alert from one of the monitoring and
surveillance components. Once the alert is validated through the procedures outlined in the
Cincinnati Pilot Operational Strategy, then the incident is Possible and CM begins.
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
The Cincinnati Pilot Consequence Management Plan used a series of nine separate decision trees
to guide personnel through the process of determining whether a Possible contamination incident,
as indicated by one of the CWS monitoring and surveillance components, is Credible and can be
Confirmed. The individual decision tree topics included threat level determination (e.g.,
Credible, Confirmed), site characterization (SC), operational responses, response partner and
public notification protocols, and remediation and recovery (R&R). The Cincinnati Pilot
Consequence Management Plan also contained five appendices of supporting material, including
the site characterization plan, forms for collecting and documenting the contamination incident
and operational procedure sheets for major response activities.
• Crisis Communications Plan. The CCP was developed to formalize public notification
procedures and guide the actions of the utility Public Information Officer (PIO) during all phases
of a potential contamination incident. It was designed from best practices in risk communication
and public notification to provide communication control internally during an incident and to
coordinate external public notification. The CCP was designed to complement the overall
Cincinnati Pilot Consequence Management Plan and corresponding incident response plans.
The CCP covered communication both within the utility and with external response partner
agencies, the media and the public. The plan included an overview of basic crisis communication
principles, detailed decision trees adapted for use by the PIO and a tools and resources section
that includes sample public notification templates, media resources and contact information.
• Confirmatory Sampling Field Decision Guide. The CSFDG consisted of charts and matrix
tables based on hydraulic models and tracer studies that describe the hydraulics of the utility's
distribution system in terms of predetermined pressure zones. It allowed utility personnel to
rapidly identify potential sampling sites in the system based on initial detection of contaminants.
2.2 Response Partner Network
The Cincinnati pilot established a response partner network to better integrate their roles and
responsibilities in the event of a drinking water contamination incident. This allowed utility personnel to
document these roles, responsibilities and communication requirements into the Cincinnati Pilot
Consequence Management Plan and allowed response partner agencies to recognize the first response
capabilities and duties of the utility. Inclusion of the response partners in exercises further enhanced the
response partner network (see Section 5.0). The list of Cincinnati pilot local response partner agencies
and their roles and responsibilities are shown in Table 2-2.
Local response partners are defined as those agencies residing close enough to the utility to be able to
provide immediate support during the investigation of a contamination incident. A broader population of
response partners would participate in the R&R phases of CM. These agencies and their roles are
described in more detail in Section 4.1.3 of this report.
Table 2-2. CM Roles and Responsibilities - Cincinnati Local Response Partner Agencies
Response Partner Organization
Method Labs
Cincinnati Fire Dept (CFD)
Metropolitan Sewer District of Greater
Cincinnati (MSDGC)
Roles & Responsibilities
Analyzes triggered samples and interacts directly with the Cincinnati
pilot WUERM for field results, event status, and reporting.
Supports site characterization activities when requested by providing
field response and HazMat support.
Provides laboratory support and input on disposal options.
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Response Partner Organization
Cincinnati Health Department (CHD),
Hamilton County Public Health (HCPH)
Drug and Poison Information Center (DPIC)
(added during evaluation period)
Hamilton County Emergency Management
Agency (HCEMA)
Ohio Department of Health (ODH)
Cincinnati Police Department (CPD) and
Ohio State Police
Ohio EPA (OEPA)
(State Drinking Water Primacy Agency)
Roles & Responsibilities
Provides local public health data, epidemiologists and disease
investigators.
Provides technical support on contaminant symptomology and
demographics during a contamination incident.
Coordinates alternate water supplies and implements Unified
Command System for expanded contamination incidents.
Provides regulatory and sampling support, investigations, and threat
assessments of Possible or Confirmed contamination incidents.
Provides support by coordinating investigation and isolation issues
with Cincinnati pilot personnel, and transports samples to ODH along
with the associated chain-of-custody.
Provides support for regulatory issues, threat assessment, risk
communication, and R&R issues.
2.3 Communication Equipment
This design element of the CM component identifies equipment required by Cincinnati pilot personnel to
effectively respond to possible drinking water contamination incidents. It was determined that the prior
approach of using cell phone communication for responding to normal water quality issues was not
adequate for handling the expanded scope of responding to a possible water contamination incident (e.g.,
site characterization activities, ICS communications). This being the case, eight Motorola XTS 5000 800
MHz radios were procured and deployed to be used during field investigation activities. The radios are
multi-channel programmable hand-held units that allow for communication between Cincinnati pilot
personnel (e.g., field personnel, team leaders, response partners). During the evaluation period, first-
responders' existing 800 MHz radio communication network was programmed into the utility 800 MHz
radios.
2.4 CM Component Roles and Responsibilities
Similar to the local response partner agencies outlined in Table 2-2, Table 2-3 summarizes the general
roles and responsibilities of major utility Incident Command System (ICS) personnel in implementing the
CM component.
Table 2-3. CM Roles and Responsibilities - Utility ICS Personnel
Personnel
Director
WUERM
Plant Supervisor
Security
Roles & Responsibilities
Manages the implementation of the CMP and the management
structure of the ICS or elements of both depending on the nature of
the contamination incident. If the ICS is activated, the Director may
assume the role of Incident Commander (1C) from the WUERM as
appropriate.
Assumes the role of 1C until relieved of these duties by the Director. If
not acting as 1C, the WUERM works directly with the Site
Characterization Team (SCT) and will also support the Director and
response activities as outlined in the CMP.
Reviews recent operational and treatment changes.
Utility security and local law enforcement conduct security surveys of
system facilities during investigations and provide security support for
field investigation personnel. Acts as the liaison for external response
partners.
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Personnel
Planning Section Chief
Operations Section Chief
PIO
Customer/ Public Information Manager
Liaison Officer
Technical Analyst
Roles & Responsibilities
Maintains the incident log, planning documents, and data
management functions during an investigation.
Assists with operational response actions such as isolating the
system.
Coordinates directly with other agency PIOs and the public.
Implements the CCP to help prepare and coordinate the release of
internal and public notifications.
Shares public information responsibilities with the PIO.
Contacts and works with external response partners as necessary.
Communicates with regulators and the scientific community
concerning water quality issues. Manages water quality analysis.
2.5 Summary of Significant CM Component Modifications
The CM component modifications discussed in Table 2-4 were implemented during the evaluation period
to improve performance. These modifications serve as a reference when discussing the results of the
evaluation (Sections 4 through 7).
Table 2-4. Significant CM Component Modifications
ID
Component Modification
Date
Incident Response Plan Modifications
1.1
1.2
1.3
Modification
Cause
Modification
Cause
Modification
The CSFDG was updated with modifications to the contaminant
transport worksheets and the addition of a section outlining the
procedures for contaminant source determination.
Updates to the CSFDG, dated June 15, 2008, were the direct
result of a drill conducted on April 15, 2008 to evaluate the
procedures in the guide, to collect baseline data on the elapsed
times required to implement them, and to provide Greater
Cincinnati Water Works (GCWW) personnel the opportunity to
practice the protocols and procedures outlined in the guide.
The draft-final version of the CMP, dated June 12, 2007, was
updated with modifications to roles and responsibilities of
Cincinnati pilot personnel as well as response partners,
including how the ICS operates within the utility, how the SCT
interacts with local law enforcement and HazMat, and how
operational responses occur during an incident (e.g., isolation
responses).
Updates to the CMP, dated September 1, 2008, were the direct
result of exercises conducted in 2007 and early 2008, including
two site characterization drills, a sampling and analysis drill, a
functional exercise, and a full scale exercise identifying that
roles and responsibilities required further clarification.
The draft-final version of the CMP, dated September 1, 2008,
was further updated, with the majority of changes pertaining to
modifying and adding new roles and responsibilities of utility
ICS personnel as well as response partners.
6/15/2008
9/1/2008
2/10/2009
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
ID
Component Modification
Date
Cause
Updates to the CMP, dated February 10, 2009, were the direct
result of the full scale exercise conducted October 1, 2008, to
evaluate the CMP, including site characterization, threat level
determinations, operational responses, public notification, and
response partner roles. Changes included adding new and
modifying existing roles and responsibilities.
Modification
The draft-final version of the CMP, dated February 10, 2009,
was updated with the majority of modifications specifically
pertaining to updating ICS roles and responsibilities and
modifying the decisions trees/process flows for site
characterization and R&R.
1.4
Cause
Updates to the CMP, dated June 5, 2009, were based on the
outcomes of several drills conducted in early 2009 identifying
the need to modify existing roles and responsibilities, decision
trees for site characterization and R&R activities. These drills
included:
• Customer Complaint Surveillance(CCS)/Site
Characterization workshop (April 2, 2009): objective was
to discuss site characterization protocols for a Possible
CCS alert including where to sample, who samples, and
the sampling process.
• SC drill (April 23, 2009): objective was to evaluate the
implementation of the revised site characterization
procedures outlined in the CMP.
• R&R workshop (May 14, 2009): objective was to discuss
the R&R decision tree and corresponding process flow.
6/5/2009
Modification
1.5
The draft-final version of the CCP, dated March 8, 2007, was
updated to be consistent with the updated version of the CMP
dated June 5, 2009. The majority of the changes specifically
pertained to modifying and adding new roles and
responsibilities, updating the decision tree figures and
reformatting the threat level phases (e.g., Credible, Confirmed).
Cause
Inconsistencies were noted between the CCP and CMP
pertaining to the roles and responsibilities of the PIO and the
Customer/Public Information Manager during the various
phases of a contamination threat and the public notification
decision tree process.
6/5/2009
Modification
1.6
The draft-final version of the CMP, dated June 5, 2009, was
further updated, with the majority of changes pertaining to
changing the term 'Confirmed' to 'Assumed/Determined
Contamination and updating ICS roles and responsibilities, as
appropriate.
Cause
Updates to the CMP, dated September 23, 2009, were based
on the outcomes of a threat level determination (TLD)
workshop, held July 1, 2009, which provided ICS personnel
with guidance on and practice for determining threat levels
(e.g., Possible, Credible and Confirmed determination) and
associated response actions for specific CWS alert scenarios.
9/23/2009
Modification
1.7
The draft-final version of the CCP, dated June 5, 2009, was
updated to be consistent with the current version of the CMP
dated September 23, 2009. The majority of the changes
pertained to modifying the threat level phases, modifying ICS
titles and roles, and updating the decision tree figures and
checklist, as appropriate.
Cause
Inconsistencies were noted between the CCP and CMP
pertaining to the threat level determination terminology used
and roles and responsibilities of the PIO and the
Customer/Public Information Manager.
9/28/2009
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
ID
Component Modification
Date
Response Partner Network Modifications
2.1
Modification
Cause
The response partner network was updated to include DPIC.
Updates to the response partner network, dated September 1,
2008, including the addition of DPIC to the response partner
network, was the direct result of exercises conducted in 2007
and early 2008, where the Cincinnati pilot recognized the
resourcefulness of DPIC in providing information on potential
poisoning inquiries that could be correlated to drinking water
contamination.
9/1/2008
Communication Equipment Modifications
3.1
Modification
Cause
The CFD programmed the Cincinnati pilot 800 MHz radios with
a communication channel that permitted network
communications with the Cincinnati emergency response
channel.
Direct communication between utility field response personnel
and CFD HazMat personnel was not possible prior to the
reprogramming. The deficiency was identified during FSE 2.
10/5/2008
2.6 Timeline of CM Development Phases and Evaluation-Related Activities
Figure 2-1 presents a summary timeline for deployment of the CM component, including milestone dates
indicating when significant component modifications and exercise evaluation activities took place. The
timeline also shows the completion date for design and implementation of the CM component including
the first completed draft of the Cincinnati Pilot Consequence Management Plan (January 2007), along
with the subsequent optimization and real-time monitoring phases of deployment.
Sep-07
SC Drill 1
Jan-07
Jul-08
SC Drill 2
Apr-09 May-09 Sep-09
SC Drill 3 R&R Workshop SC/CCS Drill
Jan-08
Jun-10
Jan-09
Jan-10
y y
J A^ A _Jl
Functional FSE 1 Wind Storm FSE 2
Exercise Sep-07 Sep-08 Oct-08
Jul-07 v
^A
Optimization
Jan-07 - Jan-09
? ^ i
TLD Workshop FSE 3
Jul-09 Oct-09
*v J
V
CMP Completed
Jan-07
Figure 2-1. Timeline of CM Component Activities
Real-time Monitoring
Jan-09 -Jun-10
End of Data
Collection
Jun-10
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Section 3.0: Methodology
The following section describes the evaluation techniques that were used to fully evaluate the CM
component metrics, which are further described in Section 4.0. Specific metrics are described for each
element of the CM component as they relate to the design objectives. The CM evaluation was conducted
using results from four evaluation techniques: training and exercise program, the simulation study,
feedback forums and analysis of life cycle costs.
3.1 Training and Exercise Program
The CM training and exercise program consisted of exercises modeled after the Department of Homeland
Security Exercise and Evaluation Program. These exercises included discussion-based exercises
consisting of seminars, workshops and tabletop exercises, and operations-based exercises that included
drills, functional exercises, and full scale exercises. After the exercises were completed, After Action
Reports (AARs) and a corresponding Improvement Plan were developed to capture comments from
participants and evaluators regarding suggested modifications to the Cincinnati Pilot Consequence
Management Plan and to outline areas for improvement of future exercises.
In general, discussion-based exercises are used to develop, refine, and train on plans, while operations-
based exercises are used to test and evaluate those plans. All of the training and exercise activities
associated with the CM component are described in this report for completeness, but most of the metrics
used for performance evaluation were extracted from operations-based exercises, particularly the full
scale exercises.
Twenty-five CM discussion-based and operations-based exercises were conducted with the Cincinnati
pilot since the inception of the pilot in 2005. The training and exercises were important in identifying
opportunities for improving the plans, evaluating participants' ability to implement the guidance of the
Cincinnati Pilot Consequence Management Plan, and optimizing response time and accuracy.
3.1.1 Discussion-Based Exercises
The CM component discussion-based exercises are described in Table 3-1. These discussion-based
exercises were essential for staff to understand and become familiar with their Cincinnati Pilot
Consequence Management Plan roles/responsibilities, which prepared them to perform well in the
operations-based exercises that followed.
Table 3-1. Discussion-Based Exercises
Training and
Exercise Title
Date(s)
Conducted
Participants
Description
CM Workshop 1
12/13/2005
Cincinnati Pilot
Discussed with GCWW key officials, CFD,and
CPD initial stages of CMP development including
decision trees and response partner
involvement. CFD provided introductory
National Incident Management System (NIMS)
briefing for utility staff.
CM Workshop 2
01/19/2006
Cincinnati Pilot
Collected data from City and County Public
Health Agencies, CFD and CPD and reviewed
draft CMP decision trees for field operation
information that would occur with Possible,
Credible and Confirmed determination decision
trees.
10
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Training and
Exercise Title
CM Workshop 3
CM Workshop 4
DHS FEMA/NIMS
IS100&IS700
Training Seminars*
CMP Orientation
Seminars
Tabletop Exercises:
Supervisor Training
Roll Out
Documentation &
Presentation
SCT Training
Seminars
CWS Management
Orientation Seminar
CCS / SCT
Workshop
R&R Workshop
Date(s)
Conducted
02/14-
02/15/2006
3/22/06
5/16-
05/18/2006
08/22 -
08/24/2006
08/29 -
08/31/2006
(4 separate
sessions)
06/27/2006
&
09/28/2006
05/11/2007
(4 separate
sessions)
06/15/2007
04/2/2009
05/14/2009
Participants
Cincinnati pilot
Cincinnati pilot
GCWW
GCWW
GCWW
Potential partners
from outside of the
utility response area
GCWW
GCWW
GCWW
GCWW
Description
Provided information to GCWW Key Officials,
PIOs from various City & County Response
Agencies, City and County Public Health
Agencies, CFD and CPD on risk communication
and message mapping for communicating with
the public during a possible contamination
incident. Developed specific message maps for
the CCP.
Collected and reviewed notification and
communication data focusing on Credible and
Confirmed phases of an incident where multiple
response agencies will be involved including City
and County Public Health Agencies, CFD, CPD,
and city, state and federal government officials.
This involved identifying key connections,
linkages and dependencies between the draft
CMP and response partner agency plans and
procedures. Collected the same data for sub-
flow decision trees (e.g., site characterization),
which was utilized in developing draft CMP.
Provided participants, including senior
management, with a basic understanding of ICS
procedures and an introduction to NIMS. The
material was formatted for use in future utility
instruction.
Provided participants, including senior
management, with an understanding of their
roles in the CMP. The material was formatted
for use in future utility instruction.
Provided participants, including senior
management and supervisors, with scenarios to
improve their knowledge of the CMP and
incident management processes.
Discussed the CM preparedness and response
process with external agencies and
organizations that might not otherwise be readily
engaged in the utility response network.
Familiarized staff with field operations associated
with site investigations and corresponding
equipment.
Provided participants, including senior
management, an overview of the CWS and CMP
including contamination scenario exercises to
familiarize attendees.
Presented scenarios to discuss and modify CMP
site characterization roles and responsibilities
following Possible determination of a CCS
alert(s). For each scenario, discussions were
based on where to sample, who will sample, and
the sample collection process.
Reviewed and discussed potential modifications
to the CMP R&R process. CMP R&R decision
tree was revised based on these discussions.
11
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Training and
Exercise Title
TLD Workshop
CM FSE 3 Follow-up
Workshop
Date(s)
Conducted
07/01/2009
05/21/2010
Participants
GCWW
Cincinnati pilot
Description
Provided participants, including senior
management, guidance and practice on
determining threat levels (e.g., Possible,
Credible and Confirmed determination) and
associated response actions for specific CWS
alert scenarios. CMP threat level terminology
was modified as a result.
Provided the utility and its response partner
agencies an opportunity to discuss the actions
following contamination of the drinking water
distribution system. This included a facilitated
discussion and corresponding changes to the
R&R section of the CMP.
Note:
*GCWW personnel have also taken other relevant ICS training which was not a formal part of the WSI pilot
Threat Level Determination Workshop
The TLD workshop, listed in Table 3-1, was of particular importance to the evaluation of the CM
component. During this workshop, utility Steering Committee members and WUERMs examined and
evaluated how the utility processed various kinds of contamination incident information to arrive at the
threat levels and response actions described in the Cincinnati Pilot Consequence Management Plan.
Six CWS alert scenarios were presented to participants in the form of incident response timelines.
Participants were asked to review each of the scenarios and identify the time or step at which Possible,
Credible and Confirmed threat levels would have occurred. They were also asked to indicate when any of
the following response actions would have occurred during each incident:
• Operational changes,
• Notification of public health partners,
• Public notification of use restrictions,
• Site characterization and sample collection, and
• Laboratory analysis of samples.
3.1.2 Operations-Based Exercises
The operations-based exercises conducted for the Cincinnati Pilot Consequence Management Plan are
described in Table 3-2. These exercises progressed from focused drills to larger functional and full scale
exercises with response partners.
Table 3-2. CM Operations-Based Exercises
Training and
Exercise Title
CM Functional
Exercise
SCT Drill 1
Date(s)
Conducted
07/31/2007
09/05 -
09/06/2007
Participants
Cincinnati pilot
GCWW
Description
Provided the utility and response partners the
opportunity to practice their roles and
responsibilities during a response to a possible
drinking water contamination incident, identify
potential revisions and corrections to the CMP,
and practice plans and procedures of various
agencies.
Provided the SCT the opportunity to practice
implementation of site characterization
procedures and equipment.
12
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Training and
Exercise Title
Date(s)
Conducted
Participants
Description
FSE 1
09/25 -
09/28/2007
Cincinnati pilot
Provided the utility and response partners the
opportunity to exercise their roles within a field
environment, test plans and procedures, and
identify opportunities for improvement and
potential revisions to the CMP. Participants also
practiced communication and coordination
techniques.
SCT Drill 2
07/15/2008
Cincinnati pilot
Provided the SCT and CFD HazMat an
opportunity to cross-train on site characterization
procedures.
FSE 2
10/1 -
10/2/2008
Cincinnati pilot
Provided a scenario-driven, real-time simulation
that implemented utility and local response
partner agency protocols related to detection of
and response to a drinking water contamination
incident.
SCT Drill 3
04/23/2009
GCWW
Conducted to evaluate the implementation of the
revised site characterization procedures outlined
in the CMP and the Standard Operating
Procedures for site characterization and
sampling.
CCS / SCT Drill
09/16/2009
GCWW
Evaluated the alert recognition and investigative
procedures associated with the CCS component
and to evaluate implementation of the site
characterization procedures as they relate to
field deployment and investigation following a
CCS alert.
FSE 3
10/21/2009
Cincinnati pilot
Provide a scenario-driven, real-time simulation
that implemented utility and local response
partner agency protocols related to detection of
and response to a drinking water contamination
incident.
FSE 3 also involved ICS implementation (for
utility second-in-commands), external
notifications resource coordination, media
relations and the execution of field investigation
procedures.
Much of the data used in this report to evaluate CM response performance by the Cincinnati pilot was
drawn from a series of three FSEs. These exercises were designed with different contamination incident
scenarios, specifically developed to test and evaluate different detection and response procedures of the
Cincinnati Pilot Operational Strategy and Cincinnati Pilot Consequence Management Plan. Thus,
response times used in the performance evaluation were influenced by the specific situations presented by
the scenarios. Additional details and distinguishing characteristics for each of the FSEs listed in Table 3-
2 is provided below.
Full Scale Exercise 1 (FSE 1)
FSE 1 was conducted for incident management and field response personnel, operating simultaneously
but independently using a common scenario, with communication between the two groups coordinated by
exercise staff in a simulation cell. The exercise was conducted in this manner in order to complete the
entire exercise within the allotted eight-hour period.
The scenario was initiated by a simulated total organic carbon (TOC) alert from a water quality
monitoring (WQM) station located at a GCWW facility. This alert was followed by a CCS alert
concerning foul smelling drinking water. Site characterization personnel were dispatched to the incident
13
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
site and discovered empty drums, along with transfer pumps and tubing that had been staged at the site to
simulate intrusion.
Local HazMat and law enforcement personnel participated in the exercise on-site, but external response
partners participated remotely from their normal work locations.
As part of all the full scale exercises, a "Hot Wash" was conducted at the conclusion of each exercise.
The Hot Wash, conducted by the exercise controllers, consisted of a presentation of the events followed
by a group discussion about the events and actions. The information collected during the Hot Wash was
incorporated into the AARs and Improvement Plan development.
Full Scale Exercise 2 (FSE 2)
FSE 2 was a conventional FSE in that the Cincinnati pilot participated without the use of a simulation
cell. The exercise was initiated with a simulated alert from a WQM station located at a GCWW facility,
and featured additional alerts from the CCS system (both interactive voice response and work order),
WQM alerts from additional locations, and ended with a public health surveillance (PHS) alert stemming
from illnesses in the affected neighborhoods. The entire exercise scenario was based on a biological
agent being intentionally introduced into the system through a pump station. One of the exercise goals
was to have the exercise be as realistic as possible which was accomplished by using the utility's
hydraulic model in developing the scenario.
Full Scale Exercise 3 (FSE 3)
FSE 3 again featured the Cincinnati pilot participating on-site in real time, but the initiation of the
exercise was timed to involve alternate ICS and back-shift personnel in the response. This exercise
scenario included recurring CCS and PHS alerts, and was designed to simulate the injection of atoxic
chemical into the system through a fire hydrant. Field sampling from a neighborhood fire hydrant and
consideration of residential sampling were among the significant objectives of this exercise.
3.2 Simulation Study
Evaluation of certain design objectives relies on the occurrence of contamination incidents with known
and varied characteristics. Because contamination incidents are extremely rare, there is insufficient
empirical data to fully evaluate the detection capabilities of the Cincinnati CWS. To fill this gap, a
computer model of the Cincinnati CWS was developed and challenged with a large ensemble of
simulated contamination incidents in a simulation study. For the CM component, simulation study data
was used to evaluate the following design objective:
• Timeliness of Response: Analyses conducted to evaluate this design objective quantify the
number of scenarios that reached Possible, Credible and Confirmed contamination and the
number of scenarios that resulted in operational response actions, public health response, and
public notification. Statistical analysis was used to characterize the time that these three threat
levels were reached and the time that response actions were implemented.
A broad range of contaminant types, producing a range of symptoms, was selected for the simulation
study in order to characterize the detection capabilities of the monitoring and surveillance components of
a CWS. For the purpose of the simulation study, a representative set of 17 contaminants was selected
from the comprehensive contaminant list that formed the basis for CWS design. These contaminants are
grouped into the following broad categories (the number in parentheses indicates the number of
contaminants from that category that were simulated during the study):
14
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
• Nuisance Chemicals (2): these chemical contaminants have a relatively low toxicity and thus
generally do not pose an immediate threat to public health. However, contamination with these
chemicals can make the drinking water supply unusable.
• Toxic Chemicals (8): these chemicals are highly toxic and pose an acute risk to public health at
relatively low concentrations.
• Biological Agents (7): these contaminants of biological origin include pathogens and toxins that
pose a risk to public health at relatively low concentrations.
Development of a detailed CWS model required extensive data collection and documentation of
assumptions regarding component and system operations including the CWS monitoring and surveillance
components (e.g., Enhanced Security Monitoring (ESM), WQM, CCS, PHS) and CM. To the extent
possible, model decision logic and parameter values were developed from data generated through
operation of the Cincinnati CWS, although input from subject matter experts and available research was
utilized as well.
The simulation study used several interrelated models, three of which are relevant to the evaluation of
CM: EPANET, Health Impacts and Human Behavior (HI/HB) and the CM model. Each model is further
broken down into modules that simulate a particular process or attribute of the model. The function of
each of these models and its relevance to the evaluation of CM is discussed below.
EPANET
EPANET is a common hydraulic and water quality modeling application widely used in the water
industry to simulate contaminant transport through a drinking water distribution system. In the simulation
study, it was used to produce contaminant concentration profiles at every node in the GCWW distribution
system model, based on the characteristics of each contamination scenario in the ensemble. The
concentration profiles were used to determine the number of miles of pipe contaminated during each
scenario, which is one measure of the consequences of the contamination scenario. EPANET was also
used to model the efficacy of operational response actions predicted by the CM model in reducing the
miles of pipe contaminated.
Health Impacts and Human Behavior Model
The HI/HB model used the concentration profiles generated by EPANET to simulate the health effects
exposure of customers in the GCWW service area to contaminated drinking water. The model analyzed
the exposed customers in three age groups including infants (five years old or younger), children (ages
five to 18) and adults (ages 18 and older). Depending on the type of contaminant, exposures occurred
during one showering event in the morning (for the inhalation exposure route) or during five consumption
events spread throughout the day (for the ingestion exposure route). The HI/HB model used the dose
received during exposure events to predict infections, onset of symptoms, health-seeking behaviors of
symptomatic customers and fatalities.
The primary output from the HI/HB model was a case table of affected customers, which captured the
time at which each transitioned to mild, moderate and severe symptom categories. Additionally, the
HI/HB model output the times exposed individuals would pursue various health-seeking behaviors, such
as visiting their doctor or calling the poison control center. The case table was used to determine the
public health consequences of each scenario, specifically the total number of illnesses and fatalities.
Furthermore, EPANET and the HI/HB model were run twice for each scenario; once without the CWS in
operation and once with the CWS in operation. The paired results from these runs were used to calculate
the reduction in consequences due to CWS operations for each simulated contamination scenario.
15
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Consequence Management Model
The CM model simulates the complex processes of determining the threat level of a Possible
contamination incident detected through one of the monitoring and surveillance components and
determining what response actions would be taken by the utility, law enforcement and the public health
community. The CM model relies on several interdependent modules to determine the threat level and
response actions.
The threat level determination module combines the functions of the WUERM and the Incident
Commander and uses the collective data generated by the monitoring and surveillance component models,
as well as the results from other investigation activities, to establish whether contamination is Possible,
Credible, or Confirmed. Because the module uses data from the component models in an integrated
fashion, a Credible contamination can be determined sooner than if one component was deployed in
isolation, modeling an important feature of the real CWS. Furthermore, the threat level determination
module incorporates site characterization and laboratory analysis activities, which produce results that are
used in the threat level determination process.
Other modules of the CM model include decision logic that governs the implementation of response
actions, such as isolation and public notification, based on the outputs of the threat level determination
module. These modules output the time, location and type of response actions, which are ultimately used
to revise parameters within EPANET and the HI/HB model to determine the consequences with a CWS in
operation.
The following assumptions used in the design of the CM model are important to consider when
evaluating the simulation study results presented in this report:
• Only scenarios that reached the Possible stage were considered in the analysis; without the
determination of Possible, the CM component is not activated and cannot be evaluated.
• Two contaminant injection location types were assumed: facility attack nodes and distribution
attack nodes. Scenarios with these contaminant injection locations were analyzed separately
because of the significant differences in timeline metrics between them.
• The simulation study results included in the CM evaluation were limited to the 1,545 scenarios
that originated at distribution attack nodes and that reached Possible contamination determination.
• Each component except CCS can produce sufficient data to independently establish that
contamination is Credible.
• The ESM component is sufficient to confirm drinking water contamination without any additional
data. This occurs when responders investigating an ESM alert observe the contaminant injection
in process.
• CCS, PHS and WQM must be supported by at least one other component to confirm a
contamination incident.
• Operational responses are limited to isolation. Flushing potentially contaminated water from the
distribution system was not considered based on observations during drills and exercises.
• Issuance of public notification occurs when the public notice is completed (e.g., 120 minutes after
contamination is deemed Possible) AND when any one of the following conditions are met:
o Threat level is Confirmed.
o Threat level is Credible and public health investigators conclude that illness in the
community may be linked to drinking water.
16
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
o Threat level is Credible and WQM alerts are due to low chlorine residual, representing a
potential risk to public health that warrants public notification, regardless of the cause of
the alert.
o Threat level is Credible due to an ESM alert (indicating tampering at a utility facility) and
another component has generated a valid alert (indicating that potentially contaminated
water has not been isolated to the utility facility).
• All public notifications are issued for the entire GCWW service area.
3.3 Feedback Forums
Two lessons learned workshops and a CM information transfer meeting were held to provide qualitative
feedback regarding the CM component deployed in Cincinnati.
• Lessons Learned Workshop, June 16, 2008 - was limited to eight EPA and contractor support
personnel responsible for the design and implementation of the CM component. The objective of the
workshop was to revisit key decisions made during the process and solicit specific feedback on the
successes and challenges encountered.
• CM Information Transfer Meeting, May 28, 2009 - included 17 participants from EPA, the
Cincinnati pilot and contract support personnel. The purpose of the meeting was to provide the
Cincinnati pilot with information necessary for maintaining the CM component.
• Lessons Learned Workshop, August 11, 2009 - included 26 participants from EPA, the Cincinnati
pilot, and contract support personnel involved in the design and implementation of the CM
component. The objective of the workshop was to elicit specific lessons learned from the pilot utility
and to gather feedback concerning how lessons learned may be shared with the drinking water sector.
3.4 Analysis of Lifecycle Costs
A systematic process was used to evaluate the overall cost of the CM component over the 20 year
lifecycle of the Cincinnati CWS. The analysis includes implementation costs, annual O&M costs,
renewal and replacement costs, and the salvage value of major pieces of equipment at the end of the
lifecycle. Data from CM operations was collected monthly from January 16, 2008 through June 15, 2010.
Data was tracked by reporting periods, and in this evaluation, the term 'reporting period' is used to refer
to a month of metrics data that spans from the 16th of one month to the 15th of the next. Thus, the
January 2008 reporting period refers to the data collected between January 16th 2008 and February 15th
2008.
Implementation costs include labor and other expenditures (equipment, supplies and purchased services)
for deploying the CM component. Implementation costs were summarized in Water Security Initiative:
Cincinnati Pilot Post-Implementation System Status (USEPA, 2008), which was used as a primary data
source for this analysis. In that report, overarching project management costs incurred during the
implementation process were captured as a separate line item. However, in this analysis, the project
management costs were equally distributed among the six components of the CWS, and are presented as a
separate line item for each component.
It should be noted that implementation costs for the Cincinnati CWS may be higher than those for other
utilities given that this project was the first comprehensive, large-scale CWS of its kind and had no
experience base to draw from. Costs that would not likely apply to any future utility project
implementation, but which were incurred for the Cincinnati CWS, include overhead for EPA and its
contractors, cost associated with deploying alternative designs and additional data collection and
17
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
reporting requirements. Other utilities planning for a similar large-scale CWS installation would have the
benefit of lessons learned and an experience base developed through implementation of the Cincinnati
CWS.
Annual O&M costs include labor and any other expenditure necessary to operate and maintain the
component. O&M labor costs were estimated from the time spent planning and participating in drills and
exercises as well as time spent in training. To account for the maintenance of documents, the costs
incurred to update documented procedures following drills and exercises conducted during the evaluation
phase of the pilot were used to estimate the annualized cost. The total O&M costs were annualized by
calculating the sum of labor and other expenditures incurred over the course of a year.
Labor hours for both implementation and O&M were tracked over the entire evaluation period. Labor
hours were converted to dollars using estimated local labor rates for the different institutions involved in
the implementation or O&M of the CM component.
The renewal and replacement costs are based on the cost of replacing major pieces of equipment at the
end of their useful life. The useful life of CM equipment was estimated using field experience,
manufacturer-provided data and input from subject matter experts. Equipment was assumed to be
replaced at the end of its useful life over the 20 year lifecycle of the Cincinnati CWS. The salvage value
is based on the estimated value of each major piece of equipment at the end of the lifecycle. The salvage
value was estimated for all equipment with an initial value greater than ~$ 1,000. Straight line
depreciation was used to estimate the salvage value for all major pieces of CM equipment based on the
lifespan of each item.
All of the cost parameters described above (implementation costs, O&M costs, renewal and replacement
costs, and salvage value) were used to calculate the total lifecycle cost for the CM component, as
presented in Section 7.1.
18
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Section 4.0: Performance of Incident Response Plans
The metric data used to measure the performance of the incident response plans design element was
gathered from the FSEs and the simulation study model. These performance metrics consisted of the
elapsed times it took the Cincinnati pilot to complete specified tasks within the following CM categories:
• Credibility Determination Actions: including time to determine threat levels.
• Response Actions: including time to notify response partners, to decide on appropriate
operational responses, and to make public notification of water use restrictions.
• R&R Actions: including time to restore the system to normal operations. Since none of the
FSEs included R&R objectives, the metrics used to evaluate this aspect of the CM component
consisted of the number and nature of the changes to the R&R section of the Cincinnati Pilot
Consequence Management Plan that resulted from several R&R workshops.
Additionally, the criteria that the Cincinnati pilot used to arrive at TLD decisions are also discussed. Note
that other CM-related metrics dealing with site characterization activities, such as time for assessing site
hazard levels, time for deploying field personnel and equipment, and time for collecting and screening
drinking water samples are discussed in Water Security Initiative: Evaluation of the Sampling and
Analysis Component of the Cincinnati Contamination Warning System Pilot (USEPA, 2013a). Table 4-1
identifies site characterization metric categories and the location of their performance evaluation.
Table 4-1. Site Characterization Metric Categories and Report Location
Site Characterization Response Activity
Time to notify response partner(s)
Time to deploy field personnel and equipment
Time to assess hazard level:
Approach site
Conduct safety screening
Time to collect and screen drinking water samples for:
Sample collection
Rapid Field Test (RFT)
Lab analysis
Data review
CM Evaluation Report
X
Sampling & Analysis
Evaluation Report
X
X
X
4.1 Design Objective: Timeliness of Response
Within the Timeliness of Response design objective, three performance criteria were identified to
evaluate the effectiveness of the incident response plans. These performance criteria included: 1) the time
to perform investigative action (i.e., Credibility determination), 2) the time to implement response actions,
and 3) the effectiveness in developing and refining the R&R process of the CM component. These
criteria are further defined in Sections 4.1.1 through 4.1.3.
Figure 4-1 demonstrates a generic incident response timeline of activities to investigate a contamination
incident that generates multiple CWS alerts. It is included here to help visualize how the CM component
contributes to this design objective. Since response actions to a contamination incident are driven by the
specific characteristics of the incident itself (e.g., contaminant, means of introduction, spatial and
volumetric scope of the contamination area, hydraulic characteristics of the system), direct comparison of
the elapsed times to complete the stages of the response timeline from one incident to another (or from
one exercise to another) is not quantitatively meaningful. Rather, the performance of this design
19
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
objective is represented by observed times for the implementation of Cincinnati Pilot Consequence
Management Plan investigative and response actions, accompanied by a qualitative discussion of the
factors used by the Cincinnati pilot in CM-related decisions.
BEGIN
Initial Component Alert(s)
Received
Additional Alert(s) Received
Additional Alert(s) Received
POSSIBLE DETERMINATION PHASE
Operational Strategy Checks
Operational Responses
CREDIBLE DETERMINATION PHASE
Response Partners Notified
SCT Deployed
SCT Positive Result Received
HazMat Activated
Sample Sentto Contract Laboratory
CONFIRMED DETERMINATION PHASE
Public Notification Issued
Sample Arrives at Laboratory
Sample Positive for "X"
CONFIRMED CONTAMINATION
BEGIN R&R
END
Figure 4-1. Generic CM Timeline
4.1.1 Credibility Determination
The amount of time required for investigative actions leading to both Credible and Confirmed (including
"Assumed Contamination") determination of a Possible contamination incident were analyzed using both
the exercise data and the simulation study data. Details of these two analyses follow.
Credible Determination
Definition: The Credible determination phase of the incident timeline begins once the initial component
alert investigation determines that contamination is Possible. It ends when evidence from follow-on
investigations either corroborates or refutes the initial alert, escalating the contamination threat to
Credible or closing the investigation. This phase includes the time required to perform multi-component
investigations and data integration, implement field investigations (such as site characterization), and
20
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
collect additional information to support the investigation.
Exercise Analysis Methodology: The analysis for the Credible determination phase of an incident
consisted of the elapsed time it took the utility (starting from Possible determination) to investigate the
available information and determine the incident as Credible during the FSEs. This information was
tabulated and examined for correlations between the number and/or type of alerts received, the
investigative information available to the utility and the amount of time required for Credible
determination.
Exercise Results: The FSE results for Credible determination are shown in Table 4-2 and Figure 4-2.
There is no statistical trend in the times to reach Credible determination among the three FSEs, which is
expected because times are dependent upon many variables, including the number of component alerts,
field investigation evidence and the fact that different participants were responsible for making the
determination. For each FSE, information from multiple CWS component alerts was required for a
Credible determination to be made. Other information that led to Credible determination included receipt
of multiple alerts from a single component with corroborating information. For example, in FSE 2 water
quality monitoring alerts were received from stations that were hydraulically connected, and in FSE 1 and
FSE 3, receipt of field investigation evidence was collected by the SCT. Finally, information that
indicated suspicious or intrusive activity led to quicker elevation of the incident to the Credible phase.
FSE1
FSE 2
FSES
50
100
150 200
Time (minutes)
250
300
350
Figure 4-2. Comparison of FSE Time from Possible to Credible Determination
Table 4-2. Exercise Times for Credible Determination
Exercise
Number of Alerts
Received
Time to Credible
Determination (minutes)
Available Information
FSE 1
300
Verified Possible determination
1 WQM alert: TOC
1 CCS alert: Odor
Suspicious activity: Leaky chemical
drums onsite
Normal RFTs
21
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Exercise
FSE2
FSE3
Number of Alerts
Received
6
3
Time to Credible
Determination (minutes)
79
191
Available Information
Verified Possible determination
5 WQM alerts: TOC, chlorine residual
1 CCS alert: Impacted neighborhood
Normal RFTs
Verified Possible determination
3 CCS alerts: Taste, same area
6 additional call complaints: Taste,
same area
• Positive RFT: Toxic chemical
Simulation Study Analysis Methodology: Of the 1,545 distribution attack scenarios that reached
Possible, 1,315 or 85% of them reached Credible. The analysis for Credible determination utilizing the
simulation study results consisted of performing separate statistical analyses of the data set according to
three scenario characteristics that had the most impact on the Credible determination process. The
scenario characteristics include contaminant type number of components detecting, and number of alerts.
Simulation Study Results by Contaminant Type: The percentile distribution for time from Possible to
Credible determination from the analysis of distribution attack scenarios involving nuisance chemicals,
toxic chemicals and biological agents is shown in Figure 4-3. The analysis by contaminant type shows a
significant difference between the nuisance chemicals, toxic chemicals and biological agents relative to
the timeliness of detection. In general, scenarios that involved toxic chemical contaminants were
determined Credible much more quickly than those that involved biological agents and nuisance
chemicals. The differences in the Credible determination time was most likely due to how the
contaminants affected the taste, odor or appearance of the drinking water, and whether the contamination
scenario generated customer complaints. Contaminants that triggered customer complaints generally
resulted in a quicker credibility determination process, compared with contaminants that did not.
Nuisance
Chemicals
Toxic
Chemicals
Biological
Agents
x=172
n=149
x=135
H n=1034
x = 623
n = 132
200
400
600
Time (minutes)
800
1000
1200
Figure 4-3. Time from Possible to Credible Determination by Contaminant Type
22
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Simulation Study Results by Number of Components Detecting: The percentile distribution of time
from Possible contamination to Credible determination for distribution attack scenarios where one, two or
three different monitoring and surveillance components detected the contaminant is shown in Figure 4-4.
The results indicate that an increase in the number of components causes a decrease in the Credible
determination time span. The data for three components shows a considerably narrower time span for
Credible determination data compared to results where one or two components detected the
contamination.
1 Component
2 Components
3 Components
C
x=153
M I in— ^^ A
x=110
i i n — i zu
x = 49
n = 71
) 100 200 300 400 500 600 700
Time (minutes)
Figure 4-4. Time from Possible to Credible Determination by Number of Components Detecting
Simulation Study Results by Number of Alerts Received: The percentile distribution of time from
Possible contamination to Credible determination for distribution attack scenarios based on the number of
alerts received from three monitoring and surveillance components (WQM, CCS and/or PHS) is shown in
Figure 4-5 and Table 4-3. The advantage that an increasing number of alerts have on the time to
Credible determination is demonstrated by a significant reduction in the time from only one alert being
received to five alerts being received. However, there was little incremental reduction in Credible
determination time as the number of alerts exceeded five.
23
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
11-100 Alerts
9 Alerts
8 Alerts
7 Alerts
6 Alerts
2 Alerts
1 Alert
C
i n
||| n JLJJ
i— | | |-i n = 295
i fn 1 1 n
i [|J i n 11
frn-H
' — Q)-*
|},n=7
a, n-6
i n o
ill 14
' 1 1 ' " 4
S^«"22
) 100 200 300 400 500 600 700 800 900 1000
Time (minutes)
Figure 4-5. Time from Possible to Credible Determination by Number of Alerts Received
Table 4-3. Number of Alerts Received - Times for Credible Determination
Number of Alerts
1
2
3
4
5
6
7
8
9
10
11 -100
>100
Median (x ) Time to Credible Determination
(minutes)
807
746
739
254
148
138
142
148
144
150
135
144
Confirmed Determination (Assumed Contamination/Determined Contamination)
Definition: The Confirmed determination phase of the incident timeline begins when a contamination
incident is deemed Possible, and ends when evidence from follow-on investigations either corroborates or
refutes the contamination incident by escalating the contamination threat to Confirmed or closing the
investigation. This includes the time required to perform laboratory analyses, collect additional
information/evidence and analyze the collective information to determine if a preponderance of evidence
confirms the incident.
The utility determined during the TLD workshops that an incident can be considered Confirmed based on
positive analytical results or a preponderance of evidence. A preponderance of evidence may include
field sample results collected during site characterization; results and observations of site characterization;
24
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
multiple connected monitoring and surveillance alerts; information from public health officials, area
hospitals, or 911 call centers; and/or targeted information from external sources (such as law enforcement
intelligence) based on the collective knowledge of the threat. This being the case, the utility decided to
change the Confirmed determination terminology so that "Assumed Contamination" would be used for
cases based on a preponderance of evidence and "Determined Contamination" would be used for cases
based on positive analytical results. The utility implemented this approach in order to escalate actions
more quickly when there is compelling evidence of contamination rather than delaying for definitive
analytical results. The metrics used for Confirmed determination performance evaluation in this report
include both types. In FSE 1, the utility went to Confirmed based on positive analytical results and in
FSE 2 and FSE 3, the utility went to Confirmed based on a preponderance of evidence. The simulation
model does not differentiate between "Assumed Contamination" and "Determined Contamination;" both
are reported as time of Confirmed determination.
Exercise Analysis Methodology: The analysis for Confirmed determination was performed by
compiling the elapsed time that it took the utility from making a Possible determination to declaring a
Confirmed (e.g., Assumed Contamination or Determined Contamination) threat level during FSEs, and
the investigative information available when the determination was made. This information was tabulated
and examined for correlations between the number or type of alerts received, the investigative
information available to the utility and the amount of time necessary for Confirmed determination.
Exercise Results: Results for Confirmed determination from the exercises are shown in Figure 4-6. As
was the case for Credible determination, no correlation could be made between the number of alerts
received and the amount of time that was required for utility personnel to determine whether the incident
was Confirmed.
The reporting of illnesses in areas of the distribution system during exercises seemed to be a motivating
factor for the investigators involved in the Cincinnati pilot to make a Confirmed determination, as shown
in Table 4-4. This was demonstrated in the case of FSE 3, where GCWW personnel determined the
incident to be Confirmed only fifteen minutes after determining that it was Credible and receiving a PHS
alert. In FSE 1 there was no public health related alert or link to the investigation, therefore a confirmed
laboratory analysis was necessary to generate the Confirmed threat level determination.
25
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
FSE1
FSE2
FSE3
100
200 300
Time (minutes)
400
500
Figure 4-6. Comparison of FSE Time from Possible to Confirmed Determination
Table 4-4. Exercise Times from Credible to Confirmed Determination
Exercise
FSE 1
FSE 2
FSE 3
Number of Alerts
Received
2
7
4
Time from Credible to
Confirmed
Determination (minutes)
60
346
15
Available Information
1 WQM alert: TOC
1 CCS IVR alert: Odor
Suspicious activity: Leaky drums onsite
Positive laboratory results
5 WQM alerts: TOC, chlorine residual
1 CCS alert: impacted neighborhood
Signs of intrusion
PHS alert: Skin irritation, nausea, vomiting
PHS alert: Verified PH impacts via DPIC report:
Increased hospital cases
• DPIC investigation: Suspect toxic chemical
Simulation Study Analysis Methodology: Of the 1,545 distribution attack scenarios that reached
Possible, 1,313 or 85% of them reached Confirmed. As with Credible determination, the analysis for
Confirmed determination utilizing the simulation study results consisted of performing separate statistical
analyses for the scenario characteristics: contaminant type, number of components detecting and number
of alerts.
Simulation Study Results by Contaminant Type: The percentile distribution of time from Possible
contamination to Confirmed determination from the analysis of distribution attack scenarios involving
nuisance chemicals, toxic chemicals and biological agents is shown in Figure 4-7. As was seen with
Credible determination, the analysis by contaminant type for Confirmed determination shows a
significant difference between the nuisance chemicals, toxic chemicals and biological agents relative to
the timeliness of detection. The toxic chemicals are determined Confirmed much more quickly than the
biological agents and nuisance chemicals. The significant difference in the Confirmed determination
timeline is most likely due to the differences between the contaminant types in time for the onset of
symptoms and/or laboratory confirmation.
26
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Nuisance
Chemicals
Toxic
Chemicals
Biological
Agents
= 528
n=149
x=159
n=1034
x=900
nn=130
200 400 600 800 1000
Time (minutes)
1200
1400
1600
Figure 4-7. Time from Possible to Confirmed Determination by Contaminant Type
Simulation Study Results by Number of Components Detecting: The percentile distribution of time
from Possible contamination to Confirmed determination for distribution attack scenarios where one, two
or three different monitoring and surveillance components detected the contaminant is shown in Figure 4-
8. As was seen with Credible determination, the Confirmed determination results indicate that an
increase in the number of components detecting the contamination causes a decrease in the Confirmed
determination time span. This observation is supported by the time from Possible contamination to
Confirmed determination for three components, which shows a considerably narrower time span
compared to the one and two component results.
27
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
1 Component
2 Components
3 Components
C
x = 531.5
H 1 n=524
x= 156.5
H. n _ -7-1 0
x = 77
^ "'71
) 100 200 300 400 500 600 700 800 900 1000
Time (minutes)
Figure 4-8. Time from Possible to Confirmed Determination by Number of Components Detecting
Simulation Study Results by Number of Alerts Received: The percentile distribution of time from
Possible contamination to Confirmed determination for distribution attack scenarios based on the number
of alerts received from three monitoring and surveillance components (WQM, CCS and/or PHS) is shown
in Figure 4-9 and Table 4-5. The advantage that an increase in the number of alerts detecting the
scenarios has on the time to Confirmed determination is significantly different from that seen with the
Credible threat level determination timeline. Here, the median time data does not show a consistent
significant reduction in the timeline until more than 10 alerts are received. However, the data does show
a wide variability in Confirmed determination times.
28
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
6 Alerts
5 Alerts
4 Alerts
2 Alerts
1 Alert
C
i 1 Hi "-11
i | ||i n 11
1 fl 1 14
i U| i n 14
i li n 5
1 \.n-7
[H n = 6
\- < n=4
n 1
S^«"22
) 100 200 300 400 500 600 700 800 900 1000
Time (minutes)
Figure 4-9. Time from Possible to Confirmed Determination by Number of Alerts Received
Table 4-5. Number of Alerts Received-Times for Confirmed Determination
Number of Alerts
1
2
3
4
5
6
7
8
9
10
11 -100
>100
Median (x ) Time to Confirmed Determination
(minutes)
807
746
742
777
804
552
547
706
870
874
166
168
4.1.2 Response A ctions
The amount of time required for response actions is based on the utility's decisions regarding
implementation of those actions, the time it takes to act on them and the time for them to take effect. For
the exercises, response times were quantified for a variety of activities, including the time to notify
response partners that a Possible contamination incident existed, the time following a Possible
contamination incident to decide on appropriate operational responses and the time to make public
notification of water use restrictions. In the simulation study analysis, the time to notify response partners
that a Possible contamination existed was not a feature. However, the time to public health response was
determined and the results were presented with the time to notify response partners. Additionally, the
simulation study analysis quantifies the time to the first operational response, not all of the responses
29
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
individually, as well as the time to a stop use public notification.
Notification of Response Partners
Definition: For exercise analysis, this metric represents the elapsed time from the determination of a
Possible contamination incident to the notification of response partners. For simulation study analysis,
the metric is the elapsed time from the determination of a Possible contamination incident to the time of
public health responses (e.g. issuance of prophylaxis, increasing hospital bed capacities).
Exercise Analysis Methodology: The analysis of the time to notify response partners was accomplished
by compiling the elapsed time recorded in AARs from one SC CCS drill and two of the three FSEs.
Other SC drill AARs were also examined, but those scenarios did not require notification of response
partners.
Exercise Results: Response partner notification time results are shown in Table 4-6. The specific times
at which individual response partners were notified of a Possible contamination incident varied with the
details of the contamination scenarios being exercised. Not all response partners participated in all
exercises, or necessarily in the same sequence. The general observation can be made that the overall
response partner notification time decreased significantly from FSE 2 to FSE 3.
Table 4-6. Time to Notify Response Partners
Exercise
FSE 1
FSE 2
FSES
SC CCS Drill
Time (minutes)
N/A*
79
47
89
254
91
194
131
428
228
34
16
47
179
170
Response Partner Notified
City of Cincinnati Manager
OEPA, ODH, CHD, HCPHD
DPIC
CFD (HazMat)/CPD
Water Information Sharing and Analysis Center
Northern Kentucky Water, Western Water, Butler
Warren County
(ISAC)
County,
EPA Region 5
Ohio Governor's Office
City of Cincinnati Manager
OEPA
CHD
DPIC, HCPHD
CFD (HazMat)/CPD
CFD (HazMat)
Notes:
This metric was not collected for this exercise
Simulation Study Analysis Methodology: The simulation study does not determine the time to notify
response partners; however, the time to public health response was analyzed. Of the 1,545 distribution
attack scenarios that reached Possible, 1,151 or 74% had a public health response. As with Credible and
Confirmed determination, the analysis for time to public health response utilizing the simulation study
results consisted of performing separate statistical analyses for the scenario characteristics: contaminant
type, number of components detecting and number of alerts.
30
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Simulation Study Results by Contaminant Type: The percentile distribution for time from Possible to
public health response from the analysis of distribution attack scenarios involving nuisance chemicals,
toxic chemicals and biological agents is shown in Figure 4-10. As was seen with Credible and
Confirmed determination, the analysis by contaminant type shows a significant difference between
nuisance chemicals, toxic chemicals and biological agents relative to the time to public health response.
Toxic chemicals elicit a response by the public health agencies much more quickly than biological agents.
This difference in time to public health response was most likely due to the differences between the
contaminant types with respect to the time for public health community awareness and how long it took to
identify the contaminant.
x=168
Nuisance
Chemicals
Toxic
Chemicals
Biological
Agents
L n = 94
x=52
= 703
= 886
n=354
500 1000 1500 2000
Time (minutes)
2500
3000
Figure 4-10. Time from Possible Contamination to Public Health Response by Contaminant Type
Simulation Study Results by Number of Components Detecting: The percentile distribution of time to
Possible to public health response from the analysis distribution attack scenarios involving one, two or
three different components detecting the contaminant is shown in Figure 4-11. The significant advantage
that an increasing number of components provide is seen in the time to public health response results
timeline where a narrowing of the range and a reduction in median time from one, two and three
components are evident.
31
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
1 Component
= 461
n=409
x=126
2 Components
n=671
3Components
D
= 52
n=71
500
1000 1500
Time (minutes)
2000
2500
Figure 4-11. Time from Possible to Public Health Response by Number of Components Detecting
Simulation Study Results by Number of Alerts Received: The percentile distribution for time from
Possible to public health response for distribution attack scenarios based on the number of alerts received
from three of the monitoring and surveillance components (WQM, CCS and/or PHS) is shown in Figure
4-12 and Table 4-7. The time to public health response shows a modest downward trend as the number
of alerts received increases from 2 to 10. However, the size of the boxes and length of whiskers in Figure
4-12 illustrate how variable the data are within this range. As was seen with Confirmed determination,
having more than 10 alerts significantly decreases the median time to public health response.
>100 Alerts i-l
11-100 Alerts hTT
-i n = 700
-i n = 379
-i n= 12
| — i n= 1
-i n= 6
-i n= 4
-i n=5
n=7
1 Alert n=0
500
1000 1500
Time (minutes)
2000
2500
Figure 4-12. Time from Possible to Public Health Response by Number of Alerts Received
32
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Table 4-7. Number of Alerts Received - Time to Public Health Response
Number of Alerts
2
3
4
5
6
7
8
9
10
11 -100
>100
Median (x ) Time to Public Health Response
(minutes)
869
870
0*
878
708
706
553
457
462
45
160
*Note: Within the simulation study results, there were only five scenarios that had four alerts received prior to public
health response time. For these five scenarios, the elapsed time was 0, 0, 0, 1,699, and 8,960 minutes.
Mathematically, the median is 0, however the range displayed on the graph more accurately illustrates the expected
value for this metric.
Identification of Utility Operational Response Actions
Definition: For exercise analysis, identification of utility operational response actions is measured by the
average elapsed time from the determination of a Possible contamination incident to the identification of
and decision to implement appropriate operational responses. For simulation study analysis, it is the
elapsed time from the determination of a Possible contamination incident to implementation of the first
operational response.
Exercise Analysis Methodology: The analysis of time taken by the Cincinnati pilot to identify
appropriate operational response was recorded in two of the FSE AARs (FSE 2 and FSE 3). These
operational responses included a variety of actions for controlling water flow in the system, such as
changing the valve patterns and starting up or shutting down pump facilities.
Exercise Results: Table 4-8 shows the average elapsed time to identify the operational response action
for each threat level phase. In this case, the average was used due to the number and different types of
operational responses that were selected by utility personnel to implement. These actions involved taking
pump stations out of service, putting others in service, changing valve configurations, etc. The actions
were assumed to be implemented almost immediately after decisions were made.
As indicated in the table, the average time for identifying appropriate operational response actions during
specific threat levels in simulated contamination incidents varied from 25 minutes to 125 minutes, but
these times were clearly influenced by the circumstances presented by the scenario. It was demonstrated
throughout the FSEs that the utility could implement operational responses very early in the incident
investigation process, and then re-evaluate and modify those responses as additional investigation
information became available. The GCWW drinking water distribution system is configured such that
major changes in water flow can be implemented without disrupting service to the customers. Whenever
utility personnel could isolate, redirect, or slow the transmission of suspected contamination in the system
without disrupting service to the customers, they did so. Thus, the elapsed time from the receipt of a
Possible contamination incident to the decision to implement an operational response such as valving or
pump changes was relatively short.
33
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Table 4-8. Time to Identify Operational Response Actions
Threat
Level
FSE2
FSE3
Possible
Number of
Operational
Response
Actions
1
5
Time to Identify
Response
Actions
(average,
minutes)
38
25
Credible
Number of
Operational
Response
Actions
1
0
Time to Identify
Response
Actions
(average,
minutes)
125
N/A1
Confirmed
Number of
Operational
Response
Actions
0
3
Time to Identify
Response
Actions
(average,
minutes)
N/A2
72
Notes:
1. For FSE 3, when the Credible determination was made, Cincinnati pilot personnel decided that the ongoing
operational actions implemented during the Possible phase were sufficient to contain the contamination, while
continuing to provide service.
2. For FSE 2, no time was projected since the exercise ended exactly at the time of Confirmed determination.
Simulation Study Analysis Methodology: Of the 1,545 distribution attack scenarios that reached
Possible, 1,253 or 81% had an operational response. As with previous metrics, the analysis for time to
the initial operational response utilizing the simulation study results consisted of performing separate
statistical analyses for the scenario characteristics: contaminant type, number of components detecting
and number of alerts.
Simulation Study Results by Contaminant Type: The percentile distribution for time from Possible to
the time of operational response from the analysis of distribution attack scenarios involving for nuisance
chemicals, toxic chemicals and biological agents is shown in Figure 4-13. The operational response data
for all contaminant type scenarios are similar, showing quick implementation and a very narrow
distribution of time since Possible determination. As such, the contaminant type does not show any
meaningful impact on the time it takes to make operational changes.
34
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Nuisance
Chemicals
Toxic
Chemicals
Biological
Agents
x=15
n=103
= 20
n=875
= 20
= 275
10 15
Time (minutes)
20
25
Figure 4-13. Time from Possible to Operational Response by Contaminant Type
Note: Within the simulation study results, all but three of the 275 scenarios involving biological agents had an elapsed
time since Possible of 20 minutes. For these three scenarios, the elapsed time was 11, 13, and 15 minutes.
Therefore, the percentiles illustrated by the "box and whisker" plot are all at the median value of 20.
Simulation Study Results by Number of Components Detecting: The percentile distribution for time
from Possible to the time of operational response from the analysis of distribution attack scenarios
involving one, two, or three different components detecting the contaminant is shown in Figure 4-14.
Similar to contaminant type, the operational response data for the number of components detecting is
similar, showing quick implementation and a narrow distribution of time since Possible determination.
As such, the number of components detecting contamination does not show any meaningful impact on the
time it takes to enact initial operational changes.
35
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
1 Component
SComponente
C
x = 20
n=568
x=15
i n — u i u
x=12
J n = 69
5 10 15 20 25
Time (minutes)
Figure 4-14. Time from Possible to Operational Response by Number of Components Detecting
Note: Within the simulation study results, all but 19 of the 568 scenarios for one component detecting had an elapsed
time since Possible of 20 minutes. For these 19 scenarios, the elapsed times were less than 20 minutes (ranging
from 8 to 15 minutes). Therefore, the percentiles illustrated by the "box and whisker" plots are all at the median value
of 20.
Simulation Study Results by Number of Alerts Received: Percentile distributions for operational
response times for distribution attack scenarios based on the number of alerts received from three
monitoring and surveillance components (WQM, CCS and/or PHS) are shown in Figure 4-15 and Table
4-9. The median time to operational response was identical (20 minutes) for all the number of alerts that
occurred, indicating that the model does not reflect an advantage with respect to timeliness of operational
response in having more alerts.
36
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
11-100 Alerts
10 Alerts
9 Alerts
8 Alerts
7 Alerts
6 Alerts
5 Alerts
4 Alerts
3 Alerts
2 Alerts
1 Alert
1
, 1
1
5 10 15
Time (minutes)
n=340
1 — ' n= 10
1 n= 14
n= 9
n= 6
n= 1
1 n=6
1 n=7
i n= 9
i n= 10
n= 1
20 25
Figure 4-15. Time from Possible to Operational Response by Number of Alerts Received
Table 4-9. Number of Alerts Received - Time to Operational Response
Number of Alerts
1 to > 1 00
(All values simulated)
Median (x ) Time to Operational Response
(minutes)
20
Time Required for Public Notification
Definition: For exercise analysis, this is the time from when the utility PIO was first instructed to
prepare public notification (e.g., do not use) through the time it was released. It includes the time
necessary for the drafting of the document, revision of the document, and coordination and approval by
appropriate agencies. It does not include the time spent on the preparation of employee notifications or
media/press conference materials. For simulation study results analysis, time required for public
notification is the time from the determination of a Possible contamination incident until the time the
public notification was released.
Exercise Analysis Methodology: Time required for public notification was extracted from the AARs of
the three FSEs.
Exercise Results: Table 4-10 shows public notification times as documented from the FSEs. The
average time required for preparation, revision and approval for release of a public notification was 169
minutes. The public notification language was prepared collaboratively among the Cincinnati utility,
local public health and Ohio EPA personnel during conference calls as the scenario unfolded. Preparation
time varied with the scope of the scenarios, as multiple alerts tended to require more frequent updating of
the information covered in the notification, and more iterations of review and approval.
37
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Table 4-10. Time to Prepare and Issue Public Notification
Exercise
FSE 1
FSE2
FSE 3
Average
Time to Prepare and Issue Public Notification
(minutes)
180*
162
165
169
*Note: Actual direction to initiate was not recorded; assumed to have started after the incident was Possible.
Significant modifications occurred to CCP and the public notification process following the FSEs,
including organizational changes to the ICS which added resources (PIO and Customer Information
Manager) to the public information function of the CM component.
Simulation Study Analysis Methodology: Of the 1,545 distribution attack scenarios that reached
Possible, 1,315 or 85% had a public notification response. As with previous metrics, the analysis for time
to public notification utilizing the simulation model consisted of performing separate statistical analyses
for the scenario characteristics: contaminant type, number of components detecting, and number of alerts.
Simulation Study Results by Contaminant Type: The percentile distribution for time from Possible to
public notification from the analysis of distribution attack scenarios involving nuisance chemicals, toxic
chemicals and biological agents is shown in Figure 4-16. The analysis shows a difference for time to
public notification between the toxic chemicals, biological agents and nuisance chemicals relative to the
timeliness of detection. Based on the simulation study analysis, toxic chemicals result in public
notification much more quickly than nuisance chemicals and biological agents. The differences in public
notification time was most likely due to the differences between the contaminant types' effect on taste,
odor, or appearance of the drinking water and whether the contaminant triggered customer complaints or
progression of symptoms leading to healthcare seeking behavior.
Nuisance
Chemicals
Toxic
Chemicals
Biological
Agents
= 528
n=149
x=140
n=1034
= 623
H n= 132
0 200 400 600 800 1000 1200
Time (minutes)
Figure 4-16. Time from Possible to Public Notification by Contaminant Type
38
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Simulation Study Results by Number of Components Detecting: The percentile distribution for time
from Possible to the time of public notification from the analysis of distribution attack scenarios with one,
two, or three different components detecting the contaminant is shown in Figure 4-17. The figure
demonstrates that an increase in the number of components results in a slight decrease in the median time
to public notification, but never less than 120 minutes. This is driven by a specific model parameter that
requires 120 minutes to develop a public notification while the decision to release the public notification
is made independently. For scenarios with only one or two components detecting, it takes longer for the
utility to acquire the information needed to determine whether or not to release the public notification
than it takes to actually develop it. When three components have detected contamination, the utility has
sufficient information to release the public notification as soon as it is ready.
1 Component
2Componente
SComponente
C
x=153
i in- ^"M
x=133
1 /; - / 20
x=120
i n=71
100 200 300 400 500 600 700 800
Time (minutes)
Figure 4-17. Time from Possible to Public Notification by Number of Components Detecting
Note: Within the simulation study results, all but 2 of the 71 scenarios involving 3 components detecting had an
elapsed time since possible of 120 minutes. For these 2 scenarios, the elapsed time was 162 and 177 minutes.
Therefore, the percentiles illustrated by the "box and whisker" plots are all at the median value of 120.
Simulation Study Results by Number of Alerts Received: Percentile distributions for public
notification times for distribution attack scenarios based on the number of alerts received from three
monitoring and surveillance components (WQM, CCS, and/or PHS) are shown in Figure 4-18 and Table
4-11. The advantage that an increasing number of alerts has on the time to public notification is clearly
demonstrated by a significant reduction in the time from only one alert being received to six alerts being
received. There is no significant advantage in an increasing number of alerts with respect to timeliness of
for the time to public notification beyond six alerts.
39
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
> 100 Alerts
11-100 Alerts
10 Alerts
9 Alerts
8 Alerts
7 Alerts
6 Alerts
5 Alerts
4 Alerts
3 Alerts
2 Alerts
1 Alert
C
f~H-i "=295
iffl i i 11
iyj inn
JTl-i n= 14
i fl 1 i n- 10
' U I
4E- n=5
O-7
H h=6
i | | i n ,
i Fl i n-T
III
| [i n=20
) 100 200 300 400 500 600 700 800 900 1000
Time (minutes)
Figure 4-18. Time from Possible to Public Notification by Number of Alerts Received
Table 4-11. Number of Alerts Received - Time to Public Notification
Number of Alerts
1
2
3
4
5
6
7
8
9
10
11 -100
>100
Median (x ) Time to Public Notification
(minutes)
807
746
741
736
448
180
149
149
144
153
144
146
4.1.3 Remediation and Recovery
None of the FSEs included R&R objectives due to scope and time constraints. Thus, response time
metrics used to evaluate this aspect of the CM component were never generated. However, several
workshops were conducted with the Cincinnati pilot that resulted in multiple changes to the R&R
decision logic in the Cincinnati Pilot Consequence Management Plan.
The original R&R decision tree incorporated into early drafts of the Cincinnati Pilot Consequence
Management Plan was modeled after the one developed for EPA's Response Protocol Toolbox, which
incorporated many aspects of the Comprehensive Environmental Response, Compensation, and Liability
Act (CERCLA) remediation process. The utility later participated in a national R&R workshop co-
40
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
sponsored by the American Water Works Association (AWWA) and EPA, conducted on March 17-18,
2009. Twenty-nine participants representing water utilities, EPA, U.S. Army Corps of Engineers, the
Department of Homeland Security (DHS), the Federal Emergency Management Agency (FEMA) and
other federal agencies convened to discuss the procedures and action associated with R&R from a
Confirmed drinking water contamination incident. The objectives of the workshop included reviewing
and refining (as appropriate) the R&R process that was outlined in the U.S. EPA Water Interim Guidance
on Developing Consequence Management Plans for Drinking Water Utilities (EPA, 2008), discussing
utility interaction with the National Response Framework, and discussing an outline for guidance on
containment/disposal of decontamination waste.
As a follow-up to the national workshop, the Cincinnati pilot conducted an internal R&R workshop on
May 14, 2009, to review their R&R decision logic relative to the results of the previous national
workshop. The major outcome of this workshop was the adoption of the national workshop R&R
decision process.
A final R&R workshop was conducted on May 21, 2010 to discuss the revised R&R process with local
response partner agencies. This was a follow-up activity related to FSE 3 where the exercise scenario was
used to facilitate discussion concerning probable R&R activities and corresponding roles and
responsibilities. The workshop resulted in the addition of 14 response partners and 60 modifications
and/or clarifications of response partner roles and responsibilities in the Cincinnati Pilot Consequence
Management Plan. In addition, participants agreed that the revised R&R decision tree: 1) contributed
significantly to utility and response partner understanding of the R&R process, 2) were compatible with
existing utility and response partner response plans, and 3) would facilitate the R&R process in the event
of a contamination incident.
4.1.4 Summary
There were no statistical trends demonstrated from the exercise data for the timeliness of detection and
response for the incident response plans. The variability seen in the data for investigative and response
actions were a direct result of the variations presented by the contamination scenarios themselves.
Although this was the case, the design elements evaluated as a part of this design objective did reveal
several significant observations.
The simulation study analysis demonstrated clear trends for timeliness of detection and response actions
in reaction to the scenario attacks evaluated. However, some of the metrics analyzed showed inconsistent
results or results that were driven by the model parameters.
Credibility Determination
Exercise Investigative Actions: For Credible and Confirmed determination, no correlation could be
made between the number of alerts received and the amount of time that was required for utility personnel
to determine whether the incident was Credible or Confirmed. Overall, a hierarchy of investigation
information types evolved, which seemed to accelerate the speed with which threat level determinations
were made, including:
• Multiple alerts, system connectivity, positive RFTs and signs of intrusion accelerated the
declaration of Credible contamination incidents and
• Signs of intrusion and health impacts accelerated the declaration of Confirmed contamination
incidents and did not necessarily depend on positive laboratory analysis.
41
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Simulation Study Investigative Actions: For Credible and Confirmed determination, a strong
correlation was observed when evaluated by the type of contaminant introduced and the number of
components detecting the contaminant:
1. The model results showed significant differences for threat level determination for toxic chemical
contaminants compared to biological agents. The toxic chemical contaminants resulted in much
quicker threat level determination. For Credible determination, the difference in timeliness is
most likely due to the contaminant types' effects on taste, odor, or appearance of the drinking
water resulting in customer complaints or progression of symptoms. For Confirmed
determination, the difference in timeliness is most likely due to the same factors that drive
Credible determination as well as being able to be analyzed and identified much more quickly.
2. The simulation study also demonstrated a significant advantage provided by an increasing
number of different monitoring and surveillance components for a reduction in threat level
determination time. Three components detecting results showed a marked advantage in the
timeline over two components detecting which generally showed an advantage over one
component detecting.
The simulation study analysis of the number of alerts on threat level determination was inconsistent for
Credible determination compared to Confirmed determination. While the Credible determination results
showed a definite advantage at five alerts and above in reducing the timeline, the Confirmed
determination results were too inconsistent to draw any conclusions.
Response Actions
Exercise Investigative Actions: The time required for the utility to notify its response partners varied
with the sequence of circumstances presented by the various exercise scenarios, but generally was
consistent from FSE 2 to FSE 3, with some slight improvements with key partners including CFD, CHD,
DPIC, and OEPA).
The average time for identifying appropriate operational response actions during specific threat levels in
the exercises varied, but these times were clearly influenced by the circumstances presented by the
scenario. Operational responses were initially driven by what actions the utility could implement quickly
to isolate or slow contamination without impacting service. As the incident progressed, investigation
evidence was subsequently used to revise those response actions as necessary.
The time to develop and implement public notification was consistent throughout the exercises, with an
average time of 169 minutes from direction to prepare and availability to release. Given the variability of
exercise scopes and the accompanying revision of the CCP, it was not possible to make statistical
inferences concerning the performance.
Simulation Study Investigative Actions: The simulation study results for time from Possible to public
health response showed a strong correlation to the contaminant type and to the number of components
detecting metrics. A comparison of the toxic chemical to the biological agents indicated that toxic
chemical contaminants resulted in much quicker implementation of public health response and a much
narrower time span. The difference in time to public health response was most likely due to differences
between contaminant types with respect to the time for the rapid onset of symptoms, public health
community awareness, and variations in the time taken to identify the contaminant.
The simulation study results for time to the first operational response were very similar for contaminant
type, number of components detecting, and number of alerts metrics. All metric results indicated very
quick first operational response implementation times and a very narrow time span.
42
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
The simulation study results for time to public notification showed a correlation for the contaminant type,
number of components detecting and number of alerts metric results. A comparison of the toxic chemical
and biological agent results showed that toxic chemical contaminants resulted in much quicker public
notification over a very narrow time span. As indicated earlier, this was most likely due to differences
between the contaminant types' effect on taste, odor, or appearance of the drinking water and whether the
contaminant triggered customer complaints or progression of symptoms leading to healthcare seeking
behavior. The results for the number of components detecting showed a slight advantage for three
components detecting followed by two components detecting with a slight advantage over one component
detecting. Finally, the number of alert results impact on the time to public notification showed a
significant reduction in the public notification time at six alerts and above.
R&R
Several workshops resulted in significant reduction of the steps in the R&R decision tree and
corresponding response actions. These workshops also resulted in the addition of 14 response partner
agencies to the R&R process, and more than 60 clarifications to the R&R roles and responsibilities. The
R&R process was never tested and evaluated through the FSEs and therefore no metric data concerning
timeliness of response was collected.
43
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Section 5.0: Performance of Integration of the Response
Partner Network
The response partner network design element was meant to provide a framework upon which the
Cincinnati pilot could coordinate their respective response actions associated with a contamination
incident. The overall goal of the network was to understand, integrate and achieve consensus regarding
roles and responsibilities during the CM process. The inclusion of the response partner network as a
design element of the CM component was intended to streamline the overall response process.
Measurement of the performance of this design element consisted of evaluating the effectiveness of
establishing and understanding roles and responsibilities, quantifying how well the network was
integrated into the CM component, and describing how the nature of the support provided by the network
changed during the pilot evaluation period.
5.1 Design Objective: Timeliness of Response
While there is no empirical data to describe how this design element contributes to timeliness of detection
and response by itself, it is assumed that the integration of the response partner network into the CM
activities increased the efficiency of overall response activities. The performance of the individual design
element for this objective specifically focuses on how well the response partner network was developed
and implemented, and how it evolved during the pilot study.
5.1.1 Understanding Response Partner Roles and Responsibilities
One of the earliest activities of the pilot study was to conduct a series of workshops with the Cincinnati
pilot to elicit input on roles and responsibilities that would be integrated into the CM component.
Evaluation criteria were then included in several subsequent exercises to determine how well the
Cincinnati pilot understood their respective roles and responsibilities.
Definition: This metric consists of the number of improvement recommendations associated with the
Cincinnati pilot that are associated with the understanding of roles and responsibilities.
Analysis Methodology: Improvement recommendations contained in the AARs from exercises,
specifically based on improving the response partner network, were analyzed for this metric. These
improvement recommendations involved changes to the Cincinnati Pilot Consequence Management Plan
and/or implementation of training.
Results: Major improvement recommendations generated from each of the exercises are described in
Table 5-1. There was no statistical trend in the number or type (plan vs. training) of improvement
recommendations found in the exercise AARs, although improvement was seen between the functional
exercise (which was conducted first) and the subsequent FSEs. The majority of the response partner
network issues reflected in the functional exercise involved the need to improve coordination and
communication between the utility and its response partners, which was not an outstanding issue during
the FSEs.
44
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Table 5-1. Response Partner Recommendations by Exercise
Exercise
Functional
Exercise
FSE 1
FSE2
FSE 3
Total Number of
Recommendations
12
(4 CMP-related
only; 8 both CMP
and Training-
related)
4
(1 CMP-related
only; 3 both CMP
and Training-
related)
4
(1 CMP-related; 3
Training-related)
1
(1 Training-
related)
Description
• Improve communication and coordination between utility and
multiple response partners
• Clarify the point of contact for the HazMat team
• Modify the list of response partner contacts and the timing of
external agency notification to allow for flexibility
• Improve communication between utility and multiple response
partners
• Add DPIC as a toxicology resource
• Modify the list of response partner contacts and the timing of
external agency notification to allow for flexibility
• Improve HazMat team access to utility facilities in emergency
situations
• Reconcile sampling protocol differences between the utility SCT
and HazMat teams
• Reconcile mission turnover procedures from the utility SCT to the
HazMat team
• Mark sampling locations at utility facilities for easy identification by
Hazmat teams
• Increase cross-training activities with HazMat teams
• Clarify utility protocols (standard language) for 91 1 requests
• Clarify response partner responsibilities during evacuation actions
5.1.2 Integration of Response Partners
Local, state, regional and federal response partner agencies played an integral role in the CM component.
Integration of the response partners into the initial development of the CM component, the reconciliation
of incident response plans and the eventual evaluation of those roles and responsibilities in exercises was
crucial to the success of the CM component. This element of the design objective addressed how well the
response partners were integrated into the component.
Definition: The degree to which the response partners were integrated into the CM component is
measured in terms of the number of times each response partner participated in the development,
execution or evaluation of a CM component exercise. The voluntary participation of response partners in
CM events indicates their willingness to actively engage in CM activities to improve overall response to
contamination incidents.
Analysis Methodology: The AARs for all exercises that involved response partner participation were
examined.
Results: The response partners that participated in each CM related exercise are shown in Table 5-2.
Overall, response partner integration into the CM component was both extensive and continuous,
beginning with the earliest workshops and continuing through the last FSE.
Table 5-2. Response Partner Exercise Participation
Response Partner
Cincinnati Fire Dept.
(CFD)/HazMat
Cincinnati Managers Office
Cincinnati Police Department
(CPD)
CM
Workshops
X
X
X
SC
Drill 2
X
Functional
Exercise
X
X
X
FSE1
X
X
FSE 2
X
X
FSE 3
X
FSE 3
Workshop
45
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Response Partner
County Wide Fire Reps
Metropolitan Sewer District
(MSD)
Cincinnati Health Department
(CHD)
Hamilton County Public Health
Department (HCPHD)
Drug and Poison Information
Center (DPIC)
Hamilton County Emergency
Management Agency (HCEMA)
Ohio Department of Health
(ODH)
Ohio Environmental Protection
Agency (OEPA)
Federal Bureau of Investigation
(FBI)
U.S. EPA
CM
Workshops
X
X
X
SC
Drill 2
X
Functional
Exercise
X
X
X
X
X
X
X
X
FSE1
X
X
X
X
X
X
FSE2
X
X
X
X
X
X
X
FSE3
X*
X
X
X
FSE3
Workshop
X
X
X
X
X
Notes:
Blank cells indicate that the partner did not participate; grey cells indicate that participation was not required.
* The CHD response partner role was assumed by an EPA employee who was a former HCPHD employee.
5.1.3 Nature of Response Partner Support
This aspect of response partner integration addresses the nature of the support provided to the Cincinnati
pilot. The different types of support included law enforcement, hazardous materials management,
emergency operations, laboratory analysis, technical support from health agencies and regulatory support.
Definition: The nature of the partner support provided to the Cincinnati pilot is defined by characterizing
the type of the support as active responder, active support or passive support.
Analysis Methodology: Actions performed by response partners during exercises were compiled from
the AARs and classified as active responder, active support or passive support using the following
criteria:
• Active Responder - response partner requested to fully or partially assume a primary
responsibility in the Cincinnati Pilot Consequence Management Plan. Example: Local HazMat
team on scene conducting sampling operations.
• Active Support - response partner implementing its individual role and responsibility to support
the utility. Example: DPIC, OEPA, CHD providing public health, toxicity, or regulatory support
during an investigation.
• Passive support - all other response partners that have been notified of an incident investigation,
but are not actively providing support. Example: MSD or Cincinnati Managers Office.
Results: Table 5-3 displays the results of the tabulation. In general, the type of support provided each
response partner agencies did not vary with the circumstances presented by the exercise scenarios. The
one exception is the FBI for FSE 3 (grey shaded cell) because there were no signs of intrusion during the
scenario and therefore the FBI would not have had an investigation role prior to the incident being
confirmed.
46
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Table 5-3. Nature of Response Partner Support
Response Partner
Cincinnati Fire Dept. (CFD)/HazMat
Cincinnati Managers Office
Cincinnati Police Department (CPD)
County Wide Fire Reps
Metropolitan Sewer District (MSD)
Cincinnati Health Department (CHD)
Hamilton County Public Health Department (HCPHD)
Drug and Poison Information Center (DPIC)
Hamilton County Emergency Management Agency
(HCEMA)
Ohio Department of Health (ODH)
Ohio Environmental Protection Agency (OEPA)
Federal Bureau of Investigation (FBI)
U.S. EPA
FSE1
AR
PS
AS
PS
PS
AS
AS
AS
AS
AS
AS
PS
FSE2
AR
PS
AS
PS
PS
AS
AS
AS
AS
AS
AS
PS
FSE3
AR
PS
AS
PS
PS
AS
AS
AS
AS
AS
PS
PS
Notes:
AR - Active Responder; AS - Active Support; PS - Passive Support
5.1.4 Summary
The inclusion of the response partner network as a design element of the CM component was intended to
streamline the overall response process. Although there was no empirical data to describe how this
design element contributed to timeliness of detection and response by itself, the design elements
evaluated as apart of this design objective did reveal several observations:
a. Understanding Response Partners Roles and Responsibilities. There was no statistical trend in
the number or type of improvement recommendations found in the exercises, although
improvement was seen between the functional exercise and the subsequent FSEs. The majority
of these recommendations focused on the need to improve coordination and communication
between the utility and its response partners, which was not an outstanding issue during the FSEs.
This improvement indicated that the utility and response partners were achieving a better
understanding of their CM-related roles and responsibilities as well as improving communication
and coordination.
b. Integration of Response Partners. A total of twelve response partner agencies with various
roles and responsibilities were extensively involved with the Cincinnati pilot CM component.
Eleven of the response partner agencies were involved from the earliest phase of development
through the conclusion of the evaluation period. DPIC was recognized early on as a vital missing
response agency and quickly engaged. These response partner agencies provided the Cincinnati
pilot a variety of significant CM support including law enforcement, hazardous materials
management, emergency operations, laboratory analysis, technical support from health agencies
and regulatory support. Overall, the different types of support provided by these response
partners contribute to a well-rounded response.
47
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Section 6.0: Performance of Communication Equipment
During the pre-implementation assessment of the CM component, it was noted that utility field response
teams did not have an established way to communicate among utility team members, with response
partners (e.g., HazMat) or between the utility and the ICS. This hindered the organization's ability to
efficiently communicate and coordinate response to a contamination incident. Prior to implementation of
the CM component, field response communications within the utility would have occurred using a facility
phone, a cell phone or over a utility radio. Communications outside the utility would have occurred using
a facility phone or cell phone.
Eight 800 MHz hand-held radios (Motorola XTS 5000) were acquired to address this deficiency. The 800
MHz radios are long-range multi-channel programmable units that are inter-operable with response
partner agencies in the City of Cincinnati and Hamilton County. The use of designated frequencies
allows utility personnel to communicate both internally and with response partners including fire, police,
health and other city workers. The radios were also programmed to operate on other agency frequencies
in the event of a Unified Command System incident response. The radios are located and deployed with
the utility SCT Leader.
6.1 Design Objective: Timeliness of Response
6.1.1 Use of 800 MHz Radios
While there is no empirical data to document how the use of the 800 MHz radios reduced the time to
investigate possible contamination incidents and implement response actions, evaluators observed during
exercises that the radios greatly facilitated communication between the utility ICS and field response
personnel. In addition to the exercises, the 800 MHz radios were used during the response to the
September 2008 wind storm and resulting power outage to address communications during the outage.
The radios were used by utility field crews and supervisors primarily because cellular phones did not
consistently work, and the radio system in utility cars was also down for several days.
Using the 800 MHz radios during the exercises also resulted in the identification of dead spots in the
coverage area and radio interference from some pumping facilities. Steps to eliminate these problems
were under consideration at the end of the evaluation period. In addition, the utility established an
internal secure website (SharePoint) to facilitate communications among incident command personnel
with access to a network computer. The consensus presented in the AARs was that the use of SharePoint
site to communicate real-time information among the utility ICS increased the overall efficiency of
response management.
6.1.2 Summary
The integration of the 800 MHz radios and SharePoint site into the utility CM related response activities
enhanced their ability to communicate information both internally and externally. Data and feedback
from the exercises indicated that the use of the communication equipment allowed faster and more
complete investigation of contamination incidents. By the end of the pilot evaluation period, utility
personnel routinely used the communication equipment for contamination incident investigations.
48
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Section 7.0: Sustainability
Sustainability is a key objective in the design of a CWS and each of its components. For the purpose of
this evaluation, Sustainability is defined in terms of the cost-benefit trade-off. Costs are estimated over
the 20 year lifecycle of the CWS and include the capital cost to implement the CWS and the cost to
operate and maintain the CWS. The benefits derived from the CWS are defined in terms of primary and
dual-use benefits. The primary benefit of a CWS is the potential reduction in consequences in the event
of a contamination incident. However, such a benefit may be rarely, if ever, realized. Thus, dual-use
benefits play an extremely important role that provide value to routine utility operations and are an
important driver for Sustainability. Ultimately, Sustainability can be demonstrated through utility and
partner compliance with the protocols and procedures necessary to operate and maintain the CWS. The
two metrics that were evaluated to assess how well the Cincinnati CWS met the design objective of
Sustainability are: Costs and Benefits. The following subsections define each metric, describe how it was
evaluated and present the results.
7.1 Costs
Definition: Costs are evaluated over the 20 year lifecycle of the Cincinnati CWS, and comprise costs
incurred to design, deploy, operate and maintain the CM component since its inception. It should be
recognized that the Cincinnati CWS was a pilot research project and as such, likely incurred costs higher
than another utility would realize.
Analysis Methodology: Parameters used to quantify the implementation cost of the CM component
were extracted from the Water Security Initiative: Cincinnati Pilot Post-Implementation System Status
(USEPA, 2008). Implementation costs include labor and other expenditures (equipment, supplies, and
purchased services) for designing and deploying the CM component. O&M costs were tracked on a
monthly basis over the duration of the evaluation period. Renewal and replacement costs, along with the
salvage value at the end of the Cincinnati CWS lifecycle were estimated using vendor supplied data, field
experience and expert judgment. Section 3.4 provides additional details regarding the methodology used
to estimate each of these cost elements.
Results: The methodology described in Section 3.4 was applied to determine the value of the major cost
elements used to calculate the total lifecycle cost of the CM component, which are presented in Table 7-
1. It is important to note that the Cincinnati CWS was a research effort, and incurred higher costs than
would be expected for a typical large utility installation. A similar CM component implementation at
another utility should be less expensive as it could benefit from lessons learned and would not incur
research-related costs. Additional information regarding the data used to determine the value each cost
element is presented below.
Table 7-1. Cost Elements used in the Calculation of Lifecycle Cost
Parameter
Implementation Costs
Annual O&M Costs
Renewal and Replacement Costs
Salvage Value
Value
$1,430,627
$33,948
$22,624
-
Table 7-2 below presents the implementation cost for each CM design element, with labor costs
presented separately from the cost of equipment, supplies and purchased services.
49
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Table 7-2. Implementation Costs1
Design Element
Project Management2
Incident Response
Plans
Response Partner
Network
Communication
Equipment
Training and Exercises
TOTAL:
Labor
$102,749
$712,828
$153,527
$16,338
$420,734
$1,406,175
Equipment, Supplies,
Purchased Services
-
$793
-
$23,584
$74
$24,451
Total
Implementation
Costs
$102,749
$713,621
$153,527
$39,922
$420,808
$1,430,627
1 All numbers rounded to the nearest dollar.
2 Project management costs incurred during implementation were distributed evenly among the CWS components.
The first design element, project management, includes overhead activities necessary to design and
implement the component. The incident response plans design element includes the cost of developing a
Cincinnati Pilot Consequence Management Plan detailing roles and responsibilities as well as developing
a preparedness and response guide. A CCP formalizing public notification procedures and guidance for
GCWW Public Information Officer was also developed. The third design element, response partner
network, includes the cost of identifying response partners and gathering their input as to roles and
responsibilities in dealing with a water contamination incident. The fourth design element,
communication equipment, includes the cost of purchasing eight 800 MHz hand held radios to improve
GCWW's ability to respond to an incident and to communicate and coordinate appropriately with
response partners in the field. The fifth design element, training and exercises, includes the cost of
designing and executing workshops, tabletop exercises, functional exercises, drills and full scale exercises
to test the Cincinnati Pilot Consequence Management Plan and to train the participants on processes and
procedures.
Overall, the incident response plans design element had the highest implementation costs (50% of the
total). Significant labor costs were involved in developing the Cincinnati Pilot Consequence
Management Plan. The total implementation cost for the training and exercises design element were
somewhat lower at 29% of the total, but also required significant labor costs for planning, coordinating,
and executing system- and component-level drills and exercises. These labor costs also involved
developing AARs for each drill and exercise to summarize the contamination scenario, partner actions,
response timelines and areas for improvement. Implementation costs for project management and
communication equipment were significantly lower at 7% and 3% of the total, respectively.
The annual labor hours and costs of operating and maintaining the CM component, broken out by design
element, are shown in Table 7-3.
Table 7-3. Annual O&M Costs
Design Element1
Procedures
TOTAL:
Total Labor
(hours/year)
771
771
Total Labor
Cost
($/year)
$33,948
$33,948
Supplies and
Purchased Services
($/year)
-
-
Total O&M Cost
($/year)
$33,948
$33,948
1 Overarching project management costs were only incurred during implementation of the CM component and are not
applicable for annual O&M costs.
50
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Most of the O&M labor hours reported under procedures was spent on ongoing coordination of drills,
exercises and trainings to maintain readiness for response to possible water contamination incidents.
Figure 7-1 shows the O&M labor hours for each reporting period over the course of the entire evaluation
period between January 2008 and May 2010. In this evaluation, the term 'reporting period' is used to
refer to a month of metrics data which spans from the 16th of one month to the 15th of the next month.
Thus, the January 2008 reporting period refers to the data collected between January 16th 2008 and
February 15th 2008.
The majority of the reporting periods experienced labor hours across all organizations of less than 400
hours. The three largest labor hour values were recorded during the September 2008, September 2009,
and October 2009 reporting periods due to preparation for and completion of the FSEs. The increase
during the December 2008 reporting period to 420 labor hours was due to extensive review of the AAR
for the previous FSE. Lessons Learned workshops and the R&R workshop accounted for other above-
average labor hours reporting periods in April 2009 and May 2010.
1
2
1400
1200
1000
800
600
400
200
GCWW
• Response Partner(s)
i Contractor
.H
i11i i iIIi ii i il
.
Start Date of Monthly Reporting Period
Figure 7-1. O&M Labor Hours per Reporting Period
Two of the major cost elements presented in Table 7-1, the renewal and replacement costs and salvage
value, were based on the costs associated with major pieces of equipment installed for the CM
component. The useful life of these items were estimated at 5 years and 10 years, respectively, based on
manufacturer-provided data. It was assumed that the items with a useful life of 5 years would need to be
replaced three times during the 20-year lifecycle of the CWS, and the items with a useful life of 10 years
were assumed to be replaced once. Because the useful life of the final installment of all equipment items
will expire at the end of the 20 year lifecycle, there is no salvage value for this component, as reported in
Table 7-1. The cost of these items is presented in Table 7-4.
51
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Table 7-4. Equipment Costs
Equipment Item
Motorola 800MHz
LCD 40 inch Flat Panel Monitor
Useful Life
(years)
10
5
Unit Capital
Costs
$2,828
$500
Quantity
(# of Units)
8
2
TOTAL:
Total Cost
$22,624
$1,000
$23,624
To calculate the total lifecycle cost of the CM component, all costs and monetized benefits were adjusted
to 2007 dollars using the change in the Consumer Price Index between 2007 and the year that the cost or
benefit was realized. Subsequently, the implementation costs, renewal and replacement costs, and annual
O&M costs were combined to determine the total lifecycle cost:
CM Total Lifecyde Cost: $2,000,828
*Actual costs were adjusted to 2007 dollars
Note that in this calculation, the implementation costs were treated as a one-time balance adjustment, the
O&M costs recurred annually, and the renewal and replacement costs for major equipment items were
incurred at regular intervals based on the useful life of each item.
7.2 Benefits
Definition: The benefits of CWS deployment can be considered in two broad categories: primary and
dual-use. Primary benefits relate to the application of the CWS to detect contamination incidents, and can
be quantified in terms of a reduction in consequences. Primary benefits are evaluated at the system-level
and are thus discussed in the Water Security Initiative: Evaluation of the Cincinnati Contamination
Warning System Pilot (USEPA, 2013b). Dual-use benefits are derived through application of the CWS to
any purpose other than detection of intentional and unintentional drinking water contamination incidents.
Dual-use benefits realized by the CM component are presented in this section.
Analysis Methodology: Information collected from forums, such as data review meetings, lessons
learned workshops and interviews were used to identify dual-use applications of the CM component of
the CWS.
Results: Operation of the CM component of the CWS has resulted in benefits beyond the response to
intentional and unintentional contamination incidents. These key dual-use benefits and examples
identified by the utility include:
1. Stronger interagency relationships with response partners
• The close coordination with response partners that is required for the CM component
translates to improved coordination during simple non-contamination incidents that can
impact the distribution system (e.g., natural disasters).
2. Strengthened incident command structure
• Efficient response to contamination incidents relies on a sound command structure to
manage multiple utility divisions as well as support from external response partners.
Development of the CM component emphasizes Incident Command Systems principles
that translate to all types of utility responses from local main breaks to multi-
jurisdictional and multi-agency emergencies.
52
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
3. Increased preparedness of utility management and staff to respond to "all-hazards"
• Through active training programs that stress classroom-based and field-based exercises,
development of the CM component stresses a step-wise process for response that equips
the utility to more effectively respond to "all-hazards". For example, GCWW personnel
indicated that they were much more comfortable and confident responding to "real"
incidents after successfully performing their response actions during the full scale
exercises.
Many of the listed dual-use benefits are illustrated in the case study below, which occurred during the
evaluation period.
Case Study: Response During Hurricane Ike Windstorm
On the afternoon of Sunday, September 14, 2008, a severe windstorm associated with the remnants of
Hurricane Ike struck the Greater Cincinnati region, resulting in a loss of power to 90 percent of the area.
This affected numerous aspects of the utility's operations, including pressure and flow in the distribution
system. Many of the utility's pumping stations were without power for a lengthy amount of time and
considerable effort was exerted to maintain pressure throughout the system.
As a result of the storm, the Cincinnati pilot utilized components of its emergency response plans and the
Cincinnati Pilot Consequence Management Plan. Several dual-use benefits were realized as a result of
this storm event including:
1. Cincinnati Pilot Consequence Management Plan. The updated version of the Cincinnati Pilot
Consequence Management Plan improved the utility's response to the overall system event
mainly through the implementation of the ICS structure. Subsequent updates to the Cincinnati
Pilot Consequence Management Plan as a result of exercises will enhance the incident response
plans and reduce response times to "all-hazard" type emergencies.
2. 800 MHz radios. The 800 MHz radios were used by utility field crews and supervisors primarily
because cellular phones did not consistently work. The radio systems in utility's cars were also
down for several days.
3. Confirmatory Sampling Field Decision Guide (CSFDG). The Water Quality and Treatment
(WQ&T) Division utilized both the CSFDG map of Pito Zones and performed modeling of a
service area at approximately 10:00 p.m. on September 14, 2008. This was performed to identify
vulnerable locations for pressure monitoring by distribution field crews.
4. Crisis Communications Plan (CCP). Principles within the CM CCP were applied during the
response to the wind storm. Although a boil water notice was never issued, a "water
conservation" notice was given to customers in several pressure zones as a precautionary measure
to discourage water consumption for non-essential uses.
7.3 Summary
Sustainability was measured by labor hours for CM implementation, O&M and dual-use benefits. The
total lifecycle cost for the CM component, which included implementation, renewal and replacement, and
annual O&M costs was $2,000,828.
Labor Hours for CM Implementation
Developing the Cincinnati Pilot Consequence Management Plan and conducting the exercises were the
most significant tasks for the implementation of this component. Both required work with utility
personnel as well response partner agencies. Implementation costs approximately amounted to $800,000
though equipment costs were minimal (approximately $40,000).
53
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Operation and Maintenance
Labor hours for O&M were steady at approximately 340 hours per month through most of the evaluation
period, with the exception of the reporting periods surrounding FSEs. These exercise required substantial
planning and participation from numerous utility personnel and response partner agencies.
Dual-Use Benefits
The Cincinnati pilot CM ICS structure, communication equipment, and crisis communication procedures
were used during a severe windstorm that occurred in the Cincinnati area on September 14, 2008. This
provided a case study example of the dual-use benefits associated with the CM component.
54
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Section 8.0: Summary and Conclusions
The evaluation of the CM component of the Cincinnati pilot CWS involved analysis of empirical data,
qualitative observations gleaned from active participants and results from the simulation study.
Highlights, limitations and considerations for interpretation of this analysis are presented here.
8.1 Design Objective: Timeliness of Response
The overall metrics used to evaluate timeliness of response were derived from three separate design
elements: the efficiency of implementing incident response plans, the degree to which the response
partner network was integrated into the CM component and the use of communication equipment. All of
the data were obtained from the AARs (where applicable) that were developed following CM exercises
and from the results of the simulation study conducted to evaluate the entire CWS. No actual
contamination incidents of the utility drinking water system occurred during the pilot period.
8.1.1 Incident Response Plans
There were no statistical trends demonstrated among the FSEs that were primary data sources for
measuring the effectiveness of the incident response plans. This was due to the variation in
contamination scenarios used for each exercise, which made direct comparison of response times
difficult. However, the design elements evaluated as a part of this design objective revealed
characteristics of how the utility investigated and responded to contamination incidents.
The simulation study results demonstrated strong correlations for the time to Credible determination, time
to Confirmed determination, time to public health response, time to first operational response, and time to
public notification metrics for the majority of the analyses performed. However, some of the metrics
analyzed showed inconsistent results or results that were driven by the model parameters. Analyses
including by contaminant type, number of components detecting and the number of alerts (some
variability in correlations) were performed.
Credibility Determination
Exercise Investigative Actions: This included the amount of time required for investigative actions
leading to both Credible and Confirmed (including "Assumed Contamination") determination of a
Possible contamination incident. A hierarchy of investigation information types evolved, which seemed
to accelerate the speed with which the CM component was implemented (progression through threat level
determinations). Two primary examples include:
1. Multiple alerts, system connectivity, positive RFTs and signs of intrusion accelerated the
declaration of Credible contamination incident and
2. Signs of intrusion and health impacts accelerated the declaration of Confirmed (Assumed
Contamination) and did not necessarily depend on positive laboratory analysis.
Simulation Study Investigative Actions: This included the amount of time required for investigative
actions leading to both Credible and Confirmed determination of a Possible contamination incident. The
type of contaminant introduced and the number of components showed a strong impact on the timeliness
of threat level determination for Credible and Confirmed. Two primary examples include:
1. The model results showed significant differences for threat level determination for toxic chemical
contaminants compared to biological agents. The toxic chemical contaminants resulted in much
quicker threat level determination. For Credible determination, the difference in timeliness is
most likely due to the contaminant types' effects on taste, odor or appearance of the drinking
water resulting in customer complaints or progression of symptoms. For Confirmed
55
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
determination, the difference in timeliness is most likely due to the same factors that drive
Credible determination as well as being able to be analyzed and identified much more quickly.
2. The simulation study also demonstrated a significant advantage provided by an increasing
number of different monitoring and surveillance components for a reduction in threat level
determination time. Three components detecting results showed a marked advantage in the
timeline over two components detecting which generally showed an advantage over one
component detecting.
Response Actions
Exercise Response Actions. This included the time to notify response partners, time to decide on
appropriate operational responses, and time to make public notification of water use restrictions. The
time to notify response partners varied with the sequence of circumstances presented through the various
exercise scenarios, but generally was consistent from FSE 2 to FSE 3, with some slight improvements
with key partners including CFD, CHD, DPIC and OEPA.
The average time for identifying appropriate operational response actions during specific threat levels in
simulated contamination incidents varied, but these times were influenced by the circumstances presented
by the scenario. Operational responses were initially driven by what actions the utility could implement
quickly to isolate or slow contamination without impacting service to customers. As the incident
progressed, investigation evidence was subsequently used to revise those response actions as necessary.
The time to develop and implement public notification was consistent throughout the exercises, with an
average time of 169 minutes from direction to prepare and availability to release. Given the variability of
exercise scopes and the accompanying revision of the CCP, it was not possible to make statistical
inferences concerning the performance.
Simulation Study Response Actions. The simulation study contamination scenario results were
analyzed to determine the time to public health response, time to the first operational response, and the
time to public notification. The data was evaluated based on the type of contaminant injected, the number
of components detecting and the number of alerts received.
Simulation Study Investigative Actions. The simulation study results for time to public health response
showed a strong correlation to the contaminant type and to the number of components detecting metrics.
A comparison of the toxic chemical to the biological agent contaminant scenarios indicated that toxic
chemical contaminants resulted in much quicker implementation of public health response and a much
narrower time span. As indicated earlier, the difference in time to public health response was most likely
due to differences between contaminant types with respect to the time for the rapid onset of symptoms,
public health community awareness, and variations in how long it took to identify the contaminant.
The simulation study results for time to the first operational response were very similar for contaminant
type, number of components detecting and number of alerts metrics. All metric results indicated very
quick first operational response implementation times and a very narrow time span.
The simulation study results for time to public notification showed a correlation for the contaminant type,
number of components detecting, and number of alerts metric results. A comparison of the toxic
chemical and biological agent contaminant scenario results showed that toxic chemical contaminants
resulted in much quicker public notification over a very narrow time span. As indicated earlier, this was
most likely due to differences between the contaminant types' effect on taste, odor or appearance of the
drinking water and whether the contaminant triggered customer complaints or progression of symptoms
leading to healthcare seeking behavior. The results for the number of components detecting showed a
slight advantage for three components detecting followed by two components detecting with a slight
56
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
advantage over one component detecting. Finally, the number of alert results impact on the time to public
notification showed a significant reduction in the public notification time at six alerts and above.
8.1.2 Integration of Response Partner Network
The inclusion of the response partner network as a design element of the CM component was intended to
streamline the overall response process. Although there was no empirical data to describe how this
design element contributed to timeliness of response by itself, the design elements evaluated as a part of
this design objective did reveal several significant observations including the effective integration of 10
response partner agencies into the CM component and progressive improvement with understanding
response partner roles and responsibilities through training.
8.1.3 Communication Equipment
The integration of the 800 MHz radios and SharePoint site into the utility response activities enhanced
their ability to communicate information between field personnel, response partner agencies and the ICS.
Data and feedback from the exercises indicated that the use of the communication equipment allowed
faster and more complete investigation of contamination incidents. By the end of the pilot evaluation
period, utility personnel routinely used the communication equipment for contamination incident
investigations.
8.2 Design Objective: Sustainability
Sustainability of the CM component was measured by labor hours for both CM implementation and
O&M and dual-use benefits. Overall, the development and implementation of a comprehensive CM
component required a considerable commitment of time and resources, from both a development and
maintenance perspective.
Developing the Cincinnati Pilot Consequence Management Plan and conducting the exercises were the
most significant tasks for the implementation of this component. Both required work with utility
personnel as well response partner agencies. Implementation costs approximately amounted to $800,000
though equipment costs were minimal (approximately $40,000).
Labor hours for O&M were steady at approximately 340 hours per month through most of the evaluation
period, with the exception of the reporting periods surrounding FSEs. Those exercise required substantial
planning and participation from numerous utility personnel and response partner agencies.
Dual-use benefits and compliance were evaluated through documentation of qualitative data during drills
and exercises and during forums with the utility including lessons learned workshops. The use of CM
procedures and equipment during a major windstorm in Cincinnati demonstrated a dual-use benefit to
Cincinnati pilot personnel. Compliance was demonstrated through 100% utility participation in full scale
exercises which required substantial effort, but were beneficial to the Cincinnati pilot as reported by
personnel who indicated that they were able to better understand CM procedures through response to
simulated water contamination incidents.
57
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Section 9.0: References
U.S. Environmental Protection Agency. 2005. WaterSentinel System Architecture, Draft for Science
Advisory Board Review.
U.S. Environmental Protection Agency. 2008. Water Security Initiative: Cincinnati Pilot Post-
Implementation System Status, EPA 817-R-08-004, September, 2008.
U.S. Environmental Protection Agency. 2013a. Water Security Initiative: Evaluation of the Sampling and
Analysis Component of the Cincinnati Contamination Warning System Pilot, EPA 817-R-13-009.
U.S. Environmental Protection Agency. 2013b. Water Security Initiative: Evaluation of the Cincinnati
Contamination Warning System Pilot, EPA 817-R-13-003
U.S. Environmental Protection Agency. 2014. Water Security Initiative: Comprehensive Evaluation of
the Cincinnati Contamination Warning System Pilot EPA 817-R-14-001
63
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Section 10.0: Abbreviations
The list below includes acronyms approved for use in the CM component evaluation. Acronyms are
defined at first use in the document.
AAR
AWWA
CCP
CCS
CERCLA
CFD
CHD
CM
CMP
CPD
CSFDG
CWS
DHS
DPIC
ESM
EPA
FBI
FEMA
FSE
GCWW
HazMat
HCEMA
HCPHD
HI/HB
1C
ICS
IP
ISAC
MSD
MSDGC
NIMS
O&M
ODH
OEPA
PHS
PIO
PN
R&R
RFT
SC
After Action Report
American Water Works Association
Crisis Communication Plan
Customer Complaint Surveillance
Comprehensive Environmental Response, Compensation, and Liability Act
Cincinnati Fire Department
Cincinnati Health Department
Consequence Management
Consequence Management Plan
Cincinnati Police Department
Confirmatory Sampling Field Decision Guide
Contamination Warning System
Department of Homeland Security
Drug and Poison Information Center
Enhanced Security Monitoring
U.S. Environmental Protection Agency
Federal Bureau of Investigations
Federal Emergency Management Agency
Full Scale Exercise
Greater Cincinnati Water Works
CFD Hazardous Materials Team
Hamilton County Emergency Management Agency
Hamilton County Public Health Department
Health Impacts and Human Behavior
Incident Commander
Incident Command System
Improvement Plan
(Water) Information Sharing and Analysis Center
Cincinnati Metropolitan Sewer District
Metropolitan Sewer District of Greater Cincinnati
National Incident Management System
Operations & Maintenance
Ohio Department of Health
Ohio Environmental Protection Agency
Public Health Surveillance
Public Information Officer
Public Notification
Remediation and Recovery
Rapid Field Test
Site Characterization
64
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
SCT Site Characterization Team
SIMCELL Simulation Cell
TOC Total Organic Carbon
TLD Threat Level Determination
WQ&T Water Quality & Treatment
WQM Water Quality Monitoring
WSI Water Security Initiative
WUERM Water Utility Emergency Response Manager
65
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Section 11.0: Glossary
Alert. Information from a monitoring and surveillance component indicating an anomaly in the system,
which warrants further investigation to determine if the alert is valid.
Alert Investigation. A systematic process, documented in a standard operating procedure, for
determining whether or not an alert is valid, and identifying the cause of the alert. If an alert cause cannot
be identified, contamination is possible.
Anomaly. Deviations from an established baseline. For example, a water quality anomaly is a deviation
from typical water quality patterns observed over an extended period.
Baseline. Normal conditions that result from typical system operation. The baseline includes predictable
fluctuations in measured parameters that result from known changes to the system. For example, a water
quality baseline includes the effects of draining and filling tanks, pump operation and seasonal changes in
water demand, all of which may alter water quality in a somewhat predictable fashion.
Benefit. An outcome associated with the implementation and operation of a contamination warning
system that promotes the welfare of the utility and the community it serves. Benefits are classified as
either primary or dual-use.
Benefit-cost analysis. An evaluation of the benefits and costs of a project or program, such as a
contamination warning system, to assess whether the investment is justifiable considering both financial
and qualitative factors.
Biotoxins. Toxic chemicals derived from biological materials that pose an acute risk to public health at
relatively low concentrations.
Box-and-whisker plot. A graphical representation of nonparametric statistics for a dataset. The bottom
and top whiskers represent the 10th and 90th percentiles of the ranked data, respectively. The bottom and
top of the box represent the 25th and 75th percentiles of the ranked data, respectively. The line inside the
box represents the 50th percentile, or median of the ranked data. Note that some data sets may have the
same values for the percentiles presented in box-and-whisker plots, in which case not all lines will be
visible.
Component response procedures. Documentation of roles and responsibilities, process flows and
procedural activities for a specified component of the contamination warning system, including the
investigation of alerts from the component. Standard operating procedures for each monitoring and
surveillance component are integrated into an operational strategy for the contamination warning system.
Confirmed. In the context of the threat level determination process, contamination is Confirmed when
the analysis of all available information from the contamination warning system has provided definitive,
or nearly definitive, evidence of the presence of a specific contaminant or class of contaminant in the
distribution system. While positive results from laboratory analysis of a sample collected from the
distribution system can be a basis for confirming contamination, a preponderance of evidence without the
benefit of laboratory results can lead to this same determination.
Consequence management. Actions taken to plan for and respond to possible contamination incidents.
This includes the threat level determination process, which uses information from all monitoring and
surveillance components as well as sampling and analysis to determine if contamination is credible or
confirmed. Response actions, including operational changes, public notification, and public health
66
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
response, are implemented to minimize public health and economic impacts and ultimately return the
utility to normal operations.
Consequence management plan. Documentation that provides a decision-making framework to guide
investigative and response activities implemented in response to a possible contamination incident.
Contamination incident. The introduction of a contaminant in the distribution system with the potential
to cause harm to the utility or the community served by the utility. A contamination incident may be
intentional or accidental.
Contamination scenario. Within the context of the simulation study, parameters that define a specific
contamination incident, including: injection location, injection rate, injection duration, time the injection
is initiated, and the contaminant that is injected.
Contamination warning system. An integrated system of monitoring and surveillance components
designed to detect contamination in a drinking water distribution system. The system relies on integration
of information from these monitoring and surveillance activities along with timely investigative and
response actions during consequence management to minimize the consequences of a contamination
incident.
Costs, implementation. Installed cost of equipment, IT components and subsystems necessary to deploy
an operational system. Implementation costs include labor and other expenditures (equipment, supplies,
and purchased services).
Cost, life cycle. The total cost of a system, component, or equipment over its useful or practical life.
Life cycle cost includes the cost of implementation, operation & maintenance, and renewal &
replacement.
Costs, operation & maintenance. Expenses incurred to sustain operation of a system at an acceptable
level of performance. Operational and maintenance costs are reported on an annual basis, and include
labor and other expenditures (e.g., supplies and purchased services).
Costs, renewal & replacement. Costs associated with refurbishing or replacing major pieces of
equipment (e.g., water quality sensors, laboratory instruments, IT hardware) that reach the end of their
useful life before the end of the contamination warning system lifecycle.
Coverage, contaminant. Specific contaminants that can potentially be detected by each monitoring and
surveillance component, including sampling & analysis, of a contamination warning system.
Coverage, spatial. The areas within the distribution system that are monitored by, or protected by each
monitoring and surveillance component of a contamination warning system.
Credible. In the context of the threat level determination process, a water contamination threat is
characterized as Credible if information collected during the investigation of possible contamination
corroborates information from the validated contamination warning system alert.
Data completeness. The amount of data that can be used to support system or component operations,
expressed as a percentage of all data generated by the system or component. Data may be lost due to QC
failures, data transmission errors, and faulty equipment among other causes.
Distribution system model. A mathematical representation of a drinking water distribution system,
including pipes, junctions, valves, pumps, tanks, reservoirs, etc. The model characterizes flow and
67
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
pressure of water through the system. Distribution system models may include a water quality model that
can predict the fate and transport of a material throughout the distribution system.
Dual-use benefit. A positive application of a piece of equipment, procedure or capability that was
deployed as part of the contamination warning system in the normal operations of the utility.
Ensemble. The comprehensive set of contamination scenarios evaluated during the simulation study.
Event detection system. A system designed specifically to detect anomalies from the various monitoring
and surveillance components of a contamination warning system. An event detection system may take a
variety of forms, ranging from a complex set of computer algorithms to a simple set of heuristics that are
manually implemented.
Evaluation period. The period from January 16, 2008 to June 15, 2010 when data was actively collected
for the evaluation of the Cincinnati contamination warning system pilot.
Field results. Field results include information collected from Site Characterization activities including
the site hazard assessment, field safety screening, water quality testing and rapid field tests. This does not
include the results of the laboratory analysis conducted on samples collected at the end of the site
characterization process.
Hydraulic connectivity. Points or areas within a distribution system that are on a common flow path.
Incident Commander. In the Incident Command System, the individual responsible for all aspects of an
emergency response, including quickly developing incident objectives, managing incident operations and
allocating resources.
Incident timeline. The cumulative time from the beginning of a contamination incident until response
actions are effectively implemented. Elements of the incident timeline include: time for detection, time
for alert validation, time for threat level determination, and time to implement response actions.
Injection location. The specific node in the distribution system model where the bulk contaminant is
injected into the distribution system for a given scenario within the simulation study.
Invalid alert. An alert from a monitoring and surveillance component that is not due to an anomaly and
is not associated with an incident or condition of interest to the utility.
Metric. A standard or statistic for measuring or quantifying an attribute of the contamination warning
system or its components.
Model. A mathematical representation of a physical system.
Model parameters. Fixed values in a model that define important aspects of the physical system.
Module. A sub-component of a model that typically represents a specific function of the real-world
system being modeled.
Monitoring & surveillance component. Element of a contamination warning system used to detect
unusual water quality conditions, potentially including contamination incidents. The four monitoring &
surveillance components of a contamination warning system include: 1) online water quality monitoring,
2) enhanced security monitoring, 3) customer complaint surveillance and 4) public health surveillance.
68
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Net present value. The difference between the present value of benefits and costs, normalized to a
common year.
Node. A mathematical representation of a junction between two or more distribution system pipes, or a
terminal point in a pipe in a water distribution system model. Water may be withdrawn from the system
at nodes, representing a portion of the system demand.
Nuisance chemicals. Chemical contaminants with a relatively low toxicity, which thus generally do not
pose an immediate threat to public health. However, contamination with these chemicals can make the
drinking water supply unusable.
Operational strategy. Documentation that integrates the standard operating procedures that guide
routine operation of the monitoring and surveillance components of a drinking water contamination
warning system. The operational strategy establishes specific roles and responsibilities for the component
and procedures for investigating alerts.
Optimization phase. Period in the contamination warning system deployment timeline between the
completion of system installation and real-time monitoring. During this phase the system is operational,
but not expected to produce actionable alerts. Instead, this phase provides an opportunity to learn the
system and optimize performance (e.g., fix or replace malfunctioning equipment, eliminate software bugs,
test procedures and reduce occurrence of invalid alerts).
Pathogens. Microorganisms that cause infections and subsequent illness and mortality in the exposed
population.
Pito zone. An area of the Greater Cincinnati Water Works distribution system in which the pressure is
fairly constant. There are 94 pito zones in the Greater Cincinnati Waterworks distribution system model.
Possible. In the context of the threat level determination process, a water contamination threat is
characterized as Possible if the cause of a validated contamination warning system alert is unknown.
Primary benefits. Benefits that are derived from the reduction in consequences associated with a
contamination incident due to deployment of a contamination warning system.
Priority contaminant. A contaminant that has been identified by the EPA for monitoring under the
Water Security Initiative. Priority contaminants may be initially detected through one of the monitoring
and surveillance components and confirmed through laboratory analysis of samples collected during the
investigation of a possible contamination incident.
Process flow. The central element of a standard operating procedure that guides routine monitoring and
surveillance activities in a contamination warning system. The process flow is represented in a flow
diagram that shows the step-by-step process for investigation alerts, identifying the potential cause of the
alert, and determining whether contamination is possible.
Public health incident. An occurrence of disease, illness or injury within a population that is a deviation
from the disease baseline in the population.
Public health response. Actions taken by public health agencies and their partners to mitigate the
adverse effects of a public health incident, regardless of the cause of the incident. Potential response
actions include: administering prophylaxis, mobilizing additional healthcare resources, providing
treatment guidelines to healthcare providers and providing information to the public.
69
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Real-time monitoring phase. Period in the contamination warning system deployment timeline
following the optimization phase. During this phase, the system is fully operational and is producing
actionable alerts. Utility staff and partners now respond to alerts in real-time and in full accordance with
standard operating procedures documented in the operational strategy. Optimization of the system still
occurs as part of a continuous improvement process, however the system is no longer considered to be
developmental.
Remediation and recovery. The stage of a contamination incident following confirmation of the
incident, which involves the implementation of characterization, remediation, and return to service with
the goal of restoring the drinking water system and returning to operational service.
Risk communication. Communication activities within an organization and with external parties that
address the impact and outcome of an incident.
Routine operation. The day-to-day monitoring and surveillance activities of the contamination warning
system that are guided by the operational strategy. To the extent possible, routine operation of the
contamination warning system is integrated into the routine operations of the drinking water utility.
Salvage value. Estimated value of assets at the end of the useful life of the system.
Simulation study. A study designed to systematically characterize the detection capabilities of the
Cincinnati drinking water contamination warning system. In this study, a computer model of the
contamination warning system is challenged with an ensemble of 2,023 simulated contamination
scenarios. The output from these simulations provides estimates of the consequences resulting from each
contamination scenario, including fatalities, illnesses, and extent of distribution system contamination.
Consequences are estimated under two cases, with and without the contamination warning system in
operation. The difference provides an estimate of the reduction in consequences.
Simulation study architecture. The interdependent models of each component of the Cincinnati
contamination warning system, integrated into a platform that allows for execution of the simulations.
The individual models describe the data processing, decision logic, and sequencing steps that represent
the activities executed by the corresponding component.
Site characterization. The process of collecting information from an investigation site to support the
investigation of a contamination incident during consequence management.
Threat level. The results of the threat level determination process, indicating whether contamination is
Possible, Credible, or Confirmed.
Threat level determination process. A systematic process in which all available and relevant
information available from a contamination warning system is evaluated to determine whether the threat
level is Possible, Credible, or Confirmed. This is an iterative process in which the threat level is revised
as additional information becomes available. The conclusions from the threat evaluation process are
considered during consequence management when making response decisions.
Threat level index. In the Cincinnati contamination warning system model, a quantitative indicator of
the threat level associated with a specific contamination scenario. The threat level index is calculated by
the Cincinnati contamination warning system model by summing the confidence indices from all
component models. A value greater than or equal to 1.0 represents possible contamination, greater than
or equal to 2.0 represents credible contamination, and greater than or equal to 3.0 represents confirmed
contamination.
70
-------
Water Security Initiative: Evaluation of the Consequence Management Component
of the Cincinnati Contamination Warning System Pilot
Time for Confirmed determination. A portion of the incident timeline that begins with the
determination that contamination is Credible and ends with contamination either being Confirmed or
ruled out. This includes the time required to perform lab analyses, collect additional information, and
analyze the collective information to determine if the preponderance of evidence confirms the incident.
Time for contaminant detection. A portion of the incident timeline that begins with the start of
contamination injection and ends with the generation and recognition of an alert. The time for
contaminant detection may be subdivided for specific components to capture important elements of this
portion of the incident timeline (e.g., sample processing time, data transmission time, and event detection
time).
Time for Credible determination. A portion of the incident timeline that begins with the recognition of
a Possible contamination incident and ends with a determination regarding whether contamination is
Credible. This includes the time required to perform multi-component investigation and data integration,
implement field investigations (such as site characterization and sampling), and collect additional
information to support the investigation.
Time for initial alert validation. A portion of the incident timeline that begins with the recognition of
an alert and ends with a determination regarding whether or not contamination is Possible.
Toxic chemicals. Highly toxic chemicals that pose an acute risk to public health at relatively low
concentrations.
Valid alert. Alerts due to water contamination, system events (i.e., work in the distribution system for
CCS or WQM) or public health incidents (for PHS)
Water Utility Emergency Response Manager. A role within the Cincinnati contamination warning
system filled by a mid-level manager from the drinking water utility. Responsibilities of this position
include: receiving notification of validated alerts, verifying that a valid alert indicates Possible
contamination, coordinating the threat level determination process, integrating information across the
different monitoring and surveillance components, and activating the consequence management plan. In
the early stages of responding to Possible contamination the Water Utility Emergency Response Manager
may serve as Incident Commander.
71
------- |