&EPA United States Environmental Protection Agency Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Monitoring and Surveillance Water Quality Monitoring Enhanced Security Monitoring Customer Complaint Surveillance ublic Health Surveillance Possible Contamination Consequence Management Sampling and Analys Response Office of Water (MC-140) EPA-817-R-14-001F April 2014 ------- ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot The Water Security Division of the Office of Ground Water and Drinking Water has reviewed and approved this document for publication. This document does not impose legally binding requirements on any party. The findings in this report are intended solely to recommend or suggest and do not imply any requirements. Neither the U.S. Government nor any of its employees, contractors or their employees make any warranty, expressed or implied, or assumes any legal liability or responsibility for any third party's use of or the results of such use of any information, apparatus, product or process discussed in this report, or represents that its use by such party would not infringe on privately owned rights. Mention of trade names or commercial products does not constitute endorsement or recommendation for use. Questions concerning this document should be addressed to: Jeff Pencil U.S. EPA Water Security Division 1200 Pennsylvania Ave, NW Mail Code 4608T Washington, DC 20460 (202)564-0818 or Steve Allgeier U.S. EPA Water Security Division 26 West Martin Luther King Drive Mail Code 140 Cincinnati, OH 45268 (513)569-7131 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Acknowledgements The Water Security Division of the Office of Ground Water and Drinking Water would like to recognize the following individuals and organizations for their assistance, contributions, and review during the development of this document. • Yeongho Lee, Greater Cincinnati Water Works • Jeff Swertfeger, Greater Cincinnati Water Works • Mike Tyree, Greater Cincinnati Water Works ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Executive Summary The goal of the Water Security Initiative (WSI) is to design and demonstrate an effective multi- component warning system for timely detection and response to drinking water contamination threats and incidents. A contamination warning system (CWS) integrates information from multiple monitoring and surveillance components to alert the water utility to possible contamination and uses a consequence management plan (CMP) to guide response actions. System design objectives for an effective CWS are: spatial coverage, contaminant coverage, alert occurrence, timeliness of detection and response, operational reliability, and sustainability. Metrics for the consequence management (CM) component were defined relative to the system metrics common to all components in the CWS, but the component metric definitions provide an additional level of detail relevant to the CM component. Evaluation techniques used to quantitatively or qualitatively evaluate each of the metrics include analysis of empirical data from routine operations, drills and exercises, modeling and simulations, forums, and an analysis of lifecycle costs (see Section 3.0). This report describes the evaluation of data collected from the CM component from the period of January 2008 - June 2010. The major outputs from the evaluation of the Cincinnati pilot include: 1. Cincinnati Pilot System Status, which describes the post-implementation status of the Cincinnati pilot following the installation of all monitoring and surveillance components. 2. Component Evaluations, which include analysis of performance metrics for each component of the Cincinnati pilot. 3. System-Level Performance Summary, which integrates the results of component evaluations, the simulation study and results of a benefit-cost analysis. The reports that present the results from the evaluation of the system and each of its six components are available in an Adobe portfolio, Water Security Initiative: Comprehensive Evaluation of the Cincinnati Contamination Warning System Pilot (USEPA 2014). Consequence Management Component Design The CM component was designed to provide guidance and equipment that would facilitate the Cincinnati pilot with the management of possible drinking water contamination incidents as detected by one or more of the CWS monitoring and surveillance components. Thus, the CM component was comprised primarily of incident response plans and communication equipment that would minimize response times, and therefore, minimize deleterious effects from contamination. The three CM design elements included: 1) incident response plans; 2) response partner network; and 3) communication equipment (see Section 2.0). Of the six design objectives identified for the CWS (contaminant coverage, spatial coverage, timeliness of detection and response, operational reliability, alert occurrence and sustainability), only two (timeliness of detection and response and sustainability) are directly applicable to the CM component. A summary of the results used to evaluate whether the CM component met each of the design objectives relevant to this component is provided below. Methodology ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Several methods were used to evaluate CM performance. Data was tracked over time to illustrate the change in performance as the component evolved during the evaluation period. Statistical methods were also used to summarize large volumes of data collected over either the entire or various segments of the evaluation period. Data was also evaluated and summarized for each reporting period over the evaluation period. In this evaluation, the term reporting period is used to refer to one month of data that spans from the 16th of the indicated month to the 15th of the following month. Thus, the January 2008 reporting period refers to the data collected between January 16th 2008 and February 15th 2008. Additionally, five drills and three full-scale exercises designed around mock contamination incidents were used to practice and evaluate the full range of procedures, from initial detection through response. Because there were no contamination incidents during the evaluation period, there is no empirical data to fully evaluate the detection capabilities of the component. To fill this gap, a computer model of the Cincinnati CWS was developed and challenged with a large ensemble of simulated contamination incidents in a simulation study. An ensemble of 2,015 contamination scenarios representing a broad range of contaminants and injection locations throughout the distribution system was used to evaluate the effectiveness of the CWS in minimizing public health and utility infrastructure consequences. The simulations were also used for a benefit-cost analysis, which compares the monetized value of costs and benefits and calculates the net present value of the CWS. Costs include implementation costs and routine operation and maintenance labor and expenses, which were assumed over a 20 year lifecycle of the CWS. Benefits included reduction in consequences (illness, fatalities and infrastructure damage) and dual-use benefits from routine operations. Design Objective: Timeliness of Response For CM, timeliness of response refers to the time it took the Cincinnati pilot to verify, characterize, and respond to a contamination incident as detected by one or more of the CWS monitoring and surveillance components. Factors that impact this objective include: time to notify response partners; time for deploying field personnel and equipment; time for assessing hazard levels; time for collecting and screening drinking water samples; time to identify and implement operational responses; time to identify and implement public health response; time to determine threat levels; time to implement public notification, and time to restore the system to normal operations (see Sections 4.0, 5.0 and 6.0). Site characterization (SC) activities, which describe the field response, investigation and sampling procedures, are described in Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Pilot (USEPA, 2013a). One key metric for CM, which is implemented after an incident has been determined to be Possible, was the time for threat level determination. This included the amount of time required for investigative actions leading to both Credible and Confirmed determination (see Section 4.0). For a simulated contamination incident exercise, a hierarchy of investigation information types evolved, which seemed to accelerate the progression through threat level determinations. Two primary examples from the exercises included: 1. Number of alerts, system connectivity, positive rapid filed tests and signs of intrusion accelerated the declaration of Credible contamination incident and 2. Signs of intrusion and health impacts accelerated the declaration of Confirmed (assumed contamination) incidents and did not necessarily depend on positive laboratory analysis. The simulation study was utilized to determine the impact of various metrics on threat level determination. The analyses resulted in a better understanding of the role that the type of contaminant, ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot the number of components detecting the incident, and the number of alerts play in threat level determination. Three primary examples of the findings include: 1. Toxic chemical contaminants are identified much quicker than biological agents, resulting in earlier declaration of Credible and Confirmed determination; 2. The number of components detecting the contaminant resulted in an accelerated declaration of Credible and Confirmed determination; and 3. The number of alerts reduced the amount of time required for Credible determination declaration. A second key metric was the average time for identifying appropriate response actions (e.g. operational responses and public notification) during a simulated contamination incident. The specific threat levels in the simulated contamination exercises varied, but these times were clearly influenced by the circumstances presented by the exercise scenario. Operational responses were initially driven by what actions the utility could implement quickly to isolate or slow contamination without impacting service to customers. As the incident progressed, new investigation evidence was used to revise response actions or implement new response actions, as necessary. The time to develop and implement public notification was consistent throughout the exercises, with an average time of 169 minutes from direction to prepare to release. Given the variability of exercise scopes, and the accompanying revision of the crisis communication plan, it was not possible to make statistical inferences concerning the improvement. Figure ES-1 is a timeline progression (in hours) showing major response action milestones for the Cincinnati Full Scale Exercise (FSE) 2, the largest of the three full scale exercises that were conducted during the pilot. Field-verified timeline information from drills and full scale exercises was used to build the CM portion of the simulation model. For more information on this topic, see the relevant subsections regarding Timeliness of Response for each CM component. 00:00 02:10 02:30 05:00 WQM WQM Alerts Alert #1 #2 02:00 Operations Checked Director Notified r i CCS Alert Expanded 05:30 Isolation Site 08:40 Public 09:30 Notification Crisis Response Characterization Response Communication Action Activities 02:40 WQM Alerts #4 03:00 Initial Isolation Response Action r^ •* V 1 Action Press Conference 06:30 V ~\ PHS Alert r ^ r i 10:30 Flushing Response Action r "^ r .,,..,,,,,, 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00 00:00 11:30 Figure ES-1. Timeline Progression for Response Actions During Full Scale Exercise 2 Design Objective: Sustainability ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Sustainability is a key objective in the design of a CWS and each of its components, which for the purpose of this evaluation is defined in terms of the cost-benefit trade-off (see Section 7.0). Costs are estimated over the life cycle of the system to provide an estimate of the total cost of ownership and include the implementation costs, enhancement costs, operation and maintenance (O&M) costs, renewal and replacement costs, and the salvage value. The benefits derived from the system are defined in terms of primary and dual-use benefits. Metrics that were evaluated under this design objective include: costs, benefits, and compliance. The costs used in the calculation of lifecycle costs for the CM component are presented in Table ES-1. These costs were tracked as empirical data during the design and implementation phase of project design, and were analyzed through a benefit-cost analysis of the Cincinnati pilot (see Section 7.0). It is important to note that the Cincinnati CWS was a pilot research project, and as such incurred higher costs than would be expected for a typical large utility installation. ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Table ES-1. Cost Elements used in the Calculation of Lifecycle Cost Parameter Implementation Costs Annual O&M Costs Renewal and Replacement Costs Salvage Value1 Value $1,430,627 $33,948 $22,624 - Calculated using major pieces of equipment. To calculate the total lifecycle cost of the CM component, all costs and monetized benefits were adjusted to 2007 dollars using the change in the Consumer Price Index between 2007 and the year that the cost or benefit was realized. Subsequently, the implementation costs, renewal and replacement costs, and annual O&M costs were combined to determine the total lifecycle cost: CM Total Lifecvcle Cost: $2,000,828* *Actual costs from the above table were adjusted to 2007 dollars to calculate the total 20 year life cycle cost. A similar CM component implementation at another utility should be less expensive when compared to the Cincinnati pilot as it could benefit from lessons learned and would not incur research-related costs. Benefits were measured by identifying applications of the CWS to any other purpose other than detection of intentional and unintentional drinking water contamination incidents. Information was collected from forums, lessons learned workshops and interviews. Key benefits that were identified included: stronger interagency relationships with response partners, strengthened incident command structure and increased preparedness of utility management and staff to respond to "all-hazards" type incidents. Compliance with protocols and procedures necessary to operate and maintain the CWS is the ultimate measure of the sustainability of the CWS, including the CM component. Compliance was evaluated through documentation of qualitative data during drills and exercises and during forums with the pilot, including lessons learned workshops. This was demonstrated through 100% utility participation in full scale exercises where utility personnel were able to obtain a better understanding of CM procedures through response to simulated water contamination incidents. For more information on this topic, see the relevant subsections regarding Sustainability for each CM component. ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Table of Contents LIST OF FIGURES 10 LIST OF TABLES 11 SECTION 1.0: INTRODUCTION 1 1.1 CWS DESIGN OBJECTIVES 1 1.2 ROLE OF CM IN THE CINCINNATI CWS 2 1.3 OBJECTIVES 3 1.4 DOCUMENT ORGANIZATION 3 SECTION 2.0: OVERVIEW OF THE CM COMPONENT 4 2.1 INCIDENT RESPONSE PLANS 4 2.2 RESPONSE PARTNER NETWORK 5 2.3 COMMUNICATION EQUIPMENT 6 2.4 CM COMPONENT ROLES AND RESPONSIBILITIES 6 2.5 SUMMARY OF SIGNIFICANT CM COMPONENT MODIFICATIONS 7 2.6 TIMELINE OF CM DEVELOPMENT PHASES AND EVALUATION -RELATED ACTIVITIES 9 SECTION 3.0: METHODOLOGY 10 3.1 TRAINING AND EXERCISE PROGRAM 10 3.1.1 Discussion-Based Exercises 10 3.1.2 Operations-Based Exercises 12 3.2 SIMULATION STUDY 14 3.3 FEEDBACK FORUMS 17 3.4 ANALYSIS OF LIFECYCLE COSTS 17 SECTION 4.0: PERFORMANCE OF INCIDENT RESPONSE PLANS 19 4.1 DESIGN OBJECTIVE: TIMELINESS OF RESPONSE 19 4.1.1 Credibility Determination 20 4.1.2 Response Actions 29 4.1.3 Remediation and Recovery 40 4.1.4 Summary 41 SECTION 5.0: PERFORMANCE OF INTEGRATION OF THE RESPONSE PARTNER NETWORK 44 5.1 DESIGN OBJECTIVE: TIMELINESS OF RESPONSE 44 5.1.1 Understanding Response Partner Roles and Responsibilities 44 5.1.2 Integration of Response Partners 45 5.1.3 Nature of Response Partner Support 46 5.1.4 Summary 47 SECTION 6.0: PERFORMANCE OF COMMUNICATION EQUIPMENT 48 6.1 DESIGN OBJECTIVE: TIMELINESS OF RESPONSE 48 6.1.1 Use of 800 MHz Radios 48 6.1.2 Summary 48 SECTION 7.0: SUSTAINABILITY 49 7.1 COSTS 49 7.2 BENEFITS 52 7.3 SUMMARY 53 SECTION 8.0: SUMMARY AND CONCLUSIONS 55 8.1 DESIGN OBJECTIVE: TIMELINESS OF RESPONSE 55 8.1.1 Incident Response Plans 55 8.1.2 Integration of Response Partner Network 57 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot 8.1.3 Communication Equipment 57 8.2 DESIGN OBJECTIVE: SUSTAINABILITY 57 SECTION 9.0: REFERENCES 63 SECTION 10.0: ABBREVIATIONS 64 SECTION 11.0: GLOSSARY 66 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot List of Figures FIGURE 2-1. TIMELINE OF CM COMPONENT ACTIVITIES 9 FIGURE 4-1. GENERIC CM TIMELINE 20 FIGURE 4-2. COMPARISON OF FSE TIME FROM POSSIBLE TO CREDIBLE DETERMINATION 21 FIGURE 4-3. TIME FROM POSSIBLE TO CREDIBLE DETERMINATION BY CONTAMINANT TYPE 22 FIGURE 4-4. TIME FROM POSSIBLE TO CREDIBLE DETERMINATION BY NUMBER OF COMPONENTS DETECTING 23 FIGURE 4-5. TIME FROM POSSIBLE TO CREDIBLE DETERMINATION BY NUMBER OF ALERTS RECEIVED 24 FIGURE 4-6. COMPARISON OF FSE TIME FROM POSSIBLE TO CONFIRMED DETERMINATION 26 FIGURE 4-7. TIME FROM POSSIBLE TO CONFIRMED DETERMINATION BY CONTAMINANT TYPE 27 FIGURE 4-8. TIME FROM POSSIBLE TO CONFIRMED DETERMINATION BY NUMBER OF COMPONENTS DETECTING 28 FIGURE 4-9. TIME FROM POSSIBLE TO CONFIRMED DETERMINATION BY NUMBER OF ALERTS RECEIVED 29 FIGURE 4-10. TIME FROM POSSIBLE CONTAMINATION TO PUBLIC HEALTH RESPONSE BY CONTAMINANT TYPE 31 FIGURE 4-11. TIME FROM POSSIBLE TO PUBLIC HEALTH RESPONSE BY NUMBER OF COMPONENTS DETECTING 32 FIGURE 4-12. TIME FROM POSSIBLE TO PUBLIC HEALTH RESPONSE BY NUMBER OF ALERTS RECEIVED 32 FIGURE 4-13. TIME FROM POSSIBLE TO OPERATIONAL RESPONSE BY CONTAMINANT TYPE 35 FIGURE 4-14. TIME FROM POSSIBLE TO OPERATIONAL RESPONSE BY NUMBER OF COMPONENTS DETECTING 36 FIGURE 4-15. TIME FROM POSSIBLE TO OPERATIONAL RESPONSE BY NUMBER OF ALERTS RECEIVED 37 FIGURE 4-16. TIME FROM POSSIBLE TO PUBLIC NOTIFICATION BY CONTAMINANT TYPE 38 FIGURE 4-17. TIME FROM POSSIBLE TO PUBLIC NOTIFICATION BY NUMBER OF COMPONENTS DETECTING 39 FIGURE 4-18. TIME FROM POSSIBLE TO PUBLIC NOTIFICATION BY NUMBER OF ALERTS RECEIVED 40 FIGURE 7-1. O&M LABOR HOURS PER REPORTING PERIOD 51 10 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot List of Tables TABLE 2-1. CONSEQUENCE MANAGEMENT DESIGN ELEMENTS 4 TABLE 2-2. CM ROLES AND RESPONSIBILITIES - CINCINNATI LOCAL RESPONSE PARTNER AGENCIES 5 TABLE 2-3. CM ROLES AND RESPONSIBILITIES - UTILITY ICS PERSONNEL 6 TABLE 2-4. SIGNIFICANT CM COMPONENT MODIFICATIONS 7 TABLE 3-1. DISCUSSION-BASED EXERCISES 10 TABLE 3-2. CM OPERATIONS-BASED EXERCISES 12 TABLE 4-1. SITE CHARACTERIZATION METRIC CATEGORIES AND REPORT LOCATION 19 TABLE 4-2. EXERCISE TIMES FOR CREDIBLE DETERMINATION 21 TABLE 4-3. NUMBER OF ALERTS RECEIVED - TIMES FOR CREDIBLE DETERMINATION 24 TABLE 4-4. EXERCISE TIMES FROM CREDIBLE TO CONFIRMED DETERMINATION 26 TABLE 4-5. NUMBER OF ALERTS RECEIVED-TIMES FOR CONFIRMED DETERMINATION 29 TABLE 4-6. TIME TO NOTIFY RESPONSE PARTNERS 30 TABLE 4-7. NUMBER OF ALERTS RECEIVED - TIME TO PUBLIC HEALTH RESPONSE 33 TABLE 4-8. TIME TO IDENTIFY OPERATIONAL RESPONSE ACTIONS 34 TABLE 4-9. NUMBER OF ALERTS RECEIVED - TIME TO OPERATIONAL RESPONSE 37 TABLE 4-10. TIME TO PREPARE AND ISSUE PUBLIC NOTIFICATION 38 TABLE 4-11. NUMBER OF ALERTS RECEIVED - TIME TO PUBLIC NOTIFICATION 40 TABLE 5-1. RESPONSE PARTNER RECOMMENDATIONS BY EXERCISE 45 TABLE 5-2. RESPONSE PARTNER EXERCISE PARTICIPATION 45 TABLE 5-3. NATURE OF RESPONSE PARTNER SUPPORT 47 TABLE 7-1. COST ELEMENTS USED IN THE CALCULATION OF LIFECYCLE COST 49 TABLE 7-2. IMPLEMENTATION COSTS' 50 TABLE7-3. ANNUALO&MCOSTS 50 TABLE 7-4. EQUIPMENT COSTS 52 11 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Section 1.0: Introduction The purpose of this document is to describe the evaluation of the consequence management (CM) component of the Cincinnati pilot, the first such pilot deployed under the U.S. Environmental Protection Agency's (EPA) Water Security Initiative (WSI). This evaluation was implemented by examining the performance of the CM component relative to the design objectives established for the contamination warning system (CWS). 1.1 CWS Design Objectives The Cincinnati CWS was designed to meet six overarching objectives, which are described in detail in WaterSentinel System Architecture (USEPA, 2005) and are presented briefly below: • Spatial Coverage. The objective for spatial coverage is to monitor the entire population served by the drinking water utility. It depends on the location and density of monitoring points in the distribution system and the hydraulic connectivity of each monitoring location to downstream regions and populations. • Contaminant Coverage. The objective for contaminant coverage is to provide detection capabilities for all EPA priority contaminants. This design objective is further defined by binning the priority contaminants into 12 classes according to the means by which they might be detected (USEPA, 2005). Use of these detection classes to inform design provides more comprehensive coverage of contaminants of concern than would be achieved by designing the system around a handful of specific contaminants. Contaminant coverage depends on the specific data streams analyzed by each monitoring and surveillance component, as well as the specific attributes of each component. • Alert Occurrence. The objective of this aspect of system design is to minimize the rate of invalid alerts (alerts unrelated to contamination or other anomalous conditions) while maintaining the ability of the system to detect real incidents. It depends on the quality of the underlying data as well as the event detection systems that continuously analyze that data for anomalies. • Timeliness of Detection and Response. The objective of this aspect of system design is to provide initial detection of a contamination incident in a timeframe that allows for the implementation of response actions that result in significant consequences reduction. For monitoring and surveillance components, this design objective addresses only the detection of an anomaly and initial investigation of the subsequent alert. The CM component of the system is evaluated with respect to timeliness of response, or the time it takes to complete investigative actions and implement response actions once a contamination has been determined to be Possible. • Operational Reliability. The objective for operational reliability is to achieve a sufficiently high degree of system availability, data completeness and data accuracy such that the probability of missing a contamination incident becomes exceedingly low. Operational reliability depends on the redundancies built into the CWS and each of its components. • Sustainability. The objective of this aspect of system design is to develop a CWS that provides benefits to the utility and partner organizations while minimizing the costs. This can be achieved through leveraging existing systems and resources that can readily be integrated into the design of the CWS. Furthermore, a design that results in dual-use applications that benefit the utility in day-to-day operations, while also providing the capability to detect intentional or accidental contamination incidents, will improve Sustainability. ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot The design objectives provide a basis for evaluating each component, in this case CM, as well as the entire integrated CWS. Because the deployment of a drinking water CWS is a new concept, design standards or benchmarks are unavailable. Thus, it is necessary to evaluate the performance of the pilot CWS in Cincinnati against the design objectives relative to the baseline state of the utility prior to CWS deployment. 1.2 Role of CM in the Cincinnati CWS CM is a key component of the Cincinnati CWS and consists of actions taken to plan for and respond to possible contamination incidents. These actions are meant to minimize response and recovery timelines through a pre-planned, coordinated effort. Investigative and response actions initiated upon determination of a Possible contamination incident are used to establish credibility, minimize public health and economic impacts and ultimately return the utility to normal operations. CM is designed to guide the utility through this process of determining whether a Possible contamination incident is Credible and can be Confirmed. CM also assists the utility in working with local partners, communicating with the public, and determining appropriate response actions. Figure 1-1 provides an overview of the surveillance and response activities which comprise the Cincinnati CWS. The CWS is divided into surveillance components (water quality monitoring, public health surveillance, customer complaint surveillance, and enhanced security monitoring) and response components (CM and sampling and analysis). As illustrated, the surveillance components help to detect water quality anomalies and potential causes, while the response components aid in the implementation of response actions that minimize adverse impacts and ultimately assist in returning the system to normal operation. Consequence management plans (CMP), as a part of the CM component, focus on integrating common elements of existing utility plans, such as the emergency response and communication plans, while also coordinating utility actions with those of its local, state and federal partners. Monitoring and Surveillance Response Enhanced Security ^ ^Customer Complaint^ Monitoring JI Surveillance ) Water Quality Monitoring Public Health Surveillance Event Detection Initial Alert Investigation Can No ! Yes V Take corrective action if necessary; continue surveillance Consequence Management Sampling and Analysis /Credibility Determination Process to \ /Field Investigation to support confirm or rule out contamination credibility determination Check alerts from surveillance components Assess outside data sources Site characterization Field and lab analyses Response Actions protect public health during the investigation process and may include: • Isolation • Flushing • Public alerts/notifications Remediation and Recovery restores a system to normal operations and may include: • System characterization • Remedial action • Post-remediation activities Figure 1-1. CWS Surveillance and Response Overview ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot 1.3 Objectives The overall objective of the CM component evaluation is to demonstrate how the component functioned as part of the Cincinnati CWS (i.e., how effectively the component achieved the two design objectives described in Section 1.1). This evaluation will describe how well the incident response plans, response partner network, and communication equipment functioned when responding to possible water contamination incidents during exercises. Data gathered from planned exercises and the simulation study yielded sufficient data for the evaluation. 1.4 Document Organization This report will present the assessment of data collected to evaluate the performance of the CM component for each of the CM design elements of the Cincinnati CWS. The report is organized as follows: • Section 2: Overview of the CM Component. This section describes the composition and purposes of the CM component of the CWS. • Section 3: Methodology. This section describes the various sources of information and methods used to evaluate the CM component. • Sections 4 through 6: Performance of the CM Design Elements. These sections each focus on the performance of each of the individual CM component design elements evaluated as a part of the Cincinnati pilot (incident response plans, response partner network, and communication equipment). In each section, a description of the performance metric is presented, as well as an evaluation of how well the overall design objectives were met. • Section 7: Sustainability. This section presents the labor hours expended in developing and refining the CM component used in the Cincinnati pilot, including the labor hours expended for implementation and O&M costs of the CM component. There is also a discussion of dual-use benefits gained by the Cincinnati pilot through implementation of the CM component during non- contamination incidents. • Section 8: Summary and Conclusions. This section summarizes key lessons learned from the evaluation and includes a description of the cost-benefit of the CM component. • Section 9: References. This section contains references for all documents cited in this report. • Section 10: Abbreviations. This section lists all acronyms approved for use in the CM component evaluation. • Section 11: Glossary. This section defines terms used throughout the CM component evaluation. ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Section 2.0: Overview of the CM Component The following section provides an overview of the Cincinnati pilot CM component including the three major design elements, which are summarized in Table 2-1. First, incident response plans included the Cincinnati Pilot Consequence Management Plan, which was developed through modification and incorporation of existing emergency response plans, partnerships and procedures to address the unique challenges associated with response to a drinking water contamination incident. The goal of the component design was first to identify existing Cincinnati pilot response plans, identify and remedy gaps in response roles and responsibilities, and then develop integrated CM protocols. Incident response plans also included supporting documents such as the Crisis Communications Plan (CCP) and the Confirmatory Sampling Field Decision Guide (CSFDG). Second, a network of response partner agencies was developed that would support implementation of the guidance contained in the integrated incident response plans. Finally, communication equipment consisting of eight hand-held 800 MHz radios was procured for the utility. These radios were to be used by the utility during field response actions associated with contamination incident investigation. Prior to the CWS pilot, utility field crews utilized cell phones for communication. Procurement of the 800 MHz radios provided the field crew secure and guaranteed communication to the water utility emergency response manager (WUERM) and local response partners such as Hazardous Materials (HazMat) team (see Sections 2.3 and 6.0). Table 2-1. Consequence Management Design Elements Design Element Incident Response Plans Response Partner Network Communication Equipment Description This included the following: • CMP to guide the utility through actions taken upon notification of a Possible contamination incident, • CCP to guide the utility and partners on when/how to make notifications, define the message, work with the media and develop a delivery system, and • CSFDG to guide operational responses, including isolation and distribution system sampling. This included a network of local, state and federal response partner agencies that had various response roles during a contamination incident. This included equipment (e.g., inter/intranet sharing sites, reverse phone information systems, 800 MHz radios) required by the Cincinnati pilot to effectively respond to contamination incidents. 2.1 Incident Response Plans Three different incident response plans were developed during the course of the Cincinnati pilot study as detailed below. • Consequence Management Plan. The Cincinnati Pilot Consequence Management Plan was developed to serve as a preparedness and response guide in the event of a Possible water contamination incident. The process of determining whether an incident is Possible was documented in the utility Cincinnati Pilot Operational Strategy. The Possible contamination evaluation process is conducted in response to an alert from one of the monitoring and surveillance components. Once the alert is validated through the procedures outlined in the Cincinnati Pilot Operational Strategy, then the incident is Possible and CM begins. ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot The Cincinnati Pilot Consequence Management Plan used a series of nine separate decision trees to guide personnel through the process of determining whether a Possible contamination incident, as indicated by one of the CWS monitoring and surveillance components, is Credible and can be Confirmed. The individual decision tree topics included threat level determination (e.g., Credible, Confirmed), site characterization (SC), operational responses, response partner and public notification protocols, and remediation and recovery (R&R). The Cincinnati Pilot Consequence Management Plan also contained five appendices of supporting material, including the site characterization plan, forms for collecting and documenting the contamination incident and operational procedure sheets for major response activities. • Crisis Communications Plan. The CCP was developed to formalize public notification procedures and guide the actions of the utility Public Information Officer (PIO) during all phases of a potential contamination incident. It was designed from best practices in risk communication and public notification to provide communication control internally during an incident and to coordinate external public notification. The CCP was designed to complement the overall Cincinnati Pilot Consequence Management Plan and corresponding incident response plans. The CCP covered communication both within the utility and with external response partner agencies, the media and the public. The plan included an overview of basic crisis communication principles, detailed decision trees adapted for use by the PIO and a tools and resources section that includes sample public notification templates, media resources and contact information. • Confirmatory Sampling Field Decision Guide. The CSFDG consisted of charts and matrix tables based on hydraulic models and tracer studies that describe the hydraulics of the utility's distribution system in terms of predetermined pressure zones. It allowed utility personnel to rapidly identify potential sampling sites in the system based on initial detection of contaminants. 2.2 Response Partner Network The Cincinnati pilot established a response partner network to better integrate their roles and responsibilities in the event of a drinking water contamination incident. This allowed utility personnel to document these roles, responsibilities and communication requirements into the Cincinnati Pilot Consequence Management Plan and allowed response partner agencies to recognize the first response capabilities and duties of the utility. Inclusion of the response partners in exercises further enhanced the response partner network (see Section 5.0). The list of Cincinnati pilot local response partner agencies and their roles and responsibilities are shown in Table 2-2. Local response partners are defined as those agencies residing close enough to the utility to be able to provide immediate support during the investigation of a contamination incident. A broader population of response partners would participate in the R&R phases of CM. These agencies and their roles are described in more detail in Section 4.1.3 of this report. Table 2-2. CM Roles and Responsibilities - Cincinnati Local Response Partner Agencies Response Partner Organization Method Labs Cincinnati Fire Dept (CFD) Metropolitan Sewer District of Greater Cincinnati (MSDGC) Roles & Responsibilities Analyzes triggered samples and interacts directly with the Cincinnati pilot WUERM for field results, event status, and reporting. Supports site characterization activities when requested by providing field response and HazMat support. Provides laboratory support and input on disposal options. ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Response Partner Organization Cincinnati Health Department (CHD), Hamilton County Public Health (HCPH) Drug and Poison Information Center (DPIC) (added during evaluation period) Hamilton County Emergency Management Agency (HCEMA) Ohio Department of Health (ODH) Cincinnati Police Department (CPD) and Ohio State Police Ohio EPA (OEPA) (State Drinking Water Primacy Agency) Roles & Responsibilities Provides local public health data, epidemiologists and disease investigators. Provides technical support on contaminant symptomology and demographics during a contamination incident. Coordinates alternate water supplies and implements Unified Command System for expanded contamination incidents. Provides regulatory and sampling support, investigations, and threat assessments of Possible or Confirmed contamination incidents. Provides support by coordinating investigation and isolation issues with Cincinnati pilot personnel, and transports samples to ODH along with the associated chain-of-custody. Provides support for regulatory issues, threat assessment, risk communication, and R&R issues. 2.3 Communication Equipment This design element of the CM component identifies equipment required by Cincinnati pilot personnel to effectively respond to possible drinking water contamination incidents. It was determined that the prior approach of using cell phone communication for responding to normal water quality issues was not adequate for handling the expanded scope of responding to a possible water contamination incident (e.g., site characterization activities, ICS communications). This being the case, eight Motorola XTS 5000 800 MHz radios were procured and deployed to be used during field investigation activities. The radios are multi-channel programmable hand-held units that allow for communication between Cincinnati pilot personnel (e.g., field personnel, team leaders, response partners). During the evaluation period, first- responders' existing 800 MHz radio communication network was programmed into the utility 800 MHz radios. 2.4 CM Component Roles and Responsibilities Similar to the local response partner agencies outlined in Table 2-2, Table 2-3 summarizes the general roles and responsibilities of major utility Incident Command System (ICS) personnel in implementing the CM component. Table 2-3. CM Roles and Responsibilities - Utility ICS Personnel Personnel Director WUERM Plant Supervisor Security Roles & Responsibilities Manages the implementation of the CMP and the management structure of the ICS or elements of both depending on the nature of the contamination incident. If the ICS is activated, the Director may assume the role of Incident Commander (1C) from the WUERM as appropriate. Assumes the role of 1C until relieved of these duties by the Director. If not acting as 1C, the WUERM works directly with the Site Characterization Team (SCT) and will also support the Director and response activities as outlined in the CMP. Reviews recent operational and treatment changes. Utility security and local law enforcement conduct security surveys of system facilities during investigations and provide security support for field investigation personnel. Acts as the liaison for external response partners. ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Personnel Planning Section Chief Operations Section Chief PIO Customer/ Public Information Manager Liaison Officer Technical Analyst Roles & Responsibilities Maintains the incident log, planning documents, and data management functions during an investigation. Assists with operational response actions such as isolating the system. Coordinates directly with other agency PIOs and the public. Implements the CCP to help prepare and coordinate the release of internal and public notifications. Shares public information responsibilities with the PIO. Contacts and works with external response partners as necessary. Communicates with regulators and the scientific community concerning water quality issues. Manages water quality analysis. 2.5 Summary of Significant CM Component Modifications The CM component modifications discussed in Table 2-4 were implemented during the evaluation period to improve performance. These modifications serve as a reference when discussing the results of the evaluation (Sections 4 through 7). Table 2-4. Significant CM Component Modifications ID Component Modification Date Incident Response Plan Modifications 1.1 1.2 1.3 Modification Cause Modification Cause Modification The CSFDG was updated with modifications to the contaminant transport worksheets and the addition of a section outlining the procedures for contaminant source determination. Updates to the CSFDG, dated June 15, 2008, were the direct result of a drill conducted on April 15, 2008 to evaluate the procedures in the guide, to collect baseline data on the elapsed times required to implement them, and to provide Greater Cincinnati Water Works (GCWW) personnel the opportunity to practice the protocols and procedures outlined in the guide. The draft-final version of the CMP, dated June 12, 2007, was updated with modifications to roles and responsibilities of Cincinnati pilot personnel as well as response partners, including how the ICS operates within the utility, how the SCT interacts with local law enforcement and HazMat, and how operational responses occur during an incident (e.g., isolation responses). Updates to the CMP, dated September 1, 2008, were the direct result of exercises conducted in 2007 and early 2008, including two site characterization drills, a sampling and analysis drill, a functional exercise, and a full scale exercise identifying that roles and responsibilities required further clarification. The draft-final version of the CMP, dated September 1, 2008, was further updated, with the majority of changes pertaining to modifying and adding new roles and responsibilities of utility ICS personnel as well as response partners. 6/15/2008 9/1/2008 2/10/2009 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot ID Component Modification Date Cause Updates to the CMP, dated February 10, 2009, were the direct result of the full scale exercise conducted October 1, 2008, to evaluate the CMP, including site characterization, threat level determinations, operational responses, public notification, and response partner roles. Changes included adding new and modifying existing roles and responsibilities. Modification The draft-final version of the CMP, dated February 10, 2009, was updated with the majority of modifications specifically pertaining to updating ICS roles and responsibilities and modifying the decisions trees/process flows for site characterization and R&R. 1.4 Cause Updates to the CMP, dated June 5, 2009, were based on the outcomes of several drills conducted in early 2009 identifying the need to modify existing roles and responsibilities, decision trees for site characterization and R&R activities. These drills included: • Customer Complaint Surveillance(CCS)/Site Characterization workshop (April 2, 2009): objective was to discuss site characterization protocols for a Possible CCS alert including where to sample, who samples, and the sampling process. • SC drill (April 23, 2009): objective was to evaluate the implementation of the revised site characterization procedures outlined in the CMP. • R&R workshop (May 14, 2009): objective was to discuss the R&R decision tree and corresponding process flow. 6/5/2009 Modification 1.5 The draft-final version of the CCP, dated March 8, 2007, was updated to be consistent with the updated version of the CMP dated June 5, 2009. The majority of the changes specifically pertained to modifying and adding new roles and responsibilities, updating the decision tree figures and reformatting the threat level phases (e.g., Credible, Confirmed). Cause Inconsistencies were noted between the CCP and CMP pertaining to the roles and responsibilities of the PIO and the Customer/Public Information Manager during the various phases of a contamination threat and the public notification decision tree process. 6/5/2009 Modification 1.6 The draft-final version of the CMP, dated June 5, 2009, was further updated, with the majority of changes pertaining to changing the term 'Confirmed' to 'Assumed/Determined Contamination and updating ICS roles and responsibilities, as appropriate. Cause Updates to the CMP, dated September 23, 2009, were based on the outcomes of a threat level determination (TLD) workshop, held July 1, 2009, which provided ICS personnel with guidance on and practice for determining threat levels (e.g., Possible, Credible and Confirmed determination) and associated response actions for specific CWS alert scenarios. 9/23/2009 Modification 1.7 The draft-final version of the CCP, dated June 5, 2009, was updated to be consistent with the current version of the CMP dated September 23, 2009. The majority of the changes pertained to modifying the threat level phases, modifying ICS titles and roles, and updating the decision tree figures and checklist, as appropriate. Cause Inconsistencies were noted between the CCP and CMP pertaining to the threat level determination terminology used and roles and responsibilities of the PIO and the Customer/Public Information Manager. 9/28/2009 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot ID Component Modification Date Response Partner Network Modifications 2.1 Modification Cause The response partner network was updated to include DPIC. Updates to the response partner network, dated September 1, 2008, including the addition of DPIC to the response partner network, was the direct result of exercises conducted in 2007 and early 2008, where the Cincinnati pilot recognized the resourcefulness of DPIC in providing information on potential poisoning inquiries that could be correlated to drinking water contamination. 9/1/2008 Communication Equipment Modifications 3.1 Modification Cause The CFD programmed the Cincinnati pilot 800 MHz radios with a communication channel that permitted network communications with the Cincinnati emergency response channel. Direct communication between utility field response personnel and CFD HazMat personnel was not possible prior to the reprogramming. The deficiency was identified during FSE 2. 10/5/2008 2.6 Timeline of CM Development Phases and Evaluation-Related Activities Figure 2-1 presents a summary timeline for deployment of the CM component, including milestone dates indicating when significant component modifications and exercise evaluation activities took place. The timeline also shows the completion date for design and implementation of the CM component including the first completed draft of the Cincinnati Pilot Consequence Management Plan (January 2007), along with the subsequent optimization and real-time monitoring phases of deployment. Sep-07 SC Drill 1 Jan-07 Jul-08 SC Drill 2 Apr-09 May-09 Sep-09 SC Drill 3 R&R Workshop SC/CCS Drill Jan-08 Jun-10 Jan-09 Jan-10 y y J A^ A _Jl Functional FSE 1 Wind Storm FSE 2 Exercise Sep-07 Sep-08 Oct-08 Jul-07 v ^A Optimization Jan-07 - Jan-09 ? ^ i TLD Workshop FSE 3 Jul-09 Oct-09 *v J V CMP Completed Jan-07 Figure 2-1. Timeline of CM Component Activities Real-time Monitoring Jan-09 -Jun-10 End of Data Collection Jun-10 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Section 3.0: Methodology The following section describes the evaluation techniques that were used to fully evaluate the CM component metrics, which are further described in Section 4.0. Specific metrics are described for each element of the CM component as they relate to the design objectives. The CM evaluation was conducted using results from four evaluation techniques: training and exercise program, the simulation study, feedback forums and analysis of life cycle costs. 3.1 Training and Exercise Program The CM training and exercise program consisted of exercises modeled after the Department of Homeland Security Exercise and Evaluation Program. These exercises included discussion-based exercises consisting of seminars, workshops and tabletop exercises, and operations-based exercises that included drills, functional exercises, and full scale exercises. After the exercises were completed, After Action Reports (AARs) and a corresponding Improvement Plan were developed to capture comments from participants and evaluators regarding suggested modifications to the Cincinnati Pilot Consequence Management Plan and to outline areas for improvement of future exercises. In general, discussion-based exercises are used to develop, refine, and train on plans, while operations- based exercises are used to test and evaluate those plans. All of the training and exercise activities associated with the CM component are described in this report for completeness, but most of the metrics used for performance evaluation were extracted from operations-based exercises, particularly the full scale exercises. Twenty-five CM discussion-based and operations-based exercises were conducted with the Cincinnati pilot since the inception of the pilot in 2005. The training and exercises were important in identifying opportunities for improving the plans, evaluating participants' ability to implement the guidance of the Cincinnati Pilot Consequence Management Plan, and optimizing response time and accuracy. 3.1.1 Discussion-Based Exercises The CM component discussion-based exercises are described in Table 3-1. These discussion-based exercises were essential for staff to understand and become familiar with their Cincinnati Pilot Consequence Management Plan roles/responsibilities, which prepared them to perform well in the operations-based exercises that followed. Table 3-1. Discussion-Based Exercises Training and Exercise Title Date(s) Conducted Participants Description CM Workshop 1 12/13/2005 Cincinnati Pilot Discussed with GCWW key officials, CFD,and CPD initial stages of CMP development including decision trees and response partner involvement. CFD provided introductory National Incident Management System (NIMS) briefing for utility staff. CM Workshop 2 01/19/2006 Cincinnati Pilot Collected data from City and County Public Health Agencies, CFD and CPD and reviewed draft CMP decision trees for field operation information that would occur with Possible, Credible and Confirmed determination decision trees. 10 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Training and Exercise Title CM Workshop 3 CM Workshop 4 DHS FEMA/NIMS IS100&IS700 Training Seminars* CMP Orientation Seminars Tabletop Exercises: Supervisor Training Roll Out Documentation & Presentation SCT Training Seminars CWS Management Orientation Seminar CCS / SCT Workshop R&R Workshop Date(s) Conducted 02/14- 02/15/2006 3/22/06 5/16- 05/18/2006 08/22 - 08/24/2006 08/29 - 08/31/2006 (4 separate sessions) 06/27/2006 & 09/28/2006 05/11/2007 (4 separate sessions) 06/15/2007 04/2/2009 05/14/2009 Participants Cincinnati pilot Cincinnati pilot GCWW GCWW GCWW Potential partners from outside of the utility response area GCWW GCWW GCWW GCWW Description Provided information to GCWW Key Officials, PIOs from various City & County Response Agencies, City and County Public Health Agencies, CFD and CPD on risk communication and message mapping for communicating with the public during a possible contamination incident. Developed specific message maps for the CCP. Collected and reviewed notification and communication data focusing on Credible and Confirmed phases of an incident where multiple response agencies will be involved including City and County Public Health Agencies, CFD, CPD, and city, state and federal government officials. This involved identifying key connections, linkages and dependencies between the draft CMP and response partner agency plans and procedures. Collected the same data for sub- flow decision trees (e.g., site characterization), which was utilized in developing draft CMP. Provided participants, including senior management, with a basic understanding of ICS procedures and an introduction to NIMS. The material was formatted for use in future utility instruction. Provided participants, including senior management, with an understanding of their roles in the CMP. The material was formatted for use in future utility instruction. Provided participants, including senior management and supervisors, with scenarios to improve their knowledge of the CMP and incident management processes. Discussed the CM preparedness and response process with external agencies and organizations that might not otherwise be readily engaged in the utility response network. Familiarized staff with field operations associated with site investigations and corresponding equipment. Provided participants, including senior management, an overview of the CWS and CMP including contamination scenario exercises to familiarize attendees. Presented scenarios to discuss and modify CMP site characterization roles and responsibilities following Possible determination of a CCS alert(s). For each scenario, discussions were based on where to sample, who will sample, and the sample collection process. Reviewed and discussed potential modifications to the CMP R&R process. CMP R&R decision tree was revised based on these discussions. 11 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Training and Exercise Title TLD Workshop CM FSE 3 Follow-up Workshop Date(s) Conducted 07/01/2009 05/21/2010 Participants GCWW Cincinnati pilot Description Provided participants, including senior management, guidance and practice on determining threat levels (e.g., Possible, Credible and Confirmed determination) and associated response actions for specific CWS alert scenarios. CMP threat level terminology was modified as a result. Provided the utility and its response partner agencies an opportunity to discuss the actions following contamination of the drinking water distribution system. This included a facilitated discussion and corresponding changes to the R&R section of the CMP. Note: *GCWW personnel have also taken other relevant ICS training which was not a formal part of the WSI pilot Threat Level Determination Workshop The TLD workshop, listed in Table 3-1, was of particular importance to the evaluation of the CM component. During this workshop, utility Steering Committee members and WUERMs examined and evaluated how the utility processed various kinds of contamination incident information to arrive at the threat levels and response actions described in the Cincinnati Pilot Consequence Management Plan. Six CWS alert scenarios were presented to participants in the form of incident response timelines. Participants were asked to review each of the scenarios and identify the time or step at which Possible, Credible and Confirmed threat levels would have occurred. They were also asked to indicate when any of the following response actions would have occurred during each incident: • Operational changes, • Notification of public health partners, • Public notification of use restrictions, • Site characterization and sample collection, and • Laboratory analysis of samples. 3.1.2 Operations-Based Exercises The operations-based exercises conducted for the Cincinnati Pilot Consequence Management Plan are described in Table 3-2. These exercises progressed from focused drills to larger functional and full scale exercises with response partners. Table 3-2. CM Operations-Based Exercises Training and Exercise Title CM Functional Exercise SCT Drill 1 Date(s) Conducted 07/31/2007 09/05 - 09/06/2007 Participants Cincinnati pilot GCWW Description Provided the utility and response partners the opportunity to practice their roles and responsibilities during a response to a possible drinking water contamination incident, identify potential revisions and corrections to the CMP, and practice plans and procedures of various agencies. Provided the SCT the opportunity to practice implementation of site characterization procedures and equipment. 12 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Training and Exercise Title Date(s) Conducted Participants Description FSE 1 09/25 - 09/28/2007 Cincinnati pilot Provided the utility and response partners the opportunity to exercise their roles within a field environment, test plans and procedures, and identify opportunities for improvement and potential revisions to the CMP. Participants also practiced communication and coordination techniques. SCT Drill 2 07/15/2008 Cincinnati pilot Provided the SCT and CFD HazMat an opportunity to cross-train on site characterization procedures. FSE 2 10/1 - 10/2/2008 Cincinnati pilot Provided a scenario-driven, real-time simulation that implemented utility and local response partner agency protocols related to detection of and response to a drinking water contamination incident. SCT Drill 3 04/23/2009 GCWW Conducted to evaluate the implementation of the revised site characterization procedures outlined in the CMP and the Standard Operating Procedures for site characterization and sampling. CCS / SCT Drill 09/16/2009 GCWW Evaluated the alert recognition and investigative procedures associated with the CCS component and to evaluate implementation of the site characterization procedures as they relate to field deployment and investigation following a CCS alert. FSE 3 10/21/2009 Cincinnati pilot Provide a scenario-driven, real-time simulation that implemented utility and local response partner agency protocols related to detection of and response to a drinking water contamination incident. FSE 3 also involved ICS implementation (for utility second-in-commands), external notifications resource coordination, media relations and the execution of field investigation procedures. Much of the data used in this report to evaluate CM response performance by the Cincinnati pilot was drawn from a series of three FSEs. These exercises were designed with different contamination incident scenarios, specifically developed to test and evaluate different detection and response procedures of the Cincinnati Pilot Operational Strategy and Cincinnati Pilot Consequence Management Plan. Thus, response times used in the performance evaluation were influenced by the specific situations presented by the scenarios. Additional details and distinguishing characteristics for each of the FSEs listed in Table 3- 2 is provided below. Full Scale Exercise 1 (FSE 1) FSE 1 was conducted for incident management and field response personnel, operating simultaneously but independently using a common scenario, with communication between the two groups coordinated by exercise staff in a simulation cell. The exercise was conducted in this manner in order to complete the entire exercise within the allotted eight-hour period. The scenario was initiated by a simulated total organic carbon (TOC) alert from a water quality monitoring (WQM) station located at a GCWW facility. This alert was followed by a CCS alert concerning foul smelling drinking water. Site characterization personnel were dispatched to the incident 13 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot site and discovered empty drums, along with transfer pumps and tubing that had been staged at the site to simulate intrusion. Local HazMat and law enforcement personnel participated in the exercise on-site, but external response partners participated remotely from their normal work locations. As part of all the full scale exercises, a "Hot Wash" was conducted at the conclusion of each exercise. The Hot Wash, conducted by the exercise controllers, consisted of a presentation of the events followed by a group discussion about the events and actions. The information collected during the Hot Wash was incorporated into the AARs and Improvement Plan development. Full Scale Exercise 2 (FSE 2) FSE 2 was a conventional FSE in that the Cincinnati pilot participated without the use of a simulation cell. The exercise was initiated with a simulated alert from a WQM station located at a GCWW facility, and featured additional alerts from the CCS system (both interactive voice response and work order), WQM alerts from additional locations, and ended with a public health surveillance (PHS) alert stemming from illnesses in the affected neighborhoods. The entire exercise scenario was based on a biological agent being intentionally introduced into the system through a pump station. One of the exercise goals was to have the exercise be as realistic as possible which was accomplished by using the utility's hydraulic model in developing the scenario. Full Scale Exercise 3 (FSE 3) FSE 3 again featured the Cincinnati pilot participating on-site in real time, but the initiation of the exercise was timed to involve alternate ICS and back-shift personnel in the response. This exercise scenario included recurring CCS and PHS alerts, and was designed to simulate the injection of atoxic chemical into the system through a fire hydrant. Field sampling from a neighborhood fire hydrant and consideration of residential sampling were among the significant objectives of this exercise. 3.2 Simulation Study Evaluation of certain design objectives relies on the occurrence of contamination incidents with known and varied characteristics. Because contamination incidents are extremely rare, there is insufficient empirical data to fully evaluate the detection capabilities of the Cincinnati CWS. To fill this gap, a computer model of the Cincinnati CWS was developed and challenged with a large ensemble of simulated contamination incidents in a simulation study. For the CM component, simulation study data was used to evaluate the following design objective: • Timeliness of Response: Analyses conducted to evaluate this design objective quantify the number of scenarios that reached Possible, Credible and Confirmed contamination and the number of scenarios that resulted in operational response actions, public health response, and public notification. Statistical analysis was used to characterize the time that these three threat levels were reached and the time that response actions were implemented. A broad range of contaminant types, producing a range of symptoms, was selected for the simulation study in order to characterize the detection capabilities of the monitoring and surveillance components of a CWS. For the purpose of the simulation study, a representative set of 17 contaminants was selected from the comprehensive contaminant list that formed the basis for CWS design. These contaminants are grouped into the following broad categories (the number in parentheses indicates the number of contaminants from that category that were simulated during the study): 14 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot • Nuisance Chemicals (2): these chemical contaminants have a relatively low toxicity and thus generally do not pose an immediate threat to public health. However, contamination with these chemicals can make the drinking water supply unusable. • Toxic Chemicals (8): these chemicals are highly toxic and pose an acute risk to public health at relatively low concentrations. • Biological Agents (7): these contaminants of biological origin include pathogens and toxins that pose a risk to public health at relatively low concentrations. Development of a detailed CWS model required extensive data collection and documentation of assumptions regarding component and system operations including the CWS monitoring and surveillance components (e.g., Enhanced Security Monitoring (ESM), WQM, CCS, PHS) and CM. To the extent possible, model decision logic and parameter values were developed from data generated through operation of the Cincinnati CWS, although input from subject matter experts and available research was utilized as well. The simulation study used several interrelated models, three of which are relevant to the evaluation of CM: EPANET, Health Impacts and Human Behavior (HI/HB) and the CM model. Each model is further broken down into modules that simulate a particular process or attribute of the model. The function of each of these models and its relevance to the evaluation of CM is discussed below. EPANET EPANET is a common hydraulic and water quality modeling application widely used in the water industry to simulate contaminant transport through a drinking water distribution system. In the simulation study, it was used to produce contaminant concentration profiles at every node in the GCWW distribution system model, based on the characteristics of each contamination scenario in the ensemble. The concentration profiles were used to determine the number of miles of pipe contaminated during each scenario, which is one measure of the consequences of the contamination scenario. EPANET was also used to model the efficacy of operational response actions predicted by the CM model in reducing the miles of pipe contaminated. Health Impacts and Human Behavior Model The HI/HB model used the concentration profiles generated by EPANET to simulate the health effects exposure of customers in the GCWW service area to contaminated drinking water. The model analyzed the exposed customers in three age groups including infants (five years old or younger), children (ages five to 18) and adults (ages 18 and older). Depending on the type of contaminant, exposures occurred during one showering event in the morning (for the inhalation exposure route) or during five consumption events spread throughout the day (for the ingestion exposure route). The HI/HB model used the dose received during exposure events to predict infections, onset of symptoms, health-seeking behaviors of symptomatic customers and fatalities. The primary output from the HI/HB model was a case table of affected customers, which captured the time at which each transitioned to mild, moderate and severe symptom categories. Additionally, the HI/HB model output the times exposed individuals would pursue various health-seeking behaviors, such as visiting their doctor or calling the poison control center. The case table was used to determine the public health consequences of each scenario, specifically the total number of illnesses and fatalities. Furthermore, EPANET and the HI/HB model were run twice for each scenario; once without the CWS in operation and once with the CWS in operation. The paired results from these runs were used to calculate the reduction in consequences due to CWS operations for each simulated contamination scenario. 15 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Consequence Management Model The CM model simulates the complex processes of determining the threat level of a Possible contamination incident detected through one of the monitoring and surveillance components and determining what response actions would be taken by the utility, law enforcement and the public health community. The CM model relies on several interdependent modules to determine the threat level and response actions. The threat level determination module combines the functions of the WUERM and the Incident Commander and uses the collective data generated by the monitoring and surveillance component models, as well as the results from other investigation activities, to establish whether contamination is Possible, Credible, or Confirmed. Because the module uses data from the component models in an integrated fashion, a Credible contamination can be determined sooner than if one component was deployed in isolation, modeling an important feature of the real CWS. Furthermore, the threat level determination module incorporates site characterization and laboratory analysis activities, which produce results that are used in the threat level determination process. Other modules of the CM model include decision logic that governs the implementation of response actions, such as isolation and public notification, based on the outputs of the threat level determination module. These modules output the time, location and type of response actions, which are ultimately used to revise parameters within EPANET and the HI/HB model to determine the consequences with a CWS in operation. The following assumptions used in the design of the CM model are important to consider when evaluating the simulation study results presented in this report: • Only scenarios that reached the Possible stage were considered in the analysis; without the determination of Possible, the CM component is not activated and cannot be evaluated. • Two contaminant injection location types were assumed: facility attack nodes and distribution attack nodes. Scenarios with these contaminant injection locations were analyzed separately because of the significant differences in timeline metrics between them. • The simulation study results included in the CM evaluation were limited to the 1,545 scenarios that originated at distribution attack nodes and that reached Possible contamination determination. • Each component except CCS can produce sufficient data to independently establish that contamination is Credible. • The ESM component is sufficient to confirm drinking water contamination without any additional data. This occurs when responders investigating an ESM alert observe the contaminant injection in process. • CCS, PHS and WQM must be supported by at least one other component to confirm a contamination incident. • Operational responses are limited to isolation. Flushing potentially contaminated water from the distribution system was not considered based on observations during drills and exercises. • Issuance of public notification occurs when the public notice is completed (e.g., 120 minutes after contamination is deemed Possible) AND when any one of the following conditions are met: o Threat level is Confirmed. o Threat level is Credible and public health investigators conclude that illness in the community may be linked to drinking water. 16 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot o Threat level is Credible and WQM alerts are due to low chlorine residual, representing a potential risk to public health that warrants public notification, regardless of the cause of the alert. o Threat level is Credible due to an ESM alert (indicating tampering at a utility facility) and another component has generated a valid alert (indicating that potentially contaminated water has not been isolated to the utility facility). • All public notifications are issued for the entire GCWW service area. 3.3 Feedback Forums Two lessons learned workshops and a CM information transfer meeting were held to provide qualitative feedback regarding the CM component deployed in Cincinnati. • Lessons Learned Workshop, June 16, 2008 - was limited to eight EPA and contractor support personnel responsible for the design and implementation of the CM component. The objective of the workshop was to revisit key decisions made during the process and solicit specific feedback on the successes and challenges encountered. • CM Information Transfer Meeting, May 28, 2009 - included 17 participants from EPA, the Cincinnati pilot and contract support personnel. The purpose of the meeting was to provide the Cincinnati pilot with information necessary for maintaining the CM component. • Lessons Learned Workshop, August 11, 2009 - included 26 participants from EPA, the Cincinnati pilot, and contract support personnel involved in the design and implementation of the CM component. The objective of the workshop was to elicit specific lessons learned from the pilot utility and to gather feedback concerning how lessons learned may be shared with the drinking water sector. 3.4 Analysis of Lifecycle Costs A systematic process was used to evaluate the overall cost of the CM component over the 20 year lifecycle of the Cincinnati CWS. The analysis includes implementation costs, annual O&M costs, renewal and replacement costs, and the salvage value of major pieces of equipment at the end of the lifecycle. Data from CM operations was collected monthly from January 16, 2008 through June 15, 2010. Data was tracked by reporting periods, and in this evaluation, the term 'reporting period' is used to refer to a month of metrics data that spans from the 16th of one month to the 15th of the next. Thus, the January 2008 reporting period refers to the data collected between January 16th 2008 and February 15th 2008. Implementation costs include labor and other expenditures (equipment, supplies and purchased services) for deploying the CM component. Implementation costs were summarized in Water Security Initiative: Cincinnati Pilot Post-Implementation System Status (USEPA, 2008), which was used as a primary data source for this analysis. In that report, overarching project management costs incurred during the implementation process were captured as a separate line item. However, in this analysis, the project management costs were equally distributed among the six components of the CWS, and are presented as a separate line item for each component. It should be noted that implementation costs for the Cincinnati CWS may be higher than those for other utilities given that this project was the first comprehensive, large-scale CWS of its kind and had no experience base to draw from. Costs that would not likely apply to any future utility project implementation, but which were incurred for the Cincinnati CWS, include overhead for EPA and its contractors, cost associated with deploying alternative designs and additional data collection and 17 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot reporting requirements. Other utilities planning for a similar large-scale CWS installation would have the benefit of lessons learned and an experience base developed through implementation of the Cincinnati CWS. Annual O&M costs include labor and any other expenditure necessary to operate and maintain the component. O&M labor costs were estimated from the time spent planning and participating in drills and exercises as well as time spent in training. To account for the maintenance of documents, the costs incurred to update documented procedures following drills and exercises conducted during the evaluation phase of the pilot were used to estimate the annualized cost. The total O&M costs were annualized by calculating the sum of labor and other expenditures incurred over the course of a year. Labor hours for both implementation and O&M were tracked over the entire evaluation period. Labor hours were converted to dollars using estimated local labor rates for the different institutions involved in the implementation or O&M of the CM component. The renewal and replacement costs are based on the cost of replacing major pieces of equipment at the end of their useful life. The useful life of CM equipment was estimated using field experience, manufacturer-provided data and input from subject matter experts. Equipment was assumed to be replaced at the end of its useful life over the 20 year lifecycle of the Cincinnati CWS. The salvage value is based on the estimated value of each major piece of equipment at the end of the lifecycle. The salvage value was estimated for all equipment with an initial value greater than ~$ 1,000. Straight line depreciation was used to estimate the salvage value for all major pieces of CM equipment based on the lifespan of each item. All of the cost parameters described above (implementation costs, O&M costs, renewal and replacement costs, and salvage value) were used to calculate the total lifecycle cost for the CM component, as presented in Section 7.1. 18 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Section 4.0: Performance of Incident Response Plans The metric data used to measure the performance of the incident response plans design element was gathered from the FSEs and the simulation study model. These performance metrics consisted of the elapsed times it took the Cincinnati pilot to complete specified tasks within the following CM categories: • Credibility Determination Actions: including time to determine threat levels. • Response Actions: including time to notify response partners, to decide on appropriate operational responses, and to make public notification of water use restrictions. • R&R Actions: including time to restore the system to normal operations. Since none of the FSEs included R&R objectives, the metrics used to evaluate this aspect of the CM component consisted of the number and nature of the changes to the R&R section of the Cincinnati Pilot Consequence Management Plan that resulted from several R&R workshops. Additionally, the criteria that the Cincinnati pilot used to arrive at TLD decisions are also discussed. Note that other CM-related metrics dealing with site characterization activities, such as time for assessing site hazard levels, time for deploying field personnel and equipment, and time for collecting and screening drinking water samples are discussed in Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Pilot (USEPA, 2013a). Table 4-1 identifies site characterization metric categories and the location of their performance evaluation. Table 4-1. Site Characterization Metric Categories and Report Location Site Characterization Response Activity Time to notify response partner(s) Time to deploy field personnel and equipment Time to assess hazard level: Approach site Conduct safety screening Time to collect and screen drinking water samples for: Sample collection Rapid Field Test (RFT) Lab analysis Data review CM Evaluation Report X Sampling & Analysis Evaluation Report X X X 4.1 Design Objective: Timeliness of Response Within the Timeliness of Response design objective, three performance criteria were identified to evaluate the effectiveness of the incident response plans. These performance criteria included: 1) the time to perform investigative action (i.e., Credibility determination), 2) the time to implement response actions, and 3) the effectiveness in developing and refining the R&R process of the CM component. These criteria are further defined in Sections 4.1.1 through 4.1.3. Figure 4-1 demonstrates a generic incident response timeline of activities to investigate a contamination incident that generates multiple CWS alerts. It is included here to help visualize how the CM component contributes to this design objective. Since response actions to a contamination incident are driven by the specific characteristics of the incident itself (e.g., contaminant, means of introduction, spatial and volumetric scope of the contamination area, hydraulic characteristics of the system), direct comparison of the elapsed times to complete the stages of the response timeline from one incident to another (or from one exercise to another) is not quantitatively meaningful. Rather, the performance of this design 19 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot objective is represented by observed times for the implementation of Cincinnati Pilot Consequence Management Plan investigative and response actions, accompanied by a qualitative discussion of the factors used by the Cincinnati pilot in CM-related decisions. BEGIN Initial Component Alert(s) Received Additional Alert(s) Received Additional Alert(s) Received POSSIBLE DETERMINATION PHASE Operational Strategy Checks Operational Responses CREDIBLE DETERMINATION PHASE Response Partners Notified SCT Deployed SCT Positive Result Received HazMat Activated Sample Sentto Contract Laboratory CONFIRMED DETERMINATION PHASE Public Notification Issued Sample Arrives at Laboratory Sample Positive for "X" CONFIRMED CONTAMINATION BEGIN R&R END Figure 4-1. Generic CM Timeline 4.1.1 Credibility Determination The amount of time required for investigative actions leading to both Credible and Confirmed (including "Assumed Contamination") determination of a Possible contamination incident were analyzed using both the exercise data and the simulation study data. Details of these two analyses follow. Credible Determination Definition: The Credible determination phase of the incident timeline begins once the initial component alert investigation determines that contamination is Possible. It ends when evidence from follow-on investigations either corroborates or refutes the initial alert, escalating the contamination threat to Credible or closing the investigation. This phase includes the time required to perform multi-component investigations and data integration, implement field investigations (such as site characterization), and 20 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot collect additional information to support the investigation. Exercise Analysis Methodology: The analysis for the Credible determination phase of an incident consisted of the elapsed time it took the utility (starting from Possible determination) to investigate the available information and determine the incident as Credible during the FSEs. This information was tabulated and examined for correlations between the number and/or type of alerts received, the investigative information available to the utility and the amount of time required for Credible determination. Exercise Results: The FSE results for Credible determination are shown in Table 4-2 and Figure 4-2. There is no statistical trend in the times to reach Credible determination among the three FSEs, which is expected because times are dependent upon many variables, including the number of component alerts, field investigation evidence and the fact that different participants were responsible for making the determination. For each FSE, information from multiple CWS component alerts was required for a Credible determination to be made. Other information that led to Credible determination included receipt of multiple alerts from a single component with corroborating information. For example, in FSE 2 water quality monitoring alerts were received from stations that were hydraulically connected, and in FSE 1 and FSE 3, receipt of field investigation evidence was collected by the SCT. Finally, information that indicated suspicious or intrusive activity led to quicker elevation of the incident to the Credible phase. FSE1 FSE 2 FSES 50 100 150 200 Time (minutes) 250 300 350 Figure 4-2. Comparison of FSE Time from Possible to Credible Determination Table 4-2. Exercise Times for Credible Determination Exercise Number of Alerts Received Time to Credible Determination (minutes) Available Information FSE 1 300 Verified Possible determination 1 WQM alert: TOC 1 CCS alert: Odor Suspicious activity: Leaky chemical drums onsite Normal RFTs 21 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Exercise FSE2 FSE3 Number of Alerts Received 6 3 Time to Credible Determination (minutes) 79 191 Available Information Verified Possible determination 5 WQM alerts: TOC, chlorine residual 1 CCS alert: Impacted neighborhood Normal RFTs Verified Possible determination 3 CCS alerts: Taste, same area 6 additional call complaints: Taste, same area • Positive RFT: Toxic chemical Simulation Study Analysis Methodology: Of the 1,545 distribution attack scenarios that reached Possible, 1,315 or 85% of them reached Credible. The analysis for Credible determination utilizing the simulation study results consisted of performing separate statistical analyses of the data set according to three scenario characteristics that had the most impact on the Credible determination process. The scenario characteristics include contaminant type number of components detecting, and number of alerts. Simulation Study Results by Contaminant Type: The percentile distribution for time from Possible to Credible determination from the analysis of distribution attack scenarios involving nuisance chemicals, toxic chemicals and biological agents is shown in Figure 4-3. The analysis by contaminant type shows a significant difference between the nuisance chemicals, toxic chemicals and biological agents relative to the timeliness of detection. In general, scenarios that involved toxic chemical contaminants were determined Credible much more quickly than those that involved biological agents and nuisance chemicals. The differences in the Credible determination time was most likely due to how the contaminants affected the taste, odor or appearance of the drinking water, and whether the contamination scenario generated customer complaints. Contaminants that triggered customer complaints generally resulted in a quicker credibility determination process, compared with contaminants that did not. Nuisance Chemicals Toxic Chemicals Biological Agents x=172 n=149 x=135 H n=1034 x = 623 n = 132 200 400 600 Time (minutes) 800 1000 1200 Figure 4-3. Time from Possible to Credible Determination by Contaminant Type 22 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Simulation Study Results by Number of Components Detecting: The percentile distribution of time from Possible contamination to Credible determination for distribution attack scenarios where one, two or three different monitoring and surveillance components detected the contaminant is shown in Figure 4-4. The results indicate that an increase in the number of components causes a decrease in the Credible determination time span. The data for three components shows a considerably narrower time span for Credible determination data compared to results where one or two components detected the contamination. 1 Component 2 Components 3 Components C x=153 M I in— ^^ A x=110 i i n — i zu x = 49 n = 71 ) 100 200 300 400 500 600 700 Time (minutes) Figure 4-4. Time from Possible to Credible Determination by Number of Components Detecting Simulation Study Results by Number of Alerts Received: The percentile distribution of time from Possible contamination to Credible determination for distribution attack scenarios based on the number of alerts received from three monitoring and surveillance components (WQM, CCS and/or PHS) is shown in Figure 4-5 and Table 4-3. The advantage that an increasing number of alerts have on the time to Credible determination is demonstrated by a significant reduction in the time from only one alert being received to five alerts being received. However, there was little incremental reduction in Credible determination time as the number of alerts exceeded five. 23 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot 11-100 Alerts 9 Alerts 8 Alerts 7 Alerts 6 Alerts 2 Alerts 1 Alert C i n ||| n JLJJ i— | | |-i n = 295 i fn 1 1 n i [|J i n 11 frn-H ' — Q)-* |},n=7 a, n-6 i n o ill 14 ' 1 1 ' " 4 S^«"22 ) 100 200 300 400 500 600 700 800 900 1000 Time (minutes) Figure 4-5. Time from Possible to Credible Determination by Number of Alerts Received Table 4-3. Number of Alerts Received - Times for Credible Determination Number of Alerts 1 2 3 4 5 6 7 8 9 10 11 -100 >100 Median (x ) Time to Credible Determination (minutes) 807 746 739 254 148 138 142 148 144 150 135 144 Confirmed Determination (Assumed Contamination/Determined Contamination) Definition: The Confirmed determination phase of the incident timeline begins when a contamination incident is deemed Possible, and ends when evidence from follow-on investigations either corroborates or refutes the contamination incident by escalating the contamination threat to Confirmed or closing the investigation. This includes the time required to perform laboratory analyses, collect additional information/evidence and analyze the collective information to determine if a preponderance of evidence confirms the incident. The utility determined during the TLD workshops that an incident can be considered Confirmed based on positive analytical results or a preponderance of evidence. A preponderance of evidence may include field sample results collected during site characterization; results and observations of site characterization; 24 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot multiple connected monitoring and surveillance alerts; information from public health officials, area hospitals, or 911 call centers; and/or targeted information from external sources (such as law enforcement intelligence) based on the collective knowledge of the threat. This being the case, the utility decided to change the Confirmed determination terminology so that "Assumed Contamination" would be used for cases based on a preponderance of evidence and "Determined Contamination" would be used for cases based on positive analytical results. The utility implemented this approach in order to escalate actions more quickly when there is compelling evidence of contamination rather than delaying for definitive analytical results. The metrics used for Confirmed determination performance evaluation in this report include both types. In FSE 1, the utility went to Confirmed based on positive analytical results and in FSE 2 and FSE 3, the utility went to Confirmed based on a preponderance of evidence. The simulation model does not differentiate between "Assumed Contamination" and "Determined Contamination;" both are reported as time of Confirmed determination. Exercise Analysis Methodology: The analysis for Confirmed determination was performed by compiling the elapsed time that it took the utility from making a Possible determination to declaring a Confirmed (e.g., Assumed Contamination or Determined Contamination) threat level during FSEs, and the investigative information available when the determination was made. This information was tabulated and examined for correlations between the number or type of alerts received, the investigative information available to the utility and the amount of time necessary for Confirmed determination. Exercise Results: Results for Confirmed determination from the exercises are shown in Figure 4-6. As was the case for Credible determination, no correlation could be made between the number of alerts received and the amount of time that was required for utility personnel to determine whether the incident was Confirmed. The reporting of illnesses in areas of the distribution system during exercises seemed to be a motivating factor for the investigators involved in the Cincinnati pilot to make a Confirmed determination, as shown in Table 4-4. This was demonstrated in the case of FSE 3, where GCWW personnel determined the incident to be Confirmed only fifteen minutes after determining that it was Credible and receiving a PHS alert. In FSE 1 there was no public health related alert or link to the investigation, therefore a confirmed laboratory analysis was necessary to generate the Confirmed threat level determination. 25 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot FSE1 FSE2 FSE3 100 200 300 Time (minutes) 400 500 Figure 4-6. Comparison of FSE Time from Possible to Confirmed Determination Table 4-4. Exercise Times from Credible to Confirmed Determination Exercise FSE 1 FSE 2 FSE 3 Number of Alerts Received 2 7 4 Time from Credible to Confirmed Determination (minutes) 60 346 15 Available Information 1 WQM alert: TOC 1 CCS IVR alert: Odor Suspicious activity: Leaky drums onsite Positive laboratory results 5 WQM alerts: TOC, chlorine residual 1 CCS alert: impacted neighborhood Signs of intrusion PHS alert: Skin irritation, nausea, vomiting PHS alert: Verified PH impacts via DPIC report: Increased hospital cases • DPIC investigation: Suspect toxic chemical Simulation Study Analysis Methodology: Of the 1,545 distribution attack scenarios that reached Possible, 1,313 or 85% of them reached Confirmed. As with Credible determination, the analysis for Confirmed determination utilizing the simulation study results consisted of performing separate statistical analyses for the scenario characteristics: contaminant type, number of components detecting and number of alerts. Simulation Study Results by Contaminant Type: The percentile distribution of time from Possible contamination to Confirmed determination from the analysis of distribution attack scenarios involving nuisance chemicals, toxic chemicals and biological agents is shown in Figure 4-7. As was seen with Credible determination, the analysis by contaminant type for Confirmed determination shows a significant difference between the nuisance chemicals, toxic chemicals and biological agents relative to the timeliness of detection. The toxic chemicals are determined Confirmed much more quickly than the biological agents and nuisance chemicals. The significant difference in the Confirmed determination timeline is most likely due to the differences between the contaminant types in time for the onset of symptoms and/or laboratory confirmation. 26 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Nuisance Chemicals Toxic Chemicals Biological Agents = 528 n=149 x=159 n=1034 x=900 nn=130 200 400 600 800 1000 Time (minutes) 1200 1400 1600 Figure 4-7. Time from Possible to Confirmed Determination by Contaminant Type Simulation Study Results by Number of Components Detecting: The percentile distribution of time from Possible contamination to Confirmed determination for distribution attack scenarios where one, two or three different monitoring and surveillance components detected the contaminant is shown in Figure 4- 8. As was seen with Credible determination, the Confirmed determination results indicate that an increase in the number of components detecting the contamination causes a decrease in the Confirmed determination time span. This observation is supported by the time from Possible contamination to Confirmed determination for three components, which shows a considerably narrower time span compared to the one and two component results. 27 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot 1 Component 2 Components 3 Components C x = 531.5 H 1 n=524 x= 156.5 H. n _ -7-1 0 x = 77 ^ "'71 ) 100 200 300 400 500 600 700 800 900 1000 Time (minutes) Figure 4-8. Time from Possible to Confirmed Determination by Number of Components Detecting Simulation Study Results by Number of Alerts Received: The percentile distribution of time from Possible contamination to Confirmed determination for distribution attack scenarios based on the number of alerts received from three monitoring and surveillance components (WQM, CCS and/or PHS) is shown in Figure 4-9 and Table 4-5. The advantage that an increase in the number of alerts detecting the scenarios has on the time to Confirmed determination is significantly different from that seen with the Credible threat level determination timeline. Here, the median time data does not show a consistent significant reduction in the timeline until more than 10 alerts are received. However, the data does show a wide variability in Confirmed determination times. 28 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot 6 Alerts 5 Alerts 4 Alerts 2 Alerts 1 Alert C i 1 Hi "-11 i | ||i n 11 1 fl 1 14 i U| i n 14 i li n 5 1 \.n-7 [H n = 6 \- < n=4 n 1 S^«"22 ) 100 200 300 400 500 600 700 800 900 1000 Time (minutes) Figure 4-9. Time from Possible to Confirmed Determination by Number of Alerts Received Table 4-5. Number of Alerts Received-Times for Confirmed Determination Number of Alerts 1 2 3 4 5 6 7 8 9 10 11 -100 >100 Median (x ) Time to Confirmed Determination (minutes) 807 746 742 777 804 552 547 706 870 874 166 168 4.1.2 Response A ctions The amount of time required for response actions is based on the utility's decisions regarding implementation of those actions, the time it takes to act on them and the time for them to take effect. For the exercises, response times were quantified for a variety of activities, including the time to notify response partners that a Possible contamination incident existed, the time following a Possible contamination incident to decide on appropriate operational responses and the time to make public notification of water use restrictions. In the simulation study analysis, the time to notify response partners that a Possible contamination existed was not a feature. However, the time to public health response was determined and the results were presented with the time to notify response partners. Additionally, the simulation study analysis quantifies the time to the first operational response, not all of the responses 29 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot individually, as well as the time to a stop use public notification. Notification of Response Partners Definition: For exercise analysis, this metric represents the elapsed time from the determination of a Possible contamination incident to the notification of response partners. For simulation study analysis, the metric is the elapsed time from the determination of a Possible contamination incident to the time of public health responses (e.g. issuance of prophylaxis, increasing hospital bed capacities). Exercise Analysis Methodology: The analysis of the time to notify response partners was accomplished by compiling the elapsed time recorded in AARs from one SC CCS drill and two of the three FSEs. Other SC drill AARs were also examined, but those scenarios did not require notification of response partners. Exercise Results: Response partner notification time results are shown in Table 4-6. The specific times at which individual response partners were notified of a Possible contamination incident varied with the details of the contamination scenarios being exercised. Not all response partners participated in all exercises, or necessarily in the same sequence. The general observation can be made that the overall response partner notification time decreased significantly from FSE 2 to FSE 3. Table 4-6. Time to Notify Response Partners Exercise FSE 1 FSE 2 FSES SC CCS Drill Time (minutes) N/A* 79 47 89 254 91 194 131 428 228 34 16 47 179 170 Response Partner Notified City of Cincinnati Manager OEPA, ODH, CHD, HCPHD DPIC CFD (HazMat)/CPD Water Information Sharing and Analysis Center Northern Kentucky Water, Western Water, Butler Warren County (ISAC) County, EPA Region 5 Ohio Governor's Office City of Cincinnati Manager OEPA CHD DPIC, HCPHD CFD (HazMat)/CPD CFD (HazMat) Notes: This metric was not collected for this exercise Simulation Study Analysis Methodology: The simulation study does not determine the time to notify response partners; however, the time to public health response was analyzed. Of the 1,545 distribution attack scenarios that reached Possible, 1,151 or 74% had a public health response. As with Credible and Confirmed determination, the analysis for time to public health response utilizing the simulation study results consisted of performing separate statistical analyses for the scenario characteristics: contaminant type, number of components detecting and number of alerts. 30 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Simulation Study Results by Contaminant Type: The percentile distribution for time from Possible to public health response from the analysis of distribution attack scenarios involving nuisance chemicals, toxic chemicals and biological agents is shown in Figure 4-10. As was seen with Credible and Confirmed determination, the analysis by contaminant type shows a significant difference between nuisance chemicals, toxic chemicals and biological agents relative to the time to public health response. Toxic chemicals elicit a response by the public health agencies much more quickly than biological agents. This difference in time to public health response was most likely due to the differences between the contaminant types with respect to the time for public health community awareness and how long it took to identify the contaminant. x=168 Nuisance Chemicals Toxic Chemicals Biological Agents L n = 94 x=52 = 703 = 886 n=354 500 1000 1500 2000 Time (minutes) 2500 3000 Figure 4-10. Time from Possible Contamination to Public Health Response by Contaminant Type Simulation Study Results by Number of Components Detecting: The percentile distribution of time to Possible to public health response from the analysis distribution attack scenarios involving one, two or three different components detecting the contaminant is shown in Figure 4-11. The significant advantage that an increasing number of components provide is seen in the time to public health response results timeline where a narrowing of the range and a reduction in median time from one, two and three components are evident. 31 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot 1 Component = 461 n=409 x=126 2 Components n=671 3Components D = 52 n=71 500 1000 1500 Time (minutes) 2000 2500 Figure 4-11. Time from Possible to Public Health Response by Number of Components Detecting Simulation Study Results by Number of Alerts Received: The percentile distribution for time from Possible to public health response for distribution attack scenarios based on the number of alerts received from three of the monitoring and surveillance components (WQM, CCS and/or PHS) is shown in Figure 4-12 and Table 4-7. The time to public health response shows a modest downward trend as the number of alerts received increases from 2 to 10. However, the size of the boxes and length of whiskers in Figure 4-12 illustrate how variable the data are within this range. As was seen with Confirmed determination, having more than 10 alerts significantly decreases the median time to public health response. >100 Alerts i-l 11-100 Alerts hTT -i n = 700 -i n = 379 -i n= 12 | — i n= 1 -i n= 6 -i n= 4 -i n=5 n=7 1 Alert n=0 500 1000 1500 Time (minutes) 2000 2500 Figure 4-12. Time from Possible to Public Health Response by Number of Alerts Received 32 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Table 4-7. Number of Alerts Received - Time to Public Health Response Number of Alerts 2 3 4 5 6 7 8 9 10 11 -100 >100 Median (x ) Time to Public Health Response (minutes) 869 870 0* 878 708 706 553 457 462 45 160 *Note: Within the simulation study results, there were only five scenarios that had four alerts received prior to public health response time. For these five scenarios, the elapsed time was 0, 0, 0, 1,699, and 8,960 minutes. Mathematically, the median is 0, however the range displayed on the graph more accurately illustrates the expected value for this metric. Identification of Utility Operational Response Actions Definition: For exercise analysis, identification of utility operational response actions is measured by the average elapsed time from the determination of a Possible contamination incident to the identification of and decision to implement appropriate operational responses. For simulation study analysis, it is the elapsed time from the determination of a Possible contamination incident to implementation of the first operational response. Exercise Analysis Methodology: The analysis of time taken by the Cincinnati pilot to identify appropriate operational response was recorded in two of the FSE AARs (FSE 2 and FSE 3). These operational responses included a variety of actions for controlling water flow in the system, such as changing the valve patterns and starting up or shutting down pump facilities. Exercise Results: Table 4-8 shows the average elapsed time to identify the operational response action for each threat level phase. In this case, the average was used due to the number and different types of operational responses that were selected by utility personnel to implement. These actions involved taking pump stations out of service, putting others in service, changing valve configurations, etc. The actions were assumed to be implemented almost immediately after decisions were made. As indicated in the table, the average time for identifying appropriate operational response actions during specific threat levels in simulated contamination incidents varied from 25 minutes to 125 minutes, but these times were clearly influenced by the circumstances presented by the scenario. It was demonstrated throughout the FSEs that the utility could implement operational responses very early in the incident investigation process, and then re-evaluate and modify those responses as additional investigation information became available. The GCWW drinking water distribution system is configured such that major changes in water flow can be implemented without disrupting service to the customers. Whenever utility personnel could isolate, redirect, or slow the transmission of suspected contamination in the system without disrupting service to the customers, they did so. Thus, the elapsed time from the receipt of a Possible contamination incident to the decision to implement an operational response such as valving or pump changes was relatively short. 33 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Table 4-8. Time to Identify Operational Response Actions Threat Level FSE2 FSE3 Possible Number of Operational Response Actions 1 5 Time to Identify Response Actions (average, minutes) 38 25 Credible Number of Operational Response Actions 1 0 Time to Identify Response Actions (average, minutes) 125 N/A1 Confirmed Number of Operational Response Actions 0 3 Time to Identify Response Actions (average, minutes) N/A2 72 Notes: 1. For FSE 3, when the Credible determination was made, Cincinnati pilot personnel decided that the ongoing operational actions implemented during the Possible phase were sufficient to contain the contamination, while continuing to provide service. 2. For FSE 2, no time was projected since the exercise ended exactly at the time of Confirmed determination. Simulation Study Analysis Methodology: Of the 1,545 distribution attack scenarios that reached Possible, 1,253 or 81% had an operational response. As with previous metrics, the analysis for time to the initial operational response utilizing the simulation study results consisted of performing separate statistical analyses for the scenario characteristics: contaminant type, number of components detecting and number of alerts. Simulation Study Results by Contaminant Type: The percentile distribution for time from Possible to the time of operational response from the analysis of distribution attack scenarios involving for nuisance chemicals, toxic chemicals and biological agents is shown in Figure 4-13. The operational response data for all contaminant type scenarios are similar, showing quick implementation and a very narrow distribution of time since Possible determination. As such, the contaminant type does not show any meaningful impact on the time it takes to make operational changes. 34 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Nuisance Chemicals Toxic Chemicals Biological Agents x=15 n=103 = 20 n=875 = 20 = 275 10 15 Time (minutes) 20 25 Figure 4-13. Time from Possible to Operational Response by Contaminant Type Note: Within the simulation study results, all but three of the 275 scenarios involving biological agents had an elapsed time since Possible of 20 minutes. For these three scenarios, the elapsed time was 11, 13, and 15 minutes. Therefore, the percentiles illustrated by the "box and whisker" plot are all at the median value of 20. Simulation Study Results by Number of Components Detecting: The percentile distribution for time from Possible to the time of operational response from the analysis of distribution attack scenarios involving one, two, or three different components detecting the contaminant is shown in Figure 4-14. Similar to contaminant type, the operational response data for the number of components detecting is similar, showing quick implementation and a narrow distribution of time since Possible determination. As such, the number of components detecting contamination does not show any meaningful impact on the time it takes to enact initial operational changes. 35 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot 1 Component SComponente C x = 20 n=568 x=15 i n — u i u x=12 J n = 69 5 10 15 20 25 Time (minutes) Figure 4-14. Time from Possible to Operational Response by Number of Components Detecting Note: Within the simulation study results, all but 19 of the 568 scenarios for one component detecting had an elapsed time since Possible of 20 minutes. For these 19 scenarios, the elapsed times were less than 20 minutes (ranging from 8 to 15 minutes). Therefore, the percentiles illustrated by the "box and whisker" plots are all at the median value of 20. Simulation Study Results by Number of Alerts Received: Percentile distributions for operational response times for distribution attack scenarios based on the number of alerts received from three monitoring and surveillance components (WQM, CCS and/or PHS) are shown in Figure 4-15 and Table 4-9. The median time to operational response was identical (20 minutes) for all the number of alerts that occurred, indicating that the model does not reflect an advantage with respect to timeliness of operational response in having more alerts. 36 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot 11-100 Alerts 10 Alerts 9 Alerts 8 Alerts 7 Alerts 6 Alerts 5 Alerts 4 Alerts 3 Alerts 2 Alerts 1 Alert 1 , 1 1 5 10 15 Time (minutes) n=340 1 — ' n= 10 1 n= 14 n= 9 n= 6 n= 1 1 n=6 1 n=7 i n= 9 i n= 10 n= 1 20 25 Figure 4-15. Time from Possible to Operational Response by Number of Alerts Received Table 4-9. Number of Alerts Received - Time to Operational Response Number of Alerts 1 to > 1 00 (All values simulated) Median (x ) Time to Operational Response (minutes) 20 Time Required for Public Notification Definition: For exercise analysis, this is the time from when the utility PIO was first instructed to prepare public notification (e.g., do not use) through the time it was released. It includes the time necessary for the drafting of the document, revision of the document, and coordination and approval by appropriate agencies. It does not include the time spent on the preparation of employee notifications or media/press conference materials. For simulation study results analysis, time required for public notification is the time from the determination of a Possible contamination incident until the time the public notification was released. Exercise Analysis Methodology: Time required for public notification was extracted from the AARs of the three FSEs. Exercise Results: Table 4-10 shows public notification times as documented from the FSEs. The average time required for preparation, revision and approval for release of a public notification was 169 minutes. The public notification language was prepared collaboratively among the Cincinnati utility, local public health and Ohio EPA personnel during conference calls as the scenario unfolded. Preparation time varied with the scope of the scenarios, as multiple alerts tended to require more frequent updating of the information covered in the notification, and more iterations of review and approval. 37 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Table 4-10. Time to Prepare and Issue Public Notification Exercise FSE 1 FSE2 FSE 3 Average Time to Prepare and Issue Public Notification (minutes) 180* 162 165 169 *Note: Actual direction to initiate was not recorded; assumed to have started after the incident was Possible. Significant modifications occurred to CCP and the public notification process following the FSEs, including organizational changes to the ICS which added resources (PIO and Customer Information Manager) to the public information function of the CM component. Simulation Study Analysis Methodology: Of the 1,545 distribution attack scenarios that reached Possible, 1,315 or 85% had a public notification response. As with previous metrics, the analysis for time to public notification utilizing the simulation model consisted of performing separate statistical analyses for the scenario characteristics: contaminant type, number of components detecting, and number of alerts. Simulation Study Results by Contaminant Type: The percentile distribution for time from Possible to public notification from the analysis of distribution attack scenarios involving nuisance chemicals, toxic chemicals and biological agents is shown in Figure 4-16. The analysis shows a difference for time to public notification between the toxic chemicals, biological agents and nuisance chemicals relative to the timeliness of detection. Based on the simulation study analysis, toxic chemicals result in public notification much more quickly than nuisance chemicals and biological agents. The differences in public notification time was most likely due to the differences between the contaminant types' effect on taste, odor, or appearance of the drinking water and whether the contaminant triggered customer complaints or progression of symptoms leading to healthcare seeking behavior. Nuisance Chemicals Toxic Chemicals Biological Agents = 528 n=149 x=140 n=1034 = 623 H n= 132 0 200 400 600 800 1000 1200 Time (minutes) Figure 4-16. Time from Possible to Public Notification by Contaminant Type 38 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Simulation Study Results by Number of Components Detecting: The percentile distribution for time from Possible to the time of public notification from the analysis of distribution attack scenarios with one, two, or three different components detecting the contaminant is shown in Figure 4-17. The figure demonstrates that an increase in the number of components results in a slight decrease in the median time to public notification, but never less than 120 minutes. This is driven by a specific model parameter that requires 120 minutes to develop a public notification while the decision to release the public notification is made independently. For scenarios with only one or two components detecting, it takes longer for the utility to acquire the information needed to determine whether or not to release the public notification than it takes to actually develop it. When three components have detected contamination, the utility has sufficient information to release the public notification as soon as it is ready. 1 Component 2Componente SComponente C x=153 i in- ^"M x=133 1 /; - / 20 x=120 i n=71 100 200 300 400 500 600 700 800 Time (minutes) Figure 4-17. Time from Possible to Public Notification by Number of Components Detecting Note: Within the simulation study results, all but 2 of the 71 scenarios involving 3 components detecting had an elapsed time since possible of 120 minutes. For these 2 scenarios, the elapsed time was 162 and 177 minutes. Therefore, the percentiles illustrated by the "box and whisker" plots are all at the median value of 120. Simulation Study Results by Number of Alerts Received: Percentile distributions for public notification times for distribution attack scenarios based on the number of alerts received from three monitoring and surveillance components (WQM, CCS, and/or PHS) are shown in Figure 4-18 and Table 4-11. The advantage that an increasing number of alerts has on the time to public notification is clearly demonstrated by a significant reduction in the time from only one alert being received to six alerts being received. There is no significant advantage in an increasing number of alerts with respect to timeliness of for the time to public notification beyond six alerts. 39 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot > 100 Alerts 11-100 Alerts 10 Alerts 9 Alerts 8 Alerts 7 Alerts 6 Alerts 5 Alerts 4 Alerts 3 Alerts 2 Alerts 1 Alert C f~H-i "=295 iffl i i 11 iyj inn JTl-i n= 14 i fl 1 i n- 10 ' U I 4E- n=5 O-7 H h=6 i | | i n , i Fl i n-T III | [i n=20 ) 100 200 300 400 500 600 700 800 900 1000 Time (minutes) Figure 4-18. Time from Possible to Public Notification by Number of Alerts Received Table 4-11. Number of Alerts Received - Time to Public Notification Number of Alerts 1 2 3 4 5 6 7 8 9 10 11 -100 >100 Median (x ) Time to Public Notification (minutes) 807 746 741 736 448 180 149 149 144 153 144 146 4.1.3 Remediation and Recovery None of the FSEs included R&R objectives due to scope and time constraints. Thus, response time metrics used to evaluate this aspect of the CM component were never generated. However, several workshops were conducted with the Cincinnati pilot that resulted in multiple changes to the R&R decision logic in the Cincinnati Pilot Consequence Management Plan. The original R&R decision tree incorporated into early drafts of the Cincinnati Pilot Consequence Management Plan was modeled after the one developed for EPA's Response Protocol Toolbox, which incorporated many aspects of the Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA) remediation process. The utility later participated in a national R&R workshop co- 40 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot sponsored by the American Water Works Association (AWWA) and EPA, conducted on March 17-18, 2009. Twenty-nine participants representing water utilities, EPA, U.S. Army Corps of Engineers, the Department of Homeland Security (DHS), the Federal Emergency Management Agency (FEMA) and other federal agencies convened to discuss the procedures and action associated with R&R from a Confirmed drinking water contamination incident. The objectives of the workshop included reviewing and refining (as appropriate) the R&R process that was outlined in the U.S. EPA Water Interim Guidance on Developing Consequence Management Plans for Drinking Water Utilities (EPA, 2008), discussing utility interaction with the National Response Framework, and discussing an outline for guidance on containment/disposal of decontamination waste. As a follow-up to the national workshop, the Cincinnati pilot conducted an internal R&R workshop on May 14, 2009, to review their R&R decision logic relative to the results of the previous national workshop. The major outcome of this workshop was the adoption of the national workshop R&R decision process. A final R&R workshop was conducted on May 21, 2010 to discuss the revised R&R process with local response partner agencies. This was a follow-up activity related to FSE 3 where the exercise scenario was used to facilitate discussion concerning probable R&R activities and corresponding roles and responsibilities. The workshop resulted in the addition of 14 response partners and 60 modifications and/or clarifications of response partner roles and responsibilities in the Cincinnati Pilot Consequence Management Plan. In addition, participants agreed that the revised R&R decision tree: 1) contributed significantly to utility and response partner understanding of the R&R process, 2) were compatible with existing utility and response partner response plans, and 3) would facilitate the R&R process in the event of a contamination incident. 4.1.4 Summary There were no statistical trends demonstrated from the exercise data for the timeliness of detection and response for the incident response plans. The variability seen in the data for investigative and response actions were a direct result of the variations presented by the contamination scenarios themselves. Although this was the case, the design elements evaluated as a part of this design objective did reveal several significant observations. The simulation study analysis demonstrated clear trends for timeliness of detection and response actions in reaction to the scenario attacks evaluated. However, some of the metrics analyzed showed inconsistent results or results that were driven by the model parameters. Credibility Determination Exercise Investigative Actions: For Credible and Confirmed determination, no correlation could be made between the number of alerts received and the amount of time that was required for utility personnel to determine whether the incident was Credible or Confirmed. Overall, a hierarchy of investigation information types evolved, which seemed to accelerate the speed with which threat level determinations were made, including: • Multiple alerts, system connectivity, positive RFTs and signs of intrusion accelerated the declaration of Credible contamination incidents and • Signs of intrusion and health impacts accelerated the declaration of Confirmed contamination incidents and did not necessarily depend on positive laboratory analysis. 41 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Simulation Study Investigative Actions: For Credible and Confirmed determination, a strong correlation was observed when evaluated by the type of contaminant introduced and the number of components detecting the contaminant: 1. The model results showed significant differences for threat level determination for toxic chemical contaminants compared to biological agents. The toxic chemical contaminants resulted in much quicker threat level determination. For Credible determination, the difference in timeliness is most likely due to the contaminant types' effects on taste, odor, or appearance of the drinking water resulting in customer complaints or progression of symptoms. For Confirmed determination, the difference in timeliness is most likely due to the same factors that drive Credible determination as well as being able to be analyzed and identified much more quickly. 2. The simulation study also demonstrated a significant advantage provided by an increasing number of different monitoring and surveillance components for a reduction in threat level determination time. Three components detecting results showed a marked advantage in the timeline over two components detecting which generally showed an advantage over one component detecting. The simulation study analysis of the number of alerts on threat level determination was inconsistent for Credible determination compared to Confirmed determination. While the Credible determination results showed a definite advantage at five alerts and above in reducing the timeline, the Confirmed determination results were too inconsistent to draw any conclusions. Response Actions Exercise Investigative Actions: The time required for the utility to notify its response partners varied with the sequence of circumstances presented by the various exercise scenarios, but generally was consistent from FSE 2 to FSE 3, with some slight improvements with key partners including CFD, CHD, DPIC, and OEPA). The average time for identifying appropriate operational response actions during specific threat levels in the exercises varied, but these times were clearly influenced by the circumstances presented by the scenario. Operational responses were initially driven by what actions the utility could implement quickly to isolate or slow contamination without impacting service. As the incident progressed, investigation evidence was subsequently used to revise those response actions as necessary. The time to develop and implement public notification was consistent throughout the exercises, with an average time of 169 minutes from direction to prepare and availability to release. Given the variability of exercise scopes and the accompanying revision of the CCP, it was not possible to make statistical inferences concerning the performance. Simulation Study Investigative Actions: The simulation study results for time from Possible to public health response showed a strong correlation to the contaminant type and to the number of components detecting metrics. A comparison of the toxic chemical to the biological agents indicated that toxic chemical contaminants resulted in much quicker implementation of public health response and a much narrower time span. The difference in time to public health response was most likely due to differences between contaminant types with respect to the time for the rapid onset of symptoms, public health community awareness, and variations in the time taken to identify the contaminant. The simulation study results for time to the first operational response were very similar for contaminant type, number of components detecting, and number of alerts metrics. All metric results indicated very quick first operational response implementation times and a very narrow time span. 42 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot The simulation study results for time to public notification showed a correlation for the contaminant type, number of components detecting and number of alerts metric results. A comparison of the toxic chemical and biological agent results showed that toxic chemical contaminants resulted in much quicker public notification over a very narrow time span. As indicated earlier, this was most likely due to differences between the contaminant types' effect on taste, odor, or appearance of the drinking water and whether the contaminant triggered customer complaints or progression of symptoms leading to healthcare seeking behavior. The results for the number of components detecting showed a slight advantage for three components detecting followed by two components detecting with a slight advantage over one component detecting. Finally, the number of alert results impact on the time to public notification showed a significant reduction in the public notification time at six alerts and above. R&R Several workshops resulted in significant reduction of the steps in the R&R decision tree and corresponding response actions. These workshops also resulted in the addition of 14 response partner agencies to the R&R process, and more than 60 clarifications to the R&R roles and responsibilities. The R&R process was never tested and evaluated through the FSEs and therefore no metric data concerning timeliness of response was collected. 43 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Section 5.0: Performance of Integration of the Response Partner Network The response partner network design element was meant to provide a framework upon which the Cincinnati pilot could coordinate their respective response actions associated with a contamination incident. The overall goal of the network was to understand, integrate and achieve consensus regarding roles and responsibilities during the CM process. The inclusion of the response partner network as a design element of the CM component was intended to streamline the overall response process. Measurement of the performance of this design element consisted of evaluating the effectiveness of establishing and understanding roles and responsibilities, quantifying how well the network was integrated into the CM component, and describing how the nature of the support provided by the network changed during the pilot evaluation period. 5.1 Design Objective: Timeliness of Response While there is no empirical data to describe how this design element contributes to timeliness of detection and response by itself, it is assumed that the integration of the response partner network into the CM activities increased the efficiency of overall response activities. The performance of the individual design element for this objective specifically focuses on how well the response partner network was developed and implemented, and how it evolved during the pilot study. 5.1.1 Understanding Response Partner Roles and Responsibilities One of the earliest activities of the pilot study was to conduct a series of workshops with the Cincinnati pilot to elicit input on roles and responsibilities that would be integrated into the CM component. Evaluation criteria were then included in several subsequent exercises to determine how well the Cincinnati pilot understood their respective roles and responsibilities. Definition: This metric consists of the number of improvement recommendations associated with the Cincinnati pilot that are associated with the understanding of roles and responsibilities. Analysis Methodology: Improvement recommendations contained in the AARs from exercises, specifically based on improving the response partner network, were analyzed for this metric. These improvement recommendations involved changes to the Cincinnati Pilot Consequence Management Plan and/or implementation of training. Results: Major improvement recommendations generated from each of the exercises are described in Table 5-1. There was no statistical trend in the number or type (plan vs. training) of improvement recommendations found in the exercise AARs, although improvement was seen between the functional exercise (which was conducted first) and the subsequent FSEs. The majority of the response partner network issues reflected in the functional exercise involved the need to improve coordination and communication between the utility and its response partners, which was not an outstanding issue during the FSEs. 44 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Table 5-1. Response Partner Recommendations by Exercise Exercise Functional Exercise FSE 1 FSE2 FSE 3 Total Number of Recommendations 12 (4 CMP-related only; 8 both CMP and Training- related) 4 (1 CMP-related only; 3 both CMP and Training- related) 4 (1 CMP-related; 3 Training-related) 1 (1 Training- related) Description • Improve communication and coordination between utility and multiple response partners • Clarify the point of contact for the HazMat team • Modify the list of response partner contacts and the timing of external agency notification to allow for flexibility • Improve communication between utility and multiple response partners • Add DPIC as a toxicology resource • Modify the list of response partner contacts and the timing of external agency notification to allow for flexibility • Improve HazMat team access to utility facilities in emergency situations • Reconcile sampling protocol differences between the utility SCT and HazMat teams • Reconcile mission turnover procedures from the utility SCT to the HazMat team • Mark sampling locations at utility facilities for easy identification by Hazmat teams • Increase cross-training activities with HazMat teams • Clarify utility protocols (standard language) for 91 1 requests • Clarify response partner responsibilities during evacuation actions 5.1.2 Integration of Response Partners Local, state, regional and federal response partner agencies played an integral role in the CM component. Integration of the response partners into the initial development of the CM component, the reconciliation of incident response plans and the eventual evaluation of those roles and responsibilities in exercises was crucial to the success of the CM component. This element of the design objective addressed how well the response partners were integrated into the component. Definition: The degree to which the response partners were integrated into the CM component is measured in terms of the number of times each response partner participated in the development, execution or evaluation of a CM component exercise. The voluntary participation of response partners in CM events indicates their willingness to actively engage in CM activities to improve overall response to contamination incidents. Analysis Methodology: The AARs for all exercises that involved response partner participation were examined. Results: The response partners that participated in each CM related exercise are shown in Table 5-2. Overall, response partner integration into the CM component was both extensive and continuous, beginning with the earliest workshops and continuing through the last FSE. Table 5-2. Response Partner Exercise Participation Response Partner Cincinnati Fire Dept. (CFD)/HazMat Cincinnati Managers Office Cincinnati Police Department (CPD) CM Workshops X X X SC Drill 2 X Functional Exercise X X X FSE1 X X FSE 2 X X FSE 3 X FSE 3 Workshop 45 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Response Partner County Wide Fire Reps Metropolitan Sewer District (MSD) Cincinnati Health Department (CHD) Hamilton County Public Health Department (HCPHD) Drug and Poison Information Center (DPIC) Hamilton County Emergency Management Agency (HCEMA) Ohio Department of Health (ODH) Ohio Environmental Protection Agency (OEPA) Federal Bureau of Investigation (FBI) U.S. EPA CM Workshops X X X SC Drill 2 X Functional Exercise X X X X X X X X FSE1 X X X X X X FSE2 X X X X X X X FSE3 X* X X X FSE3 Workshop X X X X X Notes: Blank cells indicate that the partner did not participate; grey cells indicate that participation was not required. * The CHD response partner role was assumed by an EPA employee who was a former HCPHD employee. 5.1.3 Nature of Response Partner Support This aspect of response partner integration addresses the nature of the support provided to the Cincinnati pilot. The different types of support included law enforcement, hazardous materials management, emergency operations, laboratory analysis, technical support from health agencies and regulatory support. Definition: The nature of the partner support provided to the Cincinnati pilot is defined by characterizing the type of the support as active responder, active support or passive support. Analysis Methodology: Actions performed by response partners during exercises were compiled from the AARs and classified as active responder, active support or passive support using the following criteria: • Active Responder - response partner requested to fully or partially assume a primary responsibility in the Cincinnati Pilot Consequence Management Plan. Example: Local HazMat team on scene conducting sampling operations. • Active Support - response partner implementing its individual role and responsibility to support the utility. Example: DPIC, OEPA, CHD providing public health, toxicity, or regulatory support during an investigation. • Passive support - all other response partners that have been notified of an incident investigation, but are not actively providing support. Example: MSD or Cincinnati Managers Office. Results: Table 5-3 displays the results of the tabulation. In general, the type of support provided each response partner agencies did not vary with the circumstances presented by the exercise scenarios. The one exception is the FBI for FSE 3 (grey shaded cell) because there were no signs of intrusion during the scenario and therefore the FBI would not have had an investigation role prior to the incident being confirmed. 46 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Table 5-3. Nature of Response Partner Support Response Partner Cincinnati Fire Dept. (CFD)/HazMat Cincinnati Managers Office Cincinnati Police Department (CPD) County Wide Fire Reps Metropolitan Sewer District (MSD) Cincinnati Health Department (CHD) Hamilton County Public Health Department (HCPHD) Drug and Poison Information Center (DPIC) Hamilton County Emergency Management Agency (HCEMA) Ohio Department of Health (ODH) Ohio Environmental Protection Agency (OEPA) Federal Bureau of Investigation (FBI) U.S. EPA FSE1 AR PS AS PS PS AS AS AS AS AS AS PS FSE2 AR PS AS PS PS AS AS AS AS AS AS PS FSE3 AR PS AS PS PS AS AS AS AS AS PS PS Notes: AR - Active Responder; AS - Active Support; PS - Passive Support 5.1.4 Summary The inclusion of the response partner network as a design element of the CM component was intended to streamline the overall response process. Although there was no empirical data to describe how this design element contributed to timeliness of detection and response by itself, the design elements evaluated as apart of this design objective did reveal several observations: a. Understanding Response Partners Roles and Responsibilities. There was no statistical trend in the number or type of improvement recommendations found in the exercises, although improvement was seen between the functional exercise and the subsequent FSEs. The majority of these recommendations focused on the need to improve coordination and communication between the utility and its response partners, which was not an outstanding issue during the FSEs. This improvement indicated that the utility and response partners were achieving a better understanding of their CM-related roles and responsibilities as well as improving communication and coordination. b. Integration of Response Partners. A total of twelve response partner agencies with various roles and responsibilities were extensively involved with the Cincinnati pilot CM component. Eleven of the response partner agencies were involved from the earliest phase of development through the conclusion of the evaluation period. DPIC was recognized early on as a vital missing response agency and quickly engaged. These response partner agencies provided the Cincinnati pilot a variety of significant CM support including law enforcement, hazardous materials management, emergency operations, laboratory analysis, technical support from health agencies and regulatory support. Overall, the different types of support provided by these response partners contribute to a well-rounded response. 47 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Section 6.0: Performance of Communication Equipment During the pre-implementation assessment of the CM component, it was noted that utility field response teams did not have an established way to communicate among utility team members, with response partners (e.g., HazMat) or between the utility and the ICS. This hindered the organization's ability to efficiently communicate and coordinate response to a contamination incident. Prior to implementation of the CM component, field response communications within the utility would have occurred using a facility phone, a cell phone or over a utility radio. Communications outside the utility would have occurred using a facility phone or cell phone. Eight 800 MHz hand-held radios (Motorola XTS 5000) were acquired to address this deficiency. The 800 MHz radios are long-range multi-channel programmable units that are inter-operable with response partner agencies in the City of Cincinnati and Hamilton County. The use of designated frequencies allows utility personnel to communicate both internally and with response partners including fire, police, health and other city workers. The radios were also programmed to operate on other agency frequencies in the event of a Unified Command System incident response. The radios are located and deployed with the utility SCT Leader. 6.1 Design Objective: Timeliness of Response 6.1.1 Use of 800 MHz Radios While there is no empirical data to document how the use of the 800 MHz radios reduced the time to investigate possible contamination incidents and implement response actions, evaluators observed during exercises that the radios greatly facilitated communication between the utility ICS and field response personnel. In addition to the exercises, the 800 MHz radios were used during the response to the September 2008 wind storm and resulting power outage to address communications during the outage. The radios were used by utility field crews and supervisors primarily because cellular phones did not consistently work, and the radio system in utility cars was also down for several days. Using the 800 MHz radios during the exercises also resulted in the identification of dead spots in the coverage area and radio interference from some pumping facilities. Steps to eliminate these problems were under consideration at the end of the evaluation period. In addition, the utility established an internal secure website (SharePoint) to facilitate communications among incident command personnel with access to a network computer. The consensus presented in the AARs was that the use of SharePoint site to communicate real-time information among the utility ICS increased the overall efficiency of response management. 6.1.2 Summary The integration of the 800 MHz radios and SharePoint site into the utility CM related response activities enhanced their ability to communicate information both internally and externally. Data and feedback from the exercises indicated that the use of the communication equipment allowed faster and more complete investigation of contamination incidents. By the end of the pilot evaluation period, utility personnel routinely used the communication equipment for contamination incident investigations. 48 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Section 7.0: Sustainability Sustainability is a key objective in the design of a CWS and each of its components. For the purpose of this evaluation, Sustainability is defined in terms of the cost-benefit trade-off. Costs are estimated over the 20 year lifecycle of the CWS and include the capital cost to implement the CWS and the cost to operate and maintain the CWS. The benefits derived from the CWS are defined in terms of primary and dual-use benefits. The primary benefit of a CWS is the potential reduction in consequences in the event of a contamination incident. However, such a benefit may be rarely, if ever, realized. Thus, dual-use benefits play an extremely important role that provide value to routine utility operations and are an important driver for Sustainability. Ultimately, Sustainability can be demonstrated through utility and partner compliance with the protocols and procedures necessary to operate and maintain the CWS. The two metrics that were evaluated to assess how well the Cincinnati CWS met the design objective of Sustainability are: Costs and Benefits. The following subsections define each metric, describe how it was evaluated and present the results. 7.1 Costs Definition: Costs are evaluated over the 20 year lifecycle of the Cincinnati CWS, and comprise costs incurred to design, deploy, operate and maintain the CM component since its inception. It should be recognized that the Cincinnati CWS was a pilot research project and as such, likely incurred costs higher than another utility would realize. Analysis Methodology: Parameters used to quantify the implementation cost of the CM component were extracted from the Water Security Initiative: Cincinnati Pilot Post-Implementation System Status (USEPA, 2008). Implementation costs include labor and other expenditures (equipment, supplies, and purchased services) for designing and deploying the CM component. O&M costs were tracked on a monthly basis over the duration of the evaluation period. Renewal and replacement costs, along with the salvage value at the end of the Cincinnati CWS lifecycle were estimated using vendor supplied data, field experience and expert judgment. Section 3.4 provides additional details regarding the methodology used to estimate each of these cost elements. Results: The methodology described in Section 3.4 was applied to determine the value of the major cost elements used to calculate the total lifecycle cost of the CM component, which are presented in Table 7- 1. It is important to note that the Cincinnati CWS was a research effort, and incurred higher costs than would be expected for a typical large utility installation. A similar CM component implementation at another utility should be less expensive as it could benefit from lessons learned and would not incur research-related costs. Additional information regarding the data used to determine the value each cost element is presented below. Table 7-1. Cost Elements used in the Calculation of Lifecycle Cost Parameter Implementation Costs Annual O&M Costs Renewal and Replacement Costs Salvage Value Value $1,430,627 $33,948 $22,624 - Table 7-2 below presents the implementation cost for each CM design element, with labor costs presented separately from the cost of equipment, supplies and purchased services. 49 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Table 7-2. Implementation Costs1 Design Element Project Management2 Incident Response Plans Response Partner Network Communication Equipment Training and Exercises TOTAL: Labor $102,749 $712,828 $153,527 $16,338 $420,734 $1,406,175 Equipment, Supplies, Purchased Services - $793 - $23,584 $74 $24,451 Total Implementation Costs $102,749 $713,621 $153,527 $39,922 $420,808 $1,430,627 1 All numbers rounded to the nearest dollar. 2 Project management costs incurred during implementation were distributed evenly among the CWS components. The first design element, project management, includes overhead activities necessary to design and implement the component. The incident response plans design element includes the cost of developing a Cincinnati Pilot Consequence Management Plan detailing roles and responsibilities as well as developing a preparedness and response guide. A CCP formalizing public notification procedures and guidance for GCWW Public Information Officer was also developed. The third design element, response partner network, includes the cost of identifying response partners and gathering their input as to roles and responsibilities in dealing with a water contamination incident. The fourth design element, communication equipment, includes the cost of purchasing eight 800 MHz hand held radios to improve GCWW's ability to respond to an incident and to communicate and coordinate appropriately with response partners in the field. The fifth design element, training and exercises, includes the cost of designing and executing workshops, tabletop exercises, functional exercises, drills and full scale exercises to test the Cincinnati Pilot Consequence Management Plan and to train the participants on processes and procedures. Overall, the incident response plans design element had the highest implementation costs (50% of the total). Significant labor costs were involved in developing the Cincinnati Pilot Consequence Management Plan. The total implementation cost for the training and exercises design element were somewhat lower at 29% of the total, but also required significant labor costs for planning, coordinating, and executing system- and component-level drills and exercises. These labor costs also involved developing AARs for each drill and exercise to summarize the contamination scenario, partner actions, response timelines and areas for improvement. Implementation costs for project management and communication equipment were significantly lower at 7% and 3% of the total, respectively. The annual labor hours and costs of operating and maintaining the CM component, broken out by design element, are shown in Table 7-3. Table 7-3. Annual O&M Costs Design Element1 Procedures TOTAL: Total Labor (hours/year) 771 771 Total Labor Cost ($/year) $33,948 $33,948 Supplies and Purchased Services ($/year) - - Total O&M Cost ($/year) $33,948 $33,948 1 Overarching project management costs were only incurred during implementation of the CM component and are not applicable for annual O&M costs. 50 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Most of the O&M labor hours reported under procedures was spent on ongoing coordination of drills, exercises and trainings to maintain readiness for response to possible water contamination incidents. Figure 7-1 shows the O&M labor hours for each reporting period over the course of the entire evaluation period between January 2008 and May 2010. In this evaluation, the term 'reporting period' is used to refer to a month of metrics data which spans from the 16th of one month to the 15th of the next month. Thus, the January 2008 reporting period refers to the data collected between January 16th 2008 and February 15th 2008. The majority of the reporting periods experienced labor hours across all organizations of less than 400 hours. The three largest labor hour values were recorded during the September 2008, September 2009, and October 2009 reporting periods due to preparation for and completion of the FSEs. The increase during the December 2008 reporting period to 420 labor hours was due to extensive review of the AAR for the previous FSE. Lessons Learned workshops and the R&R workshop accounted for other above- average labor hours reporting periods in April 2009 and May 2010. 1 2 1400 1200 1000 800 600 400 200 GCWW • Response Partner(s) i Contractor .H i11i i iIIi ii i il . Start Date of Monthly Reporting Period Figure 7-1. O&M Labor Hours per Reporting Period Two of the major cost elements presented in Table 7-1, the renewal and replacement costs and salvage value, were based on the costs associated with major pieces of equipment installed for the CM component. The useful life of these items were estimated at 5 years and 10 years, respectively, based on manufacturer-provided data. It was assumed that the items with a useful life of 5 years would need to be replaced three times during the 20-year lifecycle of the CWS, and the items with a useful life of 10 years were assumed to be replaced once. Because the useful life of the final installment of all equipment items will expire at the end of the 20 year lifecycle, there is no salvage value for this component, as reported in Table 7-1. The cost of these items is presented in Table 7-4. 51 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Table 7-4. Equipment Costs Equipment Item Motorola 800MHz LCD 40 inch Flat Panel Monitor Useful Life (years) 10 5 Unit Capital Costs $2,828 $500 Quantity (# of Units) 8 2 TOTAL: Total Cost $22,624 $1,000 $23,624 To calculate the total lifecycle cost of the CM component, all costs and monetized benefits were adjusted to 2007 dollars using the change in the Consumer Price Index between 2007 and the year that the cost or benefit was realized. Subsequently, the implementation costs, renewal and replacement costs, and annual O&M costs were combined to determine the total lifecycle cost: CM Total Lifecyde Cost: $2,000,828 *Actual costs were adjusted to 2007 dollars Note that in this calculation, the implementation costs were treated as a one-time balance adjustment, the O&M costs recurred annually, and the renewal and replacement costs for major equipment items were incurred at regular intervals based on the useful life of each item. 7.2 Benefits Definition: The benefits of CWS deployment can be considered in two broad categories: primary and dual-use. Primary benefits relate to the application of the CWS to detect contamination incidents, and can be quantified in terms of a reduction in consequences. Primary benefits are evaluated at the system-level and are thus discussed in the Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot (USEPA, 2013b). Dual-use benefits are derived through application of the CWS to any purpose other than detection of intentional and unintentional drinking water contamination incidents. Dual-use benefits realized by the CM component are presented in this section. Analysis Methodology: Information collected from forums, such as data review meetings, lessons learned workshops and interviews were used to identify dual-use applications of the CM component of the CWS. Results: Operation of the CM component of the CWS has resulted in benefits beyond the response to intentional and unintentional contamination incidents. These key dual-use benefits and examples identified by the utility include: 1. Stronger interagency relationships with response partners • The close coordination with response partners that is required for the CM component translates to improved coordination during simple non-contamination incidents that can impact the distribution system (e.g., natural disasters). 2. Strengthened incident command structure • Efficient response to contamination incidents relies on a sound command structure to manage multiple utility divisions as well as support from external response partners. Development of the CM component emphasizes Incident Command Systems principles that translate to all types of utility responses from local main breaks to multi- jurisdictional and multi-agency emergencies. 52 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot 3. Increased preparedness of utility management and staff to respond to "all-hazards" • Through active training programs that stress classroom-based and field-based exercises, development of the CM component stresses a step-wise process for response that equips the utility to more effectively respond to "all-hazards". For example, GCWW personnel indicated that they were much more comfortable and confident responding to "real" incidents after successfully performing their response actions during the full scale exercises. Many of the listed dual-use benefits are illustrated in the case study below, which occurred during the evaluation period. Case Study: Response During Hurricane Ike Windstorm On the afternoon of Sunday, September 14, 2008, a severe windstorm associated with the remnants of Hurricane Ike struck the Greater Cincinnati region, resulting in a loss of power to 90 percent of the area. This affected numerous aspects of the utility's operations, including pressure and flow in the distribution system. Many of the utility's pumping stations were without power for a lengthy amount of time and considerable effort was exerted to maintain pressure throughout the system. As a result of the storm, the Cincinnati pilot utilized components of its emergency response plans and the Cincinnati Pilot Consequence Management Plan. Several dual-use benefits were realized as a result of this storm event including: 1. Cincinnati Pilot Consequence Management Plan. The updated version of the Cincinnati Pilot Consequence Management Plan improved the utility's response to the overall system event mainly through the implementation of the ICS structure. Subsequent updates to the Cincinnati Pilot Consequence Management Plan as a result of exercises will enhance the incident response plans and reduce response times to "all-hazard" type emergencies. 2. 800 MHz radios. The 800 MHz radios were used by utility field crews and supervisors primarily because cellular phones did not consistently work. The radio systems in utility's cars were also down for several days. 3. Confirmatory Sampling Field Decision Guide (CSFDG). The Water Quality and Treatment (WQ&T) Division utilized both the CSFDG map of Pito Zones and performed modeling of a service area at approximately 10:00 p.m. on September 14, 2008. This was performed to identify vulnerable locations for pressure monitoring by distribution field crews. 4. Crisis Communications Plan (CCP). Principles within the CM CCP were applied during the response to the wind storm. Although a boil water notice was never issued, a "water conservation" notice was given to customers in several pressure zones as a precautionary measure to discourage water consumption for non-essential uses. 7.3 Summary Sustainability was measured by labor hours for CM implementation, O&M and dual-use benefits. The total lifecycle cost for the CM component, which included implementation, renewal and replacement, and annual O&M costs was $2,000,828. Labor Hours for CM Implementation Developing the Cincinnati Pilot Consequence Management Plan and conducting the exercises were the most significant tasks for the implementation of this component. Both required work with utility personnel as well response partner agencies. Implementation costs approximately amounted to $800,000 though equipment costs were minimal (approximately $40,000). 53 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Operation and Maintenance Labor hours for O&M were steady at approximately 340 hours per month through most of the evaluation period, with the exception of the reporting periods surrounding FSEs. These exercise required substantial planning and participation from numerous utility personnel and response partner agencies. Dual-Use Benefits The Cincinnati pilot CM ICS structure, communication equipment, and crisis communication procedures were used during a severe windstorm that occurred in the Cincinnati area on September 14, 2008. This provided a case study example of the dual-use benefits associated with the CM component. 54 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Section 8.0: Summary and Conclusions The evaluation of the CM component of the Cincinnati pilot CWS involved analysis of empirical data, qualitative observations gleaned from active participants and results from the simulation study. Highlights, limitations and considerations for interpretation of this analysis are presented here. 8.1 Design Objective: Timeliness of Response The overall metrics used to evaluate timeliness of response were derived from three separate design elements: the efficiency of implementing incident response plans, the degree to which the response partner network was integrated into the CM component and the use of communication equipment. All of the data were obtained from the AARs (where applicable) that were developed following CM exercises and from the results of the simulation study conducted to evaluate the entire CWS. No actual contamination incidents of the utility drinking water system occurred during the pilot period. 8.1.1 Incident Response Plans There were no statistical trends demonstrated among the FSEs that were primary data sources for measuring the effectiveness of the incident response plans. This was due to the variation in contamination scenarios used for each exercise, which made direct comparison of response times difficult. However, the design elements evaluated as a part of this design objective revealed characteristics of how the utility investigated and responded to contamination incidents. The simulation study results demonstrated strong correlations for the time to Credible determination, time to Confirmed determination, time to public health response, time to first operational response, and time to public notification metrics for the majority of the analyses performed. However, some of the metrics analyzed showed inconsistent results or results that were driven by the model parameters. Analyses including by contaminant type, number of components detecting and the number of alerts (some variability in correlations) were performed. Credibility Determination Exercise Investigative Actions: This included the amount of time required for investigative actions leading to both Credible and Confirmed (including "Assumed Contamination") determination of a Possible contamination incident. A hierarchy of investigation information types evolved, which seemed to accelerate the speed with which the CM component was implemented (progression through threat level determinations). Two primary examples include: 1. Multiple alerts, system connectivity, positive RFTs and signs of intrusion accelerated the declaration of Credible contamination incident and 2. Signs of intrusion and health impacts accelerated the declaration of Confirmed (Assumed Contamination) and did not necessarily depend on positive laboratory analysis. Simulation Study Investigative Actions: This included the amount of time required for investigative actions leading to both Credible and Confirmed determination of a Possible contamination incident. The type of contaminant introduced and the number of components showed a strong impact on the timeliness of threat level determination for Credible and Confirmed. Two primary examples include: 1. The model results showed significant differences for threat level determination for toxic chemical contaminants compared to biological agents. The toxic chemical contaminants resulted in much quicker threat level determination. For Credible determination, the difference in timeliness is most likely due to the contaminant types' effects on taste, odor or appearance of the drinking water resulting in customer complaints or progression of symptoms. For Confirmed 55 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot determination, the difference in timeliness is most likely due to the same factors that drive Credible determination as well as being able to be analyzed and identified much more quickly. 2. The simulation study also demonstrated a significant advantage provided by an increasing number of different monitoring and surveillance components for a reduction in threat level determination time. Three components detecting results showed a marked advantage in the timeline over two components detecting which generally showed an advantage over one component detecting. Response Actions Exercise Response Actions. This included the time to notify response partners, time to decide on appropriate operational responses, and time to make public notification of water use restrictions. The time to notify response partners varied with the sequence of circumstances presented through the various exercise scenarios, but generally was consistent from FSE 2 to FSE 3, with some slight improvements with key partners including CFD, CHD, DPIC and OEPA. The average time for identifying appropriate operational response actions during specific threat levels in simulated contamination incidents varied, but these times were influenced by the circumstances presented by the scenario. Operational responses were initially driven by what actions the utility could implement quickly to isolate or slow contamination without impacting service to customers. As the incident progressed, investigation evidence was subsequently used to revise those response actions as necessary. The time to develop and implement public notification was consistent throughout the exercises, with an average time of 169 minutes from direction to prepare and availability to release. Given the variability of exercise scopes and the accompanying revision of the CCP, it was not possible to make statistical inferences concerning the performance. Simulation Study Response Actions. The simulation study contamination scenario results were analyzed to determine the time to public health response, time to the first operational response, and the time to public notification. The data was evaluated based on the type of contaminant injected, the number of components detecting and the number of alerts received. Simulation Study Investigative Actions. The simulation study results for time to public health response showed a strong correlation to the contaminant type and to the number of components detecting metrics. A comparison of the toxic chemical to the biological agent contaminant scenarios indicated that toxic chemical contaminants resulted in much quicker implementation of public health response and a much narrower time span. As indicated earlier, the difference in time to public health response was most likely due to differences between contaminant types with respect to the time for the rapid onset of symptoms, public health community awareness, and variations in how long it took to identify the contaminant. The simulation study results for time to the first operational response were very similar for contaminant type, number of components detecting and number of alerts metrics. All metric results indicated very quick first operational response implementation times and a very narrow time span. The simulation study results for time to public notification showed a correlation for the contaminant type, number of components detecting, and number of alerts metric results. A comparison of the toxic chemical and biological agent contaminant scenario results showed that toxic chemical contaminants resulted in much quicker public notification over a very narrow time span. As indicated earlier, this was most likely due to differences between the contaminant types' effect on taste, odor or appearance of the drinking water and whether the contaminant triggered customer complaints or progression of symptoms leading to healthcare seeking behavior. The results for the number of components detecting showed a slight advantage for three components detecting followed by two components detecting with a slight 56 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot advantage over one component detecting. Finally, the number of alert results impact on the time to public notification showed a significant reduction in the public notification time at six alerts and above. 8.1.2 Integration of Response Partner Network The inclusion of the response partner network as a design element of the CM component was intended to streamline the overall response process. Although there was no empirical data to describe how this design element contributed to timeliness of response by itself, the design elements evaluated as a part of this design objective did reveal several significant observations including the effective integration of 10 response partner agencies into the CM component and progressive improvement with understanding response partner roles and responsibilities through training. 8.1.3 Communication Equipment The integration of the 800 MHz radios and SharePoint site into the utility response activities enhanced their ability to communicate information between field personnel, response partner agencies and the ICS. Data and feedback from the exercises indicated that the use of the communication equipment allowed faster and more complete investigation of contamination incidents. By the end of the pilot evaluation period, utility personnel routinely used the communication equipment for contamination incident investigations. 8.2 Design Objective: Sustainability Sustainability of the CM component was measured by labor hours for both CM implementation and O&M and dual-use benefits. Overall, the development and implementation of a comprehensive CM component required a considerable commitment of time and resources, from both a development and maintenance perspective. Developing the Cincinnati Pilot Consequence Management Plan and conducting the exercises were the most significant tasks for the implementation of this component. Both required work with utility personnel as well response partner agencies. Implementation costs approximately amounted to $800,000 though equipment costs were minimal (approximately $40,000). Labor hours for O&M were steady at approximately 340 hours per month through most of the evaluation period, with the exception of the reporting periods surrounding FSEs. Those exercise required substantial planning and participation from numerous utility personnel and response partner agencies. Dual-use benefits and compliance were evaluated through documentation of qualitative data during drills and exercises and during forums with the utility including lessons learned workshops. The use of CM procedures and equipment during a major windstorm in Cincinnati demonstrated a dual-use benefit to Cincinnati pilot personnel. Compliance was demonstrated through 100% utility participation in full scale exercises which required substantial effort, but were beneficial to the Cincinnati pilot as reported by personnel who indicated that they were able to better understand CM procedures through response to simulated water contamination incidents. 57 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Section 9.0: References U.S. Environmental Protection Agency. 2005. WaterSentinel System Architecture, Draft for Science Advisory Board Review. U.S. Environmental Protection Agency. 2008. Water Security Initiative: Cincinnati Pilot Post- Implementation System Status, EPA 817-R-08-004, September, 2008. U.S. Environmental Protection Agency. 2013a. Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Pilot, EPA 817-R-13-009. U.S. Environmental Protection Agency. 2013b. Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot, EPA 817-R-13-003 U.S. Environmental Protection Agency. 2014. Water Security Initiative: Comprehensive Evaluation of the Cincinnati Contamination Warning System Pilot EPA 817-R-14-001 63 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Section 10.0: Abbreviations The list below includes acronyms approved for use in the CM component evaluation. Acronyms are defined at first use in the document. AAR AWWA CCP CCS CERCLA CFD CHD CM CMP CPD CSFDG CWS DHS DPIC ESM EPA FBI FEMA FSE GCWW HazMat HCEMA HCPHD HI/HB 1C ICS IP ISAC MSD MSDGC NIMS O&M ODH OEPA PHS PIO PN R&R RFT SC After Action Report American Water Works Association Crisis Communication Plan Customer Complaint Surveillance Comprehensive Environmental Response, Compensation, and Liability Act Cincinnati Fire Department Cincinnati Health Department Consequence Management Consequence Management Plan Cincinnati Police Department Confirmatory Sampling Field Decision Guide Contamination Warning System Department of Homeland Security Drug and Poison Information Center Enhanced Security Monitoring U.S. Environmental Protection Agency Federal Bureau of Investigations Federal Emergency Management Agency Full Scale Exercise Greater Cincinnati Water Works CFD Hazardous Materials Team Hamilton County Emergency Management Agency Hamilton County Public Health Department Health Impacts and Human Behavior Incident Commander Incident Command System Improvement Plan (Water) Information Sharing and Analysis Center Cincinnati Metropolitan Sewer District Metropolitan Sewer District of Greater Cincinnati National Incident Management System Operations & Maintenance Ohio Department of Health Ohio Environmental Protection Agency Public Health Surveillance Public Information Officer Public Notification Remediation and Recovery Rapid Field Test Site Characterization 64 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot SCT Site Characterization Team SIMCELL Simulation Cell TOC Total Organic Carbon TLD Threat Level Determination WQ&T Water Quality & Treatment WQM Water Quality Monitoring WSI Water Security Initiative WUERM Water Utility Emergency Response Manager 65 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Section 11.0: Glossary Alert. Information from a monitoring and surveillance component indicating an anomaly in the system, which warrants further investigation to determine if the alert is valid. Alert Investigation. A systematic process, documented in a standard operating procedure, for determining whether or not an alert is valid, and identifying the cause of the alert. If an alert cause cannot be identified, contamination is possible. Anomaly. Deviations from an established baseline. For example, a water quality anomaly is a deviation from typical water quality patterns observed over an extended period. Baseline. Normal conditions that result from typical system operation. The baseline includes predictable fluctuations in measured parameters that result from known changes to the system. For example, a water quality baseline includes the effects of draining and filling tanks, pump operation and seasonal changes in water demand, all of which may alter water quality in a somewhat predictable fashion. Benefit. An outcome associated with the implementation and operation of a contamination warning system that promotes the welfare of the utility and the community it serves. Benefits are classified as either primary or dual-use. Benefit-cost analysis. An evaluation of the benefits and costs of a project or program, such as a contamination warning system, to assess whether the investment is justifiable considering both financial and qualitative factors. Biotoxins. Toxic chemicals derived from biological materials that pose an acute risk to public health at relatively low concentrations. Box-and-whisker plot. A graphical representation of nonparametric statistics for a dataset. The bottom and top whiskers represent the 10th and 90th percentiles of the ranked data, respectively. The bottom and top of the box represent the 25th and 75th percentiles of the ranked data, respectively. The line inside the box represents the 50th percentile, or median of the ranked data. Note that some data sets may have the same values for the percentiles presented in box-and-whisker plots, in which case not all lines will be visible. Component response procedures. Documentation of roles and responsibilities, process flows and procedural activities for a specified component of the contamination warning system, including the investigation of alerts from the component. Standard operating procedures for each monitoring and surveillance component are integrated into an operational strategy for the contamination warning system. Confirmed. In the context of the threat level determination process, contamination is Confirmed when the analysis of all available information from the contamination warning system has provided definitive, or nearly definitive, evidence of the presence of a specific contaminant or class of contaminant in the distribution system. While positive results from laboratory analysis of a sample collected from the distribution system can be a basis for confirming contamination, a preponderance of evidence without the benefit of laboratory results can lead to this same determination. Consequence management. Actions taken to plan for and respond to possible contamination incidents. This includes the threat level determination process, which uses information from all monitoring and surveillance components as well as sampling and analysis to determine if contamination is credible or confirmed. Response actions, including operational changes, public notification, and public health 66 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot response, are implemented to minimize public health and economic impacts and ultimately return the utility to normal operations. Consequence management plan. Documentation that provides a decision-making framework to guide investigative and response activities implemented in response to a possible contamination incident. Contamination incident. The introduction of a contaminant in the distribution system with the potential to cause harm to the utility or the community served by the utility. A contamination incident may be intentional or accidental. Contamination scenario. Within the context of the simulation study, parameters that define a specific contamination incident, including: injection location, injection rate, injection duration, time the injection is initiated, and the contaminant that is injected. Contamination warning system. An integrated system of monitoring and surveillance components designed to detect contamination in a drinking water distribution system. The system relies on integration of information from these monitoring and surveillance activities along with timely investigative and response actions during consequence management to minimize the consequences of a contamination incident. Costs, implementation. Installed cost of equipment, IT components and subsystems necessary to deploy an operational system. Implementation costs include labor and other expenditures (equipment, supplies, and purchased services). Cost, life cycle. The total cost of a system, component, or equipment over its useful or practical life. Life cycle cost includes the cost of implementation, operation & maintenance, and renewal & replacement. Costs, operation & maintenance. Expenses incurred to sustain operation of a system at an acceptable level of performance. Operational and maintenance costs are reported on an annual basis, and include labor and other expenditures (e.g., supplies and purchased services). Costs, renewal & replacement. Costs associated with refurbishing or replacing major pieces of equipment (e.g., water quality sensors, laboratory instruments, IT hardware) that reach the end of their useful life before the end of the contamination warning system lifecycle. Coverage, contaminant. Specific contaminants that can potentially be detected by each monitoring and surveillance component, including sampling & analysis, of a contamination warning system. Coverage, spatial. The areas within the distribution system that are monitored by, or protected by each monitoring and surveillance component of a contamination warning system. Credible. In the context of the threat level determination process, a water contamination threat is characterized as Credible if information collected during the investigation of possible contamination corroborates information from the validated contamination warning system alert. Data completeness. The amount of data that can be used to support system or component operations, expressed as a percentage of all data generated by the system or component. Data may be lost due to QC failures, data transmission errors, and faulty equipment among other causes. Distribution system model. A mathematical representation of a drinking water distribution system, including pipes, junctions, valves, pumps, tanks, reservoirs, etc. The model characterizes flow and 67 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot pressure of water through the system. Distribution system models may include a water quality model that can predict the fate and transport of a material throughout the distribution system. Dual-use benefit. A positive application of a piece of equipment, procedure or capability that was deployed as part of the contamination warning system in the normal operations of the utility. Ensemble. The comprehensive set of contamination scenarios evaluated during the simulation study. Event detection system. A system designed specifically to detect anomalies from the various monitoring and surveillance components of a contamination warning system. An event detection system may take a variety of forms, ranging from a complex set of computer algorithms to a simple set of heuristics that are manually implemented. Evaluation period. The period from January 16, 2008 to June 15, 2010 when data was actively collected for the evaluation of the Cincinnati contamination warning system pilot. Field results. Field results include information collected from Site Characterization activities including the site hazard assessment, field safety screening, water quality testing and rapid field tests. This does not include the results of the laboratory analysis conducted on samples collected at the end of the site characterization process. Hydraulic connectivity. Points or areas within a distribution system that are on a common flow path. Incident Commander. In the Incident Command System, the individual responsible for all aspects of an emergency response, including quickly developing incident objectives, managing incident operations and allocating resources. Incident timeline. The cumulative time from the beginning of a contamination incident until response actions are effectively implemented. Elements of the incident timeline include: time for detection, time for alert validation, time for threat level determination, and time to implement response actions. Injection location. The specific node in the distribution system model where the bulk contaminant is injected into the distribution system for a given scenario within the simulation study. Invalid alert. An alert from a monitoring and surveillance component that is not due to an anomaly and is not associated with an incident or condition of interest to the utility. Metric. A standard or statistic for measuring or quantifying an attribute of the contamination warning system or its components. Model. A mathematical representation of a physical system. Model parameters. Fixed values in a model that define important aspects of the physical system. Module. A sub-component of a model that typically represents a specific function of the real-world system being modeled. Monitoring & surveillance component. Element of a contamination warning system used to detect unusual water quality conditions, potentially including contamination incidents. The four monitoring & surveillance components of a contamination warning system include: 1) online water quality monitoring, 2) enhanced security monitoring, 3) customer complaint surveillance and 4) public health surveillance. 68 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Net present value. The difference between the present value of benefits and costs, normalized to a common year. Node. A mathematical representation of a junction between two or more distribution system pipes, or a terminal point in a pipe in a water distribution system model. Water may be withdrawn from the system at nodes, representing a portion of the system demand. Nuisance chemicals. Chemical contaminants with a relatively low toxicity, which thus generally do not pose an immediate threat to public health. However, contamination with these chemicals can make the drinking water supply unusable. Operational strategy. Documentation that integrates the standard operating procedures that guide routine operation of the monitoring and surveillance components of a drinking water contamination warning system. The operational strategy establishes specific roles and responsibilities for the component and procedures for investigating alerts. Optimization phase. Period in the contamination warning system deployment timeline between the completion of system installation and real-time monitoring. During this phase the system is operational, but not expected to produce actionable alerts. Instead, this phase provides an opportunity to learn the system and optimize performance (e.g., fix or replace malfunctioning equipment, eliminate software bugs, test procedures and reduce occurrence of invalid alerts). Pathogens. Microorganisms that cause infections and subsequent illness and mortality in the exposed population. Pito zone. An area of the Greater Cincinnati Water Works distribution system in which the pressure is fairly constant. There are 94 pito zones in the Greater Cincinnati Waterworks distribution system model. Possible. In the context of the threat level determination process, a water contamination threat is characterized as Possible if the cause of a validated contamination warning system alert is unknown. Primary benefits. Benefits that are derived from the reduction in consequences associated with a contamination incident due to deployment of a contamination warning system. Priority contaminant. A contaminant that has been identified by the EPA for monitoring under the Water Security Initiative. Priority contaminants may be initially detected through one of the monitoring and surveillance components and confirmed through laboratory analysis of samples collected during the investigation of a possible contamination incident. Process flow. The central element of a standard operating procedure that guides routine monitoring and surveillance activities in a contamination warning system. The process flow is represented in a flow diagram that shows the step-by-step process for investigation alerts, identifying the potential cause of the alert, and determining whether contamination is possible. Public health incident. An occurrence of disease, illness or injury within a population that is a deviation from the disease baseline in the population. Public health response. Actions taken by public health agencies and their partners to mitigate the adverse effects of a public health incident, regardless of the cause of the incident. Potential response actions include: administering prophylaxis, mobilizing additional healthcare resources, providing treatment guidelines to healthcare providers and providing information to the public. 69 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Real-time monitoring phase. Period in the contamination warning system deployment timeline following the optimization phase. During this phase, the system is fully operational and is producing actionable alerts. Utility staff and partners now respond to alerts in real-time and in full accordance with standard operating procedures documented in the operational strategy. Optimization of the system still occurs as part of a continuous improvement process, however the system is no longer considered to be developmental. Remediation and recovery. The stage of a contamination incident following confirmation of the incident, which involves the implementation of characterization, remediation, and return to service with the goal of restoring the drinking water system and returning to operational service. Risk communication. Communication activities within an organization and with external parties that address the impact and outcome of an incident. Routine operation. The day-to-day monitoring and surveillance activities of the contamination warning system that are guided by the operational strategy. To the extent possible, routine operation of the contamination warning system is integrated into the routine operations of the drinking water utility. Salvage value. Estimated value of assets at the end of the useful life of the system. Simulation study. A study designed to systematically characterize the detection capabilities of the Cincinnati drinking water contamination warning system. In this study, a computer model of the contamination warning system is challenged with an ensemble of 2,023 simulated contamination scenarios. The output from these simulations provides estimates of the consequences resulting from each contamination scenario, including fatalities, illnesses, and extent of distribution system contamination. Consequences are estimated under two cases, with and without the contamination warning system in operation. The difference provides an estimate of the reduction in consequences. Simulation study architecture. The interdependent models of each component of the Cincinnati contamination warning system, integrated into a platform that allows for execution of the simulations. The individual models describe the data processing, decision logic, and sequencing steps that represent the activities executed by the corresponding component. Site characterization. The process of collecting information from an investigation site to support the investigation of a contamination incident during consequence management. Threat level. The results of the threat level determination process, indicating whether contamination is Possible, Credible, or Confirmed. Threat level determination process. A systematic process in which all available and relevant information available from a contamination warning system is evaluated to determine whether the threat level is Possible, Credible, or Confirmed. This is an iterative process in which the threat level is revised as additional information becomes available. The conclusions from the threat evaluation process are considered during consequence management when making response decisions. Threat level index. In the Cincinnati contamination warning system model, a quantitative indicator of the threat level associated with a specific contamination scenario. The threat level index is calculated by the Cincinnati contamination warning system model by summing the confidence indices from all component models. A value greater than or equal to 1.0 represents possible contamination, greater than or equal to 2.0 represents credible contamination, and greater than or equal to 3.0 represents confirmed contamination. 70 ------- Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot Time for Confirmed determination. A portion of the incident timeline that begins with the determination that contamination is Credible and ends with contamination either being Confirmed or ruled out. This includes the time required to perform lab analyses, collect additional information, and analyze the collective information to determine if the preponderance of evidence confirms the incident. Time for contaminant detection. A portion of the incident timeline that begins with the start of contamination injection and ends with the generation and recognition of an alert. The time for contaminant detection may be subdivided for specific components to capture important elements of this portion of the incident timeline (e.g., sample processing time, data transmission time, and event detection time). Time for Credible determination. A portion of the incident timeline that begins with the recognition of a Possible contamination incident and ends with a determination regarding whether contamination is Credible. This includes the time required to perform multi-component investigation and data integration, implement field investigations (such as site characterization and sampling), and collect additional information to support the investigation. Time for initial alert validation. A portion of the incident timeline that begins with the recognition of an alert and ends with a determination regarding whether or not contamination is Possible. Toxic chemicals. Highly toxic chemicals that pose an acute risk to public health at relatively low concentrations. Valid alert. Alerts due to water contamination, system events (i.e., work in the distribution system for CCS or WQM) or public health incidents (for PHS) Water Utility Emergency Response Manager. A role within the Cincinnati contamination warning system filled by a mid-level manager from the drinking water utility. Responsibilities of this position include: receiving notification of validated alerts, verifying that a valid alert indicates Possible contamination, coordinating the threat level determination process, integrating information across the different monitoring and surveillance components, and activating the consequence management plan. In the early stages of responding to Possible contamination the Water Utility Emergency Response Manager may serve as Incident Commander. 71 ------- |