SEPA Water Security Initiative: Cincinnati
United States J
Pilot Post-Implementation System
Status
Environmental Protection
Agency
Covering the Pilot Period:
December 2005 through December 2007
Office of Water (MC 140) EPA 817-R-08-004 September 2008 www.epa.gov/safewater
-------
Office of Water (MC 140)
EPA817-R-08-004
September 2008
www.epa.gov/safewater
-------
Cincinnati Pilot Post-Implementation System Status
Disclaimer
The Water Security Division of the Office of Ground Water and Drinking Water has reviewed and
approved this document for publication. Neither the United States Government nor any of its employees,
contractors, or their employees make any warranty, expressed or implied, or assume any legal liability or
responsibility for any third party's use of or the results of such use of any information, apparatus, product,
or process described in this report, or represents that its use by such party would not infringe on privately
owned rights. This document is not a substitute for applicable legal requirements, nor is it a regulation
itself.
Mention of trade names or commercial products does not constitute endorsement or recommendation for
use.
Questions concerning this document should be addressed to:
Steve Allgeier
U.S. EPA Water Security Division
26 West Martin Luther King Drive
Mail Code 140
Cincinnati, OH 45268
(513)569-7131
Allgeier. Steve @epa. gov
or
Jessica Pulz
U.S. EPA Water Security Division
26 West Martin Luther King Drive
Mail Code 140
Cincinnati, OH 45268
(513)569-7918
Pulz.Jessica@epa.gov
September 2008
-------
Cincinnati Pilot Post-Implementation System Status
Acknowledgements
EPA's Office of Ground Water and Drinking Water would like to recognize the following individuals and
organizations for their assistance and contributions in development of this document and design and
implementation of the Cincinnati contamination warning system pilot:
City of Cincinnati - Greater Cincinnati Water Works
Kathy Allen
Steve Allen
Verna Arnette
Ganesh
Balasubramanian
Patty Burke
Faye Cossins
Parthajit Dastidar
Ralph Doerzbacher
Earl Einhouse
Gabe Fraley
Bill Fromme
David Hartman
Steve Hellman
Rick Hollstegge
Jim Holly
Ramesh Kashinkunti
Jay Kramer
Yeongho Lee
Bryan May
Jim McGuiness
Darla Meadows
Mark Menkhaus
Rick Merz
Debbie Metz
Kevin Moore
Larry Moster
Jeff Pieper
David Raffenburg
Mark Raffenberg
David Rager
Connie Roesch
Niranjan Selar
Todd Smith
Jeff Swertfeger
Steve Thompson
Russ Tuck
Mike Tyree
Carel Vandermeyden
Paul VonderMeulen
Kevin Weisner
Gary Wiest
Joe Zistler
Dave Bailey
Joe Barker
Denny Clark
Ed Conkey
Ed Dadosky
John Dunham
Jayson Dunn
Citv of Cincinnati - Partners
• Steve Englender
• Mike Gorman
• Beverly Head
• Dan Helm
• George Kalemanis
• Rick Lefkin
• Meg Olberding
Tony Parrot
Jim Reynolds
Dan Rottmueller
Don Theobald
Doug Ventre
Roy Winston
Tammy Bannerman
William Becker
Dave Bornino
John Colvin
Ted Folger
Katie Simon
Larry King
Local Partners
Brent Lee
Kirk Leifheit
Kathy Lordo
Marc Lowy
Bary Lusby
Cammie Mitrione
Rita Shesky
Mike Snowden
Carl Sofranko
Eric St. Germain
Dan Stine
Steve Wagner
Jared Warner
Erin Black, Centers
for Disease Control
and Prevention
Federal Partners
Vince Hill, Centers
for Disease Control
and Prevention
Rick Maier, Federal
Bureau of
Investigation
September 2008
-------
Cincinnati Pilot Post-Implementation System Status
U.S. Environmental Protection Agency - Water Security Division
Steve Allgeier • Mike Henrie • Jessica Pulz
Jeffrey Pencil • Tanya Mottley • Dan Schmelling
David Harvey • Nancy Muzzy • David Travers
Elizabeth Hedrick • Brian Pickard • Katie Umberg
U.S. Environmental Protection Agency - National Homeland Security Research Center
Dominic Boccelli • Matthew Magnuson • Cynthia Yund
Kathy Clayton • Scott Minamyer
John Hall • Regan Murray
Robert Janke • Jeff Szabo
Victoria
Blackschleger
Yildiz Chambers
John Chandler
Kevin Connell
Mike Denison
Bill Desing
Todd Elliott
Tim Ellis
Dan Gallagher
Rashmi Ghei
Darcy Gibbons
Contractor Support
• Adam Haas
• Adrian Hanley
• Gary Jacobson
• Reese Johnson
• Dan Joy
• Colm Kenny
• Timothy Kling
• Jessica Knight
• Alan Lai
• Greg Meiners
• Kim Morgan
• Tom Noble
Bill Phillips
Misty Pope
Curtis Robbins
Chris Rollo
Ellery Savage
Doron Shalvi
Marie Socha
David Watson
Scott Weinfeld
Nick Winnike
September 2008
-------
Cincinnati Pilot Post-Implementation System Status
Executive Summary
The Water Security (WS) initiative is a U.S. Environmental Protection Agency (EPA) program that
addresses the risk of intentional contamination of drinking water distribution systems. Initiated in
response to Homeland Security Presidential Directive 9, the overall goal of WS is to design and deploy
contamination warning systems for drinking water utilities. EPA is implementing the WS initiative in
three phases: (1) development of a conceptual design that achieves timely detection and appropriate
response to drinking water contamination incidents; (2) demonstration and evaluation of the conceptual
design in full-scale pilots at drinking water utilities; and (3) issuance of guidance and conduct outreach to
promote voluntary national adoption of effective and sustainable drinking water contamination warning
systems.
This report addresses the second phase of the program, specifically a description of the initial pilot as
implemented in Cincinnati, Ohio. EPA's objectives for this initial pilot were to demonstrate the
conceptual design for a contamination warning system and to gain experience that would support the
development of guidance and tools for other drinking water utilities. As shown in this report, these
objectives are being achieved.
ES.l Contamination Warning System Conceptual Design
The Cincinnati contamination warning system was designed to detect contamination incidents, both
intentional and accidental. Such incidents are presumed to be extremely rare and uncertain with respect
to contaminant type, location, time, and duration. These factors required the system to be designed
around the following robust set of objectives:
• Detect a broad spectrum of potential contaminants that could cause harm to the public or cause
disruption in service
• Provide spatial coverage of the entire distribution system
• Detect contamination incidents in sufficient time for implementation of response actions that
reduce public health and economic consequences
• Reliably indicate a contamination incident with a minimum number of false-positives
• Provide a sustainable architecture to monitor the distribution system for general water quality
objectives as well as detection of potential contamination incidents
Consideration of these objectives resulted in a multi-component approach to contaminant detection, as
shown in Figure ES-1 and briefly described below:
• Online water quality monitoring involves monitoring for typical water quality parameters
throughout the distribution system, and comparison with an established baseline to detect possible
contamination incidents.
• Sampling and analysis involves the collection of distribution system samples that are analyzed
for various contaminants and contaminant classes for the purpose of establishing a baseline of
contaminant occurrence (contaminants detected, levels detected and frequency of detections) and
method performance, and for investigating suspected incidents.
• Enhanced security monitoring includes the equipment and procedures that detect and respond
to security breaches at distribution system facilities.
• Consumer complaint surveillance enhances and automates the collection and analysis of calls
by consumers reporting unusual water quality concerns and compares trends against an
established baseline to detect possible contamination incidents.
• Public health surveillance involves the analysis of health-related data sources to identify disease
events that may stem from drinking water contamination.
September 2008 iv
-------
Cincinnati Pilot Post-Implementation System Status
Consumer
Complaint
Surveillance
Online Water
Quality
Monitoring
Sampling
and
Analysis
Contamination
Warning
System
Enhanced
Security
Monitoring
Figure ES-1. Multi-Component Approach to Contamination Warning System Design
These monitoring and surveillance strategies have a history of use in the utility industry or public health
sector. While application of these strategies to drinking water contamination warning systems is a new
concept, use of familiar systems and equipment provides a sustainable platform for contamination
warning system design. Furthermore, integration of data from multiple, independent data streams
increases contaminant and spatial coverage, timeliness of detection, and reliability of information from
the system.
The WS system architecture also includes extensive consequence management planning that describes
procedures for investigating and responding to possible contamination incidents detected through routine
monitoring and surveillance. Once a possible contamination incident has been identified, the
consequence management plan defines a process for establishing the credibility of the suspected incident.
This plan also describes response actions that may be taken during and following the investigation to
minimize public health and economic consequences and ultimately restore the system to normal
operations.
ES.2 Process for Pilot Deployment
The process for deploying the initial pilot is illustrated in Figure ES-2 and includes stages for planning
through evaluation and refinement. This report describes the status of the initial pilot following
completion of the implementation stage. Planning and pre-design activities for the Cincinnati pilot began
in December 2005. Significant implementation activities were completed in July 2007, with additional
activities continuing for some components through December 2007 in preparation for preliminary testing
of the system.
September 2008
-------
Cincinnati Pilot Post-Implementation System Status
Scope of Report
Figure ES-2. Deployment Process for the WS Contamination Warning System Pilot in Cincinnati
Because there was no experience base for deployment of a comprehensive contamination warning system
at a drinking water utility, EPA had an active and direct role in the design and implementation of the first
pilot. To facilitate this role, a Cooperative Research and Development Agreement was established
between the City of Cincinnati and EPA. This Agreement allowed the pilot to be developed as a joint
effort between the City and EPA, and provided an opportunity for EPA to evaluate the system and
compile lessons learned to the benefit of the drinking water industry.
Design and implementation of the Cincinnati pilot was coordinated through a joint management team
including staff from the Greater Cincinnati Water Works (GCWW) and EPA. This team developed an
overarching work plan and schedule to prioritize and coordinate activities across the many additional
teams established to design and implement the components of the pilot. As shown in Figure ES-3, a
number of local, state, and federal partners were involved in implementation of the pilot, and the
integrated project management team helped coordinate work with these key partners.
Centers For Disease
Control and Prevention
EPA Water Security
Division
EPA Region V
Ohio EPA
Hamilton County
Emergency
Management Agency
Neighboring Utilities
Cincinnati Health
Department
Cincinnati Fire
Dept. & HazMat
Terrorism Early
Warning Group
Metropolitan Sewer
District
Greater
Cincinnati Water
Works
City Call
Center
Cincinnati Police
Department
City Manager's
Office
Drug and Poison
Information Center
Ohio Department
of Health
Hamilton County
General Health
District
Regional Computing
Center
Department of
Homeland Security
Contract Laboratories
Federal Bureau of
Investigation
Figure ES-3. Partner Organizations Involved with Implementation of the Cincinnati Pilot
ES.3 Components of the Cincinnati Contamination Warning System Pilot
One of the overarching goals of the pilot was to integrate information across the various components to
more effectively meet system design objectives (e.g., spatial coverage, contaminant coverage, timeliness
of detection, etc.). To achieve this goal, operational procedures and information management systems
were integrated across components whenever possible. Operational procedures took the form of a
"concept of operations" that aligned alarm investigations with routine utility job functions. Information
management was integrated by first identifying commonalities among component information technology
September 2008
VI
-------
Cincinnati Pilot Post-Implementation System Status
requirements, and then implementing enhancements that met the collective requirements. Also,
centralized database and application servers were deployed to serve multiple components. These system-
level enhancements and coordination provided a foundation for developing the five (5) monitoring and
surveillance components and the consequence management plan.
The online water quality monitoring component was designed to expand GCWW's existing monitoring
capabilities to improve contaminant coverage, spatial coverage, timeliness of detection, and reliability.
Four main design elements were addressed for this component:
• Water quality monitoring equipment. Each monitoring station is equipped with sensors for
chlorine residual, total organic carbon, conductivity, pH, temperature, and oxidation reduction
potential, which provides comprehensive coverage of contaminant classes considered in the WS
design basis.
• Water quality monitoring network. Seventeen (17) water quality monitoring stations were
installed throughout the GCWW distribution system to maximize the spatial coverage and
timeliness of detection. Optimal monitoring locations were identified through application of the
Threat Ensemble Vulnerability Assessment, Sensor Placement Optimization Tool.
• Data management and communication. A dedicated communication network and SCADA
system were installed to enable the transmission, collection, and display of water quality data in
near real-time to system operators. Communication equipment was installed at each site to
transmit water quality data to the SCADA system via a digital cellular network.
• Water quality event detection. Two event detection systems were installed and
trained/configured to continually analyze data from the seventeen (17) monitoring stations in
order to detect anomalies that might be indicative of contamination. If an anomaly is detected, it
is investigated in accordance with the concept of operations.
The sampling and analysis component of the Cincinnati pilot was designed to expand GCWW's ability to
analyze for priority contaminants, as well as additional non-targeted contaminants during the
investigation of a possible contamination incident. Three main design elements were addressed for this
component:
• Laboratory capability and capacity. Through a combination of enhancements to GCWW's
laboratory capabilities, partnering with the Ohio State Health Laboratory and local utility
laboratory, and contracting with commercial laboratories, a laboratory network was established
capable of analyzing for ten (10) out of twelve (12) priority contaminant classes.
• Sampling and analysis. A baseline monitoring program was designed and implemented to
characterize contaminant occurrence (contaminants detected, levels detected and frequency of
detections) and method performance in samples throughout the distribution system. Protocols for
investigating possible contamination using the same methods implemented during baseline
monitoring were also established.
• Field screening and site characterization. Site characterization procedures were developed,
and local HazMat response teams were involved in the development of those procedures.
Additional field screening equipment was provided to both GCWW and HazMat teams, and
triggered sampling kits were assembled to allow for rapid response.
The enhanced security monitoring component was designed to provide early warning of intrusion at
critical distribution system facilities, which could provide an opportunity for contamination of large
quantities of water. Three main design elements were addressed for this component:
• Physical security equipment. Video cameras, motion sensors, and door contact switches were
installed at three (3) large pump stations to enable visual identification of a potential intruder with
intent to contaminate water. Ladder motion sensors were installed at seven (7) elevated storage
tanks to detect potential intruders. Vent housings were installed on reservoirs and tanks.
• Data management and communication. A dedicated SCADA system was installed to collect
and display security alarms and video clips in near real-time to system operators.
September 2008 vii
-------
Cincinnati Pilot Post-Implementation System Status
Communication equipment was installed at each site to transmit alarms and video clips to the
GCWW control center via a digital cellular network.
• Component response procedures. Existing partnerships with local law enforcement agencies
were leveraged to support the investigation of security breaches in accordance with the concept of
operations.
The consumer complaint surveillance component built on GCWW's existing consumer complaint
management system to improve the timeliness and reliability of detection of possible contamination
incidents. Four main design elements were addressed for this component:
• Comprehensive complaint collection. GCWW already established a single number for all
consumer calls; thus enhancements focused on procedures for funneling water quality calls
received by other agencies into the GCWW call management system.
• Electronic data management. GCWW's work order system was enhanced to categorize
customer calls related to water quality issues. Additionally, the interactive voice response system
that greets all callers was enhanced to include a category for water quality issues.
• Automated and integrated data analysis. Event detection systems were deployed to
continually analyze data from the interactive voice response and work order systems in order to
detect anomalies that might be indicative of contamination. The work order system was also
enhanced to spatially display water quality related work orders to provide more rapid
identification of clustering events that may indicate contamination.
• Component response procedures. Procedures were established for timely identification of
unusual water quality consumer calls and analysis of potential anomalies relative to an
established base-state. The investigation of the anomaly is governed by the concept of
operations.
The public health surveillance component leveraged existing syndromic surveillance programs operating
in the area, added new data streams, and improved coordination among local partners. These
enhancements extended contaminant and spatial coverage, while also improving timeliness of detection
and reliability of indicators of potential water-related health episodes. Two main design elements were
addressed for this component:
• Public health data streams. Existing syndromic surveillance systems for emergency room visits
and sale of over-the-counter pharmaceuticals were leveraged for the pilot. Similarly, existing
procedures for monitoring and analysis of poison control center calls were enhanced. In addition,
systems were deployed to capture new data streams, specifically 911 calls and emergency
medical service data applicable to drinking water exposure, in an effort to improve timeliness of
detection
• Communication and coordination. To improve coordination among the many local health
partners and GCWW, a Public Health User's Group was established and meets on a regular basis.
Automated email notifications were implemented to notify all partners when a potential water-
related health anomaly is detected. The investigation of the anomaly is governed by the concept
of operations.
The consequence management component built on existing emergency response plans and partnerships to
develop a consequence management plan with procedures that address the unique challenges associated
with response to a drinking water contamination incident. Four main design elements were addressed for
this component:
• Contaminant incident response plans. A Consequence Management Plan was developed to
serve as a preparedness and response guide, and a Crisis Communication Plan was developed to
guide public notification and risk communication procedures during all phases of a potential
contamination incident.
• Response partner network. GCWW and response partners established a network to better
integrate their roles and responsibilities in the event of a drinking water contamination incident.
September 2008 viii
-------
Cincinnati Pilot Post-Implementation System Status
• Training and exercises. Workshops, table-top exercises, drills, functional exercises, and full
scale exercises were conducted to test the Consequence Management Plan and train participants
on processes and procedures.
• Equipment. Eight 800 MHz hand-held radios were procured to improve communications
between GCWW and response partners during field activities in response to a contamination
incident.
ES.4 Cost of the Pilot Contamination Warning System
The overall cost for the design and implementation of the Cincinnati pilot, including labor and equipment,
was approximately $12.3 million. Figure ES-4 presents a breakdown of these implementation costs by
component. When interpreting the cost data in this report, it is important to consider that this project was
the first pilot of a contamination warning system, and that substantial costs were incurred as part of the
research required to design and implement this prototypical system. It is expected that the cost to
implement a contamination warning system according to the model documented in this report would be
lower relative to the cost of the Cincinnati pilot.
Cost of Cincinnati Pilot by Component (in millions of dollars)
December 2005 - December 2007
$1.3
D System Engineering
i Water Quality Monitoring
n Sampling and Analysis
D Enhanced Security Monitoring
E Consumer Complaint Surveillance
'Public Health Surveillance
l Consequence Management
Figure ES-4. Cost Breakdown for Implementation of the Cincinnati Pilot
ES.5 Future of the Pilot Contamination Warning System
The drinking water contamination warning system in Cincinnati has been installed, is fully operational,
and is currently generating data needed to establish baseline performance. Use of multiple monitoring
and surveillance components provides broad contaminant coverage throughout the core GCWW retail
service area, reduces the time for initial detection, and improves the reliability of information generated
September 2008
IX
-------
Cincinnati Pilot Post-Implementation System Status
by the system. Furthermore, use of existing monitoring and surveillance systems that are integrated with
routine utility operations provides many opportunities for dual-use applications, resulting in a sustainable
system.
Evaluation of the Cincinnati pilot will quantify system performance, derive lessons learned, and assess the
cost/benefit of deploying contamination warning systems at drinking water utilities. This information
will be critical in development of guidance for drinking water utilities nationwide on effective and
sustainable contamination warning systems.
September 2008
-------
Cincinnati Pilot Post-Implementation System Status
Table of Contents
SECTION 1.0: OVERVIEW 1
1.1 CONCEPTUAL DESIGN OF A CONTAMINATION WARNING SYSTEM 1
1.2 DEPLOYMENT OF THE PILOT CONTAMINATION WARNING SYSTEM 4
SECTION 2.0: PROJECT MANAGEMENT AND SYSTEM INTEGRATION 5
2.1 PRE-IMPLEMENTATION STATUS 5
2.1.1 Comprehensive Project Management 5
2.1.2 Integrated Operational Procedures 6
2.1.3 Integrated Information Management 7
2.1.4 Summary of Identified Gaps 8
2.2 POST-IMPLEMENTATION STATUS 8
2.2.1 Comprehensive Project Management 8
2.2.2 Integrated Operational Procedures 10
2.2.3 Integrated Information Management 11
2.2.4 Summary of Post-Implementation Status 13
SECTION 3.0: ONLINE WATER QUALITY MONITORING 16
3.1 PRE-IMPLEMENTATION STATUS 17
3.1.1 Water Quality Monitoring Stations 17
3.1.2 Water Quality Monitoring Network 17
3.1.3 Data Management and Communications 18
3.1.4 Water Quality Event Detection 18
3.1.5 Summary of Identified Gaps 18
3.2 POST-IMPLEMENTATION STATUS 19
3.2.1 Water Quality Monitoring Stations 19
3.2.2 Water Quality Monitoring Network 25
3.2.2.1 Distribution System Model Validation 25
3.2.2.2 Overview of the TEVA Sensor Placement Optimization Tool 26
3.2.2.3 Overview of the Monitoring Network Design Process 26
3.2.3 Data Management and Communications 28
3.2.3.1 Remote Data Collection and Communication 28
3.2.3.2 Data Management System Architecture 29
3.2.3.3 Data Flow 31
3.2.4 Water Quality Event Detection 32
3.2.4.1 Event Detection Systems 33
3.2.4.2 Event Detection, Deployment, Integration, and Evaluation System 34
3.2.4.3 Event Detection System Performance Evaluation 34
3.2.4.4 Event Detection System Deployment 35
3.2.5 Summary of Post-Implementation Status 36
SECTION 4.0: SAMPLING AND ANALYSIS 41
4.1 PRE-IMPLEMENTATION STATUS 42
4.1.1 Laboratory Capability and Capacity 42
4.1.2 Sampling and Analysis 47
4.1.3 Field Screening and Site Characterization 48
4.1.4 Summary of Identified Gaps 48
4.2 POST-IMPLEMENTATION STATUS 49
4.2.1 Laboratory Capability and Capacity 50
4.2.2 Sampling and Analysis 53
4.2.3 Field Screening and Site Characterization 54
4.2.4 Summary of Post-Implementation Status 55
SECTION 5.0: ENHANCED SECURITY MONITORING 61
5.1 PRE-IMPLEMENTATION STATUS 61
5.1.1 Physical Security Equipment 62
5.1.1.1 Pump Stations 62
September 2008 xi
-------
Cincinnati Pilot Post-Implementation System Status
5.1.1.2 Reservoirs and Ground-Level Storage Tanks 62
5.1.1.3 Elevated Storage Tanks 62
5.1.2 Data Management and Communications 62
5.1.3 Component Response Procedures 63
5.1.4 Summary of Identified Gaps 63
5.2 POST-IMPLEMENTATION STATUS 64
5.2.1 Physical Security Equipment 64
5.2.1.1 Pump Station Equipment 65
5.2.1.2 Reservoir and Ground Level Storage Tank Equipment 66
5.2.1.3 Elevated Storage Tank Equipment 66
5.2.2 Data Management and Communications 67
5.2.3 Component Response Procedures 67
5.2.4 Summary of Post-Implementation Status 68
SECTION 6.0: CONSUMER COMPLAINT SURVEILLANCE 72
6.1 PRE-IMPLEMENTATION STATUS 72
6.1.1 Comprehensive Complaint Collection 73
6.1.1.1 GCWW Call Center: Consumer Calls during Regular Business Hours 73
6.1.1.2 Distribution Division Dispatcher: Consumer Calls during Non-Business Hours 74
6.1.2 Electronic Data Management 74
6.1.2.1 Banner 74
6.1.2.2 Distribution Work Creation 74
6.1.2.3 Water Quality Database 74
6.1.3 Automated and Integrated Data Analysis 75
6.1.3.1 Event Detection 75
6.1.3.2 Spatial Analysis 75
6.1.4 Component Response Procedures 75
6.1.5 Summary of Identified Gaps 75
6.2 POST-IMPLEMENTATION STATUS 76
6.2.1 Comprehensive Complaint Collection 77
6.2.2 Electronic Data Management 77
6.2.2.1 Interactive Voice Response 79
6.2.2.2 Water Quality Work Requests 79
6.2.2.3 Water Quality Work Orders 80
6.2.3 Automated and Integrated Data Analysis 80
6.2.3.1 Event Detection 80
6.2.3.2 Notifications 81
6.2.3.3 Spatial Analysis 82
6.2.4 Component Response Procedures 83
6.2.5 Summary of Post-Implementation Status 84
SECTION 7.0: PUBLIC HEALTH SURVEILLANCE 87
7.1 PRE-IMPLEMENTATION STATUS 88
7.1.1 Public Health Data Streams 88
7.1.2 Communication and Coordination 89
7.1.3 Summary of Identified Gaps 89
7.2 POST-IMPLEMENTATION STATUS 90
7.2.1 Public Health Data Streams 90
7.2.2 Communication and Coordination 96
7.2.3 Summary of Post-Implementation Status 97
SECTION 8.0: CONSEQUENCE MANAGEMENT 101
8.1 PRE-IMPLEMENTATION STATUS 102
8.1.1 Contamination Incident Response Plans 102
8.1.2 Response Partner Network 104
8.1.3 Training and Exercises 105
8.1.4 Equipment 106
8.1.5 Summary of Identified Gaps 106
8.2 POST-IMPLEMENTATION STATUS 106
8.2.1 Contamination Incident Response Plans 106
8.2.1.1 Consequence Management Plan 106
September 2008 xii
-------
Cincinnati Pilot Post-Implementation System Status
8.2.1.2 Crisis Communication Plan 107
8.2.1.3 Site Characterization Plan 107
8.2.2 Response Partner Network 108
8.2.3 Training and Exercises 109
8.2.4 Equipment 110
8.2.5 Summary of Post-Implementation Status 110
SECTION 9.0: SUMMARY AND FUTURE WORK 113
REFERENCES 115
September 2008 xiii
-------
Cincinnati Pilot Post-Implementation System Status
List of Tables
Table 1-1. Overview of the Deployment Process for the WS Contamination Warning System 4
Table 2-1. System-Level Design Objectives 5
Table 2-2. GCWW Divisions and Responsibilities 6
Table 2-3. Examples of Existing GCWW Procedures Relevant to Contamination Warning System Operation 6
Table 2-4. GCWW Information Technology Systems Relevant to Contamination Warning System Operations 7
Table 2-5. System-Level Gap Analysis 8
Table 2-6. Local Partners Providing Support to GCWW during Implementation of the WS Pilot 9
Table 2-7. Summary of IT Systems Used in the Cincinnati Contamination Warning System 12
Table 2-8. System-Level Post-Implementation Status 14
Table 3-1. Water Quality Monitoring Component Design Objectives 16
Table 3-2. Water Quality Monitoring Gap Analysis 19
Table 3 -3. Water Quality Parameters and Equipment Selected for Water Quality Monitoring Stations 20
Table 3-4. Water Quality Monitoring Post-Implementation Status 36
Table 3 -5. Comparing Implementation Costs for Each Type of Water Quality Monitoring Station 40
Table 4-1. Sampling and Analysis Component Design Objectives 41
Table 4-2. Cincinnati Pilot Contaminant Classes and Method Types 43
Table 4-3. GCWW Pre-Implementation Equipment and Methods 44
Table 4-4. Pre-Implementation Status of GCWW Laboratory Methods 45
Table 4-5. Sampling and Analysis Frequency 47
Table 4-6. Summary of Pre-Implementation Field Screening and Site Characterization Capability 48
Table 4-7. Sampling and Analysis Gap Analysis 49
Table 4-8. WS Detection Classes and Current Sampling and Analysis Capabilities for the GCWW Pilot 50
Table 4-9. Sampling and Analysis Equipment Enhancements at GCWW 50
Table 4-10. Laboratory Contracts and Letters of Intent 51
Table 4-11. Laboratories Supporting the Cincinnati Pilot, Analytical Methods, and Analytes 52
Table 4-12. Cincinnati Pilot Safety Screening and Field Testing Equipment 54
Table 4-13. Sampling and Analysis Post-Implementation Status 56
Table 5-1. Enhanced Security Monitoring Component Design Objectives 61
Table 5-2. Enhanced Security Monitoring Component Gap Analysis 63
Table 5-3. Enhanced Security Monitoring Design and Implementation Approach 64
Table 5-4. Enhanced Security Components at GCWW Distribution Facilities 64
Table 5-5. Enhanced Security Monitoring Component Post-Implementation Status 68
Table 5-6. Enhanced Security Monitoring Equipment Unit Cost 71
Table 6-1. Consumer Complaint Surveillance Component Design Objectives 72
Table 6-2. Consumer Complaint Surveillance System Gap Analysis 76
Table 6-3. Consumer Complaints Surveillance Data Management System Inventory 77
Table 6-4. Event Detection Systems Deployed for Consumer Complaint Surveillance 81
Table 6-5. Consumer Complaint Surveillance System Post-Implementation Status 84
Table 7-1. Public Health Surveillance Component Design Objectives 87
Table 7-2. Public Health Surveillance Gap Analysis 90
Table 7-3. Public Health Surveillance Post-Implementation Status 97
Table 8-1. Consequence Management Plan Design Objectives 102
Table 8-2. Pre-Implementation Evaluation of Utility Plans, Procedures, Equipment, and Training 102
Table 8-3. Response Partner Workshops 105
Table 8-4. Consequence Management Component Gap Analysis 106
Table 8-5. Summary of Pilot Response Partner Network and CMP Role 108
Table 8-6. Consequence Management Training and Exercises 109
Table 8-7. Consequence Management Component Post-Implementation Status 110
September 2008
XIV
-------
Cincinnati Pilot Post-Implementation System Status
List of Figures
Figure ES-1. Multi-Component Approach to Contamination Warning System Design v
Figure ES-2. Deployment Process for the WS Contamination Warning System Pilot in Cincinnati vi
Figure ES-3. Partner Organizations Involved with Implementation of the Cincinnati Pilot vi
Figure ES-4. Cost Breakdown for Implementation of the Cincinnati Pilot ix
Figure 1-1. Overview of EPA's Water Security Initiative 1
Figure 1-2. Architecture of the WS Contamination Warning System 2
Figure 2-1. Level of Effort for the Cincinnati Pilot Project Management and System Integration Design and
Implementation Activities (December 2005 - December 2007) 14
Figure 2-2. Extramural Costs Associated with Design and Implementation of Project Management and System
Integration Activities (December 2005 - December 2007) 15
Figure 3-1. Type-A Water Quality Monitoring Station 21
Figure 3-2. Type-B Water Quality Monitoring Station 22
Figure 3-3. Type-C Water Quality Monitoring Station 23
Figure 3-4. WS Water Quality and Enhanced Security Monitoring Communication Architecture 29
Figure 3-5. WS Water Quality Monitoring Data Management System Architecture 30
Figure 3-6. WS Water Quality Monitoring Data Flow Diagram 32
Figure 3-7. Level of Effort for Design and Implementation of the Online Water Quality Monitoring Component
(December 2005 - December 2007) 38
Figure 3-8. Extramural Costs Associated with Design and Implementation of the Online Water Quality Monitoring
Component (December 2005 - December 2007) 39
Figure 4-1. Level of Effort for Design and Implementation of the Sampling and Analysis Component (December
2005 - December 2007) 57
Figure 4-2. Extramural Costs Associated with Design and Implementation of the Sampling and Analysis
Component (December 2005 - December 2007) 58
Figure 4-3. Breakdown of Expenditures per Laboratory 59
Figure 4-4. Costs Associated with Establishing Pathogen Monitoring Capabilities 60
Figure 5-1. Level of Effort for Design and Implementation of the Enhanced Security Monitoring Component
(December 2005 - December 2007) 69
Figure 5-2. Extramural Costs Associated with Design and Implementation of the Enhanced Security Monitoring
Component (December 2005 - December 2007) 70
Figure 5-3. Total Costs by Facility Type Associated with Installation of the Enhanced Security Monitoring
Equipment (October 2006) 71
Figure 6-1. The Filter, Funnel, and Focus Model 77
Figure 6-2. Information Flow for the Consumer Complaints Surveillance Component 78
Figure 6-3. Example Email Alarm Notification 82
Figure 6-4. GCWW Hydra Map - Detailed Map with Alarm Data and Work Orders and Work Requests 83
Figure 6-5. Level of Effort for Design and Implementation of the Consumer Complaint Surveillance Component
(December 2005 - December 2007) 85
Figure 6-6. Extramural Costs Associated with Design and Implementation of the Consumer Complaint Surveillance
Component (December 2005 - December 2007) 86
Figure 7-1. Public Health Surveillance User Interface Summary Screen 91
Figure 7-2. Information Flow for 911 Calls 92
Figure 7-3. Information Flow for Emergency Medical Service Logs 93
Figure 7-4. Drug and Poison Information Center Drinking Water Surveillance Process Flow 95
Figure 7-5. Level of Effort for Design and Implementation of the Public Health Surveillance Component
(December 2005 - December 2007) 98
Figure 7-6. Extramural Costs Associated with Design and Implementation of the Public Health Surveillance
Component (December 2005 - December 2007) 99
Figure 7-7. Breakout of Costs Associated with Deployment of 911 Call Logs and Emergency Medical Service
Records 100
Figure 8-1. Level of Effort for Design and Implementation of the Consequence Management Component
(December 2005 - December 2007) Ill
Figure 8-2. Extramural Costs Associated with Design and Implementation of the Consequence Management
Component (December 2005 - December 2007) 112
September 2008 xv
-------
Cincinnati Pilot Post-Implementation System Status
List of Acronyms
The list below includes acronyms approved for use in the System Status Report. Acronyms are defined at
first use in the document.
CDC
CL
CMP
COND
CUSUM
CSV
DMZ
EARS
EDDIES
EPA
EMS
GCWW
CIS
HMI
IVR
J2EE
LAN
LOE
NELAC
NHSRC
ORD
ORP
PLC
PTZ
RAM-W™
RODS
SCADA
SMTP
TEMP
TEVA
TOC
TURB
UPS
VOCs
WAN
WS
WSDR
WUERM
Centers for Disease Control and Prevention
Chlorine Residual (Total/Free)
Consequence Management Plan
Conductivity
Cumulative Sum calculation
Comma Separated Value
Demilitarized Zone
Early Aberration Reporting System
Event Detection, Deployment, Integration, and Evaluation System
Environmental Protection Agency
Emergency Medical Service
Greater Cincinnati Water Works
Geographic Information System
Human Machine Interface
Interactive Voice Response
Java 2 Platform, Enterprise Edition
Local Area Network
Level of Effort
National Environmental Laboratory Accreditation Conference (standards)
National Homeland Security Research Center
Office of Research and Development
Oxidation Reduction Potential
Programmable Logic Controller
Pan-Tilt-Zoom
Risk Assessment Methodology - Water™ (Sandia Methodology)
Real-time Outbreak and Disease Surveillance
Supervisory Control and Data Acquisition
Simple Mail Transfer Protocol
Temperature
Threat Ensemble Vulnerability Assessment
Total Organic Carbon
Turbidity
Uninterruptible Power Supply
Volatile Organic Compounds
Wide Area Network
Water Security (initiative)
Water Security Data Repository
Water Utility Emergency Response Manager
September 2008
XVI
-------
Cincinnati Pilot Post-Implementation System Status
Section 1.0: Overview
The Water Security (WS) initiative is a U.S. Environmental Protection Agency (EPA) program that
addresses the risk of intentional contamination of drinking water distribution systems. Initiated in
response to Homeland Security Presidential Directive 9, the overall goal of WS is to design and deploy
contamination warning systems for drinking water utilities.
As shown in Figure 1-1, EPA is implementing the WS initiative in three phases: (1) development of a
conceptual design that achieves timely detection and appropriate response to drinking water
contamination incidents; (2) demonstration and evaluation of the conceptual design in full-scale pilots at
drinking water utilities; and (3) issuance of guidance and conduct outreach to promote voluntary national
adoption of effective and sustainable drinking water contamination warning systems.
Phase
Approach
Scope
Design
Specificity
Funding
DESIGN
System Architecture
DEMONSTRATE
Initial Pilot
Additional Pilots
EXPAND
Voluntary National Adoption
V-7X Applied bVT/
Apply to single Evaluate multiple Evaluate
Conceptual . N pilot utlity => ^ , => Cotlvert '°
design — • /\ n /\> I"! gu-arwifor
/ c* II / r K// any utility
AC r / *\x r /
\ 7 Refine <, V 7 Refine \ ^
x-/ and ^ and
enhance enhance
Not
applicable
Low
ll
High-
Applies to pilot utility only
ilk jlk
jA
High-
Applies to each pilot
EPA Funds
(Sggj^
:^§?W
Medium -
Applies to range of utilities
Utility Funds
Figure 1-1. Overview of EPA's Water Security Initiative
Conceptual design of a contamination warning system began in 2004 and culminated in the development
of the WaterSentinel System Architecture (USEPA, 2005a). In the second phase, Cincinnati, Ohio was
chosen to demonstrate the initial pilot; implementation of the Cincinnati contamination warning system
was complete in December, 2007. This document presents the post-implementation status of the initial
pilot system and captures the effort invested to achieve this milestone.
To provide a context for the design of the initial pilot, the conceptual design developed during the first
phase is briefly described in the following section. Additional detail on the conceptual design can be
found in WaterSentinel System Architecture (USEPA, 2005a).
1.1 Conceptual Design of a Contamination Warning System
Significant contamination incidents in drinking water distribution systems are extremely rare, and their
characteristics are uncertain with respect to contaminant type, location, time, and duration. Thus, a
contamination warning system capable of detecting a wide range of contamination scenarios is designed
around the following broad objectives:
September 2008
-------
Cincinnati Pilot Post-Implementation System Status
• Detect a broad spectrum of contaminant classes. As part of the contamination warning system
design basis, contaminants were prioritized and then binned into twelve (12) detection classes
(USEPA, 2005b). Use of the detection classes to inform design provides more robust detection
capability than analyzing for only a select number of contaminants and also avoids the challenge
associated with designing a system around a list containing hundreds of potential contaminants.
• Achieve spatial coverage of the entire distribution system. Monitoring and surveillance
strategies should be deployed throughout the distribution system in a manner that provides broad
coverage on both a spatial and population basis.
• Detect contamination in sufficient time for effective response. The system should be designed
in a manner to reduce the time to initial detection and thus increase the time available for
implementation of response actions that reduce public health and economic consequences.
• Reliably indicate a contamination incident with a minimum number of false-positives.
Information produced by the contamination warning system should lead decision makers to
successfully infer that contamination has or has not occurred.
• Provide a sustainable architecture to monitor distribution system water quality. The
contamination warning system should be designed as a dual-use application to benefit the utility
in day-to-day operations while also providing the capability to detect intentional or accidental
contamination incidents.
Consideration of these objectives resulted in the WS system architecture shown in Figure 1-2. The
system is defined by two distinct operational phases: routine operation and consequence management.
Routine operation includes the monitoring and surveillance components detailed below in addition to
event detection systems designed to mine the large amounts of data produced by the system in order to
identify patterns indicative of possible contamination. Consequence management is initiated once a
possible contamination incident is identified and includes processes for investigating credibility of
possible contamination, confirming an incident, and remediating a contaminated system.
Response actions
Operational response
Public health response
Site Characterization
Laboratory confirmation
Risk communication
Figure 1-2. Architecture of the WS Contamination Warning System
September 2008
-------
Cincinnati Pilot Post-Implementation System Status
Routine operation integrates the following multiple monitoring and surveillance components:
• Online water quality monitoring comprises stations located throughout the distribution system
that measure multiple water quality parameters that have been shown to change in the presence of
various contaminants (Hall, 2007a). Algorithms analyze the monitoring data to establish a water
quality base-state. Possible contamination is indicated when a significant, unexplained deviation
from the base-state occurs.
• Sampling and analysis involves the collection of distribution system samples that are analyzed
for various contaminant classes as well as specific contaminants. Sampling is both routine to
establish a base-state and triggered to respond to an indication of possible contamination from
any of the other monitoring and surveillance components. Field and laboratory-based analyses
are conducted for chemicals, radiochemicals, pathogens, and toxins using a team of utility
personnel and a laboratory network.
• Enhanced security monitoring includes the equipment and procedures that detect and respond
to security breaches at distribution system facilities. Security equipment may include cameras,
motion activated lighting, door contact alarms, ladder and window motion detectors, area motion
detectors, and access hatch contact alarms.
• Consumer complaint surveillance enhances and automates the collection and analysis of calls
by consumers reporting unusual odor, taste, or visual characteristic of the drinking water.
Algorithms analyze the call data to establish a base-state. Possible contamination is indicated
when a significant, unexplained deviation from the base-state occurs.
• Public health surveillance involves the analysis of health-related data sources to identify disease
events that may stem from drinking water contamination. Public health data may include over-
the-counter sales of pharmaceuticals, hospital admission reports, infectious disease surveillance,
emergency medical service (EMS) reports, 911 calls, and poison control center calls. Algorithms
analyze the various data streams to establish a base-state and identify deviations from the base-
state that could be indicative of a public health incident.
Data from each of these components are captured, managed, analyzed, and interpreted to identify
potential contamination incidents. Information from the multiple components is used in combination to
expand contaminant and spatial coverage, provide more timely detection, and improve reliability of
detections. Furthermore, use of common or existing monitoring and surveillance systems improves
acceptance and sustainability of the system.
Event detection is the process or mechanism by which an anomaly or deviation from the baseline or base-
state is detected. The approach utilized for event detection may vary significantly from component to
component and can range from analysis of data using sophisticated statistical algorithms to comparison of
data against simple control limits. In many cases, event detection will also include manual investigation
procedures to validate the alarm and establish whether or not contamination is possible. Once a possible
contamination incident has been identified through routine monitoring and surveillance, operations shift
to the first stage of consequence management - credibility determination.
Credibility determination procedures are performed using information from all monitoring and
surveillance components, as well as available external resources. Through the credibility determination
process, response partners may be notified and some preliminary response actions may be initiated to
limit or minimize impacts of suspected contamination. If contamination is determined to be credible,
additional confirmatory and response actions are initiated; if not, the system returns to routine monitoring
and surveillance activities. In the confirmatory stage of consequence management, additional information
is gathered and assessed in an attempt to provide definitive evidence of contamination. Once
contamination is confirmed and the immediate crisis has been addressed through response, remediation
and recovery actions defined in the consequence management plan are implemented to restore the system
to normal operations.
September 2008
-------
1.2
Cincinnati Pilot Post-Implementation System Status
Deployment of the Pilot Contamination Warning System
The initial WS pilot was designed according to the conceptual design described in Section 1.1 and
illustrated in Figure 1-2. The process for deploying the initial pilot, summarized in Table 1-1, includes
stages for planning through evaluation and refinement. Additional details on the contamination warning
system deployment process are available in Water Security Initiative: Interim Guidance on Planning for
Contamination Warning System Deployment (USEPA, 2007a). This document describes the status of the
initial pilot following completion of the implementation stage in December 2007.
Table 1-1. Overview of the Deployment Process for the WS Contamination Warning System
Stage of Approach
Planning and pre-
design
Design
Implementation
Preliminary testing
Operation and
maintenance
Evaluation and
refinement
Description
Development of a core implementation team, definition of design objectives to guide
implementation, and a preliminary assessment of existing capabilities relative to design
objectives.
Development of a preliminary concept of operations, and development of a detailed work
plan and schedule to guide implementation.
Implementation of enhancements, installation of equipment, and training according to the
plan.
Operation of the contamination warning system for the purpose of collecting data
necessary to understand system performance, and finalization of the concept of
operations to optimize system.
Operation of the contamination warning system for the purpose of monitoring for
contamination incidents and other water quality issues.
Analysis of data and information generated during full operation to refine and optimize the
system.
This document is intended to serve as a summary of the Cincinnati contamination warning system pilot.
The intended audience for this document includes drinking water utility and public health managers and
decision officials who wish to gain a broad yet comprehensive understanding of this initial contamination
warning system pilot. It may also serve as a case study for contamination warning system
implementation. However, this document does not provide detailed technical specifications on the
various components, which are available through other documentation referenced throughout this report.
Sections 2 through 8 describe the various components of the initial pilot: overall system integration; each
of the five (5) monitoring and surveillance components; and consequence management. For each of these
components, the corresponding section describes the pre-existing capability of the initial pilot utility,
Greater Cincinnati Water Works (GCWW), and partner agencies, the gap between existing capabilities
and the design objectives for the pilot, and the post-implementation status that reflects enhanced
capabilities realized through the pilot.
September 2008
-------
Cincinnati Pilot Post-Implementation System Status
Section 2.0: Project Management and System Integration
The contamination warning system design involves the integration of information from multiple
monitoring and surveillance components, coordination across multiple disciplines and organizations, and
a planned transition from routine operations to consequence management. Such a complex system
requires an integrated, interdisciplinary approach to design and implementation that extends beyond the
basic tenets of project management in order to ensure that the components and procedures are integrated
into a functional system that meets the design objectives discussed in Section 1.1. Furthermore, the
contamination warning system should be implemented in a manner that is consistent with drinking water
utility organization and procedures to promote adoption by the large number of staff with a role in its
operation.
The desired outcome of the project is an integrated contamination warning system aligned with existing
utility operations and partner organizations to the extent possible. To achieve this outcome, three
overarching, system-level design objectives were defined for the Cincinnati pilot as shown in Table 2-1.
Table 2-1. System-Level Design Objectives
Project Element
1. Comprehensive
Project Management
2. Integrated Operational
Procedures
3. Integrated Information
Management
Design Objective
Establish an overarching project management structure to guide implementation,
establish priorities, and ensure that project goals are met.
Develop procedures and define roles and responsibilities that guide routine
monitoring and trigger investigation for each of the monitoring and surveillance
components in an effective and efficient manner that is aligned with normal utility
activities to the extent possible.
Identify data needed for the investigation of alarms and credibility determination.
Ensure that the data are available in a timely and effective manner (preferably in
an electronic format). To the extent possible, integrate information from different
sources to improve overall efficiency in the review process.
2.1 Pre-lmplementation Status
The initial pilot was deployed in the City of Cincinnati, with GCWW as the principle local organization
in the pilot effort. GCWW is a public drinking water utility serving retail and wholesale customers in the
greater Cincinnati area, including wholesale customers in northern Kentucky. On average, GCWW
distributes approximately 136 million gallons of water per day through 3,000 miles of distribution mains
to approximately 1.2 million retail customers. GCWW operates two treatment plants to provide a reliable
source of safe drinking water to all of its customers. The Richard Miller Treatment Plant treats surface
water from the Ohio River and provides water to 88 percent of GCWW's customers. The remainder of
GCWW's customers are served by the C. M. Bolton Treatment Plant, which treats ground water from the
Great Miami Aquifer. Additional information about GCWW can be found at http: //www. Cincinnati -
oh.gov/water/pages/-3026-/.
Prior to implementation of the contamination warning system, GCWW had a management structure in
place for overseeing complex projects, procedures for guiding routine utility operations, and an extensive
information technology infrastructure. These existing capabilities, described in more detail below,
provided an excellent platform for implementation of a complex project such as a contamination warning
system.
2.1.1 Comprehensive Project Management
GCWW has a standing Steering Committee chaired by the utility director with leadership from all utility
divisions listed in Table 2-2. This committee provides a forum for establishing priorities for the utility
and directing cross-divisional projects. Because of the multidisciplinary nature of contamination warning
system deployment, the Steering Committee served a critical role in coordinating GCWW resources
during WS pilot design and implementation. In addition, GCWW also had existing procedures that rely
September 2008 5
-------
Cincinnati Pilot Post-Implementation System Status
on partnerships with local agencies such as those from public health and law enforcement, some of which
are described in Section 2.1.2. These existing partnerships provided a starting point for the development
of many new cross-organizational relationships that were necessary for successful implementation of the
pilot.
Table 2-2. GCWW Divisions and Responsibilities
Division
Business Services
Commercial Services
Distribution
Engineering
Supply
Water Quality & Treatment
Information Technology*
Responsibilities
Houses the office of the Director, administrative staff, and clerical support;
maintains the utility's vehicles, runs the internal storerooms for parts/supplies,
and manages the total enterprise asset management system.
Customer service and billing operations, manages the customer call center,
responsible for meter reading and premises inspections; includes
administrative and technical support personnel for these functions.
Maintenance and repair of the distribution system; including underground
water mains, service branches, valves, fire hydrants, and other appurtenances
within the right-of-way; respond to leak investigations, low pressure or water
quality complaints.
Planning, design, and inspection of new and replacement lines in the system
and at the treatment plants; also includes records management, survey, field
investigation, and contract administration personnel.
24/7 operation and maintenance of both water treatment plants, and all
pumping and storage facilities.
Defines treatment at both plants 24/7, responsible for water quality at
treatment plants and in distribution system. Performs studies and research to
ensure high quality of water and compliance with all regulations at the
treatment plants and throughout the distribution system; includes certified
laboratory capabilities.
Maintenance, upgrade and development of the software and hardware
systems to support the IT needs of all divisions.
The Information Technology
Technology Roundtable was
Division was formed in 2007, during implementation of the pilot. An Information
in existence prior to the start of the pilot (see Section 2.1.3).
2.1.2 Integrated Operational Procedures
Procedures for routine utility operations within each GCWW division were well established, and many
were applicable to contamination warning system operations. For example, GCWW had established
procedures with local law enforcement agencies to support investigation of security breaches. This and
other examples of existing GCWW procedures relevant to contamination warning system operations are
listed in Table 2-3, while more detailed discussion of existing operational procedures relevant to
component operations is included in Sections 3 through 8.
Table 2-3. Examples of Existing GCWW Procedures Relevant to Contamination Warning System
Operation
Existing Procedure
Investigation of Security
Breaches
Cryptosporidium Action
Plan
Routine Sampling Routes
Daily Chlorine Level Trend
Analysis
Customer Service
Representatives
Description
Close partnership established with local law enforcement agencies that had
jurisdiction in areas where GCWW facilities were located to coordinate timely
response to an investigation of security alarms at these facilities.
Plan developed in coordination with local public health agencies to respond to
waterborne or water-related threats to public health.
Plans and routes that were used for routine sampling and analysis of regulated
drinking water parameters.
On a daily basis, staff from the water quality and treatment division would
analyze trends in chlorine levels monitored at distribution system facilities
(tanks, reservoirs and pump stations) to assess water quality and water age.
Consumer complaints relating to water quality were directed to specific,
appropriately trained/knowledgeable personnel, and a threshold number of
calls was established to indicate a possible water quality issue in the
distribution system.
While the above procedures have some applicability to contamination warning system operation, they
were not developed or optimized for rapid detection of contamination incidents of unknown origin. Nor
September 2008
-------
Cincinnati Pilot Post-Implementation System Status
were the procedures always integrated across divisions and local partners that may need to become
involved in the investigation of possible contamination. However, a thorough understanding of these
procedures facilitates integration of contamination warning system operations with routine utility
operations.
2.1.3 Integrated Information Management
Effective information management and integration is at the heart of a contamination warning system, and
thus information technology plays a key role. GCWW utilizes a number of information technology
systems to support utility operations; however, many of these systems are isolated from one another
making integration a challenge. GCWW recognized the importance of system interoperability for more
efficient utility operations, and took steps towards this goal independent of the requirements of a
contamination warning system. For example, an Information Technology Roundtable was formed with
representatives from all GCWW divisions and information technology systems to improve coordination
and planning for information technology related projects. In 2007, GCWW took this a step further and
formed an Information Technology Division. GCWW also developed an Information Technology
Strategic Plan (GCWW, 2006), with one key goal being improved information integration and system
interoperability. While the goals of this master plan are consistent with those of a contamination warning
system, the timeline for implementation of this plan was not aligned with the schedule for pilot
implementation. Thus, a more modest integration strategy had to be developed for the pilot.
A number of existing GCWW information technology systems were leveraged to support the WS pilot;
the relevant systems are briefly described in Table 2-4. While these systems were not fully integrated,
GCWW had implemented a number of standards that made it easier to combine data from multiple
systems to support the contamination warning system pilot. One such standard is the use of Oracle as the
database platform for all information technology systems, which set the standard for data management in
the pilot.
Table 2-4. GCWW Information Technology Systems Relevant to Contamination Warning System
Operations
GCWW Information
Technology System
Information
Technology
Infrastructure and
Networks
Water Quality
Database
Supervisory Control
and Data Acquisition
(SCADA)
EMPAC
DWC / Hydra
Banner
Description
GCWWs information technology environment is served by a combination of Local and Wide
Area Networks (LANs and WANs). Ethernet-based LANs are deployed in all major GCWW
facilities to provide network services to user workstations and servers, permitting access to
the databases for SCADA, EMPAC, and Banner, among others. Individual LANs are
interconnected with a WAN operated and maintained by the Cincinnati Regional Computer
Center behind a firewall that protects the content, function, and integrity of all systems
operating on the WAN.
An Oracle database that collects, manages, and reports water quality data related to
regulatory compliance, consumer water quality complaint investigations, and results from
water quality and treatment studies. A separate Oracle Gas Chromatograph database,
containing over 20 million records, stores laboratory analytical results.
GCWWs SCADA system provides real-time control and monitoring of the treatment plants,
pump stations and storage facilities. During the pilot, the system underwent a major upgrade
to a Citect system that runs on a Windows platform.
EMPAC is a software application that provides integrated functionality for work orders,
inventory, and fixed asset management. The system interfaces with the utility billing system,
the PeopleSoft financial system, the City's geographic information system (CIS), the City's
human resources system, and the City's financial systems.
DWC is a custom application that integrates CIS functionality with EMPAC, created to
manage work in the distribution system not associated with a fixed asset, such as main
repairs. The Commercial, Water Quality and Treatment, and Distribution divisions use DWC
to create and search for work requests and work orders relating to maintenance of the
GCWW distribution infrastructure. Hydra is a web browser user interface to the work order
system that replaced DWC over the course of the pilot.
The Banner billing system is an Oracle-based, enterprise application that manages
customer-based information and issues a combined utility billing statement for water,
sewerage, fire, and stormwater. It is operated and maintained by the Commercial Services
Division.
September 2008
-------
Cincinnati Pilot Post-Implementation System Status
GCWW Information
Technology System
Description
Geographical
Information System
(CIS)
GIS displays maps and models of the water distribution system that enables users to make
better decisions by revealing trends and patterns that are not easily seen in other data
presentation formats. GCWW runs ESRI Arclnfo 9x, ArcView 3x, and Gen7, but the ESRI
application suite has been extensively customized by the Engineering Division. DWC/Hydra
uses some of the shape files that are created by the GIS system, but beyond this there is no
integration or direct interfaces between the GCWW GIS and the DWC/Hydra, Banner, and
EMPAC applications and databases. GCWW also participates in the Cincinnati Area GIS
consortium.
2.1.4 Summary of Identified Gaps
GCWW's existing programmatic and procedural capabilities coupled with an extensive information
technology infrastructure provided a solid foundation for the system-level design and management of a
contamination warning system; however, modifications and enhancements were necessary to meet the
design objectives summarized in Table 2-1. To achieve comprehensive project management for
contamination warning system deployment, plans needed to be developed, teams established, and
mechanisms put in place to monitor progress and keep the effort aligned with the overarching goals of the
pilot. While existing procedures were in place that addressed some elements of contamination warning
system operations, they had not been optimized to include all relevant participants nor to resolve possible
contamination incidents in a relatively short period of time (e.g., less than a day). GCWW had many
information technology systems that were leveraged for the contamination warning system; however,
many of these systems operated independent of one another and some lacked capacity or key functionality
needed for the pilot. The specific gaps identified for each project element are summarized in Table 2-5.
Table 2-5. System-Level Gap Analysis
Project Element
1. Comprehensive
Project Management
2. Integrated Operational
Procedures
3. Integrated Information
Management
Description of Gap
A dedicated project management structure needed to be created to support
deployment of a contamination warning system, including: developing a project
plan and schedule, establishing project teams, establishing agreements, and
tracking progress.
Existing procedures were not comprehensive for all monitoring and surveillance
components and were not optimized for rapid investigation of possible signs of
contamination. In many cases, multiple divisions and external agencies needed
to be integrated into the operational procedures.
Many information technology systems essential to operation of the contamination
warning system were not interoperable. There was also a lack of
standardization in some key areas, such as storage of geo-spatial information.
Some information technology systems, such as SCADA, lacked capacity to
support contamination warning system operations. There was not a central
repository for contamination warning system information and there were no
automated event detection systems.
2.2 Post-Implementation Status
The following section provides the post-implementation status of the contamination warning system pilot
with respect to project management and system integration. The gaps identified at the conclusion of the
previous section were addressed to the extent possible. The major products resulting from the system
integration efforts include: integrated operational procedures, a dedicated data repository for the
contamination warning system, and most importantly, an operational contamination warning system
consisting of monitoring and surveillance components integrated through procedures that transition
seamlessly to consequence management once a possible contamination incident is identified.
2.2.1 Comprehensive Project Management
Deployment of the first WS contamination warning system pilot was a cooperative effort between the
City of Cincinnati, led by GCWW, and EPA. This working relationship was formalized through a
Cooperative Research and Development Agreement between the City of Cincinnati and EPA (USEPA,
September 2008
8
-------
Cincinnati Pilot Post-Implementation System Status
2006a). This agreement established overall project goals, expectations, and responsibilities for the
project. The agreement vehicle also allowed EPA to provide equipment and assistance to GCWW, while
collecting data from the pilot for use in evaluation and expansion of the program to the benefit of the
entire drinking water industry.
As an addendum to this formal agreement a system-level work plan was developed, which established the
scope of the project, approach to implementation, and a high-level schedule. While more detailed work
plans were necessary for design and implementation of the individual components, this overarching work
plan provided an effective means of coordinating, tracking, and prioritizing activities across the entire
pilot.
Due to the magnitude of the project, dedicated teams were formed to design and implement each
component. Parallel teams were formed by EPA and Cincinnati, and these teams were multidisciplinary
to ensure that all aspects of component design and operation were considered. This was particularly
important for the information technology elements of each component because many share common
hardware or software, and inclusion of information technology representatives on each team helped to
ensure that common systems met the needs of all components relying on those systems. For example, a
SCADA system was installed parallel to GCWW's existing system to manage and display information for
both the water quality and enhanced security monitoring components.
While the Cincinnati component teams were largely made up of staff and supervisors from the various
GCWW divisions listed in Table 2-2, a number of local partners were involved in various aspects of pilot
implementation. Table 2-6 lists the partners with the most substantial involvement during design and
implementation of the contamination warning system and provides a summary description of their roles.
By far, consequence management involved the largest number of external partners; a more complete
listing of these partners and their respective roles and responsibilities in consequence management is
provided in Section 8.
Table 2-6. Local Partners Providing Support to GCWW during Implementation of the WS Pilot
Local Partner
Cincinnati Fire Department
Cincinnati Health Department
City Call Center
City Council
City Facilities and Maintenance
City Manager's Office
Contract Laboratories
Role in Contamination Warning System Implementation
• Host several water quality monitoring stations
• Host sampling locations for the sampling and analysis component
• Data provider for public health surveillance (91 1 and EMS data)
• Supported development of the consequence management plan
• Provide field response and HazMat support during consequence
management
• Informed design of public health surveillance systems to meet
pilot objectives
• Monitor public health surveillance systems and investigate
anomalies detected by the systems
• Supported development of the consequence management plan
• Provide information and consultation during consequence
management
• Implemented procedures to funnel water quality calls to GCWWs
call center
• Approved the charter to enter into a cooperative research and
development agreement with EPA to launch the first WS pilot
• Facilitated site selection and installation activities for water quality
monitoring stations located at city facilities
• Supported the installation of wireless routers at Cincinnati Fire
Department stations for public health surveillance
• Entered the City of Cincinnati into the cooperative research and
development agreement with EPA to launch the first WS pilot
• Supported development of the consequence management plan
• Provides support for risk communication during consequence
management
• Provide support and surge capacity for chemical analyses
September 2008
-------
Cincinnati Pilot Post-Implementation System Status
Local Partner
Drug and Poison Information Center
Greater Cincinnati HazMat Team
Hamilton County Emergency
Management Agency
Hamilton County Public Health
Local Fire Departments
Local Law Enforcement Agencies
Metropolitan Sewer District
Ohio Department of Health
Ohio EPA
Regional Computer Center
Terrorism Early Warning Group
Role in Contamination Warning System Implementation
• Data provider for public health surveillance
• Informed design of public health surveillance systems to meet
pilot objectives
• Monitor poison control center calls and investigate anomalies as
appropriate
• Provide information and consultation during investigation of a
suspected contamination incident
• Provide field response and HazMat support during consequence
management
• Supported development of the consequence management plan
• Coordinate alternate water supplies during consequence
management
• Informed design of public health surveillance systems to meet
pilot objectives
• Monitor public health surveillance systems and investigate
anomalies detected by the systems
• Supported development of the consequence management plan
• Provide information and consultation during consequence
management
• Host several water quality monitoring stations
• Host one water quality monitoring station
• Investigate reported security breaches at GCWW facilities
• Supported development of the consequence management plan
• Lead the criminal investigation and related support during
consequence management
• Provide analytical support for baseline and triggered sampling
• Supported development of the consequence management plan
• Perform analysis of baseline and triggered samples for select
biological agents and radiochemicals
• Supported development of the consequence management plan
• Supported development of the consequence management plan
• Provide support for risk communication and remediation/ recovery
issues
• Reviewed and approved all information technology architecture
using the City's wide area network
• Supported the design and implementation of comprehensive call
management for the consumer complaint surveillance component
• Serve as a hub of law enforcement intelligence in the region, and
provide support during credibility determination
During implementation of the Cincinnati pilot, a large amount of equipment was procured and installed
throughout the GCWW service area. The project management team implemented a comprehensive
tagging and tracking system to ensure that all equipment was accounted for as well as to assist in
monitoring maintenance activities. Each piece of equipment with substantial value, typically over $1,000,
was tagged with both EPA and GCWW property tags. The information for each piece of equipment was
entered into a database used for periodic inventories. This information was also entered into GCWW's
asset management system to facilitate transfer of ownership to Cincinnati.
More than 70 documents were developed during design and implementation of the Cincinnati pilot. They
include assessment reports, requirements documents, design documents, as-built drawings, operating
procedures, operations and maintenance guides, and training materials. In order to track and maintain
version control of this extensive compilation of documentation, a document inventory / library was
developed. The library serves as a valuable resource by allowing all GCWW and EPA personnel to
quickly locate specific documentation and verify that the version is current.
2.2.2 Integrated Operational Procedures
To support routine operation of the Cincinnati pilot, it was necessary to develop an operational strategy
that fulfilled the primary objective of a contamination warning system - timely detection of
September 2008
10
-------
Cincinnati Pilot Post-Implementation System Status
contamination incidents - that was also aligned with utility operational procedures. Consideration of
typical utility operational procedures during development of a contamination warning system operational
strategy will help to integrate the system into routine operations, which is essential to the long-term
sustainability of the system. For the Cincinnati pilot, the operational strategy took the form of a
document titled Concept of Operations for the GCWW Contamination Warning System (USEPA, 2007b).
The overarching objective of the Cincinnati concept of operations is to describe the processes and
procedures involved in routine operation of the contamination warning system, including the initial
investigation of alarms. To achieve this objective, it establishes specific roles and responsibilities,
process and information flows, and checklists to support the systematic review of alarms. Process flow
diagrams describe the sequence of activities that are performed during the detection and investigation of
alarms and system anomalies. The information flow diagrams identify information technology systems,
databases, and user interfaces used during routine monitoring and surveillance, event detection, and alarm
investigation. Checklists were developed for each job function involved in contamination warning system
operations and are intended to guide users through the investigation of alarms.
While the operational strategy does deal with monitoring and surveillance activities for individual
components, the overall operational strategy had to be integrated across the entire system to ensure that
investigational procedures and job functions were consistently defined throughout the system. This was
achieved through a three-stage development process that included: 1) conducting a system-level resource
assessment; 2) developing a concept of operations for each individual component; and 3) conducting a
system-level review and revision to ensure that common elements of the concept of operations are
consistent across all components.
Once the operational strategy for the Cincinnati contamination warning system was developed, training
materials were developed and used to orient front-line staff and supervisors on system operations.
2.2.3 Integrated Information Management
One of the primary design objectives of the Cincinnati pilot was to provide information used in routine
operations and alarm investigation in a timely and efficient manner to the end users. The requirements for
the information technology systems supporting the pilot were derived in part from the information flows
in the concept of operations, which identified key users and their information needs as well as existing
systems that could be leveraged to support operations (see Table 2-4). Key gaps were identified with
respect to information integration, limitations in functionality or capability of existing systems, and event
detection. Many of these gaps were addressed at the component level, as discussed in Sections 3 through
8. Efforts at the system-level focused on coordinating data management across the components and
filling key gaps in capability that could be best addressed from a broader perspective.
The contamination warning system pilot generates large quantities of new data continuously, in addition
to mining data already collected by GCWW and other local agencies. To support data management for
several components, two dedicated servers were installed on the GCWW LAN: one an application server
and the other a database server. The application server hosts Oracle Application Server software and
other custom application software to support event detection for consumer complaint surveillance and
public health surveillance. The database server hosts Oracle Database Management System software and
the Water Security Data Repository (WSDR), which supports data storage needs in cases where existing
systems cannot serve this function for all monitoring and surveillance components. Additional detail on
the design and operations of these centralized IT systems can be found in Water Security Initiative Data
Management at GCWW- Centralized Hardware and Software Operations and Maintenance Manual
(USEPA, 2007c). While these two centralized data systems do not achieve complete information
integration across all components, they are an important step towards that objective, which may
ultimately be realized through implementation of GCWW's Information Technology Strategic Plan
(GCWW, 2006).
September 2008 11
-------
Cincinnati Pilot Post-Implementation System Status
In addition to the application and centralized database servers, a number of existing systems were
leveraged for the pilot. In cases where an existing system could not meet the requirements, new systems
were designed and implemented. A comprehensive summary of all major information technology
systems used in the Cincinnati pilot, including which contamination warning system component(s) each
system supports, is presented in Table 2-7.
Table 2-7. Summary of IT Systems Used in the Cincinnati Contamination Warning System
Existinc
System Name*
GCWW SCADA
System
Water Quality
Database
EMPAC and Hydra
Banner
Avaya Interactive
Voice Response
CIS
Priority Dispatch
Emergency Medical
Service Tablet
Fireweb Database
WQM
^
•/
S&A
^
S
•/
Systems and Tools Operated by GCWW
ESM
^
•/
CCS
V
s
•/
V
s
•/
PHS
V
•/
•/
s
s
Role in Contamination Warning System
A control system that collects and displays
operational data and security alarms for
distribution facilities. Provides information
used in the investigation of any component
alarm.
An Oracle database that serves as a
repository of data for analytical results, field
test results, and information related to water
quality consumer complaints.
EMPAC is an asset management system that
is used to track work orders, which are
accessed via a web application called Hydra
that includes a GIS application for display of
work orders. Used to track work orders created
in response to customer water quality calls,
and provides information about ongoing
distribution system work, which is used in the
investigation of component triggers.
A customer information and billing system from
which EMPAC draws customer premises data.
A call menu routing system used to triage and
direct customer calls. The menu was revised
to facilitate tracking of customer water quality
calls.
GCWW maintains its own GIS database using
ESRI Arclnfo and ArcView, and this system is
used with DWC for spatial display of customer
water quality calls. The City of Cincinnati
maintains a separate system, CAGIS, which is
used for spatial analysis of data from public
health surveillance.
When 911 calls are received with medical
emergencies, Cincinnati Fire Department uses
Priority Dispatch software to prioritize EMS
runs. This software also serves to standardize
information collected by dispatch. Call records
are transferred from Priority Dispatch to the
91 1/EMS Database.
EMS technicians enter run data on their
handheld EMS tablet while in the field. The
data are uploaded from the tablet to the
91 1/EMS Database via a wireless link when
the vehicle returns to a Cincinnati Fire
Department station.
Database maintained by Cincinnati Fire
Department for storage of 91 1 and Emergency
Medical Service (EMS) data. Databases
reside on the 91 1/EMS Cincinnati Fire
Department Server. A replica database has
been created to support WS data transfer and
analysis.
September 2008
12
-------
Cincinnati Pilot Post-Implementation System Status
Existinc
System Name*
Drug and Poison
Information Center
Systems
National Retail Data
Monitor
Real-time Outbreak
Disease
Surveillance
WQM
S&A
Systems and Tools Operated by GCWW
ESM
CCS
PHS
s
s
V
Role in Contamination Warning System
Drug and Poison Information Center system
alerts are phoned to Local Public Health. Data
from these systems will be managed
independent of the WS contamination warning
system and will not be transferred to the
WSDR.
National Retail Data Monitor alerts are sent by
email to Local Public Health.
Real-time Outbreak Disease Surveillance
(RODS) system alerts are sent by email to
Local Public Health.
Dedicated WS Systems and Tools
System Name*
Water Security Data
Repository
Water Security
Application Server
Consumer
Complaint
Surveillance Event
Detection System
WS SCADA System
EDDIES
Early Aberration
Reporting System
Event Detection Tool
SaTScan™Tool
WQM
•/
S
S
S&A
•/
ESM
•/
S
CCS
•/
s
s
PHS
•/
s
•/
V
Role in Contamination Warning System
The WSDR is an Oracle 10g database that
stores data from all components of Cincinnati
pilot. WSDR does not store all data for the
system; rather, it stores data that are not
stored in another utility or dedicated database.
The WS Application Server is a general-
purpose application server deployed to host
custom applications developed for consumer
complaint and public health surveillance.
Algorithms deployed on the WS Application
Server that continuously analyze customer
water quality Interactive Voice Response (IVR)
selection, work requests, and work orders in
search of anomalies.
A control system that collects and displays
data from water quality monitoring stations and
enhanced security locations. Alarms for both of
these components are displayed on an
associated user interface. The system also
includes remote clients that allow users
outside of GCWW's control center to view
alarm data from these two components.
The Event Detection Deployment, Integration,
and Evaluation System (EDDIES) is a custom
application designed to broker data between
event detection tools and a SCADA system. It
hosts two event detection tools that
continuously analyze data from the WS
SCADA system in search of anomalies.
Early Aberration Reporting System (EARS)
event detection algorithms will reside on the
WS Application Server at GCWW and will
analyze EMS and 91 1 data received from the
911/EMS Database.
SaTScan™ spatial analysis program will
reside on the WS Application Server at
GCWW and perform cluster analysis on 91 1
call data received from the 91 1/EMS
Database.
*System names: Water Quality Monitoring
(ESM), Consumer Complaint Surveillance
(WQM), Sampling and Analysis (S&A), Enhanced Security Monitoring
(CCS), and Public Health Surveillance (PHS)
2.2.4 Su/n/nary of Post-Implementation Status
The drinking water contamination warning system in Cincinnati has been installed, is fully operational,
and is currently generating data needed to establish baseline performance. A comprehensive project
management strategy was implemented to guide and prioritize work across all components, track
equipment, and compile supporting documentation. Integrated operational procedures were developed in
the form of a concept of operations, which was aligned with routine utility function to the extent possible.
September 2008
13
-------
Cincinnati Pilot Post-Implementation System Status
Data management activities were coordinated across components, and a centralized data system was
deployed to fill critical gaps in existing capabilities. Table 2-8 provides a summary of the post-
implementation status of the system-level elements of the Cincinnati pilot.
Table 2-8. System-Level Post-Implementation Status
Project Element
1. Comprehensive
Project Management
2. Integrated Operational
Procedures
3. Integrated Information
Management
Description of Implemented Element
A cooperative research and development agreement was established between
the City of Cincinnati and EPA. A dedicated project management structure was
established for implementation of the contamination warning system pilot,
including: a project plan and schedule, project teams, and mechanisms for
tracking progress. Systems are in place for tracking equipment and
documentation.
A comprehensive concept of operations was developed to guide routine
operation of the contamination warning system and efficient investigation of
alarms from any component. Procedures were integrated across components
and with typical utility operations.
A centralized data system was deployed to support data management activities
for all components. Data management strategies for individual components
were implemented in a coordinated manner that leveraged existing systems to
the extent possible.
Figure 2-1 provides a summary of the level of effort associated with design and implementation of
project management and system integration activities for the Cincinnati pilot. Comprehensive project
management activities, as summarized in Table 2-8, relied on support from EPA, GCWW, and local
partners. Local partners also contributed to the effort associated with integrated information management
activities to ensure that the hardware, software, and supporting code deployed as part of the pilot did not
introduce any vulnerabilities to the City's Metropolitan Area Network.
1 .0
1 R -
.4
1.2 '
12
S5 1 o -
>•
<5 °-8
+j
(/>
OR -
OA -
A
.2
0.1
n Local Partners LOE
ncc
HEP
)WWLOE
ALOE
0.1
Comprehensive Project Integrated Operational Integrated Information
Management Procedures Systems
Figure 2-1. Level of Effort for the Cincinnati Pilot Project Management and System Integration
Design and Implementation Activities (December 2005 - December 2007)
September 2008
14
-------
Cincinnati Pilot Post-Implementation System Status
Figure 2-2 presents a summary of the costs associated with design and implementation of project
management and system integration activities for the Cincinnati pilot. Extramural labor costs include
contractor activities associated with coordination of system components and implementation of the
centralized data system, including development of software, database applications, and hardware
installation. Figure 2-2 also includes equipment and consumable supplies costs associated with the
procurement of components of the centralized data system, as well as contracted services. Contractor
travel costs were not included in these calculations.
$600
„ $500
TJ
C
| $400
o
(A
O
$100
Equipment,
Consumables,
and Purchased
Services
n Extramural LOE
Comprehensive
Project Management
Integrated
Operational
Procedures
Integrated Information
Systems
Figure 2-2. Extramural Costs Associated with Design and Implementation of Project Management
and System Integration Activities (December 2005 - December 2007)
September 2008
15
-------
Cincinnati Pilot Post-Implementation System Status
Section 3.0: Online Water Quality Monitoring
Online water quality monitoring is included as a component of the contamination warning system due to
its demonstrated potential to rapidly detect contamination through changes in several commonly
measured water quality parameters. These changes may result from the aqueous chemistry of the
contaminant (e.g., dissolution of an organic compound may result in an increase in the concentration of
total organic carbon) or from reactions with the disinfectant residual (e.g., oxidation of a reactive
contaminant consumes the free chlorine residual). While there are limited empirical data regarding the
impact of many contaminants of concern on conventional water quality parameters, there has been a
substantial amount of research over the past few years demonstrating that many contaminants of concern
can produce measurable changes in conventional water quality parameters. Furthermore, many of these
contaminants have been shown to impact water quality at concentrations well below reported lethal dose
concentrations, as will be discussed below.
In the Cincinnati pilot, the following water quality parameters are monitored at seventeen locations (two
entry points to and fifteen locations within the distribution system): chlorine residual (CL), total organic
carbon (TOC), oxidation-reduction potential (ORP), pH, conductivity (COND), turbidity (TURB), and
temperature (TEMP). Data from these monitoring stations are transmitted to the SCADA system at the
utility's control center where an event detection system analyzes the data to monitor for water quality
anomalies that might indicate a possible contamination incident. If the event detection system detects a
water quality anomaly, it will generate an alarm to alert operators who initiate an investigation into a
possible contamination incident.
The water quality monitoring component design objectives are shown in Table 3-1, and were derived
from the overarching performance objectives of the contamination warning system as described in
WaterSentinel System Architecture (USEPA, 2005a). GCWW's pre-existing capability with respect to
each design element listed in Table 3-1 is summarized in Section 3.1.
Table 3-1. Water Quality Monitoring Component Design Objectives
Design Element
Design Objective
1. Water Quality
Monitoring Equipment
Deploy monitoring stations consisting of a suite of water quality sensors that
provide broad contaminant coverage. For water systems using free chlorine
residual, current research shows that CL, TOC, ORP, and conductivity (COND) are
the most reliable parameters for contaminant detection. The sensors and
equipment used in the design of the water quality monitoring stations must function
within specifications and consistently produce accurate data. Proper instrument
maintenance and routine calibration are essential to meeting this design objective,
and the effort required to maintain the equipment must be acceptable to the utility.
2. Water Quality
Monitoring Network
Deploy several water quality monitoring stations at locations in the distribution
system that optimize spatial coverage and timeliness of detection. This objective
can be achieved through use of calibrated distribution system models along with
sensor placement optimization tools.
3. Data Management
and Communications
Deploy a communication system that transfers data from remote monitoring
stations to a centralized data repository, such as a SCADA system, with minimal
delay (i.e., less than two minutes from the time of measurement). The SCADA and
communication network must be operational at least 99 percent of the time.
4. Water Quality Event
Detection
Deploy an automated event detection system that continuously analyzes the large
amount of water quality data produced by the monitoring network to search for
anomalies that may be indicative of contamination. The event detection system
will detect anomalies that are not due to contamination, and thus may be
considered false positives that require some expenditure of resources. The rate of
false alarms can be reduced by providing ancillary data to the event detection
system, such as sensor alarms and tank/pump operational data. Furthermore,
written procedures can guide the timely and systematic investigation of water
quality alarms, which can further reduce the unnecessary expenditure of
resources.
September 2008
16
-------
Cincinnati Pilot Post-Implementation System Status
3.1 Pre-lmplementation Status
Prior to installation of the contamination warning system, water quality monitoring equipment installed in
the GCWW distribution system included only instruments commonly used for drinking water
applications, such as CL analyzers. The sensors transmitted data, mostly using standard telephone lines,
to the existing GCWW SCADA system located at the utility's operations center. Water quality plots were
produced continuously and reviewed daily by staff from the Water Quality & Treatment Division, but
there was no real-time event detection system deployed for early identification of water quality
anomalies.
3.1.1 Water Quality Monitoring Stations
Pre-existing water quality monitoring equipment in the GCWW distribution system included forty (40)
chlorine monitors, three (3) pH meters and two (2) turbidimeters. These sensors were located at twenty-
two (22) sites across the distribution system. Twenty (20) of these sites were GCWW facilities and the
remaining two (2) were utility boxes permanently installed alongside a public road in critical locations.
Each of the utility-owned storage facilities included one or more analyzers, monitoring residual chlorine
levels at various points in the pump station flow stream. For example, multiple source feeds and
discharge locations at one pump station require the use of multiple analyzers.
At the Miller and Bolton water treatment plants, water quality monitoring at the plant discharge conforms
to standards for the industry and regulatory requirements. Specifically, the parameters monitored at the
plant discharge are CL, fluoride concentration, pH, TURB, and TEMP.
All operations and maintenance of the online water quality monitors in the distribution system was
performed by utility personnel, primarily consisting of monthly refilling of reagents for the CL analyzers,
and calibration when necessary.
3.1.2 Water Quality Monitoring Network
The primary objective of the sensor network prior to contamination warning system implementation was
to monitor chlorine residual levels at key transmission and storage facilities for water quality degradation
as indicated by reduced chlorine residual levels. Monitors were first installed at the tanks furthest from
the treatment plants based on the assumption that these locations had the oldest water. Recognizing the
benefit of these sensors, and due to their successful application in the field, GCWW expanded the
deployment of chlorine sensors in the early 2000 's to include the installation of new sensors in all future
tank and facility upgrades. By the commencement of the WS pilot at GCWW in 2006, twenty (20) of
GCWW's storage tanks and pump stations were equipped with chlorine sensors. At that time, the utility
also maintained two (2) additional sites that were deployed in stand-alone utility boxes permanently
installed alongside the public road in critical locations. There were no water quality monitoring systems
at non-utility owned sites, and thus there were no monitoring capabilities on distribution lines closer to
consumers.
Water quality monitoring efforts prior to the implementation of the contamination warning system were
supported by a hydraulic and water quality model of the utility's distribution system, which was built and
evaluated using the widely-accepted EPANET software platform. The model was a skeletonized version
of GCWW's distribution network, with 59 percent of the 3,000 miles of pipe in the system being
represented. This included all pipes in the utility's retail distribution area that had diameters 12 inch and
larger (as well as all significant 6 and 8 inch diameter pipes), and wholesale demands simply included at
the hydraulically appropriate nodes. One version of this model was maintained by the Water Quality and
Treatment Division personnel who used it to predict water quality parameters, particularly chlorine
residual, in the distribution system. This informed the utility's efforts to deploy chlorine monitors at
remote facilities.
September 2008 17
-------
Cincinnati Pilot Post-Implementation System Status
3.1.3 Data Management and Communications
A well-developed data management and communication system was in place at GCWW primarily to
support system operations and control, as well as monitoring water quality, flows, and pressures at select
sites. The existing communications network included the following:
• Frame relay digital data telephone lines to remote terminal units at the Bolton Plant and major
pump stations.
• Dedicated point-to-multipoint analog telephone lines with modulator/demodulator interfaces to
remote terminal units at most other remote facilities.
• Unlicensed frequency hopping spread spectrum connections to several water storage tanks.
The analog telephone lines provide relatively low data transfer rates and are somewhat unreliable
primarily because they are not monitored by the telephone company and are difficult and expensive to
maintain. GCWW had previously conducted a tabletop radio path analysis based on their desire to
migrate the existing analog telephone line network to a private radio network. This path analysis
indicated that it would be economically challenging to deploy a private radio network throughout the
distribution system due to the hilly topography of the greater Cincinnati area.
Data from the remote terminal units are transferred over the communications network to a centralized
SCADA system. The SCADA system is located at the operations center, which is co-located with the
Richard Miller Treatment Plant. At the start of the pilot, the existing GCWW SCADA system was in the
process of being upgraded and migrated to a new Microsoft Windows based system, using Citect's
graphical user interface application software. However, the communications links were not being
upgraded. Therefore, GCWW determined early in the WS pilot that the existing communications network
would not support the communications and data storage requirements of the project. The video
communication and storage requirements of the enhanced security monitoring component of the pilot
were a key consideration in GCWW's decision (see Section 5). Therefore, a separate or parallel SCADA
network, complete with a separate communications network, was required for the WS pilot.
3.1.4 Water Quality Event Detection
In the context of the water quality monitoring component of a contamination warning system, an event
detection system is defined as one or more algorithms that continually analyze water quality and related
data to monitor for anomalous conditions. The application of event detection systems to near real-time
water quality data is a relatively recent innovation, and thus has not been used in drinking water utilities
with the exception of a handful of research and demonstration projects.
GCWW did not have an event detection system that met the requirements described above. However, the
utility did have a method for identifying significant excursions from an expected range of water quality
conditions, most notably CL. This included the establishment of process limit set-points configured in the
local programmable logic controllers (PLC) which, when exceeded, activate alarms in GCWW's SCADA
system. In addition, staff from the Water Quality & Treatment Division periodically performed
retrospective analysis of water quality data from the distribution system monitoring locations to identify
trends and methods of improving water quality, typically by reducing water age in storage facilities.
3.1.5 Summary of Identified Gaps
GCWW's existing water quality monitoring system was extensive and well designed for the purpose of
monitoring and managing water age, as indicated by CL levels. This existing system provided a solid
foundation and experience-base for the design of a contamination warning system; however, it did not
meet the design objectives summarized in Table 3-1. Specifically, the pre-existing water quality
monitoring system did not provide the required level of contaminant coverage, spatial coverage, timely
detection of contamination incidents, reliable indication of potential contamination, or degree of
automation necessary for near real-time detection. The specific gaps identified for each design element
are summarized in Table 3-2. These gaps provided the design basis for enhancements to the GCWW
September 2008 18
-------
Cincinnati Pilot Post-Implementation System Status
water quality monitoring system, and the post-implementation status of the resulting system is described
in Section 3.2.
Table 3-2. Water Quality Monitoring Gap Analysis
Design Element
Description of Gap
1. Water Quality
Monitoring Equipment
The existing network of monitors, consisting primarily of chlorine sensors, is
insufficient to provide coverage of the contaminants of concern.
There was limited experience with other types of water quality sensors
deployed for distribution system monitoring applications. Thus, there was
insufficient information regarding the operating and maintenance
requirements of water quality monitoring equipment necessary for a
functional contamination warning system.
2. Water Quality
Monitoring Network
The existing monitoring network was based on utility-owned locations:
primarily pump stations, reservoirs, and storage tanks. These facilities are
typically located on large transmission lines; thus, spatial coverage of the
distribution system was limited.
Existing monitoring locations were selected primarily to monitor chlorine
residual as an indication of water age. Locations were not selected to
optimize detection of contamination incidents and reduce the time to
detection.
3. Data Management
and Communications
The utility was in the midst of a major SCADA upgrade at the time of the
project, making it impractical to add a significant number of new monitoring
locations under the schedule for the pilot.
The existing communication network was not being upgraded along with
SCADA. While the existing network likely could have supported the
expanded water quality monitoring network, it lacked the bandwidth to
transmit video data that was essential to the enhanced security monitoring
component.
4. Water Quality Event
Detection
The utility did not have an event detection system capable of rapidly detecting
water quality anomalies that could be indicative of contamination.
The utility did not have a formal procedure for the timely and systematic
investigation of water quality anomalies detected through online monitoring.
3.2 Post-Implementation Status
The design and installation of the water quality monitoring component of the contamination warning
system was performed in two phases. Phase 1 included the design and installation of two types of water
quality monitoring stations that allowed for comparison of sensors from different manufacturers. During
Phase 1 a parallel communications network and SCADA system were deployed specifically to support
both the water quality and enhanced security monitoring components of the contamination warning
system. Phase 2 of the project utilized lessons-learned from Phase 1 to design and install a third type of
water quality monitoring station that leveraged the best attributes of the two prototypes. The parallel
communications network was expanded to include the additional stations, and data from all stations were
continuously transmitted to the parallel SCADA system. Deployment of two event detection system and
a custom interface application also occurred during Phase 2. The following subsections describe the post-
implementation status of the: water quality monitoring stations; water quality monitoring network; data
management and communication system; and event detection system.
3.2.1 Water Quality Monitoring Stations
Three types of water quality monitoring stations were designed and installed, which are referred to as
Types-A, B, and C monitoring stations (see Figures 3-1, 3-2, and 3-3). Specific sensors were selected
based on the results of pipe-loop testing and evaluation studies (Hall, 2007a; Hall, 2007b), and GCWW
experience. Additional design features of the monitoring stations were based on recommendations from
the Interim Voluntary Guidelines for Designing an Online Contaminant Monitoring System (Pikus, 2004).
The main difference among the three types is the manufacturer and model of installed instrumentation.
All station types include instrumentation to measure the following water quality parameter: pH, TURB,
COND, CL, TEMP, and TOC. Type-B and C systems also include sensors which measure ORP. During
Phase 1 three (3) Type-A and five (5) Type-B systems were installed, while nine (9) Type-C stations were
September 2008
19
-------
Cincinnati Pilot Post-Implementation System Status
installed during Phase 2, resulting in a total of seventeen (17) water quality monitoring stations installed
during the pilot. The specific sensors installed in each type of monitoring station are described in Table
3-3 below.
Table 3-3. Water Quality Parameters and Equipment Selected for Water Quality Monitoring
Stations
Parameter
PH
COND
TURB
ORP
TEMP
CL
TOC
Type-A
Hach GLI pHD
Hach GLI 3422
Hach 1720D
-
Hach GLI pHD
HachCL-17
Hach 1950Plus
Type-B
US Filter Depolox 3+,
YSI 6500 multiparameter probe
YSI 6500 multiparameter probe
YSI 6500 multiparameter probe
YSI 6500 multiparameter probe
YSI 6500 multiparameter probe
US Filter Depolox 3+,
YSI 6500 multiparameter probe
GE-Sievers 900
Type-C
Hach pHD sc
Hach D3422C3
Hach 1720E
Hach pHD sc
Hach pHD sc
HachCL-17
GE-Sievers 900
The Type-A system instruments, with the exception of TOC, were provided pre-configured on Hach
WDMP monitoring panels. The Type-C system instruments, with the exception of TOC, were provided
pre-configured on Hach WDMP-sc monitoring panels, which are updated versions of the panels included
on the Type-A systems.
One each of the Type-A and Type-B systems also includes an S::CAN "carbo::lyser" TOC/TURB
analyzer with "con::stat" transmitter. The carbo::lyser is an optical (visible-ultraviolet range) instrument,
as contrasted with the standard, chemically-based methodology used by the Hach TOC instrument. It was
provided as a redundant TOC analyzer so that its performance could be evaluated relative to the standard,
yet more complicated, TOC instrumentation.
Each Type-A systems also includes a Hach Event Monitor. This is a stand alone computer which analyzes
the measured water parameters and may register an Alert or Alarm locally. The event monitor alarm is
transferred back to the SCADA system at the California Control Center to alert operators. The event
detection systems are discussed further in Section 3.2.4.
Each Type-A station is equipped with an air compressor and a Parker-Balston gas generator. The gas
generator removes carbon compounds (carbon dioxide, etc.) from the compressed air stream to provide
carbon-free air to the Hach TOC analyzer, as required for its operation. Each compressor has an
automatic drain valve which periodically purges condensed water from the air storage tank. Both the
drain valve and the gas generator can be powered from the monitoring station uninterruptible power
supply (UPS) convenience receptacle on the system. However, due to the starting load of the generator,
and some associated experience with blown system fuses, the generator is powered from a separate power
circuit, and not from the water quality monitoring station's electrical system.
The Sievers TOC analyzer does not require compressed air for operation, so air compressors are not
provided with Type-B or Type-C stations. The Sievers instruments, however, are supplied with inorganic
carbon removal systems to ensure that the resulting measurement is representative of only the organic
carbon content.
September 2008
20
-------
Cincinnati Pilot Post-Implementation System Status
Hach Event Monitor
TOC analyzer
Electrical and
PLC cabinet
s::can TOC analyzer not shown
Transmitters and
local
displays
Turbidity analyzer
pH/Temp. sensor
Chlorine analyzer
Conductivity sensor
Water supply
manifold
Sample collection
Bottle (see Figure 3-
3 for current
configuration)
Figure 3-1. Type-A Water Quality Monitoring Station
September 2008
21
-------
Cincinnati Pilot Post-Implementation System Status
TOC analyzer -
GE/Sievers 900
Chlorine/pH analyzer-
US Filter Depolox 3+
Multiparameter
Probe - YSI
S::CAN TOC analyzer not shown
Transmitters and
local displays
Electrical and
PLC cabinet
Water supply
manifold
Sample collection
bottles - not shown
Figure 3-2. Type-B Water Quality Monitoring Station
September 2008
22
-------
Cincinnati Pilot Post-Implementation System Status
Transmitter and
local display
TOC analyzer -
GE/Sievers 900
Water supply
manifold
Sample collection
bottles
ORP sensor
pH sensor
Conductivity sensor
Electrical and
PLC cabinet
Chlorine analyzer
Turbidity analyzer
Figure 3-3. Type-C Water Quality Monitoring Station
September 2008
23
-------
Cincinnati Pilot Post-Implementation System Status
The water quality monitoring stations are part of a demonstration pilot, and thus are configured as free
standing systems mounted on casters for easy set-up and relocation. They are neither hard-wired to a
power source nor hard-piped to a water source or drain. Each system includes an electric cord compatible
with a standard 120 VAC power receptacle. Each system is powered from an UPS which provides
approximately 24 hours of operation of all instruments, local PLC, and communication equipment in the
event the main power supply fails.
Water supply and drains are routed through flexible hoses. Influent water flows through a Y-type strainer
to prevent large particles from plugging the small diameter flow ports of some of the instruments. A
regulator reduces the pressure from the distribution main to approximately 30 psig as verified through a
pressure gauge (0-60 psig) mounted directly downstream of the regulator (Type-C units also include a 0-
100 psig gauge upstream of the regulator). Water then flows through a pipe or manifold from which
several 1A inch diameter branch connections are provided to carry water to each sensor. One of the branch
connections is fitted with a manual ball valve and flexible tube to facilitate manual sample collection.
Another connection with ball valve is provided at the end of the sample supply pipe/manifold, with a
flexible hose connected directly to the system drain. This ball valve is normally closed, but may be
opened temporarily to flush the supply header or left partially open as a bypass to reduce the residence
time in the service connection between the distribution main and monitoring station. Two of the
connections are used for the remote sampling systems discussed below.
Each monitoring station also includes two remote sampling systems, each of which consists of an
automatically controlled solenoid valve and a 5-gallon bottle containing a small quantity of sodium
thiosulfate to quench any CL, thus preserving chlorine sensitive contaminants. Each sampling system
operates independently and only one of the two sample bottles may be filled at a time. A sampling event
can be initiated through an operator command from the WS SCADA system, or when the event detection
system alarms for a predetermined time interval. When a sample bottle has been successfully filled, the
operator is notified through the WS SCADA system and another sample cannot be collected from that
specific remote sampling system until the sample is retrieved and the solenoid valve is locally reset.
Bristol Babcock ControlWave Micro PLCs are used for collection and transmission of sensor data, and
for control of the solenoid valves that are part of the remote sampling systems. This type of PLC was
selected for compatibility with GCWW's existing equipment and is a model upgrade that GCWW plans
to switch over to completely. The PLCs are housed in an electrical panel that also contains circuit
breakers and fused terminal blocks for system isolation and protection as well as the UPS. Two- and
four-outlet electrical receptacles are provided on the monitoring stations, some of which are used for
ancillary equipment while others are available as convenience outlets for small electrical loads such as
laptop computers. Digital cellular radios, with associated firewall protected Ethernet switches and UPS,
are provided in separate enclosures.
Each water quality monitoring station is equipped with a Normal/Calibrate switch, which is used to
indicate when a station is being serviced or calibrated. The state of this switch is transmitted back to the
parallel WS SCADA, and alerts operators that the data from that monitoring location should be
considered suspect until the status is returned to "normal." When the switch is in the "calibrate" position,
the remote sampling systems are disabled, and the event detection system will not generate an alarm for
that location.
Recommended calibration and maintenance activities and schedules for Types-A, B, and C systems have
been provided to service technicians, and are included in manuals stored at each monitoring location as
well as the GCWW service technician's office. Maintenance activities are provided in tabular format,
with separate documents provided for each of the three monitoring station types. Each list includes
maintenance activities for each instrument, along with recommended intervals noted as "monthly,"
"quarterly," "semi-annually," and "annually."
September 2008 24
-------
Cincinnati Pilot Post-Implementation System Status
3.2.2 Water Quality Monitoring Network
As discussed in Section 3.1.2, existing monitoring locations within GCWW's monitoring network were
selected based on objectives different from those that guide the design of a contamination warning
system. For the pilot, the water quality monitoring network was designed to minimize average
consequences to the public over thousands of possible contamination scenarios. The methodology used to
develop the network design relies on an accurate model of the utility's distribution system, and uses a
sophisticated, optimization process that is part of the Threat Ensemble Vulnerability Assessment (TEVA)
software. The following three subsections describe: steps taken to validate the GCWW distribution
system model, the TEVA methodology, and the process for finalizing the physical monitoring network.
3.2.2.1 Distribution System Model Validation
The TEVA software used to optimize the placement of monitoring stations relies on an accurate
distribution system model. Thus, a significant part of the network design included efforts aimed at
assessing and validating the accuracy of the GCWW distribution system model.
The methodology devised was to validate the utility's model by comparing its predictions of the mass
transport of a conservative tracer through the distribution system to actual field data gathered during a
tracer study. This approach required the planning and execution of complex field work to capture the
movement of an injected tracer through the utility's distribution system followed by significant efforts to
modify the GCWW model to match the operational conditions that existed during the field study. The
final step involved analysis and comparison of the field study data to the model predictions. The goal was
to gain confidence in the accuracy of the distribution system model, or to identify modifications to the
model that were necessary to improve confidence that the monitoring network design would be as close to
optimal as practical.
Following more than nine months of planning and preparation, the field activities collected data over an
intense six-week timeframe in the fall of 2006. The tracer chemical (a calcium chloride solution injected
at a concentration that doubled the background conductivity of the water) was injected at four locations,
effectively testing the entire service area of the Richard Miller Treatment Plant, which serves
approximately 88 percent of GCWW's customers. Each injection consisted of at least six, one-hour
pulses over a 24-hour period to maximize data collection. The injections were done in series,
approximately a week apart, and targeted specific regions within the Richard Miller Treatment Plant
service area. Following each injection, specially designed conductivity meters were deployed to measure
and record the conductivity signal at approximately forty (40) locations throughout each study region.
Before the field data was analyzed and compared, significant effort was spent modifying the utility's
original model to reflect the exact distribution system conditions during the study, as captured in data
from the GCWW SCADA system. Unfortunately, this effort yielded limited success. The field testing
and model recalibration effort needed for this study to be successful was beyond the scope of this project.
Instead, the field results were compared against the original model provided by the utility that had been
calibrated for a typical summer day in 2005. Performance of the model was assessed using three metrics
that characterized how well the model predictions matched measured values with respect to: 1) profiles of
each pulse; 2) peak of each pulse; and 3) travel time to the monitored location in the distribution system.
The complete analysis and results are summarized in the Water Security Initiative Distribution System
Studies and Modeling: Tracer Study Report (USEPA, 2007d).
The results of the field study and subsequent data analysis uncovered two omissions in the model that had
important impacts on the accuracy of model predictions. These items were incorporated into the model
and this updated version was used in the second phase of network design. Additionally, the tracer study
enhanced GCWW's knowledge of water quality and hydraulics in their distribution system. Finally, the
tracer study results will also be used in a retrospective analysis of the water quality monitoring network
design during evaluation of the pilot.
September 2008 25
-------
Cincinnati Pilot Post-Implementation System Status
3.2.2.2 Overview of the TEVA Sensor Placement Optimization Tool
The TEVA Sensor Placement Optimization Tool was developed by EPA's National Homeland Security
Research Center (NHSRC), Sandia National Laboratories, University of Cincinnati, and Argonne
National Laboratory to address the challenge of optimally placing a limited number of sensors throughout
a distribution system in order to protect public health (USEPA, 2007e). The TEVA Sensor Placement
Optimization Tool uses a utility's distribution system model to simulate the transport of contaminants in
the distribution system from the point of injection to consumers. Consequences of contamination are
estimated by predicting exposure and the resulting public health impacts using contaminant-specific dose-
response curves. In order to develop a robust design, TEVA uses an ensemble of thousands of potential
contamination scenarios in which every distribution system node is considered a potential injection
location (unless known physical constraints preclude injection at a specific node). The TEVA Sensor
Placement Optimization Tool then uses the consequence assessment from this ensemble to place a pre-
defined number of sensors at locations that maximize public health protection across all modeled
scenarios.
Without constraints on monitoring station locations, the TEVA Sensor Placement Optimization Tool will
select the optimal locations, even if those sites are impractical for installation. Therefore, the design was
constrained to a pre-selected set of feasible installation sites and a maximum number of monitoring
stations, as further described in the following section. The impact of these constraints on monitoring
network performance was evaluated by comparison with the unconstrained design (in which a monitoring
station could be installed at any node). Comparison between the two designs showed that the constrained
design reduced the potential reduction in public health impacts from 49 to 44 percent.
3.2.2.3 Overview of the Monitoring Network Design Process
Design of the online water quality monitoring network followed a three-step process. The first step was
to determine the number of monitoring stations that could be deployed within the allocated budget and
identify feasible installation locations. The second step was to run the TEVA Sensor Placement
Optimization Tool with the constraints identified under the first step to determine the optimal installation
sites that maximize public health protection. The final step was to validate the site conditions and local
hydraulics at each potential installation site identified through the second step. This three step process
was applied in an iterative fashion until a complete, validated set of installation sites was developed.
Additional details of the monitoring network design process are discussed in the Preliminary Sensor
Network Design for Greater Cincinnati Water Works (USEPA, 2007f).
The monitoring network design for both Phase 1 and 2 followed the three-step process described above.
During Phase 1, three (3) Type-A and five (5) Type-B prototype monitoring stations were installed and
operated for approximately three months to collect data on the various equipment used in each prototype.
As discussed in Section 3.2.1, this information led to the design of the Type-C monitoring stations, nine
(9) of which were installed during Phase 2. These two phases of monitoring network design are described
in more detail below.
During the first phase of sensor network design, GCWW identified a set of desirable installation
locations. In addition to all GCWW-owned facilities, other municipal buildings to which GCWW
personnel could gain 24/7/365 access were considered. This initial pool of potential locations included all
police stations and fire stations in the county and many governmental and academic institutions. EPA
personnel then translated the physical addresses of these potential locations to a specific node in the
distribution system model. At this phase of the design, the number of monitoring stations that could be
deployed was unknown, but based on experience with other monitoring network designs, it was decided
that an upper bound of thirty (30) stations was reasonable for a distribution system of this size.
The TEVA Sensor Placement Optimization Tool was then used with the existing GCWW distribution
system model to develop the first network design. Because all thirty (30) stations would not be installed
in the first phase, the thirty locations in the design were ranked in order of importance. The intent of this
September 2008 26
-------
Cincinnati Pilot Post-Implementation System Status
approach was to determine an overall network design consisting of up to thirty (30) monitoring locations,
and install them in phases. Note that ultimately only seventeen (17) monitoring stations were installed,
but the initial network design was based on the upper bound of thirty.
Following the development of detailed cost estimates for the Type-A and B prototypes, it was determined
that eight (8) monitoring stations could be fabricated and installed in Phase 1. Furthermore, it was
decided that two (2) of the water quality monitoring stations would be installed at the two (2) water
treatment plants to monitor the water quality entering the distribution system, which provides a
benchmark for distribution system water quality. The remaining six (6) locations were selected from the
prioritized list of thirty potential locations identified by the TEVA Sensor Placement Optimization Tool.
In addition to the rank of each location, consideration was given to the expected variability of water
quality at the monitored location. This would provide an opportunity to evaluate how event detection
systems, as discussed in Section 3.2.4, would respond to different baseline water quality conditions.
Next, the hydraulic connectivity for each of the proposed installation locations was verified. Using GIS
records available from GCWW, or plans provided by the facility owner, the physical location of the
proposed site was compared to the model representation to ensure that the facility was actually being
supplied water from the distribution system model node.
Finally, a site visit was conducted to locate the exact installation location within the facility, estimate the
hydraulic residence time in the pipes from the distribution main to the monitoring equipment inside the
building, and address any outstanding concerns with that specific location. Installation sites were also
verified to confirm accessibility, physical security, available sample water and drainage, a reliable power
supply, and data communications.
If at any point in this process it was discovered that a site being investigated was unsuitable, it was
discarded and another site from the ranked list was evaluated. The result of Phase 1 deployment was a
network of six (6) stations installed across the distribution system, in addition to the two (2) at the
GCWW treatment plants, to provide broad protection as well as monitoring locations with different water
quality variability.
During Phase 2 of water quality monitoring network design, the station location selection process
followed a similar path, with a few modifications. To begin with, the pool of potential installation sites
was further verified and refined from the pool used in the first phase. However, the facility types
considered were still constrained to GCWW-owned facilities, police stations and fire stations in the
county, city facilities, post offices, and many other governmental and academic institutions. This resulted
in a pool of 193 potential installation locations.
It was also possible to define the precise number of monitoring stations to be installed under the pilot -
seventeen (17). Furthermore, it was decided that the eight (8) monitoring stations installed under the first
phase would remain at the same locations. Using these additional constraints, the TEVA Sensor
Placement Optimization Tool was used with the updated GCWW distribution system model to identify
potential installation locations for the nine (9) Type-C monitoring stations.
The suitability of potential installation was then verified through site visits, during which the hydraulic
connectivity and physical constraints on installation were evaluated. If this review showed the potential
installation site to be acceptable, it was included in the final monitoring network design. Any site that
was found to be unsuitable was removed from further consideration, and the TEVA Sensor Placement
Optimization Tool was re-run to determine optimal locations under the new set of constraints. For
example, if the first four (4) locations were found acceptable but the remaining five (5) deemed
unsuitable, the TEVA Sensor Placement Optimization Tool was re-run with the eight (8) original and four
(4) new locations fixed so that the remaining five (5) locations could be optimized under these refined
constraints. This process was repeated until nine (9) acceptable locations were identified for the Type-C
September 2008 27
-------
Cincinnati Pilot Post-Implementation System Status
monitoring stations. The end result was a water quality monitoring network that provides improved
contaminant coverage, spatial coverage, and timeliness of detection.
3.2.3 Data Management and Communications
While GCWW's existing data management and communication system was able to support the existing
water quality monitoring network, it was determined that a parallel system would be needed to support
the water quality and enhanced security monitoring components of the contamination warning system
pilot. As discussed in Section 3.1.3, this decision was driven by two considerations: 1) GCWW was in
the process of updating its SCADA at the same time the water quality monitoring network was being
deployed; and 2) the existing communication system was deemed inadequate to support communication
of video data produced by the enhanced security monitoring component. Improvements to the data
management and communications systems are discussed in the following subsections.
3.2.3.1 Remote Data Collection and Communication
Communication of data from sixteen (16) of the seventeen (17) water quality monitoring stations to the
parallel WS SCADA system was established using digital cellular telephone links to the public telephone
system. (As discussed in Section 3.2.2, one (1) monitoring station is located at the Richard Miller
Treatment Plant and utilizes fiber optic cable for communications). This design uses digital telephone
infrastructure, similar to that used by common cellular telephones, but optimized for use with data
applications. It represents a substantial upgrade in both technology and reliability compared to the pre-
existing analog telephone network.
The digital cellular network is composed of a Cincinnati Bell Telephone LAN Advantage wireline link
and integrated service firewall/router connecting the WS SCADA to the digital cellular network. Figure
3-4 shows the communication network with nodes for both water quality and enhanced security
monitoring locations.
Each water quality monitoring station is equipped with a radio panel that includes an Ethernet switch with
built-in firewall and security features to protect data stored on the PLC and during transmission to the WS
SCADA system. Each radio panel is also equipped with an UPS which can power the Ethernet switch,
radio, and other communication equipment for approximately 24 hours. Water quality parameter data
collected at each water quality monitoring station are automatically communicated in two-minute
intervals to the WS SCADA system.
While data are communicated to the WS SCADA system and stored on a local server, all measured data
are also stored every two minutes in flash memory on the Bristol Babcock ControlWave Micro PLC.
More than thirty days of data are stored in the flash memory on a chronological basis, and the oldest
records are automatically overwritten by the newest records. This data file is available for download in
the event that communication to the WS SCADA system is lost.
The enhanced security monitoring devices communicate with the WS SCADA network using the existing
GCWW communications system. These new devices provide access alarming for reservoir and tank
assets. The existing GCWW SCADA network and the parallel WS SCADA network are separated by a
network security device, a firewall, to eliminate unwanted traffic.
September 2008 28
-------
Cincinnati Pilot Post-Implementation System Status
DIGITAL CELLUAR
NETWORK
CINCINNATI BELL PRIVATE
NETWORK
Water Quality Monitoring
Stations
Typical for:
- GCWW Facilities
- Non-Utility Facilities
* Note: Security cameras at
only 3 locations.
Network
Security
Device
ENHANCED SECURITY MONITORING
DEVICES USING EXISTING GCWW
SCADA PROTECTED NETWORK
Enhanced
Security
Monitoring
Typical for:
- Reservoirs
- Tanks
Existing GCWW
Communications
System
LEGEND
| \\ Video camera(s)
gj$ Cellular based radio
-------
Cincinnati Pilot Post-Implementation System Status
GCWW SCADA
DMZ
GCWW & CITY NETWORK
PRE-EXISTING GCWW SCADA PROTECTED
NETWORK
Remote WS
SCADA
Workstations (2)
GCWW
SCADA
Workstations (6)
GCWW
SCADA
Servers (2)
GCWW
Reporting
Historians (3)
WS Database
Server
WS Application
Server
Network
Security
Device
Network
Security
Device
WS EDDIES
Workstation
WATER
SECURITY
PARALLEL
SCADA DMZ
WATER SECURITY PARALLEL
SCADA PROTECTED NETWORK
Network
Security
Device
Network
Security
Device
WS SCADA
Servers (2)
r a -, au
WS SCADA
Workstation
DAT
WS Historian &
Terminal Services
Server
Figure 3-5. WS Water Quality Monitoring Data Management System Architecture
The pre-existing GCWW SCADA Protected Network consists of two (2) SCADA communication
servers, six (6) SCADA workstations, and a communications system as depicted in Figure 3-4. The
communications system communicates with all the GCWW assets that are off-site such as pump stations,
reservoirs, and elevated tanks. Off-site SCADA data are constantly being polled by the SCADA servers
and being displayed on the SCADA workstations. Operators use this information to monitor the entire
GCWW water system and to make control changes.
The GCWW and City of Cincinnati Network consists of hundreds of workstations and servers used to
provide business services to the city. These include email, file storage, and database engines. The WS
project added two servers and one workstation to the network. The WS Database Server stores data from
many components of the contamination warning system pilot, including SCADA data. The WS
Application Server provides program services for some components, such as public health surveillance.
The servers were installed on the business network to allow other systems on the City network easy
access to data. A workstation located in the security guard shack and another in a GCWW water quality
laboratory provides remote, read-only access to the WS Protected SCADA Network. These workstations
can monitor data from individual water quality monitoring stations, analysis results from the event
detection system, and enhanced security cameras and access controls.
The GCWW SCADA DMZ houses servers which archive GCWW SCADA data and summarize it for
reporting purposes. The DMZ provides access to this data via public networks such as the GCWW and
City Business Network. One of the reporting servers is used to temporarily hold operational data from
September 2008
30
-------
Cincinnati Pilot Post-Implementation System Status
the GCWW SCADA private network. It is later picked up by the WS Application Server and written to
the WS Database Server for long-term storage.
The EDDIES network consists of one (1) firewall and one (1) workstation. The EDDIES workstation
runs the EDDIES application and the event detection software. It receives SCADA data from both the
WS and GCWW SCADA Protected Networks. A firewall is used to secure each SCADA Protected
Network and ensure that no communication occurs directly between the two SCADA systems.
The water quality monitoring stations are monitored by the WS SCADA system. This system consists of
two SCADA servers and one SCADA workstation. The SCADA servers communicate with the PLCs for
each of the water quality monitoring stations, polling each for local data such as pH, TOC, and chlorine
values. One of the WS SCADA servers also provides information to the Terminal Services server in the
WS SCADA DMZ.
The WS SCADA servers host Human Machine Interface (HMI) software to monitor and control the water
quality monitoring stations and event detection system via the SCADA workstations. The HMI software
is written by Citect and provides the following functionality to users: monitoring of SCADA data, control
of field devices, and alarming of values that are out of a specific range. The WS System HMI application
provides user interfaces to view to following data representations of the system: a system map detailing
the location and status of each monitoring system, real-time values and alarm status from each monitoring
system, and event detection system results. The WS SCADA workstation also provides operators with
the ability to initiate remote sample collection at any water quality monitoring location. For more
information about the WS SCADA HMI application, reference the WS SCADA HMI Users' Guide
(USEPA, 2007g).
The WS SCADA DMZ includes one (1) SCADA Historian and Terminal Services Server. This server
provides the following services: file storage for water quality and enhanced security data; a tape backup
device to archive data; and Terminal Services for providing SCADA information to remote users as
described above. The WS SCADA DMZ allows for remote SCADA workstations, located on the
GCWW and City Network, to view the WS SCADA system without compromising system security.
3.2.3.3 Data Flow
Data flow throughout the data monitoring and event detection system uses two standards, Object Linking
and Embedding for Process Control and text files in comma separated value (CSV) format. As noted in
Section 3.2.3.2 there are six (6) networks making up the data management system supporting the water
quality monitoring component. In order to comply with industry and City of Cincinnati security
standards, numerous network security devices were deployed. The City required the use of static port
addresses in firewalls, which eliminated the possible use of Object Linking and Embedding for Process
Control communications between networks. This required the use of CSV files to transfer data from one
network to another. Figure 3-6 illustrates the path of data flow between networks.
September 2008 31
-------
Cincinnati Pilot Post-Implementation System Status
PRE-EXISTING GCWW SCADA
PROTECTED NETWORK £CWW SCA DA
F9h
" ' GCWW JJS&'y
c Communication ^^^^x'
X Servers (2}^Jf ~^
^^>
' * — ^ EfJ
^i
EDDIES DMZ
] Network
ws ^ I Secur'lv
EDDIES ^^^A1, L\J D^ice
Workstation
i
WATER SECURITY
PARALLEL SCADA
PROTECTED NETWORK ws
*~^ Communications
fj^/l Servers (2)
L ^^^^ \"\ 1 DAS S DA6
4 "\X
WQ Monitoring f • ^^
Station PLCs / ' ^ \ fr\
" \
WS SCADA
Workstation
GCWW & CITY NETWORK
Database r^|x &Sx5
^ ' ' Remote WS
/~> SCADA
s^S^ Workstations (2)
"^l 1
LEGEND
=£
-••*
6: x 1 Network
1 L Securily
^X^J Device
i
i
i
• Network
SJ. Security
^ Device
PLC Data 1 * Firewall
2min.Da.aFi,. ( Etheme,
EDS Atarm File f^^ Switch
EDS Alarm \IO ]]|||||nE| Programmable
Data Archiving III ''1 logic controllor
GCWW
SCADA
DMZ
GCWW
Reporting
Historians (3)
WATER
SECURITY
PARALLEL
SCADA DMZ
WS Historian
& Terminal
Services
Server
Figure 3-6. WS Water Quality Monitoring Data Flow Diagram
Each of the protected SCADA networks have private communication systems to talk to their SCADA
devices, such as PLCs. Because these networks are private, they employ Object Linking and Embedding
for Process Control as the communication standard. This line of communication is illustrated in Figure 3-
6 by the bold blue arrows. Communication occurs in anywhere from millisecond to minute intervals
depending on the device.
Each of the firewalls in Figure 3-6 require static port address and thus cannot use Object Linking and
Embedding for Process Control communications. This requires SCADA data to be transferred using CSV
files. CSV file movement is represented by gray, purple, orange and black lines.
On a two minute interval data are sent from both the GCWW and WS SCADA Communication Servers to
the EDDIES workstation for processing by the event detection system. Once analyzed, EDDIES returns
the status of each monitoring location to the WS Communication Server for display on the WS SCADA
workstations, including the remote workstations in the GCWW and City Network.
Nightly, SCADA data, represented by gray lines, are moved from each of the protected SCADA networks
to their respective DMZs. Then the WS Database Server, located on the GCWW and City Network, polls
each DMZ for their SCADA data and archives it for long-term storage.
3.2.4 Water Quality Event Detection
As discussed in Section 3.1.4, GCWW did not have a pre-existing water quality event detection system
that met the design objectives of a contamination warning system. For the pilot, three (3) event detection
systems were deployed to allow for comparison among these new and novel technologies. Two (2) of
these event detection systems were integrated with the WS and GCWW SCADA systems through a
September 2008
32
-------
Cincinnati Pilot Post-Implementation System Status
custom interface - EDDIES. An off-line study was performed to test the ability of these two event
detection systems to detect simulated contamination incidents without producing an unmanageable
number of false alarms. Finally, the two (2) systems were deployed for real-time operation at the utility
using the EDDIES interface, and a concept of operations was developed to guide routine operations and
the investigation of water quality alarms.
3.2.4.1 Event Detection Systems
In the context of the water quality monitoring component of a contamination warning system, an event
detection system is defined as one or more algorithms that continually analyze water quality data, along
with metadata such as sensor alarms and data quality flags, to monitor for changes in water quality
triggered by abnormal conditions. Three (3) event detection systems were deployed as part of the
GCWW contamination warning system pilot: the Event Monitor developed by Hach; Canary developed
by Sandia National Laboratories; and H2O Sentinel™ developed by Frontier Technology Incorporated.
The Hach Event Monitor is an ancillary piece of hardware that is compatible only with Hach sensors. As
Hach's Event Monitor must be deployed at each monitoring location and ties directly into the Hach
Distribution Monitoring Panel, it was not practical to include the Event Monitor in the offline evaluation
study discussed in Section 3.2.4.3. For this reason, the focus was on the two centralized event detection
systems, Canary and H2O Sentinel™.
Canary: The Canary event detection system, developed by the Sandia National Laboratories in
cooperation with USEPA - National Homeland Security Research Center, uses three different algorithms
to detect possible contamination events based on water quality values (Hart, 2007). The algorithms were
developed and tested using empirical data relating water quality response to specific contaminants, as
well as historic baseline data from large water utilities.
The three algorithms used in Canary are described below:
• Time Series Increment: This algorithm looks at the differences between successive values of each
water quality parameter separately. This difference between the current parameter value and the
previous is represented as n standard deviations, and an alarm is raised if n surpasses the set
threshold. Differences across all water quality sensors can be fused to create a combined difference
value.
• Linear Filter: For the linear filter algorithm, the expected value of a parameter for a given time-
step is predicted based on a linear combination of its previous values in the time series. Similar to
the time series increment algorithm, differences between the current water quality parameter value
and the predicted value are recorded and compared to a threshold value. Also, differences across
all sensors can be fused to create a combined difference value (Klise, 2006).
• Multivariate Distance: All parameters are considered together in this algorithm. The Euclidean
distance in multivariate space between the current measurement vector and all previous vectors
held in the moving time history is calculated, the minimum of all these distances is determined, and
if this distance is above the set threshold, an alarm is raised.
A recent addition to the Canary tool is the Binomial Event Discriminator that takes the multiple,
sequential outputs from any one of the event detection algorithms and determines whether or not an event
is beginning based on those multiple results (McKenna et al., 2007).
H2O Sentinel™: The H2O Sentinel™ Event Detection System, developed by Frontier Technology
Incorporated, is specifically designed to detect abnormalities in drinking water quality data. The software
is an extension of NormNet™, the company's patented event detection technology. This technology uses
a statistical / signal processing / pattern recognition approach (Frontier Technology Incorporated, 2006).
The training procedure for H2O Sentinel™ is fully automated. Training data that represents normal
conditions is provided to the software, and multiple statistical models are built that capture a wide variety
of normal operating conditions (Frontier Technology Incorporated, 2006). Each model combines current
September 2008 33
-------
Cincinnati Pilot Post-Implementation System Status
and past values of the input parameters to uniquely characterize sensor performance over a subset of the
training data. These models are stored in a database and are later used to assess new water quality
conditions as anomalous or normal.
Once the models are built, new data are automatically analyzed by selecting the best statistical model
from the existing database using a rapid nearest neighbor search. Next, H2O Sentinel™ generates
predicted values for each parameter that are compared with measured values. At each time-step, the tool
produces a probability of anomalous water quality conditions.
3.2.4.2 Event Detection, Deployment, Integration, and Evaluation System
Canary and H2O Sentinel™ were designed as software applications that could be deployed at a central
location and monitor data from water quality monitoring sensors produced by any manufacturer. To
facilitate the evaluation of these two tools, as well and their integration with the SCADA systems
discussed in Section 3.2.3, an interface application was developed: Event Detection, Deployment,
Integration, and Evaluation System (EDDIES). Two versions of EDDIES were developed to facilitate
and manage the evaluation and deployment of the two event detection systems.
• EDDIES 2.0: The off-line evaluation, discussed in Section 3.2.4.3, was conducted using
EDDIES 2.0. Through EDDIES 2.0, test datasets were stored, managed, and provided to the EDS
tools in simulated real-time. EDDIES 2.0 also managed the output from each event detection
system, which was used in the evaluation of the event detection systems (USEPA, 2007h).
• EDDIES 3.0: Canary and H2O Sentinel™ were deployed at GCWW using EDDIES 3.0. This
custom application manages the event detection systems, processes and stores data from the WS
SCADA system in an Oracle database, makes it available to the event detection system in real-
time, receives and stores event detection system output, and transfers the output back to the WS
SCADA system in order to display alarm status. The supporting system architecture is shown in
Figure 3-5. Similar to EDDIES 2.0, EDDIES 3.0 eliminates the need for event detection systems
to read and write data from a variety of sources with disparate formats. EDDIES 3.0 is currently
operational at the Cincinnati pilot, processing data and providing output to the WS SCADA
system in near real-time (USEPA, 2007i).
3.2.4.3 Event Detection System Performance Evaluation
Two preliminary evaluations have been performed on Canary and H2O Sentinel™ using data from the
Cincinnati pilot: an off-line evaluation was completed to support the selection of tool(s) for deployment at
the pilot, and an on-line evaluation was performed to quantify the performance of the tools as they operate
in near real-time at the Cincinnati pilot.
In spring of 2007, an off-line evaluation of Canary and H2O Sentinel™ was conducted using EDDIES
2.0. Data from GCWW was analyzed by each tool, both in its original state and with simulated
contamination events, to test the tools' detection capability. Approximately thirteen (13) weeks of 2-
minute data from early 2007 for each of the eight (8) water quality monitoring stations installed in Phase
1 was used for the evaluation. Superposition of simulated events on the original baseline data allows for
the evaluation of the tools' detection capability in addition to the false alarm rate. Contamination events
were simulated at each monitoring location, and were defined by the following parameters.
• Contaminant concentration pattern: Two (2) to three (3) patterns showing concentration as a
function of time were chosen for each monitoring location. The patterns were obtained from a
large-scale tracer study conducted at the pilot utility in fall of 2006.
• Peak extension: Two (2) peak extensions were used for each pattern. The patterns are made
longer by extending the peak or plateau of the event.
September 2008 34
-------
Cincinnati Pilot Post-Implementation System Status
• Peak concentration: Two (2) to three (3) peak concentrations were identified for each
contaminant. The LD50 (the dose that would be lethal for 50 percent of the exposed population)
of toxic chemicals (and infections dose for pathogens) was used as the benchmark for selection of
contaminant peak concentrations in order to provide some equivalence among simulations.
• Contaminant: Eight (8) contaminants were selected for this evaluation based on EPA's analysis
of contamination threats. Five (5) of the contaminants were identified by the EPA as
contaminants of concern because of their ability to cause harm if injected into a water system,
ability to be dispersed in water, and availability. In addition, this set of contaminants was chosen
because they capture various combinations of water quality parameter responses. Laboratory data
from pipe-loop studies and bench-top experiments was used to develop models for the change in
water quality parameter values as a function of contaminant concentration.
• Start times: Four (4) event start times were used in the design of the simulations. Events may
manifest quite differently depending on the time of day and background water quality variability
at that time, and the four dates and start times selected each exhibit different baseline conditions.
The range of values for the above parameters was selected to create realistic events and as well as a wide
variety of water quality changes. Every combination of these six (6) variables was simulated, producing
an experimental matrix of 3,872 test datasets. More details on the evaluation can be found in the event
detection system evaluation framework (USEPA, 2006b) and plan (USEPA, 2007J).
Once the event detection tools were executed on the test data, a variety of analyses were performed on the
Event Detection System outputs. Performance measures calculated included false alarm rates, median
time to detect, and sensitivity (ratio of events correctly identified). The results of this evaluation were
very promising. At some monitoring stations, over 70 percent of the simulated events were detected at
false positive rates (specificity of 0.004 translates to less than 3 false positives a day for this utility). Note
that while this rate may seem high, these false positives are often grouped together into one false alarm,
which would require only one investigation. In addition to the good detection performance at low false
alarm rates, those detections occur in less than an hour, on average. More comprehensive results can be
found in the Evaluation of Tools to Detect Distribution System Water Quality Anomalies (Umberg and
Allgeier, 2007).
The on-line evaluation provided a summary of event detection tool performance at GCWW during the
month of October 2007 using EDDIES 3.0 (USEPA, 2007k). As this was an on-line evaluation, only
real-time, non-event data was used; thus, only false alarm rates could be evaluated, and no information
was gained on how effectively the tools would detect contamination events. During this period, EDDIES
3.0 and Canary ran on all seventeen (17) stations with no errors or unexpected down-time. Also, both
Canary and H2O Sentinel™ produced considerably fewer false alarms than was observed in the offline
evaluation, with several stations producing fewer than eight (8) alarms during the entire month of October
2007.
3.2.4.4 Event Detection System Deployment
Deployment of the event detection system at GCWW involved training the tools, setting them up on a
dedicated workstation, establishing data flows, and establishing procedures to investigate and respond to
water quality alarms.
Approximately three (3) months of water quality data from each of the seventeen (17) monitoring stations
was used to train the event detection system tools. The event detection system developers used this data
to establish baseline water quality and determine normal variability at each location. Each tool was
optimized for each monitoring station by developing location-specific settings tuned to the water quality
variability and patterns present in the training data for that location.
September 2008 35
-------
Cincinnati Pilot Post-Implementation System Status
These "trained" tools, along with EDDIES 3.0, were installed on a dedicated workstation at GCWW. As
discussed in Section 3.2.3, this workstation exists on a DMZ that is only connected to the GCWW and
WS SCADA systems. EDDIES 3.0 facilitates the processing of data from both SCADA systems for use
by the event detection tools.
For each timestep, the running event detection tool processes the data provided by EDDIES 3.0 and
outputs two values for each location after analyzing the water quality data for that timestep. First, it
outputs the probability that conditions are anomalous as a value between 0 and 1 (with higher numbers
indicating greater cause for alarm). A definitive Alarm I No Alarm indicator is also produced. In the case
of an alarm, additional information is given as to which water quality parameter(s) triggered the alarm.
EDDIES 3.0 then sends the Event Detection System results back through the WS SCADA system to be
displayed on the SCADA HMI. In addition to visual alarms on the screen, the system issues an audible
alarm to alert operators in case they are away from the graphical user interface.
When an Event Detection System alarm indicates abnormal conditions for a particular monitoring station,
GCWW implements the concept of operations that guides the initial investigation into potential causes of
the alarm (USEPA, 2007b). The concept of operations includes steps that are intended to rule out benign
causes. For example, system operations and ongoing distribution system work activities would be
reviewed as potential causes. In some cases, the water quality monitoring station that produced the alarm
may be investigated to determine if a sensor malfunction was the cause. If the initial investigation does
not reveal an obvious cause, contamination is considered possible and the investigation is turned over to
the Water Utility Emergency Response Manager (WUERM), who will take additional steps to determine
whether or not contamination is credible.
3.2.5 Summary of Post-Implementation Status
The water quality monitoring system at GCWW has been installed, is fully operational, and is currently
generating data needed to establish baseline performance. The water quality monitoring network includes
a total of seventeen (17) monitoring stations placed at locations optimized to detect contamination events
in a manner that will limit public health consequences. Three (3) monitoring station prototypes have been
deployed to allow for comparison of different vendor technologies. A digital cellular network was
designed and installed to transmit data back to a SCADA system developed specifically for the needs of
this project. Three (3) event detection systems were deployed to provide for continuous analysis of the
data produced by the monitoring network and provide early warning of anomalies that may be indicative
of contamination. Table 3-4 provides a summary of the post-implementation status of the water quality
monitoring component of the contamination warning system.
Table 3-4. Water Quality Monitoring Post-Implementation Status
Design Element
1. Water Quality
Monitoring Equipment
2. Water Quality
Monitoring Network
3. Data Management
and Communications
Description of Installed Component
• Each monitoring station includes sensors for chlorine residual, TOC,
conductivity, pH, and temperature (fourteen (14) locations also monitor ORP).
Collectively, these parameters provide broad coverage of potential
contaminants.
• Each monitoring station is equipped with two sampling devices that allow for
remote sample collection in the event of suspected contamination.
• A water quality monitoring network consisting of seventeen (17) water quality
monitoring stations has been deployed and is operational.
• Fifteen (15) monitoring stations have been strategically placed throughout the
distribution system to optimize public health protection.
• A monitoring station has been place at the finished water for each of the two
(2) treatment plants, providing reference water quality values for the rest of
the stations in the network.
• Data collection and transmittal to central facility is operating using a secure,
digital cellular system via public telephone utility.
• Water quality is continuously displayed to system operators, including real-
time parameter values, instrument alarms, and event detection alarms.
September 2008 36
-------
Cincinnati Pilot Post-Implementation System Status
Design Element
Description of Installed Component
Data are formatted and transmitted to a dedicated workstation for use by the
even detection system.
Data are automatically archived for historical data analysis and easy recall.
4. Water Quality Event
Detection
Two (2) event detection tools, Canary and H2O Sentinel™, are installed and
operational on all seventeen (17) water quality monitoring stations.
A comprehensive performance evaluation of the two (2) event detection tools
has been completed.
EDDIES manages real-time data transfer between the SCADA network and
the event detection system providing continuous monitoring for anomalies.
A concept of operations has been developed and guides the day-to-day
operation of the component at GCWW.
Figure 3-7 provides a summary of the level of effort associated with design and implementation of the
online water quality monitoring component for the Cincinnati pilot. Implementation of the online water
quality monitoring component, as summarized in Table 3-4, relied on support from EPA, GCWW, and
local partners. Much of the effort associated with the design element "water quality monitoring
equipment" involved the design, implementation, and shakedown of the monitoring stations. For design
of the water quality monitoring network, most of the level of effort was expended in the design,
implementation, and analysis of the tracer study. The local partner effort associated with data
management and communications resulted from reviews and approvals necessary for deploying new
systems on Cincinnati's Metropolitan Area Network. A significant portion of level of effort for the water
quality event detection design element was associated with integrating two software tools - H2O
Sentinel™ and Canary - with the GCWW and parallel SCADA systems, and training both tools on the
water quality base-state at each of the seventeen (17) monitoring stations.
September 2008
37
-------
Cincinnati Pilot Post-Implementation System Status
.0
3.0 •
2.5 '
(/) 2.0
(0
0) ...
fc 1 q -
5-
0)
1.0 •
0.5 '
0.2
DLo
• GC
•EP
:al Partners LOE
;WWLOE
ALOE ~
0.1
h
0.1
—
.01 i i i i
Water Quality Monitoring Water Quality Monitoring Data Management and Water Quality Event
Equipment Network Communications Detection
Figure 3-7. Level of Effort for Design and Implementation of the Online Water Quality Monitoring
Component (December 2005 - December 2007)
Figure 3-8 presents a summary of the extramural costs associated with design and implementation of the
online water quality monitoring component for the Cincinnati pilot. The most significant extramural
labor costs include contractor activities associated with design, procurement, and installation of the water
quality monitoring stations. Purchased services include costs associated with routine maintenance and
repair of the online water quality monitoring stations, as well as the licensing and technical support
contract for the H2O Sentinel™ software. Another event detection system, Canary, was developed by
EPA's NHSRC, and is being used as part of the Cincinnati pilot. As this software is available in the
public domain, no procurement costs were incurred. Costs associated with contractor travel were not
included in this calculation. The majority of the equipment and extramural LOE costs for the water
quality monitoring network design element were associated with the large-scale tracer study. Forty (40)
portable conductivity and ten (10) chlorine monitors had to be procured and fabricated to support this
study, in addition to the equipment and supplies necessary to inject the tracer. The extramural costs
attributable to data management and communications were expended primarily on the parallel WS
SCADA system deployed to support online water quality and enhanced security monitoring.
September 2008
38
-------
Cincinnati Pilot Post-Implementation System Status
$1,800
$1,600
$1,400
(A
C
ra
(A
3
$1,200
$1,000
•- $800
•4-1
(A
O
° $600
$400
$200
• Equipment, Consumables,
and Purchased Services
n Extramural LOE
$51
Water Quality
Monitoring Equipment
Water Quality
Monitoring Network
Data Management and Water Quality Event
Communications Detection
Figure 3-8. Extramural Costs Associated with Design and Implementation of the Online Water
Quality Monitoring Component (December 2005 - December 2007)
The cost breakdown in Table 3-5 is intended to provide additional details regarding the total
implementation costs for the water quality monitoring equipment deployed at the Cincinnati pilot.
Specifically, Table 3-5 focuses on the costs for fabrication and installation of the three water quality
monitoring station prototypes described in Section 3.2.1, including the local electrical, data storage, and
communication system installed with each station.
September 2008
39
-------
Cincinnati Pilot Post-Implementation System Status
Table 3-5. Comparing Implementation Costs for Each Type of Water Quality Monitoring Station
(reference 3.2.1 and Table 3-3 for Water Quality Monitoring Station Specifications)
Activity
Administrative
Support
Procurement:
Primary Components
Procurement: Other
Design
Fabrication
Fabrication:
Technical Support
Installation
Installation:
Technical Support
Startup
Communications
System
Description
Procurement support, equipment tracking, and
contractual services
TYPE A: Hach WDMP ($12,400), Hach Astro TOC
($18,450), Hach Event Monitor ($8,300) TYPE B:
YSI 6500 ($10,700), USF Depolox 3+ ($3,700), GE
TOC 900 ($24,950) TYPE C: Hach WDMPsc
($14,950), GE TOC 900 ($24,950)
PLC, UPS, radio panel, sampling system, misc.
electrical and plumbing connections
Selection of primary/secondary components, and
arrangement of components onto a portable frame
Placement of components on frame, materials for
frame, assembly of radio panel, routing of interior
plumbing and electrical wiring, and WQM delivery
Development of fabrication specifications, and
technical inspection/oversight of fabrication process
WQM placement and connections to electricity,
plumbing, and the communications network
Development of installation specifications, and
technical inspection/oversight of installation process
Initial calibration/troubleshooting required to
produce accurate WQ data, and cost of
reagents/supplies
Field study, design, technical support, startup, and
troubleshooting required to establish
communication link
TOTAL COST PER WQM STATION
Type A
Costs
$3,000
$39,150
$10,500
$3,500
$17,000
$4,000
$5,000
$2,000
$4,000
$2,500
$90,650
Type B
Costs
$3,000
$39,350
$10,500
$3,500
$17,000
$4,000
$5,000
$2,000
$4,000
$2,500
$90,850
TypeC
Costs
$3,000
$39,900
$10,500
$3,000
$11,000
$3,500
$5,000
$2,000
$4,000
$2,500
$84,400
Cost savings in the areas of "Fabrication" and "Fabrication Technical Support" for the Type C water
quality monitoring station can be attributed to the use of a local, lower-cost fabrication company in Phase
2. Type C stations also experienced cost savings in the area of "Design," primarily due to templates and
processes that were developed during Phase 1.
These cost estimates are illustrative and not intended to inform future decisions regarding equipment
selection for similar projects, which should be based on analysis of comprehensive life-cycle costs among
the various options. The costs presented in this document do not include expenditures associated with
routine operation and maintenance activities or depreciation of equipment.
September 2008
40
-------
Cincinnati Pilot Post-Implementation System Status
Section 4.0: Sampling and Analysis
Sampling and analysis plays a critical role in the contamination warning system due to the potential to
detect contaminants in drinking water samples collected throughout GCWW's distribution system.
During a suspected contamination event, water samples can be collected and analyzed with the goal of
confirming or ruling out actual contamination. Though results from sample analyses may not be
generated until several hours or longer after sample collection, results can supplement a triggered
investigation of alarms from other components of the system.
Prior to implementation of the Cincinnati pilot, EPA compiled a list of high priority contaminants of
interest to water security (USEPA, 2005b). From the high priority contaminant list, EPA identified a
subset of chemical, radiochemical and microbiological contaminants to monitor during the Cincinnati
pilot at GCWW. To supplement the in-house analytical capability of the GCWW utility laboratory, EPA
identified a network of partner laboratories that would provide additional analytical support.
During implementation of the sampling and analysis component at GCWW, the initial phases of sample
collection and analysis, referred to as 'baseline monitoring', were designed to establish a baseline of
contaminant occurrence (contaminants detected, levels detected, and frequency of detections) and method
performance in samples collected throughout the distribution system over a one year period of time.
Additionally, baseline monitoring provided the opportunity to practice, under low-stress conditions, the
methods and protocols that would be used in the high-stress scenario of a triggered sampling event in
which contamination is possible.
Triggered samples are those collected and analyzed in response to "triggers" from any other component
(e.g., public health surveillance, consumer complaint surveillance, online water quality monitoring, or
enhanced security monitoring). Initially, triggered samples are analyzed using the same methods and
protocols employed routinely during baseline monitoring. If a suspected contamination event occurs, the
credibility determination would involve comparison of triggered sample results to baseline data to assess
whether background levels have been exceeded or method performance has been compromised. This
process will assist the utility to confirm or rule out a wide array of known contaminants or refer the
sample for additional analysis. In special circumstances, it may be necessary to analyze triggered samples
using methods not previously employed during baseline monitoring.
The sampling and analysis component design objectives are shown in Table 4-1, and were derived from
the overarching performance objectives of the contamination warning system as described in
WaterSentinel System Architecture (USEPA, 2005a). GCWW's pre-existing capability with respect to
each attribute of the sampling and analysis component listed in Table 4-1 is summarized in Section 4.1
and the utility's capability after implementation of the contamination warning system is summarized in
Section 4.2.
Table 4-1. Sampling and Analysis Component Design Objectives
Attribute
1. Laboratory capability
and capacity
2. Sampling and analysis
3. Field screening and
site characterization
Design Objective
Build laboratory capability and capacity to perform screening and confirmatory
analyses for a wide range of contaminants under routine and non-routine
scenarios
Select sampling locations, frequencies, quality assurance and data quality
objectives for routine sampling and analysis to establish baseline data for
contaminant occurrence in the distribution system and to evaluate method
performance
Establish roles that will be assumed by the utility and others investigating a
potential contamination incident and identify testing equipment that will be
necessary to conduct field and safety screening
September 2008 41
-------
Cincinnati Pilot Post-Implementation System Status
4.1 Pre-lmplementation Status
This section describes the processes and tools that were used to evaluate existing sampling and analysis
capabilities that could support the Cincinnati pilot and summarizes the findings. The evaluation included
review of existing in-house sampling and analysis capabilities as well as areas where analytical support
could be provided by partner laboratories and agencies.
Implementation of the sampling and analysis component at GCWW involved an assessment of existing
capabilities and identification of potential enhancements to enable the detection of a wide range of
contaminants from EPA's priority contaminant list (USEPA, 2005b). The primary objective was to
determine if GCWW's existing system could support routine monitoring and triggered sampling and
analysis objectives.
Gap analyses were used to determine appropriate enhancements and/or modifications to support
implementation of the sampling and analysis component at GCWW and to identify key equipment or
training to increase the utility's capability and capacity. Additionally, local partner laboratories or
agencies that could provide support to sampling and/or analytical objectives were also identified. The
assessment process demonstrated that method support laboratories would be needed for biological,
radiochemical, and some chemical analyses. Also, local HazMat resources would be required during
sampling events when hazardous materials were suspected to be present.
EPA used a multifaceted evaluation and assessment process to determine the sampling and analysis
capabilities of the utility and local partner laboratories and agencies. On-site tours, interviews, cataloging
of inventories, and surveys were used to assess the pre-implementation capabilities and resources of
GCWW and other local resources. These activities included:
• Tours of GCWW, Metropolitan Sewer District, Cincinnati Health Department Laboratory, and
Ohio Department of Health facilities, and interviews with staff
• Catalog of GCWW field and laboratory equipment and respective methods currently performed
• Review of current certifications held by GCWW
• Determination of GCWW chain of custody practices and data management capabilities through
survey questions
• Determination of existing GCWW sampling routes, locations, and sampling frequencies that
could be leveraged for baseline monitoring
• Review of existing GCWW response capabilities and plans
• Review of existing relationships between GCWW and the Cincinnati Fire Department and
Greater Cincinnati HazMat Unit
• Determination of field capabilities of Cincinnati Fire Department and Greater Cincinnati HazMat
Unit
• Determination of current Cincinnati Health Department Laboratory methods performed,
equipment, and capabilities
The assessment process led to the identification of improvements or enhancements that were necessary to
provide the broad level of contaminant coverage desired for the Cincinnati pilot.
4.1.1 Laboratory Capability and Capacity
Field and laboratory-based methods to detect a subset of contaminants from the priority contaminant list
were identified and GCWW was assessed to determine if they possessed the capability (instrumentation
and staff), qualifications (training, certifications and accreditations), and systems (Quality
Assurance/Quality Control program, Standard Operating Procedures, protocols, data management) to
implement the methods. Due to the sensitive nature of revealing specific detection capabilities at the
Cincinnati pilot, Table 4-2 presents only the contaminant classes for which capabilities were identified
and implemented.
September 2008 42
-------
Cincinnati Pilot Post-Implementation System Status
Table 4-2. Cincinnati Pilot Contaminant Classes and Method Types
Contaminant Class
Free cyanide
Free chlorine
pH and conductivity
Turbidity
Organophosphate chemical warfare agents and pesticides
Radioactivity
Volatile Organic Compounds
Toxicity
Metals
Polychlorinated biphenyls
Organophosphate pesticides
Carbamate pesticides
Volatile Organic Compounds
Gross alpha
Gross beta
Gamma emitters
Select Agents and Toxins
Method Type
Field and Screening Test Methods/Kits
Laboratory-Based Chemical Methods
Laboratory-Based Radiochemical Methods
Laboratory-Based Biological Methods
Capabilities for routine monitoring and surveillance using the approaches listed in Table 4-2 were desired
for the Cincinnati pilot so that a wide range of contaminants and classes could be ruled in or out by
comparison of baseline data to triggered sample results. In the initial phases of a contamination threat
investigation, triggered sampling and analysis protocols will be almost identical to routine sampling and
analysis protocols. Unless there is specific evidence leading the investigation, the same methods used for
routine sampling and analysis will be utilized during a triggered event.
For regulated chemical or radiochemical contaminants, EPA drinking water certification standards
(USEPA, 2005c) or National Environmental Laboratory Accreditation Conference (NELAC) standards
were used for assessment. NELAC standards were used for field methods, and Clinical Laboratory
Improvement Amendments standards or Laboratory Response Network standards were used for
biological contaminants. It was necessary to establish new standards for some contaminants and methods,
as standards did not exist (e.g., ultrafiltration quality control recovery, field method quality control
practices). The document Assessment Report of Sampling and Analysis Activities at the GCWW WS-CWS
Pilot contains more detailed information regarding assessment criteria, pre-implementation status of
sampling and analysis at GCWW and partner laboratories, and how existing gaps were addressed
(USEPA, 20071).
GCWW Pre-lmplementation Sampling and Analysis Equipment and Capabilities
Prior to implementation of the Cincinnati pilot, GCWW's analytical capabilities were primarily
associated with compliance monitoring and water treatment process monitoring. In general, compliance
samples were analyzed by GCWW; however, some compliance monitoring samples (for semivolatiles
and carbamates) were contracted to Mobile Analytical Services, Inc. Radiochemical analyses were
conducted by the Ohio Department of Health in Columbus, OH.
One of the first steps conducted during the assessment process was a full inventory of sampling and
analysis equipment at GCWW. The findings are summarized in Table 4-3. GCWW had sufficient staff
September 2008
43
-------
Cincinnati Pilot Post-Implementation System Status
and standard operating procedures for most analyses conducted in-house. For some of the portable
instruments, the instruction manuals were used as the standard operating procedures, as indicated in Table
4-3.
Table 4-3. GCWW Pre-lmplementation Equi
Instrumentation
Radiochemical
Nuclear Radiation
Monitor
Inorganic
Portable
Colorimeter
Portable pH
Meter
Conductivity meter
Graphite furnace
AA
Flame AA
Cold Vapor
Mercury Analyzer
Hot Block
Nitrate
Fluoride
02
Portable
Turbidimeter
1C- Conductivity
detector
Amperometric
titrator
Spectrophotometer
Organic
Fluorometer
GC-FID
GC-MS
GC-PIDAND
ELCD
GC/MS
TOC
Model
Radalert 50
Hach DR890
Fisher Accumet
AP61
920A
Orion 150AT
Varian SpectrAA
640Z
Varian SpectrAA
640
Varian VGA 77
Environmental
Express
Accumet AR 50
920At
Orion 97-08-00
Hach 21 OOP
Metrohm819 1C
Wallace and
Tiernan
Hach DR 4000 and
5000
10-AU Turner
Designs
Varian 3400cx
Varian Saturn 2000
VARIAN 3400
Varian Saturn 2000
TeledyneTekmar
Phoenix 8000
Qty
1
4
3
2
1
1
1
1
1
2
1
1
1
1
1
2
1
1
1
2
1
1
pment and Methods
Methods Used
General scan
Chlorine Hach Method
Cyanide CHEMetrics Vacu-vial Method
PH
SM 4500 H Ammonia as Nitrogen
SM2510
SM3113-B
Pb, Cu, Al, As
SM3111-B
Zn, Na
SM3112-B
Hg
EPA 200.7
Digestion
SM 4500-NOs D
SM 4500-F C
SM 4500-O G
Turbidity
EPA 314, 300
SM 4500 Cl D Chlorine residual by
amperometric titration
NH3, PO4, Fe, NO3, UV254, SO42"
SM 10200H Chlorophyll
EPA 502.2
EPA 524.2
EPA 502.2
SM 6040D
SM5310C
SOP
Yes
Yes
No
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Manual
Manual
Yes
No
Mfrs Method
Yes
Yes
Yes
Yes
Yes
Yes
September 2008
44
-------
Cincinnati Pilot Post-Implementation System Status
Instrumentation
TOX
GC-ECD
Biological
Autoclave
Water Bath
Incubator
Refrigerator
Class II Biosafety
Cabinet
Shaking Water
Bath
Phase Contrast
Microscope
Colony Counter
Dissecting
Microscope
Particle Counter
Model
Rosemont
Dohrman
Varian 3800
Consolidated
SSR38PB
Lindberg BlueM
1110
Precision 6LM
LabLine Ambi-HiLo
Chamber
LabConCo Model
36208 / 36209
Type A/B3
Precision Scientific
Model 50
Bausch & Lomb
Quebec Colony
Counter
AO Spencer
Met One 250
Qty
1
1
1
1
3
1
1
1
1
1
1
1
Methods Used
SM
EPA 552.2
Waste, media, and glassware
sterilization
Tempering agar
SM 9223B Colisure®
Presence/Absence
Sample Storage
N/A
SM 9218 B Endospores
N/A
Colony Counting
N/A
Manufacturer's Instructions
SOP
Yes
Yes
Yes
No
No
No
Manual
No
No
No
No
Yes
While some of the equipment and methods listed are not directly relevant for detecting the specific
contaminants targeted for monitoring during the Cincinnati pilot, documenting GCWW's existing
capabilities was essential for performing gap analysis and identifying necessary enhancements.
GCWW Pre-Implementation: Laboratory Methods and Certifications/Accreditations
The majority of the drinking water methods GCWW conducted were certified by Ohio EPA. Laboratory
staff had access to fume hoods and biosafety cabinets and in some cases could analyze potentially
hazardous samples. GCWW did not have capability to analyze for all the targeted priority contaminants
(e.g., select agents, radiochemicals).
Table 4-4 contains a list of analytical methods for chemical and biological contaminants that GCWW was
certified by the Ohio EPA to perform prior to implementation of the contamination warning system.
Table 4-4. Pre-lmplementation Status of GCWW Laboratory Methods
Method Number
Title/Instrumentation
Analyte or Analytical
Parameter
Methods with Certification from Ohio EPA
502.2
VOCs by Purge and Trap Capillary GC with Photoionization and
Electrolytic Conductivity Detectors in Series
Regulated volatile organic
compounds (VOCs)
September 2008
45
-------
Cincinnati Pilot Post-Implementation System Status
Method Number
524.2
552.2
SM5310-B
SM3113-B
SM 9223-B
SM2130-B
SM 4500-H+
SM 2320
SM 2330
SM 2340-C
SM 4500 F-C
SM 4500 NO3-D
SM 4500 CI-D
SM 4500 CI-G
Title/Instrumentation
Purgeable Organic Compounds by Capillary Column GC/Mass
Spectrometry
Haloacetic Acids and Dalapon by Liquid-Liquid Extraction,
Derivatization and GC with Electron Capture Detector
Total Organic Carbon by High-Temperature Combustion
Metals by Graphite Furnace Atomic Absorption
MMO-MUG Colisure®
Nephelometric Method
pH Value
Alkalinity
Calcium Carbonate Saturation
Hardness
Ion-Selective Electrode Method
Nitrate Electrode Method
Potentiometric Method
Mercuric Thiocyanate Flow Injection Analysis
Analyte or Analytical
Parameter
Regulated VOCs
Halogenated organic
compounds
Total organic carbon and
dissolved organic carbon
Certified for lead and copper
Total coliforms and E. coli
Turbidity
PH
Alkalinity
Stability
Hardness
Fluoride
Nitrate
Chlorine
Chlorine
Methods without Certification from Ohio EPA
SM3113-B
NA
Various
Metals by Graphite Furnace Atomic Absorption
In-line GC/FID monitoring of all water coming from the Miller
Treatment Plant sampled every few minutes
Spectrophotometric methods
Monitoring for metals other
than lead and copper
VOCs
Various
Several contaminant classes [e.g., biological (select agent) and radiochemical contaminants] were not
within GCWW's capability. EPA determined that partner laboratories would be necessary to provide
analytical coverage for the targeted priority contaminants for routine and triggered sampling and analysis.
Laboratory Partners: Pre-lmplementation Capabilities and Relationships with GCWW
During the assessment process, EPA evaluated the capability of various partner laboratories to process
samples for targeted contaminant classes/methods outside of GCWW's laboratory capability. EPA also
determined whether existing relationships were established between GCWW and the partner laboratories.
The following partner laboratories were identified by EPA as entities that could provide analytical support
to the GCWW utility laboratory.
Metropolitan Sewer District of Greater Cincinnati: Prior to the Cincinnati pilot, no formal arrangement
existed between GCWW and Metropolitan Sewer District for sample analyses. Metropolitan Sewer
District possessed capabilities to analyze samples for metals by inductively coupled plasma - mass
spectrometry. Although Metropolitan Sewer District was not certified by Ohio EPA for drinking water
compliance monitoring, however, they participate in the Discharge Monitoring Report Quality Assurance
program and analyze performance evaluation samples on a quarterly basis. EPA determined that the
laboratory's qualifications were sufficient to support baseline monitoring.
September 2008
46
-------
Cincinnati Pilot Post-Implementation System Status
Ohio Department of Health Laboratory: All radiochemical analyses as required by National Primary
Drinking Water Regulations were analyzed by the drinking water certified Ohio Department of Health
Radiochemical Laboratory. The Cincinnati pilot did not require the use of different methods by Ohio
Department of Health Laboratory for radiochemical analyses, however, the number of samples submitted
for analyses increased significantly during the baseline monitoring phase.
GCWW did not have an established formal arrangement with the Ohio Department of Health Laboratory
to conduct microbiological analyses for select agents and toxins. EPA noted that Ohio Department of
Health Laboratory had the required experience to concentrate and analyze large volumes of drinking
water using the BT Agent Screening Protocol through participation in the Laboratory Response Network
Multi-Center Validation Study conducted in 2006.
Mobile Analytical Services, Incorporated: Mobile Analytical Services, Incorporated had provided
analytical support to GCWW for several years. During the Cincinnati pilot, Mobile Analytical Services,
Incorporated was contracted to analyze samples for carbamates using an EPA approved method. Mobile
Analytical Services, Incorporated was certified by Ohio EPA to analyze drinking water samples using an
EPA approved method.
Test America - Savannah, formerly Severn-Trent Laboratories: Prior to the pilot, no contract existed
between GCWW and Test America-Savannah for chemical analyses. Test America-Savannah is certified
to conduct drinking water analyses using a wide variety of EPA approved methods. Test America-
Savannah was contracted to analyze samples as necessary using these methods.
4.1.2 Sampling and Analysis
Sampling and analysis encompasses existing sampling and analysis plan(s), including sample locations,
frequencies, and procedures as well as roles and responsibilities. It also includes data management
activities and appropriate Quality Assurance/Quality Control policies and procedures.
At the time of this assessment, GCWW performed routine sampling for all regulated drinking water
parameters and had established sampling routes and plans. GCWW's routine sampling and analysis
frequencies for each method or group of methods is provided in Table 4-5.
Table 4-5. Sampling and Analysis Frequency
Contaminants
Total coliforms and E. coli
Method 524.2 VOCs
Copper and lead
Chlorine
Wet chemistry parameters
Sampling and Analysis Frequency
80 samples per week
30 samples per week
20 samples per week
2 samples per day at plant effluent plus 80
samples per week at microbiological
distribution system locations
1 sample per day
GCWW had developed a document called the "Section IIF4- Distribution Water Contamination Plan"
which included sampling standard operating procedures for incidents if contamination of the water system
was suspected. This plan addressed compliance monitoring sampling and sampling for unknowns, but
did not address scenarios involving analysis for unknown contaminants. Additionally, this plan did not
contain field screening procedures and only addressed sampling from taps. The pilot study consequence
management team later developed a more comprehensive response plan to address the different possible
warnings and responses to a contamination event.
Only a few GCWW employees had received 40 hour Occupational Safety and Health Administration
HAZWOPER training prior to implementation of the Cincinnati pilot. GCWW planned to rely on the
local HazMat units (e.g., Cincinnati Fire Department and Greater Cincinnati HazMat Unit) for sample
September 2008
47
-------
Cincinnati Pilot Post-Implementation System Status
collection during a potential contamination event. Although GCWW field personnel were experienced
samplers, they did not have the appropriate training, expertise, or equipment to respond to a potentially
hazardous contamination incident.
The GCWW laboratory was divided into inorganics, organics, microbiology, and wet chemistry sections;
each section had a quality assurance plan and standard operating procedures. These quality assurance
plans were reviewed and approved by the Ohio EPA auditor; therefore, EPA did not review them for
completeness or accuracy, but did verify that specific laboratory procedures were documented in the
quality assurance plan. GCWW did not have a quality assurance program or quality control practices for
the new methods that would be implemented during the Cincinnati pilot (e.g., new field methods,
laboratory-based chemical methods, or ultrafiltration). The document Assessment Report of Sampling and
Analysis Activities at the GCWW WS-CWS Pilot contains a more in-depth discussion of quality assurance
practices that were evaluated at GCWW (USEPA, 20071).
Although GCWW did not have a laboratory information management system, an in-house database was
used for data management. This water quality and treatment database captured all of the sampling
parameters (date, time, location, sample identification, and technician/analyst) and analytical results.
However, analytical quality control data are not captured in the database. All compliance data was
required to be maintained and stored for at least 10-12 years. A questionnaire, "Data Management for
Sampling and Analysis" was used by EPA for an initial assessment of GCWW capabilities. The results
of this survey showed that GCWW uses hardcopy chain of custody forms. Analysts check the sample
storage area daily to determine if new samples had arrived. It is the responsibility of the analyst to track
the samples in the laboratory. All hardcopy records pertaining to sampling and analysis are kept for a
minimum often (10) years.
4.1.3 Field Screening and Site Characterization
Table 4-6 describes GCWW's existing field screening and site characterization procedures, protocols,
and equipment. It also includes equipment available through external partners.
Table 4-6. Summary of Pre-lmplementation Field Screening and Site Characterization Capability
Organization
GCWW field
screening
methods
Emergency
response
coordination with
HazMat units
Cincinnati Fire
Department
Greater Cincinnati
HazMat Unit
Description of Capability
Cyanide analysis
Free chlorine, total chlorine, and iron analysis
pH testing
Turbidity analysis
Fluorometerto analyze for PCBs and fuels. However, these instruments were not used on a
routine basis.
• Existing SOPs for all screening equipment
• Prior to the pilot study, GCWW had not developed a response plan that included the local
HazMat teams affiliated with the Cincinnati Fire Department and the Greater Cincinnati
HazMat Unit.
• Full OSHA Level B response capabilities, and possesses a wide variety of field screening
instruments.
• Possessed several multi-gas meters for confined space entry that contain detectors for
VOCs, explosive gases, carbon monoxide, oxygen, and hydrogen sulfide.
• Similar capabilities to Cincinnati Fire Department.
• Field screening van that contains hundreds of different detection kits and instruments. An
inventory of the equipment in this van was not provided.
4.1.4 Summary of Identified Gaps
Table 4-7 summarizes the gaps identified during the initial assessment of sampling and analysis
capability at GCWW. Section 4.2 describes the post-implementation status for each of the attributes
listed below.
September 2008
48
-------
Cincinnati Pilot Post-Implementation System Status
Table 4-7. Sampling and Analysis Gap Analysis
Attribute
Gap Description
1. Laboratory capability
and capacity
GCWW did not have the in-house capability to conduct any biological analyses for
select agents. GCWW would require instrumentation and training to conduct the
LRN ultrafiltration procedure necessary to concentrate drinking water samples for
analysis by ODH using the LRN BT Agent Screening Protocol.
GCWW did not have any significant capability to screen or analyze for
radiochemicals.
GCWW did not have the in-house capability to perform carbamate pesticide analysis.
Contract laboratory support would be required.
GCWW had personnel to perform GC/MS analysis, but did not have a GC/MS
instrument dedicated to semivolatile organic compound analysis. Instrumentation
would be required for this function.
GCWW did not have ICP-MS capabilities for metals analysis.
MSP did not have a turbidity meter to check turbidity of drinking water samples.
2. Sampling and analysis
A QA program was needed for concentration of large volumes of water using
ultrafiltration.
QC practices and standards needed to be developed for field methods.
A baseline monitoring program needed to be designed, including development of
Sampling and Analysis Plans (SAPs).
Sample tracking equipment such as bar code readers was needed.
COC forms for baseline monitoring and triggered sampling and analysis were
needed.
SOPs for special activities associated with routine and triggered sampling and
analysis were needed (e.g., sample collection, packaging, and shipping procedures).
Additional data elements needed to be added to the WQ&T database.
A separate database was required for EPA to statistically analyze baseline data.
CSC was contracted to establish and maintain this database.
3. Field screening and
site characterization
GCWW did not have the equipment or personnel necessary to mobilize multiple
teams that could perform site (safety) characterization and field screening. Purchase
of field equipment was needed.
The screening equipment possessed in-house at GCWW, CFD, and GCHMU were
inconsistent. If each organization possessed the same basic screening
instrumentation, then communication of field screening information between these
organizations during a response event would be greatly improved.
CFD and GCHMU needed to gain familiarity with the potential sampling sites at
GCWW.
Coordination of roles and responsibilities of GCWW, CFD, and GCHMU to conduct
field screening and site characterization was needed.
4.2 Post-Implementation Status
This section provides a summary of enhancements to the sampling and analysis component which were
implemented during the Cincinnati pilot. These enhancements include resource documents and sampling
and analysis plans, training sessions, equipment purchases, standard operating procedures, and outreach to
partner laboratories and agencies. The following sections also summarize the post-implementation status
of laboratory capability and capacity, sampling and analysis, field screening, and site characterization for
both baseline monitoring and triggered sampling and analysis.
Gap analyses described in Section 4.1 were used to determine the appropriate enhancements and
modifications that would be required to implement the desired sampling and analysis capabilities. Table
4-8 summarizes post-implementation status of the sampling and analysis component with respect to
analytical capabilities for detection of potential drinking water contamination.
September 2008
49
-------
Cincinnati Pilot Post-Implementation System Status
Table 4-8. WS Detection Classes and Current Sampling and Analysis Capabilities for the GCWW
Pilot
Detection
Class
1
2
3
4
5
6
7
8
9
10
11
12
Contaminant Class
Petroleum products
Pesticides (chlorine reactive)
Inorganic compounds
Metals
Pesticides (chlorine resistant)
Chemical warfare agents
Radiochemicals
Bacterial toxins
Plant toxins
Pathogens causing diseases
with unique symptoms
Pathogens causing diseases
with common symptoms
Persistent chlorinated organic
compounds
Laboratory-Based Analytical
Method(s)
V
V
V
V
V
Method not available
V
Method not available
V
V
Method not available
V
Field-Based Screening Method
or Rapid Field Test(s)
V
V
V
Method not available
Method not available
V
V
Method not available
Method not available
Method not available
Method not available
Method not available
4.2.1 Laboratory Capability and Capacity
This section describes equipment and support laboratory enhancements implemented at GCWW. If
enhancements were needed to provide the capability to analyze for targeted contaminants, GCWW
determined if building in-house capability or capacity was sustainable. If in-house enhancements were
deemed sustainable, then consideration was given to providing instrumentation and training to GCWW
staff. Otherwise, a partner laboratory with relevant capability was identified to provide the analysis.
Table 4-9 contains a list of enhancements made at the GCWW laboratory.
Table 4-9. Samplinc
and Analysis Equipment Enhancements at GCWW
Equipment
Justification
Gas chromatograph /
mass spectrometer
and extraction
equipment
EPA and GCWW determined that it would be more beneficial to purchase a GC/MS and
the associated extraction equipment and supplies rather than compensate a contract
laboratory for semivolatile organic compound analyses.
Adequate laboratory space was available and GCWW personnel were familiar with the
instrument.
This enhancement is sustainable because GCWW, once certified, can analyze its own
compliance samples and possibly generate new revenue by providing analyses for other
utilities.
September 2008
50
-------
Cincinnati Pilot Post-Implementation System Status
Equipment
Ultrafiltration
equipment
Incubator
Barcode reader
Justification
• EPA and Ohio Department of Health determined that ODH did not have personnel
available to perform the Ultrafiltration concentration procedure for baseline monitoring.
• Transportation of large volume samples (up to 100 liters each) would be impractical. CDC
agreed to allow GCWW use the LRN filter concentration protocol to concentrate samples
and ship retentates to Ohio Department of Health for analysis.
• GCWW identified staff to conduct the Ultrafiltration procedure on-site and Ohio
Department of Health provided training. EPA purchased two Ultrafiltration units and
associated supplies for the GCWW microbiology laboratory.
• To support initial and ongoing proficiency demonstrations, EPA procured an incubator for
Quality Control procedures requiring bacterial culture at41°C.
• The barcode reader was purchased for GCWW to help the laboratory incorporate
electronic tracking of laboratory samples. Currently, laboratory staff are integrating
barcode reading into their sample tracking procedures.
The addition of new equipment and the use of new methods at GCWW (i.e., Ultrafiltration, and gas
chromatograph / mass spectrometer semivolatile analysis) required training and instructional
documentation. GCWW was already familiar with the gas chromatograph / mass spectrometer
instrumentation and operation; however, EPA provided training for extraction and analysis of SVOCs in
water.
GCWW did not have any prior experience with the Ultrafiltration concentration procedure, so
implementation of this capability required drafting standard operating procedures and training. The
Standard Operating Procedure for Routine Pathogen and Toxin Sampling, Packaging, and Shipping
(USEPA, 2007m) describes procedures for large volume sample collection (100 L), transport, packaging,
and shipping of concentrated retentate. This document was developed in collaboration with Ohio
Department of Health to support the use of the Laboratory Response Network Ultrafiltration procedure at
GCWW. This standard operating procedure provides guidance to GCWW staff collecting and preparing
Ultrafiltration retentates for shipment to Ohio Department of Health.
The Standard Operating Procedure for the Demonstration of Capability Using the CDC/LRN
Ultrafiltration Protocol: Determining Recovery of Enterococcus faecalis from Water (USEPA, 2007n)
describes the procedure used by GCWW to demonstrate initial and ongoing demonstration of capability
using the filter concentration/ultrafiltration procedure. Proficiency is based on acceptable percent
recovery (>50 percent) of Enterococcus faecalis spikes using the Ultrafiltration procedure and EPA
Method 1600. Ohio Department of Health, in conjunction with EPA, conducted training on the use of the
Laboratory Response Network filter concentration procedure for GCWW personnel in May of 2006.
In order to analyze for additional contaminant classes, GCWW and EPA built a laboratory network to
provide additional and contingency analysis during baseline monitoring and triggered response events.
Table 4-10 lists the partner laboratories that were identified to support analysis of samples and a
description of the type of agreement that was utilized.
Table 4-10. Laboratory Contracts and Letters of Intent
Laboratory
Metropolitan Sewer District
of Greater Cincinnati
Mobile Analytical Services
Incorporated
Ohio Department of Health
Radiochemical Laboratory
Description of Agreement
The letter of intent between Metropolitan Sewer District and the U.S. EPA made it
possible for GCWW to send samples to Metropolitan Sewer District for metals analysis
during routine and triggered sampling, as GCWW could not perform these analyses in-
house. In addition, a turbidity meter was purchased for Metropolitan Sewer District.
A contract with Mobile Analytical Services Incorporated allowed for analyses of
samples for carbamate pesticides during baseline monitoring. GCWW could not
conduct these analyses in-house.
A contract with Ohio Department of Health allowed for analyses of samples for
radiochemicals during baseline monitoring. GCWW could not conduct these analyses
in-house.
September 2008
51
-------
Cincinnati Pilot Post-Implementation System Status
Laboratory
Ohio Department of Health
Laboratory Response
Network Laboratory
Public Health Foundation
Enterprises
Test America - Savannah
Cincinnati Health
Department Laboratory
Description of Agreement
A letter of intent with Ohio Department of Health and the U.S. EPA enabled the
analysis of samples filtered by GCWW in accordance with the LRN ultrafiltration
procedure by the LRN BT Agent Screening Protocol. The letter of intent also made
available the necessary LRN reagents for this analysis.
A contract provided funding for overtime work in the event that laboratory analysts
would have to complete analyses for BT agents at the Ohio Department of Health
laboratory outside of normal working hours.
A laboratory contract with Test America, Savannah was established in the event that
other local partner chemistry laboratories could not analyze samples for any reason.
Test America, Savannah is a large full service environmental laboratory that has a
wide variety of drinking water certifications.
Real time PCR instrumentation was purchased and installed at the Cincinnati Health
Department Laboratory to build detection and response capabilities for non-select
pathogens (Bordetella spp., Staphylococcus aureus, norovirus). This is a longer-term
project that can hopefully provide support in the second and third year of the pilot.
Contaminant class analytical capabilities, by laboratory, are listed in Table 4-11. GCWW analyzed
drinking water samples for volatile organic compounds (VOCs), semivolatile organic compounds
(SVOCs), free cyanide, and concentrated large-volume samples using ultrafiltration for biological
analyses. The Ohio Department of Health performed the radiochemical analyses and analyzed the
ultrafiltrate prepared by GCWW for biological agents (select agents and toxins). The Metropolitan Sewer
District performed the metals analysis. Mobile Analytical Services, Inc. analyzed all the samples for
carbamate pesticides. These laboratories were selected based on proximity to GCWW and the ability to
use the methods recommended by EPA for the analysis of the priority contaminant classes.
Test America-Savannah performs contingency analyses for the Cincinnati pilot. Contingency analysis
serves two purposes: the contingency laboratory can substitute for other laboratories when capacity is
exceeded, and the contingency laboratory can process samples in the event that Cincinnati is disabled
(e.g., natural disaster). Test America-Savannah was selected as a contract laboratory through a
competitive bidding process to perform the analyses listed in Table 4-11. Test America-Savannah is
located in Savannah, Georgia, so samples must be shipped via overnight delivery to the laboratory.
Table 4-11. Laboratories Supporting the Cincinnati Pilot, Analytical Methods, and Analytes
Laboratory
Contaminant Class
Instrumentation
GCWW Utility Laboratory and Local Partner Laboratories
Greater Cincinnati Water Works
Greater Cincinnati Water Works
Metropolitan Sewer District of
Greater Cincinnati
Mobile Analytical Services Inc.
VOCs
SVOCs
Cyanide
Sample Concentration for the LRN BT-
Agent Screening Protocol
Metals
Carbamates
Gas Chromatography with Mass
Spectrometry Detection using
purge and trap
Gas Chromatography with Mass
Spectrometry Detection using
liquid-solid extraction (LSE)
Colorimeter
Pumps and Filters
Inductively Coupled Plasma -
Mass Spectrometry
High Performance Liquid
Chromatography with
fluorescence detection
September 2008
52
-------
Cincinnati Pilot Post-Implementation System Status
Laboratory
Contaminant Class
Instrumentation
GCWW Utility Laboratory and Local Partner Laboratories
Ohio Department of Health
Test America - Savannah
BT Agents
Radiochemicals
Metals
Total Cyanide
VOCs
SVOCs
Carbamates
Real-time PCR and
Immunoassay (TRF) platforms
Alpha Beta Scintillation Sealer or
Gas Flow Low-Background
Proportional Detector
High Purity Germanium Gamma
Spectrometry System
Inductively Coupled Plasma -
Mass Spectrometry
Colorimetry with Reflux
Distillation Extraction
Gas Chromatography with Mass
Spectrometry Detection using
purge and trap
Gas Chromatography with Mass
Spectrometry Detection using
liquid-solid extraction (LSE)
High Performance Liquid
Chromatography with
fluorescence detection
Test America - Savannah will process samples using these methods when Cincinnati area laboratories' capacity is
exceeded. This laboratory is also contracted to analyze total cyanide.
4.2.2 Sampling and Analysis
This section describes enhancements implemented for sampling and analysis activities at GCWW as part
of the pilot. This includes baseline monitoring design, sampling and analysis plans, standard operating
procedures, forms, and other supporting resource documents.
EPA designed a baseline monitoring program at GCWW to determine contaminant occurrence
(contaminant, levels detected and frequency of detection) and method performance of the chemical,
biological, and radiochemical methods implemented. This design is summarized in the document,
Baseline Monitoring at the Greater Cincinnati Water Works Water Security - Contamination Warning
System Pilot (USEPA, 2007o). The document identifies six phases of baseline monitoring, presents the
statistical design of specific sampling efforts, and identifies sampling locations and frequencies to provide
adequate spatial and temporal coverage of the GCWW distribution system. The document provides
detailed sampling and analysis plans for field measurements, chemical contaminants, radiochemical
contaminants, and select agents and toxins. Prior to the development of the baseline monitoring plan,
laboratory specific study plans were prepared for Ohio Department of Health and Metropolitan Sewer
District.
GCWW requested toxicity information for the targeted contaminants being monitored for during baseline
monitoring in case any were detected in drinking water samples. Existing EPA toxicity data for these
contaminants was compiled and provided to GCWW in a document titled Supplemental Information for
Water Security - Contamination Warning System Targeted Contaminants Monitored During the Pilot at
the Greater Cincinnati Water Works (USEPA, 2007p).
Several day to day sampling and analysis tools were developed to aid in the execution of sampling and
analysis during Cincinnati pilot. These include chain of custody forms for baseline monitoring that
September 2008
53
-------
Cincinnati Pilot Post-Implementation System Status
contained pre-populated baseline monitoring analyses and laboratory contact information, and Method
1600 data reporting forms developed to standardize the quality control data generated by GCWW when
conducting ultrafiltration quality control samples.
EPA created a data management system to store and manipulate data received from the laboratories
supporting the pilot. Currently GCWW does not have a laboratory information management system, and
many of the support laboratories are inflexible as to the type of electronic deliverables they are able to
produce. Therefore, EPA established a separate mechanism to collect electronic data from each source.
The various data streams and the database used to house them are summarized in the WS-CWS Pilot Study
Data Management Plan (USEPA, 2007q).
4.2.3 Field Screening and Site Characterization
This section describes enhancements implemented for field screening and site characterization activities
at the Cincinnati pilot. EPA, GCWW, the Cincinnati Fire Department, and the Greater Cincinnati
Hazardous Materials Response Unit participated in a series of conference calls between January and April
2006 to discuss and identify field screening equipment that would be most appropriate for a water utility.
The equipment and methods were selected to screen for as wide a variety of contaminant classes as
possible using available technologies. Preference was given to equipment that had been tested by EPA's
Environmental Technology Verification Program. Due to the sensitive nature of revealing specific
detection capabilities at the GCWW pilot, Table 4-12 instead summarizes the contaminant classes for
which capabilities were identified and implemented. Ease of use in the field was also considered.
Table 4-12. Cincinnati Pilot Safety Screening and Field Testing Equipment
Safety Screening
Contaminant Class
Radioactivity (alpha,
beta, and gamma)
General hazards
VOCs and combustible
gases
Methodology
Hand-held device
HazCat (explosives, oxidants, etc.)
Hand-held device
Comments
May be expanded to water testing with a
special probe
Should be performed by trained HazMat
responder
Detects chemicals in air
Rapid Field Testing
Contaminant Class
Cyanide
Chlorine residual
pH/conductivity/ORP
Methodology
Portable colorimeter
Portable colorimeter
Portable electrochemical detector
Comments
Tests water for cyanide ion, but not
combined forms
Absence of residual may indicate a
problem
Abnormal pH or conductivity may indicate
a problem
Rapid Field Testing
Turbidity
Chemical Warfare
Agents (VX, sarin, etc.)
Toxicity
Portable Turbidimeter
Test Kit
Test Kit
High turbidity may indicate a problem
May also detect some pesticides and
common chemicals
Need to establish a baseline
September 2008
54
-------
Cincinnati Pilot Post-Implementation System Status
Once the field screening equipment was purchased, standard operating procedures were developed for
each instrument listed in Table 4-12; these standard operating procedures are available in the Site
Characterization Plan (Appendix O of the Consequence Management Plan). Field data reporting forms
were developed for the new equipment and streamline collection of field screening data and capture time-
of-use quality control. GCWW also developed a training program for field equipment. In addition, each
analyst is required to perform ongoing quality control. Field screening is practiced regularly during
baseline monitoring sampling and would be performed during triggered sampling events. Triggered
sample collection activities are identical to baseline monitoring, except that additional health and safety
precautions are taken in the field and additional samples may be collected in case additional or
contingency analyses are required.
Additional standard operating procedures were drafted for sampling and safety procedures required for
triggered sample site characterization activities which were developed by EPA and reviewed by GCWW.
These standard operating procedures encompass a wide variety of sample collection activities that might
be encountered at GCWW, and include necessary procedures that support sample collection. They
include: site approach including hazard awareness, pre-sampling guidelines from drinking water sources,
sample container labeling and packaging, decontamination of personnel and equipment, sampling from
accessible water taps, sampling from fire hydrants, sampling from water towers, and sampling from
underground tanks or reservoirs. The standard operating procedures are also available in the Site
Characterization Plan (Appendix O of the Consequence Management Plan).
GCWW, in conjunction with EPA, conducted a sampling and field screening training session in August of
2006 for GCWW, Cincinnati Fire Department, and Greater Cincinnati Hazardous Materials Unit
personnel. The training covered all of the standard operating procedures listed above for field screening
equipment and triggered field sampling and safety procedures. Follow-up training exercises occurred in
April 2007 for GCWW and first responders on appropriate use of the site characterization standard
operating procedures. The training also provided guidance to GCWW regarding instances when HazMat
Units should be contacted for sample collection. This training is described in more detail by the
consequence management component section of this report.
4.2.4 Summary of Post-Implementation Status
EPA and GCWW have worked collaboratively to expand in-house utility laboratory capabilities and
developed standard operating procedures for new procedures at GCWW. These include sample
concentration using the Laboratory Response Network ultrafiltration procedure, analysis of semivolatile
organic compounds, and development of triggered sampling and field screening site characterization
procedures. EPA and GCWW have also developed a laboratory network for a wide variety of
contaminants for which GCWW does not possess capability to analyze. GCWW site characterization
procedures and screening techniques have been developed with cooperation from the Cincinnati Fire
Department and the Greater Cincinnati HazMat Unit.
One year of baseline monitoring for all field and laboratory methods should be completed by the end of
April 2008. Initial demonstrations of capability were performed for each laboratory analysis that was not
part of the state drinking water certification program. Baseline data has been collected from the treatment
plants and strategic locations. Data was collected over an extended period of time to evaluate seasonal
and temporal trends, and spike recovery studies were pursued to determine matrix interference. A survey
study of water from over sixty (60) locations throughout the distribution system was conducted to
determine whether local trends could be detected. Maintenance monitoring will continue beyond the
baseline monitoring period to maintain response capabilities and to periodically update baseline
contaminant occurrence and method performance data. Through implementation of a maintenance
monitoring program, the utility will sustain the capability to respond to triggered events.
Table 4-13 lists all of the most significant sampling and analysis capabilities implemented for the
Cincinnati pilot.
September 2008 55
-------
Cincinnati Pilot Post-Implementation System Status
Table 4-13. Sampling and Analysis Post-Implementation Status
Attribute
Description of Installed Component
1. Laboratory capability
and capacity
• Semivolatile organic compound analysis at GCWW
• Ultrafiltration concentration capability at GCWW
• Laboratory response network capable of analyzing drinking water samples for
EPA priority contaminant classes
2. Sampling and analysis
Baseline monitoring design
In April 2008, GCWW will have completed one year of baseline monitoring.
Results from the baseline monitoring period will be used to determine a
maintenance monitoring schedule. EPA is working with GCWW to address
unique sampling and analysis scenarios such as sample apportionment for
analyses when sample volume is limited and evidentiary chain of custody
procedures.
3. Field screening and
site characterization
Purchased field screening equipment for GCWW and local HazMat teams
Developed site characterization procedures compatible with local HazMat
response teams
Integrated local HazMat response teams into the GCWW triggered sampling
procedures
Four triggered sampling kits were assembled by EPA, in case a triggered
event occurred
Currently the consequence management component has performed two field
drills for triggered sampling, one including CFD and Greater Cincinnati
Hazmat Unit. The drills are described in more detail in the consequence
management section.
Figure 4-1 provides a summary of the level of effort associated with design and implementation of the
sampling and analysis component for the Cincinnati pilot. Sampling and analysis activities, as
summarized in Table 4-13, relied on support from EPA, GCWW, and local partners. The most significant
effort was associated with coordination of sampling and analysis activities during implementation of
baseline monitoring at GCWW. Utility laboratory personnel at GCWW were responsible for collecting
all baseline monitoring samples, and conducted some sample analyses for targeted biological and
chemical analytes. Local partner laboratories expended a considerable amount of effort during
implementation of the pilot study to analyze samples that could not be processed at the utility laboratory.
EPA also expended effort to conduct training at the utility to increase analytical capability and familiarize
personnel with field screening/site characterization activities.
September 2008
56
-------
Cincinnati Pilot Post-Implementation System Status
4.5 T
4.0
n Local Partners LOE
GCWW LOE
EPA LOE
Laboratory Capability and
Capacity
Sampling and Analysis
Field Screening and Site
Characterization
Figure 4-1. Level of Effort for Design and Implementation of the Sampling and Analysis
Component (December 2005 - December 2007)
Figure 4-2 presents a summary of the extramural costs associated with design and implementation of the
sampling and analysis component for the Cincinnati pilot. The most significant extramural labor costs
include contractor activities associated with coordination of sampling and analysis activities during
implementation of baseline monitoring at GCWW. Baseline monitoring sample data was transmitted to
and analyzed by EPA contractors. The EPA contractors were also integrally involved in the
establishment of a laboratory network to expand and enhance the analytical capabilities of the utility
laboratory. The most significant equipment costs included procurement of analytical laboratory
equipment, including a GC/MS, which was installed at GCWW. Chemistry standards, reagents, and
consumable supplies (i.e., ultrafiltration equipment) were purchased for GCWW and several local partner
laboratories during baseline monitoring, to allow for analysis of targeted priority contaminants. Costs
associated with contractor travel were not included in this calculation.
September 2008
57
-------
Cincinnati Pilot Post-Implementation System Status
$1,000
$900
•Equipment, Consumables
and Purchased Services
D Extramural LOE
Laboratory Capability and
Capacity
Sampling and Analysis
Field Screening and Site
Characterization
Figure 4-2. Extramural Costs Associated with Design and Implementation of the Sampling and
Analysis Component (December 2005 - December 2007)
Costs associated with establishing laboratory capability and capacity and conducting baseline monitoring
represent a significant portion of costs associated with sampling and analysis for the Cincinnati pilot.
Figure 4-3 shows the expenditures by laboratory to build capability and perform baseline monitoring for
the Cincinnati pilot. The largest investment was made at GCWW. Costs included purchase of a GC-MS
and equipment for SVOC extraction, bar code readers, pathogen concentration equipment, etc., (see Table
4-9) as well as reagents and supplies for sample collection, packaging, shipping and analysis. Contract
laboratory support is the next largest expenditure. Contract laboratories provided analyses of samples for
metals, carbamates, radiochemicals and provided contingency analyses. Refer to Table 4-11 for a
description of partner laboratory roles. Expenses at the Cincinnati Health Department Laboratory were
primarily for PCR equipment to build future capabilities to analyze for non-select agents in water.
Expenditures for select agent analyses at the Ohio Department of Health Laboratory were primarily for
reagents and consumables.
September 2008
58
-------
Cincinnati Pilot Post-Implementation System Status
D Contract laboratories
(analytical costs)
DGCWW
(consumables, equipment, and warranties)
DOhio Department of Health - Pathogen Analysis
(consumables and equipment)
D Cincinnati Health Department Laboratory
(consumables, equipment, and warranties)
Figure 4-3. Breakdown of Expenditures per Laboratory
Expenditures were made to build capability for pathogen analysis in three (3) laboratories; the GCWW
Miller Treatment Plant Laboratory, the Ohio Department of Health Laboratory and the Cincinnati Health
Department Laboratory. Figure 4-4 presents the costs associated with this effort. During baseline
monitoring for select agents, GCWW utility personnel collected and concentrated by ultrafiltration large
volume water samples. Special equipment had to be purchased for this purpose. Consumable expenses at
the GCWW laboratory included sample collection containers, preservatives, reagents and supplies for
concentration of samples as well as reagents and supplies to analyze samples for quality control purposes.
The Ohio Department of Health Laboratory performed secondary filtration and analysis for five select
agents. The bulk of expenditures for the Ohio Department of Health were for consumables to perform
these expensive analyses as well as for method optimization experiments. Equipment purchases for the
Ohio Department of Health were for small items, e.g., automatic pipets to increase processing efficiency.
An investment was made in the Cincinnati Health Department Laboratory to build future capabilities for
non-select agent analysis using real-time PCR. Work is still in progress to evaluate methods for the
analysis of non-select agents in water at the Cincinnati Health Department Laboratory.
September 2008
59
-------
Cincinnati Pilot Post-Implementation System Status
$90,000 n
$80,000 -
$70,000 -
D.OOO -
000 -
',000 -
$20,000 -
$10,000 -
$-
S1.300
$83,000
Equipment
$10,000
Consumables
• GCWW
Ohio Department
of Health
D Cincinnati Health
Department
Laboratory
Figure 4-4. Costs Associated with Establishing Pathogen Monitoring Capabilities
September 2008
60
-------
Cincinnati Pilot Post-Implementation System Status
Section 5.0: Enhanced Security Monitoring
The enhanced security monitoring component includes the systems, equipment, and procedures that
detect, delay and respond to security breaches at distribution system facilities such as pump stations,
reservoirs and storage vessels that are vulnerable to contamination. The monitoring strategy includes
detection by physical security systems such as alarms and cameras, witness accounts, notifications by
perpetrators, media, and law enforcement, as well as associated response methods. Alarms and video are
transmitted to the central control facility at the Miller Plant (California Control Center), where they are
monitored by GCWW personnel. Under the contamination warning system model, enhanced security
monitoring is designed to help discriminate between notifications that may be related to a contamination
incident and those resulting from other activities.
The enhanced security monitoring component design objectives are shown in Table 5-1, and were
derived from the overarching performance objectives of the contamination warning system as described
in WaterSentinel System Architecture (USEPA, 2005a). GCWW's pre-existing capability with respect to
each attribute of the enhanced security monitoring component listed in Table 5-1 is summarized in
Section 5.1 and the utility's capability after implementation of the contamination warning system is
summarized in Section 5.2.
Table 5-1. Enhanced Security Monitoring Component Design Objectives
Design Element
1. Physical Security
Equipment
2. Data Management
and Communications
S.Component
Response Procedures
Design Objective
Contact alarms, motion sensors and cameras designed and installed to detect
intrusion and help discriminate between potential contamination threats and routine
access to facilities.
Communication technology which would allow video of sufficiently good quality to
be transmitted in a time frame which could assist in stopping a contamination event
or limiting the spread of contamination.
Written standard operating procedures exist for every step in responding to a
security monitoring alarm. These procedures outline effective and timely
communications, including clear guidance on appropriate response actions
5.1 Pre-lmplementation Status
After completing a vulnerability assessment, as required under the Public Health Security and
Bioterrorism Preparedness and Response Act of 2002 (USFDA, 2002), GCWW chose to concentrate their
limited funding on intruder delay and intrusion detection capabilities for distribution system facilities,
rather than more advanced assessment technologies, such as video surveillance. Prioritization of the
security enhancements for distribution system facilities focused on addressing the facilities with the
highest consequence of attack. Intruder delay measures, which limited access to facilities with perimeter
fencing and locks on points of entry, were installed at all facilities. Detection and assessment for these
facilities would only occur through periodic site visits by GCWW maintenance personnel or by GCWW
security staff. Intrusion detection equipment, such as contact alarms or limit switches on points of entry,
was installed on all stations and tanks. Motion sensors were installed at one of the major pumping
facilities.
Thus, prior to the implementation of the contamination warning system, the physical security systems and
processes in place for GCWW distribution facilities were primarily intended to delay entry of
unauthorized individuals and detect intrusion. GCWW did install motion sensors in one facility parking
lot to deter theft from outdoor storage areas. Further details on the assessment of the existing systems,
identification of gaps and proposed enhancements, and the prioritization process for final selection of
improvements can be found in the WaterSentinel Enhanced Security Assessment (USEPA, 2006c).
September 2008 61
-------
Cincinnati Pilot Post-Implementation System Status
5.1.1 Physical Security Equipment
During the physical security equipment assessment, GCWW's facilities were categorized into three
groups that presented distinct vulnerabilities: pump stations, reservoirs/ground-level storage tanks, and
elevated storage tanks. The equipment and systems in place at each type of facility prior to
implementation of the contamination warning system are discussed in the following sub-sections.
5.1.1.1 Pump Stations
All pump station doors were fitted with contact switches which would trip any time the door was opened.
The switches would send a door alarm signal via the utility SCAD A system to California Control Center
every time the door was opened. No contact or motion sensors were provided on windows or ventilation
louvers to detect intrusion.
Additionally, an access control system (key card) was in place on the entrance doors at the two largest
pump stations. The access control system provides for keyless entry, but is completely separate from the
door alarm system. Contact switches on the station doors signal an alarm upon any entry to the facility
whether or not a key card was used for entry. The access control system has a monitor in the guard post
at the Richard Miller Treatment Plant that displays whether a key card was used to open a door, but this
data was not communicated to the SCADA system. Therefore, entry via a key card triggers the alarm in
the same manner as access at locations without the keyless entry system.
5.1.1.2 Reservoirs and Ground-Level Storage Tanks
All reservoirs and ground storage tanks were fitted with contact alarms on access hatches and doors.
Metal ground-level storage tanks are cylindrical with a sizable side wall depth. Access to the top of the
tank and the water surface was typically provided by an enclosed ladder. In most locations, a contact
alarm or limit switch was provided on the access door to the ladder compartment. Cast-in-place concrete
reservoirs are typically below grade. Personnel access to the reservoirs is provided through a stairway in
an adjoining pump station or through sidewalk style access hatches. In most locations, a contact alarm or
limit switch was present on the stairway access door or access hatch. Cylindrical post tensioned
reservoirs have a low profile such that access to the top can be gained using a portable ladder or in some
cases a portion of the tank top is at grade with the surrounding hillside. Access hatches are provided on
the domed roof for access to the interior of the tank. In many locations, contact switches were present on
the doors and access hatches.
All reservoirs and tanks have passive vent structures allowing air to enter and exit the storage area as the
tank emptied or filled. The vent structures on reservoirs typically consisted of a box with louvers on at
least one side. The vents on the cylindrical concrete tanks are manufactured aluminum or fiberglass
reinforced plastic units with a mushroom shape. The openings on both type vents are covered with a
screen to prevent animals from entering the storage tank, but the screens would provide little barrier to
intruder access.
5.1.1.3 Elevated Storage Tanks
Access to the interior of the elevated storage tanks is provided via a ladder enclosed in the tank base or in
an enclosed access shaft. The access door to the area containing the ladder was fitted with a contact
alarm.
5.1.2 Data Management and Communications
Security data was managed through a custom Citect SCADA application installed during an upgrade in
2006. Door and hatch contact alarm signals were transmitted to the California Control Center and
displayed on the human/machine interface to the Citect SCADA system along with operational data and
alarm signals. The alarm signals on the HMI were common alarms indicating only that a door or access
hatch alarm was open without identification of the specific door at that facility. The screens on the
operations monitors accessed by the operators at the control center were configured to highlight
operations data and system status parameters needed by the operators to maintain target water quality and
September 2008 62
-------
Cincinnati Pilot Post-Implementation System Status
pressure. The monitors and screens were judged by GCWW to display and manage nearly the maximum
amount of data that operations staff could process in performing their operations duties.
A well-developed data communication system was in place at GCWW prior to implementation of the
contamination warning system, primarily to support operational needs for water quality and pressure
monitoring and control. Tl lines were used for communications between the California Control Center
and the two largest pump stations. Private radio was used for some other remote locations; however the
hilly topography of the Cincinnati area prevented direct radio communications from many locations to the
California Control Center, so standard phone lines were used to transmit information in many situations.
The communications system in place for the monitoring and control systems for the distribution systems
facilities was developed to provide reliable yet efficient transfer of operational data and control signals
between the operations center and the remote facilities. The data transfer capacity of the system was
adequate for existing operational needs with some additional available capacity for future operational-
related parameters. The communications system had insufficient capacity for high bandwidth additions,
such as video communications.
The door access system in place at the two large pump stations is a Simplex brand system that provides
for key card access to a door and identification of the status of individual key card access doors. The
control system maintains a record on the time of entry of each key card controlled door. The Simplex
system data record is accessible at the Richard Miller Water Treatment Plant guard house.
5.1.3 Component Response Procedures
Security alarms were indicated at the operations station in the California Control Center. Prior to the
introduction of the contamination warning system enhancements, the security alarms at distribution
system facilities were confined to door access or hatch access alarms. For all such alarms, the GCWW
protocol was that a utility employee entering a facility was required to call the California Control Center
within two minutes of entry to inform the operators that they had accessed the door. When the staff
followed protocol and called in to report their visit, the alarm was dismissed. If the California Control
Center did not receive a call within 2 minutes, the operator would contact local law enforcement and
notify GCWW security staff of the potential unauthorized intrusion.
The security staff of GCWW is charged with securing the central utility office, treatment facilities and
distribution facilities. GCWW closely partners with all law enforcement agencies/jurisdictions in their
large geographic service area for investigation of intrusion incidents. These partnerships were based on a
concerted effort by the utility to minimize the number of false alarms.
5.1.4 Summary of Identified Gaps
Table 5-2 presents the gaps identified during the initial assessment. Section 5.2 describes the post-
implementation status for each of the attributes listed below.
Table 5-2. Enhanced Security Monitoring Component Gap Analysis
Design Element
1. Physical Security
Equipment
2. Data Management
and Communications
3. Component
Response Procedures
Description of Gap
• Existing ground storage tank vent structures provided an opening near the
ground surface through which a contaminant could be introduced to treated
water stored at atmospheric pressure; no system was in place to minimize or
detect intrusion at or around the vents.
. The security system in place for the pump station facilities provided for detection
of intrusion only and not assessment.
. The communications system had insufficient capacity for high bandwidth
additions such as video communications.
. A formalized, documented process for responding to security alarms was in
place. However, due to the security improvements recommended as part of the
pilot project, the existing process needed to be revised.
September 2008 63
-------
5.2
Cincinnati Pilot Post-Implementation System Status
Post-Implementation Status
In conjunction with GCWW, EPA first reviewed the results of their initial vulnerability assessment,
completed as required under the "Bioterrorism Act" (USFDA, 2002). The approach described in the
WaterSentinel Enhanced Security Assessment (USEPA, 2006c) and shown in Table 5-3 was used to
prioritize facilities which would receive security system enhancements as part of the WS program.
Table 5-3. Enhanced Security Monitoring Design and Implementation Approach
Activity
1. Preliminary
Facility List
2. Site
Assessment
3. Design Basis
Threat
4. Risk Ranking
5. Improvements
Recommendations
Description
Determination of which GCWW distribution system facilities (elevated storage tanks,
ground storage reservoirs and pump stations) may be at the highest risk for
contamination. The list was developed primarily based on simulations of contamination
at 28 of the facilities represented in the GCWW water quality and hydraulic model, as
well as knowledge of system hydraulics for two facilities that are not in the model. In
addition, other factors were considered, such as site location, accessibility and visibility.
A detailed site assessment was performed for the facilities, including investigation of
structural attributes and physical locations of the facilities and existing security systems
and practices. These included physical security, proximity to the public, terrain,
adjacent land uses, site access, site lighting, alarm and detection systems, and physical
barriers such as fencing and hardened structures.
The threats faced were defined, including types and capabilities of adversaries and
quantity and type of contaminants that could be used.
The contamination risk faced by each facility was evaluated by estimating the
effectiveness of existing security systems, the consequence of contamination at that
facility, and the probability that a facility may be attacked. This approach used a
modified approach to the Risk Assessment Methodology for Water (RAM-W™),
developed by Sandia National Laboratory.
For facilities ranked highest, a more detailed evaluation was done of existing security
systems, and recommendations were made for potential security system
improvements, including cost estimates. A cost-to-benefit analysis was done by
estimating cost in terms of the amount that contamination risk would be reduced by
implementing the security improvements. Improvements were made at facilities with the
best cost-to-benefit ratios that could fit within the component budget of $400,000.
5.2.1 Physical Security Equipment
Table 5-4 provides a complete list of the major security equipment installed as part of the contamination
warning system, organized by location. The sub-sections following the table discuss the equipment
installed at each group of facilities.
Table 5-4. Enhanced Security Components at GCWW Distribution Facilities
Component
ESM PLC
Local Facility
Switch
Digital Cellular
Modem
Remote Video
Engine
Cameras
Manufacturer
Bristol
Cisco
Digi
Longwatch™
Pelco
Model
Controlwave Micro
PLC
2955
Connectport WAN
VPN 3G 300 KBPS*
RVE-100
CC377OUH-6-Fixed
SD53CBW-PG-1-PTZ
Quantity
3
3
3
3
6
5
Site(s) Installed (Quantity)
3 Pump Stations
3 Pump Stations
3 Pump Stations
3 Pump Stations
3 Pump Stations
2 Pump Stations
September 2008
64
-------
Cincinnati Pilot Post-Implementation System Status
Component
Door Status
Switch
Hatch Status
Level Switch
Ladder Motion
Sensor (MSL)
Indoor Motion
Sensor (MSC)
Rollup Door
Status (RS)
Manufacturer
GE Security
GE Security
Omega
Protech
Honeywell
GE Security
Model
2507AD
2507AH
LV-40
Piramid SDI-77XL-
DIR-LT
Intellisense DT-906
120'x10' Pattern
2317AH
Quantity
35
13
5
7
10
2
Site(s) Installed (Quantity)
3 Pump Stations
4 Reservoirs
4 Reservoirs
2 Stand pipes at pump stations
4 Elevated tanks
1 Ground level storage tank
3 Pump Stations
1 Pump Station
* This is the modem capacity on a 3G network. The CBT network is currently a 2.5G network using the EDGE
technology. Actual network throughput is around 50 kbps.
5.2.1.1 Pump Station Equipment
The video monitoring equipment was the most important piece of the security enhancements installed at
the three (3) pump stations. The Longwatch™ video system was selected because it includes a unique
feature that provides event-based video versus continuously streaming video typically found in
conventional closed circuit TV systems. This allows the Longwatch™ video system to function on low
bandwidth communication networks, and so may have applicability beyond the Cincinnati pilot. The
intent of the event-based system is that while the Longwatch™ system continuously records the video
from the cameras on a local video recorder, it only sends short duration video clips in response to a
detected security incident. Thus, while streaming video continuously consumes considerable bandwidth
on the communications system, the event-based video clips imposes only a brief data load on the system.
The continuous video stored on the video recorders at each of the pump stations provides an additional
assessment tool for intrusion after the event, without imposing a heavy communications load. Event
detection at the pump stations is provided by contact switches on external doors and by area motion
detectors on interior walls that have windows or ventilation louvers. The video clips are transmitted to
the California Control Center for assessment by the operations staff monitoring the human/machine
interface consoles. This approach provides a very communications-efficient means of using video.
Except for entrance gate monitoring (discussed below), the video systems deployed at the pump stations
were entirely confined to the interior of the stations. The intent was to detect and respond to intrusion
where the perpetrator has access to water-bearing equipment and piping. The video cameras are intended
to capture video clips of the area where intrusion occurs in order to provide an initial assessment of the
intruder. To achieve this level of functionality, two different types of video cameras were used: pan-tilt-
zoom (PTZ) and fixed cameras with zoom capability. The PTZ cameras can rotate to provide coverage
over a large area and zoom in on a specific location. These cameras provided video coverage of multiple
intrusion points more cost effectively than individual fixed cameras for each entry point. The fixed
cameras were used for pump stations with four (4) or fewer points or areas of entry. One (1) camera was
enclosed in a sealed and pressurize cylinder because the access area atmosphere contained a high
concentration of chlorine that would be corrosive to a camera in a standard enclosure.
Supplemental lighting systems were also installed in the pump stations and tied to the video controls. The
lighting systems provided sufficient illumination for resolution of the video images, whether taken in
daylight or nighttime conditions. The supplemental lights are energized any time a door contact or
motion sensor alarm is tripped. Halogen or fluorescent bulbs were used for supplemental lighting because
of the slower illumination time associated with metal halide or sodium vapor lights. A detailed
description of the function and operation of the video systems is provided in Water Security Initiative
Enhanced Security Monitoring Video Intrusion Alarm Assessment Operations Manual (USEPA, 2007r).
September 2008
65
-------
Cincinnati Pilot Post-Implementation System Status
Because of the frequency of deliveries and visits by non-utility personnel to two (2) pump stations,
GCWW requested that fixed cameras be installed outside these locations to provide video coverage of the
vehicle gates. The vehicle gates allow access to employees with key cards. The video cameras allow the
California Control Center operators to take a video clip when a call is received to verify the identity of the
caller before opening the gate. The vehicle gate cameras also provide for intrusion assessment if a large
vehicle was used to break through the gate.
Two (2) of the pump station facilities also have large standpipes that provide access to finished water
from the treatment plant. Motion sensors were installed for the ladders that provide access to the top of
the standpipes. These sensors send an intrusion alarm back to the California Control Center, but are not
tied into the video control systems.
5.2.1.2 Reservoir and Ground Level Storage Tank Equipment
Enhanced security equipment was installed at five (5) reservoirs, including four (4) circular tanks with
walls projecting above ground level and shallow domed concrete roofs, and one (1) cast-in-place reservoir
that is entirely below the ground surface. For reservoirs with above ground roofs, rectangular fabricated
aluminum structures were added over the existing vent structures to eliminate direct access to the vents.
The structures contained screened holes in the walls of at least equal area to the openings in the vents to
allow the vents to function as intended. However, the openings were covered with a screen and inverted
L-shaped louvers to block the line of sight to the vent. The structures were also fitted with sidewalk style
hatches to provide access to the vents for inspection and cleaning.
Two (2) types of intrusion detection devices were installed at each vent enclosure. The first was a contact
switch on the hatch that sends an alarm when the hatch is opened. The second was a liquid-level switch
installed at the base of the enclosure sidewall. The enclosures have drain holes at the base to allow rain
water that enters through the louvers to drain out. The liquid-level switches were provided to detect a
situation where a terrorist would plug the drain holes and the louvers and introduce a liquid contaminant
into the enclosure. The liquid level switch inside the enclosure will send an alarm to the California
Control Center if the fluid level in the structure rises to the detection level. The reservoirs have existing
access hatches which were previously fitted with contact switches. The alarm signals from the vent
enclosure are wired in series with the existing access hatch alarms.
The single below-grade reservoir is accessed by a stairwell in an adjacent pump station. The stairwell has
only an exterior door, which is already protected by an alarm. A fixed camera was installed inside the
stairwell and enclosed in a sealed container to protect against chlorine vapors that may corrosive to the
camera (see Section 5.2.1.1).
5.2.1.3 Elevated Storage Tank Equipment
Four (4) elevated storage tanks and one (1) ground storage tank were fitted with enhanced security
equipment. Each of the tanks has a ladder to provide access to the top of the storage bowl. The ladder is
enclosed either in the center column or in a side column (in the case of the ground level storage tank).
The entrance doors to the enclosures were protected by existing contact switches, so motion sensors were
installed for the ladders to provide an added level of security. The ladder motion sensors have both
microwave and infrared sensors; both sensors must be activated to trigger an alarm. The enclosures
typically contain equipment in addition to the ladder, so the motion sensors were installed at least thirty
feet up the ladder to reduce false alarms from floor-level personnel accessing this equipment (i.e., only an
intruder climbing the ladder would be sensed). Both the door contact switch and the ladder motion sensor
alarms are relayed to the operators at the California Control Center. This approach provides two (2)
sources of alarms, which helps to screen-out false alarms arising from the motion sensors and from utility
staff entering the tank base who neglect to call the California Control Center.
September 2008 66
-------
Cincinnati Pilot Post-Implementation System Status
5.2.2 Data Management and Communications
The existing contacts and sensors installed at the storage tanks and reservoirs require minimal data flow
capacity for communication to the California Control Center, and have low data capacity needs for
storage of event records. The data management and communications needs for these systems were
readily accommodated by the existing GCWW SCADA system. However, it was decided that other
alarm signals, video clips and live video from the pump stations would be best transmitted to the
California Control Center via a separate digital cellular network. Thus, a parallel WS SCADA system
was installed for the pilot project to assist in data transmission for both the enhanced security monitoring
component and the water quality monitoring component. Section 3.2.3 provides more details on the
architecture of this parallel WS SCADA system. A HMI was also deployed at the California Control
Center as part of this parallel WS SCADA system to allow the operators to assess and respond to
intrusion alarms for the pump station sites. The SCADA HMI also provided an interface for the operators
to manage the video alarm assessment.
The normal operating mode for Longwateh™ is to transmit single frame images from the cameras at each
monitored facility to the California Control Center at regular intervals, while also retaining continuous
video files in digital video recorders, termed remote video engines. When a security event is indicated by
a tripped door contact switch or a motion sensor signal, the fixed camera remains stationary, while the
PTZ camera focuses on the preset view dedicated to the particular intrusion location. The Longwateh™
video system accesses the video data from the camera covering the intrusion location and assembles a
short video clip of adjustable duration (initially set at ten seconds) beginning at the time of the alarm for
the PTZ cameras and three seconds before the alarm for the fixed cameras. The intrusion event clip is
transmitted to the operators in the California Control Center for assessment.
The Longwateh™ system Video Control Center software at the California Control Center maintains a
record of alarm events and stores attributes of the alarms and any video clips that result. This system
allows the video data to be reviewed after an event for further assessment. A full video record is
maintained in the video recorders located at each facility. Video data from the remote recorders must be
downloaded manually and are intended for use in post-event investigations to support criminal
prosecution and provide a more complete record of the actions of an intruder. More details on the
operation of the Longwateh™ video systems can be found in the Enhanced Security Monitoring Video
Intrusion Alarm Assessment Operations Manual (USEPA, 2007r).
5.2.3 Component Response Procedures
To ensure GCWW timely response to the newly installed enhanced security alarms, EPA in conjunction
with GCWW reviewed the existing response procedures for investigating a security alarm and modified
the procedures based on new security equipment, such as the video cameras. Distinct protocols were
developed depending on the whether the security alarm was from a video or non-video site. The
procedures were documented in the Concept of Operations for the GCWW Contamination Warning
System (USEPA, 2007b). The Concept of Operations guides initial alarm validation actions, defines
specific roles and responsibilities and outlines process and information flows for the enhanced security
monitoring component. It also provides a formalized process that concludes with the determination if an
alarm or a witness account is or is not a "possible" contamination incident. If a possible contamination
incident is identified through the enhanced security monitoring component, the process outlined in the
Concept of Operations would transition the utility personnel to the credibility determination process
outlined in Consequence Management Plan. EPA provided training to GCWW mid-level managers in
June 2006 on the use of the enhanced security monitoring Concept of Operations and the associated
procedures.
The enhanced security monitoring systems that involve video provide for a considerably more rapid
assessment of security breaches at critical pump stations. Without the video, the sole means of assessing
a breach is to dispatch personnel to the remote facility. The Longwateh™ system begins to capture the
video clip the instant a security alarm is tripped, and within two minutes a video clip is received at the
September 2008 67
-------
Cincinnati Pilot Post-Implementation System Status
California Control Center. This allows the operator to verify whether the person entering the station is an
unauthorized intruder and if so, to develop a description and begin to assess the actions of the intruder.
The information will allow security personnel who are dispatched to prepare for response actions and
minimize assessment time when arriving at the site. If the video clip or subsequent video monitoring
clearly confirms a possible contamination event, this information will allow utility personnel to begin
preparation for response actions such as isolation of the facility suspected to have been the target of
tampering.
At the elevated storage tanks, the addition of the ladder motion sensors provides more definitive
information that an intruder has entered the site and has moved towards the finished water supply.
Although a thorough assessment must still be completed by personnel dispatched to the site, the dual
alarm signals provide a more complete record of the possible intrusion, and may communicate a higher
level of urgency to the security team dispatched to the site.
5.2.4 Summary of Post-Implementation Status
Enhanced security monitoring equipment was installed at three (3) pump stations, four (4) finished water
reservoirs, four (4) elevated tanks and one (1) ground storage tank. A digital cellular network was
installed to accommodate data communication needs for both video surveillance equipment and online
water quality monitoring equipment (see Section 3.2.3). Response protocols, which document the
necessary human actions to both existing and newly installed security equipment alarms, were written and
mid-level manager training on these protocols was completed. Table 5-5 provides a summary of the
post-implementation status of the enhanced security monitoring component of the contamination warning
system.
Table 5-5. Enhanced Security Monitoring Component Post-Implementation Status
Design Element
1. Physical Security
Equipment
2. Data Management
and Communications
3. Component
Response Procedures
Description of Installed Component
. Completed an enhanced security assessment report outlining recommended
security improvements.
. Completed engineering designs for security improvements and coordinated
project management activities.
• Installed video cameras, motion sensors, door contact switches and
transmission equipment at 3 pump stations to enable visual identification of a
potential intruder with intent to contaminate water.
. Installed vent covers on 3 ground level storage tanks with liquid level sensors
and hatch contact switches to deter the introduction of contaminates.
. Installed ladder motion sensors on 7 water storage tanks to provide additional
detection of potential intruders climbing towards the finished water.
• Video data are transmitted via a secure digital cellular network system in
conjunction with water quality monitoring data.
• Video data are displayed on a dedicated user interface with menu drive
commands to request additional video clips and remotely control pan-tilt-zoom
cameras.
. Contact alarm data are shown for pump stations and elevated tanks as distinct
alarms, which provides better data for security assessments.
• Developed and trained enhanced security personnel on the Concept of
Operations, which documented security roles and responsibilities for specific
alarm types (video, ladder motion sensor, door contact).
Figure 5-1 provides a summary of the level of effort associated with design and implementation of the
enhanced security monitoring component for the Cincinnati pilot. Enhanced security monitoring
activities, as summarized in Table 5-5, relied on support from EPA, GCWW, and local partners. Early
efforts for the enhanced security monitoring component focused on identification of recommended utility
locations for security enhancements, and coordination of installation of security devices. Data
management and communication activities were coordinated with related efforts to support online water
quality monitoring. The level of effort displayed below reflects those activities specific to the enhanced
security monitoring component.
September 2008
68
-------
Cincinnati Pilot Post-Implementation System Status
S2
ra
0)
fc
ra
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0.1
n Local Partners LOE
l GCWW LOE
1 EPA LOE
Physical Security Equipment
Data Management and
Communications
Component Response
Procedures
Figure 5-1. Level of Effort for Design and Implementation of the Enhanced Security Monitoring
Component (December 2005 - December 2007)
Figure 5-2 presents a summary of the extramural costs associated with design and implementation of the
enhanced security monitoring component for the Cincinnati pilot. As illustrated, the most significant
contractor costs dealt with the procurement and installation of visible security equipment. Equipment
costs included procurement of physical security equipment, such as video cameras, motion sensors, door
contact switches, and transmission equipment. Costs associated with contractor travel were not included
in this calculation.
September 2008
69
-------
Cincinnati Pilot Post-Implementation System Status
-p 1 ,UUU
-pyuu
-poUU
-p/ UU
(A
C $600
ra
(A
3
" $500
1-
c
^_, $400
(A
O
$300
$200 '
$100 •
$- '
• Equipment, Consumables,
and Purchased Services
G Extramural LOE
^KXE^^I
Physical Security Equipment Data Management and Component Response
Communications Procedures
Figure 5-2. Extramural Costs Associated with Design and Implementation of the Enhanced
Security Monitoring Component (December 2005 - December 2007)
Figure 5-3 shows a breakdown of the total installed equipment cost by facility type, based on October
2006 competitive bid pricing for the design improvements identified in the assessment report. The
average installation cost by facility type for the Cincinnati pilot is also illustrated.
September 2008
70
-------
Cincinnati Pilot Post-Implementation System Status
$113,912
( 28%)
D Pump Stations
• Elevated Tanks
D Ground Storage Tanks
v-
$18,772
(5%)
$268,033
(67%)
/ TOTAL = $400,717
/Average Installed Cost by Facility Type
Pump Station: $89,300 per station
Elevated Tanks: $4,700 per tank
Ground Storage Tanks: $28,500 per tank
Figure 5-3. Total Costs by Facility Type Associated with Installation of the Enhanced Security
Monitoring Equipment (October 2006)
Table 5-6 presents a further cost breakdown for typical equipment installed as part of the enhanced
security upgrades. The 2006 figures represent equipment list pricing only, and do not include installation.
Table 5-6. Enhanced Security Monitoring Equipment Unit Cost
Equipment
Fixed Video Camera
Dome Pan-Tilt-Zoom Video Camera
Door Status Switch
Indoor Motion Sensor
Outdoor Motion Sensor (Ladder)
Video Transmission Hardware and Viewing Software
Digital Cellular Modem, Antenna and Cabling
Digital Cell Antenna Amplifier
Programmable Logic Controller (PLC)
PLC Programming Software
Cost
$1,050
$4,050
$260
$450
$1,660
$11,000
$1,200
$300
$2,600
$2,400
Unit
each
each
each
each
each
per site
each
each
each
per project
September 2008
71
-------
Cincinnati Pilot Post-Implementation System Status
Section 6.0: Consumer Complaint Surveillance
Located throughout a utility's distribution network, consumers can provide near real-time input regarding
changes in water characteristics discernable through the senses. Consumers may detect contaminants
with characteristics that impart an odor, taste, or visual change to the drinking water. Complaints from
residential, commercial and industrial consumers are routinely reported to water utilities on a very timely
basis. As such, consumer complaints may provide one of the earliest warnings of a possible
contamination incident, if an effective system is in place to detect anomalous trends in complaints and to
respond quickly. The consumer complaint surveillance component extends beyond just managing
complaints by providing near real-time collection and analysis of call management and work management
system data, along with automated notification of utility personnel when anomalous conditions are
detected.
The consumer complaint surveillance component design objectives are shown in Table 6-1, and were
derived from the overarching performance objectives of the contamination warning system, as described
in WaterSentinel System Architecture (USEPA, 2005a). GCWW's pre-existing capability with respect to
each design element of the consumer complaint surveillance component listed in Table 6-1 is summarized
in Section 6.1 and the utility's capability after implementation of the contamination warning system is
summarized in Section 6.2.
Table 6-1. Consumer Complaint Surveillance Component Design Objectives
Design Element
1. Comprehensive
Complaint Collection
2. Electronic Data
Management
3. Automated and
Integrated Data
Analysis
4. Component
Response Procedures
Design Objective
A "funnel" for collecting all water quality complaints into the consumer complaint
surveillance system. For example, a unified call center with a widely publicized
telephone number in place to capture the largest percentage of potential
complaints. Procedures must also be in place to capture complaints that are
directed to other points inside the utility or that are initially received by other
agencies.
All water quality complaints are entered into an electronic database as they are
received and categorized by type. A complaint record is carried through the process
with information added to it as it is received or investigations are conducted.
As data are captured in electronic format, automated event detection algorithms
indicate when consumer complaint surveillance data reach pre-determined
thresholds, signaling the need for human involvement in the assessment process.
When thresholds are exceeded, notifications are sent to appropriate personnel, and
complaint spatial information is displayed for easy identification of clustering events.
Written standard operating procedures exist for every step in the water quality
complaint handling process and for alarms. These procedures outline effective and
timely communications, including clear guidance on appropriate response actions.
6.1 Pre-lmplementation Status
Prior to implementation of the contamination warning system, GCWW had systems and processes in
place that could be enhanced and used to create an integrated consumer complaint surveillance system.
Data supporting this component resided in call management data systems, customer information data
systems, asset management data systems, and water quality data systems, but were not automatically
aggregated and analyzed and notifications of potential events were not sent. Although call volume reports
were reviewed periodically by supervisors, event detection relied exclusively on staff experience to
recognize trigger events involving customer water quality complaints. Through assessments and
demonstrations of existing GCWW Customer Complaint Management System operations, along with
interviews with GCWW staff, gaps were identified between current operations and a functional consumer
complaint surveillance system. Consumer complaint processes, procedures and data management
systems that existed prior to the Cincinnati pilot implementation are summarized below. Further details
on the assessment of the existing systems can be found in the Consumer Complaint Surveillance
Assessment Report (USEPA, 2006d).
September 2008 72
-------
Cincinnati Pilot Post-Implementation System Status
6.1.1 Comprehensive Complaint Collection
GCWW routinely received consumer calls from residential, commercial, and industrial consumers via a
single, widely published utility phone number. During business hours, customer calls are routed to the
GCWW call center, where a customer service representative from the Commercial Services Division
handles the call. During non-business hours, customer calls are routed to the distribution dispatcher, a
Distribution Division position, who directs and coordinates the work being done on the distribution
system. GCWW used informal call management protocols and operating procedures, but many of the
procedures were not documented.
Some water quality related complaints were received outside of the GCWW Call Center. The City of
Cincinnati's Customer Service and Communications Center received the largest portion of these calls,
while the GCWW Director's office would receive occasional calls directly from City Hall, public health
organizations, the media, and concerned citizens.
6.1.1.1 GCWW Call Center: Consumer Calls during Regular Business Hours
During normal business hours, all calls are directed to individual customer service representatives through
an automated call management system using an interactive voice response (IVR) system. The pre-
existing IVR system script prompted the customer to select a menu option as follows:
"Welcome to Greater Cincinnati Water Works
For automated bill payment using your credit or debit card, press 1;
To speak with a service representative about billing or account information, or if you are moving, press 2;
To speak with a service representative if your water has been turned off or to report a water leak, press 3;
To speak with a service representative about meter reading, press 4;
For recorded directions to the GCWW business office and hours of operation, press 5;
For all other services or information, press 0 to speak to a service representative."
The call management system then routes the call to a customer service representative who has been
trained in that area. The existing IVR system script did not have a menu item specifically for water
quality concerns or complaints. In general, there were three types of water distribution system-related
(non-billing) calls handled by the customer service representatives:
• General Inquires. General inquiries, including questions on basic water parameters like pH and
hardness, were most often answered by the customer service representative. If the customer
service representative could not satisfactorily answer the question, the call was transferred to the
Water Quality and Treatment Division.
• Infrastructure Issues. For distribution system infrastructure issues, such as main breaks and
leaky hydrants, the customer service representative created a "work request" using the
Distribution Work Center application. The work request was then forwarded electronically to the
Distribution Division for review prior to generation of a work order by the dispatcher to begin
field activities.
• Water Quality Issues. Water quality complaints (discoloration, taste, and odor concerns) were
transferred by the customer service representative directly to the Water Quality and Treatment
Division receptionist without generating a work request. During normal business hours the
receptionist transferred the call to the on-duty customer water quality representative to initiate a
response. Customer service representatives that received what they perceived to be an unusual
water quality complaint, or began seeing several water quality issues over a short period of time,
were instructed to inform the call center supervisor. However, it was unlikely that one customer
service representative would receive all water quality complaints due to the large number of
agents fielding calls. Therefore, the Water Quality and Treatment Division receptionist and the
customer water quality representative would likely be the first to recognize a potential water
contamination event. The WQT customer water quality representative would conduct an
investigation in the event that a predefined threshold of water quality calls was exceeded.
September 2008 73
-------
Cincinnati Pilot Post-Implementation System Status
6.1.1.2 Distribution Division Dispatcher: Consumer Calls during Non-Business Hours
The IVR system functioned during non-business hours, but customers are told to call back during normal
business hours for all non-emergency issues. For emergencies, such as water main breaks or water
quality issues, the call is transferred to distribution dispatch because this position is staffed 24 hours a day
and is capable of responding to emergencies. The distribution dispatcher also received emergency calls
directly from other GCWW staff and from the City of Cincinnati's Customer Service and
Communications Center. Non-emergency calls would also be received by the dispatcher, since customers
could select '0' on the IVR system if they desire to talk with someone at GCWW. If so, the dispatcher
evaluated the type and urgency of the call.
The dispatcher is an experienced staff member capable of answering general inquiries and handling
infrastructure issues. However, for water-quality issues, the dispatcher would contact Distribution
Division Senior Management, who would then consult with the Water Quality and Treatment Division
Senior Management to determine whether a work order is needed to investigate the incident. For calls
requiring field action, the dispatcher created a work order directly using the Distribution Work Creation
application, and bypassed the step of generating a work request.
6.1.2 Electronic Data Management
GCWW used call management, customer information and asset management data systems, along with
water quality databases to manage and respond to consumer calls. The IVR system was part of the call
management system, while the generation of work requests and work orders resided in the asset
management systems. Water quality data concerning water quality related complaints, collection of field
samples, and associated analytical results were captured on various paper forms and then transferred to
the water quality database. These data systems are described below.
6.1.2.1 Banner
A customer service software application called Banner was used by the customer service representatives
to link incoming calls to a GCWW account, and acted as the platform for the utility's customer
information system. The IVR software used a table in the Banner database to write the caller's initial
IVR menu choice.
6.1.2.2 Distribution Work Creation
The Distribution Work Creation (DWC) application existed to manage and track the scheduled and
unscheduled work that was being performed in the distribution system. The primary users were
Distribution Division personnel, particularly the distribution dispatcher, and Water Quality and Treatment
personnel. The dispatcher and Water Quality and Treatment personnel would create work orders that
directed crews into the field to perform the necessary work. Customer service representatives were able
to enter the application to create a work request for distribution system infrastructure issues (main breaks,
etc.) or water quality issues related to maintenance work (rusty or cloudy water, etc.). These requests
were reviewed by the distribution dispatcher, triaged, and either converted into a work order or declined,
depending on the dispatcher's assessment of the current conditions. Work requests and work orders were
periodically archived. Water quality work orders were monitored to conduct investigations in the event
that a predefined threshold for water quality was exceeded.
6.1.2.3 Water Quality Database
If a customer was transferred from the customer service representative to the Water Quality and
Treatment Division, additional complaint information was collected over the phone by the customer water
quality representative or shift chemist using a paper form. This information was then entered into the
Water Quality Database (an Excel spreadsheet), which was used to track and record every water quality
complaint received, and the Distribution Work Creation application was used to create a work order, as
described above. To ensure that work orders were not missed, the representative printed a hardcopy and
emailed the Distribution Dispatcher to indicate that a new work order was in the system.
September 2008 74
-------
Cincinnati Pilot Post-Implementation System Status
If a sample was collected from the location of the complaint, the results were recorded in both the Sample
Request Form (hard copy) and the Water Quality Database. Information on the Sample Request Form
was used to complete a notes field in the utility's asset management system, Enterprise Maintenance
Planning and Control, and then the paper form was filed. Finally, a form letter and survey were sent to
the customer as a follow-up.
6.1.3 Automated and Integrated Data Analysis
Event detection systems were developed and deployed to analyze data from the IVR system, work
requests, and work orders. Additionally, some GIS capability was developed to facilitate spatial analysis
of water quality related consumer calls.
6.1.3.1 Event Detection
Detection of anomalous water quality related call volumes relied on manual, experience-based processes
from call center and Water Quality and Distribution staff. As mentioned previously, customer service
representatives were trained to inform the call center supervisor if they received what they perceived to be
an unusual water quality complaint, or began seeing several water quality issues over a short period of
time. However, it was unlikely that the same customer service representative would receive all water
quality complaints due to the large number of agents fielding calls. To address this, the utility captured
the water quality related complaints on paper logs, transferred the information to an Excel worksheet and
manually determined whether a threshold number of calls had been exceeded. If more than four (4) calls
were recorded in the Water Quality Database within a 24-hour period an investigation call type and
spatial patterns was performed, and appropriate Water Quality and Treatment management was notified.
Customer water quality representatives recognizing a trend developing (same complaint area or type)
were, however, trained to alert a supervisor before waiting for the entire 24-hour period.
6.1.3.2 Spatial Analysis
The Distribution Work Creation application provided some GIS ability when creating work requests and
work orders. Although limited in scope, the GIS application allowed the user to view a pre-defined area
around the premises location and showed areas of the distribution system construction in the immediate
vicinity of the work request or work order location.
6.1.4 Component Response Procedures
GCWW has established informal protocols and operating procedures that are not formally documented.
While GCWW employees exhibit an ability to deal effectively and professionally with consumer
complaints, identification of water quality related call volumes may be hindered by the numerous
individuals within the same division or call center that may be involved in processing requests
concurrently. Informal procedures can also break down during emergencies, shift changes, and as
institutional knowledge is lost through personnel retirements, promotions, or job changes.
6.1.5 Summary of Identified Gaps
GCWW's existing call management system and procedures were robust and functioned efficiently for the
addressing routine customer service issues. The existing systems provided a solid foundation to address
the design objectives summarized in Table 6-1. The specific gaps identified for each design element are
summarized in Table 6-2. These gaps provided the design basis for enhancements to the GCWW
consumer complaint system, and the post-implementation status of the resulting system is described in
Section 6.2.
September 2008 75
-------
Cincinnati Pilot Post-Implementation System Status
Table 6-2. Consumer Complaint Surveillance System Gap Analysis
Design Element
Gap Description
1. Comprehensive
Complaint Collection
A defined procedure for transferring water quality related calls from City Call
Center to the GCWW Call Center is needed.
2. Electronic Data
Management
Multiple data streams from call management and asset management systems
should be developed to serve as indicators of possible contamination incidents.
Mechanisms to capture electronic data from the interactive voice response (IVR)
system are needed to provide the earliest indication of possible water quality
problems.
3. Automated and
Integrated Data
Analysis
Automated event detection algorithms are needed to analyze data in near real-
time.
Automated notifications to key utility personnel are needed to alert them when
water quality related consumer complaint thresholds are surpassed.
A GIS user interface is needed to spatially display alarm data.
4. Component
Response Procedures
Standard operating procedures for operating the CCS component are needed
for Customer Service Representatives and Distribution Dispatch personnel.
A specific definition concerning calls that should be categorized as potential
indicators of a significant water quality concern.
Some procedures need altering to produce discrete data streams for the CCS
component.
Training materials are needed for consumer complaint surveillance standard
operating procedures.
6.2 Post-Implementation Status
Modifications to the GCWW consumer complaint surveillance component were based on the "Funnel,
Filter and Focus" model (Figure 6-1). This model provided an efficient process flow and filtering
mechanism for consumer complaint calls, where non-water quality related calls were quickly removed
from consideration.
• Funnel. All customer calls directed to the GCWW call center.
• Filter. GCWW customer service representatives respond to billing, meter reading and general
water quality concerns. Calls with water quality issues based on a specified definition are
forwarded to customer water quality representatives.
• Focus. Customer water quality representatives gather in-depth information from the consumer
and determine whether the situation requires field sampling.
September 2008
76
-------
Cincinnati Pilot Post-Implementation System Status
LU
LU
I-
Primary Source:
Water Utility Call Center
Secondary Source: ^ I * Secondary
Other Agencies ^^^ | ^^ Source:
* Other Utility
Departments
Non Water Quality
Related Calls
CO
^
O
o
Non Water Quality
I Related Calls
' (As determined by
Call Center Staff)
Water Quality Calls
Related to System
Operations
(Main Breaks,
Maintenance, etc.)
Water Quality Specialist
analyzes remaining
complaints for indications
of possible contamination
Figure 6-1. The Filter, Funnel, and Focus Model
6.2.1 Comprehensive Complaint Collection
A "backdoor number" was implemented, with the assistance of the City of Cincinnati Call Center, to
transfer all water quality related calls to the GCWW Call Center. The backdoor number not only transfers
the call from the City Call Center, but also places the caller to the head of the queue for immediate
attention by GCWW customer service representatives. A standardized definition of a water quality issue
was also developed for the City call center personnel to aid in the proper identification of the calls that
should be forwarded to the GCWW Call Center.
6.2.2 Electronic Data Management
In general, data management activity for the consumer complaint surveillance component involves the
automated, near real-time (every 1-minute) collection of customer complaint data from utility call
management and asset management data systems, and transferring and transforming (where necessary)
that data to the event detection software. A summary of the data management systems involved in this
component is provided in Table 6-3.
Table 6-3. Consumer Complaints Surveillance Data Management System Inventory
Data System
Call Management System
(CMS)
Banner
Pre-Existing
System?
Yes
Yes
Description
Includes interface used by customer service representatives and
the interactive voice response (IVR) system
Customer information system with account and address
information
September 2008
77
-------
Cincinnati Pilot Post-Implementation System Status
Data System
Enterprise Maintenance
Planning and Control
(EMPAC) System
Distribution Work Creation
(DWC) System
Hydra
Microsoft Exchange Server
(Email)
WS Simple Mail Transfer
Protocol (SMTP) Server
Hydra Map Display
WS Database Server
WS Application Server
Pre-Existing
System?
Yes
Yes
Yes
Yes
No
No
No
No
Description
Primary GCWW asset/work order system
Interface to EMPAC that creates work requests and work orders,
with limited GIS mapping feature
Web-based interface to EMPAC that creates work requests and
work orders, with more advanced GIS mapping capability.
GCWW is in the process of replacing DWC with Hydra.
GCWW email system used to send alarm notifications to groups
of pre-defined utility personnel
System used to send alarm notifications from the event detection
system to the utility's Microsoft Exchange Server
CIS-based map to spatially display work request and work order
information in near real-time (2-minute updates)
Server responsible for storage of WS-CWS data
Server containing code to process data, including the event
detection algorithms
The consumer complaint surveillance component includes an automated, real-time data collection process
to obtain data from GCWW systems using a web services application, a mechanism for machine-to-
machine interaction over the network application. Once retrieved, the data are stored locally and
analyzed using a variety of statistical event detection algorithms on the Water Security Application
Server. Data management elements of the consumer complaint surveillance component have been
gradually deployed at GCWW, beginning in December 2006. As of April 2007, the major data
management aspects of this component were functional at GCWW, including data collection, data
analysis, and the distribution of notifications. Figure 6-2 illustrates data management systems used by
the component and how they are accessed by utility staff throughout the process.
Customer phones
utility interacts —
with IVR
Call routed
CSR collects to CSR „
WQ complaint data Distribution Dispatch
from customer
WQ&T Chemist col
detailed WQ complaii
from consi
ler x
llects N. /r4\
mtdata T^M K
7 M\
CSR or Distribution
Dispatcher
GCWW WQST Dl
Appropriate GCWW
Supervisor for
Investigation
Figure 6-2. Information Flow for the Consumer Complaints Surveillance Component
September 2008
78
-------
Cincinnati Pilot Post-Implementation System Status
The consumer complaint surveillance component has implemented three "tiers" of alarms, based on the
following discrete data streams:
1. Interactive Voice Response. GCWW customers, when placing a call to the utility, have the
opportunity to indicate that their call relates to "water quality information, questions or
concerns." By selecting this newly added option in the IVR system, calls are flagged as relating
to water quality and are moved to the head of the call center queue.
2. Water Quality Work Requests. All calls that are not general inquiries or do not relate to rusty
or cloudy water are termed "water quality requests." For these calls, the customer service
representative, or the distribution dispatcher after hours, generates a work request in the Hydra
work management interface. All such work requests are tracked through the process and can
trigger an alarm when a threshold value is reached.
3. Water Quality Work Orders. Water quality work requests can be converted into work orders,
which direct field personnel to collect water samples as part of a response to a suspected water
quality problem. Work orders, like work requests, are tracked through the process and can trigger
an alarm when a threshold value is reached.
6.2.2.1 Interactive Voice Response
The interactive voice response (IVR) menu was modified to add the following option for water quality
related concerns (note that the former option #5 was moved to #6):
"For water quality information, questions, or concerns, press 5 to speak with a service representative"
Addition of this option created the data stream necessary for analysis by the event detection algorithms.
Callers who push #5 are also forwarded to the top of the call queue for immediate attention by the next
available customer service representative.
Because a selection in the IVR system represents the first contact between the customer and the water
utility, it is the timeliest indication of a possible contamination incident. However, the IVR system does
not provide any complaint description or location information. The IVR web services client collects a
count of water quality (#5) selections, aggregates the selections, applies a date/time stamp to each, and
exposes the data for retrieval by the Water Security initiative web service client. This data are retrieved
from the call management system. The Water Security initiative web services client queries the latest
IVR selection data from the web service every minute. Additionally, the web services client collects all
IVR selection data once per day. This complete data set is collected for the purpose of EPA evaluation
and is not analyzed by an event detection algorithm.
6.2.2.2 Water Quality Work Requests
In 2007, GCWW upgraded its distribution system work management software (Distribution Work
Creation) and renamed it "Hydra." One function included in the upgrade was the ability to classify work
requests according to "problem type." Now a work request can be generated in Hydra by a customer
service representative or the distribution dispatcher and classified as one of three new problem types that
relate to potential water quality problems ("rusty water", "cloudy water" and "water quality request").
The creation of such a work request electronically captures additional information regarding the
customer's complaint, including its location, and indicates the likely need for a sample to be collected in
the field to investigate a possible water quality problem. Work requests classified as such have been
made available via web services to the consumer complaint surveillance event detection algorithms since
March 2007. Currently, only work requests classified as a "water quality request" are analyzed by the
consumer complaint surveillance event detection algorithms.
September 2008 79
-------
Cincinnati Pilot Post-Implementation System Status
A work request web services client collects counts of work requests generated in the Distribution Work
Creation/Hydra application, aggregates them, applies a date/time stamp to each, and exposes the data for
retrieval by the Water Security initiative web service client. The work request data are retrieved from the
EMPAC system via the Hydra interface. The work request web services client queries the latest work
request data from the work request web service every minute.
6.2.2.3 Water Quality Work Orders
Each water quality work request is reviewed by staff in the Water Quality and Treatment Division, and is
converted to a work order if a water quality specialist confirms the need for sampling or other field
investigation. Using the work request already generated by the customer service representative, the Water
Quality and Treatment representative or distribution dispatcher creates a work order in EMPAC, using
Hydra, to initiate the field sample process. The conversion of the work request to a work order signals an
even greater likelihood that the complaint is related to a water quality issue because the call has been
reviewed by a water quality specialist. This data stream is therefore the most reliable, but the time
required to gain this confidence and to perform this review makes it the least timely indication of a
possible contamination event.
As with work requests, a work order web services client collects counts of work orders generated in
Hydra, aggregates them, applies a date/time stamp to each, and exposes the data for retrieval by the Water
Security initiative web service client. The work order data are retrieved from the EMPAC system via the
Hydra interface. The work order web services client queries the latest work order data from the work
order web service every minute. Before March 2007, GCWW categorized work orders related to water
quality as "taste and odor complaint." Since then, GCWW has been categorizing work orders related to
water quality as "water quality request" and analyzing them with the consumer complaint surveillance
event detection algorithms.
6.2.3 Automated and Integrated Data Analysis
All consumer complaint surveillance data are captured in a consistent electronic database format to
facilitate access to the data and the use of automated analysis tools. Automated anomaly detection
algorithms detect when critical indicators of a potential contamination event reach pre-determined
thresholds. Human involvement in the assessment process is then initiated. Quantitative triggers are
based on analysis of historic data and statistical analysis. The use of pre-determined threshold levels is
advantageous in that they can be adjusted as more data becomes available. Additionally, the triggers can
be changed depending on the threat level intelligence. If the threat level is higher, fewer calls might be
needed to trigger an alarm and begin an investigation. Further details regarding the functional and
technical specifications for the data implementation elements of CCS improvements can be found in the
Water Security Contamination Warning System, Consumer Complaint Surveillance, Detailed Design
Specification, Version 2.0 (USEPA, 2007s).
6.2.3.1 Event Detection
The event detection system is more appropriately thought of as a collection of algorithms which analyze
data collected from the utility's source databases. The event detection system automates the data analysis
and alarm process based on pre-established frequencies of water quality related calls, work requests and
work orders. The automated event detection system provides a dependable, robust surveillance system
that is not affected by human errors that may occur in maintaining continuity of observations across the
large number of staff involved in the process and transferring information across shift changes. Once an
alarm is triggered, a notification is generated and sent to appropriate personnel, bringing in the human
element for the subsequent investigation actions.
There are five (5) algorithms applied to three (3) data streams (the numbers of IVR #5 selections, water
quality related work requests, and water quality related work orders), for a total of fifteen (15) potential
triggers. Because some algorithms are designed to operate only on weekends or only on weekdays, not
all triggers are active at the same time. The algorithms are described in Table 6-4.
September 2008 80
-------
Cincinnati Pilot Post-Implementation System Status
Table 6-4. Event Detection Systems Deployed for Consumer Complaint Surveillance
Algorithm
Scan Statistics, 1 day weekday
Scan Statistics, 2 day
Scan Statistics, 7 day
Scan Statistics, 1 day weekend
CUSUM
Description
A prospective scan statistic monitors current data and evaluates it
against past data in a time window. Separate trigger, or threshold,
levels were established for 1, 2 and 7 days.
A separate scan statistic is applied to weekend and holiday consumer
complaint data, where call volumes drop significantly. Separate
threshold, levels were developed for these days.
A cumulated sum (CUSUM) accumulates the difference between an
observed number of complaints per day and a reference value. If the
accumulation exceeds the trigger, and alarm is given. Because the
observations can accumulate overtime, the CUSUM method can
detect slowly worsening situations earlier than a single day trigger.
Initial trigger threshold values were developed based on statistical analyses using existing historical data.
The final selection of the trigger values also took into account the risk position and the level of effort
required to investigate alarms. During baseline assessment, the trigger thresholds are set artificially low
to facilitate fine tuning of the email and text notification systems (discussed below) by EPA personnel.
Once GCWW enters the full deployment phase, revised trigger thresholds will be adopted based on the
data collected during the baseline assessment phase, and notifications will be sent to GCWW users.
Trigger parameters, such as algorithm thresholds, are implemented within text files separate from the core
code of the algorithms themselves. This programming structure allows the utility to fine-tune trigger
parameters without accessing and re-writing complicated computer code or disrupting the central
functioning of the automated algorithms.
6.2.3.2 Notifications
If a trigger level is not reached, an alarm will not be generated. The event detection algorithms will
continue to monitor the data streams. Once the trigger level has been reached, however, an electronic
alarm will be generated and notification of the appropriate GCWW staff will follow through email and
cell phone text message. The consumer complaint surveillance component harnessed the existing
GCWW Microsoft Exchange server to deliver email notifications of alarms generated by the new event
detection system. Using defined email groups, targeted notification emails for each type of consumer
complaint surveillance alarm are sent to the utility personnel responsible for further investigation of that
data stream. Since GCWW's mobile workforce will not always have access to email, text messaging was
also incorporated as part of a redundant notification strategy. Email notifications are divided into four
general groups: business hours, business hours (cell format), after hours, and after hours (cell format). An
example email alarm notification is included in Figure 6-3.
September 2008
81
-------
Cincinnati Pilot Post-Implementation System Status
Interactive Voice Response Alarm (Water Quality Selection by Consumer High)
Business Hours
Subject: CCS ALARM NOTICE: Water Quality Selection by Consumer High
Body:
Over the past the threshold number () of water quality consumer self-
identified calls in the Interactive Voice Response (IVR) system has been surpassed.
ALARM DETAILS
Alarm ID:
The algorithm is responsible for triggering this notification.
The customer service representative supervisor should:
1) Check on the call waiting queue time
2) Discuss the nature of the calls with customer service representatives or the distribution
dispatcher
3) Assess the need for additional personnel to handle the increased call volume
4) Inform the Utility Director's office and City's call center supervisor of increased water quality
call volume (if appropriate)
5) Inform the customer water quality representative supervisor (Water Quality and Treatment
Division) of increasing water quality call volume and nature (if appropriate)
Figure 6-3. Example Email Alarm Notification
6.2.3.3 Spatial Analysis
To quickly determine whether there is a geographic correlation within a consumer complaint surveillance
alarm, an internet map service display, referred to as the Hydra map, was built using a custom GIS web
application. A separate web services application, similar to the one developed to collect interactive voice
response, work request, and work order data, is used to transfer location information from the work orders
and work requests. The Hydra map then displays these in near real-time (2-minutes) so that spatial
clustering of complaints can more easily be identified. Note that the IVR data cannot be analyzed
spatially, since there is no location information associated with the calls. An example Hydra map screen
shot is shown in Figure 6-4.
September 2008 82
-------
Cincinnati Pilot Post-Implementation System Status
Figure 6-4. GCWW Hydra Map - Detailed Map with Alarm Data and Work Orders and Work
Requests
6.2.4 Component Response Procedures
The integration of the consumer complaint surveillance component with GCWW's call management
system required additional procedures and protocols to direct human response to a notification from the
consumer complaint surveillance component. A detailed Concept of Operations was developed to cover
the routine operations of the component leading up to and after issuance of an alarm notification. It
describes the process and procedures involved in the operation of the consumer complaint surveillance
component, including the initial investigation and validation of an alarm. The Concept of Operations also
establishes specific roles and responsibilities, and detailed procedural and information flow descriptions
(see Figure 6-2). In addition, a series of trigger validation checklists are included in the Concept of
Operations to support the investigation of consumer complaint alarms. This process concludes with the
determination of whether or not an alarm generated from several water quality consumer complaints is
indicative of a possible contamination incident.
Further details on the Concept of Operations protocols can be found in the Concept of Operations for the
GCWW Contamination Warning System (USEPA, 2007b).
Initial training of utility staff, in particular with respect to their roles and responsibilities during the
investigation of a possible contamination event, was needed to prepare for response to such an event.
Training was conducted by EPA for front line and supervisory GCWW personnel involved in the routine
operation of the consumer complaint surveillance component. These trainings were based on the
consumer complaint surveillance section of the Concept of Operations for the GCWW Contamination
Warning System (USEPA, 2007b).
September 2008
83
-------
Cincinnati Pilot Post-Implementation System Status
Future training to maintain the capabilities of the consumer complaint surveillance component will be
integrated with the existing GCWW training program for new personnel in the call center. If new
protocols and procedures are developed for the consumer complaint surveillance component, the
customer service representatives and supervisors, distribution dispatchers and customer water quality
representatives from the Water Quality and Treatment Division would be involved in their review and
development.
6.2.5 Summary of Post-Implementation Status
EPA, with the assistance of GCWW, completed a comprehensive assessment of the consumer complaint
call management, customer information, asset management and water quality data systems in place at
GCWW. During this assessment, the existing practices and systems were compared to an ideal consumer
complaint surveillance concept (Table 6-1) which could assist a utility in the early detection of a
contaminant event. Following the completion of the assessment, gaps were identified where EPA and
GCWW believed changes or additions to the practices and systems in place at the utility to process
consumer complaint calls could be improved (Table 6-2). EPA and GCWW worked collaboratively to
develop and implement solutions which would address the identified gaps. A summary of the installed
consumer complaint surveillance is presented in Table 6-5.
Table 6-5. Consumer Complaint Surveillance System Post-Implementation Status
Design Element
Description of Installed Component
1. Comprehensive
Complaint Collection
Completed the CCS assessment report to identify and prioritize needed
improvements.
Established a "back door" number for the transfer and prioritization of consumer
water quality complaint calls from the City wide call number to GCWW.
2. Electronic Data
Management
Modified the interactive voice response (IVR) system to include water quality
concern as a selection option.
Established a water quality complaint categorization in the utility's work request
and work order form that can be tracked.
Included a problem type characterization field in the work order form to establish
a uniform description of a water quality complaint to assist in the determination
of a possible contamination event.
3. Automated and
Integrated Data
Analysis
Completed historic assessment of water quality related calls to set initial alarm
trigger levels.
Development, testing and deployment of automated event detection algorithms
to count the number of water quality related interactive voice response (IVR)
selections, work request and work orders.
Establish a method for automatic notification of key utility personnel when water
quality consumer complaint thresholds were surpassed.
Worked jointly with GCWW to ensure consumer complaint alarm data are
displayed on the utility wide geographic information system to allow a faster
determination of geographically related water quality problems.
4. Component
Response Procedures
Developed and trained GCWW on the Concept of Operations for the CCS
component which included:
- The standardization of a water quality complaint definition.
- Modified procedures for customer service representatives and
distribution dispatchers to generate work requests for water quality
related complaints.
- Checklist to assist the utility in determining whether an alarm is a
possible contamination event.
Figure 6-5 provides a summary of the level of effort associated with design and implementation of the
consumer complaint surveillance component for the Cincinnati pilot. Consumer complaint surveillance
activities, as summarized in Table 6-5, relied on support from EPA, GCWW, and local partners. The
most significant effort was associated with incident response procedures for this component, which
entailed standardization of a water quality complaint definition and modification of response procedures
for utility customer service representatives and distribution dispatchers. A considerable effort as also
associated with establishment of a "back-door" number for the transfer and prioritization of consumer
September 2008
84
-------
Cincinnati Pilot Post-Implementation System Status
water quality complaint calls from the City wide call number to GCWW - part of the comprehensive
complaint collection design element.
(A
ra
n Local Partners
Labor
• GCWW LOE
D EPA LOE
Comprehensive
Complaint Collection
Electronic Data
Management
Automated and
Integrated Data
Analysis
Component Response
Procedures
Figure 6-5. Level of Effort for Design and Implementation of the Consumer Complaint
Surveillance Component (December 2005 - December 2007)
Figure 6-6 presents a summary of the extramural costs associated with design and implementation of the
consumer complaint surveillance component for the Cincinnati pilot. It is important to note that there
were not significant equipment or consumable costs associated with this component due to the fact that
existing software and hardware systems employed by GCWW and the City of Cincinnati were leveraged
for data collection, processing, and analysis. Costs associated with contractor travel were not included in
this calculation.
September 2008
85
-------
Cincinnati Pilot Post-Implementation System Status
(/) $120 "
TJ
C
ra
0
.G
H>
o
• Equipr
and P
n Fvtrar.
nent, Consumables,
jrchased Services
lural LOE
P
Comprehensive Complaint Electronic Data Management Automated and Integrated Component Response
Collection Data Analysis Procedures
Figure 6-6. Extramural Costs Associated with Design and Implementation of the Consumer
Complaint Surveillance Component (December 2005 - December 2007)
September 2008
86
-------
Cincinnati Pilot Post-Implementation System Status
Section 7.0: Public Health Surveillance
Public health surveillance is included as a contamination warning system component due to the tools,
procedures, and lessons learned from the public health sector that can be leveraged to enhance detection
capabilities for a broad range of contaminants. Lessons learned from recent outbreaks of waterborne
disease, including the 1993 outbreak of cryptosporidiosis in Milwaukee, and an outbreak in Walkerton,
Ontario, in May of 2000 caused by E. coli, illustrate how the integration of environmental, health care,
and other types of data can provide earlier warning or more robust validation of public health issues than
clinical signs and symptoms alone (Foldy, 2004; Hrudey, 2002).
As defined by the Centers for Disease Control and Prevention (CDC), public health surveillance is the
ongoing and systematic collection, analysis, interpretation, and dissemination of data about a health-
related event for use in public health action to reduce morbidity and mortality and to improve health
(German, 2001). Syndromic surveillance is a specific type of public health surveillance that relies on
electronic data such as 911 calls, emergency room visits, emergency medical service logs, over-the-
counter medication sales, laboratory test orders, workplace or school absenteeism, and other types of data
that may be available in the early stages of an outbreak. Syndromic surveillance systems seek to use
existing health data in real-time to identify changes in community health status, facilitating notification to
those charged with investigation and follow-up of a potential public health issue (Henning, 2004).
In the Cincinnati pilot, the following public health surveillance data streams are leveraged as part of the
contamination warning system: 911 calls, emergency medical service logs, over-the-counter drug sales,
Poison Control Center calls, emergency room chief complaints, and infectious disease reporting systems.
If an alarm is generated through one of these systems and it appears that the alarm indicates possible
water contamination, the local health departments work collaboratively with GCWW utility staff to
conduct an investigation to determine whether or not the public health issue is related to an actual
drinking water contamination.
The public health surveillance component design objectives are shown in Table 7-1 and were derived
from the overarching performance objectives of the contamination warning system as described in
WaterSentinel System Architecture (USEPA, 2005a). Cincinnati's pre-existing capability with respect to
each attribute of the public health surveillance component listed in Table 7-1 is summarized in Section
7.1, and the post-implementation capability of the contamination warning system is summarized in
Section 7.2.
Table 7-1. Public Health Surveillance Component Design Objectives
Attribute
Design Objective
1. Public Health Data
Streams
Assess existing public health surveillance data streams and modify or enhance them to
meet the objectives of the contamination warning system. This may include expanding
notification of alerts to the local drinking water utility or modification of algorithms to
include symptoms related to exposure to contaminated drinking water. Also, identify
approaches for detection of fast-acting contaminants such as review or tracking of 911
calls, emergency medical service logs, or Poison Control Center calls. The spatial
coverage of the public health surveillance data streams relative to the service area of the
water utility should also be considered.
2. Communication and
Coordination
Develop a mechanism and protocol for communication and coordination between the
appropriate local public health organizations and the drinking water utility. This may
include establishing a dedicated workgroup for the contamination warning system or
integrating the utility into existing public health groups and forums. The local public health
departments and drinking water utility should also establish procedures, roles, and
responsibilities for investigation of alarms generated through the public health surveillance
data streams.
September 2008 87
-------
Cincinnati Pilot Post-Implementation System Status
7.1 Pre-lmplementation Status
Through pre-existing programs and initiatives, the City of Cincinnati had several public health
surveillance data streams in place that were leveraged as part of the contamination warning system pilot.
However, these data streams were not optimized or comprehensively integrated in a manner to support the
contamination warning system objectives. Furthermore, communication and coordination between the
local health departments and GCWW was inconsistent and not well defined beyond a few specific issues,
such as the utility's Cryptosporidium Action Plan.
7.1.1 Public Health Data Streams
As indicated previously, a number of public health surveillance data streams existed in the Cincinnati
metropolitan area prior to implementation of the WS pilot. A description of the existing public health
surveillance systems along with their pre-implementation status is summarized below.
• 911 calls. The City of Cincinnati captures 911 calls electronically through a Computer Aided
Dispatch system. The 911 call center receives over one million calls a year that are routed
initially to the police department; those calls identified as medical emergencies are subsequently
routed to the Cincinnati Fire Department for triage and response. To assist with the triage of
medical emergency calls, dispatchers used paper guidecards based on procedures described in the
Association of Public Safety Communications Officials Fire Service Dispatch Guidecards.
• Emergency medical service logs. The Cincinnati Fire Department collects information using a
computerized tablet system and software that is compliant with Ohio state reporting standards for
emergency medical billing. The tablets were procured under an Urban Area Security Initiative
grant - a homeland security grant designed to improve activities during a weapon of mass
destruction incident. Patient information captured includes patient age, gender, vital signs, chief
medical complaint, medical observations made by the emergency medical technician or
paramedic, medication, and incident zip code. Data was manually uploaded to the centralized
Cincinnati Fire Department server on a daily basis and were used for training, patient tracking,
billing, reporting requirements, and quality assurance.
• Poison Control Center calls. The Drug and Poison Information Center of the Cincinnati
Children's Medical Center is a 24-hour emergency and technical information telephone service
resource for use by the public regarding concerns involving drugs or poisons. A specially trained
staff of pharmacists, pharmacologists, nurses, paramedics, and students within related medical
professions answer questions about poisonings, environmental contaminants and drugs, including,
drug abuse, product contents, substance identification, and adverse reactions. Toxicall®, a
specialized medical database, is used to capture information received on the 24-hour hotline. In
addition, an existing contract between Drug and Poison Information Center and the Southwest
Ohio Public Health Departments, provided a mechanism for evening, weekend and holiday
infectious disease reporting. Under this contract, protocols existed for reporting potential food or
waterborne outbreaks, notification of public health officials to a potential biological terrorist
incident, reporting of unusual disease reporting, and physician consultation.
• Over-the-counter drug sales. Local public health officials receive over-the-counter drug sale
data through the National Retail Drug Monitor. Through this tool, retail pharmacy, grocery, and
mass merchandise operations provide certain over-the-counter sales data which are aggregated
into eighteen (18) product categories, analyzed, and displayed via a secure website. Users are
able to view timeline sales by product category or geographically by zip code on a map. Regions
with unusually high sales are 'flagged' using wavelet time series prediction models. Automated
alerts are not generated and the number of participating retailers is proprietary, thus the exact
population coverage is unknown.
• Emergency room chief complaints. Greater Cincinnati area hospitals provide healthcare
registration data in real-time through the Real-Time Outbreak and Disease Surveillance system.
Specific data provided include age, gender, home zip code, date/time of admission, and a free-text
chief complaint of the patient. A natural language processing program is used to classify the
chief complaint into one of eight (8) categories: gastrointestinal, constitutional, respiratory, rash,
hemorrhagic, botulinic, or neurological. Four (4) algorithms (moving average, recursive least
September 2008 88
-------
Cincinnati Pilot Post-Implementation System Status
square, wavelet, cumulative sum calculation (CUSUM) with weighted moving average) are used
to determine deviations from an established baseline level of emergency department visits. Alerts
to Ohio Department of Health are automatically generated when the threshold is four standard
deviations above the mean. Once received by the Ohio Department of Health, alerts are
forwarded to the appropriate health jurisdiction for investigation. This system, along with the
National Retail Data Monitor, is financially supported by the Ohio Department of Health through
use of public health infrastructure funds.
• Infectious Disease Reporting. Mechanisms for standard infectious disease reporting to the Ohio
Department of Health and/or the CDC are in place for notifiable diseases such as acquired
immune deficiency syndrome, anthrax, botulism, cholera, and others (CDC, 2007). Reporting of
these diseases typically occurs after receipt of diagnostic test results and while more reliable, is
not as timely of an indicator as the syndromic surveillance tools.
7.1.2 Communication and Coordination
Thirteen (13) health jurisdictions are included in the GCWW service area with the primary jurisdictions
serviced by GCWW being the City of Cincinnati and the Hamilton County Public Health. In
collaboration with Cincinnati Health Department and Hamilton County Public Health, GCWW developed
a Cryptosporidium Action Plan that described roles and responsibilities and general protocols in the event
of a Cryptosporidium outbreak that could be associated with contaminated drinking water. However,
there was no mechanism in place for routine communication and coordination between GCWW and the
health departments for the jurisdictions within GCWW's service area.
Several public health committees with responsibilities related to surveillance activities, outbreaks, and
terrorism preparedness existed prior to implementation of the pilot. The most relevant committees are
those established in response to a Public Health Infrastructure grant awarded to the State of Ohio from the
CDC. A Health Commissioners Executive Steering Committee was created to manage and discuss
requirements for deliverables related to the grant and established the following sub-committees:
• Regional Epidemiologists and Disease Investigators. Responsible for epidemiological and
disease investigation grant requirements. This group developed a protocol, implemented by local
health jurisdictions, for responding to infectious disease outbreaks.
• Environmental Surety. Formed to focus on the response to West Nile Virus and other
environmentally-related outbreaks.
• Bioterrorism Coordinators. Consists of both regional and local bioterrorism coordinators.
Their primary focus is to coordinate regional and local response to a bioterrorism event and to
ensure that response plans were in place.
• Public Health Information Officers. Responsible for creating talking points and message maps
related to specific outbreaks or public health crises and collaborating with the Joint Information
Center during an event.
GCWW was not a member or active participant in any of these committees.
7.1.3 Summary of Identified Gaps
The existing public health surveillance data streams and infrastructure provided a solid foundation for this
component of the contamination warning system pilot. However, a number of enhancements and
modifications were needed to fully develop and/or optimize the data streams and communication and
coordination protocols to meet the design objectives described in Table 7-1. Specifically, the public
health surveillance data streams that could provide early indication of drinking water contamination with
a fast-acting contaminant (911 calls, emergency medical service logs, and Poison Control Center calls)
were not fully optimized for timely detection of contamination incidents or the degree of automation
necessary for near real-time detection. In addition, the lack of a consistent and reliable mechanism for
communication and coordination with local health departments presented a challenge in terms of defining
roles and responsibilities and developing a protocol to investigate alarms generated through the public
health surveillance data streams. The specific gaps identified for each attribute are summarized in Table
September 2008 89
-------
Cincinnati Pilot Post-Implementation System Status
7-2. These gaps provided the design basis for enhancements to public health surveillance, and the post-
implementation status of the resulting system is described in Section 7-2.
Table 7-2. Public Health Surveillance Gap Analysis
Attribute
Description of Gap
1. Public Health Data
Streams
911 calls: Although calls were captured electronically, information was gathered
using a manual triage process resulting in a data set that was not standardized and
difficult to analyze for trends. There was no mechanism for automated or real-time
analysis of the data.
Emergency medical service logs: Patient information was recorded using
standardized software; however, manual uploads were required to transfer data to a
centralized location thereby reducing the timeliness of any analyses. There was no
mechanism for automated or real-time analysis of the data.
Poison Control Center calls: The Drug and Poison Information Center captured
standardized data electronically on a 24/7/365 basis. There was no defined approach
for filtering and analyzing data that could be indicative of drinking water
contamination.
Emergency room chief complaints: An automated mechanism for simultaneous
notification of local public health departments and GCWW utility staff did not exist.
2. Communication and
Coordination
No reliable link or consistent mechanism for data sharing had been established
between GCWW and local public health partners.
7.2 Post-Implementation Status
The design and implementation of the public health surveillance component for the contamination
warning system in Cincinnati focused on two primary objectives 1) enhancing data streams for the
detection of fast-acting contaminants and 2) improving communication and coordination between
GCWW and the local public health community. In order to enhance the 911 calls and emergency medical
service logs to meet the contamination warning system objectives, a significant development effort that
required support from multiple City partners was necessary. Modifications to mechanisms for analyzing
Poison Control Center calls and enhancing the notifications for emergency room chief complaints were
more modest in scope and effort. The following subsections describe the post-implementation status of
the public health data streams and communication and coordination between GCWW and the local public
health community.
7.2.1 Public Health Data Streams
Design and implementation activities related to the public health surveillance data streams focused on
enhancements and development of 911 calls and emergency medical service logs into data streams that
could support the contamination warning system objectives for detection of fast-acting contaminants.
Collaboration with the local Poison Control Center provided the opportunity to implement modifications
to protocols for processing and analyzing these calls to support the contamination warning system
objectives as well. Based on the assessment of the emergency room chief complaints data stream, the
only modifications made through the Cincinnati pilot were related to notifications as discussed in Section
7.2.2.
Public Health Surveillance User Interface
The use of 911 calls and emergency medical service logs as a surveillance tool was a new application for
these data streams. In order to support the analysis of data by local public health partners and the utility,
it was necessary to develop a Public Health User Interface to display data and results. The Public Health
Surveillance User Interface, deployed on Cincinnati's wide area network, contains three sources of data
for event investigation: Early Aberration Reporting System (EARS) Results Summary, SaTScan™
Results Summary, EARS Data, and WSDR Data. As illustrated in Figure 7-1, the initial page of the
Public Health Surveillance User Interface includes a combined EARS Results and SaTScan™ Results
Summary page.
September 2008 90
-------
Cincinnati Pilot Post-Implementation System Status
3 PHS Message Viewer - Microsoft Internet Explorer
File Edit View Favorites Tools Help
* \*\ 0\ "j P Search ^ Favorites
€? [.:> 5. H - D
Sddross|Bhttp:H2-g™H-raappl.g™w.ds.rK.org/PHS_Analvsis/ears.do;isessionid-acl2293dce6659dfd21d53c40adbl76819cfOba791e.e3BMcheRbhuQaOOMa38KbxOQaNaRe6f!nA5Pp7ftdbGmkTy _J Q Go
• City of Cincinnati
EARS Data WSDRData EARS Results
02/08/200801:16 -.,„,„
EARS Summary
rM
Location Syndrome Date C1 C2 C3
45213 neurons 02/08/2008 100
45219 neurons 02/08/2008 001
45226 gicat 02/08/2008 111
5226 mater 02/08/2008 111
5202 cardiaccat 02/07/2008 100
5204 unexplained 02/07/2008 100
5204 upperresp 02/07/2003 Oil
5213 unexplained 02/07/2008 111
5217 upperresp 02/07/2008 111
5219 neurons 02/07/2008 100
5223 cardiaccat 02/07/2008 Oil
45227 poison 02/07/2008 111
5227 water 02/07/2008 111
5229 unexplained 02/07/2008 100
5230 gicat 02/07/2008 100
5232 neurons 02/07/2008 111
5234 gicat 02/07/2008 111
5234 water 02/07/2008 111
5238 gicat 02/07/2008 100
5238 psychcat 02/07/2008 111
45238 upperresp 02/07/2008 100
/stem
Help
02/08/200S 01:31 PM SalScan Summary
Event H Mas Ct Lit 1 Lug Start Date End Date Radios P-Valae Observed Cases Deserved ,'Btptaed
1 2 39. 1328 / -84. 5356 02/08/2008 02/08/2008 1.32 0.82900 9 3.21
1 3
1 3
1 1
1 1
3 3
1 2
1 2
_
1 1
3 3
2 3
1 2
1 1
1 1
1 1
1 2
1 2
1 2
* Please see the Help link for hints and explanation of the content above *
^J Done
*j Local intranet
Jn|_x|
;••
Links
A
d
ff
Figure 7-1. Public Health Surveillance User Interface Summary Screen
This summary page provides a table of the latest analysis results, color-coded according to severity for
each data source; the color code corresponds to that established by the Department of Homeland Security
Advisory System (Department of Homeland Security, 2006). According to the current business rules,
only conditions that merit a "red threat level" for each data source that also applies to the entire City of
Cincinnati would initiate an email alert notification to the appropriate local public health partners and
utility staff. The EARS Data page provides access to multiple Excel® worksheets that collectively
contain a detailed alert table by syndrome category and zip code as well as time-series graphs of the
syndromes. The WSDR Data page provides a user link to the de-identified patient information as
obtained from the EMS run. The Water Security Initiative Cincinnati Pilot Public Health Surveillance
User Interface Guide provides a detailed explanation for navigation of the user interface (USEPA, 2007t).
911 Calls
To improve standardization and electronic call tracking for 911 calls a commercial software package,
Priority Dispatch ProQA software was deployed. This software package assists 911 dispatchers in
effective triage of calls by gathering a variety of health data in a systematic manner. Data elements
include chief complaint and geographic location (latitude / longitude) of the call. The caller initiates the
complaint (i.e., breathing problems) and more specific information is attained through the questions asked
September 2008
91
-------
Cincinnati Pilot Post-Implementation System Status
using prompts provided by the software program. In addition, instructions are provided to the caller until
medical assistance arrives.
Cincinnati Fire Department began using the ProQA 'card' set in December 2006 to triage all medical 911
calls and implemented the software package to support this analysis upon installation of the City's new
Computer Assisted Dispatch software. This Priority Dispatch ProQA software directs the call-taker
through a series of questions for the purpose of triage and incident coding.
In order to meet the objectives of the contamination warning system, the 'Incident Type' field was
identified as the most appropriate filter for the 911 call data analysis. Priority Dispatch ProQA contains
366 unique incident type codes, i.e., caller's chief complaint. A review of all potential incident types by
the User Group (see Section 7.2.2) identified 107 that are relevant for WS 911 data analysis. Incident
types include symptoms such as abdominal pain, breathing problems, cardiac or respiratory alert, and
seizures as well as certain events such as chemical spill and carbon monoxide alert that can be used to
quickly rule out drinking water contamination as a potential cause.
Figure 7-2 illustrates the information flow for the 911 calls data stream. The 911 dispatcher uses the
Priority Dispatch ProQA software to document information, including the incident type. Select data are
then exported from the Computer Aided Dispatch server to a database table developed by the Cincinnati
Fire Department. Java 2 Platform, Enterprise Edition (J2EE)-compliant software components were
developed to acquire and reformat the table row contents uniquely and evaluate the data against the 107
selected incident types. Only those records that contain one of the 107 incident types are transferred to
the WSDR, hosted at GCWW, for analysis. Additional Data Access components retrieve the last 21 days
of 911 call data from the WSDR in reverse chronological order on an hourly basis to populate input files
for SaTScan™ analysis. Once the latest data set is retrieved, SaTScan™ is queried to provide the latest
analysis results. Upon analysis completion, data display and alerting tools publish a new analysis
summary to the Public Health User Interface (discussed below) for user review. Through an automated
process, analysis results are compared against business rules to determine if there is a deviation from the
baseline and users should be notified. If such conditions exist, an email notification is sent to public
health partners including the Cincinnati Health Department and Hamilton County Public Health and
GCWW as a prompt for further investigation in accordance with the procedures defined in the Concept of
Operations for the GCWW Contamination Warning System (USEPA, 2007b).
911 Operator
"Receives Call"
Hs°atah I 911 Ca"record uP'oaded to CFD Server —'J.
WUlRM
WUERM and LPH
Investigate alert
WS Application Server
Subscription Filtering
Filtered data sent to
SatScan
SatScan Analysis
CFD WS Server
Data filtered for
HIPAAand copied into
911/EMS Database
ri Alert
T
. _ _ _ _ .Data transformed into XML
911 Results
displayed on
PHS User Interface
Filtered Data
Sent to WSDR"
Local
Public
Health
CFD
WSDR Oracle DB
Server
Figure 7-2. Information Flow for 911 Calls
September 2008
92
-------
Cincinnati Pilot Post-Implementation System Status
The SaTScan™ prospective space-permutation method is used as the event detection tool for this public
health data stream. More information is available at http://www.satscan.org. This method detects a
cluster in a time space setting using the latitude and longitude coordinates of the call location. A cluster is
defined as a high proportion of cases over a twenty-four hour period in one geographic location (zip code)
compared to another; the Cincinnati pilot has established notification rules for clusters at a significance
level of p < 0.025 and display requirements of clusters at a significance level of p < 0.10.
Emergency Medical Service Logs
To improve the timeliness of data collection from the Cincinnati Fire Department's emergency medical
service tablets, wireless CISCO WISM devices and Aironet routers were installed in all twenty-six (26)
Cincinnati Fire Department stations. Installation of these wireless routers allowed for automatic upload
of patient information to the Cincinnati Fire Department server upon return of the vehicle to the firehouse.
Figure 7-3 presents an illustration of the information flow for emergency medical service logs, beginning
with entry of data into the tablets in the field. The provider's impressions are categorized into syndromes
using the Early Aberration Reporting System software developed and maintained by the CDC (CDC,
2006). The syndromes are analyzed temporally based on zip code of the location of the run. Data are
collected and results are generated on an hourly basis. Upon analysis completion, data display and
alerting tools publish a new analysis summary to the Public Health User Interface (discussed below) for
user review. Through an automated process, analysis results are compared against business rules to
determine if there is a deviation from the baseline and users should be notified. If such conditions exist,
an email notification is sent to public health partners including the Cincinnati Health Department and
Hamilton County Public Health and GCWW as a prompt for further investigation in accordance with the
procedures defined in the Concept of Operations for the GCWW Contamination Warning System
(USEPA, 2007b).
Emergency
Medical Technician
EMS Tablet
T
0
Run Detail uploaded to
^=— rr^
WS CFD Server
Data filtered for
HIPAA and copied into
91 1/EMS Database
WUERM and LPH
Investigate alert
WS Application Server
Subscription Filtering
Filtered data sent to EARS
EARS input spreadsheet
EARS Analysis
riS Alert
Data transformed into XML,
assigned EARS syndrome
EMS results displayed on
PHS User Interface
Filtered Data _
"Sent to WSDFT
WSDR Oracle DB
Server
Local
Public
Health
CFD
Figure 7-3. Information Flow for Emergency Medical Service Logs
September 2008
93
-------
Cincinnati Pilot Post-Implementation System Status
The EARS software allows for customization of syndrome categories. A list of provider impressions that
might indicate a drinking water contamination incident was developed through discussions with local
health partners. The provider impressions are categorized into the following eight syndromes:
• Cardiac category: angina pectoris, cardiac arrest, chest pain/discomfort, congestive heart
failure, dysrhythmia, hypertension, hypotension, myocardial infarction, unconscious (unknown
etiology)
• Gastrointestinal category: abdominal pain minor, abdominal pain severe, appendicitis,
dehydration, diarrhea, food poisoning, lower gastrointestinal bleeding, nausea/vomiting, upper
gastrointestinal bleeding
• Neurological category: altered level of consciousness, cerebrovascular accident or stroke,
dizziness/vertigo, headache, numbness/tingling, paralysis/loss of motion, seizures/convulsions
unknown, syncope/fainting, transient ischemic attack
• Poison category: abuse/dependency, alcohol related, drug induced emotional, drug overdose,
food poisoning, hematuria, ingestion, inhalation, renal failure
• Psychological category: abuse/dependency, alcohol related, anxiety, behavioral disorder,
depression, drug induced emotional, drug overdose, psychiatric disorder, suicide attempt (not
dead on arrival)
• Unexplained category: blank (provider impression not provided), dead on arrival, other,
respiratory arrest, unconscious (unknown etiology)
• Upper respiratory category: airway obstruction/choking, cold/flu, croup, epiglottitis,
respiratory distress, respiratory distress (acute), respiratory involvement, smoke inhalation
• Water category: altered level of consciousness, abdominal pain minor, abdominal pain severe,
diarrhea, dizziness/vertigo, ingestion, nausea/vomiting, seizure/convulsions febrile,
seizures/convulsions (unknown)
The mechanism for event detection employed by the EARS software relies on a cumulative sum or
CUSUM method. A CUSUM is best defined as a measure of how much higher a current observation is
than a reference baseline. The EARS software utilizes three CUSUM algorithms to analyze the data: Cl,
C2, and C3. Each of these algorithms is explained below:
• Cl algorithm. Includes data from the current day only [that is, hits from one (1) day of data], but
uses the prior seven (7) days worth of data to calculate the (baseline) reference mean. Cl is the
least sensitive of the three (3) algorithms. A flag or alert generated by the Cl algorithm is most
useful when evaluating acute events.
• C2 algorithm. Analyzes data from the current day but shifts the data start back two (2) days
from the current day; the baseline mean is calculated using the seven (7) prior days of data from
that initial point. C2 is also based on one day of information. C2 can help define the length of an
outbreak's acceleration and when the outbreak has peaked. C2, however, does not include the
initial start of the upward swing of an outbreak. A flag or alert generated by a C2 algorithm is
slightly more sensitive than a flag or alert generated by a Cl algorithm.
• C3 algorithm. Uses the same timeframe as C2 (it shifts the starting point of the data to start from
two (2) days prior to the current day and then the baseline mean is calculated using the seven (7)
prior days of data from that initial point). However, C3 includes an average of three (3) days of
events which it compares against the seven (7) day baseline mean. C3 is useful when there is not
a sharp increase above the baseline. For example, in cases where data from the past three (3)
days have been above the mean but not far enough above the mean to produce a flag from a Cl or
C2 algorithm. A flag or alarm generated by a C3 algorithm is considered to be the most sensitive
of the three.
Based upon historical analyses of emergency medical service logs, descriptions of the CUSUM
algorithms, and the protocol for regional surveillance in the Cincinnati area, alerts generated by the C1
algorithm were considered to be the most useful for the Cincinnati pilot.
September 2008 94
-------
Cincinnati Pilot Post-Implementation System Status
Poison Control Center Calls
As part of the Cincinnati pilot effort, a contract for testing the feasibility and benefits of integrating local
poison control center calls into the contamination warning system framework was funded by EPA's
Office of Research and Development (ORD) NHRSC. The purpose of the project was to determine how
local poison control centers can contribute to early detection, notification, and rapid response for a more
robust drinking water contamination warning system.
The Drug and Poison Information Center implemented a multi-tiered approach to event detection that
included statistical, non-statistical, and human surveillance as illustrated in Figure 7-4.
Calls received on hotline
are entered into Toxicall
by call center staff
Data uploaded to the
National Poison Surveillance
System continuously via the
AutoUpload process
Automated statistical and
non-statistical analysis
performed
Local health departments
and utility staff sent email
notification for further
investigation
ger identify ^^
al risk? s/*
T
Toxicosurveillance
Team reviews trigger
Is
Email notification sent to
Toxicosurveillance Team
^^— ^
/^ Return t
( normal
V^ope ration
n
Figure 7-4. Drug and Poison Information Center Drinking Water Surveillance Process Flow
The American Association of Poison Control Centers National Poison Data System aggregates data
collected from poison control centers across the nation, with data uploaded on a near real-time basis and
hourly statistical analysis, alert processing, and communication of findings. The system allows poison
control centers to develop customized statistical analysis parameters for three individual types of
toxicosurveillance definitions including: total call volume, human exposure call volume, and clinical
effect count. Zip codes or telephone numbers can be fixed for each definition to provide a more targeted
regional approach for surveillance. For the Cincinnati pilot, this included all Ohio zip codes in the
GCWW service area.
The statistical surveillance approach for the Cincinnati pilot relies on the human exposure call volume
and clinical effect count definitions. For human exposure call volume, statistical analysis is performed
only on the total number of human exposure calls uploaded to the National Poison Data System. Analysis
on the clinical effect count definitions are performed on each individual symptom that is coded in the
Toxicall medical database used by the Drug and Poison Information Center. Aberrations greater than
three times the standard deviation from the baseline that involve at least two (2) cases trigger an email to
be sent to the toxicosurveillance team (on call 24/7/365) for further investigation.
Non-statistical analysis methods leverage the National Poison Data System's Syndromic Definition
Module. The toxicosurveillance team developed a customized search through this module that
incorporates specific substances thought to be most likely related to a water contamination event (e.g.,
arsenic, cyanide, dioxin, ricin, botulinum toxin, etc.) and eliminates records where the reason for exposure
to the substance is understood and unrelated to water (e.g., intentional suicidal exposures, occupational
injuries).
September 2008
95
-------
Cincinnati Pilot Post-Implementation System Status
The third surveillance method deployed by the Drug and Poison Information Center relies on human
surveillance. The Drug and Poison Information Center is staffed by trained physicians, pharmacists,
nurses, paramedics, and students within nursing and pharmacy programs. In addition, a Certified
Specialist in Poison Information staff member is consulted on every exposure call received to ensure
consistency in recommendations from the call center and physician toxicologists are on call 24/7/365 for
more complicated or critical human poisonings. The human surveillance method for the pilot relies on
this expertise along with the open call center environment that facilitates ongoing discussion and
consultation among staff members.
In addition to the surveillance methods described above, as part of the project the Drug and Poison
Information Center established a "Water Safety Hotline" that is dedicated for water contamination
queries. Physicians, nurses, pharmacists, public health providers, and utility staff seeking toxicology
consultation or related services can access this number in the event of unusual water testing results, water-
related health effects or other threats.
7.2.2 Communication and Coordination
As discussed in Section 7.1.2, while a number of public health committees and organizations existed in
the Cincinnati area, communication and coordination with the utility was inconsistent and limited. To
improve this area a Public Health User's Group was established. The User's Group is comprised of
members from EPA, GCWW, Cincinnati Fire Department, law enforcement, the Drug and Poison
Information Center, the Environmental Health and Safety Officer from Northern Kentucky Health
Department, and a representative from the Executive Steering Committee, Bioterrorism Coordinators,
Environmental Surety, and Regional Epidemiologists and Disease Investigators of the Southwest Ohio
public health committees described in Section 7.1.2. These representatives report the progress of the
Cincinnati pilot to the members of their respective committees and organizations to ensure awareness of
the program.
During the early stages of design and deployment, the Public Health User's Group met on a monthly basis
to inform the design, use, and evaluation of tools proposed for the public health surveillance component
of the Cincinnati pilot drinking water contamination warning system. With the completion of
implementation activities the group has transitioned to a quarterly meeting schedule. The User's Group
provided a forum to discuss not only issues related to the pilot, but other issues that impacted both the
public health community and the drinking water utility. Through participation in these meetings, an
ongoing dialogue has been established that improves communication and coordination between GCWW
and its local public health partners.
In addition to the Public Health User's Group, development of the GCWW Concept of Operations
(USEPA, 2007b) provided an opportunity to better define the protocols and procedures for how GCWW
would work with local public health partners to investigate alarms generated through the contamination
warning system. As part of this development effort the User's Group identified improvements and/or
defined protocols for issuing notifications of alarms. For the 911 calls and emergency medical service
logs this involved developing an automatic notification tool to send emails to GCWW, Cincinnati Health
Department, Hamilton County Public Health, and Drug and Poison Information Center representatives
simultaneously so each organization could begin the process of trigger investigation as described in the
Concept of Operations. Similarly, the Drug and Poison Information Center includes the same health and
utility contacts on email notifications related to anomalies detected through the analysis of poison control
center call data. Although the Real-Time Outbreak and Disease Surveillance system for analyzing
emergency room chief complaint data was not customized for the objectives of the contamination warning
system pilot, the notification protocol was modified to include staff from GCWW for email alerts
generated in response to elevated gastrointestinal illness in Hamilton County, OH (including the City of
Cincinnati).
September 2008 96
-------
Cincinnati Pilot Post-Implementation System Status
7.2.3 Summary of Post-Implementation Status
The public health surveillance component of the contamination warning system has been installed, is fully
operational, and is currently generating data needed to establish baseline performance. The public health
surveillance component includes the following data streams: 911 calls, emergency medical service logs,
Poison Control Center calls, over-the-counter medication sales, emergency room chief complaints, and
infectious disease reporting. A user interface was developed and deployed on the City's WAN to view
and analyze data generated by 911 calls and emergency medical service logs. Customized syndrome
categories were developed to support analysis of 911 calls and emergency medical service logs by spatial
and statistical algorithms, respectively. A user's group was created to better integrate GCWW into the
public health community and joint notification protocols were implemented to improve coordination and
communication relative to potential drinking water contamination incidents. Table 7-3 provides a
summary of the post-implementation status of the public health surveillance component of the
contamination warning system.
Table 7-3. Public Health Surveillance Post-Implementation Status
Attribute
Description of Installed Component
1. Public Health Data
Streams
911 call information collection is standardized through the use of Priority Dispatch
software and transferred to a centralized server where it is processed for analysis
Wireless routers have been installed at the City's 26 fire stations to support automatic
upload of emergency medical service data
Customized syndrome categories defined for spatial and statistical analysis of 911
calls and emergency medical service logs
Public Health User Interface developed and deployed on the City's Metropolitan Area
Network to view and analyze 911 calls and emergency medical service data
Enhanced algorithms and protocols implemented for handling of Poison Control
Center calls
2. Communication and
Coordination
Public Health User's Group with representatives from key public health partners and
GCWW meets on a quarterly basis to discuss issues related to the public health
surveillance component of the contamination warning system
An automated email notification mechanism is in place to notify local public health
and GCWW of alarms generated through the analysis of 911 calls and emergency
medical service logs
Local public health departments and GCWW receive simultaneous automated email
notifications of alarms generated through the analysis of emergency room chief
complaints for gastrointestinal illness
24-hour Water Safety Hotline established by the Poison Control Center to provide
toxicological support to GCWW and local public health on issues related to drinking
water contamination
A concept of operations has been developed that guides the day-to-day operation of
the component
Figure 7-5 provides a summary of the level of effort associated with design and implementation of the
public health surveillance component of the Cincinnati pilot. Public health surveillance activities, as
summarized in Table 7-3, relied on support from EPA, GCWW, and local partners. The level of effort for
both design elements - available data streams and communication and coordination - was comparable.
Local public health partners participated in regular Public Health User Group meetings, and received and
reviewed automated email notification of alarms generated through analysis of 911 calls and emergency
medical service logs.
September 2008
97
-------
Cincinnati Pilot Post-Implementation System Status
.u
.0
1.6 -
"1 A -
!_ 1-2
ra
« ,.
1
^ n« -
U.o
.6
.4
.z
.0 1
0.01
\^
0.6
0.04
0.5
Available Data Streams Communication and Coordination
D Local Partners
LOE
•GCWWLOE
• EPA LOE
Figure 7-5. Level of Effort for Design and Implementation of the Public Health Surveillance
Component (December 2005 - December 2007)
Figure 7-6 presents a summary of the extramural costs associated with design and implementation of the
public health surveillance component for the Cincinnati pilot. Significant EPA contractor labor was
associated with design and deployment of the 911 call and emergency medical service data flows. EPA
contractor support to communication and coordination activities included design and deployment of
notification procedures, development of the concept of operations, and support to the Public Health
User's Group. Costs associated with contractor travel were not included in this calculation.
September 2008
98
-------
Cincinnati Pilot Post-Implementation System Status
-------
Cincinnati Pilot Post-Implementation System Status
Other
activities,
$86,000
Emergency
Medical
Service
records,
$273,000
Figure 7-7. Breakout of Costs Associated with Deployment of 911 Call Logs and Emergency
Medical Service Records
The costs associated with significant enhancements for a public health surveillance data stream relative to
leveraging data streams already in place for the purpose of syndromic surveillance may be an important
consideration for contamination warning system design and deployment in other jurisdictions. Through
evaluation of the Cincinnati pilot, the return on investment will be considered and may further influence
the decision-making process for contamination warning system design.
September 2008
100
-------
Cincinnati Pilot Post-Implementation System Status
Section 8.0: Consequence Management
Consequence management is a key aspect of the GCWW contamination warning system and consists of
actions taken to plan for and respond to potential drinking water contamination incidents in the
distribution system. These actions are meant to minimize response and recovery timelines through a pre-
planned, coordinated effort between the utility and a network of local response partners. Investigative
and response actions initiated upon determination of a "possible" contamination threat are used to
establish credibility, minimize public health and economic impacts, and ultimately return the utility to
normal operations.
The Consequence Management Plan (USEPA, 2006e) serves as a guide for GCWW that describes the
actions that should be taken upon determination of a "possible" contamination threat, as detected by one
of the contamination warning system monitoring and surveillance components. In the event of a
confirmed contamination incident, the plan can be used by utility staff and/or local response partners to
guide remediation and recovery steps to return the utility to normal operation. The Consequence
Management Plan relies on extensive pre-planning efforts to establish clear roles and responsibilities with
local, state, and federal response organizations, and to define strategies for communicating with the
public.
The GCWW Consequence Management Plan was developed to be a component of the utility's existing
emergency response plan and focuses specifically on a contamination threat to the distribution system.
The Consequence Management Plan includes sections and response guidelines (e.g., decision trees) to
address all phases of consequence management including Credible Determination, Confirmed
Determination, and Remediation and Recovery. In addition, information is included that describes how to
address utility and risk communication issues.
The pre-implementation assessment process used to initiate the development of the Consequence
Management Plan consisted of the following activities:
• Evaluation of Plans, Processes and Equipment - Existing materials and response
plans/procedures were reviewed to determine their applicability to contamination warning system
consequence management.
. Interviews with Stakeholders - Interviews were conducted to identify the existing status of
utility incident management operations for a contamination incident. Interviews were used to
assess strengths and weaknesses of the utility's emergency response plan, as well as to identify
current needs, including GCWW response partner support. Three groups were identified and
interviewed to assess existing procedures: utility group consisting of representatives from
GCWW's operations, distribution, laboratory, engineering, business services and commercial
services divisions; local group consisting of city, county, state and regional federal agency
representatives (police/fire departments, city/state public health, etc.); and technical consultant
group consisting of individuals with water utility or public health expertise and experience in
contamination warning system processes similar to the Cincinnati pilot.
. Workshops with Stakeholders - Workshops with GCWW, response partner agencies and
representatives from other utilities were used to determine the appropriate level of stakeholder
involvement during Consequence Management Plan development. This included identifying
roles and responsibilities in the pilot and the status of external response partner coordination
including their response plans and procedures (e.g., Hazardous Materials responders' protocols,
law enforcement response protocols).
Coupled with the assessment process, a gap analysis was conducted to compare existing response
capabilities against the objectives of the GCWW Consequence Management Plan to determine missing
components. Potential solutions to close these gaps were then evaluated for impact and cost, taking into
September 2008 101
-------
Cincinnati Pilot Post-Implementation System Status
account possible leveraging of existing system plans and procedures already integrated into daily
procedures.
The Consequence Management Plan design objectives are shown in Table 8-1, and were derived from the
overarching performance objectives of the contamination warning system as described in WaterSentinel
System Architecture (USEPA, 2005a). GCWW's pre-existing capability with respect to each attribute of
the Consequence Management Plan listed in Table 8-1 is summarized in Section 8.1 and the utility's
capability after implementation of the contamination warning system is summarized in Section 8.2.
Table 8-1. Consequence Management Plan Design Objectives
Attribute
1. Contaminant
Incident Response
Plans
2. Response Partner
Network
3. Training and
Exercises
4. Equipment
Design Objective
• Review existing utility emergency preparedness plans and operations to
determine potential elements of a CMP.
• Identify and engage response partner agencies and stakeholders that may be
involved in the development of the GCWWCMP and corresponding response
activities.
• Assess the existing utility training program to determine potential elements that
address a system-wide contamination incident.
• Assess existing incident response equipment to identify additional equipment
needed to carry out response activities outlined in the GCWW CMP; acquire
equipment, as necessary.
8.1 Pre-lmplementation Status
The following section provides the pre-implementation status of the GCWW consequence management
program. As stated above, the pre-implementation assessment process used to initiate the development of
the Consequence Management Plan consisted of evaluating existing response plans/procedures/equipment
and conducting interviews and workshops with key stakeholders. The results of the pre-implementation
assessment are described below including a summary of gaps identified relative to the objectives of a
functional Consequence Management Plan.
8.1.1 Contamination Incident Response Plans
Prior to the development of the GCWW Consequence Management Plan, existing utility preparedness
plans and written procedures, including standard operating procedures, policies, and checklists, were
collected and evaluated according to their applicability and contribution to the development of a
Consequence Management Plan. During the review, a matrix was created that identified the type of plan,
response resources available, areas where gaps occurred, and possible interaction points with the
Consequence Management Plan and external response partner plans. A summary of the matrix is shown
in Table 8-2.
Table 8-2. Pre-lmplementation Evaluation of Utility Plans, Procedures, Equipment, and Training
GCWW Plan
Emergency
Response Plan
Contains
protocols for
drinking water
contamination
response?
^
Contains
a list of
external
partners?
^
Plan up
to date?
•^
Contains
steps that can
be included in
the CMP?
^
Notes
This plan covers water
contamination and has a
good list of potential
response partners, including
a contact list.
September 2008
102
-------
Cincinnati Pilot Post-Implementation System Status
GCWW Plan
Security Standard
Operating
Procedure (SOP)
Cryptosporidium
Action Plan
Alternate Water
Supply Plan
CWWFIowsheetl 0
42503. pp
MassContamPlan.d
oc
REACT.doc
(may be called
illness1.doc by
GCWW)
SecllF4 Distribution
Water
Contamination
BacT Route
Locations.doc
Depressurization 0
42503.doc
Contains
protocols for
drinking water
contamination
response?
•^
•^
v'
•^
^
^
Contains
a list of
external
partners?
^
^
Plan up
to date?
y'
^
S
y'
Contains
steps that can
be included in
the CMP?
^
^
^
Notes
This plan contains detailed
information on working with
local law enforcement that
could be included in the
CMP, although contact
numbers need to be
updated.
This plan contains specific
protocols for responding to
water contaminated by
Cryptosporidium. The
process portion will link to
CMP and has excellent
response information.
This plan is currently being
developed by Hamilton
County and City of Cincinnati
with agreements with Ohio
and Kentucky National
Guard. When completed, it
should be linked to the CMP.
These SOPs vaguely identify
water flow rates. Not useful
for planning, response or
recovery.
This plan provides little
information and does not
address the issues of
response and recovery to a
mass contamination incident.
This plan provides the steps
and activities involved in
responding to a single
contamination point, such as
crossed sewer/water lines in
a residence. This process is
important since it reflects day
to day operations and should
serve as the basis for larger
contamination incident
response.
This plan is incomplete and
does not address full
response, recovery or
integration with other
response partners.
This plan covers BacT
processes and probable
locations for its growth.
This plan contains SOPs with
instructions concerning how
to depressurize a water line.
September 2008
103
-------
Cincinnati Pilot Post-Implementation System Status
GCWW Plan
Directions for
routes.doc
Emergencycontacts
.doc
Public Notice.doc
Contains
protocols for
drinking water
contamination
response?
'
S
Contains
a list of
external
partners?
-
Plan up
to date?
'
'
Contains
steps that can
be included in
the CMP?
-
'
Notes
This plan includes general
descriptions of the larger
routes.
This list provides a starting
point of contacts for GCWW
and integration. Contacts
have not been updated in a
long time; some
names/numbers no longer
apply.
This is critical for the public
notification portion of
planning but consists only of
three flyers: Do Not Use, Do
Not Drink and Boil. Meets
Ohio and water guidelines.
Consider whether to
formalize and document the
public notification procedures
into a plan or SOP.
The documents in this matrix were relevant at the time the matrix was created but may now be outdated.
As indicated in Table 8-2, most of the existing plans and procedures served as independent plans that
focused on normal or routine operational conditions at the utility and were not fully applicable to the
intent of the Consequence Management Plan. Therefore, the analysis indicated the need for GCWW and
stakeholders to develop a comprehensive Consequence Management Plan that addressed the deficiencies
identified during review of existing response plans. Specifically, the deficiencies included guidance for
contamination threat level determination (e.g., credible, confirmed), a formal risk or crisis communication
plan, and a site characterization plan or similar process.
8.1.2 Response Partner Network
The degree of GCWW integration with response partner agencies was evaluated and compared to the
goals of a Consequence Management Plan. Integration addressed the level and degree to which GCWW
plans, procedures and operations were coordinated with those of the external response partners.
Integration was evaluated and addressed through surveys, interviews, and workshops with GCWW and
response partner agencies.
Preliminary surveys and interviews with GCWW revealed a lack of participation directly with local
emergency planning committees. To some degree, key officials at GCWW had active relationships with
response partners, in particular with the Fire and Police departments, through the functional organization
of the city government itself. In addition, the GCWW Public Information Officer had established
relationships and lines of communication with other public information officers at the city, county and
public health levels. However, with the exception of the Cryptosporidium Action Plan and REACT
procedures, a formal plan or agreement had not been established to define the roles, responsibilities,
processes, coordination and lines of communication between the utility and local partners in the event of a
contamination incident.
Integration was further evaluated and addressed through a series of four workshops involving external
response partner agencies. The workshops provided a forum to identify roles and responsibilities,
procedural gaps and to pursue integration of plans and procedures. Response partner agencies from the
two largest jurisdictions in the GCWW service area - the City of Cincinnati and Hamilton County - were
included in the workshops. A description of the four (4) workshops is included in Table 8-3.
September 2008
104
-------
Cincinnati Pilot Post-Implementation System Status
During the initial workshops, it was clear that most external agencies did not view the utility as a "first
responder", even when presented with a scenario of a water contamination incident. External partners did
view the water utility as a critical part of the city's infrastructure, but were unaware of its susceptibility to
contamination and the potential consequences of such an incident. Most of the primary response agencies
(e.g., local law enforcement and fire/HazMat) considered first response to a water contamination incident
as belonging to their jurisdiction, although these agencies did not have water contamination experience or
training. Since the utility was not viewed as a first response agency, the integration of plans and
procedures had never been considered by local agencies and the interoperability of procedures and
equipment had not previously been identified as an important issue.
Table 8-3. Response Partner Workshops
Workshop
Consequence
Management
Workshop #1
Consequence
Management
Workshop #2
Consequence
Management
Workshop #3
Consequence
Management
Workshop #4
Date
12/13/05
01/19/06
02/14/06-
02/15/06
03/22/06
Participants
GCWW Key Officials,
Cincinnati Fire and Police
Departments
GCWW Key Officials, City
and County Public Health
Agencies, Cincinnati Fire
and Police Departments
GCWW Key Officials,
Public Information
Officers from various City
and County response
agencies, City and County
Public Health Agencies,
Cincinnati Fire and Police
Departments
GCWW Key Officials and
Department Heads, City
and County Public Health
Agencies, Cincinnati Fire
and Police Departments,
and Government
Representatives from the
City, State, and Federal
levels.
Description
Discussed initial stages of CMP development
including decision tree and response partner
involvement. Cincinnati Fire Department provided
introductory NIMS briefing for GCWW staff.
Collected data from response partners and
reviewed draft CMP decision trees for field
operation information that would occur with
possible, credible, and confirmed determination
decision trees.
Provided information on risk communication and
message mapping for communicating with the
public during a possible contamination threat.
Developed GCWW message maps.
Collected and reviewed notification and
communication data focusing on credible and
confirmed stages of an emergency where multiple
response agencies will be involved This involved
identifying key connections, linkages, and
dependencies between the draft CMP and external
response partner plans and procedures. Collected
the same data for sub-flow decision trees (e.g.,
site characterization).
8.1.3 Training and Exercises
Pre-implementation assessment of training activities revealed that GCWW did not have a formal training
program in place to address a distribution system-wide contamination incident. Rather, their existing
training program only focused on specific standard operating procedures used in daily operations.
GCWW also did not participate with outside organizations in training, cross training or general
preparedness; however, on several occasions, a limited number of utility engineers and managers
participated in city-wide emergency response exercises that focused on public health issues rather than
utility procedures.
In addition, the utility was aware of the State and Federal preparedness training requirements, but had not
yet addressed them. This included federal compliance and training under Homeland Security Presidential
Directive 5, the National Incident Management System, and the State of Ohio requirements that utilities
meet the National Incident Management System requirements for Incident Command System training
including IS 100, IS 200, ICS 300, ICS 400, IS 700 and IS 800.
September 2008
105
-------
Cincinnati Pilot Post-Implementation System Status
8.1.4 Equipment
As part of the preliminary assessment, GCWW staff were interviewed to assess existing incident response
equipment. Throughout the development of the GCWW Consequence Management Plan, equipment
needs were reviewed to ensure that any changing roles and responsibilities that required additional
equipment were considered. Aside from equipment used on a daily basis for water system maintenance
and field sampling, the utility lacked field response equipment for Consequence Management Plan
procedures including site characterization and field screening, sample concentration, and sample analysis.
In addition, GCWW did not have communication equipment (e.g., 800 MHz radios) that was
interoperable with the City and County response partner agencies. Without this equipment, key GCWW
officials and engineers did not have the capability to be immediately alerted by outside agencies to
incidents or potential threats of water contamination. Furthermore, GCWW field response teams did not
have an established way to communicate with the Incident Commander and that the Incident Command
System did not have a way to communicate with response partners in the field.
8.1.5 Summary of Identified Gaps
Table 8-4 presents the gaps identified during the initial assessment. Section 8.2 describes the post-
implementation status for each of the attributes listed below.
Table 8-4. Consequence Management Component Gap Analysis
Attribute
1. Contaminant
Incident Response
Plans
2. Response Partner
Network
3. Training and
Exercises
4. Equipment
Gap Description
• Assessment of existing plans, procedures, and policies revealed that GCWW did
not have a comprehensive response plan to address system wide contamination.
Additional guidance was needed for contamination threat level determination,
site characterization and risk communication.
• There was minimal participation/communication between GCWW and
response partners. The integration of response plans and procedures had never
been considered by local agencies and the interoperability of procedures and
equipment had not previously been identified as an important issue.
. Assessment of training activities at GCWW revealed that they did not have a
formal training program in place to address a system-wide contamination
incident. Existing training also did not meet the NIMS/ICS and State of Ohio
training requirements.
. GCWW did not have necessary field response equipment for site
characterization and field screening, sample concentration, and sample analysis.
. GCWW did not have communication equipment (e.g., 800 MHz radios) that was
interoperable with the City and County response partner agencies and other
GCWW response groups in the field.
8.2 Post-Implementation Status
The following section provides the post-implementation status concerning development of the GCWW
consequence management program. The gaps and issues identified during the pre-implementation
assessment were tracked and addressed during the development of the GCWW Consequence
Management Plan and corresponding documents. The major products that resulted from the consequence
management implementation efforts included the Consequence Management Plan, the Crisis
Communication Plan, the Site Characterization Plan, a comprehensive training and exercise program, and
the purchase of necessary response equipment.
8.2.1 Contamination Incident Response Plans
Response planning documents developed during implementation of the Cincinnati pilot are described
below.
8.2.1.1 Consequence Management Plan
The GCWW contamination warning system pilot Consequence Management Plan was developed to serve
as a preparedness and response guide in the event of a water contamination incident. The Consequence
September 2008 106
-------
Cincinnati Pilot Post-Implementation System Status
Management Plan is intended to guide GCWW through the process of determining whether a "possible"
contamination threat, as indicated by one of the contamination warning system monitoring and
surveillance components, is "credible" and can be "confirmed." The Consequence Management Plan also
assists the utility in working with local partners, communicating with the public, and determining
appropriate response actions. In the event of actual contamination, the plan provides information on
remediation and recovery steps to return the utility to normal operation.
The Consequence Management Plan was developed as a stand-alone document incorporating elements of
the GCWW Emergency Response Plan and the U.S. EPA Response Protocol Toolbox to address gaps
identified in assessment of existing utility response plans. The Consequence Management Plan also
incorporated elements of the National Incident Management System-based Incident Command System
protocols to provide incident command and control guidance.
The Consequence Management Plan consists of nine (9) separate decision trees which provide response
guidance to track the evolution of a contamination incident from a "possible' contamination threat, as
detected by one of the contamination warning systems monitoring and surveillance components, through
system remediation and recovery. The decision trees address response topics which include threat level
("credible" and "confirmed") determination, site hazard characterization, operational responses, response
partner and public notification protocols, and remediation and recovery.
In addition, the Consequence Management Plan contains five (5) appendices of supporting material,
including the Site Characterization Plan, various forms for collecting and documenting the contamination
incident, and operational procedure sheets for major response activities.
8.2.1.2 Crisis Communication Plan
A Crisis Communication Plan (USEPA, 2007u) was developed to formalize public notification
procedures and guide the actions of the GCWW Public Information Officer during all phases of a
potential contamination incident. It was designed from best practices in risk communication and public
notification to provide communication control internally during an incident and to coordinate external
public notification. The Crisis Communication Plan is designed to complement the overall Consequence
Management Plan and corresponding response procedures outlined in Decision Tree 400.0: Public
Notification.
The Crisis Communication Plan covers communication both within GCWW and with external response
partner agencies, the press, and the public. The plan includes an overview of basic crisis communication
principles, detailed Consequence Management Plan decision trees adapted for use by the Public
Information Officer, and a tools and resources section that includes sample public notification templates,
media resources, and contact information.
8.2.1.3 Site Characterization Plan
The Site Characterization Plan (USEPA, 2006f) was developed as part of the Consequence Management
Plan to provide guidance to GCWW field personnel for the preliminary investigation phase of a
contamination incident. The objective of the Site Characterization Plan is to describe the steps needed to
collect sufficient information from an investigation site(s) to help characterize an incident site(s) once a
threat, accidental or intentional, has been suspected. It is based on principles contained in Module 3 of
the U.S. EPA Response Protocol Toolbox: Site Characterization and Sampling Guide (USEPA, 2003).
The Site Characterization Plan is outlined in Decision Tree 30.0 and Appendix B of the Consequence
Management Plan and describes the roles and responsibilities of GCWW management and field response
personnel, and local response support (law enforcement, HazMat, health and emergency management),
for site characterization activities. This includes site evaluation, field safety screening, sample collection,
and rapid field testing. In addition, the Site Characterization Plan consists of seventeen (17) SOPs for
approaching and characterizing an investigation site. The standard operating procedures cover site safety
screening, rapid field tests of drinking water, drinking water sampling techniques for laboratory analysis
September 2008 107
-------
Cincinnati Pilot Post-Implementation System Status
and operation of the field equipment used for conducting the field investigation. Also included in the Site
Characterization Plan are forms for reporting investigation observations and data, sample documentation,
chain-of-custody, and a template for preparing incident-specific Site Characterization Plans.
8.2.2 Response Partner Network
Through face-to-face workshops (as described in Table 8-3 of Section 8.1.2), GCWW and response
partner agencies were better able to understand, confirm, and integrate their roles and responsibilities in
the event of a drinking water contamination incident. This allowed GCWW to document these roles,
responsibilities, and communication requirements into the Consequence Management Plan and, in turn,
allowed outside agencies to recognize the "first response" capabilities and duties of the utility.
Through each of the four (4) workshops, gaps were identified and information was gathered for use in
development of Consequence Management Plan and corresponding procedures. The workshop process
required each participant to describe their perception of the roles and responsibilities in a given incident.
Responses from participants were compared and gaps in responsibilities were identified and addressed.
Existing response plans, contact lists, and draft plans were revised according to the findings subsequent to
the workshops.
A summary of the response partner network that evolved from the workshops is shown in Table 8-5.
Table 8-5. Summary of Pilot Response Partner Network and CMP Role
Response Partner Organization
Method Laboratories
Cincinnati Fire Department (CFD)
Local public health agencies (LPH) in GCWW
service area
Hamilton County Emergency Management Agency
Ohio Department of Health
Local, State and Federal law enforcement
State Drinking Water Primacy Agency (Ohio EPA)
CMP Role
Analyzes triggered samples and interacts directly with the
GCWW Incident Commander for field results, event status
and reporting
Supports GCWW consequence management activities by
providing field response and HazMat support
Provides local public health data; epidemiologists and disease
investigators
Supports CMP by coordinating alternate water supplies and
implementing Unified Command System for expanded
contamination incidents
Provides regulatory and sampling support, investigations and
threat assessments of potential or confirmed contamination
events
Provides support by coordinating investigation and isolation
issues with GCWW Incident Commander and GCWW
Security
Provides support for risk communication and remediation/
recovery issues
In addition to the workshops for response partner agencies in the City of Cincinnati and Hamilton County,
roll-out presentations on the Water Security initiative were provided to other jurisdictions served by
GCWW to ensure that all local partners have working knowledge of the program, benefits of the program
to their community, and their roles and responsibilities under the program. The roll out strategy was
designed to achieve an efficient and comprehensive approach for reaching all jurisdictions through each
of the disciplines (e.g., all county, city, and community public health agencies). This goal was
accomplished as representatives from each agency (within one, or both, of the two primary jurisdictions)
acted as primary facilitators and used the roll-out presentations to provide training within their respective
agencies.
Overall, the workshop and roll-out process resulted in expanded preparedness and response capability
amongst local partners and stakeholders and, as a result, a stronger local network was established. The
process was also important to the development and strengthening of the utility Consequence Management
September 2008
108
-------
Cincinnati Pilot Post-Implementation System Status
Plan and response process. Ultimately, planning, communication and coordination procedures developed
with input from local response partners were tested and refined through a series of training and exercise
activities.
8.2.3 Training and Exercises
Training and exercises were conducted to test the Consequence Management Plan and to train participants
on processes and procedures. Training consisted of workshops, table-top exercises, functional exercises,
drills, and full scale exercises. Training was also important in identifying opportunities for improving the
plans, evaluating participants' ability to implement the guidance of the Consequence Management Plan,
and increasing response time and accuracy. After completion of the exercise, "After Action Reports"
were developed that captured comments from participants and evaluators about suggested modifications
to the Consequence Management Plan and contained a critique regarding areas for improvement to future
exercises.
Table 8-6 presents the training that was delivered to address gaps and enhance preparedness and response
capabilities by the utility and response partners.
Table 8-6. Consequence Management Training and Exercises
Training Title
DHS FEMA/NIMS
IS100&IS700
CMP Orientation
Training Implementation
Roll Out Documentation
and Presentation
Table-Top Exercise:
Supervisor Training-
(4 separate sessions)
Site Characterization
Training
(4 separate sessions)
Contamination Warning
System Management
Training
Consequence
Management Functional
Table-Top Exercise
Site Characterization
Drills
Consequence
Management Full Scale
Field Exercise
Date
Received
5/16/06 -
05/18/06
08/22/06 -
08/24/06
06/27/06
and
09/28/06
08/29/06 -
08/31/06
05/11/07
06/15/07
07/31/07
09/05/07 -
09/06/07
09/25/07 -
09/28/07
Participants
GCWW
GCWW
Potential partners
outside the GCWW
response area
GCWW
GCWW
GCWW
GCWW and
Response Partners
GCWW
GCWW and
Response Partners
Description
Provided participants with a basic understanding of
ICS procedures and an introduction to NIMS. The
material was formatted for delivery to GCWW for
use in future instruction.
Senior level staff training ensured participant
understanding of their roles in the CMP. The
material was formatted for delivery to GCWW for
use in future instruction and updates.
Discussed the consequence management
preparedness and response process with external
agencies and organizations that might not
otherwise be readily engaged in the GCWW
response network.
Provided GCWW supervisors with scenarios to
improve their knowledge of the CMP and incident
management processes.
Familiarized staff with field operations associated
with site investigations and corresponding
equipment.
Provided an overview of the contamination warning
system and CMP including contamination scenario
exercises.
Provided GCWW and response partners the
opportunity to practice their roles and
responsibilities during a response to a possible
drinking water contamination incident, identify
potential revisions and corrections to the CMP, and
practice plans and procedures of various agencies.
This was practice for the Full Scale Field Exercise.
Provided GCWW field repose personnel the
opportunity to practice implementation of Site
Characterization procedures and equipment.
Provided GCWW and response partners the
opportunity to exercise their roles within a field
environment, test plans and procedures and
identify opportunities for improvement and potential
revisions to the CMP. Participants also practiced
communication and coordination techniques.
September 2008
109
-------
Cincinnati Pilot Post-Implementation System Status
8.2.4 Equipment
During the pre-implementation assessment, it was also noted that field response teams did not have an
established way to communicate with the Incident Commander and that the Incident Command System
did not have a way to communicate with response partners in the field. This greatly hindered the
organization's ability to respond to an incident and to communicate and coordinate appropriately. To
address this deficiency, eight (8) 800 MHz hand-held radios (Motorola XTS 5000) were acquired.
The 800 MHz radios are long-range multi-channel programmable units that are interoperable with
response agencies in the City of Cincinnati and Hamilton County. They can also be programmed to
operate on other agency frequencies in the event of a Unified Command System incident response. The
radios are located and deployed with, as well as maintained by, the GCWW Site Characterization Team
Leader.
8.2.5 Summary of Post-Implementation Status
EPA, with the assistance of GCWW, completed a comprehensive assessment of the existing response
plans, procedures, and equipment in place at GCWW. During this assessment, the existing practices and
systems were evaluated according to their applicability and contribution to the development of a
Consequence Management Plan. Following the completion of the assessment, gaps were identified where
EPA and GCWW felt changes or additions to the practices and systems in place at the utility to handle a
contamination threat to the distribution system could be improved (Table 8-4). EPA and GCWW worked
collaboratively to develop and implement solutions which would address the identified gaps. Table 8-7
presents a summary of the post-implementation status of utility's capability with respect to consequence
management activities.
Table 8-7. Consequence Management Component Post-Implementation Status
Attribute
1. Contaminant
Incident Response
Plans
2. Response Partner
Network
3. Training and
Exercises
4. Equipment
Description of Consequence Management Program Component
• Response planning documents developed during implementation of the
Cincinnati pilot were the Consequence Management Plan and the Crisis
Communication Plan.
• The Consequence Management Plan was developed to serve as a
preparedness and response guide in the event of a water contamination
incident.
. The Crisis Communication Plan was developed to formalize public notification
procedures and guide the actions of the GCWW Public Information Officer
during all phases of a potential contamination incident.
• GCWW and response partners established a network to better understand,
confirm, and integrate their roles and responsibilities in the event of a drinking
water contamination incident.
• Specific GCWW and response partner roles, responsibilities, and
communication requirements were documented in the CMP.
• Training and exercises were conducted to test the CMP and to train participants
on processes and procedures.
• Training consisted of workshops, table-top exercises, functional exercises, drills,
and full scale exercises.
. GCWW acquired eight (8) 800 MHz hand-held radios (Motorola XTS 5000) to
address the organization's ability to respond to an incident and to communicate
and coordinate appropriately with response partners in the field.
Figure 8-1 provides a summary of the level of effort associated with design and implementation of the
consequence management component for the Cincinnati pilot. Consequence management activities, as
summarized in Table 8-7, relied on support from EPA, GCWW, and local partners. The LOE for local
partners in all aspects of consequence management represents the combined efforts of more than a dozen
organizations from the local and state levels. The most significant effort was expended to develop the
GCWW Consequence Management Plan which outlines response plans for each component of the
contamination warning system. Considerable effort was also associated with training and exercises to test
the Consequence Management Plan, involving utility staff and all relevant local partners.
September 2008
110
-------
Cincinnati Pilot Post-Implementation System Status
n Local Partners LOE
• GCVWVLOE
DEPA LOE
Contaminant Incident Response Partner Network Training and Exercises
Response Plans
Equipment*
"Note: For equipment, level of effort for EPA and local partners is 0.1 staffyears, and GCWW LOE is 0.01 staffyears
Figure 8-1. Level of Effort for Design and Implementation of the Consequence Management
Component (December 2005 - December 2007)
Figure 8-2 presents a summary of the extramural costs associated with design and implementation of the
consequence management component for the Cincinnati pilot. As illustrated, extramural labor costs were
significant for development of the Consequence Management Plan and coordination efforts associated
with drills and exercises to test the Consequence Management Plan. Equipment costs included purchase
of hand-held radios for communication and coordination during emergency response. Costs associated
with contractor travel were not included in this calculation.
September 2008
111
-------
Cincinnati Pilot Post-Implementation System Status
pbOO •
5>4UU
(A
TJ
C
ra
(A
3
O $300 "
1-
_c
*->
(A
(\ q>zUU
$100
$0.8
S
• Equipment, Consumables,
and Purchased Services
n Extramural LOE
$0.07
2
$7
/
$- i i i i
Contaminant Incident Response Partner Network Training and Exercises* Equipment
Response Plans*
"Note: The equipment, consumables, and purchased service cost was $800 for Contaminant Incident Response Plans and $70 for
Training and Exercises
Figure 8-2. Extramural Costs Associated with Design and Implementation of the Consequence
Management Component (December 2005 - December 2007)
September 2008
112
-------
Cincinnati Pilot Post-Implementation System Status
Section 9.0: Path Forward
Implementation of the first contamination warning system pilot deployed under EPA's Water Security
Initiative in Cincinnati, Ohio was completed at the end of 2007. This deployment resulted in an
operational system consisting of multiple monitoring and surveillance strategies, including online water
quality monitoring, sampling and analysis, enhanced security monitoring, consumer complaint
surveillance, and public health surveillance.
The strength of this design lies in the integration of information from these multiple monitoring and
surveillance strategies through a comprehensive concept of operations and consequence management
plan. During the first phase of the Water Security initiative, simulations of the conceptual design
indicated that this approach is capable of achieving timely and reliable detection of a broad array of
contamination incidents. The Cincinnati pilot provides an opportunity to test and validate the conceptual
design developed during that initial phase of the program.
With the completion of design and installation activities, the Cincinnati pilot entered a period of
preliminary testing. During this phase, data is being collected from all system components to assess and
optimize performance. This data collection is one aspect of a robust evaluation process, which will
continue through the duration of the pilot. The results will be used to assess the overall performance and
sustainability of the system, as well as potential revisions or alternate approaches to contamination
warning system design. Preliminary findings from the Cincinnati pilot have already been incorporated
into three interim guidance documents:
• Water Security Initiative: Interim Guidance on Planning for Contamination Warning System
Deployment (EPA-817-R-07-002; 2007). Experience from implementation of the Cincinnati pilot
formed the basis for guidance on developing a comprehensive plan for a drinking water
contamination warning system, considering both the individual components and integrated
system. Common concepts emphasized in the document include an integrated project
management team; engaging IT staff early in the planning process; and component-specific
considerations that influence design and implementation activities.
• Water Security Initiative: Interim Guidance on Developing and Operational Strategy for
Contamination Warning Systems (EPA-817-R-08-002; 2008). The operational strategy
developed for the Cincinnati pilot provided a model for guidance on developing an integrated
operational strategy and component-specific standard operating procedures to guide routine
operation of the contamination warning system and the initial investigation of triggers generated
from monitoring and surveillance components. A case study based on a generalized example for
the Cincinnati pilot is included as an appendix.
• Water Security Initiative: Interim Guidance on Developing a Consequence Management Plan for
Contamination Warning Systems (EPA-817-R-08-001; 2008). The consequence management
plan developed for the Cincinnati pilot provided a model for guidance on developing a
consequence management plan for a contamination warning system. It includes concepts related
to site characterization, determination of credible and confirmed contamination, local response
partner roles and responsibilities, crisis communications, and public notification. The approach
recommended in the guidance is based on lessons learned from the Cincinnati pilot and relies on a
series of decision trees to guide response actions.
These guidance documents will play a key role in the deployment of up to four additional drinking water
contamination warning system pilots scheduled to begin in mid and late 2008. Collectively, the data from
all pilots implemented under the Water Security initiative will allow for a thorough evaluation of
contamination warning system performance and sustainability under a variety of conditions. Not only do
September 2008 113
-------
Cincinnati Pilot Post-Implementation System Status
the pilot utilities represent a range of distribution system designs and water quality baselines, but also a
variety of organizational structures, both within the utility and across partner organizations that play a key
role in system operation. Such a robust set of pilots will provide a basis for the development of products
and guidance that have broad applicability across the drinking water treatment and supply industry.
September 2008 114
-------
Cincinnati Pilot Post-Implementation System Status
References
Centers for Disease Control and Prevention. 2006. "Early Aberration Reporting System."
.
Centers for Disease Control and Prevention. 2007. "Nationally Notifiable Infectious Diseases."
.
Department of Homeland Security. 2006. "Homeland Security Advisory System—Guidance for Federal
Departments and Agencies."
.
Frontier Technology Incorporated. 2006. "H2O Sentinel Users Guide."
Foldy, S. 2004. "Linking Better Surveillance to Better Outcomes." Centers for Disease Control and
Prevention: Morbidity and Mortality Weekly Report, Vol. 53 (Supplement): 12-17.
GCWW. 2006. "GCWW Information Technology Strategic Plan." Internal GCWW document.
German, R.R. 2001. "Updated Guidelines for Evaluating Public Health Surveillance Systems
Recommendations from the Guidelines Working Group Guidelines Working Group." Centers for
Disease Control and Prevention: Morbidity and Mortality Weekly Report. Vol. 50: RR-13.
Hall, J., et al. 2007a. "Online Water Quality Parameters as Indicators of Distribution System
Contamination." Journal AWWA. Vol. 99, Iss. 1:66-77.
Hall, J. and J. Szabo. 2007b. "Proposed Design and Basis of Selection for WaterSentinel Online Water
Quality Monitoring Stations." USEPA-NHSRC.
Hart, D.B., S.A. McKenna, K.A. Klise, V. Cruz and M. Wilson. 2007. "CANARY: A Water Quality
Event Detection Algorithm Development Tool." Submitted to: ASCE World.
Henning, K. J. 2004. "Overview of Syndromic Surveillance: What is Syndromic Surveillance?" Centers
for Disease Control and Prevention: Morbidity and Mortality Weekly Report. Vol. 53
(Supplement): 5-11.
Hrudey, S.E., Huck P.M., Payment, P., Gillham, R.W., and Hrudey, E. J. 2002. "Walkerton: Lessons
learned in comparison with waterborne outbreaks in the developed world." J. Environ. Eng.
SciVRev. 1(6), 397-407.
Klise, K.A. and S.A. McKenna. 2006. "Water quality change detection: multivariate algorithms." In
Proceedings of 2006 SPIE (International Society for Optical Engineering), Defense and Security
Symposium. Orlando, Florida.
McKenna, S.A., D.B. Hart, K.A. Klise, V.A. Cruz and M.P. Wilson. 2007. "Event Detection from Water
Quality Time Series." In Proceedings of 2007 ASCE World Environmental and Water Resources
Congress. Tampa, FL.
National Environmental Laboratory Accreditation Conference (NELAC) Quality Systems. June 2003.
State of Ohio, Rule number 2745-89-03. Ohio Administrative Code (OAC), 2007.
September 2008 115
-------
Cincinnati Pilot Post-Implementation System Status
Pikus, I. M. 2004. Interim Voluntary Guidelines for Designing an Online Contaminant Monitoring
System. ASCE, USEPA-Sponsored Report.
SatScan™: Software for the Spatial, Temporal, and Space-time Scan Statistics. 2005.
.
Umberg, K. and Allgeier, S.C. 2007. "Evaluation of Tools to Detect Distribution System Water Quality
Anomalies." In Proceedings of 2007 AWWA Water Quality Technology Conference. Charlotte,
NC.
U.S. Environmental Protection Agency. 2003. "Response Protocol Toolbox: Planning for and
Responding to Drinking Water Contamination Threats and Incidents Module 3: Site
Characterization and Sampling Guide." Interim Final.
U.S. Environmental Protection Agency. 2005a. WaterSentinel System Architecture, Draft for Science
Advisory Board Review.
U.S. Environmental Protection Agency. 2005b. WaterSentinel Contaminant Selection, Draft for Science
Advisory Board Review. For Official Use Only.
U.S. Environmental Protection Agency. 2005c. Manual for the Certification of Laboratories Analyzing
Drinking Water Criteria and Procedures Quality Assurance (5th Edition).
U.S. Environmental Protection Agency. 2006a. Cooperative Research and Development Agreement with
the United States Environmental Protection Agency.
U.S. Environmental Protection Agency. 2006b. Framework for the Evaluation of Event Detection
Software for Drinking Water Contamination Warning Systems.
U.S. Environmental Protection Agency. 2006c. WaterSentinel Enhanced Security Assessment.
Proprietary document.
U.S. Environmental Protection Agency. 2006d. Consumer Complaint Surveillance Assessment Report.
Proprietary document.
U.S. Environmental Protection Agency. 2006e. Greater Cincinnati Water Works WaterSentinel
Consequence Management Plan, Draft July 7, 2006. Proprietary document.
U.S. Environmental Protection Agency. 2006f Greater Cincinnati Water Works Consequence
Management Plan, Appendix B: Site Characterization Plan, Draft November 2, 2006.
Proprietary document.
U.S. Environmental Protection Agency. 2007a. Water Security Initiative: Interim Guidance on Planning
for Contamination Warning System Deployment. EPA-817-R-07-002.
U.S. Environmental Protection Agency. 2007b. Concept of Operations for the GCWW Contamination
Warning System. Proprietary document.
U.S. Environmental Protection Agency. 2007c. Water Security Initiative-GCWW Centralized Hardware
and Software Operations and Maintenance Manual. Proprietary document.
U.S. Environmental Protection Agency. 2007d. Water Security Initiative Distribution System Studies and
Modeling: Tracer Study Report. Proprietary document.
September 2008 116
-------
Cincinnati Pilot Post-Implementation System Status
U.S. Environmental Protection Agency. 2007e. Threat Ensemble Vulnerability Assessment Research
Program .
U.S. Environmental Protection Agency. 2007f. Preliminary Sensor Network Design for Greater
Cincinnati Water Works. Proprietary document.
U.S. Environmental Protection Agency. 2007g. WS SCADAHM1Users' Guide. Proprietary document.
U.S. Environmental Protection Agency. 2007h. EDDIES 2.0 EDS Interface Specification.
U.S. Environmental Protection Agency. 2007i. EDDIES 3.0 EDS Interface Specification.
U.S. Environmental Protection Agency. 2007J. Plan for the Evaluation of Event Detection Software for
Drinking Water Contamination Warning Systems.
U.S. Environmental Protection Agency. 2007k. "October EDS Performance Summary." Proprietary
document.
U.S. Environmental Protection Agency. 20071. Assessment Report of Sampling and Analysis Activities at
the GCWW WS-CWSPilot. Proprietary document.
U.S. Environmental Protection Agency. 2007m. Standard Operating Procedure for Routine Pathogen
and Toxin Sampling, Packaging, and Shipping. Proprietary document.
U.S. Environmental Protection Agency. 2007n. Standard Operating Procedure for the Demonstration of
Capability Using the CDC/LRN Ultrafiltration Protocol: Determining Recovery of Enterococcus
faecalisfrom Water.
U.S. Environmental Protection Agency. 2007o. Baseline Monitoring at the Greater Cincinnati Water
Works Water Security - Contamination Warning System Pilot (Rev. 1.0). Proprietary document.
U.S. Environmental Protection Agency. 2007p. Supplemental Information for Water Security -
Contamination Warning System Targeted Contaminants Monitored During the Pilot at the
Greater Cincinnati Water Works. Proprietary document.
U.S. Environmental Protection Agency. 2007q. WS-CWS Pilot Study Data Management Plan.
Proprietary document.
U.S. Environmental Protection Agency. 2007r. Water Security initiative Enhanced Security Monitoring
Video Intrusion Alarm Assessment Operations Manual. Proprietary document.
U.S. Environmental Protection Agency. 2007s. Water Security Contamination Warning System,
Consumer Complaint Surveillance, Detailed Design Specification, Version 2.0, Draft July 23,
2007. Proprietary document.
U.S. Environmental Protection Agency. 2007t. Water Security Initiative Cincinnati Pilot Public Health
Surveillance User Interface Guide. Proprietary document.
U.S. Environmental Protection Agency. 2007u. Greater Cincinnati Water Works Consequence
Management Crisis Communication Plan, March 9, 2007. Proprietary document.
U.S. Food and Drug Administration. 2002. "Bioterrorism Act of 2002."
.
September 2008 117
------- |