EPA
United States
Environmental Protection
Agency
Water Security Initiative: Evaluation of the Customer
Complaint Surveillance Component of the Cincinnati
Contamination Warning System Pilot
Monitoring and Surveillance
Water Quality Monitoring
Enhanced Security Monitoring
Customer Complaint Surveillance
Public Health Surveillance
Possible Contamination
Consequence Management
Sampling and Analysis
Response
Office of Water (MC-140)
EPA-817-R-14-001D
April 2014
-------
-------
Disclaimer
The Water Security Division of the Office of Ground Water and Drinking Water has reviewed and
approved this document for publication. This document does not impose legally binding requirements on
any party. The findings in this report are intended solely to recommend or suggest and do not imply any
requirements. Neither the U.S. Government nor any of its employees, contractors or their employees
make any warranty, expressed or implied, or assumes any legal liability or responsibility for any third
party's use of or the results of such use of any information, apparatus, product or process discussed in this
report, or represents that its use by such party would not infringe on privately owned rights. Mention of
trade names or commercial products does not constitute endorsement or recommendation for use.
Questions concerning this document should be addressed to:
Nelson Mix, PE, CHMM
U.S. EPA Water Security Division
MC 4608T, 1200 Pennsylvania Ave, NW
Washington, DC 20460
(202)564-7951
Mix.Nel son@epa. gov
or
Steve Allgeier
U.S. EPA Water Security Division
26 West Martin Luther King Drive
Mail Code 140
Cincinnati, OH 45268
(513)569-7131
Allgeier.Steve@iepa.gov
-------
Acknowledgments
The Water Security Division of the Office of Ground Water and Drinking Water would like to recognize
the following individuals and organizations for their assistance, contributions, and review during the
development of this document.
Bill Fromme, Greater Cincinnati Water Works
Mark Menkhaus, Greater Cincinnati Water Works
Jeff Swertfeger, Greater Cincinnati Water Works
Yves Mikol, New York City Department of Environmental Protection
Tim Martin, New York City Department of Environmental Protection
Rita Kopansky, Philadelphia Water Department
John Vogtman, Philadelphia Water Department
David Harvey, U.S. Environmental Protection Agency
IV
-------
Executive Summary
The goal of the Water Security Initiative (WSI) is to design and demonstrate an effective multi-
component warning system for timely detection and response to drinking water contamination threats and
incidents. A contamination warning system (CWS) integrates information from multiple monitoring and
surveillance components to alert the water utility to possible contamination, and uses a consequence
management plan to guide response actions.
System design objectives for an effective CWS are: spatial coverage, contaminant coverage, alert
occurrence, timeliness of detection and response, operational reliability and sustainability. Metrics for the
customer complaint surveillance (CCS) component were defined relative to the system metrics common
to all components in the CWS, but the component metric definitions provide an additional level of detail
relevant to the CCS component. Evaluation techniques used to quantitatively or qualitatively evaluate
each of the metrics include analysis of empirical data from routine operations, drills and exercises,
modeling and simulations, forums and an analysis of lifecycle costs. This report describes the evaluation
of data collected from the CCS component from the period of January 2008 - June 2010.
The major outputs from the Cincinnati pilot evaluation include:
1. Cincinnati Pilot System Status, which describes the post-implementation status of the Cincinnati
pilot following the installation of all monitoring and surveillance components.
2. Component Evaluations, which include analysis of performance metrics for each component of
the Cincinnati pilot.
3. System Evaluation, which integrates the results of the component evaluations, the simulation
study and the benefit-cost analysis.
The reports that present the results from the evaluation of the system and each of its six components are
available in an Adobe portfolio, Water Security Initiative: Comprehensive Evaluation of the Cincinnati
Contamination Warning System Pilot (USEPA 2014a).
Customer Complaint Surveillance Component Design
A key component of the WSI design is CCS, which consists of the following design elements:
comprehensive complaint collection, electronic data management, automated and integrated data analysis
and component response procedures. As part of the Cincinnati pilot, the monitoring system developed for
the CCS component deployed at the Greater Cincinnati Water Works (GCWW) was based on the utility's
existing customer complaint management system. Prior to implementation of the CWS, GCWW had
systems and processes in place that could be enhanced and used to create an integrated CCS monitoring
system, but data was not automatically aggregated and analyzed, and notifications of potential incidents
were not sent. Although call volume reports were reviewed periodically by supervisors, event detection
relied exclusively on staff experience to recognize anomalies in customer complaint activity.
Three data streams were utilized for the CCS component: Interactive Voice Response (IVR) data, work
request data and work order data. As part of the CCS component, a new IVR option was added to the
GCWW automated call management system; a prompt specifically for water quality-related issues was
added to the menu. Additionally, work request classifications, generated after a Customer Service
Representative (CSR) interview with the customer, were expanded to include a water quality-related
option to aid data tracking. After generation of the work request, the CSR forwards the water quality calls
to a Water Quality & Treatment (WQ&T) chemist, who then determines if a work order is needed. The
work order represents the final data stream tracked as part of the CCS component. These data streams are
-------
analyzed independently using algorithms to identify anomalous patterns in the customer complaint data.
Once anomalies are identified, automated email alerts are sent to GCWW personnel who conduct an
investigation according to the Cincinnati Pilot Operational Strategy. For more information on this topic,
see Section 2.0.
A summary of the results used to evaluate whether the CCS component met each of the design objectives
relevant to the component is provided below.
Methodology
Several methods were used to evaluate CCS performance. Data was tracked over time to illustrate the
change in performance as the component evolved during the evaluation period. Statistical methods were
also used to summarize large volumes of data collected over either the entire or various segments of the
evaluation period. Data was also evaluated and summarized for each reporting period over the evaluation
period. In this evaluation, the term reporting period is used to refer to one month of data that spans from
the 16th of the indicated month to the 15th of the following month. Thus, the January 2008 reporting
period refers to the data collected between January 16th 2008 and February 15th 2008. Additionally, four
drills and two full-scale exercises designed around mock contamination incidents were used to practice
and evaluate the full range of procedures, from initial detection through response.
Because there were no contamination incidents during the evaluation period, there is no empirical data to
fully evaluate the detection capabilities of the component. To fill this gap, a computer model of the
Cincinnati CWS was developed and challenged with a large ensemble of simulated contamination
incidents in a simulation study. An ensemble of 2,015 contamination scenarios representing a broad range
of contaminants and injection locations throughout the distribution system was used to evaluate the
effectiveness of the CWS in minimizing public health and utility infrastructure consequences. The
simulations were also used for a benefit-cost analysis, which compares the monetized value of costs and
benefits and calculates the net present value of the CWS. Costs include implementation costs and routine
operation and maintenance labor and expenses, which were assumed over a 20 year lifecycle of the CWS.
Benefits included reduction in consequences (illness, fatalities and infrastructure damage) and dual-use
benefits from routine operations.
Design Objective: Spatial Coverage
Spatial coverage is the cumulative area of the distribution system where a detectable change in water
characteristics could potentially be reported via a customer complaint call. Spatial coverage is measured
by the metrics of area coverage, spatial extent of alerts and population coverage. The CCS component
covers the pilot utility's entire retail service area (100% area coverage). For the spatial extent of work
request and work order alerts, the area encompassed within the alerts tended to be larger for algorithms
with longer event detection timeframes, due to the greater number of complaints included in each alert.
Population coverage, as measured by available empirical data, is estimated as the percentage of GCWW
customers with access to a telephone (approximately 96%). For more information on this topic, see the
relevant subsections regarding spatial coverage in Sections 4.0 - 8.0.
Design Objective: Contaminant Coverage
Contaminant coverage is the ability to detect a wide range of water contaminants and is measured by the
following metrics: contamination scenario coverage and contaminant detection threshold. CCS is limited
to detecting contaminants that cause a discernible change in drinking water taste, odor, and/or appearance.
Simulation study results are used for metrics when there is no observed or recorded empirical data for
VI
-------
evaluation. Since there were no contamination incidents during the evaluation, water contamination was
simulated with six theoretically detectable contaminants to assess contaminant coverage and the
contaminant detection threshold. CCS detected every scenario which was theoretically detectable by the
component (i.e. contaminants present in the drinking water in sufficient concentrations to produce a
noticeable change in taste, odor or appearance). The simulation scenarios were selected to model water
contamination incidents which would have a significant impact on public health or infrastructure. At the
necessary concentrations for these impacts, CCS will most likely detect the contamination. IVR alerts
preceded work order alerts in 559 of 564 simulated scenarios.
The overall average of the number of exposures (simulated customers who have been exposed to a
contaminant in the drinking water) before the first IVR or work order alert is 195 with a median of 81
exposures. There is a median of 81 exposures before the first IVR alert and a median of 171 exposures
before the first work order alert. The difference between the numbers of exposures is an outcome of the
time delay between a customer calling (entering the IVR data stream) and creation of a work order.
While the work order data stream is timely, the delay does result in more customers potentially being
exposed to contamination.
Over 92% of the scenarios have fewer than 500 exposures prior to the first CCS alert. Figure ES-1
displays the distribution of exposures prior to the first CCS alert. The scenarios with a greater number of
exposures represent injections of large volumes of a contaminant in areas of dense population. In these
scenarios, the contaminant is able to reach a large number of people before the customers first begin
consuming the water.
1 n nnn
1 nnn -
-------
variability in the underlying data. Metrics for this design objective include invalid alerts, valid alerts and
sensitivity. Invalid and valid alert rates were characterized using empirical data gathered during the real-
time monitoring phase. Invalid alerts occurred frequently at the beginning of the evaluation period due to
intentionally low threshold levels which provided opportunities to rehearse alert investigation procedures.
Following threshold adjustment, invalid alerts occurred at a sustainable level of five to seven alerts per
month or fewer. Two valid alerts were observed over the evaluation period, representing 0.38% of the
total number of valid and invalid alerts. The CCS component was able to detect valid alerts in the form of
various system events including high chlorine levels and distribution system maintenance activities. For
more information on this topic, see the relevant subsections regarding alert occurrence in Sections 4.0 -
8.0.
Design Objective: Timeliness of Detection
For CCS, timeliness of detection and response refers to the time from when the component generates an
alert during a contamination incident to the time that an alert is confirmed via investigation. Factors that
impact this objective include: time for data transmission, time for event detection, time to recognize alerts
and time to investigate alerts. These metrics were characterized using empirical data. Data from drills
and exercises was used to evaluate the time to investigate valid alerts. The automated portions of the
CCS monitoring system (time for data transmission and time for event detection) occurred within
milliseconds and had a negligible impact on the overall CCS timeline. The median time for alert
recognition was 10 minutes for the IVR data stream and 14 minutes for the work order data stream. For
invalid alerts, the median CCS alert investigation time to determine whether water contamination was
possible ranged from 5-10 minutes based on empirical data. For valid alerts, the alert investigation time
ranged from 19-40 minutes based on CCS drill data, a timeframe greater than invalid alerts due to the
time needed for data confirmation and consultation with an increased number of utility personnel. Figure
ES-2 shows the timeline progression of the key alert investigation activities, as characterized by CCS
Drill 1. Note that the timeline was normalized so the start of the investigation occurs at time 0. This drill
scenario ends prior to a determination of Possible contamination because the WQ&T Chemist,
responsible for making the final determination, was not involved in CCS Drill 1.
00:15
00:29
Need for
Emergency IVR
Message
Determined
00:28
00:05 WQ&T Customer 00:20 Need for
00:00 Email query Water Quality Alert Verification Additional Call
1 -Day IVR sent to Supervisor Checklist Handling Staff
Alert CS
Rs Cont;
F 1
acted Comp
' 1
leted Determinec
r i
F 1
00:40
Call Center
Activities End
00:30 00:35
Work Request Work Order
Alert Developed for
- i
- i
Distribution Staff
' 1
00:00
00:40
Figure ES-2. Timeline of CCS Drill 1
For more information on this topic, see the relevant subsections regarding timeliness of detection in
Sections 4.0-8.0.
Design Objective: Operational Reliability
The metrics for operational reliability quantify the percent of time that the CCS component is working as
designed. Availability of the CCS component was measured to quantify operational reliability through
VIM
-------
analysis of empirical data. The CCS component exhibited excellent operational reliability and was
available during 99.6% of the evaluation period. There was one significant event in May 2008, where the
EDAs were taken offline for 67 hours due to a malfunction. This event contributed the majority of
downtime for the system. The limited amount of downtime experienced by the FVR component was due
to isolated, transient Web service failures. The work order and work request data streams downtime was
the result of a few isolated power outages. Table ES-1 presents the CCS subcomponent downtime hours
during the overall evaluation period, and the corresponding availability percentage.
Table ES-1. CCS Downtime Hours and Percent Available
Metric
Availability in terms of
Downtime Events
(Total Hours)
Percent Available
CCS Subcomponent
IVR
579 Hours
96.9%
Work Requests
57 Hours
99.8%
Work Orders
57 Hours
99.8%
For more information on this topic, see the relevant subsections regarding operational reliability in
Sections 4.0-8.0.
Design Objective: Sustainability
Sustainability is a key objective in the design of a CWS and each of its components, which for the
purpose of this evaluation is defined in terms of the cost-benefit trade-off. Costs are estimated over the
life-cycle of the system to provide an estimate of the total cost of ownership and include the
implementation costs, enhancement costs, operation and maintenance costs, renewal and replacement
costs, and the salvage value. The benefits derived from the system are defined in terms of primary and
dual-use benefits. Metrics that were evaluated under this design objective include: costs, benefits, and
acceptability. The costs used in the calculation of lifecycle costs for the CCS component are presented in
Table ES-2. These costs were tracked as empirical data during the design and implementation phase of
project design, and were analyzed through a benefit-cost analysis. It is important to note that the
Cincinnati CWS was a pilot research project, and as such incurred higher costs than would be
expected for atypical large utility installation.
Table ES-2. Cost Elements used in the Calculation of Lifecycle Cost
Parameter
Implementation Costs
Annual O&M Costs
Renewal and Replacement Costs1
Salvage Value1
Value
$1,037,591
$8,086
$231,419
-
Calculated using major pieces of equipment.
To calculate the total lifecycle cost of the CCS component, all costs and monetized benefits were adjusted
to 2007 dollars using the change in the Consumer Price Index (CPI) between 2007 and the year that the
cost or benefit was realized. Subsequently, the implementation costs, renewal and replacement costs, and
annual operation and maintenance costs were combined to determine the total lifecycle cost:
CCS Total Lifecyde Cost: $1,353,331
IX
-------
A similar CCS component implementation at another utility should be less expensive when compared to
the Cincinnati pilot as it could benefit from lessons learned and would not incur research-related costs.
Dual-use benefits and acceptability were evaluated through documentation of qualitative data during drills
and exercises, and during forums with the utility including lessons learned workshops. The use of rusty
water complaints captured through customer calls to identify potential distribution system issues
demonstrated a dual-use benefit. Acceptability was demonstrated through 100% utility participation in
drills and exercises, which required substantially more effort than routine investigations. Feedback from
GCWW personnel indicated that they were able to better understand standard operating procedures by
responding to the simulated water contamination incidents presented during the drills and exercises.
Furthermore, acceptability was evidenced by a high rate of alert investigations completed by utility
personnel during the evaluation period (87% alerts investigated or higher for all data streams). For more
information on this topic, see the relevant subsections regarding sustainability in Sections 4.0 - 8.0.
-------
Table of Contents
LIST OF FIGURES XIV
LIST OF TABLES XVI
SECTION 1.0: INTRODUCTION 1
1.1 CWS DESIGN OBJECTIVES 1
1.2 ROLE OF CCS IN THE CINCINNATI CWS 2
1.3 OBJECTIVES 2
1.4 DOCUMENT ORGANIZATION 2
SECTION 2.0: OVERVIEW OF THE CCS COMPONENT 4
2.1 COMPREHENSIVE COMPLAINT COLLECTION 5
2.2 ELECTRONIC DATA MANAGEMENT 6
2.3 AUTOMATED & INTEGRATED DATA ANALYSIS 6
2.4 COMPONENT RESPONSE PROCEDURES 8
2.5 SUMMARY OF SIGNIFICANT CCS COMPONENT MODIFICATIONS 9
2.6 TIMELINE OF CCS DEVELOPMENT PHASES AND EVALUATION-RELATED ACTIVITIES 11
SECTION 3.0: METHODOLOGY 12
3.1 ANALYSIS OF EMPIRICAL DATA FROM ROUTINE OPERATIONS 12
3.2 DRILLS AND EXERCISES 12
3.2.1 Full Scale Exercise 2 (October 1, 2008) 12
3.2.2 CCS-Site Characterization Drill (September 16, 2009) 13
3.2.3 Full Scale Exercise 3 (October 21, 2009) 13
3.2.4 CCS Drills (1-3) 13
3.3 SIMULATION STUDY 14
3.4 FORUMS 17
3.5 ANALYSIS OF LIFECYCLE COSTS 17
SECTION 4.0: PERFORMANCE OF IVR DATA STREAM 19
4.1 DESCRIPTION OF THE IVR DATA STREAM 19
4.2 DESIGN OBJECTIVE: SPATIAL COVERAGE 20
4.3 DESIGN OBJECTIVE: ALERT OCCURRENCE 20
4.3.1 Invalid Alerts 20
4.3.2 Summary 24
4.4 DESIGN OBJECTIVE: TIMELINESS OF DETECTION 24
4.4.1 Time for Alert Recognition 24
4.4.2 Time to Investigate Alerts 25
4.4.3 Summary 27
4.5 DESIGN OBJECTIVE: OPERATIONAL RELIABILITY 27
4.5.1 Availability 27
4.5.2 Summary 28
4.6 DESIGN OBJECTIVE: SUSTAINABILITY 28
4.6.1 Acceptability 28
4.6.2 Summary 30
4.7 IVR SUBMENU AND KEYSTROKE DATA ANALYSIS 30
4.7.1 IVR Menu 5 Analysis 30
4.7.2 IVR 5-4 Submenu Analysis 34
4.7.3 Summary 36
SECTION 5.0: PERFORMANCE OF WORK REQUEST DATA STREAM 37
5.1 DESCRIPTION OF THE WORK REQUEST DATA STREAM 37
5.2 DESIGN OBJECTIVE: SPATIAL COVERAGE 37
XI
-------
5.2.1 Spatial Extent of Invalid Alerts 38
5.2.2 Summary 39
5.3 DESIGN OBJECTIVE: ALERT OCCURRENCE 39
5.3.1 Invalid Alerts 39
5.3.2 Summary 42
5.4 DESIGN OBJECTIVE: TIMELINESS OF DETECTION 42
5.4.1 Time for Alert Recognition 43
5.4.2 Time to Investigate Alerts 43
5.4.3 Summary 44
5.5 DESIGN OBJECTIVE: OPERATIONAL RELIABILITY 44
5.5.1 Availability 44
5.5.2 Summary 45
5.6 DESIGN OBJECTIVE: SUSTAINABILITY 45
5.6.1 Acceptability 45
5.6.2 Summary 47
SECTION 6.0: PERFORMANCE OF WORK ORDER DATA STREAM 48
6.1 DESCRIPTION OF THE WORK ORDER DATA STREAM 48
6.2 DESIGN OBJECTIVE: SPATIAL COVERAGE 48
6.2.1 Spatial Extent of Invalid Alerts 49
6.2.2 Summary 50
6.3 DESIGN OBJECTIVE: ALERT OCCURRENCE 50
6.3.1 Invalid Alerts 50
6.3.2 Summary 52
6.4 DESIGN OBJECTIVE: TIMELINESS OF DETECTION 52
6.4.1 Time for Alert Recognition 52
6.4.2 Time to Investigate Alerts 53
6.4.3 Summary 54
6.5 DESIGN OBJECTIVE: OPERATIONAL RELIABILITY 54
6.5.1 Availability 54
6.6 DESIGN OBJECTIVE: SUSTAINABILITY 55
6.6.1 Acceptability 55
6.6.2 Summary 56
SECTION 7.0: PERFORMANCE OF THE INTEGRATED COMPONENT 57
7.1 DESIGN OBJECTIVE: SPATIAL COVERAGE 57
7.1.1 Area Coverage 57
7.1.2 Population Coverage 57
7.1.3 Summary 58
7.2 DESIGN OBJECTIVE: CONTAMINANT COVERAGE 58
7.2.7 Contamination Scenario Coverage 58
7.2.2 Contaminant Detection Threshold 59
7.2.3 Valid Alerts 60
7.2.4 Summary 61
7.3 DESIGN OBJECTIVE: TIMELINESS OF DETECTION 61
7.3.1 Time for Data Transmission 61
7.3.2 Time for Event Detection 62
7.3.3 Time for Initial Detection 62
7.3.4 Time to Validate Possible Contamination 63
7.3.5 Summary 66
7.4 DESIGN OBJECTIVE: OPERATIONAL RELIABILITY 66
7.4.1 Rate of Work Requests Producing Work Orders 66
7.4.2 Rate of Work Orders Preceded by Work Requests 67
7.4.3 Component Availability 68
7.4.4 Summary 68
7.5 DESIGN OBJECTIVE: SUSTAINABILITY 68
XII
-------
7.5.1 Costs 69
7.5.2 Benefits 72
7.5.3 Summary 74
SECTION 8.0: SUMMARY & CONCLUSIONS 75
8.1 DESIGN OBJECTIVE: SPATIAL COVERAGE 75
8.2 DESIGN OBJECTIVE: CONTAMINANT COVERAGE 75
8.3 DESIGN OBJECTIVE: ALERT OCCURRENCE 76
8.4 DESIGN OBJECTIVE: TIMELINESS OF DETECTION 76
8.5 DESIGN OBJECTIVE: OPERATIONAL RELIABILITY 77
8.6 DESIGN OBJECTIVE: SUSTAINABILITY 77
SECTION 9.0: REFERENCES 79
SECTION 10.0: ABBREVIATIONS 80
SECTION 11.0: GLOSSARY 81
XIII
-------
List of Figures
FIGURE 2-1. CUSTOMER COMPLAINT CALL PROCESS 5
FIGURE 2-2. TIMELINE OF CCS COMPONENT ACTIVITIES 11
FIGURE 4-1. IVR INVALID ALERTS PER REPORTING PERIOD 21
FIGURE 4-2. HISTOGRAM OF IVR INVALID ALERTS PER REPORTING PERIOD 22
FIGURE 4-3. EXPECTED ALERTS AT VARIOUS THRESHOLDS FOR HISTORICAL IVR 5 DATA 23
FIGURE 4-4. EXPECTED ALERTS AT VARIOUS THRESHOLDS FOR HISTORICAL IVR 5-4 DATA 24
FIGURE 4-5. IVR ALERT RECOGNITION TIME, BY ALGORITHM 25
FIGURE 4-6. TIMELINE PROGRESSION OF THE CCS ALERT INVESTIGATION DURING CCS DRILL 3 26
FIGURE 4-7. IVR AVERAGE INVESTIGATION TIME PER ALERT, BY ALGORITHM 27
FIGURE 4-8. IVR AND CIS DOWNTIME 28
FIGURE 4-9. INVESTIGATED IVR ALERTS BY ALGORITHM 29
FIGURE 4-10. IVR ALERT INVESTIGATIONS COMPLETED AND ALERTS NOT INVESTIGATED 30
FIGURE 4-11. ANNUAL IO-DAY MOVING AVERAGE OF DAILY IVR 5 SELECTIONS 31
FIGURE 4-12. HISTOGRAM OF DAILY KEYSTROKE 6 COUNTS 32
FIGURE 4-13. Box AND WHISKERS PLOT OF IVR 5 AND KEYSTROKE 6 DATA 32
FIGURE 4-14. PROPORTION OF IVR SUBMENU OPTIONS 34
FIGURE 4-15. FREQUENCY OF DAILY NUMBER OF WATER QUALITY COMPLAINTS FOR EACH DATA STREAM 35
FIGURE 4-16. 10-DAY MOVING AVERAGE OF IVR 5-4 AND CSR KEYSTROKES 36
FIGURE 5-1. SPATIAL EXTENT OF WORK REQUEST FALSE ALERTS 39
FIGURE 5-2. WORK REQUEST FALSE ALERTS PER REPORTING PERIOD 40
FIGURE 5-3. HISTOGRAM OF WORK REQUEST INVALID ALERTS PER REPORTING PERIOD 41
FIGURE 5-4. EXPECTED ALERTS AT VARIOUS THRESHOLDS FOR HISTORICAL WORK REQUEST DATA 42
FIGURE 5-5. TIME FOR WORK REQUEST ALERT INVESTIGATION 44
FIGURE 5-6. DOWNTIME OF THE HYDRA SYSTEM 45
FIGURE 5-7. WORK REQUEST ALERT INVESTIGATIONS COMPLETED BY ALGORITHM 46
FIGURE 5-8. INVESTIGATIONS COMPLETED AND ALERTS NOT INVESTIGATED, BY REPORTING PERIOD 46
FIGURE 6-1. SPATIAL EXTENT OF WORK ORDER ALERTS 49
FIGURE 6-2. WORK ORDER INVALID ALERTS PER REPORTING PERIOD 51
FIGURE 6-3. HISTOGRAM OF WORK ORDER INVALID ALERTS PER REPORTING PERIOD 51
FIGURE 6-4. EXPECTED ALERTS AT VARIOUS THRESHOLDS FOR HISTORICAL WORKORDERDATA 52
FIGURE 6-5. WORK ORDER ALERT RECOGNITION TIMES DURING REAL-TIME MONITORING 53
FIGURE 6-6. AVERAGE TIME TO INVESTIGATE WORK ORDER INVALID ALERTS 54
FIGURE 6-7. WORK ORDER ALERT INVESTIGATIONS CONDUCTED 55
FIGURE 6-8. WORK ORDER ALERTS INVESTIGATED AND NOT INVESTIGATED BY REPORTING PERIOD 56
FIGURE 7-1. POPULATION DENSITY OF THE GCWW SERVICE AREA 58
XIV
-------
FIGURE 7-2. NUMBER OF EXPOSURES PRIOR TO THE FIRST CCS ALERT 60
FIGURE 7-3. TIME FOR INITIAL DETECTION 63
FIGURE 7-4. TIMELINE PROGRESSION OF THE CCS ALERT INVESTIGATION DURING CCS DRILL 1 65
FIGURE 7-5. TIMELINE PROGRESSION OF THE CCS ALERT INVESTIGATION DURING FSE 2 65
FIGURE 7-6. TIMELINE PROGRESSION OF THE CCS ALERT INVESTIGATION DURING CCS DRILL 2 65
FIGURE 7-7. TIMELINE PROGRESSION OF THE CCS ALERT INVESTIGATION DURING FSE 3 66
FIGURE 7-8. PERCENT OF WORK REQUESTS CONVERTED TO WORK ORDERS 67
FIGURE 7-9. PERCENT OF WORK ORDERS PRECEDED BY WORK REQUESTS 68
FIGURE 7-10. EXPECTED ALERTS AT VARIOUS THRESHOLDS FOR RUSTY WATER COMPLAINTS 73
xv
-------
List of Tables
TABLE 2-1. CUSTOMER COMPLAINT SURVEILLANCE DESIGN ELEMENTS 4
TABLE 2-2. ED A DEPLOYED FOR THE CCS COMPONENT 7
TABLE 2-3. ROLES AND RESPONSIBILITIES FOR ROUTINE OPERATION OF THE CCS COMPONENT 8
TABLE 2-4. CCS MAJOR COMPONENT MODIFICATIONS 9
TABLE 3-1. CCS DRILL VARIATIONS 14
TABLE 3-2. ASSUMED CONTAMINANT CHARACTERISTICS FOR CONTAMINANTS DETECTABLE BY THE CCS
COMPONENT 16
TABLE 4-1. TIME FOR IVR ALERT RECOGNITION (MINUTES) 25
TABLE 4-2. TIME FOR IVR ALERT INVESTIGATION (MINUTES) 26
TABLE 4-3. BUSINESS HOURS AND NON-BUSINESS HOURS INVESTIGATIONS 29
TABLE 4-4. SUMMARY STATISTICS OF IVR 5 AND KEYSTROKE 6 DATA 31
TABLE 4-5. DAYS WITH GREATEST VOLUME OF IVR 5 SELECTIONS CORRELATED TO KEYSTROKE 6 COUNTS 33
TABLE 4-6. SUMMARY STATISTICS OF IVR AND KEYSTROKE DATA 34
TABLE 4-7. CORRELATION SUMMARY: IVR 5-4 AND KEYSTROKE 6 DATA 35
TABLE 5-1. TIME FOR WORK REQUEST ALERT INVESTIGATION 43
TABLE 6-1. TIME FOR WORK ORDER ALERT RECOGNITION (MINUTES) 53
TABLE 6-2. TIME FOR WORK ORDER ALERT INVESTIGATION (MINUTES) 54
TABLE 7-1. ELEVATED CHLORINE ALERTS 60
TABLE 7-2. DISTRIBUTION WORK ALERTS 61
TABLE 7-3. CCS DRILL METRICS 64
TABLE 7-4. COST ELEMENTS USED IN THE CALCULATION OF LIFECYCLE COST 69
TABLE 7-5. IMPLEMENTATION COST 70
TABLE 7-6. ANNUAL O&M COSTS 71
TABLE 7-7. EQUIPMENT COSTS 71
TABLE 7-8. POSITIVE PREDICTIVE VALUE OF RUSTY WATER WORK REQUEST ALERTS 73
TABLE 8-1. EVALUATION OF SPATIAL COVERAGE METRICS 75
TABLE 8-2. EVALUATION OF CONTAMINANT COVERAGE METRICS 76
TABLE 8-3. EVALUATION OF ALERT OCCURRENCE METRICS 76
TABLE 8-4. EVALUATION OF TIMELINESS METRICS 77
TABLE 8-5. EVALUATION OF OPERATIONAL RELIABILITY METRICS 77
XVI
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 1.0: Introduction
The purpose of this document is to describe the evaluation of the customer complaint surveillance (CCS)
component of the Cincinnati pilot, the first such pilot deployed by the US Environmental Protection
Agency's (EPA) Water Security Initiative. This evaluation was implemented by examining the
performance of the CCS component relative to the design objectives established for the contamination
warning system (CWS).
1.1 CWS Design Objectives
The Cincinnati CWS was designed to meet six overarching objectives, which are described in detail in
WaterSentinel System Architecture (USEPA, 2005) and are presented briefly below:
Spatial Coverage. The objective for spatial coverage is to monitor the entire population served
by the drinking water utility. It depends on the location and density of monitoring points in the
distribution system and the hydraulic connectivity of each monitoring location to downstream
regions and populations. Metrics evaluated under this design objective include: spatial extent of
alert and population coverage.
Contaminant Coverage. The objective for contaminant coverage is to provide detection
capabilities for all priority contaminants. This design objective is further defined by binning the
priority contaminants into 12 classes according to the means by which they might be detected
(USEPA, 2005). Use of these detection classes to inform design provides more comprehensive
coverage of contaminants of concern than would be achieved by designing the system around a
handful of specific contaminants. Contaminant coverage depends on the specific data streams
analyzed by each monitoring and surveillance component, as well as the specific attributes of
each component. Metrics evaluated under this design objective include: contaminant detection
potential, contamination scenario coverage and contaminant detection threshold.
Alert Occurrence. The objective for alert occurrence is to minimize the rate of invalid alerts
(alerts unrelated to contamination or other anomalous conditions) while maintaining the ability of
the system to detect real incidents. It depends on the quality of the underlying data as well as the
event detection systems that continuously analyze that data for anomalies. Metrics evaluated
under this design objective include: invalid alerts and valid alerts.
Timeliness of Detection. The objective of this aspect of system design is to provide initial
detection of a contamination incident in a timeframe that allows for the implementation of
response actions that result in significant consequences reduction. For monitoring and
surveillance components, such as CCS, this design objective addresses only detection of an
anomaly and investigation of the subsequent alert. Metrics associated with timeliness of
detection include: time for initial detection and time to investigate an alert. Timeliness of
response is addressed under consequence management (CM) and sampling and analysis (S&A).
Operational Reliability. The objective for operational reliability is to achieve a sufficiently high
degree of system availability, data completeness and data accuracy such that the probability of
missing a contamination incident becomes exceedingly low. It depends on the redundancies built
into the CWS and each of its components. Availability is evaluated under this design objective.
Sustainability. The objective of this aspect of system design is to develop a CWS that provides
benefits to the utility and partner organizations while minimizing the costs. This can be achieved
through leveraging of existing systems and resources that can readily be integrated into the design
of the CWS. Furthermore, a design that results in dual-use applications that benefit the utility in
day-to-day operations, while also providing the capability to detect intentional or accidental
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
contamination incidents, will also improve sustainability. Metrics evaluated under this design
objective include: lifecycle costs, benefits and acceptability.
The design objectives provide a basis for evaluation of each component, in this case CCS, as well as the
entire integrated system. Because the deployment of a drinking water CWS is a new concept, design
standards or benchmarks are unavailable. Thus, it is necessary to evaluate the performance of the pilot
CWS in Cincinnati against the design objectives relative to the baseline state of the utility prior to CWS
deployment.
1.2 Role of CCS in the Cincinnati CWS
CCS is one of the monitoring and surveillance components of the CWS that also includes online water
quality monitoring (WQM), enhanced security monitoring (ESM) and public health surveillance (PHS).
CCS involves the systematic tracking and analysis of customer complaint calls in order to detect potential
contamination within a water system. Integration of the customer complaint data with the observations
from the other components aims to detect water contamination in an efficient and effective manner for
better protection of the public.
Drinking water customers can provide near real-time input regarding changes in water characteristics
discernible through the senses. Changes in the taste, color, clarity and odor of water may elicit
complaints from water utility customers, indicating the possible presence of water contamination. These
complaints are continually collected and monitored by the water utility; hence, customer complaints may
provide one of the earliest warnings of a possible contamination incident. Since drinking water customers
are spread throughout the distribution system, an effective method for monitoring complaints monitors
not only the type of contamination, but also the area affected. By itself, and coupled with the other
components, CCS is valuable to a successful CWS.
1.3 Objectives
The overall objective of this report is to demonstrate how well the CCS component functioned as part of
the CWS deployed in Cincinnati (i.e., how effectively the component achieved the design objectives).
This evaluation will describe how the deployed CCS component could reliably detect a possible drinking
water contamination incident based on the standard operating procedures (SOPs) established for the
Cincinnati pilot. Although no known contamination incidents occurred during the pilot period, data
collection during routine operation, drills and exercises, and computer simulations yielded sufficient data
to evaluate performance of the CCS component against each of the stated design objectives. In summary,
this document will discuss the approach for analysis of this information and present the results that
characterize the overall operation, performance and sustainability of the CCS component of the Cincinnati
CWS.
1.4 Document Organization
This document discusses the approach for analysis and integration of this information to assess the overall
operation, performance and sustainability of the CCS component as part of the Cincinnati CWS.
Evaluation is addressed in the following sections:
Section 2: Overview of the CCS Component. This section introduces the CCS component of
the Cincinnati CWS and describes each of the major design elements that make up the
component. A summary of significant modifications to the component that had a demonstrable
impact on performance is presented at the end of this section.
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 3: Methodology. This section describes the techniques and methods used to evaluate
the CCS component.
Sections 4 through 6: Performance of the CCS Data Streams. These sections include an
evaluation of each CCS data stream individually (interactive voice response (IVR), work request
and work order data). Each section describes the data stream and offers an evaluation of its
function based on the metrics of contaminant coverage, spatial coverage, timeliness of detection
and reliability.
Section 7: Performance of the Integrated Component. This section includes a thorough
evaluation of the integrated functionality of the CCS data streams used in the Cincinnati pilot,
including how the component operates as a whole to identify possible contamination incidents.
Section 8: Summary & Conclusions. This section summarizes the overall operation,
performance and sustainability of the CCS component.
Section 9: References. This section lists all sources and documents cited throughout this report.
Section 10: Abbreviations. This section lists all acronyms approved for use in the CCS
component evaluation.
Section 10: Glossary. This section defines terms used throughout the CCS component
evaluation.
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 2.0: Overview of the CCS Component
The Cincinnati CWS involved a modification of the existing customer complaint management system.
Prior to implementation of the CWS, Greater Cincinnati Water Works (GCWW) had systems and
processes in place that could be enhanced and used to create an integrated CCS monitoring system, but
data was not automatically aggregated and analyzed. Also, notifications of potential incidents were not
automatically sent to appropriate personnel. Although call volume reports were reviewed periodically by
supervisors, event detection relied exclusively on staff experience to recognize anomalies in customer
complaint activity.
Major enhancements for the CCS component included the automation of data collection, identification of
water quality complaints from the IVR system, and automated analysis of data using event detection
algorithms (EDAs). Enhancements also included the automatic alert notification via email and text
message, the addition of spatial analysis capabilities, and the development the Cincinnati Pilot
Operational Strategy to streamline the processes of alert investigation. The Cincinnati Pilot Operational
Strategy describes both the roles and responsibilities of various job functions within the CCS as well as
the processes and procedures involved in its operation.
The CCS component of the Cincinnati CWS was fully deployed and operational by the end of 2007 and a
detailed description of the system at this point in the project can be found in Water Security Initiative:
Cincinnati Pilot Post-Implementation System Status (USEPA, 2008). During the next phase of the pilot,
the evaluation period from January 2008 through June 2010, the system was modified in an effort to
optimize performance and then analyzed.
The four main design elements for the Cincinnati pilot are described in Table 2-1. Sections 2.1 through
2.4 provide an overview of each of the four CCS design elements, with an emphasis on changes to the
component during the evaluation period. Section 2.5 summarizes all significant modifications to the CCS
component that are relevant to the interpretation of the evaluation results presented in this report.
Table 2-1. Customer Complaint Surveillance Design Elements
Design Element
Comprehensive
Complaint Collection
Electronic Data
Management
Automated & Integrated
Data Analysis
Component Response
Procedures
Description
A "funnel" for collecting all water quality complaints into the CCS monitoring system.
An example is a unified call center with a widely publicized telephone number in
place to capture the largest percentage of potential complaints.
All water quality complaints are entered into an electronic database when received
and categorized by type. A complaint record is carried through the process with
information added to it as it is received or as investigations are conducted.
As data is captured in electronic format, automated EDAs indicate when CCS data
reach pre-determined thresholds, signaling the need for human involvement in the
assessment process. When thresholds are exceeded, notifications are sent to
appropriate personnel, and complaint spatial information is displayed for easy
identification of clustering events.
Written SOPs exist for every step in the water quality complaint handling process and
for assessing alerts. These procedures outline effective and timely communications,
including clear guidance on appropriate response actions.
Many users with different job functions are involved in continued operation and maintenance of the CCS
component.
-------
2.1
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Comprehensive Complaint Collection
GCWW routinely receives customer calls from residential, commercial and industrial customers via a
single, widely published utility phone number. During business hours, customer calls are routed to the
GCWW call center, where a CSR from the Commercial Services Division handles the call. During non-
business hours, customer calls are routed to the distribution dispatcher, a Distribution Division position,
who directs and coordinates the work performed on the distribution system. Prior to implementing the
CCS component, GCWW used informal call management protocols and operating procedures, and many
of the procedures were not documented.
During normal business hours, all customer calls are directed to individual CSRs through an automated
call management system using an IVR system. If the CSR is unable to resolve the problem, or the
customer indicates that their water may be contaminated, the call is forwarded to Water Quality and
Treatment (WQ&T) personnel who determine whether it is necessary to generate a work order. This
process and the approximate times required for each step are shown in Figure 2-1.
Person perceives
change(s) in drinking
water characteristics
Customer calls
GCWW
CSR identifies
WQ issue
WQ&T Chemist
confirms WQ
issue
5 minutes
4 minutes
15 minutes
Data available for analysis
Work Order
data
Figure 2-1. Customer Complaint Call Process
During non-business hours, the IVR system informs customers to call back during normal business hours
for all non-emergency issues. For emergencies, such as water main breaks or water quality issues, the call
is transferred to distribution dispatch because this position is staffed 24 hours a day and is capable of
responding to emergencies. The distribution dispatcher also receives emergency calls directly from other
GCWW staff and from the City of Cincinnati's 311 Customer Service and Communications Center. Non-
emergency calls are also received by the dispatcher, since customers can select '0' on the IVR system if
they desire to talk with someone at GCWW. In either case, the dispatcher evaluates the call type and
urgency of the call.
The original IVR system script did not have a menu selection specifically for water quality concerns or
complaints. Since the CCS design required that calls be filtered for water quality complaints, an IVR
selection (number "5") was added to the menu for water quality issues in September 2006. Subsequently,
from April - May 2009, four submenu selections were added under IVR 5 to include screening for
rusty/cloudy water; pressure issues; general inquiries; and taste, odor or appearance complaints. The IVR
5 and submenu IVR 5-4 for taste, odor or appearance were integral additions to the water quality call
tracking process.
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
2.2 Electronic Data Management
Prior to implementation of the CWS, a customer service software application called Customer
Information System (CIS) was used by the CSRs to link incoming calls to a GCWW account. The IVR
software used a table in the CIS database to write the caller's IVR menu choice. Additionally, when
CSRs completed a session with a customer, they entered a note in the customer's CIS account and hit a
keystroke on their phone to code the types of issues handled during the call. This was captured by the call
management system as a 'stroke' count, and was used by the call center supervisor to track the types of
calls received in the Call Center. The CSRs would also enter the call into the GCWW's Distribution
Work Creation (DWC) work management application to create a work request for distribution system
infrastructure issues (main breaks, etc.).
As part of the CWS, an automated, near real-time data collection process to obtain water quality
complaint data from GCWW systems was developed using a Web services application, which is a
mechanism for machine-to-machine interaction over a network application. Once retrieved, the data was
stored locally and analyzed using a variety of statistical ED As.
IVR 5-4 selections (water quality complaints), water quality work request and water quality work order
information passes through the Web services every 60 seconds and is stored in a central location. This
information can then be analyzed independently to identify anomalous patterns in the customer complaint
data. Collectively, the data represent customer complaints from customer-identified water quality issues
to WQ&T treatment identified potential issues (work orders).
2.3 Automated & Integrated Data Analysis
EDAs, a collection of algorithms that analyze data collected from the utility's source databases, were
deployed as part of implementation of the CCS monitoring system. The EDAs automate the data analysis
and alert process based on pre-established frequencies of water quality-related calls, work requests and
work orders. The automated EDAs provide a dependable, robust surveillance system that is not
susceptible to human errors that can occur when information is transferred among a large number of staff
and across shift changes. Once an alert is generated, a notification is automatically sent to appropriate
personnel who perform the subsequent investigation.
Five algorithms, described in Table 2-2, were applied to three data streams (IVR 5-4 counts, water
quality-related work request counts and water quality-related work order counts), for a total of 15
methods of event detection. Because some algorithms are designed to operate only on weekends or only
on weekdays, not all event detection methods are operating at all times.
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Table 2-2. EDA Deployed for the CCS Component
Algorithm
Description
One-day Scan, Weekday
A prospective scan statistic monitors current data and evaluates it
Two-day Scan
against past data occurring within a specified time window. Separate
alert, or threshold, levels were established for one, two and seven days.
Seven-day Scan
A separate scan statistic is applied to weekend and holiday customer
complaint data, where call volumes are significantly less than during
One-day bean, Weekend normal business hours. Separate threshold levels were developed for
these days.
CUSUM
A cumulative sum (CUSUM) accumulates the difference between an
observed number of complaints per day and a reference value. If the
accumulation exceeds the threshold, an alert is given. Because the
observations can accumulate overtime, the CUSUM method can
detect slowly worsening situations earlier than a single day alert.
If the threshold is not reached, an alert is not generated and the EDA will continue to monitor the data
streams. Once the threshold has been reached, an electronic alert is generated and automated notification
of the CSR Supervisor and Distribution Dispatcher Supervisor occurs via email and cell phone text
message.
Several changes were made to the alert algorithms throughout the evaluation period. Initially, thresholds
were set artificially low to test the level of acceptability (i.e., how many alerts the utility could reasonably
respond to in a day without compromising regular service), then raised to levels deemed acceptable by the
utility. During the January 2009 reporting period, the utility disabled the seven-day scan and CUSUM
algorithm alert notifications, reasoning that the seven-day scan algorithm was not needed because it
exceeded the average age of water in the system and was redundant with the one and two-day scan alerts.
The utility favored the simplicity of interpretation of the scan window algorithm alerts over the complex
variable windows of the CUSUM algorithm alerts.
The DWC application provided some geographic information system (GIS) ability when creating work
requests and work orders. The GIS application, called Hydra Map, allowed the user to view a pre-defined
area around the location and showed areas of the distribution system construction in the immediate
vicinity of the work request or work order location. This application was modified to include a GIS layer
which was automatically updated with the locations of work requests, work orders and their
corresponding alert algorithm, to aid investigation.
GCWW's existing email system was utilized to distribute alert notifications. Using defined email groups,
targeted notification emails for each category of EDA alerts were sent to the utility personnel responsible
for further alert investigation. As a mobile workforce, GCWW personnel do not always have access to
email. Consequently, text messaging was also incorporated as part of a redundant notification strategy.
Email notifications were divided into four general groups: business hours, business hours (cell format),
after-hours and after-hours (cell format).
Numerous enhancements to the email notifications were requested during the evaluation period. The
majority of requests asked for more information on the work request or work orders generating the alert to
be included in the email notification. Additionally, the utility personnel instructions included in the
notification were clarified to ensure proper execution of all investigations.
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
2.4
Component Response Procedures
To capture the routine operation of the CCS component leading up to and after issuance of an alert
notification, GCWW developed detailed Operational Strategy procedures. The Cincinnati Pilot
Operational Strategy describes the process and procedures involved in the operation of the CCS
component, including the initial investigation and validation of a CCS alert. The Cincinnati Pilot
Operational Strategy establishes specific roles and responsibilities, and details procedural and information
flow descriptions. The procedural flow concludes with the determination of whether or not the CCS alert
is indicative of a possible water contamination incident. When a possible contamination incident is
identified, CM actions begin. For alert investigation, a series of checklists were established to support the
investigation of CCS alerts.
The CSR Supervisor investigates FVR alerts, while the WQ&T Chemist investigates work request and
work order alerts. Essentially, for all alerts, the primary investigator determines if the complaints have
similar descriptions and are spatially clustered based on utility knowledge of the distribution system. The
WQ&T Chemist examines the work request or work order information through the Hydra Map. The CSR
Supervisor polls the CSRs as to the nature and locations of the complaints. Since implementation, the
majority of alerts have had complaints that were not clustered or have had different complaint
descriptions. Investigations are closed once the determination is made that contamination is not possible.
If the complaints have similar descriptions and are clustered, the investigator determines if there has been
work in the area that could explain the complaints. If no work is being done in the area, the WQ&T
Chemist inspects water quality data and operational changes to explain the changes in water quality. If no
benign explanation is found for the clustered complaints, contamination is deemed Possible. The Water
Utility Emergency Response Manager (WUERM) is then notified and Cincinnati Pilot Consequence
Management Plan (CMP) is implemented. Table 2-3 lists key roles and responsibilities for the CCS
component.
Table 2-3. Roles and Responsibilities for Routine Operation of the CCS Component
GCWW User
WUERM
WQ&T Chemist
WQ&T Shift Chemist
WQ&T Customer Water
Quality Representative
Role in Operation of Customer Complaint Surveillance
Receives water quality work request and work order alerts;
Assumes the lead in the credibility determination process, as outlined in the CMP,
once notified of a Possible contamination incident; and,
Implements the CMP as necessary.
Receives and leads the investigation of the water quality work request and work
order alerts;
Coordinates support from Control Operators, Distribution Dispatchers and the CSR
Supervisor during investigation of the CCS alert;
Notifies the WUERM if the determination is made that contamination is possible.
Assumes CWS responsibilities of WQ&T Chemist during off-hours; support sample
analysis.
Collects detailed information from customers reporting water quality concerns.
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
GCWW User
Distribution Dispatcher
Distribution Dispatch
Supervisor
Control Center Operator
Customer Service
Representative
CSR Supervisor
Role in Operation of Customer Complaint Surveillance
Receives water quality calls during after-hours and weekends/holidays;
Advises customers after-hours about water quality concerns related to typical
distribution system issues (i.e. rusty water, chlorine odor, etc.) without additional
support unless requested by customer;
Contacts WQ&T and creates a water quality work request for all water quality
request calls after hours;
Creates water quality work orders after hours, if requested by WQ&T. (If after
consultation with WQ&T on the work request it is determined by WQ&T that field
investigation is required, WQ&T will request the Distribution Dispatcher to create
the work order); and,
Reviews distribution system work request and work orders during investigation of a
CCS Alert.
Receives and leads the investigation of IVR, water quality work request and work
order alerts during non-business hours and weekends/holidays;
Coordinates support from Control Operator, Distribution Dispatcher and CSR
Supervisor during investigation of the CCS alert, as appropriate;
Notifies the WUERM if the determination is made that contamination is Possible;
and,
Notifies alert notification group of investigation close out.
Monitors supervisory control and data acquisition alerts, and reviews operational
data to support the investigation of alerts.
Receives water quality calls during normal business hours;
Advises customers during normal business hours about water quality concerns
related to typical distribution system issues (i.e., rusty water, chlorine odor, etc.)
without additional support, unless requested by the customer; and,
Transfers water quality request calls to WQ&T Division after generating a water
quality work request.
Receives and leads the investigation all IVR alerts during normal business hours in
conjunction with WQ&T personnel and/or the WUERM; and,
Notifies alert notification group of investigation close out.
2.5
Summary of Significant CCS Component Modifications
The modifications discussed in the previous subsections were implemented during the evaluation period
to improve the performance of the CCS component. The impact of these component modifications on
performance can be observed in the metrics used to evaluate the degree to which the CCS component met
the design objectives described in Section 1.1. Table 2-4 summarizes these modifications and serves as a
reference when discussing the results of the evaluation presented in Sections 4 through 7.
Table 2-4. CCS Major Component Modifications
ID
1
Component Modification
Modification
Cause
CUSUM alert notifications included water quality work request
and work order event sets and details.
There was uncertainty how to accurately capture the work
requests/work orders that contributed to the occurrence of a
CUSUM alert.
Date
March 4, 2008
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
ID
2
3
4
5
6
7
8
Component Modification
Modification
Cause
Modification
Cause
Modification
Cause
Modification
Cause
Modification
Cause
Modification
Cause
Modification
Cause
Removed midnight EDA alert reminders for two-day and seven-
day alerts that were continuously alerting.
GCWW found midnight reminder emails for two-day and seven-
day scan alerts above thresholds not useful and believed they
caused confusion.
Changed all EDA thresholds from testing levels to production
levels.
Thresholds that were initially set artificially low to test the level
of acceptability were no longer needed.
Directed EDAs to send weekend alert notifications to an "After-
hours" group.
Business- and non-business-hours notifications groups were
originally conceived as "Day" and "Night" personnel. Weekend
alerts were being sent to the business-hours group and the
correct utility personnel were not being notified of EDA alert
occurrence.
Water quality work requests were supplemented with
information whether or not a work order was generated,
problem description, and related work order ID.
GCWW indicated that including information whether a work
order was generated or not, work order ID and problem
description would be useful during water quality alert
investigation. This reduced the need to access the work
management system.
Notification for Work Request alerts was turned off.
Water quality work requests were generated at a rate lower
than that of water quality work orders, contrary to the initial
projection.
Procedures were modified so all after-hours CCS alerts would
be investigated by the distribution dispatch supervisor.
The Cincinnati Pilot Operational Strategy did not adequately
describe after-hours investigation procedures.
Discontinued receipt of seven-day and CUSUM alerts.
GCWW requested to no longer receive seven-day or CUSUM
alerts. The initial system could turn off alert notifications by
data stream and day or night recipients only.
Date
March 4, 2008
April 29, 2008
May 30, 2008
September 9, 2008
December 17, 2008
January 22, 2009
January 23, 2009
10
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
ID
9
Component Modification
Modification
Cause
Added submenus for rusty/cloudy water, pressure issues,
general inquiry, and taste, odor, or appearance complaints to
IVR 5 selection and directed the IVR EDA to analyze the taste,
odor, and appearance submenu, IVR 5-4.
The message for IVR 5 was general. Complaints related to
water quality issues that were the result of common utility
operations and general questions about water quality were
captured in this menu.
Date
May 28, 2009
Additionally, while not a modification of the CCS component, high turnover in the CSR Supervisor
position (the lead investigator for IVR alerts) significantly impacted the early deployment phase as
demonstrated by inconsistent CCS alert investigations. This was rectified with the appointment of a
permanent CSR Supervisor in May 2009.
2.6
Timeline of CCS Development Phases and Evaluation-related Activities
Figure 2-2 presents a summary timeline for deployment of the CCS component, including milestone
dates for when significant component modifications and drill and exercise evaluation activities took place.
The timeline also shows the completion date for design and implementation, along with the subsequent
optimization and real-time monitoring phases of deployment.
Jan-08
Alert Details Added
to Notifications Apr-08
Thresholds
Jan-08 / Raised
Nov-08
Discontinue
7-day/CUSUM
Nov-08
Discontinue Work
Request Alerts
Apr-09
IVR 5-4 Submenu
Added
Design &
Implementation
Complete
Jan-08
Jun-10
Drill 1
Aug-08
Wind Storm
Sep-08
FSE2
Oct-08
Optimization
Jan-08 - Jan-09
Drill 2
Apr-09
Permanent
CSR Supvr
May-09
CCS-Site
Char. Drill
Sep-09
FSE3
Oct-09
Drill 3
Apr-10
V
Real-time Monitoring
Jan-09 -Jun-10
End of Data
Collection
Jun-10
Figure 2-2. Timeline of CCS Component Activities
11
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 3.0: Methodology
The following section describes the evaluation techniques for the CCS component. The analysis of the
CCS component was conducted using four evaluation techniques to fully assess each data stream and the
overall integrated component: empirical data from routine operations, results from drills and exercises,
results from the simulation study, findings from CCS feedback forums and an analysis of lifecycle costs.
3.1 Analysis of Empirical Data from Routine Operations
Data from CCS operations was collected monthly from January 16, 2008 through June 15, 2010. Data
was tracked by reporting periods, beginning on the 16th of a month and ending on the 15th of the following
month. In this evaluation, the term 'reporting period' is used to refer to a month of data that spans from
the 16th of one month to the 15th of the next. Where applicable, results are summarized by reporting
period to illustrate temporal trends in the data. Additionally, source IVR, work request and work order
data were collected for retrospective analysis.
Investigation data and timelines were provided through CCS investigation checklists. To facilitate and
document CCS alert investigations, lead investigators were required to fill out an investigation checklist
indicating completion of procedures, summarizing findings and detailing the investigation time. The CCS
component was modified, as needed, to optimize performance from January 2008 to January of 2009.
While some investigation checklists were completed during this optimization period, CCS investigators
were not required to respond to alerts in real-time nor complete an investigation checklist during this
time. After turning off the work request, seven-day scan and CUSUM alert notifications in January 2009,
the component transitioned to real-time monitoring and investigation of alerts. Investigators were
expected to respond to alerts immediately and complete investigation checklists.
3.2 Drills and Exercises
Findings from drills and exercises were used to evaluate the alert investigation process, as conducted by
system users, and to determine whether timely and accurate conclusions resulted from the investigation.
One main objective of the drills and exercises was to provide GCWW the opportunity to practice SOPs
associated with recognition of, and response to CCS alerts. Drills and exercises also provided an
opportunity to identify which CCS elements required modification to be more representative of observed
investigation and communication procedures. Four component drills and two Full Scale Exercises (FSEs)
involving CCS were conducted for the purpose of component evaluation. These are discussed below and
include:
CCS Drill 1 (August 19, 2008)
CWS FSE 2 (October 1, 2008)
CCS Drill 2 (April 29, 2009)
CCS-Site Characterization Drill (September 16, 2009)
CWS FSE 3 (October 21, 2009)
CCS Drill 3 (April 15, 2010)
3.2.1 Full Scale Exercise 2 (October 1, 2008)
Description: FSE 2 was conducted on October 1, 2008 to test all Cincinnati pilot CWS components. The
scenario involved the intentional injection of 2,000 gallons of a biological agent, along with an organic
solvent extracting agent, into the GCWW drinking water system at a pumping station. The CCS alert
12
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
investigation procedures associated with the exercise were observed and investigation times were
captured.
Simulated customer complaints of a chemical odor resulted in FVR and work request alerts. The player
actions provided information about the incident to the WUERM and on to the Incident Commander, who
elevated the threat level to Credible.
Relevant Participants:
GCWW WQ&T Chemist, GCWW WQ&T Division
GCWW CSR Supervisor, GCWW Commercial Services Division
GCWW Distribution Dispatch Supervisor, GCWW Distribution Division
GCWW WUERM, GCWW WQ&T Division
3.2.2 CCS-Site Characterization Drill (September 16, 2009)
Description: The objectives of this drill were to evaluate the alert recognition and investigative
procedures associated with the CCS component and to evaluate implementation of the site
characterization procedures related to field deployment and investigation following a CCS alert. The
scenario involved the receipt of odor complaints from customers separated by a few city blocks.
CCS played an integral role in this drill, which was initiated by two work order alerts. The WQ&T
Chemist investigated the alerts and determined that contamination was Possible and notified the
WUERM. The WUERM and the WQ&T Chemist also determined the locations of an appropriate fire
hydrant, and subsequently, residence, to sample for the CMP field investigation of the incident.
Relevant Participants: CCS relevant participants are listed in Table 3-1.
3.2.3 Full Scale Exercise 3 (October 21, 2009)
Description: The FSE 3 design was based on simulating a contamination incident in the GCWW drinking
water distribution system. The scenario involved the intentional injection of a large quantity of a
pesticide into the GCWW drinking water system through a fire hydrant in a western neighborhood of
Cincinnati. The contaminant compound selected was expected to result in CCS alerts, and produce
sufficient public illness to generate delayed PHS involvement.
CCS was the primary component influencing player action in FSE 3. FVR and work order alerts initiated
the drill. A continuous stream of customer complaints throughout the morning provided information to
consequence management. The CSR Supervisor and WQ&T Chemist communication with the WUERM
and Incident Commander was continuous throughout the exercise.
Relevant Participants:
GCWW WQ&T Chemist, GCWW WQ&T Division
GCWW CSR Supervisor, GCWW Commercial Services Division
GCWW WUERM, GCWW WQ&T Division
3.2.4 CCS Drills (1-3)
Description: The objectives of the CCS drills were to evaluate the component alert recognition and
investigation procedures through various alert notification methods and hours of operations at the utility.
The simulated customer complaints produced both IVR and work request alerts and Drill 3 specifically
monitored interactions with the CSR Supervisor and CSRs as calls were placed directly to the utility call
center. In addition to evaluating the implementation of the procedures, the elapsed time between drill
actions was recorded to establish baseline data for future drill activities.
13
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Relevant Participants: CCS relevant participants are listed in Table 3-1.
Table 3-1. CCS Drill Variations
Variations
Time of Drill (N = Normal business hours, A = After hours)
Drill Participants
GCWW CSR 1, GCWW Commercial Services Division
GCWW CSR Supervisor, GCWW Commercial Services Division
GCWW Distribution Dispatcher, GCWW Distribution Division
GCWW Distribution Dispatch Supervisor, GCWW Distribution Division
GCWW WQ&T Chemist, GCWW WQ&T Division
GCWW WUERM, GCWW WQ&T Division
Drill 1
N
0
1
0
0
1
0
Drill 2
A
0
0
1
1
0
0
Drill 3
N
4
1
0
0
0
0
3.3
Simulation Study
Evaluation of certain design objectives relies on the occurrence of contamination incidents with known
and varied characteristics. Because contamination incidents are extremely rare, there is insufficient
empirical data to fully evaluate the detection capabilities of the Cincinnati CWS. To fill this gap, a
computer model of the Cincinnati CWS was developed and challenged with a large ensemble of
simulated contamination incidents in a simulation study. For the CCS component, simulation study data
was used to evaluate the following design objectives:
Contaminant Coverage: Analyses conducted for this design objective quantify the ratio of
contamination scenarios actually detected by the CCS component versus those that could
theoretically be detected.
Timeliness of Detection: Analyses conducted to evaluate this design objective quantify the time
between the start of contaminant injection and the first CCS alert.
A broad range of contaminant types, producing a range of symptoms, was utilized in the simulation study
to characterize the detection capabilities of the monitoring and surveillance components of a CWS. For
the purpose of the simulation study, a representative set of 17 contaminants was selected from the
comprehensive contaminant list that formed the basis for CWS design. These contaminants are grouped
into the broad categories listed below (the number in parentheses indicates the number of contaminants
from that category that were simulated during the study). A description of the manner in which the
critical concentration, which is the concentration that would produce adverse health effects (or aesthetic
problems in the case of the nuisance chemicals), was derived is also provided for each contaminant
category.
Nuisance Chemicals (2): these chemical contaminants have a relatively low toxicity and thus
generally do not pose an immediate threat to public health. However, contamination with these
chemicals can make the drinking water supply unusable. The critical concentration for nuisance
chemicals was selected at levels that would make the water unacceptable to customers, e.g.,
concentrations that result in objectionable aesthetic characteristics.
Toxic Chemicals (8): these chemicals are highly toxic and pose an acute risk to public health at
relatively low concentrations. The critical concentration for toxic chemicals was based on the
14
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
mass of contaminant that a 70 kg adult would need to consume in one liter of water to have a 10%
probability of dying (LDi0).
Biological Agents (7): these contaminants of biological origin include pathogens and toxins that
pose a risk to public health at relatively low concentrations. The critical concentration for
biological agents was based on the mass of contaminant that a 70 kg adult would need to
consume in one liter of water, or inhale during a showering event, to have a 10% probability of
dying (LD10).
Development of a detailed CWS model required extensive data collection and documentation of
assumptions regarding component and system operations. To the extent possible, model decision logic
and parameter values were developed from data generated through operation of the Cincinnati CWS,
although input from subject matter experts and available research was utilized as well.
The simulation study used several interrelated models, three of which are relevant to the evaluation of
CCS: EPANET, Health Impacts and Human Behavior (HI/HB), and the CCS component model. Each
model is further broken down into modules that simulate a particular process or attribute of the model.
The function of each of these models, and their relevance to the evaluation of CCS, is discussed below.
EPANET
EPANET is a common hydraulic and water quality modeling application widely used in the water
industry to simulate contaminant transport through a drinking water distribution system. In the simulation
study, it was used to produce contaminant concentration profiles at every node in the GCWW distribution
system model, based on the characteristics of each contamination scenario in the ensemble. The
concentration profiles were used to determine the number of miles of pipe contaminated during each
scenario, which is one measure of the consequences of that contamination scenario.
Health Impacts and Human Behavior Model
The HI/HB model used the concentration profiles generated by EPANET to simulate exposure of
customers in the GCWW service area to contaminated drinking water. Depending on the type of
contaminant, exposures occurred during one showering event in the morning (for the inhalation exposure
route), or during five consumption events spread throughout the day (for the ingestion exposure route).
The HI/HB model used the dose received during exposure events to predict infections, onset of
symptoms, health-seeking behaviors of symptomatic customers and fatalities.
The primary output from the HI/HB model was a case table of affected customers, which captured the
time at which each transitioned to mild, moderate and severe symptom categories. Additionally, the
HI/HB model outputted the times at which exposed individuals would pursue various health-seeking
behaviors, such as calling GCWW to report unusual water quality characteristics. These calls to GCWW
were input data to the CCS component and were processed by the event detection systems included in the
CCS component model.
The case table was used to determine the public health consequences of each scenario, specifically the
total number of illnesses and fatalities. Furthermore, EPANET and the HI/HB model were run twice for
each scenario; once without the CWS in operation and once with the CWS in operation. The paired
results from these runs were used to calculate the reduction in consequences due to CWS operations for
each simulated contamination scenario.
Customer Complaints Surveillance Component Model
The CCS component model is based on the component as deployed and currently operating in the
Cincinnati CWS. The CCS model contains three modules: a Work Order Generation module, Event
15
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Detection Algorithm and Alert module, and CCS Alert Validation module. In the CCS component, all
customers in the GCWW service area are potential detectors for contaminants that change the aesthetic
characteristics of the drinking water. Thus, each node (area where water is withdrawn from the
distribution system) in the GCWW distribution system model with a non-zero demand represents tens to
hundreds of potential detectors. Customers exposed to water contaminated at concentrations above the
contaminant-specific threshold will detect the contaminant based on a contaminant-specific percentage.
The model assumes a percentage of those customers that detected the contaminant will call GCWW.
These calls are turned into water quality-work orders in the Work Order Generation module. The CCS
Event Detection Algorithm and Alert module generates alerts based on the number of calls or work
orders. The CCS Alert Validation module includes logic for the investigation of these alerts according to
procedures used in the Cincinnati CWS.
Of the 17 contaminants included in the simulation model, 6 contaminants were determined to be
theoretically detectable by the CCS model, based on their organoleptic properties and detection
thresholds. Table 3-2 includes these 6 contaminants, their organoleptic properties and indicates the ratio
of the critical concentration to the detection threshold for each contaminant. The ratio was calculated to
determine whether the detection threshold was sufficient to detect water contaminated at concentrations
equal to or greater than the critical concentration. Large ratios demonstrate the contaminants that can be
detected at concentrations significantly lower than the critical concentration. The detection threshold
values for CCS are based on the contaminant concentration that could be detected by the human senses
(taste, odor, irritation) by a significant percentage of the population, ranging from 50 - 80%, and were
obtained from a literature review and input from subject matter experts. These values were used to
parameterize the CCS model.
Table 3-2. Assumed Contaminant Characteristics for Contaminants Detectable by the CCS
Component
Type1
Nuisance Chemical 1
Toxic Chemical 1
Toxic Chemical 2
Toxic Chemical 3
Toxic Chemical 4
Biological Agent 1
Organoleptic
Property
Odor
Odor
Taste
Taste
Taste
Taste
Critical
Concentration/
Detection Threshold
20.0
5.86
50.5
22.78
4.03
88.2
Note that the contaminants being modeled in the simulation study were assigned generic IDs for security purposes.
The CCS model uses the following assumptions for generation of complaint data:
A contaminant-specific percentage of exposed customers detect the contaminant.
A percentage of the customers that detect the contaminant call the utility.
All calls to the utility are treated as water quality calls (IVR 5-4) and are converted to work
orders.
All water quality complaint calls above the baseline number of calls are due to the simulated
contamination incident.
It takes approximately 15 minutes for a WQ&T Chemist to manage the complaint, determine that
action is needed and create a work order for distribution personnel to collect a sample.
16
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
During non-business hours, capacity is reduced to one shift chemist who can create a work order.
The simulated CCS investigation reflects the procedures used by GCWW personnel to investigate a CCS
alert. Investigators assess the underlying complaints for clustering and similar problem descriptions as
well as possible benign explanations for the alerts, such as distribution work or operational changes. The
CCS investigation portion of the model assumes:
Because the simulated CCS component involved analysis of only contamination-related
complaints from all calls, and thus all alerts, contained similar problem descriptions.
Because of the hydraulic connectivity of the contamination scenarios, all complaints in an alert
were clustered.
No benign explanations could be found for alerts during investigations.
The practical implication of these assumptions is that in the simulated scenarios, the investigation was not
prematurely terminated. In the model, all CCS alerts resulted in a determination that contamination was
Possible.
3.4 Forums
Throughout implementation and optimization of the CCS component, EPA, GCWW personnel and
contract support personnel met monthly to review data, component operation, and discuss modifications.
The monthly meetings provided regular, timely feedback on the CCS component. In addition to monthly
meetings, two lessons learned workshops and exit interviews were conducted.
Lessons Learned Workshop, June 16, 2008, was limited to eight EPA and contractor support
personnel responsible for the design and implementation of the CCS component. The objective of the
workshop was to revisit key decisions made during the process and solicit specific feedback on
successes and challenges encountered.
Lessons Learned Workshop, August 11, 2009, included 16 EPA, GCWW and contract support
personnel involved in the design, implementation and daily operation of the CCS component. The
objective of the workshop was to elicit specific lessons learned information from the pilot utility
through discussions and to gather feedback concerning how lessons learned may be shared with the
drinking water sector.
Exit Interviews, June 2010, were limited to the CCS component leads to provide a final opportunity
to learn and document GCWW's perceptions and experiences with the CWS since assuming full
ownership in June 2009.
3.5 Analysis of Lifecycle Costs
A systematic process was used to evaluate the lifecycle cost of the CCS component, which represents the
overall cost of the CCS component over the lifecycle of the Cincinnati CWS, which is assumed to be 20
years for the purpose of this analysis. The analysis includes implementation costs, component
modification costs, annual operations and maintenance (O&M) costs, renewal and replacement costs, and
the salvage value of major pieces of equipment.
Implementation costs include labor and other expenditures (equipment, supplies and purchased services)
for installing the CCS component. Implementation costs were summarized in Water Security Initiative:
Cincinnati Pilot Post-Implementation System Status (USEPA, 2008), which was used as a primary data
source for this analysis. In that report, overarching project management costs incurred during the
implementation process were captured as a separate line item. However, in this analysis, the project
17
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
management costs were equally distributed among the six components of the CWS, and are presented as a
separate line item for each component. Component modification costs include all labor and expenditures
incurred after the completion of major implementation activities in December 2007 that were not
attributable to O&M costs. These modification costs were tracked on a monthly basis, summed at the end
of the evaluation period and added to the overall implementation costs.
It should be noted that implementation costs for the Cincinnati CWS may be higher than those for other
utilities given that this project was the first comprehensive, large-scale CWS of its kind and had no
experience base to draw from. Costs that would not likely apply to future implementers (but which
were incurred for the Cincinnati CWS) include overhead for EPA and its contractors, cost associated
with deploying alternative designs and additional data collection and reporting requirements. Other
utilities planning for a similar large-scale CWS installation would have the benefit of lessons learned
and an experience base developed through implementation of the Cincinnati CWS.
Annual O&M costs include labor and other expenditures (supplies and purchased services) necessary to
operate and maintain the component and investigate alerts. O&M costs were obtained from procurement
records, maintenance logs, investigation checklists and training logs. Procurement records provided the
cost of supplies, repairs and replacement parts, while maintenance logs tracked the staff time spent
maintaining the CCS component. To account for the maintenance of documents, the cost incurred to
update documented procedures following drills and exercises conducted during the evaluation phase of
the pilot was used to estimate the annualized cost. Investigation checklists and training logs tracked the
staff hours spent on investigating alerts and training, respectively. The O&M costs were annualized by
calculating the sum of labor and other expenditures (supplies and purchased services).
Labor hours for both implementation and O&M were tracked over the entire evaluation period. Labor
hours were converted to dollars using estimated local labor rates for the different institutions involved in
the implementation or O&M of the CCS component.
The renewal and replacement costs are based on the cost of replacing these major pieces of equipment at
the end of their useful life. The useful life of CCS equipment was estimated using field experience,
manufacturer-provided data and input from subject matter experts. Equipment was assumed to be
replaced at the end of its useful life over the 20 year lifecycle of the Cincinnati CWS. The salvage value
is based on the estimated value of each major piece of equipment at the end of the lifecycle of the
Cincinnati CWS. The salvage value was estimated for all equipment with an initial value greater than
~$1,000. Straight line depreciation was used to estimate the salvage value for all major pieces of CCS
equipment based on the lifespan of each item.
All of the cost parameters described above, (implementation costs, enhancement costs, O&M costs,
renewal and replacement costs, and salvage value) were used to calculate the total lifecycle cost for the
CCS component, as discussed in Section 7.5.
18
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 4.0: Performance of IVR Data Stream
The following section provides a description of the IVR data stream followed by the results of the
evaluation of this data stream. This analysis includes an evaluation of metrics that characterize how the
IVR data stream achieves the design objectives described in Section 1.1. Specific metrics are described
for each of the design objectives.
4.1 Description of the IVR Data Stream
GCWW routinely receives calls from residential, commercial and industrial customers via a single,
widely published utility phone number. As part of the CCS component, GCWW's automated call
management system transfers all water quality-related calls to the GCWW call center. These calls are
directed to CSRs using an IVR system. A new IVR menu option specifically for water quality-related
issues (IVR 5) was implemented in September 2006 to filter out water quality complaints from other
inquiries, creating in essence, a data stream that would give the earliest indication of a possible water
quality anomaly. Information on the IVR selection is stored in GCWW's CIS application. IVR data does
not have customer location information associated with it.
CCS data management of IVR calls involves the automated, near real-time (every minute) collection and
analysis of customer complaint data using event detection software. Data collection occurs via Web
services, a mechanism for machine-to-machine interaction over the network application. Once analyzed
using a variety of statistical EDAs, the data is stored locally to support periodic offline manual analysis to
verify the algorithm is functioning properly.
IVR Submenu
The IVR menu option (IVR 5) was successful at filtering water quality calls initially. However, analysis
of a year of data indicated that some of the filtered calls may not be indicative of possible water
contamination. The IVR 5 script tells customers: "For water quality questions or concerns, press five."
This generally worded script tended to group calls that may be indicative of water contamination with
other more benign calls, such as those related to general inquiries, water pressure complaints, and calls
about rusty or cloudy water, which are often associated with regular utility maintenance and operations.
Consequently, calls most likely associated with a possible contamination incident (those indicating a
change in taste, odor or appearance) made up a relatively small proportion of total IVR 5 calls.
In response, a new submenu was added on April 3, 2009 to better filter customer complaint calls most
indicative of water contamination. The new IVR 5 submenu provided separate selections for the
following: 1) general inquiries; 2) water pressure issues; 3) rusty or cloudy water; and 4) taste, odor, and
appearance. The taste, odor and appearance submenu (5-4) was specifically targeted for surveillance.
IVR Event Detection
A collection of algorithms was used to automate the IVR data analysis process to provide a dependable,
robust surveillance system that produces alerts for call volume above an established threshold. These
initial threshold values were developed based on statistical analyses of existing historical data; thresholds
were subsequently adjusted for routine operations using data collected during the optimization phase.
Five algorithms were originally applied to the IVR 5 data stream. These algorithms included different
permutations of scan statistics and CUSUM functions, as presented previously in Table 2-2. As
significantly fewer calls are received on weekends, a different algorithm with a different threshold was
used. Since different algorithms and varying thresholds are used, all five algorithms do not operate at all
19
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
times. The EDAs operate continuously in near real-time (every minute); any count above the
predetermined threshold generates an automated email alert. If the threshold is not reached, the
algorithms continue to monitor the IVR data stream.
During the optimization period, evaluators observed that using five algorithms produced overlapping
coverage; therefore, the CCS event detection system was refined to include only algorithms most
indicative of more acute possible water contamination incidents. Because water contamination would
likely produce acute effects, a seven-day alert would not likely be related to water contamination, and the
variable nature of the CUSUM algorithm made interpretation less direct. Additionally, GCWW staff had
little confidence in identifying health effects caused by slowly worsening water quality conditions
because the age of the water in the distribution system was typically only 48 hours or less. Hence, the
seven-day scan statistic and CUSUM algorithms were excluded from alert notification in January 2009.
4.2 Design Objective: Spatial Coverage
Spatial coverage is the cumulative area of the distribution system where a detectable change in water
characteristics could be reported via a customer complaint call. Since there is no location data associated
with IVR alerts, spatial extent of alerts cannot be analyzed. Population coverage and area coverage are
discussed for the IVR, work request and work order data streams at the component level (Section 7.0).
4.3 Design Objective: Alert Occurrence
Alert occurrence addresses aspects of system performance, including the frequency of invalid alerts, in
order to determine how well the IVR data stream discriminates between real incidents (water
contamination or valid alerts) and normal variability in the underlying data. The following two metrics
are evaluated as described in the following sections: invalid alerts and sensitivity.
4.3.1 Invalid Alerts
Definition: The invalid alert metric is an indication that an alert was generated from IVR call volume
EDAs when there is no contamination present or system event occurring. The number of invalid alerts is
dependent on the threshold level set for each EDA.
Analysis Methodology: The total number of invalid alerts is equal to the number of total alerts minus the
number of valid alerts. To date, there have been no valid alerts generated by the IVR data stream;
therefore, all empirical data from the evaluation period represent an invalid alert. Alert rates at various
thresholds were also evaluated using the Alarm Estimation Tool (AET), a Microsoft Excel macro that
simulates and illustrates the behavior of various scan algorithms at different thresholds (USEPA, 2011).
Results: A total of 202 invalid alerts were generated by all IVR EDAs deployed, including the seven-day
scan and CUSUM algorithms, during the evaluation period. The total number of alerts per reporting
period varied, as depicted in Figure 4-1. The average and median number of invalid alerts per reporting
period was seven and four, respectively.
The Cincinnati Fire Department performs routine hydrant maintenance in early spring of each year. The
maintenance includes flushing activities that may result in elevated turbidity in the drinking water,
generating customer water quality complaints. This pattern is observed in the March and April reporting
periods of each year. On April 30, 2008, thresholds were increased from testing levels set artificially low
to test the system, to production levels that were based on historical data. The figure also shows that the
CUSUM and seven-day scan alerts do not significantly alter the shape of the bar graph, and primarily
contributed to the volume of alerts.
20
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Additionally, while there are no problem descriptions associated with IVR calls, there were some
activities identified in the work request problem descriptions which may explain tendencies in the data.
For example, a spike in invalid alerts occurred in the January 2009 reporting period, which corresponded
with customer participation in a regulatory compliance monitoring program. Thus, customers who called
the utility to schedule collection of the program's household samples contributed to the rise in invalid
alerts during the reporting period. Customers may have also selected IVR 5 when contacting GCWW
during this time, thereby also affecting the number of IVR alerts. The same regulatory compliance
monitoring program occurred again during the September 2009 reporting period, where another spike in
invalid alerts is observed.
35
30
25
is
10
1
begmnmgot
Real-time Analysis
1 $ "
It £ |
CUSUM
Seven-dayScan
Two-day Scan
One-day Scan, Weekend
One-day Scan, Weekday
|
I
i 1
i,
ll
1 1
1. ill.
\
Start Date of Monthly Reporting Period
Figure 4-1. IVR Invalid Alerts per Reporting Period
Figure 4-2 presents a histogram of the total number IVR alerts which occurred during each reporting
period. The majority of reporting periods had five or fewer invalid alerts. There were no reporting
periods within the 11-15 range for number of alerts.
21
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
6-10
11-15
Invalid Alerts
16-20
21-33
Figure 4-2. Histogram of IVR Invalid Alerts per Reporting Period
Figure 4-3 displays how the scan algorithms behave at various thresholds based on analysis of historical
IVR 5 data. The y-axis displays the number of alerts that are generated at each threshold on a logarithmic
scale. Arrows indicate the thresholds utilized by GCWW prior to April 3, 2009, when the algorithms
ceased analyzing IVR 5 data and switched to IVR 5-4 data. The shapes of the curves differ significantly
among the reset (one-day scan) and continuous (two-day and seven-day scan) algorithms. The number of
alerts for the one-day scan reset algorithm declines as the threshold increases. At a threshold of one,
algorithms alert for each IVR complaint received, and the number of complaints resets to zero after each
alert. However, the two-day and seven-day scan continuous algorithms remain in alert as long as the
number of complaints over the scan window remains above the threshold. Thus, one complaint can
produce an alert that lasts the entire EDA duration period.
22
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
10000
One-day Scan, Weekday
Two-day Scan
Seven-day Scan
Threshold
Figure 4-3. Expected Alerts at Various Thresholds for Historical IVR 5 Data
The continuous two-day scan deployed at GCWW only operates on weekdays, from Monday, 12:00 AM
until Friday, 23:59 PM. Thus, for low thresholds an alert is issued when the threshold is crossed the first
time each week. The number of alerts remains steady until the threshold reaches a level where multiple
alerts can occur in a week. The number of alerts continues to rise until threshold peaks at the maximum
number of alerts that could have occurred during the evaluation period. The number of alerts then
declines as the threshold increases.
Since the continuous seven-day scan operates every day, the weekly issuing of alerts seen in the two-day
scan does not affect the lower thresholds. For the continuous seven-day scan, an alert is issued at lower
thresholds that persist throughout the evaluation period. This is one long continuous alert. Multiple alerts
do not occur until the threshold is greater than the fewest number of complaints occurring in any seven-
day period. Beyond this threshold, the continuous seven-day scan follows the pattern of the continuous
two-day scan. The number of continuous seven-day scan alerts increases as the threshold increases until
reaching a peak of 89 and declining. The previous IVR 5 threshold levels were: one-day scan, weekday =
52, two-day scan = 92 and seven-day scan = 216.
Figure 4-4 depicts the expected number of alerts per EDA at varying thresholds, based on historical IVR
5-4 data. Arrows indicate the thresholds utilized by GCWW at the end of the evaluation period (June 15,
2010). Note that thresholds can be set much lower than with the IVR 5 data, due to the improved
specificity and lower overall volume of IVR 5-4 selections. The current threshold levels are: one-day
scan, weekday = four, two-day scan = six and seven-day scan = eleven.
23
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
120
100
One-day Scan, Weekday
Two-day Scan
Seven-day Scan
10 11 12 13 14 15 16
Figure 4-4. Expected Alerts at Various Thresholds for Historical IVR 5-4 Data
4.3.2
Summary
Although invalid alerts initially occurred frequently, once the EDA thresholds were raised in April 2008
the number of invalid alerts per reporting period did not exceed 22. Most reporting periods had five or
fewer invalid alerts. Routine hydrant flushing by the Cincinnati Fire Department causes an increase in
complaints during the spring of each year. CUSUM and seven-day scan alerts only contributed to the
volume of alerts, and did not cause any alerts during months when other algorithms did not alert.
Analysis of the EDA behavior at various thresholds indicates that thresholds can be set much lower for
the IVR 5-4 data as opposed to IVR 5 data. The AET exploited patterns that emerge when plotting
thresholds using continuous and reset algorithms.
4.4
Design Objective: Timeliness of Detection
Timeliness of detection involves the time for the IVR data stream to detect possible water contamination
through routine analysis and the time necessary for investigation of this alert. Factors impacting this
objective include: time for data transmission, time for event detection, time to recognize alerts, and time
to investigate invalid alerts. The latter two metrics are defined, their analysis discussed, and results
presented in the following subsections. Data transmission and time for event detection are shared across
all data streams, and are discussed in Section 7.0.
4.4.1
Time for Alert Recognition
Definition: Time for alert recognition quantifies the time for investigators to recognize the alert
notification and begin to act upon it. The CSR Supervisor is responsible for IVR alert investigations
during business hours, while the Distribution Dispatch Supervisor is responsible for investigations during
non-business hours.
Analysis Methodology: The average and median time between delivery of the alert email and start of
alert investigation, as recorded on investigation checklists is presented. The number of alert
investigations conducted is also presented.
24
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Results: Table 4-1 shows the average and median times for recognition of an IVR alert by the CSR
supervisor. Analysis of the data shows a significant difference between the median and average times for
alert recognition, as well as the overall range (minimum and maximum) for alert recognition. It should be
noted that there was only one one-day scan, weekend alert investigation during the monitoring period.
Table 4-1. Time for IVR Alert Recognition (minutes)
Algorithm
One-Day Scan, Weekday
Two-Day Scan
One-Day Scan, Weekend
Total
Investigations
20
12
1
33
Average
16.7
24.8
3.0
19.2
Median
3.5
9.0
3.0
7.5
Range
0-155
0-155
3
0-155
Figure 4-5 illustrates a noticeable difference between the median and average time for alert recognition.
The average is affected by a few outlying data points, none of which were greater than 155 minutes.
Most alerts were recognized in fewer than 10 minutes. After the appointment of apermanent CSR
Supervisor, the time for recognition of IVR alerts improved.
One-day Scan,
Weekday
Two-day Scan
One-day Scan,
Weekend
Total
Figure 4-5. IVR Alert Recognition Time, by Algorithm
4.4.2
Time to Investigate Alerts
Definition: Investigation time measures the time spent by staff to investigate invalid alerts. Since there
were no actual contamination incidents in the evaluation period, this metric represents all time spent by
GCWW staff on alert investigation.
Analysis Methodology: Average and median time between the beginning and end of an alert
investigation as recorded on the investigation checklists, is calculated. Additionally, the timeline for CCS
Drill 3 is presented.
25
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Results: CCS Drill 3 was conducted on April 29, 2010. The drill included actual calls to the utility Call
Center to evaluate the collection of customer complaints and internal processing of call data leading to an
IVR alert and to evaluate alert recognition and investigation procedures. The IVR alert was generated
approximately eight minutes after the fourth and final drill-related call was received and approximately 30
minutes after the start of the drill. The drill was completed shortly after the CSR Supervisor contacted the
WQ&T Chemist, 10 minutes after completing the IVR alert investigation checklist. Figure 4-6 shows the
timeline progression of the key activities completed during the CCS alert investigation for CCS Drill 3.
Note that the timeline was normalized so the start of the investigation occurs at time 0.
00:00
First Drill-related
Call Received at
Call Center
00
Final Dri
Call Rec
CallC
30
l-related
eived at
enter
00:38
IVR 1 -Day Alert
Received
00:42
IVR Alert Verification
Checklist Completed
1 r
00:52
WQ&T Chemis
Notified o
Complaints
00:00
00:52
Figure 4-6. Timeline Progression of the CCS Alert Investigation during CCS Drill 3
Table 4-2 shows that most of the investigated invalid alerts were in response to the one-day scan,
weekday algorithm. Furthermore, the average investigation time did not vary greatly by EDA, and the
total average alert investigation time was 12.3 minutes. Prior to May 2009, staff turnover in the CSR
Supervisor position was high, and investigating IVR alerts was not an immediate priority for the
temporary CSR Supervisors. Once the CSR Supervisor position was filled permanently in May 2009,
alerts were investigated consistently and thoroughly. The average investigation time spent per alert (12.3
minutes) indicates that the IVR data stream is a timely method for detection of a possible contamination
incident.
Table 4-2. Time for IVR Alert Investigation (Minutes)
Algorithm
One-Day Scan, Weekday
Two-Day Scan
One-Day Scan, Weekend
Total
Investigations
20
12
1
33
Average
12.6
10.1
34.0
12.3
Median
11.5
9.5
34.0
10.0
Range
4-30
3-22
34
3-34
As shown in Figure 4-7, the average investigative times per false alert for the one-day scan, weekday and
two-day scan algorithms were similar, at 12.6 and 10.1 minutes, respectively. Average investigation time
of the one-day scan, weekend algorithm appears artificially high because the investigation is based on
only one occurrence of that alert.
26
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
40
35
30
25
20
15
10
5
0
I
One-day Scan,
Weekday
Two-day Scan
One-day Scan,
Weekend
Total
Figure 4-7. IVR Average Investigation Time per Alert, by Algorithm
4.4.3
Summary
GCWW staff identified most alerts in fewer than 10 minutes, indicating these alerts were taken seriously.
IVR alert weekday investigations took approximately 10 to 13 minutes, on average, to complete.
4.5
Design Objective: Operational Reliability
Analysis of the operational reliability of the IVR data stream involves quantifying the percent of time that
the IVR surveillance is working as designed. In order to evaluate how well the IVR data stream met this
design objective, the following two metrics were evaluated: availability and data completeness. These
metrics are described in the following sections.
4.5.1
Availability
Definition: Availability refers to the amount of time that all components of the IVR data stream are
operating correctly, including the availability of the IVR system, CIS application, Web services for data
transfer, and EDAs. Since the EDAs are shared with the other data streams, it is addressed in Section 7.0.
Analysis Methodology: Any downtime experienced by the IVR or CIS systems and Web service was
measured.
Results: Figure 4-8 displays the downtime for the two systems constituting the IVR data stream of CCS.
When these systems are down, IVR event detection is not possible. Additionally, event detection is not
possible if the Web service transferring data to the EDAs fails. Only once did the Web service fail in
November 2008. There was a 1 minute, 21 second-long network outage on October 31, 2008 that resulted
in data disruption for IVR 5, work request and work orders. Due to this brief outage, the work request
and work order processes timed out and then returned to normal behavior, while the IVR 5 process failed
to reset. This failure was unnoticed for nearly three weeks, resulting in 572 hours of IVR downtime. This
was an isolated event.
27
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
60
50
40
20
10
1
Beginningof
Real-time Analysis
lVR
Banner
Start Date of Monthly Reporting Period
Figure 4-8. IVR and CIS Downtime
Since the IVR and CIS systems are closely tied together, they are regularly maintained or fail in
conjunction with each other. During most reporting periods, downtime is minimal or up to one hour for
regular maintenance, however, there were two planned and one unplanned exceptions to this trend. The
IVR and CIS systems were taken offline in the May and September 2008 reporting periods for scheduled
upgrades. While these upgrades resulted in significant downtime for the data stream, they were scheduled
over weekends to minimize the impact on customer service. Customers typically wait until business
hours to contact GCWW to report problems or request information. The one unplanned event resulting in
significant downtime occurred in July 2008. In this third instance, a storm interrupted power at the
GCWW facility housing the IVR systems. The loss of power outlasted the uninterrupted power supply
for the systems, resulting in approximately seven hours of downtime.
4.5.2
Summary
The small percent of unavailability for the IVR data stream underscores the reliability of this data stream.
The IVR system is very stable, with an overall availability of 96.9%. Scheduled maintenance is
conducted during non-business hours, when customers are least likely to call to report water quality
issues, to minimize service impacts.
4.6 Design Objective: Sustainability
4.6.1 A cceptability
Definition: Acceptability is the willingness of persons and organizations to monitor, maintain and
actively participate in a program. The acceptability of the investigation procedures is quantified through
investigation checklist usage.
Analysis Methodology: Alert investigations are documented in investigation checklists. The percentage
of alerts investigated is calculated for data from the real-time monitoring period. Since the seven-day
28
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
scan and CUSUM algorithm notifications were disabled during this time, the investigation checklists
were not expected for those alerts.
Results: A total of 33 alerts were investigated during the evaluation period. A majority of these were
one-day scan, weekdays, as seen in Figure 4-9. There were no seven-day scans or CUSUM alerts
investigated prior to disabling alert notifications for seven-day scan and CUSUM algorithms in January
2009.
lOne-day Scan, Weekday Two-day Scan One-day Scan, Weekend
Figure 4-9. Investigated IVR Alerts by Algorithm
Table 4-3 presents the business hours versus non-business hours alert investigation percentages. The low
percentage of non-business hours IVR investigations may indicate problems defining clear
responsibilities for non-business hours investigations, even after the Cincinnati Pilot Operational Strategy
was updated to encompass investigations of these alerts.
Table 4-3. Business Hours and Non-business Hours Investigations
Timeframe
Business Hours
Non-business Hours
Total
Alerts
48
24
72
Investigations
31
3
34
% Investigated
65%
13%
47%
Figure 4-10 displays the number of IVR alert investigations completed and the number of alerts not
investigated. In January of 2009, alerts were defined for this metric as all alerts received and were
expected to be investigated. Early on, the turnover in the lead investigator position, the CSR Supervisor,
resulted in a large number of uninvestigated alerts. After a permanent CSR Supervisor was appointed in
May of 2009, investigations that occurred during business hours were completed timely and regularly.
From the May 2009 reporting period onward, 89% of IVR alerts during business hours were investigated.
29
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
V2. -
10
5
3 Q -
Number of/
^ CD C
re
s
o -
I 1 *
Alerts Not
P Alerts Inve
1
nvestigated
sti gated
i
1 P
I' I P
1
Oj Oj Oj Oj Oj Oj Oj Oj Oj ^ Q) Q) ^\ ^\ O
vi vi vi vi vi vi vi vi vi /. \i / \^ f \» ^^ ^^ ^^
Start Date of Monthly Reporting Period
I,
.1
x/
Figure 4-10. IVR Alert Investigations Completed and Alerts Not Investigated
4.6.2 Summary
Most business-hours IVR alerts were investigated during the evaluation period, in particular after a
permanent CSR Supervisor was appointed. Alerts that occurred after-hours (nights and weekends) were
less likely to be investigated.
4.7
IVR Submenu and Keystroke Data Analysis
As discussed in Section 4.1, a new menu option was added to the IVR system to capture water quality-
related complaints (menu option 5). Originally, this menu prompt instructed callers, "For water quality
information, questions or concerns press five." In September 2008, analysis of the IVR 5 data was
conducted to assess the effectiveness of the IVR data to capture complaints indicative of possible water
contamination. Prior to CCS implementation, CSRs tracked water-quality related calls using a
"keystroke" process. At the conclusion of all customer complaint calls, the CSR denotes all calls using a
keystroke identifier. For actual water quality complaints, the number 6 is pushed on the telephone to
indicate the call type. Using this keystroke dataset, the correlation between IVR menu selections and
actual complaints, as determined by the CSR who interviewed the caller, can be calculated.
After the first sixth months of deployment (January 2008 - June 2008), evaluators suspected that the IVR
5 menu option was capturing many calls, such as pressure or turbidity complaints, that were not related to
water contamination. Thus, an analysis of historical IVR data was conducted in September 2008.
4.7.1
IVR Menu 5 Analysis
Data for IVR 5 selections was available from December 2006 onward, and keystroke 6 data was provided
from June 2007 through the beginning of July 2008 (GCWW stores keystroke counts for up to a year).
IVR 5 selection analysis focuses on trends and potential seasonality as well as traditional background
frequencies. Since there is a clear difference in the volume of customer complaint calls on weekdays vs.
holidays and weekends, this analysis focuses on non-holiday weekday data only.
30
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Summary statistics on the initial analysis of 393 weekdays from December 13, 2006 to July 10, 2008 are
presented in Table 4-4. The mean approximates the median and mode, indicating that the data is not
skewed or affected by a large number of extreme outliers.
Table 4-4. Summary Statistics of IVR 5 and Keystroke 6 Data
Summary Statistics
Mean
Median
Mode
Standard Deviation
Range
IVR 5
(12/13/06-7/10/08)
31.9
31
27
9.2
11-70
Keystroke 6
(6/26/07 - 7/7/08)
1.8
1
1
1.7
0-7
To filter random noise and identify trends in the data, the IVR 5 counts were smoothed using a ten-day
moving average, as shown in Figure 4-11. Throughout most of the year, IVR 5 selections record around
30 complaints a day. However, in all years there is a temporary rise in late March through early April.
This is most likely the result of spring maintenance activities conducted by the City of Cincinnati Fire
Department, which creates "rusty" or "dirty" water complaints due to scheduled flushing of fire hydrants.
The increase seen in March - April of 2008 may have also been influenced by a March 2008 Associated
Press report regarding pharmaceuticals in drinking water. Although GCWW was not named in the story,
this jump could represent an increase in water quality questions, not complaints. This example shows
how the original IVR 5 prompt was too broad to adequately filter water quality-related complaints.
o 30
Figure 4-11. Annual 10-day Moving Average of Daily IVR 5 Selections
As indicated in Table 4-4, keystroke 6 counts were much less frequent than IVR 5 calls. A histogram of
keystroke 6 counts per day, shown in Figure 4-12, indicates that most days receive only one or two
keystroke 6 counts.
31
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
2345
CSR Keystroke 6 Totals
Figure 4-12. Histogram of Daily Keystroke 6 Counts
The box plot of IVR 5 and keystroke 6 data, shown in Figure 4-13, indicates that increased IVR 5 calls
may correlate with higher keystroke 6 counts.
a: o
> ^r
3 4
Keystroke #6
Figure 4-13. Box and Whiskers Plot of IVR 5 and Keystroke 6 Data
The histogram of the keystroke 6 counts is heavily weighted to few counts. Days with one or no
keystroke 6 hits account for 52% of all work days. The box plot of the IVR 5 data against keystroke 6
counts indicates that IVR 5 may increase with keystroke 6 hits. During the initial design of the CCS
32
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
monitoring system, the CCS data analysis focused on keystroke 6 hits, since the IVR system had not been
established at the time. The CCS design assumed that IVR 5 choices by customers would correlate with
keystroke hits. To test the validity of this assumption, a statistical analysis was performed using a
generalized linear model.
Throughout the study period, CSR-identified water quality complaints (keystroke 6) were much lower
than the customer-selected water quality complaints (IVR 5 selections). The statistical analysis indicated
IVR 5 selection is a statistically significant predictor (95% confidence interval) of CSR keystroke 6 hits.
However, for the purposes of identifying extreme events this analysis may be misleading. To investigate
further, the days with the greatest volume of IVR 5 selections were compared to their associated
keystroke 6 counts, as shown in Table 4-5.
Table 4-5. Days with Greatest Volume of IVR 5 Selections Correlated to Keystroke 6 Counts
Date
07/23/07
08/30/07
04/7/08
04/22/08
07/9/07
11/5/07
05/19/08
02/4/08
03/10/08
04/14/08
09/4/07
07/7/08
04/23/08
12/10/07
04/21/08
04/3/08
10/22/07
05/5/08
04/1/08
04/28/08
IVR 5 Selections
47
47
47
47
48
48
49
50
50
50
51
52
55
59
60
65
67
67
68
70
Keystroke 6 Selections
1
1
0
1
2
2
1
4
6
1
1
7
4
1
5
1
6
1
4
3
33
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Half of the days with the highest IVR counts occurred in the spring of 2008, which is likely the product of
the news report and Cincinnati Fire Department maintenance activities. Of these days, only two have an
associated keystroke 6 count that is among the three highest ranks. The statistical significance may be a
product of the data falling into the IVR 5 inter-quartile range. The bulk of the correlation may be a
product of days with zero, one or two keystroke 6 counts and corresponding moderate IVR 5 selections.
4.7.2 IVR 5-4 Submenu Analysis
On April 3, 2009, when the new IVR submenu was implemented to further filter complaints more
indicative of possible water contamination; this submenu became available during after-hours calls
starting May 28, 2009. Like the IVR 5 calls, a CSR would log the IVR 5-4 calls with a keystroke
identifier at the completion of each call, using keystroke 6 for water quality complaints.
Analysis of the first year of data from the submenu was performed in the spring of 2010. Figure 4-14
gives a breakdown of the IVR submenu choices. As shown, the number of taste, odor and appearance
calls makes up a relatively small percentage (4%) of the overall IVR 5 calls. It should be noted that 30%
of all calls were dropped, or caller hang-ups. The new IVR submenu does not have a "zero-out" option
(to exit the menu) if the customer does not feel that the IVR options describe the issue to be addressed.
30%
50%
5-1 General Inquiry
5-2 Water Pressure
5-3 Rusty/Cloudy
5-4 Taste, Odor, Appearance
^Dropped
9%
Figure 4-14. Proportion of IVR Submenu Options
Table 4-6 compares summary statistics for IVR 5, IVR 5-4 and CSR keystrokes from April 3, 2009
through February 26, 2010. The taste, odor, and appearance submenu option has similar mean, median,
and range to the keystroke 6.
Table 4-6. Summary Statistics of IVR and Keystroke Data
Summary Statistics
Mean
Median
Mode
Standard Deviation
IVR 5
31.5
31
26
9.0
IVR 5-4
1.1
1
1
1.3
Keystroke 6
1.2
1
0
1.7
34
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Summary Statistics
Range
IVR5
10-64
IVR 5-4
0-5
Keystroke 6
0-6
The similarity in their distributions can be seen in Figure 4-15, where the distributions are largely
comparable, with the only key difference being the slightly heavier tail of the keystroke data (greater
occurrence of more than three water quality complaints recorded in a day) while IVR 5-4 selections are
more heavily weighted to two or fewer complaints per day.
IVR 5-4
Key stroke 6
12345
Number of Water Quality Calls Received
Figure 4-15. Frequency of Daily Number of Water Quality Complaints for Each Data Stream
The similarities in the above table and figure suggest that IVR 5-4 submenu selections may be a better
proxy for actual water quality complaints than the broader IVR 5 category. To further explore the
similarity of the data, a correlation analysis using daily counts of IVR 5-4 and CSR Keystroke 6 counts
was performed. The results of the Pearson's product-moment correlation of the IVR and keystroke data
are summarized in Table 4-7. While the correlation is statistically significant (p = 0.004) the correlation
is not particularly strong with r = 0.1892.
Table 4-7. Correlation Summary: IVR 5-4 and Keystroke 6 Data
Correlation
N
227
R Estimate
0.1892
2.5%
0.0605
97.5%
0.3118
P Value
0.004
The temporal correlation between IVR 5-4 selections and keystroke 6 counts can be seen in Figure 4-16,
which depicts the moving averages of each data stream counts. While at times the two data sets exhibit
similar trends, there are other times where they are discordant; hence, the relatively low correlation
coefficient. In particular, the average of IVR 5-4 calls was consistently above that of keystroke 6 counts
for June through mid-July 2009.
35
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
10 per. Mov. Avg. (IVR 5-4)
10 per. Mov. Avg. (Keystroke 6)
\N" \N? \N?
Nn>N ^N &
Figure 4-16. 10-Day Moving Average of IVR 5-4 and CSR Keystrokes
4.7.3
Summary
During the initial design of the CCS monitoring system, the CCS data analysis focused on keystroke 6
hits, since the IVR system had not been established at the time. The original IVR 5 menu option captured
many customer complaints that were not water quality-related. Addition of the IVR 5-4 submenu more
effectively filtered out complaints that were more indicative of possible water quality contamination. If
CSR keystroke 6 counts are used as a standard of measure for true water quality complaints, then self-
identified water quality complaints by callers using the IVR 5-4 submenu are more indicative of true
water quality complaints than the broader IVR 5 menu.
Comparison of the IVR submenu 5-4 and keystroke 6 datasets showed that while the linear relationship
between the two was statistically significant (p = 0.004), the correlation between them (i.e. how well the
data moved together over time) was not particularly strong (r = 0.1892). In general, however, the IVR 5-
4 submenu was found to be an effective means of filtering true water quality complaints from other types
of customer complaints.
36
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 5.0: Performance of Work Request Data Stream
The following section provides a description of the work request data stream followed by the results of
the evaluation of this data stream. This analysis includes an evaluation of metrics that characterize how
the work request data stream achieves the design objectives described in Section 1.1. Specific metrics are
described for each of the design objectives.
5.1 Description of the Work Request Data Stream
As discussed in Section 4.1, calls made to GCWW's automated call management system are directed
using an IVR system. The IVR 5-4 (and previously, the IVR 5) menu option filters out water quality calls
and directs them to a CSR, who asks the caller further questions. A work request is generated when the
CSR determines the call received via the IVR system constitutes a water quality concern (i.e., not a
general inquiry or call relating to rusty/cloudy water). Work requests, including information on time,
date, location and type of water quality concern, are entered by the CSR into GCWW's work order
system and stored in the GCWW's Enterprise Maintenance Planning and Control (EMPAC) database,
where it can be accessed by the Water Security application server for analysis by EDAs.
Work Request Event Detection
EDAs automate the analysis of work request data. The frequency of data upload and analysis provides a
timely surveillance mechanism to detect work request volumes above established thresholds. These
thresholds were determined after an assessment of historical work request data, and can be adjusted as
necessary based on current data.
The algorithms applied to the work request data stream are the same as those used in IVR data analysis:
Scan Statistics, one-day scan weekday; Scan Statistics, two-day scan; Scan Statistics, seven-day scan;
Scan Statistics, one-day scan, weekend; and CUSUM. A description of these EDAs can be found in
Table 2-2. Work request volume above predetermined thresholds produce automated email and text
message alerts, which are sent to the WQ&T Chemist and CSR Supervisor for review. Like the IVR
analysis, the CUSUM algorithm and seven-day scan were excluded from alert notification in January
2009; in addition, no one-day scan weekend alerts were identified although the algorithm continued to
function. If the threshold is not reached, the algorithms continue to monitor the work request data stream.
Originally, alert investigation was executed following email or text message alerts according to the
Cincinnati Pilot Operational Strategy that outlines the investigative protocols. These investigations used
spatial analysis via Hydra Map to identify clusters of work requests. However, it was determined that
work request alert investigation was redundant with work order alert investigation, and all work request
email alert notifications were discontinued in December 2008. Note that work request data continued to
be collected and analyzed for evaluation purposes. This information is still available to GCWW staff if
necessary for investigation into possible water contamination.
Current management of the work request data stream consists of CSRs creating work requests in the
Hydra system with information from customer water quality calls. These work requests are stored in the
EMPAC database and analyzed using EDAs.
5.2 Design Objective: Spatial Coverage
The spatial coverage is the cumulative area of the distribution system where a detectable change in water
characteristics can be reported via customer complaint calls and the alerts from these calls can be
37
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
detected. Spatial coverage depends on spatial distribution, spatial extent of alerts and population
coverage. Analysis of the areas included in work request alerts can help determine spatial distribution
and extent of alerts. Since population coverage and area coverage are the same for FVR, work request and
work order data streams, this metric is discussed at the integrated component level (Section 7.0).
5.2.1 Spatial Extent of Invalid Alerts
Definition: This metric refers to the geographical coverage of work request alerts, that is, whether the
work requests occur in a small or large portion of the GCWW service area. Customer complaints are
expected to spatially cluster during true contamination incidents.
Analysis Methodology: For each work request alert, the area of the bounding rectangle of the work
requests involved is calculated. The bounding rectangle is the smallest rectangle possible that contains all
of the work requests contributing to the alert within its borders. Effective bounding reduces the
coordinates to the furthest east, west, north and south from this set of work requests. Since thresholds
were set low initially, the small number of work requests involved in many alerts do not give an accurate
depiction of the spatial extent of invalid alerts. Thus, distribution of spatial extent of alerts is only
calculated for alerts after the thresholds were raised on April 30, 2008. For reference, the GCWW retail
distribution system covers an area of approximately 356 square miles. Given the irregular shape of
GCWW's retail area, the calculated area of the bounding rectangle has an area greater than GCWW's
retail area.
Results: Figure 5-1 displays the spatial extent of work request alerts for the one-day scan, weekday, two-
day scan and seven-day scan. Since all of the alerts were false alerts, tight clustering is not exhibited.
The area of alerts is greater in the two-day and seven-day scans compared to the one-day scan, weekday
alerts. This is attributed to the greater number of work requests necessary to generate alerts for these
algorithms. Overall, the average spatial extent was 113 square miles.
The distribution of the areas of the one-day scan, weekday and two-day scans are similar, which is
expected since the number of work requests producing an alert differs by one additional work request
(three and four respectively). The alert areas for the two-day scan are slightly greater than the alert areas
for the one-day scan. The seven-day scan has the greatest distribution of alert areas as well as greater
alert areas generally. This suggests that the alert area increases as the number of work requests generating
the alert increases. Additionally, the average alert area increases as the number of work requests
increases.
38
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
450
_£_
o 35°
X
=
V)
«*" 9nn
D
tr
*v 1 nn
2
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
alerts are included in this analysis, despite the lack of investigations. Using actual evaluation data, alert
rates at various thresholds were generated using the AET.
Results: A total of 157 invalid alerts were generated by the work request EDAs during the evaluation
period. Note that the number of invalid alerts generated differed from the number of alerts investigated,
since alert notification ceased in December 2008. While the number of invalid alerts varied by reporting
period, each reporting period experienced an average and median of 5.4 and 3 invalid alerts, respectively.
The distribution of invalid alerts per reporting period can be seen in Figure 5-2. Early on, thresholds
were set low to generate more alerts to aid in system optimization. The thresholds were raised at the end
of April 2008, after the early optimization period was complete. Alerting rates decreased immediately
and remained low for several months before an abrupt jump in the December 2008-January 2009
reporting period. The jump occurred when customers participating in a regulatory compliance monitoring
program called the utility to schedule collection of the household samples. While these were not water
quality complaints, the CSRs categorized them as water quality work requests. The same activity
occurred again during the September 2009 reporting period, causing the highest number of invalid alerts
(22) during the evaluation period. In general, as with IVR alerts, CUSUM and seven-day scan alerts for
work requests only contributed to the volume of alerts, and did not cause any investigations during
months when other algorithms did not alert.
V.
Beginningof
Real-time Analysis
1
II 1 .
II,
,
.1
CUSUM
Seven-day Scan
Two-day Scan
One-dayScan, Weekday
ll
>\^\^fe^\^\^\^\^\^\^^^
Start Date of Monthly Reporting Period
Figure 5-2. Work Request False Alerts per Reporting Period
As shown in the Figure 5-3 histogram, the majority of the reporting periods experienced five or fewer
invalid alerts.
40
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
\
6-10 11-15
Invalid Alerts
16-22
Figure 5-3. Histogram of Work Request Invalid Alerts per Reporting Period
Figure 5-4 displays how the scan algorithms behave at various thresholds based on analysis of historical
work request data. The y-axis displays the number of alerts that are generated at each threshold. Arrows
indicate the thresholds utilized by GCWW at the end of the evaluation period (June 15, 2010). The reset
one-day scan, weekday and continuous two-day scan algorithms generally exhibit similar patterns. Both
experienced the maximum number of alerts at the lowest tested threshold, although the two-day scan did
exhibit a slight uptick at a threshold of 6. Note this is in contrast to the historical IVR 5 data, where the
reset one-day scan, weekday variable displayed a much different pattern than the continuous two-day scan
algorithm (Figure 4-3). This is due to the lower volume of water quality work request complaints
available and the different threshold levels for the work request data stream. The maximum number of
alerts expected for the reset one-day scan, weekday and continuous two-day scan algorithms are 106 and
80, respectively at a threshold of 2. Expected alerts decrease as the threshold increases until no alerts
should be produced, at a threshold of 7 for the reset one-day weekday scan and a threshold of 9 for the
continuous two-day scan.
41
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
120
100
Scan 1 Day, Weekday
Scan 2 Day
Scan? Day
3
5
5
-------
5.4. 1
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Time for Alert Recognition
There is no metric data for time for alert recognition that reflects performance during the real-time
monitoring period, since work request alert notifications were turned off starting December 17, 2008.
While GCWW was receiving work request notifications during January 1, 2008 - December 16, 2008, the
utility was not expected to immediately respond to alerts during this time period.
5.4.2 Time to Investigate Alerts
Definition: Investigation time measures the time spent by staff to investigate invalid alerts. Since there
were no true contamination incidents, this metric represents all time spent by GCWW staff on alert
investigations.
Analysis Methodology: Average and median times between the start and end of an investigation, as
noted in investigation checklists are calculated. Unlike the FVR and work order data streams,
investigation data is only available from January 1, 2008 - December 16, 2008. Even though the utility
was not expected to immediately respond to alerts, the utility did investigate alerts by utilizing the
checklists when they had time to do so.
Results: Table 5-1 shows the average, median and range of investigation times. A total of 50 work
request alerts were investigated, with each alert investigation taking an average of 8.1 minutes to
complete. While the recorded range of investigation times was zero to 65 minutes, the majority of
investigations took five minutes. A few outliers influenced the averages, and may have been the result of
investigators pausing in the middle of investigations to complete other work. Had the investigators been
required to immediately respond to the alerts, they likely would not pause to complete other work.
Table 5-1 . Time for Work Request Alert Investigation
Algorithm
Seven-day Scan,
Weekday
Two-day Scan
Seven-day Scan
CUSUM
Total
Average
4.6
5.5
20
9.6
8.1
Median
5
5
5
5
5
Range
1 -10
1 -10
2-65
2-40
1 -65
The average time for alert investigation for each algorithm is shown in Figure 5-5. Although the seven-
day scan investigations, on average, took the longest, the median investigation time was five minutes.
43
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
One-day Scan, Two-day Scan Seven-day Scan
Weekday
CUSUM
Total
Figure 5-5. Time for Work Request Alert Investigation
Overall, work request alert investigations take approximately five minutes. The work request
investigation procedures, as well as work order investigations, are designed to quickly identify invalid
alerts. If the customer complaints are not clustered or do not have the similar problem descriptions, then
the alert is determined to be an invalid alert. The bulk of work request alerts meet these criteria. The
alert notifications list all of the work requests involved in the alert, the problem descriptions entered in a
free text field, and the neighborhood and street address of each work request. Thus, invalid alerts can
often be identified from viewing the alert email alone. Because of this, the labor hours required to
investigate work request alert emails is minimal.
5.4.3
Summary
Median time to investigate alerts was five minutes. This relatively short investigation time can be
attributed to the details provided in the alert email, which were often enough to quickly rule-out possible
water contamination as the source of the complaints.
5.5
Design Objective: Operational Reliability
Operational reliability considers metrics that address aspects of operation of the work order data stream.
These metrics quantify the percent of time that the tool is working as designed. In order to evaluate how
well the work order data stream met this design objective, the following two metrics are evaluated:
availability and data completeness. These metrics are described in the following sections.
5.5.7
Availability
Definition: Availability measures the amount of time that useful data is accessible for analysis during
alert investigations. For the work request data stream, availability is based on the usability of the Hydra
system; downtime of the Hydra system causes work request data to be unavailable. Since the EDAs are
shared with the other data streams, they will be discussed on the integrated component level (Section 7.0).
Analysis Methodology: Any downtime for the Hydra system is reported.
Results: As shown in Figure 5-6, the Hydra system experienced some downtime, particularly during the
July 2008 reporting period. This was due to a power outage that outlasted the uninterrupted power
supply, resulting in multiple problems across GCWW information technology systems during the reboot.
44
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
In an unrelated problem later that same reporting period, the Hydra Map became unresponsive and did not
display alerts for nearly 24 hours. In February 2009, an issue intermittently arose when two users logged
into the system at the same time. This resulted in several instances of downtime lasting approximately 45
minutes each. Additionally, there was an instance similar to the July 2008 reporting period where the
Hydra Web service became unresponsive for three hours. Aside from these unanticipated downtime
events, the work request data stream experienced little unavailability. Once a month, the servers are taken
down for maintenance. Maintenance is scheduled for the evening of the fourth Sunday of every month to
minimize the impact of these brief periods of unavailability.
^
o
« 15
.1
?
§1 n
o
5
1
Beginningof
Real-time Monitoring
1.
Start Date of Monthly Reporting
1
i _^ -^ _*& -^ -^ -^ -^ -^ -^ -^
IO-* N^* l-v^ N^-* l^* *O-* N^* V^ N^* >^
Period
Figure 5-6. Downtime of the Hydra System
5.5.2
Summary
The work request data stream had high availability due to limited downtime of the Hydra system. The
disruptions to this data stream were minimal. Overall, the work request data stream is very reliable
(99.75%) with respect to availability.
5.6 Design Objective: Sustainability
5.6.1 A cceptability
Definition: Acceptability is the willingness of persons and organizations to monitor, maintain and
actively participate in a program. The acceptability of the investigation procedures is quantified in the
analysis through investigation checklist usage, where completed checklists demonstrate that the
investigation processes is feasible and not burdensome.
Analysis Methodology: Alert investigations are documented in investigation checklists. The percentage
of alerts investigated is calculated based on the actual number of investigation checklists used divided by
the total number of investigation checklists expected. Data from January 2008 through December 2008
was used.
Results: Fifty-one work request data alert investigations were performed during the optimization period.
Figure 5-7 shows the distribution of alerts among the one-day scan, weekday; two-day scan; seven-day
45
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
scan; and CUSUM algorithms. The one-day scan, weekday algorithm accounted for about half of the total
alerts received.
25
One-day Scan, Weekday
% Two-day Scan
Seven-day Scan
CUSUM
11
Figure 5-7. Work Request Alert Investigations Completed by Algorithm
Figure 5-8 displays investigation checklist usage during the period which GCWW was receiving and
responding to work request alert notifications (January 1, 2008 - December 16, 2008). Overall, 88% of
work request data alerts were investigated. The lowest percentage of alerts investigated occurred in the
first reporting period, when investigators first began receiving alert notifications. After January 2008, the
majority of alerts were investigated, indicating acceptance of the investigation procedures by the WQ&T
chemists.
Not Invest!gated
Investigated
^ nT &
Start Date of Monthly Reporting Period
Figure 5-8. Investigations Completed and Alerts Not Investigated, by Reporting Period
46
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
5.6.2 Summary
Most work request alert investigations were performed on one-day scan, weekday alerts, as these were the
most frequent alerts. While the work request data stream was dropped in December 2008, prior to this
time work request alerts were investigated 88% of the time.
47
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 6.0: Performance of Work Order Data Stream
The following section provides a description of the work order data stream followed by the results of the
evaluation of this data stream. This analysis includes an evaluation of metrics that characterize how the
work order data stream achieves the design objectives described in Section 1.1. Specific metrics are
described for each of the design objectives.
6.1 Description of the Work Order Data Stream
After generation of a water quality work request by the CSR, the water quality complaint call is
forwarded to the WQ&T Customer Water Quality Representative. The WQ&T representative may ask
further questions about the specific taste, odor or color complaint and will make a decision based on
professional experience and knowledge whether sampling is needed. If the WQ&T representative
determines the complaint cannot be explained by presence of illness, cross-connection or loss of pressure
and that field sampling is warranted, a work order is created to initiate the sampling process. Caller
information (including time, date, location and type of water quality concern) is logged into the Hydra
system and analyzed by EDAs.
Work order data is collected and analyzed in near real-time (every minute) via Web services. Post-
analysis data are stored for historical evaluation. As with the other CCS data streams, the application of
EDAs automate the analysis of work order data. Work order data is uploaded and analyzed frequently to
quickly identify work order data sets above a predetermined threshold over a particular time period.
Thresholds were determined after analysis of historical work order data, and can be adjusted based on
recent trends.
The algorithms applied to the work order data stream are the same as those used in FVR and work request
data analysis: one-day scan, weekday, one-day scan, weekend, two-day scan, seven-day scan, and
CUSUM. A description of these EDAs can be found in Table 2-2. Work order volume above
predetermined thresholds produces automated email and text message alerts sent to the WQ&T Chemist,
CSR Supervisor, Distribution Dispatch Supervisor and WUERM for review. The WQ&T Chemist then
completes an investigation checklist to determine whether water contamination is possible. If the
threshold is not reached, the algorithms continue to monitor the work order data stream.
Work order alert complaint locations are visually displayed on the GIS Hydra Map to facilitate
identification of spatial clustering. Similarities in complaints between callers were also considered when
making the determination of Possible water contamination. Alert notifications for the CUSUM algorithm
were turned off January 2009 due to the difficulty in its interpretation and redundancies with other alerts;
this is consistent with the analysis of the IVR and work request data streams. In addition, the seven-day
scan algorithm alert notifications were also turned off in January 2009, as it was determined that these
alerts were unlikely to be related to water contamination.
6.2 Design Objective: Spatial Coverage
The spatial coverage is the cumulative area of the distribution system where a detectable change in water
characteristics could be reported via customer complaint calls and the alerts from these calls could be
detected. Spatial coverage depends on spatial extent of alerts and population coverage. Analysis of the
areas included in work request alerts can help determine spatial distribution and extent of alerts. To
determine how well the work request data stream achieves this design objective, the spatial extent of
48
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
alerts was evaluated. Since population coverage and area coverage is the same for FVR, work request and
work order data streams, this metric is discussed at the component level (Section 7.0).
6.2.1
Spatial Extent of Invalid Alerts
Definition: This metric refers to the geographical coverage of work order alerts in the GCWW service
area. Customer complaints are expected to cluster during actual contamination incidents.
Analysis Methodology: For each work order alert, the area of the bounding rectangle of the work order
involved is calculated. The bounding rectangle is the smallest rectangle possible that contains all of the
work orders contributing to the alert within its borders. Effectively, the bounding rectangle reduces to the
coordinates furthest east, west, north and south from this set of work orders. Given the irregular shape of
GCWW's retail area, the calculated area of the bounding rectangle may have an area greater than
GCWW's retail area. Since thresholds were set low early during the optimization period, the small
number of work orders involved in many alerts do not give an accurate depiction of the spatial extent of
invalid alerts. Thus, the distribution of spatial extent of alerts since the thresholds were raised in April
2008 is calculated.
Results: Figure 6-1 displays the spatial extent of work order alerts for the one-day scan, weekday, two-
day, and seven-day scans. Since most of the alerts were invalid, clustering was not exhibited. The
distribution of alert areas mirrors the work request alerts. There were two alerts (see Section 7.2.3) which
were valid alerts.
450
One-day Scan, Weekday
Two-day Scan
Algorithm
Seven-day Scan
Figure 6-1. Spatial Extent of Work Order Alerts
The seven-day scan has the greatest distribution of alert areas as well greater alert areas generally. This
follows a similar pattern to the work request data stream. For reference, the GCWW distribution system
covers an area of approximately 356 square miles. Given the irregular shape of GCWW's retail area, the
calculated area of the bounding rectangle has an area greater than GCWW's retail area.
49
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
6.2.2 Summary
Spatial clustering was rare in work order alerts. The analysis showed that unrelated work orders that
contributed to invalid alerts were regularly distributed across large areas. While valid alerts, which are a
result of system events, had smaller areas for the one-day scan, the work orders were not clustered.
Overall, the average area was 118 square miles.
6.3 Design Objective: Alert Occurrence
Alert occurrence addresses aspects of system performance, including the frequency of invalid alerts, in
order to ascertain the accuracy of the tool in discriminating between real incidents (water contamination,
system event or public health incident) and normal variability in the underlying data. In order to evaluate
how well the work order data stream met this design objective, the following two metrics were evaluated:
invalid alerts and sensitivity. These metrics are described in the following sections.
6.3.1 Invalid Alerts
Definition: Any work order alerts determined not to be an indication of contamination or a system event
are considered invalid alerts. The quantity of invalid alerts is dependent on the threshold level set for
each EDA. To date, there have been no valid alerts generated by the work order data stream indicative of
water contamination; therefore, all empirical data represent an invalid alert or a valid alert not related to
water contamination. Two valid alerts, not related to water contamination, are discussed in Section 7.2.3.
Analysis Methodology: Summary statistics and temporal analysis of the number of invalid alerts are
presented, both in total and per reporting period. Alert rates at various thresholds were generated using
the AET and are discussed.
Results: During the evaluation period, 160 invalid alerts were generated through the work order data
stream. The number of invalid alerts generated for each algorithm per reporting period can be seen in
Figure 6-2. Invalid alerts were more frequent initially in the evaluation period due to threshold levels
being set purposefully high in order to practice investigational procedures. Note that reporting periods
with the highest counts of work order invalid alerts do not necessarily correspond to reporting periods
with the highest work request alerts. This is due to different threshold settings for the work order and
work request data streams.
50
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
9 "^
Invalid Alerts
-1 |_1 |S
D Ln C
0 -
Beginning of
Real-time Monitoring
CUSUM
1 Seven-day Scan
Two-day Scan
One-day Scan, Weekend
One-day Scan, Weekday
' ii i i ii ii
jX! :! ;:H
,1 ||||.
Start Date of Monthly Reporting Period
Figure 6-2. Work Order Invalid Alerts per Reporting Period
Over half (52%) of reporting periods experienced five or fewer alerts, as indicated in the histogram in
Figure 6-3. Reporting periods, on average, experienced 5.5 invalid alerts each.
5 10
Invalid Alerts
More
Figure 6-3. Histogram of Work Order Invalid Alerts per Reporting Period
Figure 6-4 displays how the scan algorithms behave at various thresholds based on analysis of historical
work order data. The y-axis displays the number of alerts that are generated at each threshold. Arrows
indicate the thresholds utilized by GCWW at the end of the evaluation period (June 15, 2010).
51
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
120
100
One-day Scan Weekday
Two-day Scan
Seven-day Scan
10 11
12 13
Threshold
Figure 6-4. Expected Alerts at Various Thresholds for Historical Work Order Data
The number of alerts decreases from a maximum of 116 alerts as the threshold increases for the reset one-
day scan, weekday algorithm. For the continuous two-day scan, the alert rate remains at its maximum of
88 until a threshold of four is reached, at which point the alert rate also decreases with increasing the
threshold. The continuous seven-day scan algorithm starts at an alert rate of 19 with a threshold of two,
and then increases to its maximum alert count of 43 at a threshold of 4, before decreasing as the threshold
increases. As mentioned in previous sections, the pattern for the continuous seven-day scan algorithm
differs from the others due to a longer scan window operating every day.
6.3.2 Summary
The number of invalid alerts for the work order data stream varied by reporting period, with a slight
majority of reporting periods experiencing five or fewer invalid alerts. An analysis using the AET
showed that alert rates per threshold varied by algorithm, but after a certain point for each alert, rates
decrease as threshold increases.
6.4
Design Objective: Timeliness of Detection
This objective consists of the time for the work order data stream to detect a possible water contamination
via standard operation, analysis and investigation. Metrics under this objective include: time for data
transmission, time for event detection, time for alert notification, and time to investigate invalid alerts.
Time of data transmission and event detection are discussed at the component level, as they are the same
for all three customer complaint data flows. The following two metrics were evaluated and described
below to determine how well the work order data stream achieves this objective: time for alert
recognition, and time to investigate alerts.
6.4.1 Time for Alert Recognition
Definition: Time for alert notification quantifies the time for investigators to recognize the email or text
message alert notification and begin their investigation. Like the work request investigations, the
responsibility for work order data alerts investigation belongs to the WQ&T Chemist.
52
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Analysis Methodology: Average and median time between delivery of the alert email or text message
and start of alert investigation is calculated.
Results: The work order alert recognition times were analyzed for the evaluation period, with the
average, median and range for each algorithm shown in Table 6-1. On average, GCWW staff took 11.6
minutes to recognize each alert. After the system was fully developed in January 2009, investigators were
expected to respond to alerts in real-time. This is representative of normal daily operation. There was
one outlier of 1,446 minutes, which was excluded from the statistical analysis.
Table 6-1. Time for Work Order Alert Recognition (Minutes)
Algorithm
One-day Scan, Weekday
Two-day Scan
Overall
Average
19.7
7.3
13.8
Median
5
3
3
Range
0-1,446
1 -27
0-1,446
Figure 6-5 demonstrates that most alert investigations were initiated within four minutes of receiving the
alert based on the median values, indicating that email is an effective method for alert notification for
work order alerts.
One-day Scan,
Weekday
Two-day Scan
Total
Figure 6-5. Work Order Alert Recognition Times During Real-time Monitoring
The response times during normal operations indicate that work order alerts can be a timely indicator of
water contamination. Additionally, 90% of work order alerts were investigated during the entire
evaluation period demonstrating the value utility personnel place in the data stream.
6.4.2
Time to Investigate Alerts
Definition: This metric describes the time spent by staff to investigate invalid alerts. Since there were no
actual contamination incidents in the evaluation period, this metric represents all time spent by GCWW
staff on work order alert investigation.
Analysis Methodology: Average and median time between the start and end of an investigation, as noted
in the investigation checklists, is presented.
53
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Results: Alert investigation times were analyzed for the evaluation period, with the average, median and
range for each algorithm shown in Table 6-2. On average, GCWW staff spent 7.7 minutes investigating
each alert.
Table 6-2. Time for Work Order Alert Investigation (Minutes
Algorithm
One-day Scan, Weekday
Two-day Scan
Overall
Average
7
8.6
7.7
Median
5
7
6
Range
3-15
5-20
3-20
As shown in Figure 6-6, average investigation times were similar for each of the EDAs, ranging from 7
to 8.6 minutes per alert. No invalid alert investigations took longer than 20 minutes.
16
14
12
!310
lAverage
i Median
One-day Scan, Weekday Two-day Scan
Total
Figure 6-6. Average Time to Investigate Work Order Invalid Alerts
6.4.3 Summary
Most email alerts were investigated within four minutes and took an average of approximately eight
minutes to investigate. Investigation times ranged from three to twenty minutes.
6.5
Design Objective: Operational Reliability
As with the other data streams, the work order subcomponent reliability considers the performance and
operation of a data stream. Measures of performance quantify the accuracy of the tool in discriminating
between an actual water contamination incident and normal variability in the underlying data. Reliability
is characterized by availability.
6.5.1
Availability
This measures the amount of time that useful data is accessible for analysis during alert investigations.
As with the work request data stream, work order data stream availability is based on the usability of the
54
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Hydra system and the ED As; downtime of the Hydra system causes work order data to be unavailable.
Hydra availability is discussed in Section 5.5.1. Since the EDA is shared with the other data streams,
EDA is addressed in Section 7.0.
6.6 Design Objective: Sustainability
6.6.1 A cceptability
Definition: Acceptability is the willingness of persons and organizations to monitor, maintain and
actively participate in a program. The acceptability of the investigation procedures is quantified in the
analysis through investigation checklist usage.
Analysis Methodology: Alert investigations are documented in investigation checklists. The percentage
of alerts investigated is calculated by dividing the number of investigation checklists completed by the
number of investigation checklists expected. Data from the real-time monitoring period is used. Since
the seven-day scan and CUSUM algorithms were disabled at that time, investigation checklists were not
expected for those alerts.
Results: During the evaluation period, a total of 27 work order investigations were conducted. As shown
in Figure 6-7, the number of investigations per algorithm was similar. No weekend alerts occurred
during the analysis period.
12
15
One-day Scan, Weekday
Two-day Scan
Figure 6-7. Work Order Alert Investigations Conducted
Figure 6-8 shows that the majority (87%) of work order alerts were investigated during the evaluation
period. Alerts investigated during the first reporting period contained valid alerts and were investigated
as described in Section 7.2.3. WQ&T Chemists were still responsive to work order alerts in most
reporting periods in which alerts were received. Alerts that were not investigated were correctly
identified by investigators as duplicate alerts which were dismissed.
55
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
I
3
i Alerts Not Investigated
I Alerts Investigated
I
Start Date of Monthly Reporting Period
Figure 6-8. Work Order Alerts Investigated and Not Investigated by Reporting Period
6.6.2 Summary
Nearly all work order alerts (87%) were investigated during the entire evaluation period, demonstrating
the value utility personnel place in the work order data stream. During six reporting periods, there were
no alerts, but alerts occurred often enough to allow personnel to maintain their familiarity with the
investigation process. Investigations were performed during more than half of the monthly reporting
periods.
Note: Currently at GCWW, email is now automatically generated, which contains information about all of
the work orders that are part of the alert. In essence, this allows anyone receiving the email to conduct a
cursory investigation and the WQ&T Chemist would not necessarily need to support the investigation.
This process would likely result in more timely detection of a contamination incident (especially on
weekends) and builds additional layers of redundancy into the system.
56
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 7.0: Performance of the Integrated Component
Sections 4 through 6 describe metrics related to evaluating specific CCS data streams. This section
describes metrics not covered in the previous sections which were used to evaluate the integrated CCS
component, including all data streams.
7.1 Design Objective: Spatial Coverage
The spatial coverage is the cumulative area of the distribution system where a detectable change in water
characteristics could be reported via a customer complaint call. Factors contributing to spatial coverage
include area coverage, spatial extent of alerts and population coverage. Spatial extent of alerts is
discussed in Sections 5 and 6. To assess how well the integrated CCS component achieves this objective,
population coverage is described below.
7.1.1 Area Coverage
Definition: This metric consists of the physical area of GCWW's distribution system with potential
access to telephone service to reach the IVR system.
Analysis Methodology: Review of GCWW service area and regional telephone service availability.
Results: The entire GCWW service area of 356 square miles theoretically has access to telephone service,
providing the CCS component with extensive area coverage.
7.1.2 Population Coverage
The population coverage for CCS data encompasses anyone within the GCWW service area who can
place a customer complaint call; essentially, anyone with a telephone. Coverage is also affected by the
population density of the GCWW service area.
Analysis Methodology: The methodology used to assess population coverage was an assessment of
empirical data analysis of U.S. Census data.
Results: The US Census Bureau estimates that 96.3% of Hamilton County, Ohio, occupies housing units
with telephone access (2006-2008 American Community Survey table S2504). This is slightly higher
than the Ohio and US percentage, both at 95.7%. Assuming the demographics of all GCWW customers
are similar to that of Hamilton County, then population coverage for the IVR data stream is very robust,
including nearly all customers.
Population coverage is also affected by the population density of the GCWW service area. Areas with a
higher population density should be more likely to generate complaint call volume that exceeds the CCS
analysis thresholds. As seen in Figure 7-1, the service area with the highest population density is in
downtown Cincinnati. Population density gradually decreases farther from the city core.
57
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
E Treatment Plant
| Service Areas
Figure 7-1. Population Density of the GCWW Service Area
7.13
Summary
Because of the high percentage of customers with access to a telephone, IVR data provide excellent
population coverage for detection of a possible contamination incident. Areas around downtown
Cincinnati may be more likely to generate an IVR alert due to higher population density.
7.2
Design Objective: Contaminant Coverage
The CCS component monitors customer complaint calls that could potentially be associated with certain
contaminants. This design objective aims to define the type and amount of contaminant that would need
to be present to elicit an alert from the CCS component. The following metrics are analyzed to ascertain
how well the CCS component achieves this design objective: contamination scenario coverage,
contaminant detection threshold, invalid alerts and valid alerts. These metrics are described in the
following sections.
7.2.1
Contamination Scenario Coverage
Definition: Contamination scenario coverage is defined as the ratio of contamination incidents that are
actually detected to those that are detectable based on design detection capability of the data stream.
Theoretical detection is based on the type of contaminant and the area this component operates within
(GCWW service area). The CCS component is designed to detect contaminants that result in a
discernible change in water properties, as discussed above.
Analysis Methodology: The simulation study was used to quantify the CCS contamination scenario
coverage. Of the 17 contaminants utilized in the simulation study, six were theoretically detectable by
58
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
CCS (Table 3-2). Scenarios in which the injection was at a facility where ESM would detect the intrusion
quickly were excluded from the analysis. If there was an IVR or work order alert generated by the
simulation, the scenario is considered to be detected by CCS.
Results: The six theoretically detectable contaminants produced 564 simulated scenarios at distribution
attack nodes. In every scenario, CCS detected the contamination with either IVR or work order alerts.
IVR alerts preceded the work order alerts in 559 of the scenarios. This is expected, since the model
assumes that every water quality complaint called into the utility will eventually be converted to a work
order.
In five scenarios, the work order data stream was the first or only CCS data stream to detect; there are two
scenarios for which there were no IVR alerts. All of these were low impact, low casualty and little
infrastructure damage scenarios. In the three remaining scenarios of which the work order alert preceded
an IVR alert, the IVR alert followed the work order alert an average of 99 minutes later. The work order
thresholds are slightly lower than the IVR thresholds, since many self-identified water quality complaints
are screened out by the CSRs. However, in the simulation model, all complaints are converted to work
orders. Thus, the scenarios in which the work order system detected contamination before the IVR
system involve very few callers to the utility in the early stages of the event.
CCS detected every scenario which was theoretically detectable by the component (i.e. had a taste, odor
or appearance characteristics above a detectable threshold). The simulation scenarios were selected to
model water contamination doses which would pose a significant risk on public health or infrastructure.
At the necessary exposure concentrations during these risk scenarios, CCS will most likely detect the
contamination.
7.2.2 Contaminant Detection Threshold
Definition: The contaminant detection threshold for CCS is related to the number of exposures prior to an
alert. This metric is intended to characterize the size of the smallest contamination incident, expressed in
terms of the number of people affected, that can be detected through the CCS component.
Analysis Methodology: The simulation study was used for this analysis by first identifying all
contamination scenarios (n=564) that were detected by the CCS component during the simulation study
and then quantifying the number of exposures before the first EDA alert was generated for each scenario.
This information was statistically analyzed across all scenarios (e.g., calculation of mean, standard
deviation, interquartile range, etc.). Analyses were also performed on sub-sets of scenarios, in particular
by contaminant.
Results: Figure 7-2 shows the distribution of exposures prior to the first CCS alert. The overall average
of the number of exposures before the first alert is 195 with a median of 81 exposures. There is a median
of 81 exposures before the first IVR alert and a median of 171 exposures before the first work order alert.
The difference between the numbers of exposures is a product of the time delay between a customer
calling (entering the IVR data stream) and creation of a work order. While the work order data stream is
timely, the delay does result in potentially more customers exposed to contamination.
While these values may seem high, they represent a small percentage of overall exposures for a scenario.
The five scenarios in which the work order data stream was the first or only CCS data stream to detect
had few exposures before producing the alert. The maximum number of exposures in these scenarios was
30, with a minimum of 8. As discussed in the previous section, these are low casualty and infrastructure
damage scenarios, resulting in fewer people detecting the contaminant by the time each CCS EDA
analyzes the number of complaints within the algorithm window.
59
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Over 92% of the scenarios have fewer than 500 exposures prior to the first CCS alert. These scenarios
typically represented injections of large volumes of a contaminant in population dense areas. The
contaminant was able to reach a large number of people before the customers first begin consuming the
water.
-I n nnn
1 nnn -
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Date, Time
8/28/08, 13:35
8/28/08, 14:01
8/28/08, 14:01
8/28/08, 16:52
Data Stream
Work Request
Work Order
Work Order
Work Order
Algorithm
One-day Scan,
Weekday
Two-day Scan
CUSUM
Seven-day Scan
January 16, 2009: Distribution Work
Two customers on the same street complained of cloudy water with bad taste and odor. The work orders
were four days apart. The investigator attributed the complaints to GCWW work being performed in the
area. The second customer explained that the flushing done as a result of the first call had helped improve
the water quality, but the taste/odor problems continued. The two related work orders contributed to
elevating the algorithms past the threshold, which resulted in three alerts over the ensuing week. Table 7-
2 provides details for these alerts.
Table 7-2. Distribution Work Alerts
Date, Time
1/16/09, 10:07
1/16/09, 10:07
1/16/09, 10:07
Data Stream
Work Order
Work Order
Work Order
Algorithm
Two-day Scan
CUSUM
One-day Scan,
Weekday
7.2.4
Summary
CCS is designed to detect contaminants that cause a discernible change in the taste, odor or appearance of
water. Though no contamination incidents occurred during the evaluation period, system events
concerning elevated chlorine levels and distribution work generated valid alerts. In 564 simulation study
scenarios, the overall average of the number of exposures before the first alert was 195 with a median of
81 exposures, which is a small percentage of the overall exposures for each scenario.
7.3
Design Objective: Timeliness of Detection
Timeliness of detection describes the overarching timeline from receipt of the customer complaint,
transmission of the complaint data, processing of the data by the EDAs and recognition of the alert by the
investigator. The CCS application manages the data transmission and analysis functions for all of the
data streams together. Thus, this portion of the timeline is discussed in this section. Alert recognition
times for each data stream are summarized in their respective sections.
7.3.1
Time for Data Transmission
Definition: Time for data transmission is the amount of time it takes customer complaint call entered
into the system to be available for analysis by the CCS application. As discussed in the description of the
CCS data streams, the customer complaint data is accessed through Web services by the CCS application,
which handles all of the data streams. Data is queried by the application every minute.
61
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Analysis Methodology: Over the course of the evaluation, the CCS application has queried data over one
million times; thus, only a small sample of data transmission times was selected for analysis.
Transmission time is measured as the difference between when the data is requested and when the data is
received. Average and median transmission times are calculated.
Results: The time for data transmission for algorithms is less than 300 milliseconds (ms). Given other
aspects of the event detection timeline, the data transmission time is negligible. A two minute delay is
parameterized in the application. For example, at 1:53 PM the application asks for data that entered the
system at 1:51 PM. This allows time to completely process new data before the data is available to the
application, resolving a programming error which resulted in missing complaint data.
The work request and work order data streams have additional delays associated with handling
complaints. The IVR data is available as soon as the customer calls GCWW to report a water quality
issue. However, the customer must speak with a CSR before a work request can be generated, which can
take between two and four minutes, as verified in CCS Drill 3. Work orders are only created after the
customer has spoken to a WQ&T Chemist, who determines that the customer complaint requires action.
This process takes approximately 15 minutes.
The time for data transmission for all data streams is negligible when compared to other factors
influencing the time for event detection. Water quality customer complaints are available for analysis
within two minutes for the IVR data stream. The other data streams involve various delays attributed to
the complaint handling process that extend the timeline for data to become available for analysis. Thus,
the complaint handling process introduces delays which reinforce the timeliness of the IVR data stream.
7.3.2 Time for Event Detection
Definition: Time for event detection measures the amount of time necessary to analyze the data and
identify an anomaly. The EDAs execute every minute.
Analysis Methodology: Over the course of the evaluation, the CCS application has queried data over
one million times; thus, only a small sample of data transmission times was selected for analysis.
Execution time is measured as the difference between when the execution begins and ends. Average and
median is also calculated.
Results: As with the time for data transmission, the EDA's execution time is negligible. In most cases,
each data stream is analyzed within an average of 15 milliseconds. The CCS component relies on a
multi-threaded application, which expedites the processing of the data. There was one outlier in the IVR
data stream execution which influences the average. However, this is still less than a fifth of a second.
7.3.3 Time for Initial Detection
Definition: Time for initial detection measures the time between the start of a simulated contamination
incident and the first CCS alert.
Analysis Methodology: The simulation study was used to quantify the time for initial detection.
Average, median and interquartile ranges of the difference between the simulated injection time and the
first CCS alert were calculated for simulated water contamination scenarios that CCS detects. Only
scenarios from distribution attack nodes were included in the analysis.
Results: Evaluating the time for initial detection for all theoretically detectable scenarios can be
misleading since contamination incidents which start during the low demand time (midnight) do not have
a chance to generate a CCS alert until after the first simulated customers ingest the water in the morning.
To account for this, the analysis was filtered by high demand (n = 441) and low demand (n = 123)
62
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
injection scenarios, shown in Figure 7-3. As expected, there is a significant difference between the
averages of the high (32 minutes) and low (404 minutes) demand scenarios. However, the interquartile
ranges are very narrow and spread around the first consumption time, whether they are high or low
demand scenarios. This indicates that once a contaminant reaches customers who are consuming the
water, enough customers will call the utility in a short time to quickly surpass the EDA thresholds.
800
TOO
ROO
U) DUU
§
c
= EJOO
o
+ AC\C\
0
0
300
.£S
|^
'c
<- 900
o
q-
I ± =l
I ^
> 1
Full Ensemble LowDemand High Demand
Scenario Type
Figure 7-3. Time for Initial Detection
The CCS component is not only quick to detect contamination (as measured from the time of injection),
but it is also often the first component to detect contamination. Of the 564 theoretically detectable
scenarios, CCS was the first component to detect the contamination in 547 scenarios (97%). Of the
remaining 17 scenarios, WQM was the first component to detect in every scenario except one. Fifteen of
these 17 scenarios (88%) involved low demand injection times, which is to be expected since the first
consumption by customers does not occur until several hours after the injection. In these distribution
attack scenarios, WQM has several hours to detect contamination before the first customers drink the
water, providing the first opportunities for CCS and PHS to detect. Additionally, 14 of the 17 scenarios
(83%) were Nuisance Chemical 1 scenarios, in which the contaminant is injected in a high bulk volume,
creating a greater chance that it will hit a WQM station in sufficient concentrations to produce an alert.
The simulation results indicate that for contaminants with taste, odor or appearance properties, CCS can
be critical to early detection of water contamination.
7.3.4
Time to Validate Possible Contamination
Response time metrics during routine operation of the CCS component were discussed in Sections 4, 5
and 6 for the IVR, work request and work order data streams, respectively. Aside from the few valid
alerts received during the evaluation period, none of the routine operation alert investigations reached the
Possible stage, so the investigation time to reach a Possible determination could not be captured. This
section covers alert response investigation times observed during CCS drills, all of which reached the
Possible stage. Table 7-3 compares the key metrics among all three drills. Information on each drill
scenario is presented in Section 3.2. Additionally, CCS alerts contributed to FSE 2, FSE 3 and the CCS
63
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
site-characterization drill, details of which can be found in the Water Security Initiative: Evaluation of the
Consequence Management Component of the Cincinnati Contamination Warning System Pilot (USEPA,
2014b).
Table 7-3. CCS Drill Metrics
Metrics (Times in minutes from alert)
Time for call waiting in IVR queue
Time for CSR Supervisor follow-up with CSRs
Time forWQ&T Chemist to examine Hydra Map
Time to contact WQ&T Supervisor
Time for completion of IVR investigation checklist
Time for Distribution Dispatcher to complete alert
investigation
Time for Distribution Dispatch Supervisor to contact CCS
alert group
Time for CSR Supervisor to contact Distribution Dispatch
Supervisor
Time for WQ&T Chemist to complete alert verification
checklist
Time for WQ&T Chemist to review work requests
Time for WQ&T Chemist to consult WUERM regarding
site characterization
Time for WQ&T Chemist to develop work order
Time for additional call handling staff determined
Time for consideration of emergency message
deployment on IVR
Time for the investigators to conclude that contamination
was Possible
Drill 1
8/19/08
2
5
8
15
16
20
21
21
22
28
29
40
FSE2
101/08
2
10
32
23
16
10
57
57
Drill 2
2/29/09
1
4
11
11
FSE3
10/21/09
1
9
35
20
9
47
32
33
44
Drill 3
4/15/10
1
4
19
19
Since CCS Drill 3 only involved the IVR data stream, its timeline is discussed in Section 4.4.2. CCS
Drills 1 and 2 and FSE 2 and 3 involved multiple data streams, so their timelines are discussed here.
CCS Drill 1 was conducted on August 20, 2008. The drill started at 9:03 am with the injection of an IVR
alert and concluded at 9:35 AM, although a few drill activities at the utility Call Center continued until
approximately 9:43 AM. The drill was concluded when the CCS alerts were investigated, including a
second inject of three additional work requests generated by the CSRs. The results of the investigation
were simulated to be communicated to the WUERM by the WQ&T Chemist. The time to fully investigate
the alert was 40 minutes. Figure 7-4 shows the timeline progression of the key activities completed
during the CCS alert investigation for CCS Drill 1. Note that the timeline was normalized so the start of
the investigation occurs at time 0.
64
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
00:00
IVR
Alert
00:05
Email query
sent to
GSRs
00:29
Need for
Emergency IVR
Message
Determined
00:15 00:28
WQ&T Customer 00:20 Need for
Water Quality Alert Verification Additional Call
Supervisor Checklist Handling Staff
Cont
acted Comf
r i
)leted Determinec
' i
' 1
00:40
Possible
Contamination
Determinatior
00:30 00:35
Work Request Work Order
Alert Developed for
- *
i
Distribution Staff
' i
00:00
00:40
Figure 7-4. Timeline Progression of the CCS Alert Investigation During CCS Drill 1
On October 1, 2008 a full-scale exercise was conducted that evaluated several components including
CCS. FSE 2 featured an IVR alert at 9:33 AM followed by a work request alert at 10:00 AM. The initial
IVR alert was investigated in 16 minutes. Possible contamination determination was made 40 minutes
after receipt of the work request alert. Figure 7-5 shows the timeline progression of the key activities
completed during the CCS alert investigation for FSE 2. Note that the timeline was normalized so the
start of the investigation occurs at time 0.
00:00
IVR Alert
00:10 00:16 .
Distribution
Dispatcr
Supervise
Contactec
IVR
Invest
r Che
Com
f i
Alert 00
igation Work R
cklist Alert R
pleted
' i
77
zl . WQ&T
e(lues' Review*
3celved Reqi
' i
Dhemist
;d Work
ests
00:57
Possible
Contamination
Determinatior
00:00
00:57
Figure 7-5. Timeline Progression of the CCS Alert Investigation During FSE 2
CCS Drill 2 was conducted on April 29, 2009 to evaluate alert recognition and investigative procedures
associated with the component during the non-business hours of the utility. The CCS alert was received
at 10:20 PM and the Distribution Dispatch Supervisor completed the IVR alert investigation checklist one
minute later. The time to fully investigate the alert was 11 minutes. Figure 7-6 shows the timeline
progression of the key activities completed during the CCS alert investigation for CCS Drill 2. Note that
the timeline was normalized so the start of the investigation occurs at time 0.
00:00
IVR Alert
00
IVR
Invest
Che
Comf
01
Alert
gallon
;klist
Jleted
00
Distribut
Invest
Che
Com
04
on Alert
gallon 00
=klist Work F
Dieted Al
00:11
Possible
05 Contamination
equest Determination
srt Reached
' i
00:00
00:11
Figure 7-6. Timeline Progression of the CCS Alert Investigation during CCS Drill 2
65
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
On October 21, 2009, FSE 3 was held at GCWW. CCS was the primary alerting component of FSE 3.
IVR and work order alarms initiated the drill. A steady stream of customer complaints continued
throughout the morning. Figure 7-7 shows the timeline progression of the key activities completed
during the CCS alert investigation for FSE 3.
: Initial IVR Aler
£rstCa" . Investigation
Received at Completec
Call Center v
00:20
WQ&TC
Notified (
Complai
f i
hemist 00:30
3f Work C
its Alert R
- i
t
WQ&T Ch
Reviewed
3rder °
eceived
r 1
50:35
=mist 00:44
Work Possible
rtjers Contamination
Determination
Reachec
00:47
Distnibution
Dispatch
Supervisor
Contactec
i
00:00
00:47
Figure 7-7. Timeline Progression of the CCS Alert Investigation during FSE 3
In general, the time for the investigators to conclude that contamination was Possible decreased for
successive drills. However, different drill players and variations in the drill scenarios limit the ability to
draw conclusions as to whether investigator proficiency increased as more CCS drills were performed.
7.3.5
Summary
Time for data transmission was just a fraction of a second across all data sources (IVR data, work
requests and work orders). However, while IVR data is available as soon as a call is received, processing
generally takes between two and four minutes to create a work request and approximately 15 minutes for
staff to determine whether a work request is warranted. Event detection for all data streams takes less
than a second, and is negligible when considering the entire CCS timeline. The simulation data indicated
CCS detection was very timely once customers had been exposed to contamination. In 97% of the CCS
simulated scenarios CCS was the first component to detect.
7.4 Design Objective: Operational Reliability
7.4.1 Rate of Work Requests Producing Work Orders
Definition: This metric is unique to the work request data stream and measures how often a work request
created by the CSR meets the qualifications to generate a work order; that is, the complaint warrants
water sampling as determined by the WQ&T Chemist. Analysis of this metric helps determine the
relationship between work requests and work orders. A high percentage of work requests that generate
work orders validates the work request data, verifying that the customer complaint was indeed a water
quality concern that required follow up action by a WQ&T Chemist. A high percentage of work requests
that generate work orders may also suggest redundancy in the data streams.
Analysis Methodology: The percent of work requests that generate work orders, by reporting period and
in total is calculated. The work order field was not added to the work request data stream until September
2008. Thus, data is only available from the August 2008 reporting period onward.
Results: Overall, work requests resulted in the generation of a work order 88% of the time. As seen in
Figure 7-8, the percent conversion to work orders remained fairly consistent through the evaluation
period. Even during the reporting periods with the fewest work requests converted to work orders, 75%
of all work requests still generated work orders. Four reporting periods saw all of their work requests
converted to work orders. This demonstrates that in most instances, the volume of work orders will
closely mimic the volume of work requests.
66
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
100%
de
Wo
Requests Converted
Wo
50%
40%
20%
0%
% By Reporting Period
Overall Average
^ ^ ^ ^ ^ «^
Start Date of Monthly Reporting Period
Figure 7-8. Percent of Work Requests Converted to Work Orders
7.4.2 Rate of Work Orders Preceded by Work Requests
Definition: This measures how often a work order created by the WQ&T Chemist was produced from a
work request, or how often sampling requests are a result of work requests. While related, this metric is
unique from the rate of work requests producing work orders metric discussed in Section 7.4.1, as work
orders can also be produced outside of the work request business flow.
Analysis Methodology: The percent of work orders generated from work requests, by reporting period
and in total, is calculated from empirical data.
Results: The percent of work orders preceded by work requests can be seen in Figure 7-9. Overall, 78%
of work orders were generated from work requests, although over half (54%) of reporting periods saw
80% or more of work orders preceded by work requests. This is because of an unusually low percentage
of work orders preceded by work alerts in the 4/16/10 reporting period.
67
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
_ 100%
~ 90%
I 80%
tr
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
implement the system and the cost to operate and maintain the system. The benefits derived from the
system are defined in terms of primary and dual-use benefits. The primary benefit of a CWS is the
potential reduction in consequences in the event of a contamination incident; however, such a benefit may
be rarely, if ever, realized. Thus, dual-use benefits that provide value to routine utility operations are an
important driver for sustainability of the system. Ultimately, the sustainability of the system can be
demonstrated through utility and partner acceptance of protocols and procedures necessary to operate and
maintain the CWS. The three metrics that will be evaluated to assess how well the Cincinnati CWS met
the design objective of sustainability are: Costs, Benefits and Acceptability. The following subsections
define each metric, describe how it was evaluated, and present the results.
7.5.1 Costs
Definition: Costs are evaluated over the 20 year lifecycle of the Cincinnati CWS, and comprise costs
incurred to design, deploy, operate and maintain the CCS component since its inception.
Analysis Methodology: Parameters used to quantify the implementation cost of the CCS component
were extracted from the Water Security Initiative: Cincinnati Pilot Post-Implementation System Status
(USEPA, 2008). The cost of modifications to the CCS component made after the completion of
implementation activities were tracked as they were incurred. O&M costs were tracked on a monthly
basis over the duration of the evaluation period. Renewal and replacement costs, along with the salvage
value at the end of the Cincinnati CWS lifecycle were estimated using vendor supplied data, field
experience and expert judgment. Note that all costs reported in this section are rounded to the nearest
dollar. Section 3.5 provides additional details regarding the methodology used to estimate each of these
cost elements.
Results: The methodology described in Section 3.5 was applied to determine the value of the major
cost elements used to calculate the total lifecycle cost of the CCS component, which are presented in
Table 7-4. It is important to note that the Cincinnati CWS was a research effort, and as such
incurred higher costs than would be expected for a typical large utility installation. A similar CCS
component implementation at another utility should be less expensive as it could benefit from lessons
learned and would not incur research-related costs.
Table 7-4. Cost Elements used in the Calculation of Lifecycle Cost
Parameter
Implementation Costs
Annual O&M Costs
Renewal and Replacement Costs1
Salvage Value
Value
$1,037,591
$8,086
$231,419
-
Calculated using equipment presented in Table 7-7.
Table 7-5 below presents the implementation cost for each CCS design element, with labor costs
presented separately from the cost of equipment, supplies, and purchased services.
69
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Table 7-5. Implementation Cost
Design Element
Project Management2
Comprehensive
Complaint Collection
Electronic Data
Management
Automated and
Integrated Data Analysis
Procedures
Shared IT Systems
TOTAL:
Labor
$102,7491
$202,127
$70,675
$182,689
$91,266
$283,923
$933,429
Equipment, Supplies,
Purchased Services
-
-
$5,567
-
-
$77,140
$82,707
Component
Modifications
-
-
-
$21,455
-
-
$21,455
Total
Implementation
Costs
$102,749
$202,127
$76,242
$204,144
$91,266
$361,063
$1,037,591
1 Project management costs incurred during implementation were distributed evenly among the CWS components.
The first design element, project management, includes overhead activities necessary to design and
implement the component. The comprehensive complaint collection design element includes the cost of
developing a procedure for transferring calls from the City Call Center to the GCWW Call Center. A
process for collecting and evaluating all water quality complaints was implemented. The third design
element, electronic data management, includes the cost of evaluating GCWW Call Center IVR
information and the GCWW Work Order system data, and implementing a procedure for capturing this
data. The fourth design element, automated and integrated data analysis, includes the cost of developing
automated event detection algorithms for analyzing data from the GCWW Call Center IVR and the
GCWW Work Order system for identifying unusual conditions. An automated email notification system
was also developed. An existing GIS map was modified to show the locations of the alerts. The fifth
design element, procedures, includes the cost of developing procedures that guide the routine operation of
the component and alert investigations, along with training on those procedures.
The final design element, shared IT systems, includes the cost of developing a system which utilizes the
WS Application and Database servers to monitor the data needed for alert investigations and credibility
determination. As this system is utilized by CCS and PHS, the associated cost was split evenly between
these two components.
Overall, the shared IT systems design element had the highest implementation cost (35%). The total
implementation costs for comprehensive complaint collection and automated and integrated data analysis
were lower at 19% and 20%, respectively. Implementation costs for electronic data management and for
development of the procedures for routine operation and training on those procedures were significantly
lower at 7% and 9%, respectively. Project management costs distributed across all components accounted
for 10% of the overall cost. There were no significant equipment or consumable costs associated with
this component due to existing software and hardware systems employed by GCWW and the City of
Cincinnati that were leveraged for data collection, processing and analysis.
The component modification costs represent the labor, equipment, supplies and purchased services
associated with enhancements to the CCS component after completion of major implementation activities
in December 2007. The modification costs include additional expenses incurred to enhance the event
detection system based on GCWW feedback and the implementation of the IVR 5-4 sub-menu.
70
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
The annual labor hours and costs of operating and maintaining the CCS component, broken out by design
element, are shown in Table 7-6.
Table 7-6. Annual O&M Costs
Design Element1
Event Detection
System
Procedures
TOTAL:
Total Labor
(hours/year)
36
125
161
Total Labor
Cost
($/year)
$1,259
$6,827
$8,086
Supplies and
Purchased Services
($/year)
-
-
-
Total O&M Cost
($/year)
$1,259
$6,827
$8,086
1 Overarching project management costs were only incurred during implementation of the CCS component and are
not applicable for annual O&M costs.
O&M for the event detection system requires a low level of monitoring and troubleshooting of the IT
infrastructure. Most of the O&M labor hours reported under procedures were spent on the routine
investigation of CCS alerts. Development and execution of the drills and exercises contributed the
most to O&M procedures labor. Drills often involved over 100 total person-hours to development
and conduct. However, in lessons-learned forums, GCWW identified drills and exercises as one of
the most beneficial elements of a CCS component. Drills improved CCS operation in several ways
including clarifying investigation end-points (CCS Drill 1), identifying issues with remote access to
CCS data necessary for investigations (CCS Drill 2), and identifying the need for additional training
of CSRs to ensure all necessary information for an investigation is captured (CCS Drill 3).
Two of the major cost elements presented in Table 7-4, the renewal and replacement costs and salvage
value, were based on the cost associated with one major piece of equipment installed for the CCS
component (the WS Database Server/WS Application Server). For CCS, there is no salvage value for the
equipment deployed over a 20-year period. The useful life of this item was estimated at 5 years based on
manufacturer-provided data and input from subject matter experts. It was assumed that this item would
need to be replaced three times during the 20-year lifecycle of the CWS. The cost of this item is
presented in Table 7-7.
Table 7-7. Equipment Costs
Equipment Item
WS Database Server/WS Application Server1
Useful Life
(years)
5
Unit Capital
Costs
$77,140
Quantity
(# of Units)
1
TOTAL:
Total Cost
$77,140
$77,140
1 Equipment utilized by CCS and PHS; costs evenly split between two components.
To calculate the total lifecycle cost of the CCS component, all costs and monetized benefits were adjusted
to 2007 dollars using the change in the Consumer Price Index (CPI) between 2007 and the year that the
cost or benefit was realized. Subsequently, the implementation costs, renewal and replacement costs, and
annual O&M costs were combined to determine the total lifecycle cost:
CCS Total Lifecvcle Cost: $1,353,331
Note that in this calculation the implementation costs were treated as a one-time balance adjustment, the
O&M costs recurred annually, and the renewal and replacement cost of one equipment item was incurred
at regular intervals based on the useful life of that item.
71
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
7.5.2 Benefits
Definition: The benefits of CWS deployment can be considered in two broad categories: primary and
dual-use. Primary benefits relate to the application of the CWS to detect contamination incidents, and can
be quantified in terms of a reduction in consequences. Primary benefits are evaluated at the system-level
and are thus discussed in the report titled Water Security Initiative: Evaluation of the Cincinnati
Contamination Warning System Pilot (USEPA, 2014c). Dual-use benefits are derived through
application of the CWS to any purpose other than detection of intentional and unintentional drinking
water contamination incidents. Dual-use benefits realized by the CCS component are presented in this
section.
Analysis Methodology: Information collected from forums, such as data review meetings, lessons
learned workshops and interviews were used to identify dual-use applications of the CCS component of
the CWS.
Results: Operation of the CCS component of the CWS resulted in benefits beyond the detection of
intentional and unintentional contamination incidents. These key dual-use benefits and examples
identified by the utility include:
1. Improved awareness of distribution interruptions
Distribution issues such as main breaks and flushing events can be quickly identified through
monitoring of the rusty water complaints
2. Improved customer service
The impact of operational changes on customer's perception of water quality can be monitored by
the CCS component. A CCS alert resulting from an operational change can signal to the utility
that customers are noticing the changes. Contact center managers can instruct the CSRs that the
situation is routine and customers do not need to be concerned.
Case Study 1: Rusty Water Complaints
Taste, odor, or discoloration complaints can be indicators of water contamination. These complaint types
are the monitoring target of a CCS component. However, other customer complaint types may provide
early warnings of other problems within the distribution system. "Rusty" or "dirty" water complaints may
be indicative of turbidity issues in the distribution system, which are often the result of utility activity
such as flushing. Utilities may consider monitoring for these more benign complaints as a dual-use
benefit to enhance knowledge of water quality in the distribution system.
GCWW has a category of work requests labeled "Rusty Water" to track turbidity complaints.
Retrospective analysis using the AET generated dates and times of alerts that would have been generated
if EDAs were analyzing the data. A positive predictive value for rusty water work requests, or precision
rate reflecting the true probability that rusty water events are occurring can be calculated for alerts
generated under the operating thresholds of water quality work requests. Here, the positive predictive
value is calculated as the number of valid alerts divided by the number of valid alerts and false positives.
A high positive predictive value indicates that alerts from the data stream are often generated by real
changes in the condition of the water and, thus, are a reliable marker of potential contamination. Since
the complaint descriptions are by definition similar, if any two complaints contributing to an alert are
from the same pressure zone then the alert can be determined to be the result of a valid alert. However,
other than two cases (7.2.3), there are no investigation results which can verify any system events.
Results: The number of rusty water complaint alerts received for various threshold levels is displayed in
Figure 7-10. The alert pattern over increasing thresholds differs from the other threshold analyses. The
reset one-day scan, weekday algorithm follows the continuous two-day scan algorithm, although at a
72
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
slightly flatter decline with the number of alerts consistent with the water quality work requests data
stream. However, the continuous seven-day scan has a brief uptick between a threshold of two and three,
before gradually declining. The continuous seven-day scan features fewer alerts overall than the water
quality work request alert. This pattern is indicative of the rusty water complaints tendency to cluster
temporally, most likely the result of actual rusty water complaints.
SO
70
One-day Scan, Weekday
Two-day Scan
Seven-day Scan
Threshold
Figure 7-10. Expected Alerts at Various Thresholds for Rusty Water Complaints
Table 7-8 shows the positive predictive values for rusty water alerts for the one-day scan, weekday, two-
day, and seven-day scan algorithms. The positive predictive value is high for rusty water work requests,
with 61% of one-day scan alerts, 78% of two-day scan alerts and 86% of seven-day scan alerts the result
of system events. While the longer length scan algorithms have a higher predictive value, they may not
be as timely as the one-day scan, weekday algorithm. The cluster of complaints may have passed for days
before additional work requests generated an alert. At three work requests, a one-day scan alert is more
likely to be generated by complaints of rusty water. Early warnings from rusty water alerts can allow the
utility to respond better to customer concerns and address the conditions causing the alerts more
promptly.
Table 7-8. Positive Predictive Value of Rusty Water Work Request Alerts
System Events
Invalid Alerts
Total
Positive Predictive Value
One-day Scan,
Weekday
17
11
28
0.61
Two-day Scan
14
4
18
0.78
Seven-day Scan
6
1
7
0.86
73
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Additionally, turbidity problems affect aesthetic properties of the water in the distribution system,
potentially affecting many households. Distribution issues mirror the impact of contamination with a
contaminant having strong aesthetic characteristics. Alerts that would have been generated by monitoring
rusty water work requests had a high predictive value. Monitoring of this data stream can alert utilities to
developing water quality problems within the distribution system. Assuming customers respond to
contamination as they do to turbidity issues, the rusty water data indicates CCS may be an effective
mechanism to detect water contamination with contaminants that have aesthetic properties.
7.5.3 Summary
The time required to operate and maintain the CCS component varied by month, due mostly to drills and
implementation of further enhancements to the system. Although drills and exercises required
considerable effort, they were identified as highly beneficial to understanding the role and function of
CCS within a CWS. Feedback from CCS personnel indicates that the typical labor hours required for
alert investigations (~1 hour per month) is acceptable and should be sustainable in the long-term. The
majority of costs associated with the CCS during the evaluation period are related to staff effort for
system operation.
A dual use benefit of CCS is utilizing rusty water complaints to detect potentially deteriorating water
quality in the distribution system. Analysis of these complaints yielded a high positive predictive value,
indicating alerts generated by the data stream are likely the result of changing distribution conditions,
caused by activities such as hydrant flushing.
74
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 8.0: Summary & Conclusions
The evaluation of the CCS component of the CWS involved analysis of empirical data, data from drills
and exercises, results from the simulation study, qualitative observations gleaned from participants during
forums, and the benefit-cost analysis. An overall results summary for each design objective is presented
in this section.
8.1 Design Objective: Spatial Coverage
The spatial extent of invalid alerts for the work request and work order data streams is large, due to the
absence of clustering around a system event. As the number of complaints necessary to generate an alert
increases, the spatial extent of invalid alerts also expectedly increases. Almost the entire population has
access to a telephone, giving CCS broad spatial coverage. These metrics are presented in Table 8-1.
Table 8-1. Evaluation of Spatial Coverage Metrics
Metric
Average Spatial Extent
of Invalid Alerts
Population Coverage
CCS Subcomponent
IVR
N/A1
Work Requests
113 square miles
Work Orders
118 square miles
96.3%
The IVR data did not contain spatial references and there is no analysis over the 356 square mile service area.
8.2 Design Objective: Contaminant Coverage
The CCS component monitors customer complaint calls that could potentially be associated with certain
contaminants. These contaminants would exhibit a noticeable change in the taste, odor or appearance of
drinking water if present in high enough amounts. CCS detected 100% of the theoretically detectable
simulated contamination scenarios, using 6 of 17 contaminants modeled as part of the simulation study.
The simulation scenarios were selected to model water contamination incidents which would pose a
significant risk to public health and infrastructure. At concentrations manifesting these risks, CCS will
most likely detect the contamination. IVR alerts preceded the work order alerts in all but five of the 564
scenarios.
The overall average of the number of exposures before the first alert is 195 with a median of 81
exposures. Over 92% of the scenarios have fewer than 500 exposures prior to the first CCS alert. There
is a median of 81 exposures before the first IVR alert and a median of 171 exposures before the first work
order alert. The difference between the number of exposures is a product of the time delay between a
customer calling (entering the IVR data stream) and creation of a work order. While the work order data
stream is timely, the delay does result in potentially more customers exposed to contamination.
The scenarios with a greater number of exposures represent injections of large volumes of a contaminant
in population dense areas. The simulated contaminant was able to reach a large number of people before
the customers first begin consuming the water. These results illustrate that while CCS often provides
early detection, it could still be possible to expose a significant number of customers to a dose of
contaminant before an alert is issued resulting in significant risk to the public. These metrics are
presented in Table 8-2.
75
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Table 8-2. Evaluation of Contaminant Coverage Metrics
Metric
Contamination Scenario
Coverage
Contaminant Detection
Threshold (Median)
CCS Subcomponent
IVR
99.65%
81 Exposures
Work Orders
100%
171 Exposures
8.3
Design Objective: Alert Occurrence
During the evaluation period, alert occurrence consisted primarily of invalid alerts. Invalid and valid alert
rates were characterized using empirical data gathered during the real-time monitoring phase. Invalid
alerts occurred frequently at the beginning of the evaluation period due to intentionally low threshold
levels which provided opportunities to rehearse alert investigation procedures. Following threshold
adjustment, invalid alerts occurred at a sustainable level of five to seven alerts per month or fewer.
Feedback forum data indicated that the number of invalid alerts was acceptable to GCWW, and did not
hinder regular operations. Two valid alerts were observed over the evaluation period, which was just
0.38% of the total number of valid and invalid alerts. The CCS component was able to detect valid alerts
in the form of various system events including high chlorine levels and distribution system maintenance
activities. These results are seen in Table 8-3.
Table 8-3. Evaluation of Alert Occurrence Metrics
Metric
Average Invalid Alerts per
Reporting Period
CCS Subcomponent
IVR
6.9
Work Requests
5.4
Work Orders
5.5
8.4
Design Objective: Timeliness of Detection
The processing time of the CCS algorithms is negligible compared to other factors involved in the
complaint management process. Work requests take approximately four minutes to enter, while the work
orders can take up to 15 minutes to enter. Since IVR alerts do not have location and problem descriptions
for the underlying complaints, investigators must ask for this information directly from the CSRs. This
interview process time prolongs the IVR investigation. Because of modifications to the alert
notifications, work request and work order investigations are often completed in less than five minutes,
while the average IVR investigation time is ten minutes. These metric results are shown in Table 8-4.
The simulation study was used to quantify the time for initial detection of valid alerts. Of the 564
theoretically detectable scenarios, CCS was the first component to detect the contamination in 97% of the
simulations. There is an expected difference for time for initial detection for all theoretically detectable
scenarios since contamination incidents which start during the low demand time (midnight) do not have a
chance to generate a CCS alert until after the first simulated customers ingest the water in the morning.
The median time for initial detection for low demand scenarios was 400 minutes; high demand scenarios
were detected with a median of 27.5 minutes. The interquartile ranges were very narrow and spread
around the first consumption time, whether they are high or low demand scenarios. This indicates that
76
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
once a contaminant reaches customers who are consuming the water, enough customers will call the
utility in a short time to quickly surpass the EDA thresholds.
Table 8-4. Evaluation of Timeliness Metrics
Metric
Median Time for Alert
Recognition
Median Time to
Investigate Invalid Alerts
Average Time for Data
Transmission
Average Time for Event
Detection
CCS Subcomponent
IVR
10 Minutes
10 Minutes
226ms
29 ms
Work Requests
Not Applicable*
5 Minutes
52 ms
15 ms
Work Orders
14 Minutes
6 Minutes
276ms
16 ms
Investigators were not expected to respond to alerts in real-time.
8.5
Design Objective: Operational Reliability
The CCS component exhibited excellent operational reliability and was available during 99.6% of the
evaluation period. There was one significant event in May 2008, where the EDAs were taken offline for
67 hours due to a malfunction. This event contributed to the majority of the downtime for the system.
The limited amount of downtime experienced by the IVR component was due to isolated Web service
failures. The work order and work request data streams downtime was the result of a few isolated power
outages. Availability is shown in Table 8-5.
Table 8-5. Evaluation of Operational Reliability Metrics
Metric
Availability in terms of
Downtime Events
(Total Hours)
Percent Available
CCS Subcomponent
IVR
579 Hours
96.9%
Work Requests
57 Hours
99.8%
Work Orders
57 Hours
99.8%
8.6
Design Objective: Sustainability
Costs are estimated over the 20 year life-cycle of the system to provide an estimate of the total cost of
ownership and include the implementation costs, enhancement costs, operation and maintenance costs,
renewal and replacement costs and the salvage value. The benefits derived from the system are defined in
terms of primary and dual-use benefits. Metrics that were evaluated under this design objective include:
costs, benefits and acceptance. The costs used in the calculation of lifecycle costs for the CCS component
are presented in Table 8-6. These costs were tracked as empirical data during the design and
implementation phase of project design, and were analyzed through a benefit-cost analysis of the
Cincinnati pilot.
77
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Table 8-6. Cost Elements used in the Calculation of Lifecycle Cost
Parameter
Implementation Costs
Annual O&M Costs
Renewal and Replacement Costs1
Salvage Value1
Value
$1,037,591
$8,086
$231,419
-
1 Calculated using major pieces of equipment.
The implementation costs, renewal and replacement costs, and annual O&M costs were combined to
determine the total lifecycle cost:
CCS Total Lifecvcle Cost: $1,353,331
Dual-use benefits and acceptance were evaluated through documentation of qualitative data during drills
and exercises, and during forums with the utility including lessons learned workshops. The use of rusty
water complaints captured through customer calls to identify potential distribution system issues
demonstrated a dual-use benefit. Acceptance was demonstrated through 100% utility participation in
drills and exercises which required substantially more effort than routine investigations, but was
beneficial to GCWW as reported by personnel who indicated that they were able to better understand
standing operating procedures through response to simulated water contamination incidents.
Furthermore, acceptance was evidenced by a high rate of alert investigations completed by utility
personnel during the evaluation period (87% alerts investigated or higher for all data streams). While not
implemented at GCWW, analyses of rusty water complaint data indicated that extension of the rusty
water complaint category to the CCS design is viable, and would likely assist the utility in more quickly
identifying aesthetic water issues.
Most business-hours FVR alerts were investigated during the evaluation period, in particular after a
permanent CSR supervisor was appointed. Alerts that occurred after-hours (nights and weekends) were
less likely to be investigated. While the work request data stream was dropped in December 2008, prior
to this time work request alerts were investigated 88% of the time. Work order alerts were investigated
87% of the time during the period of real-time monitoring.
78
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 9.0: References
U.S. Environmental Protection Agency. 2005. WaterSentinel System Architecture, EPA 817-D-05-003.
U.S. Environmental Protection Agency. 2008. Water Security Initiative: Cincinnati Pilot Post-
Implementation System Status. EPA 817-R-08-004.
U. S. Environmental Protection Agency. 2011. Alarm Estimation Tool.
http://water.epa.gov/infrastructure/watersecurity/techtools/upload/alarmestimationtool.xls
U.S. Environmental Protection Agency. 2014a. Water Security Initiative: Comprehensive Evaluation of
the Cincinnati Contamination Warning System Pilot EPA 817-R-14-001.
U.S. Environmental Protection Agency. 2014b. Water Security Initiative: Evaluation of the
Consequence Management Component of the Cincinnati Contamination Warning System Pilot. EPA
817-R-14-001F.
U.S. Environmental Protection Agency. 2014c. Water Security Initiative: Evaluation of the Cincinnati
Contamination Warning System Pilot. EPA 817-R-14-001A.
79
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 10.0: Abbreviations
The list below includes acronyms approved for use in the CCS component evaluation. Acronyms are
defined at first use in the document.
AET Alarm Estimation Tool
CCS Customer Complaint Surveillance
CIS Customer Information System
CM Consequence Management
CMP Consequence Management Plan
CPI Consumer Price Index
CSR Customer Service Representative
CUSUM Cumulative Sum
CWS Contamination Warning System
DWC Distribution Work Creation
EDA Event Detection Algorithm
EMPAC Enterprise Maintenance Planning and Control
EPA/USEPA United States Environmental Protection Agency
ESM Enhanced Security Monitoring
FSE Full Scale Exercise
GCWW Greater Cincinnati Water Works
GIS Geographic Information System
IVR Interactive Voice Response
O&M Operation and Maintenance
PHS Public Health Surveillance
S&A Sampling and Analysis
SCADA Supervisory Control and Data Acquisition
SOP Standard Operating Procedure
WQ&T Water Quality and Treatment
WQM Water Quality Monitoring
WS Water Security
WSI Water Security Initiative
WUERM Water Utility Emergency Response Manager
80
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Section 11.0: Glossary
Alert. Information from a monitoring and surveillance component indicating an anomaly in the system,
which warrants further investigation to determine if the alert is valid.
Alert Investigation. A systematic process, documented in a standard operating procedure, for
determining whether or not an alert is valid and identifying the cause of the alert. If an alert cause cannot
be identified, contamination is Possible.
Anomaly. Deviations from an established baseline. For example, a water quality anomaly is a deviation
from typical water quality patterns observed over an extended period.
Baseline. Normal conditions that result from typical system operation. The baseline includes predictable
fluctuations in measured parameters that result from known changes to the system. For example, a water
quality baseline includes the effects of draining and filling tanks, pump operation, and seasonal changes
in water demand, all of which may alter water quality in a somewhat predictable fashion.
Benefit. An outcome associated with the implementation and operation of a contamination warning
system that promotes the welfare of the utility and the community it serves. Benefits are classified as
either primary or dual-use.
Benefit-cost analysis. An evaluation of the benefits and costs of a project or program, such as a
contamination warning system, to assess whether the investment is justifiable considering both financial
and qualitative factors.
Biotoxins. Toxic chemicals derived from biological materials that pose an acute risk to public health at
relatively low concentrations.
Bulk volume (of contaminant). The total volume of a contaminant solution that is injected into the
distribution system during a contamination scenario.
Component response procedures. Documentation of roles and responsibilities, process flows and
procedural activities for a specified component of the contamination warning system, including the
investigation of alerts from the component. Standard operating procedures for each monitoring and
surveillance component are integrated into an operational strategy for the contamination warning system.
Consequence management. Actions taken to plan for and respond to possible contamination incidents.
This includes the threat level determination process, which uses information from all monitoring and
surveillance components as well as sampling and analysis to determine if contamination is Credible or
Confirmed. Response actions, including operational changes, public notification, and public health
response, are implemented to minimize public health and economic impacts, and ultimately return the
utility to normal operations.
Consequence management plan. Documentation that provides a decision-making framework to guide
investigative and response activities implemented in response to a possible contamination incident.
Contamination incident. The introduction of a contaminant in the distribution system with the potential
to cause harm to the utility or the community served by the utility. A contamination incident may be
intentional or accidental.
81
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Contamination scenario. Within the context of the simulation study, parameters that define a specific
contamination incident, including: injection location, injection rate, injection duration, time the injection
is initiated and the contaminant that is injected.
Contamination warning system. An integrated system of monitoring and surveillance components
designed to detect contamination in a drinking water distribution system. The system relies on integration
of information from these monitoring and surveillance activities along with timely investigative and
response actions during consequence management to minimize the consequences of a contamination
incident.
Costs, implementation. Installed cost of equipment, IT components and subsystems necessary to deploy
an operational system. Implementation costs include labor and other expenditures (equipment, supplies
and purchased services).
Cost, life cycle. The total cost of a system, component, or equipment over its useful or practical life.
Life cycle cost includes the cost of implementation, operation & maintenance, and renewal &
replacement.
Costs, operation & maintenance. Expenses incurred to sustain operation of a system at an acceptable
level of performance. Operational and maintenance costs are reported on an annual basis, and include
labor and other expenditures (supplies and purchased services).
Costs, renewal & replacement. Costs associated with refurbishing or replacing major pieces of
equipment (e.g., water quality sensors, laboratory instruments, IT hardware, etc.) that reach the end of
their useful life before the end of the contamination warning system lifecycle.
Coverage, contaminant. Specific contaminants that can potentially be detected by each monitoring and
surveillance component, including sampling & analysis, of a contamination warning system.
Coverage, spatial. The areas within the distribution system that are monitored by, or protected by, each
monitoring and surveillance component of a contamination warning system.
Credible. In the context of the threat level determination process, a water contamination threat is
characterized as Credible if information collected during the investigation of Possible contamination
corroborates information from the validated contamination warning system alert.
Data completeness. The amount of data that can be used to support system or component operations,
expressed as a percentage of all data generated by the system or component. Data may be lost due to QC
failures, data transmission errors and faulty equipment among other causes.
Distribution system model. A mathematical representation of a drinking water distribution system,
including pipes, junctions, valves, pumps, tanks, reservoirs, etc. The model characterizes flow and
pressure of water through the system. Distribution system models may include a water quality model that
can predict the fate and transport of a material throughout the distribution system.
Dual-use benefit. A positive application of a piece of equipment, procedure or capability that was
deployed as part of the contamination warning system, in the normal operations of the utility.
Ensemble. The comprehensive set of contamination scenarios evaluated during the simulation study.
82
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Event detection system. A system designed specifically to detect anomalies from the various monitoring
and surveillance components of a contamination warning system. An event detection system may take a
variety of forms, ranging from a complex set of computer algorithms to a simple set of heuristics that are
manually implemented.
Evaluation period. The period from January 16, 2008 to June 15, 2010 when data was actively collected
for the evaluation of the Cincinnati contamination warning system pilot.
Exposure. In the simulation model, any person who ingests or detects contaminated water.
Hydraulic connectivity. Points or areas within a distribution system that are on a common flow path.
Invalid alert. An alert from a monitoring and surveillance component that is not due to an anomaly and
is not associated with an incident or condition of interest to the utility.
Job function. A description of the duties and responsibilities of a specific job within an organization.
Metric. A standard or statistic for measuring or quantifying an attribute of the contamination warning
system or its components.
Model. A mathematical representation of a physical system.
Module. A sub-component of a model that typically represents a specific function of the real-world
system being modeled.
Monitoring & surveillance component. Element of a contamination warning system used to detect
unusual water quality conditions, potentially including contamination incidents. The four monitoring &
surveillance components of a contamination warning system include: 1) online water quality monitoring,
2) enhanced security monitoring, 3) customer complaint surveillance and 4) public health surveillance.
Node. A mathematical representation of a junction between two or more distribution system pipes, or a
terminal point in a pipe in a water distribution system model. Water may be withdrawn from the system
at nodes, representing a portion of the system demand.
Nuisance chemicals. Chemical contaminants with a relatively low toxicity, which thus generally do not
pose an immediate threat to public health. However, contamination with these chemicals can make the
drinking water supply unusable.
Operational strategy. Documentation that integrates the standard operating procedures that guide
routine operation of the monitoring and surveillance components of a drinking water contamination
warning system. The operational strategy establishes specific roles and responsibilities for the component
and procedures for investigating alerts.
Optimization phase. Period in the contamination warning system deployment timeline between the
completion of system installation and real-time monitoring. During this phase the system is operational,
but not expected to produce actionable alerts. Instead, this phase provides an opportunity to learn the
system and optimize performance (e.g., fix or replace malfunctioning equipment, eliminate software bugs,
test procedures and reduce occurrence of invalid alerts).
Pathogens. Microorganisms that cause infections and subsequent illness and mortality in the exposed
population.
83
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Possible. In the context of the threat level determination process, a water contamination threat is
characterized as Possible if the cause of a validated contamination warning system alert is unknown.
Precision. The degree to which a set of measurements obtained under similar conditions are the same.
Precision is usually expressed as standard deviation, variance, or range, in either absolute or relative
terms.
Primary benefits. Benefits that are derived from the reduction in consequences associated with a
contamination incident due to deployment of a contamination warning system.
Priority contaminant. A contaminant that has been identified by the EPA for monitoring under the
Water Security Initiative. Priority contaminants may be initially detected through one of the monitoring
and surveillance components and confirmed through laboratory analysis of samples collected during the
investigation of a Possible contamination incident.
Real-time monitoring phase. Period in the contamination warning system deployment timeline
following the optimization phase. During this phase, the system is fully operational and is producing
actionable alerts. Utility staff and partners now respond to alerts in real-time and in full accordance with
standard operating procedures documented in the operational strategy. Optimization of the system still
occurs as part of a continuous improvement process, however the system is no longer considered to be
developmental.
Routine operation. The day-to-day monitoring and surveillance activities of the contamination warning
system that are guided by the operational strategy. To the extent possible, routine operation of the
contamination warning system is integrated into the routine operations of the drinking water utility.
Salvage value. Estimated value of assets at the end of the useful life of the system.
Simulation study. A study designed to systematically characterize the detection capabilities of the
Cincinnati drinking water contamination warning system. In this study, a computer model of the
contamination warning system is challenged with an ensemble of 2,023 simulated contamination
scenarios. The output from these simulations provides estimates of the consequences resulting from each
contamination scenario, including fatalities, illnesses, and extent of distribution system contamination.
Consequences are estimated under two cases, with and without the contamination warning system in
operation. The difference provides an estimate of the reduction in consequences.
Site characterization. The process of collecting information from an investigation site to support the
investigation of a contamination incident during consequence management.
Time for initial alert validation. A portion of the incident timeline that begins with the recognition of
an alert and ends with a determination regarding whether or not contamination is Possible.
Timestep. In the Cincinnati contamination warning system model, a set interval of time (i.e., every 15
minutes) at which the computational platform performs calculations, reads inputs, or generates outputs.
Toxic chemicals. Highly toxic chemicals that pose an acute risk to public health at relatively low
concentrations.
Valid alert. Alerts due to water contamination, system events (i.e., work in the distribution system for
CCS or WQM) or public health incidents (for PHS)
84
-------
Water Security Initiative: Evaluation of the Customer Complaint Surveillance Component
of the Cincinnati Contamination Warning System Pilot
Water Utility Emergency Response Manager. A role within the Cincinnati contamination warning
system filled by a mid-level manager from the drinking water utility. Responsibilities of this position
include: receiving notification of validated alerts, verifying that a valid alert indicates possible
contamination, coordinating the threat level determination process, integrating information across the
different monitoring and surveillance components, and activating the consequence management plan. In
the early stages of responding to possible contamination, the Water Utility Emergency Response Manager
may serve as Incident Commander.
85
------- |