Environmental Protection
Agency
-------
Office of Water (MC 140)
EPA817-R-08-002
September 2008
www.epa.gov/safewater
-------
Water Security Initiative: Operational Strategy Guidance
Foreword
The Water Security initiative is a U.S. Environmental Protection Agency (EPA) program that addresses
the risk of intentional contamination of drinking water distribution systems. Initiated in response to
Homeland Security Presidential Directive 9, the overall goal is to design and deploy contamination
warning systems for drinking water utilities. EPA is implementing the Water Security initiative in three
phases: (1) development of a conceptual design that achieves timely detection and appropriate response to
drinking water contamination incidents; (2) demonstration and evaluation of the conceptual design in full-
scale pilots at drinking water utilities; and (3) issuance of guidance and conduct outreach to promote
voluntary national adoption of effective and sustainable drinking water contamination warning systems.
Figure F-l summarizes this process.
Phase
Approach
Scope
Design
Specificity
Funding
DESIGN
System Architecture
DEMONSTRATE
Initial Pilot
Additional Pilots
EXPAND
Voluntary National Adoption
"V^TX Applied bV"~X
Apply to single Evaluate multiple Evaluate
Conceotual , N pilot utility i > utilities i > Convert to
' p ' i/ . . — i AS. i — | guidance for
9n /> /I A> N/J a"^utility
A \ N/ / ^\ \ K /
\ 7 Refine |__i \J Refine (__i
\/ and ^ and
enhance enhance
Not
applicable
Low
ll
High-
Applies to pilot utility only
t^l
High-
Appliesto each pilot
EPA Funds
fHf
Medium -
Applies to range of utilities
Utility Funds
Figure F-1. Overview of EPA's Water Security Initiative
A contamination warning system should be a proactive approach to managing threat warnings that uses
advanced monitoring technologies/strategies and enhanced surveillance activities to collect, integrate,
analyze, and communicate information. However, it should not be merely a collection of monitors and
equipment placed throughout a water distribution system to alert of intrusion or contamination, but rather
an exercise in information acquisition and management. Different information streams are captured,
managed, analyzed, and interpreted to recognize potential contamination incidents in time to respond
effectively. While the contamination warning system should be designed by the drinking water utility,
some data sources may be outside of the utility, and in this case, cooperation with partners would likely
be important to the success of a contamination warning system. Figure F-2 illustrates the recommended
components of a contamination warning system, as briefly described below:
• Online water quality monitoring involves monitoring for typical water quality parameters
throughout the distribution system, and comparison with an established base-state to detect
possible contamination incidents.
• Sampling and analysis involves the collection of distribution system samples that are analyzed
for various contaminants and contaminant classes for the purpose of establishing a baseline of
contaminant occurrence (contaminants detected, levels detected, and frequency of detections) and
method performance, as well as for the purpose of investigating suspected contamination
incidents triggered by other monitoring and surveillance components.
September 2008
-------
Water Security Initiative: Operational Strategy Guidance
Enhanced security monitoring includes the
equipment and procedures that detect and respond
to security breaches at distribution system
facilities.
Consumer complaint surveillance enhances and
automates the collection and analysis of consumer
calls reporting unusual water quality concerns and
compares trends against an established base-state
to detect possible contamination incidents.
Public health surveillance involves the analysis
of health-related data sources to identify illness in
the community that may stem from drinking water
contamination.
Online Water
Quality
Monitoring
Figure F-2. Multi-Component Approach
to a Contamination Warning System
Developing a contamination warning system should also include extensive consequence management
planning to develop procedures for investigating and responding to possible contamination incidents
detected through the recommended routine monitoring and surveillance components. Once a possible
contamination incident has been identified, the consequence management plan should define a process for
establishing the credibility of the suspected incident, the response actions that may be taken to minimize
public health and economic consequences, and a strategy to ultimately restore the system to normal
operations.
In the context of the Water Security initiative, the deployment of a contamination warning system should
include the six phases illustrated in Figure F-3. EPA is developing a suite of guidance to assist utilities
through this process, all of which will be available at EPA's Water Security initiative website
(http://cfpub.epa.gov/safewater/watersecurity/initiative.cfm) upon publication.
APPLICATION OF
Implementation
Preliminary
Testing
Operation and
Maintenance
Evaluation and
Refinement
THIS GUIDANCE
Figure F-3. Recommended Stages of Contamination Warning System Deployment
The document that follows, Interim Guidance on Developing an Operational Strategy for Contamination
Warning Systems, was written to assist utilities with the development of recommended standard operating
procedures for day-to-day operations of the monitoring and surveillance components of a contamination
warning system. This interim guidance manual will be revised as needed based on findings of the
demonstration pilots and public comment prior to being issued in final form. The guidance emphasizes
development of an operational strategy in a manner that integrates the monitoring and surveillance
components to provide a timely indication of a possible contamination incident in the distribution system.
Development of an operational strategy would typically begin in the design phase of contamination
warning system deployment, as indicated in Figure F-3. Once the components of the contamination
warning system have been designed and implemented, the preliminary operational strategy developed
during the design phase should be refined through subsequent phases of deployment. The ultimate use of
the operational strategy developed according to this guidance is to guide day-to-day operation of the
contamination warning system during the operation and maintenance phase. A companion document,
Interim Guidance on the Development of a Consequence Management Plan, was written to assist utilities
with the development of plans to guide the utility and partner agencies through the processes of
validating, responding to, and recovering from a contamination incident in the distribution system
(USEPA, 2008a). Together, the operational strategy and the consequence management plan should
comprehensively document the procedures that guide operation of the contamination warning system.
September 2008
-------
Water Security Initiative: Operational Strategy Guidance
Disclaimer
Note to Readers: The U.S. Environmental Protection Agency (EPA) prepared this guidance to help you
enhance the security of your water system. This document does not impose legally binding requirements
on EPA, states, tribes, or the regulated community, and it may or may not apply to a particular situation,
depending on the circumstances. EPA, state decision-makers, and drinking water utilities retain the
discretion to adopt approaches that may differ from this guidance. Any decisions regarding a particular
community water system should be made based on applicable statutes and regulations. Therefore,
interested parties are free to raise questions and objections about the appropriateness of the application of
this guide to a particular situation, and EPA will consider whether the recommendations or interpretations
in the guidance are appropriate in that situation based on the law and regulations. EPA may change this
guidance in the future. To determine whether EPA has revised this guide or to obtain additional copies,
contact the Safe Drinking Water Hotline at 1-800-426-4791 or visit the EPA's Water Security website at
www.epa.gov/watersecurity.
Any mention of trade names, companies, products, or services in this guidance does not constitute an
endorsement by the Environmental Protection Agency of any non-federal entity, its products, or its
services.
Questions concerning this document should be addressed to:
Steve Allgeier
U.S. EPA Water Security Division
26 West Martin Luther King Drive
Mail Code 140
Cincinnati, OH 45268
(513)569-7131
Allgeier.Steve(g),epa.gov
or
Jessica Pulz
U.S. EPA Water Security Division
26 West Martin Luther King Drive
Mail Code 140
Cincinnati, OH 45268
(513)569-7918
Pulz.Jessica@epa.gov
Request for Comments
EPA is soliciting suggestions and recommendations to make this interim guidance manual more complete
and user-friendly. Commenters are encouraged to be as specific as possible and to provide references
where appropriate. Submit suggestions by e-mail to: watersecuritv(giepa.gov and indicate that the
message relates to the "Interim Guidance on Developing an Operational Strategy for Contamination
Warning Systems."
September 2008
-------
Water Security Initiative: Operational Strategy Guidance
Acknowledgements
EPA's Office of Ground Water and Drinking Water would like to recognize the following individuals and
organizations for their assistance and contributions in development of this document:
City of Cincinnati - Greater Cincinnati Water Works
Steve Allen • David Hartman •
Verna Arnette • Jim Holly •
Faye Cossins • Yeongho Lee •
Bill Fromme • Mark Menkhaus
Jeff Pieper
Jeff Swertfeger
Mike Tyree
U.S. Environmental Protection Agency - Office of General Counsel
Leslie Darman
U.S. Environmental Protection Agency - Water Security Division
Steve Allgeier • Mike Henrie • Jessica Pulz
Jeffrey Pencil • Tanya Mottley • Dan Schmelling
David Harvey • Nancy Muzzy • David Travers
Elizabeth Hedrick • Brian Pickard • Katie Umberg
U.S. Environmental Protection Agency - National Homeland Security Research Center
Hiba Ernst • Matthew Magnuson • Jeff Szabo
John Hall • Regan Murray • Cynthia Yund
Robert Janke • Benjamin Packard
Victoria
Blackschleger
John Chandler
Kevin Connell
Mike Denison
Bill Desing
Darcy Gibbons
Adam Haas
Contractor Support
• Adrian Hanley
• Yakir Hasit
• Gary Jacobson
• Reese Johnson
• Dan Joy
• Alan Lai
• Greg Meiners
• Bill Phillips
Curtis Robbins
Doron Shalvi
David Watson
Scott Weinfeld
Nick Winnike
Utility Reviewers
Manouchehr Boozarpour, San Francisco Public Utilities Commission
Don Christie, Los Angeles Department of Water and Power
Ricardo DeLeon, Metropolitan Water District of Southern California
Ron Hunsinger, East Bay Municipal Utility District
Bart Koch, Metropolitan Water District of Southern California
David Lipsky, New York City Department of Environmental Protection
Dan Quintanar, Tucson Water
Steve Rhode, Massachusetts Water Resources Authority
Stanley States, Pittsburgh Water and Sewer Authority
September 2008
IV
-------
Water Security Initiative: Operational Strategy Guidance
Table of Contents
SECTION 1.0: INTRODUCTION 1
1.1 KEY CONCEPTS AND DEFINITIONS 1
1.2 DOCUMENT OVERVIEW 3
SECTION 2.0: CONSTRUCTING THE OPERATIONAL STRATEGY 5
2.1 STEP 1: SYSTEM-WIDE ASSESSMENT OF RESOURCES 6
2.2 STEP 2: COMPONENT-LEVEL ANALYSIS - STANDARD OPERATING PROCEDURES 8
2.3 STEPS: SYSTEM-WIDE INTEGRATION 9
SECTION 3.0: STANDARD OPERATING PROCEDURES 12
3.1 COMPONENT DESCRIPTION 12
3.2 ROLES AND RESPONSIBILITIES 13
3.3 PROCESS FLOW 14
3.4 USER-SPECIFIC CHECKLISTS 15
SECTION 4.0: IMPLEMENTATION AND MAINTENANCE 16
4.1 BASELINE AND PRELIMINARY TESTING 16
4.2 FULL DEPLOYMENT 17
4.3 EVALUATION AND REFINEMENT 18
SECTION 5.0: REFERENCES 20
APPENDIX CASE STUDY: OPERATIONAL STRATEGY FOR THE CINCINNATI CONTAMINATION
WARNING SYSTEM 21
A.1 OVERVIEW AND OBJECTIVES 22
OVERVIEW 22
OBJECTIVES 23
OVERVIEW OF ROLES AND RESPONSIBILITIES 23
OVERVIEW OF TRIGGER INVESTIGATION PROCESS FLOWS 24
A.2: ONLINE WATER QUALITY MONITORING STANDARD OPERATING PROCEDURES 26
COMPONENT DESCRIPTION 26
ROLES AND RESPONSIBILITIES 27
PROCESS FLOW 27
CHECKLISTS 31
A.3: SAMPLING AND ANALYSIS STANDARD OPERATING PROCEDURES 32
COMPONENT DESCRIPTION 32
ROLES AND RESPONSIBILITIES 33
PROCESS FLOW 33
CHECKLISTS 38
A.4: ENHANCED SECURITY MONITORING STANDARD OPERATING PROCEDURES 39
COMPONENT DESCRIPTION 39
ROLES AND RESPONSIBILITIES 40
PROCESS FLOW 40
CHECKLISTS 45
A.5: CONSUMER COMPLAINT SURVEILLANCE STANDARD OPERATING PROCEDURES 46
COMPONENT DESCRIPTION 46
ROLES AND RESPONSIBILITIES 47
PROCESS FLOW 48
CHECKLISTS 54
September 2008
-------
Water Security Initiative: Operational Strategy Guidance
A.6: PUBLIC HEALTH SURVEILLANCE STANDARD OPERATING PROCEDURES 56
COMPONENT DESCRIPTION 56
ROLES AND RESPONSIBILITIES 56
PROCESS FLOWS 57
CHECKLISTS 62
A.7: EXAMPLES OF CONTAMINATION WARNING SYSTEM, TRIGGER INVESTIGATION
CHECKLISTS 63
CHECKLIST A-1: CONTAMINATION WARNING SYSTEM TRIGGER INVESTIGATION 64
CHECKLIST A-2: DISTRIBUTION SYSTEM SITE INVESTIGATION 67
CHECKLIST A-3: DISTRIBUTION SYSTEM OPERATIONS REVIEW 69
CHECKLIST A-4: DISTRIBUTION SYSTEM WORK ORDER REVIEW 70
CHECKLIST A-5: SECURITY INCIDENT INVESTIGATION 71
CHECKLIST A-6: WATER QUALITY CONSUMER COMPLAINT INVESTIGATION 74
CHECKLIST A-7: PUBLIC HEALTH SURVEILLANCE TRIGGER INVESTIGATION 76
September 2008 vi
-------
Water Security Initiative: Operational Strategy Guidance
List of Tables
Table 2-1. Example IT System Inventory 7
Table A-1. Summary of Primary Roles in Routine Contamination Warning System Operations 24
Table A-2. Summary of Process Flows and Trigger Validation Process for Contamination Warning
System Components 25
Table A-3. Summary of the Example Online Water Quality Monitoring Network 26
Table A-4. Roles and Responsibilities for Routine Operation of Online Water Quality Monitoring 27
Table A-5. Example Timeline for Validation of a Water Quality Trigger in the Context of an Operational
Contamination Warning System 31
Table A-6. Example Checklists used During Investigation of a Water Quality Monitoring Trigger 31
Table A-7. Local Laboratory Network and Sampling Frequency for Maintenance Monitoring 32
Table A-8. Field Methods for Safety and Contaminant Screening 32
Table A-9. Roles and Responsibilities for Routine Operation of Sampling and Analysis 33
Table A-10. Example Timeline for Validation of a Sampling and Analysis Trigger in the Context of an
Operational Contamination Warning System 37
Table A-11. Example Checklists used During Investigation of a Sampling and Analysis Trigger 38
Table A-12. Summary of Enhanced Security Monitoring Equipment per Location 39
Table A-13. Roles and Responsibilities for Routine Operation of Enhanced Security Monitoring 40
Table A-14. Example Timeline for Validation of a Security Trigger in the Context of an Operational
Contamination Warning System 44
Table A-15. Example Checklists used During Investigation of an Enhanced Security Trigger 45
Table A-16. Summary of the Algorithms used in the Consumer Complaint Surveillance Event Detection
System 46
Table A-17. Roles and Responsibilities for Routine Operation of Consumer Complaint Surveillance 47
Table A-18. Example Timeline for Validation of a Consumer Complaint Surveillance Trigger in the
Context of an Operational Contamination Warning System 54
Table A-19. Example Checklists used During Investigation of a Consumer Complaint Surveillance Trigger
55
Table A-20. Data Streams, Public Health Partners, and Detection Capabilities 56
Table A-21. Roles and Responsibilities for Routine Operation of Public Health Surveillance 57
Table A-22. Example Timeline for Validation of a Public Health Surveillance Trigger in the Context of an
Operational Contamination Warning System 62
Table A-23. Example Checklists used During Trigger Investigation for the Public Health Surveillance
Component 62
Table A-24: Example Checklists used During Investigation of Contamination Warning System Triggers 63
September 2008 vii
-------
Water Security Initiative: Operational Strategy Guidance
List of Figures
Figure F-1. Overview of EPA's Water Security Initiative i
Figure F-2. Multi-Component Approach to a Contamination Warning System ii
Figure F-3. Stages of Contamination Warning System Deployment ii
Figure 1-1. Contamination Warning System Architecture 1
Figure 1-2. Overarching Structure of EPA's Recommended Operational Strategy 2
Figure 2-1. Operational Strategy Development Process 6
Figure 3-1. Roles within a Generic Utility Organizational Hierarchy 13
Figure 3-2. Generic Process Flow and Initial Trigger Validation Process 15
Figure A-1. Process Flow for Online Water Quality Monitoring: Routine Monitoring and Initial Trigger
Validation 28
Figure A-2. Process Flow for Sampling and Analysis: Routine Monitoring and Initial Trigger Validation..34
Figure A-3. Process Flow for Enhanced Security Monitoring: Routine Monitoring and Initial Trigger
Validation 41
Figure A-4. Process Flow for Consumer Complaint Surveillance: Routine Monitoring 49
Figure A-5. Process Flow for Consumer Complaint Surveillance: Initial Trigger Validation 52
Figure A-6. Process Flow for Public Health Surveillance: Routine Monitoring and Initial Trigger Validation
58
September 2008 viii
-------
Water Security Initiative: Operational Strategy Guidance
List of Acronyms
The list below includes acronyms approved for use in the Operational Strategy Guidance. Acronyms are
defined at first use in the document.
BT Bioterror (Agent)
CSR Customer Service Representative
EPA Environmental Protection Agency
ER Emergency Room
CIS Geographic Information System
GUI Graphical User Interface
IT Information Technology
IVR Interactive Voice Response
LAN Local Area Network
LRN Laboratory Response Network
ORP Oxidation-reduction potential
PCS Polychlorinated Biphenyl
QA Quality Assurance
QC Quality Control
SCADA Supervisory Control and Data Acquisition
SVOC Semi-volatile Organic Compound
TOC Total Organic Carbon
VOC Volatile Organic Compound
WS Water Security (initiative)
WUERM Water Utility Emergency Response Manager
September 2008
IX
-------
Water Security Initiative: Operational Strategy Guidance
Section 1.0: Introduction
This document is part of a series of guidance documents developed to support EPA's Water Security
(WS) initiative (formerly known as WaterSentinel). Initiated in response to Homeland Security
Presidential Directive 9, the overall goal of the Water Security initiative is to design, deploy, and evaluate
contamination warning systems for drinking water utilities. Additional information on the objectives of
the Water Security initiative and contamination warning systems can be found in Water Sentinel System
Architecture (USEPA, 2005). Additional information is also available on the Water Security initiative
website at: http://cfpub.epa.gov/safewater/watersecurity/initiative.cfm.
1.1 Key Concepts and Definitions
Figure 1-1 provides an overview of EPA's recommended contamination warning system architecture. It
illustrates the role of the operational strategy in guiding routine operation of the monitoring and
surveillance components, and the transition to a credibility determination process in the event a
contamination threat is deemed possible. Typically, this aspect of the contamination warning system is
guided by a consequence management plan, which provides a decision-making framework that should be
used to establish credibility, implement response actions, minimize public health and economic impacts,
and ultimately return the system to normal operations. (See Interim Guidance for Developing a
Consequence Management Plan, USEPA, 2008a.)
Credible Determination Confirmed Determination Remediation and Recovery
Figure 1-1. Contamination Warning System Architecture
In the context of this guidance, an operational strategy is the system-wide integration of the standard
operating procedures that guide routine operation of the monitoring and surveillance components of a
drinking water contamination warning system. Generally, the standard operating procedures establish
specific roles and responsibilities, process flows, and procedural activities for each component and the
processes for investigating a trigger and determining whether or not an anomaly is indicative of a possible
contamination threat, as described in the EPA's Response Protocol Toolbox (USEPA, 2003). An
operational strategy may also include checklists that support specific users in the implementation of the
standard operating procedures. Figure 1-2 illustrates the high-level structure of an operational strategy,
which is made up of component-level standard operating procedures that in turn are supported by user-
September 2008
1
-------
Water Security Initiative: Operational Strategy Guidance
specific checklists. The purpose of this document is to assist drinking water utilities in development of
EPA's recommended operational strategy for a contamination warning system based on the high-level
structure shown in this figure.
Integrated Operational Strategy
r
Component-level Standard Operating Procedures
Water Quality
Monitoring
Sampling &
Analysis
Enhanced
Security
Monitoring
Consumer
Complaint
Surveillance
Public
Health
Surveillance
r
User-specific Checklists
Water
Quality
Supervisor
Water Quality
Field
Technician
SCADA
Operator
Distribution
Work
Supervisor
Customer
Service
Representative
Security
Personnel
Public
Health
Agency
Figure 1-2. Recommended Overarching Structure of Operational Strategy
Additional key concepts and definitions used in this guidance include the following:
• Routine Operation. Routine operation refers to the day-to-day monitoring and surveillance
activities that are guided by the operational strategy for the contamination warning system. To
the extent possible, routine operation of the contamination warning system should be integrated
into the routine operations of the drinking water utility.
• Process Flow. A process flow is the central element of a standard operating procedure. It
describes how routine monitoring and surveillance, event detection, and trigger validation lead to
a determination of possible contamination, prior to the implementation of the consequence
management plan. Because each component uses different data sources and generates different
triggers, the detailed process flow for each component is unique. However, all component
process flows should include a common set of process elements.
• Standard Operating Procedure. A standard operating procedure should establish specific roles
and responsibilities, process flows, and procedural activities for a specified component of the
contamination warning system. It should also establish the initial alarm investigation processes
that conclude with the determination whether or not a trigger is indicative of a possible
contamination threat.
• Operational Strategy. The system-wide integration of the standard operating procedures for the
routine operation of monitoring and surveillance components of a drinking water contamination
warning system. In the event a contamination threat is deemed possible, the operational strategy
can facilitate transition to the credibility determination process of a consequence management
plan.
September 2008
-------
Water Security Initiative: Operational Strategy Guidance
• Job Function. A description of the duties and responsibilities of a specific job within an
organization.
• User. In the context of a contamination warning system, a user refers to a specific individual
within the drinking water utility or local partner organization who has a defined role and
responsibility in the operational strategy.
• Base-state. Typical pattern of a parameter, which represents the range of normal conditions
observed in a system and captures known causes of variability (such as seasonal or operational
changes).
• Anomaly. Deviation from an established base-state. For example, a water quality anomaly is a
deviation from typical water quality patterns observed over an extended period (i.e., a base-state).
• Event Detection. The process by which a deviation from established base-state is identified as
an anomaly. The anomaly could be a pattern of unusual water quality readings, a cluster of
unusual consumer complaints, or unusual symptoms picked up by a public health surveillance
program. For most monitoring and surveillance components, event detection utilizes algorithms
to continuously analyze the data stream and filter out perturbations that are part of the established
base-state, and signal only those anomalies that are likely to be possible contamination threats. In
short, the purpose of the event detection algorithms is to reduce the false positive rate without
missing potential contamination incidents.
• Trigger. Information from a monitoring and surveillance component indicating an anomalous or
unusual condition within the system, which warrants further investigation to determine if it is
benign or a possible contamination threat. The nature of a trigger can vary by component and
may take the form of an alarm, alert, threshold excursion, or warning. Event detection algorithms
are the tool by which triggers can be identified for most monitoring and surveillance components.
• Possible Contamination Threat. In the context of the contamination warning system
operational strategy, water contamination should be characterized as possible if the cause of a
contamination warning system trigger cannot be identified and/or determined to be benign.
• Initial Trigger Validation. The process of investigating potential causes of a contamination
warning system trigger to either rule out contamination or determine that contamination is
possible. This process is related to event detection, but the latter is typically automated and
produces the trigger that is investigated during initial trigger validation as guided by a standard
operating procedure.
• Credibility Determination. Investigation of a possible contamination threat to determine
whether or not additional information, including data from other monitoring and surveillance
components, corroborates the information from the validated trigger. If the additional
information corroborates the trigger, contamination should be considered credible.
• Water Utility Emergency Response Manager (WUERM). A utility may refer to this position
by another title, but regardless, this role should generally be filled by a mid-level manager who
can integrate information from multiple monitoring and surveillance components, receive
notification of possible contamination events, coordinate the credibility determination process,
and initiate the consequence management plan. Additionally, the Water Utility Emergency
Response Manager may serve as Incident Commander early on in an investigation.
1.2 Document Overview
This document provides guidance for developing, implementing, and maintaining an operational strategy
for a drinking water contamination warning system based on EPA's recommended approach. It provides
details and background on the content of EPA's recommended operational strategy; a framework or
approach for developing, implementing, and testing the operational strategy; and discusses how to align
routine operations of a contamination warning system with existing utility operations to achieve a
September 2008 3
-------
Water Security Initiative: Operational Strategy Guidance
sustainable system that realizes dual-use applications. Throughout the document, tips and success stories
from the initial contamination warning system pilot in Cincinnati are highlighted to draw attention to
useful points for consideration.
The following sections are included in this document:
• Section 2.0: Constructing the Operational Strategy. This section describes a step-wise
process for developing the operational strategy for contamination warning system deployment.
• Section 3.0: Standard Operating Procedures. This section provides an overview of the
structure and content of standard operating procedures for monitoring and surveillance activities
in a contamination warning system.
• Section 4.0: Implementation and Maintenance. This section describes activities associated
with implementation and maintenance of the operational strategy including training and exercises.
• Section 5.0: References. This section lists references cited throughout the document.
• Appendix: Case Study. The appendix provides a case study in development of a contamination
warning system operational strategy, which was generalized from the operational strategy
developed for the initial pilot in Cincinnati. The case study includes example standard operating
procedures and corresponding checklists for each of the monitoring and surveillance components.
September 2008
-------
Water Security Initiative: Operational Strategy Guidance
Section 2.0: Constructing the Operational Strategy
An operational strategy should integrate the standard operating procedures that guide routine operation of
each component of a contamination warning system, and in the event a contamination threat is deemed
possible, facilitate transition to the consequence management plan. An equally important application of
the operational strategy is to support the development of
system requirements during the design phase of / LESSON LEARNED
contamination warning system deployment. A preliminary
operational strategy developed during the design phase of the
system that describes how it is envisioned to operate once
implemented can help to identify key users and their
requirements for access to information, procedures to guide
system operation, information systems that may be leveraged
to support system development, and requirements for
notifications to key users and decision-makers. This is also
an opportunity to ensure that the overall processes defined by
the component standard operating procedures are compatible
with the utility's organizational structure and current job
functions to the extent possible.
At the initial Water Security initiative
pilot, development of the operational
strategy did not begin until late in the
design phase. This resulted in some
delays as a point was reached in
which further progress could not be
made without first clearly defining
users and their information needs.
Recommendation: Develop a
preliminary operational strategy early
in the design process!
In order to develop the preliminary operational strategy, the recommended steps include the following
three steps as illustrated in Figure 2-1:
1. System-wide assessment of resources
2. Component-specific analysis to develop standard operating procedures
3. System-wide integration of component-specific standard operating procedures into a
comprehensive operational strategy for the contamination warning system
While the standard operating procedures developed during Step 2 should be the central element of the
operational strategy, the system-wide analyses performed at the beginning and end of the development
process can ensure that the system functions as an integrated whole.
September 2008
-------
Water Security Initiative: Operational Strategy Guidance
1. System-wide Assessment
of Resources
k Inventory available A
operational procedures
and IT systems
2. Component-Level
Analysis
Develop a preliminary
standard operating
procedure for each
component
3. System-wide
Integration
Develop
integrated operational strategy
for entire system
Figure 2-1. Operational Strategy Development Process
The operational strategy should be developed with full and active participation of the project management
team, which includes the Water Utility Emergency Response Manager, information technology (IT) staff,
and representatives from each division or organization involved in the design or operation of the system.
Furthermore, it is important that front-line staff be engaged in the component-level analysis and
development of the standard operating procedures to build acceptance of responsibilities for system
operation as well as to accurately portray system operations. For some components, this will also include
working with local partners outside of the utility who have a critical role in operation. Sections 2.1
through 2.3 provide additional detail on each step of the development process.
2.1 Stepl: System-wide Assessment of Resources
The first step in developing a preliminary operational strategy should be to conduct an initial resource
assessment. The resource assessment should include development of an IT system inventory and a review
of existing procedures. Although these activities are considered Step 1 of the development process, they
may be incomplete or subject to revision based on remaining steps of the development process.
Development of IT System Inventory
Information management is a fundamental aspect of a successful contamination warning system.
-------
Water Security Initiative: Operational Strategy Guidance
should document the network environment in which each system is deployed, as this may impact the
feasibility or ease of data integration across systems. If information is collected from external partners,
related IT systems should be identified and included as part of the summary. Table 2-1 provides an
example IT system inventory.
Table 2-1. Example IT System Inventory
System Name
General Description
Users
Network
Environment
Call Management
System
Provides comprehensive management of
customer calls received by the utility. Includes
an Interactive Voice Response to triage and
direct calls.
Customer Service
Representative
Local Area
Network
Supervisory Control
and Data
Acquisition
(SCADA)
SCADA system collects, displays, and stores
operational data collected from treatment
plants, pumping facilities, and other monitoring
points throughout the distribution system.
Treatment Plant or
Distribution System
Operators
Protected
Network
Work Order System
Contains information related to work activities
in the distribution system, including work
orders and work requests.
Distribution Work
Supervisor
Local Area
Network
Laboratory
Information
Management
System
Contains sample information and detailed
analytical results for all water quality analytical
data. Information includes data generated in-
house as well as data provided by external
laboratories.
Laboratory Supervisor,
Laboratory Chemist,
Laboratory Microbiologist,
Managers and Supervisors
Local Area
Network
Water Quality
Database
Repository for all water quality related data.
Includes results associated with investigation
of customer water quality complaints,
summary analytical results, field investigation
results, and special investigations requested
from other divisions and departments, such as
the Health Department. May be part of a
Laboratory Information Management System.
Laboratory Supervisor,
Laboratory Chemist,
Laboratory Microbiologist,
Managers and Supervisors
Local Area
Network
Review of Existing Procedures
The second part of the resource assessment should be a review of existing procedures relative to the
objectives of the contamination warning system. This includes a review of procedures internal to the
utility as well as those of local partners who have a role in operation of the contamination warning
system.
Procedures for routine operations at the utility are well established, and some may be applicable to
contamination warning system operations. Building on these procedures should help to integrate the
contamination warning system with existing procedures and thereby significantly improve the
sustainability of the contamination warning system. For example, the initial pilot utility in Cincinnati had
established procedures with local law enforcement agencies /^- ~N^
to support investigation of security breaches at un-staffed
facilities, and these procedures were leveraged for enhanced
security monitoring. For this component, additional
security monitoring capabilities, including video cameras,
were installed at facilities already monitored by door or
hatch alarms. The video from these facilities is used to
remotely assess the security breach and determine if
notification to law enforcement was necessary. If so, the
existing procedures for notifying law enforcement and
investigating the alarm were followed.
HELPFUL HINT
Building on existing procedures for
routine operations helps integrate the
contamination warning system at the
utility and will significantly improve the
system's sustainability.
Recommendation: Maximize dual-use
applications and leverage existing utility
procedures!
To identify existing procedures, determine whether there are established procedures for responding to and
investigating alarms generated through SCADA, abnormal analytical results for finished drinking water
September 2008
-------
Water Security Initiative: Operational Strategy Guidance
samples, security breaches, consumer calls (e.g., taste and odor), and/or public health inquiries.
Additional procedures that may be useful include protocols for coordination with: support laboratories,
public health agencies, law enforcement, and Hazmat. During this review of existing procedures,
enhancements or modifications necessary for contamination warning system operation should be
documented to support development of the preliminary operational strategy.
At the conclusion of Step 1, the utility should have sufficient information regarding available resources to
facilitate component-level analysis and development of standard operating procedures in Step 2.
2.2 Step 2: Component-Level Analysis - Standard Operating Procedures
The objective of Step 2 is to develop a preliminary standard operating procedure for each of the
monitoring and surveillance components. Development of the preliminary standard operating procedures
can be facilitated by a multi-disciplinary team with representatives from water quality, IT staff,
supervisors from participating divisions, and front-line staff who may have a role in operation of the
component. Project management team members such as the /* >
Water Utility Emergency Response Manager, senior managers, ( HELPFUL HINT
and IT system administrators may participate in the
development of the standard operating procedures for all
components, thus providing some continuity to the process and
facilitating integration of the individual procedures into an
operational strategy during Step 3.
The component-level standard operating procedures should
contain the elements described in Section 3.0: component
description; roles and responsibilities; process flow; and user-
To develop detailed, component-specific
standard operating procedures, form
multi-disciplinary teams with
representatives from water quality,
distribution, engineering, IT, and other
divisions as appropriate. Include
managers, supervisors, and front-line
staff! ,
- /
specific checklists. In general, development will begin with development of component description
followed by establishment of roles and responsibilities. This basic information can then be used to build
a process flow, which is the central element of a standard operating procedure.
Section 3.3 provides a general template for a process flow which includes the following elements: routine
monitoring and surveillance; event detection; notifications; trigger investigations; and determination
regarding possible contamination. Considerations for development of each recommended element of a
component-specific standard operating procedure are provided below:
Routine Monitoring and Surveillance
• Identify the users who will be responsible for routine monitoring and surveillance activities of the
component, along with the IT systems these users access as part of existing job duties.
• Review routine operations to identify opportunities to effectively integrate contamination warning
system monitoring and surveillance activities.
• Determine how users will be alerted to triggers. Options may include visual alarms, audible
alarms, email notifications, and text messaging alerts.
• Identify options for data storage and retrieval to support monitoring and surveillance activities.
• Consider how operation of the contamination warning system and staff roles and responsibilities
may change during non-business hours.
Event Detection
• Identify potential event detection tools that may be used in the deployment of the component.
• Identify the data sources that will be used by event detection system.
September 2008 8
-------
Water Security Initiative: Operational Strategy Guidance
• Identify data output from the event detection system.
• Identify the hardware platform that will host the event detection system and determine how the
relevant data streams will be moved to that platform.
• Determine how event detection alarm information will be displayed or otherwise provided to
users responsible for routine monitoring.
Notifications
• Determine how users will be notified when an alarm has been received such that they can initiate
investigation procedures. Options may include direct notification from users responsible for
routine monitoring to more sophisticated and automated notification mechanisms.
• Identify who needs to receive trigger information during each stage of operations: alarm
notification, trigger investigation, and determination of possible contamination.
Trigger Investigation
• Identify the users who will be responsible for investigation and validation of triggers for the
component, along with the IT systems these users access as part of existing job duties.
• Define the process for conducting the trigger investigation, including all data sources that will be
used during the investigation. Determine the data requirements for each specific user, the data
system from which each user can access the required data, and the process by which the
investigation occurs.
• Evaluate approaches to consolidate data and information used during trigger validation in order to
streamline the process and reduce the time required for the investigation.
Determination Regarding Possible Contamination
• Identify who will make the determination regarding possible contamination and the information
needed to make this determination.
Once the process flow has been developed, checklists can be derived from the activities outlined in the
process flow. In general, checklists should be developed to support specific end-users in fulfilling their
role in routine operation of the monitoring and surveillance component.
At the conclusion of Step 2, a preliminary standard operating procedure should exist for each monitoring
and surveillance component. Furthermore, these procedures should have been vetted with the front-line
staff responsible for day-to-day operation of the system.
2.3 Step 3: System-wide Integration
The final step in developing a preliminary operational strategy is to determine how to effectively integrate
all component-level operating procedures into a functional contamination warning system. In order to
accomplish this step, the project management team should conduct an analysis of each component-
specific standard operating procedure to identify inconsistencies as well as opportunities to streamline and
optimize procedures across components. The basic framework for this analysis should include a cross-
component evaluation of roles and responsibilities, process flows, timelines, notifications, and checklists.
This analysis should result in improved consistency across components as well as more effective
leveraging of resources.
Considerations during the system-wide integration step should include the following:
September 2008 9
-------
Water Security Initiative: Operational Strategy Guidance
• Roles and Responsibilities. For each identified user, verify that their roles and responsibilities
are consistent across all components with respect to routine job functions as well as their role in
each component.
• Process Flow. Verify consistent application of the standard contamination warning system
process: routine monitoring and surveillance, event detection, notifications, trigger investigation,
and determination of possible contamination. In particular, verify that there is consistency in
terms of the level of trigger validation to determine if contamination is possible and timing of
notification to the Water Utility Emergency Response Manager. Each standard operating
procedure ends with a determination of possible contamination, at which point they include a step
to transition to the consequence management plan if contamination is deemed possible.
• Timelines. The time to investigate a trigger may vary across components, but should generally
reach the point of determining whether or not contamination is possible within a few hours.
Through the evaluation of the process flows, similarities may be identified across components. It
is important to evaluate and reconcile the timelines for these components so that similar steps and
processes occur in a similar timeframe. In addition, opportunities to streamline the process flows
may also result in more timely decisions. In a preliminary operational strategy, these timelines
are estimates that should be refined through preliminary testing and operation of the system.
• Notifications. Based on the process flows, all component-level standard operating procedures
should generally conclude with notification of the Water Utility Emergency Response Manager
when contamination has been deemed possible. Other notifications occur throughout each step of
the process. It is important to consider whether the same individual(s) may receive notifications
based on information generated from multiple components. Where this is the case, the
mechanism for notification, as well as the information provided, should be consistent. It is also
possible that through this analysis, it may be necessary to expand notifications to other
individuals or departments within the utility in order to facilitate timely investigation of alarms.
• Checklists. During Step 2 of the development process, checklists may have been developed to
support implementation of component-level standard operating procedures. During the system-
wide integration, these checklists should be analyzed and combined when possible. The resulting
checklists should be designed to support specific users in their investigation of triggers for all
components in which they have a role. This will help to ensure that user roles and responsibilities
are aligned across components and should generally streamline the investigation process.
The system-level analysis can be facilitated by developing summary tables that compile similar
information across all components. A tabular summary of roles and responsibilities might include a
listing of all identified users, the description of their role/responsibility, and an indication of the
component(s) for which they have an operational role. A tabular summary of component-level process
flows might include the process for event detection, a description of the trigger, a summary of the
investigation process, and the definition of a validated trigger. Similar summary tables can be developed
for timelines, notifications, and checklists. The case study in the Appendix includes examples of some of
these summary tables.
Once this system-wide analysis is complete, the component-level standard operating procedures should be
revised to improve consistency and integration of the system components. These component-level
standard operating procedures can then be compiled into an integrated operational strategy, which should
include the following:
• Description of the objectives and use of the integrated operational strategy
• Description of the general operational strategy for the contamination warning system
• Comprehensive listing of users and their roles and responsibilities in operation of the
contamination warning system
September 2008 10
-------
Water Security Initiative: Operational Strategy Guidance
• Summary of the trigger investigation process, and associated timelines, across all components
• Revised, component-level standard operating procedures
• User-specific checklists
After the contamination warning system has been installed and is considered operational, system
deployment should enter the baseline and preliminary testing phase. At this point, the operational
strategy is revised to guide operation of the system "as-built." The objective during the baseline and
preliminary testing phase of deployment is to characterize the system and ensure that procedures,
equipment, software, and other components function / •
adequately. During this period, it may be necessary to
deviate from the operational strategy that would be
implemented in a fully tested, functional system. For
example, more time may be spent investigating the cause of
triggers in order to understand the source of false alarms.
The knowledge and experience gained during the baseline
and preliminary testing phase should be used to refine
system operations and update the operational strategy in
preparation for full deployment.
REMINDER
A primary goal of the operational
strategy is to integrate monitoring and
surveillance for potential contamination
with day-to-day activities to promote
sustainability and identify dual-use
applications.
During the full deployment phase, the operational strategy should be applied in a manner aligned with the
overarching objective stated above: to guide day-to-day operations of the contamination warning system
in a manner that can quickly detect and validate triggers indicative of possible contamination. At this
phase of deployment, it is critical to integrate the operational strategy into routine operations at the utility
and local partner organizations. Otherwise, the system may be difficult, if not impossible, to sustain.
Additional guidance on implementation and maintenance of the operational strategy is included in Section
4.
September 2008
11
-------
Water Security Initiative: Operational Strategy Guidance
Section 3.0: Standard Operating Procedures
This section describes the recommended content and general structure of standard operating procedures
for monitoring and surveillance components of a contamination warning system. The purpose of the
standard operating procedures is to describe routine operation of each monitoring and surveillance
component and a step-by-step process for the initial investigation and validation of triggers. A standard
operating procedure for each of the five monitoring and surveillance components should include the
following elements:
• Component Description. A summary-level description of the monitoring and surveillance
component, initially as conceptualized, but ultimately as-built.
• Roles and Responsibilities. A summary listing of all users that have a role in operation of the
component along with a description of their responsibilities in operation of the contamination
warning system.
• Process Flow. A flow diagram illustrating the process for routine operation and investigation of
triggers from the component.
• User-specific Checklists. Simple forms intended to guide specific users during the initial
investigation of a contamination warning system trigger in a manner consistent with the process
flows in the standard operating procedures.
The remainder of this section provides additional detail regarding each element of a recommended
standard operating procedure. An example operational strategy, including standard operating procedures
for each monitoring and surveillance component, is presented as a case study of the Cincinnati pilot in the
Appendix.
3.1 Component Description
A standard operating procedure should generally begin with a summary description of the component at
its current state of development. The component description is not intended to present detailed design
information, rather it is a high-level description to provide the user with the necessary context to
understand the remaining elements of the standard operating procedure.
The component description may include the following information:
• The general functionality or objective of the component within the context of the contamination
warning system
• A description of major pieces of equipment, such as water quality monitoring stations or security
monitoring systems
• A description of the major information systems or software applications supporting the
component, such as a Supervisory Control and Data Acquisition (SCADA) system, Geographical
Information System (GIS) system, or public health surveillance platform
• A listing of methods that support the component, such as laboratory methods used in baseline
sampling and analysis
• The locations of spatially distributed systems, such as enhanced security or water quality
monitoring sites
September 2008 12
-------
Water Security Initiative: Operational Strategy Guidance
The example component descriptions included in the Appendix (Case Study) are loosely based on the
Cincinnati pilot and are intended to illustrate the level of detail that may be useful for this section of the
operational strategy.
3.2 Roles and Responsibilities
Many users with different job functions are involved in some aspect of contamination warning system
operations. The roles and responsibilities section of the standard operating procedures should provide a
comprehensive listing of all users involved in routine operation of the component. Figure 3-1 illustrates
a generic utility organizational hierarchy and indicates which levels of the organization are anticipated to
be involved in routine operations of the contamination warning system. While Figure 3-1 focuses on the
utility structure, the standard operating procedures should also include representatives from organizations
beyond the utility with a role in contamination warning system operations, such as public health, Hazmat,
and law enforcement.
Senior
Management
v /
Division
Management
v /
No involvement in
routine operations;
roles defined within
the Consequence
Management Plan
-I
_
WUERM
V J
.
Supervisors
v )
Front-line
Staff
v J
Transitions from routine
operation to consequence
management
Primary users of the
standard operating
procedures included in the
Operational Strategy
Figure 3-1. Roles within a Generic Utility Organizational Hierarchy
Once users are identified, their specific roles in contamination warning system operations should be
defined. As shown, in Figure 3-1, the front-line staff who are responsible for day-to-day monitoring of
each component and their supervisors are the primary users of the standard operating procedures.
Therefore, it is critical that their responsibilities are detailed therein. Further, it is important to designate
one or more individuals with specific, overarching responsibilities for coordinating certain aspects of
contamination warning system operation, and in this document these responsibilities are fulfilled by the
Water Utility Emergency Response Manager. A utility /. ^
may refer to this position as something different, but
regardless of the title, this role should generally be filled
by a mid-level manager who can integrate information
from multiple monitoring and surveillance components,
assess the threat of contamination, communicate possible
contamination events to division and senior management,
and initiate the consequence management plan.
HELPFUL HINT
The project management team, as
defined in Planning for Contamination
Warning System Deployment, should be
actively involved in the development of
the component-level standard operating
procedures!
September 2008
13
-------
Water Security Initiative: Operational Strategy Guidance
Figure 3-1 depicts the critical role of the Water Utility Emergency Response Manager in guiding the
transition from routine operations to consequence management. It is important to note that the
responsibilities of the Water Utility Emergency Response Manager may evolve during investigation of a
suspected contamination incident; this manager may initially serve as the Incident Commander and later
serve in a supportive role as the investigation progresses. A more detailed description of this transitional
role is provided in the document Interim Guidance on Developing a Consequence Management Plan
(USEPA, 2008a).
All roles should detail actions during both routine monitoring and surveillance and trigger investigation
and, to the extent possible, be aligned with typical job functions. In some cases, gaps may be identified
that can only be filled by assigning new responsibilities to certain users. Assignment of user
responsibilities during off-hours, weekends, and holidays should also be considered, as the majority of
contamination warning system functions should be covered 24/7/365. Other alternatives include
assignment of some users to be on call or providing key users with remote access to various information
and notification systems. Regardless, efforts to align the standard operating procedures with existing
responsibilities will greatly facilitate integration of the contamination warning system into the utility or
partner organizations.
3.3 Process Flow
The process flow should be the central element of a standard operating procedure. It describes how
routine monitoring and surveillance, event detection, and trigger validation lead to a determination of
possible contamination. Because each component uses different data sources and generates different
triggers, the detailed process flow for each component is unique. However, all component process flows
should include a common set of process elements:
• Routine Monitoring. Typically the process flow will begin with routine monitoring of the
component.
• Initial Trigger. The process flow should illustrate the manner in which triggers are recognized.
Triggers may take the form of an alarm, an alert, an external notification, or an excursion above
an established threshold.
• Notifications. Throughout the process flow, all necessary notifications should be shown at the
point in the process where they would occur. Following initial recognition of the trigger,
notifications are typically made to those individuals who would support the investigation of a
trigger.
• Trigger Investigation. The steps detailing trigger investigation represent a systematic process
for ruling out possible benign causes of the trigger. Typically, these steps will comprise the
majority of the component process flow, and include information collection and analysis that can
be completed in less than two hours.
• Determination Regarding Possible Contamination. Process flows should generally conclude
with a determination regarding whether or not contamination is possible. If contamination is
possible, the Water Utility Emergency Response Manager is notified and the process flow
illustrates a transition to credibility determination and consequence management. If not, the
process returns to routine operation and documentation of the alarm.
Figure 3-2 illustrates a generic process flow showing the basic steps from routine monitoring and
surveillance to event detection and possible determination.
September 2008 14
-------
Water Security Initiative: Operational Strategy Guidance
Monitoring and
Surveillance
Online water
quality monitoring
Sampling and
analysis
Enhanced
security
monitoring
Consumer
complaint
surveillance
Public health
surveillance
Event Detection and Possible Determination
Anomaly
detected;
alarm
generated
i
Data
a
nalysis
i
Initial notifications Review available data: Expand notifications,
and coordination • Operational data onsite investigation,
of initial alarm • Work orders assess other available
investigation • Water quality data information
Rule out
contamination Rule out
as cause of contamination
alarm as cause of
alarm
, i •
No deviations
from baseline
or base-state
Contamination is unlikely,
document investigation
1
Return to routine monitoring and surveillance
IBim consequence 1
management 1
Figure 3-2. Generic Process Flow and Initial Trigger Validation Process
Specific process flows should be developed for each of the monitoring and surveillance components.
Process flows will typically include a flow diagram illustrating the process, along with text describing
each step of the process. Example process flows for each of the monitoring and surveillance components
can be found in the case study discussed in the Appendix. These examples may serve as a starting point
for development of process flows to support operation of a specific contamination warning system
component, but would be modified and probably expanded to reflect the component as-built and operated
in the specific system.
During the development of a process flow it is also useful to estimate the time necessary to investigate
and validate a trigger. Time estimates can be used for planning response actions associated with
consequence management. For example, a different set of response actions might be considered in the
case of a 30 minute validation time compared to those available in the case of a four hour validation time.
The case study in the Appendix includes example timeline estimates for each of the components. The
time, both average and a range, are estimated for each significant step of the process flow, with
consideration given to methods for streamlining the overall timeline. For example, some activities may
take place concurrently, while the time required to perform some aspects of the investigation may be
reduced through improvements to information systems. The timeline developed for a preliminary
operational strategy will likely be based on estimates and should be viewed as goals for system
performance that may influence the design of the system. During the baseline and preliminary testing
phase, drills and exercise may yield more accurate estimates of the time required to complete the trigger
investigation process. Finally, during full deployment, the timeline should be optimized to the extent
possible.
3.4 User-specific Checklists
Checklists can complement the standard operating procedures and serve as an aid to specific users during
investigation of a contamination warning system trigger. These checklists should be derived from the
process flow and serve to prompt the user to check resources, evaluate information, and perform actions
as described in the operational strategy. Unlike process flows that are generally component-centric, the
checklists are user-centric and organized by job function. Furthermore, a well designed standard
operating procedure will generally yield checklists that guide users through a similar set of investigative
activities regardless of the source of the trigger. This integration of checklists is achieved through Step 3
of the operational strategy development process - system-wide integration - as discussed in Section 2.3.
September 2008
15
-------
Water Security Initiative: Operational Strategy Guidance
Section 4.0: Implementation and Maintenance
This section describes recommended activities associated with implementation and maintenance of the
operational strategy throughout the phases of contamination warning system deployment (USEPA, 2007).
After the contamination warning system has been designed and implemented, the operational strategy
should be revised to incorporate any changes based on the "as-built" system. The contamination warning
system should then enter a period of baseline and preliminary testing, which provide users with an
opportunity to learn the system and make modifications to optimize performance. After the system has
been optimized, operations should enter the full deployment phase during which the system is actively
monitored for the purpose of contaminant detection. Over the long-term, the system may undergo
periodic cycles of evaluation and refinement. The following subsections discuss the role of the
operational strategy in these phases of system deployment: 1) baseline and preliminary testing; 2) full
deployment; and 3) evaluation and refinement.
4.1 Baseline and Preliminary Testing
Baseline and preliminary testing begins after design and implementation activities are complete. The
objective of this phase of deployment is to operate the contamination warning system for the purpose of
collecting data necessary to understand and optimize system performance. It should be noted that the
timeline for baseline and preliminary testing phase may vary by component based on the complexity of
component operations and the amount of data generated. It may be necessary to conduct drills and
exercises in order to generate sufficient data for analysis of system performance during this phase of
deployment.
As the system will not be fully operational at this time, it
may be desirable to make some adjustments to the
operational strategy. For example, while alarms and
triggers generated through baseline and preliminary testing
may be investigated and documented for the purposes of
assessment and optimization of the system, notifications
leading to consequence management activities may be
limited to drills and exercises. The operational strategy
should support baseline and preliminary testing in the
following activities: communicating goals and objectives,
training, documentation of performance, and refining the
operational strategy. Each of these activities is discussed in
further detail below.
HELPFUL HINT
Buy-in from all levels, including front-
line staff, supervisors, managers, and
the project management team prior to
initiating baseline and preliminary
testing is critical!
Identification of dual-use applications
of the contamination warning system
components is a powerful way to
garner buy-in.
• Communicating Goals and Objectives. As discussed, the primary objective of the operational
strategy during this phase of deployment is to assess the system and determine whether or not it
operates as designed and intended. The operational strategy should guide routine operations and
trigger investigations; however, response actions are generally not implemented during this
phase. This allows those operating the system on a day-to-day basis to gain a better
understanding of the performance, and possible limitations, of the system. These goals and
objectives should be clearly communicated to all users involved with system operations.
• Training. Training on the operational strategy should occur early in the baseline and preliminary
testing phase, once the standard operating procedures have been revised to reflect the "as-built"
system. To ensure that training approaches and materials are geared to the appropriate audience,
it is recommended that training sessions be divided into training for managers and supervisors
and training for front-line staff. Local partners who have a role in operation of the system should
also be included in training sessions as appropriate.
September 2008
16
-------
Water Security Initiative: Operational Strategy Guidance
o Manager and Supervisor Training. The primary objective for this training is to present
the integrated operational strategy to managers and supervisors and clearly define the
goals and objectives of baseline development. The format for this training may vary, but
a classroom setting should be appropriate. It may also be helpful to develop training
materials to reference during the training, allowing managers and supervisors to stay
engaged in the discussions.
o Front-Line Staff Training. The primary
LESSON LEARNED
Training for front-line staff should be
hands-on and focused on how
contamination warning system activities
fit in with their routine job activities.
objective of this training is to familiarize front-
line staff with their role in the operational
strategy. Emphasis should be placed on
activities that are different from their normal job
duties as well as how normal job duties may
serve a contamination warning system function.
In contrast to the classroom training recommended
for managers and supervisors, training for front-line staff should be hands-on and perhaps
scenario-based. This will allow front-line staff to focus on the operational activities that
are directly applicable to them and begin to assess how to integrate new responsibilities
into day-to-day activities.
• Documentation of Performance. As the primary objective of the baseline and preliminary
testing phase of deployment is to generate data to characterize system performance,
documentation of alarms, triggers, and subsequent actions is critical. Front-line staff should
utilize checklists to log and document triggers, the results of investigations, and possible causes
of triggers. This information will be analyzed to identify modifications to the operational strategy
and/or the contamination warning system to optimize performance prior to full deployment.
• Optimization of Operational Strategy. Near the conclusion of the baseline and preliminary
testing phase, another system-wide analysis should be conducted by the project management
team. The purpose of this analysis is to assess documentation and data generated during the
baseline and preliminary testing phase. Based on this analysis, and on lessons learned through
baseline and preliminary testing of the system, the operational strategy should be revised to
optimize performance. Modifications and enhancements to the contamination warning system
components may also be necessary to support system optimization.
4.2 Full Deployment
J
During the full deployment phase, EPA considers the contamination
warning system to be "operational" with active monitoring and / HELPFUL HINT
surveillance for indications of drinking water contamination. The
operational strategy, revised to reflect system optimization and to
reflect lessons learned from the baseline and preliminary testing
phase, should be distributed to staff with an active role in system
operation, including local partners. It may be helpful to include a
summary of what has changed from the previous version of the
operational strategy. In addition, establishing a "go-live" date for
Establishing a "go-live" date - the point
from which alarms and triggers will be
investigated in accordance with the
operational strategy - should help to
ensure everyone is ready to begin full
deployment and routine operation of the
system!
when the system will be fully operational may help to ensure that V /
everyone clearly understands the change in system operation. From the go-live date forward, alarms and
triggers should be investigated in accordance with the operational strategy, and the consequence
management plan should be enacted when appropriate (e.g., when a trigger is validated and contamination
is considered possible). Thus, it is also important to notify local partners who may have a role in response
and consequence management of the go-live date.
Training on the operational strategy will be critical to the success of the system as it enters full
deployment. During the transition to full deployment, all users should be trained on the operational
September 2008 17
-------
Water Security Initiative: Operational Strategy Guidance
strategy that was revised following baseline and preliminary testing. Furthermore, a maintenance training
program should be established to provide refresher training to current staff on a periodic basis and
comprehensive training for new staff. Furthermore, drills and exercises can be a highly effective method
of training as well as a method for evaluating current procedures as discussed in the next section.
4.3 Evaluation and Refinement
As the contamination warning system is periodically evaluated and refined, it may be appropriate to
update the operational strategy to reflect modifications to the system. An annual system-wide review of
the operational strategy and system performance is recommended. A process for conducting this review
should be established concurrent with full deployment of the system. Factors that may influence
revisions to the operational strategy include evaluation of system performance, enhancements or
modifications to the system, identification of dual-use applications, or other factors external to the system.
In the absence of actual contamination incidents, drills and exercises provide a means for assessing
system performance. In addition, drills and exercises should be considered a part of maintenance training
as discussed in the previous section. To address both .,
objectives of drills and exercises, various approaches /^ REMINDER
may be employed. Tabletop exercises or focused
Throughout the life-cycle of the contamination
warning system, remember to ...
drills may be conducted for specific components at
greater frequency than drills and exercises designed
to assess performance of the entire integrated system.
At a minimum these drills and exercises should he Conduct periodic system-wide analyses and
At a minimum, tnese anils ana exercises snouia be evaluations to optimize operation and performance
conducted annually to coincide with the system-wide
review of the operational strategy. Greater frequency
may be desired and important to ensure that
functionality of the system is maintained. These
.••.•,,,, i i _4_ • i j • Document trigger investigations
activities should also engage local partners involved ., ... , , ,. r ...
0 ° ^ • Identify dual-use applications and other
of the system:
• Conduct routine drills and exercises and
integrate operations with response actions in
the consequence management plan
benefits derived from operation of the system
in operation as appropriate. It may also be
advantageous to integrate operational drills and
exercises with those planned for consequence
management. However, objectives should be clearly defined and agreed to by all participants to minimize
confusion and help ensure success of the drill or exercise. Lessons learned through drills and exercises as
well as routine operation of the system should be documented and reviewed as part of the annual system-
wide analysis.
While drills and exercises are useful for evaluating the system, they are not the only tools available.
Routine water quality, operational, or public health excursions can provide a valuable opportunity for
system evaluation and training. During full deployment, it is expected that at least a few triggers for each
component could result in a conclusion that contamination is possible. The subsequent investigation and
implementation of response actions may involve implementation of not only the operational strategy but
also the consequence management plan. Post-incident review and documentation could potentially
provide some of the most useful information for evaluation, refinement, and identification of dual-use
application.
Over time, additional monitoring and surveillance tools may be available or there may be a desire to
upgrade or modify certain systems or processes within the utility. It is important to evaluate how these
changes may impact or enhance the contamination warning system prior to moving forward with
implementation. Some may involve substantial revisions to the operational strategy whereas others might
involve a preliminary testing phase to assess the performance of the new tool prior to continuing without
modifications to the operational strategy. These instances may be identified in preparation for the annual
system-wide review, or pending the timeframe, may result in an ad hoc system-wide review of the
operational strategy.
September 2008 18
-------
Water Security Initiative: Operational Strategy Guidance
Other factors external to the contamination warning system that may influence revisions to operational
strategy include changing priorities within the utility; dual-use applications of tools, technology, or
information; and/or organizational and management changes. By conducting an annual system-wide
review of the operational strategy, these factors can be identified and addressed while maintaining the
functionality and sustainability of the contamination warning system.
September 2008 19
-------
Water Security Initiative: Operational Strategy Guidance
Section 5.0: References
U.S. Environmental Protection Agency. 2003. Response Protocol Toolbox: Planning for and
Responding to Drinking Water Contamination Threats and Incidents. Interim Final.
U.S. Environmental Protection Agency. 2005. WaterSentinel System Architecture, Draft for Science
Advisory Board Review. EPA 817-D-05-003.
U.S. Environmental Protection Agency. 2007. Water Security Initiative: Interim Guidance on Planning
for Contamination Warning System Deployment. EPA-817-R-07-002.
U.S. Environmental Protection Agency. 2008a. Water Security Initiative: Interim Guidance on
Developing a Consequence Management Plan. EPA-817-R-08-001.
U.S. Environmental Protection Agency. 2008b. Water Security Initiative: Cincinnati Pilot Post-
Implementation System Status. EPA-817-R-08-004.
September 2008 20
-------
Water Security Initiative: Operational Strategy Guidance
APPENDIX
Case Study: Operational Strategy for the Cincinnati
Contamination Warning System
This appendix presents a case study of the operational strategy developed for the Cincinnati
contamination warning system pilot. A detailed description of the post-implementation status of each
component of the Cincinnati contamination warning system is provided in the document, Cincinnati Pilot
Post-Implementation System Status (USEPA, 2008b). In this case study, the operational strategy has been
generalized by simplifying process flows and utilizing nonspecific roles and job functions in an attempt to
make the example more universal. The intent of the case study is to illustrate the application of the
recommendations in this guidance document through presentation of a real-world example, specifically,
the experience gained during the initial Water Security initiative pilot in Cincinnati. The Operational
Strategy for the Cincinnati Contamination Warning System is built around a series of standard operating
procedures and supporting checklists that guide the investigation of triggers from each component of the
contamination warning system.
This case study may provide a useful reference for the development of an operational strategy customized
to a specific locality's contamination warning system. For example, the checklists and process flows
provided in this appendix could be tailored to the specific objectives and organizational structure of a
utility developing its own contamination warning system. However, it is important to recognize that
while the case study has been generalized from the Operational Strategy for the Cincinnati
Contamination Warning System, many artifacts specific to Cincinnati's contamination warning system
remain. Thus, the example should be viewed as illustrative of the concept and not as guidance or
recommendations on the detailed content of a specific operational strategy.
September 2008 21
-------
Water Security Initiative: Operational Strategy Guidance
A.1 Overview and Objectives
EPA's Water Security initiative contamination warning system model can be used to monitor and
integrate a variety of information sources in order to detect conditions that might indicate a contamination
incident. The Cincinnati contamination warning system includes five components, each of which
monitors a different set of information sources:
• Online Water Quality Monitoring (WQM)
• Sampling and Analysis (S&A)
• Enhanced Security Monitoring (ESM)
• Consumer Complaint Surveillance (CCS)
• Public Health Surveillance (PHS)
The operational strategy for this system describes how the five components are operated in a
complementary manner to function as an integrated contamination warning system.
Overview
The Operational Strategy for the Cincinnati Contamination Warning System is organized into seven
sections. This section, A.I, provides an overview of the document and states the objectives of the
operational strategy. It also provides an overarching summary of the roles and responsibilities of the
various users (identified by "job function") in system operations, indicating each component for which a
specific job function has a responsibility in routine operations.
Sections A.2 though A.6 include standard operating procedures for each of the five monitoring and
surveillance components. Each standard operating procedure describes the process for identification and
investigation of triggers, and is organized as follows:
• Component Description: provides a high-level summary of the as-built component, with
sufficient detail to provide the necessary context to understand the procedures that follow.
• Roles and Responsibilities: identifies each job function with a role in routine operation of the
component, and provides a description of their responsibilities.
• Process Flow: presents a step-by-step process for systematically investigating a trigger. The
process flow is presented as a flow diagram with supporting text and a tabular summary of time
estimates to complete each step.
• Checklists: provides a listing of the checklists that support implementation of the standard
operating procedure for the specific component.
Finally, Section A.7 includes the checklists referenced throughout Sections A.2 through A.6. These
checklists are generally developed to support specific job functions across all components in which that
job function has a role. Thus, while the standard operating procedures are developed around each
component of the system, the checklists are developed around the user.
The scope of the standard operating procedures that comprise the operational strategy are limited to the
identification and initial investigation of triggers. Generally, the investigation ends with the conclusion
that either the trigger was a false alarm or indicative of a possible contamination incident. If the latter,
operations shift from routine operations to consequence management as described in the Consequence
Management Plan for the Cincinnati Contamination Warning System Pilot. The first major activity under
the consequence management paradigm is an investigation into the credibility of the possible
contamination incident. This investigation relies upon information obtained from each of the monitoring
and surveillance components, and thus there is an important linkage between the standard operating
procedures that guide routine activities and the consequence management plan that guides the credibility
determination process. For the Cincinnati pilot, these two documents (the Operational Strategy and the
September 2008 22
-------
Water Security Initiative: Operational Strategy Guidance
Consequence Management Plan) have been thoroughly reviewed and integrated to facilitate a smooth
transition from routine operations to consequence management in the event of a possible contamination
incident.
Objectives
The Cincinnati contamination warning system was designed to integrate with existing systems and
procedures such that system operations can be performed by existing staff and front-line supervisors.
Contamination warning system monitoring and surveillance operations will provide benefits beyond
contamination warning, by enabling the utility to rapidly detect and respond to routine water quality
problems.
The objectives of this operational strategy are to document the users, roles and responsibilities, and
procedures used in routine operation of the Cincinnati contamination warning system. The intended
audience for the operational strategy includes the staff and supervisors from the utility and local partner
organizations with responsibility for routine operation of the system. Additionally, upper-management
from participating organizations can use this document to plan for integration of contamination warning
system operations into normal activities, and explore opportunities for dual-use application of the system.
Overview of Roles and Responsibilities
Effective operation of the Cincinnati contamination warning system involves a variety of personnel from
the utility and local partner organizations, each having well-defined responsibilities. Each of the standard
operating procedures presented in this document lists the personnel (identified by job function) that have
roles and responsibilities with respect to routine operation of that component. Table A-l is a
comprehensive listing of job functions and shows that many have roles in multiple monitoring and
surveillance components. The table describes the general role of each job function in contamination
warning system operations and identifies each component in which that job function has a role.
Table A-l was derived from a system-level analysis of the standard operating procedures for all
components and was used to ensure that responsibilities were defined consistently across components. It
is important to note that the roles described in this table will likely evolve as a suspected contamination
event transitions from routine operation to consequence management. A more detailed description of the
responsibilities of each of these roles in the credibility determination process of consequence management
is provided in the document Interim Guidance on Developing a Consequence Management Plan
(USEPA, 2008a).
September 2008 23
-------
Water Security Initiative: Operational Strategy Guidance
Table A-1. Summary of Primary Roles in Routine Contamination Warning System Operations
Job Function
Water Utility Emergency
Response Manager
Water Quality Supervisor
Water Quality Technician
Water Quality Customer
Service Representative
Customer Service
Representative
Laboratory Supervisor
Laboratory Chemist
Laboratory Microbiologist
SCADA Operator
Distribution Work Supervisor
Distribution Field Crews
Utility Security Personnel
Local Law Enforcement
Local Public Health Agencies
Poison Control Center
Fire Department
State Health Department
Contract Laboratory
General Role in CWS Operations
Receive notification of possible
contamination incidents and transition to
consequence management
Supervise the water quality component of
any trigger investigation and coordinate
synthesis of information from other utility
personnel
Investigate the site of WQM or CCS
triggers; inspect WQ monitor stations;
collect samples; and perform field tests
Monitor for CCS triggers and serve as
subject matter expert during investigation
of a water quality complaint
Respond to customer calls and identify
those with unique water quality concerns
Manage laboratory sampling & analysis
activities at the utility and coordinate use
of laboratory results during a trigger
investigation
Perform analysis and QC review of
results from chemical analyses
Perform analysis and QC review of
results from biological analyses
Monitor for WQM and ESM triggers; and
review distribution system operations to
support the investigation of triggers
Monitor for CCS triggers during non-
business hours, and review ongoing and
recent distribution system work to support
the investigation of triggers
Perform field investigations in response
to CCS and ESM triggers
Lead the investigation of all ESM triggers
Lead the criminal aspect of the
investigation of security breach
Provide local public health data;
epidemiologists and disease investigators
Monitor for and investigate PHS triggers
resulting from calls to the center
Manage the IT system that provides 91 1
and EMS data
Analyze samples for select biological
agents and radiochemicals
Analyze samples for designated chemical
analytes
WQM*
S
s
s
s
s
S&A
s
s
s
s
s
s
s
s
s
s
ESM
s
s
s
s
s
s
CCS
•/
V
V
s
•/
V
V
•/
PHS
s
s
s
V
V
V
•/
V
*System names: Water Quality Monitoring (WQM), Sampling and Analysis (S&A), Enhanced Security Monitoring
(ESM), Consumer Complaint Surveillance (CCS), and Public Health Surveillance (PHS)
Overview of Trigger Investigation Process Flows
A similar system-level analysis was conducted for the process flows to ensure that the decision points,
notifications, and end points are consistently defined across components. Table A-2 provides a summary
across all components that includes: the data source, event detection, trigger, investigation, and trigger
validation for each component.
September 2008
24
-------
Water Security Initiative: Operational Strategy Guidance
Table A-2. Summary of Process Flows and Trigger Validation Process for Contamination Warning System Components
CWS Component
Event Detection
Trigger
Investigation
Validated Trigger
Online Water Quality
Monitoring
Event detection system determines
if an anomaly is present in the
water quality data
Water quality alarm is displayed on
a graphical user interface to the
SCADA system located in a control
room staffed 24/7
• Analyze monitoring station status
• Review operational data and
ongoing work in the distribution
system
• Review water quality data from
spatially related locations
• Investigate monitoring station
that witnessed the anomaly
Trigger is validated if alarm is not
explained by a monitoring
equipment problem, operational
changes, distribution system work,
or water quality data review
Sampling and
Analysis
Sample analysis results exceed
control levels or contain non-target
analytes
Internal or external analytical
laboratory contacts the designated
water quality point of contact at the
utility
• Review data to determine if
results exceed baseline control
values
• Review operational data, ongoing
work in the distribution system, and
other water quality data
• Perform confirmatory analysis (if
appropriate)
Trigger is validated if baseline
exceedence cannot be explained
by benign causes
Enhanced Security
Monitoring
Security monitoring systems detect
intrusion at a utility facility
Intrusion alarm is displayed on a
graphical user interface to the
SCADA system located in a control
room staffed 24/7
• Review video feed (if available)
• Conduct field investigation
• Assess witness legitimacy (if
trigger is from witness account)
Trigger is validated if field
investigation indicates that an
intrusion occurred that provided
access to the water supply, or
reveals that hazardous conditions
exist or are suspected
Consumer Complaint
Surveillance
Event detection system determines
if an anomaly exists in the number
and location of consumer complaint
calls
Consumer complaint alarm text
message sent to designated water
quality point of contact at the utility
• Review water quality data
• Review operational data and
ongoing work in the distribution
system
• Spatially analyze consumer
complaint data to identify
clustering
Trigger is validated if the consumer
complaint alarm is not explained by
review of water quality data,
operational changes, or distribution
system work
Public Health
Surveillance
Public health agency detects an
anomaly in EMS/911 data,
emergency room chief complaints,
poison control center cases, or
analysis of infectious disease
cases
Public health alert is sent to
designated water quality point of
contact at the utility via email or a
phone call
• Review data from other
monitoring and surveillance
components
• Review operational data and
ongoing work in the distribution
system
• Review of other pertinent test
results, e.g., coliform
Trigger is validated if the public
health agency determines the
trigger could be related to drinking
water, and the utility determines
contamination is possible, based
on results of investigation
September 2008
25
-------
Water Security Initiative: Operational Strategy Guidance
A.2: Online Water Quality Monitoring Standard Operating
Procedures
Component Description
As a component of EPA's contamination warning system model, online water quality monitoring may
provide an indication of contamination through detection of a water quality anomaly as indicated by
deviations from an established water quality base-state. The online water quality monitoring network
deployed in this contamination warning system is comprised of the monitoring stations deployed at
specific locations throughout the drinking water distribution system, a supervisory control and data
acquisition (SCADA) system to transmit and manage data in a centralized location, and an event detection
system to analyze data for anomalies and possible contamination incidents. The following is a brief
description of the water quality monitoring component.
Twelve water quality monitoring stations were installed throughout the distribution system as shown in
Table A-3. A tiered sensor network design was used, employing two types of monitoring stations. The
Type A water quality monitoring stations monitor for: total organic carbon (TOC), chlorine residual,
oxidation-reduction potential (ORP), conductivity, pH, turbidity, and temperature. The Type B water
quality monitoring stations are similar to the Type A stations, but have a UVA spectrophotometer instead
of a TOC analyzer.
Table A-3. Summary of the Example Online Water Quality Monitoring Network
Location
Fairview Pump Station
Greenville Fire Dept.
Glenn Township Police Dept.
Plum Street Pump Station
Madison Reservoir
City University, North Campus
Hillcrest Reservoir
Main Street US Post Office
Conner Pump Station
Northbrook Fire Dept.
Highland Street Police Dept.
Golf Drive Storage Tank
Monitoring Station Type
Type A (TOC, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Type B (UVA, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Type B (UVA, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Type A (TOC, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Type A (TOC, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Type A (TOC, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Type A (TOC, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Type B (UVA, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Type A (TOC, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Type B (UVA, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Type A (TOC, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Type A (TOC, Chlorine, ORP, conductivity, pH, turbidity, temperature)
Data from the water quality monitoring stations are transferred to the SCADA system at the utility control
center via a digital cellular network. A graphical user interface (GUI) provides real-time data from the
entire water quality monitoring network as well as alarms, operational data, and information from the
event detection system described below. In addition to the primary SCADA workstation installed in the
control center, several remote workstations were installed throughout the utility to provide key personnel
with direct access to data from the water quality monitoring network.
Once the data from the water quality monitoring stations are collected within the SCADA system, it is
analyzed for anomalies that could be indicative of contamination by an event detection system. The event
detection system is installed on a high-performance workstation in a dedicated protected zone that is
connected to the SCADA system through a firewall.
September 2008
26
-------
Water Security Initiative: Operational Strategy Guidance
Roles and Responsibilities
Table A-4 shows the roles that utility personnel and managers have in water quality monitoring. For the
Cincinnati pilot, the Water Quality Supervisor has the lead role in the investigation of water quality
alarms while the SCADA Operator has primary responsibility for routine monitoring for triggers.
Table A-4. Roles and Responsibilities for Routine Operation of Online Water Quality Monitoring
Job Function
Water Quality Supervisor
SCADA Operator
Distribution Work
Supervisor
Water Quality Field
Technician
Water Utility Emergency
Response Manager
(WUERM)
Role in Online Water Quality Monitoring Component Operations
• Assume the lead in the investigation of a water quality trigger.
• Coordinate support from appropriate utility personnel during investigation of a
water quality trigger.
• Review water quality data and related information during the investigation of a
water quality trigger.
• Make the determination regarding whether or not the water quality trigger is a
"possible" contamination threat.
• Decide whether to initiate remote sample collection at the site of the water
quality monitoring station that detected the anomaly.
• Decide whether or not to send field technicians to the field to inspect the water
quality monitoring station.
• Notify the Water Utility Emergency Response Manager if the determination is
made that the contamination threat is "possible."
• Monitor all SCADA alarms 24/7/365, including water quality triggers.
• Notify Water Quality Supervisor in the event of a water quality trigger.
• Review distribution system operations to support the investigation of a water
quality trigger.
• Review distribution system work orders to support the investigation of a water
quality trigger.
• Lead the investigation of a water quality trigger if the Water Quality Supervisor
(or alternate) is unavailable.
• Notify the Water Utility Emergency Response Manager if the abbreviated
investigation cannot rule-out contamination.
• Inspect online water quality monitoring stations.
• Perform field verification of online water quality sensor readings.
• Collect samples from the distribution system in support of a water quality trigger
investigation.
• Review water quality data and related information during the investigation of a
water quality trigger when the Water Quality Supervisor (or alternate) is
unavailable.
• Implement the consequence management plan as necessary.
Process Flow
The process flow in Figure A-l illustrates the steps taken during the investigation of an online water
quality monitoring trigger. The process begins with recognition of a water quality alarm as displayed on
the SCADA GUI and ends with notification of the Water Utility Emergency Response Manager (if
contamination is deemed "possible"), or with logging the incident (if contamination is deemed "not
possible"). The anticipated timeline for validation of a trigger from water quality monitoring is presented
in Table A-5.
September 2008
27
-------
Water Security Initiative: Operational Strategy Guidance
1: SCADA Operator monitors for water
quality alarms 24/7/365
2: Water quality alarm displayed on
the SCADA GUI
3: Notify the Water Quality Supervisor
I
4: Water Quality Supervisor coordinates the
investigation of the water quality alarm
5: Review distribution
system operations data
6: Review re
distribution sy
tor ongoing
m work orders
7: Review water quality
and related data
LEGEND
C 1 Start of Process
| | Action Performed
< > Decision Step
I J End of Decision Tree
8: Is the water quality alarm a result
of operational changes, distribution
system work, or other known and
benign causes?
I
NO
9: The monitoring station that produced the
alarm is sampled and inspected
10: Is the water quality anomaly due
to an equipment problem?
NO
I
Contamination is 'possible'
111: Notify the WUERM and begin |
the credibility determination
process
-YES-
Contamination is unlikely
i12: Close investigation, log
ncident, and return to normal
operation
Figure A-1. Process Flow for Online Water Quality Monitoring: Routine Monitoring and Initial
Trigger Validation
1. SCADA Operator monitors for water quality alarms 24/7/365.
• The event detection system operates in real-time, continuously updating the alarm status shown
on the SCADA GUI for each water quality monitoring location.
• Routine monitoring for water quality alarms is integrated into the procedures typically used for
monitoring of other high priority SCADA alarms.
2. Water quality alarm displayed on the SCADA GUI.
• When a water quality anomaly is detected by the event detection system, the SCADA GUI
displays alarm status, date/time, location, and the parameters suspected of triggering the alarm.
September 2008
28
-------
Water Security Initiative: Operational Strategy Guidance
• The alarm is acknowledged to turn off the audible alert.
3. Notify the Water Quality Supervisor of the water quality alarm.
• If the water quality alarm occurs during non-business hours, a pre-assigned alternate is notified.
4. The Water Quality Supervisor coordinates the investigation of the water quality alarm.
• Request the assistance of the appropriate utility personnel (i.e., SCADA Operator, Distribution
Work Supervisor, etc.) in the investigation of a water quality alarm.
• Review distribution system operations data, as described under Step 5.
• Review recent and ongoing distribution system work, as described under Step 6.
• Review water quality data and related information, as described under Step 7.
5. Use Checklist A-3: Distribution System Operations Review to check for distribution system
operating conditions that could have influenced water quality at the monitoring location,
including the following:
• Pump operation.
• Tank levels and fill/drain status.
• Valve open/close status.
• Relevant system alarms (e.g., control limit, loss of power, loss of communications, intrusion, etc.)
• Unusual demands (e.g., due to fire flow).
• Pressure anomalies in the vicinity of the monitoring station that detected the water quality
anomaly.
• Additional flow and pressure data in the vicinity of the monitoring station that detected the water
quality anomaly, if available.
6. Use Checklist A-4: Distribution System Work Order Review to check ongoing or recent
distribution system work that could have influenced water quality at the monitoring location,
including the following:
• Main breaks, repairs, and replacement.
• Flushing operations.
• Water outages.
• Power outages.
• Work that may have interfered with proper operation of the online monitoring instrumentation.
• Other distribution system work that could have impacted water quality in the vicinity of the
monitoring station(s).
7. Use Checklist A-l: Contamination Warning System Trigger Investigation to check water quality
data and other information potentially related to the trigger, including the following:
• Water quality trend lines from the monitoring location(s) that detected the water quality event.
• Water quality trend lines from other water quality monitoring stations in the distribution system.
Approximate travel time between monitoring stations should be considered when selecting the
time period for display of water quality data from other distribution system monitoring stations.
September 2008 29
-------
Water Security Initiative: Operational Strategy Guidance
• Water quality trend lines for the finished water leaving the treatment plant that supplies the region
in which the monitoring station is located. Approximate travel time from the plant to the
monitoring station that witnessed the water quality anomaly should be considered when selecting
the time period for display of data from the treatment plant.
• Recent treatment plant operating conditions, process water quality, or source water quality.
• Change in the source water supplying the monitoring location that witnessed the anomaly.
• Historic water quality trends, such as seasonal patterns, which are not automatically factored into
the event detection system configuration.
• A log of previously observed water quality anomalies.
• Maintenance and calibration records for the monitoring station(s) that detected the anomaly.
• Attributes, configuration, and settings of the event detection system.
8. Is the water quality alarm a result of operational changes, distribution system work, or other
known and benign causes?
• If "YES," contamination is considered unlikely, the investigation is closed and the trigger is
logged. Go to Step 12.
• If "NO," the water quality EDS alarm is considered valid. Go to Step 9.
9. The water quality monitoring station that produced the alarm is inspected.
• Initiate remote sample collection at the monitoring station that detected the anomaly.
• Based on available information, the appropriate utility supervisor decides whether or not utility
personnel can be sent to the site to conduct the investigation. If conditions are considered too
hazardous for utility personnel, contamination is deemed "possible." Go to Step 11 and seek
additional support (e.g., aHazmat responder) to investigate the site.
• If no obvious hazards are apparent, water quality field technicians are dispatched to the site to
retrieve the sample and investigate the water quality monitoring station. Precautions outlined in a
Site Characterization Plan should be followed during the field inspection.
• Record results of the inspection in Checklist A-2: Distribution System Site Investigation.
• Report the results of the site investigation to the Water Quality Supervisor from the field as they
become available.
10. Is the water quality anomaly due to an equipment problem? Equipment problems may include
sensors, communication systems, IT systems, or ancillary components such as plumbing or
electric.
• If "YES," contamination is considered unlikely. Go to Step 12.
• If "NO," contamination is considered "possible." Go to Step 11.
11. Notify the Water Utility Emergency Response Manager and initiate the credibility
determination process.
• Once all reasonable explanations for the water quality anomaly have been assessed and ruled out,
the water quality trigger is considered valid, and contamination is considered "possible."
• The Water Quality Supervisor notifies the Water Utility Emergency Response Manager.
• The Water Utility Emergency Response Manager implements the credibility determination
process, thereby transitioning to consequence management activities as described in the
September 2008 30
-------
Water Security Initiative: Operational Strategy Guidance
consequence management plan. This includes review of data from other contamination warning
system components.
12. Close investigation, log the incident, and return to normal operation.
• At the conclusion of the investigation, if contamination can be ruled out, return to normal
operation. However, some level of investigation may continue if the anomaly is indicative of an
operational or water quality problem.
• Return to routine monitoring and operating activities.
• The Water Quality Supervisor documents the review and assessment of the water quality trigger
by compiling the checklists used in the investigation.
Table A-5. Example Timeline for Validation of a Water Quality Trigger in the Context of an
Operational Contamination Warning System
Process
Activity ID
Number
2
3
4
5-7
8
9-10
11
Process Activity Description
Water quality alarm displayed on the SCADA GUI
Notify the Water Quality Supervisor of the water
quality alarm
Initiate the water quality trigger investigation and
requests support from the appropriate utility personnel
5: Review distribution system operations data
6: Review recent or ongoing distribution system work
7: Review water quality and related data
Evaluate initial data and determine the validity of the
water quality trigger
Inspect the water quality monitoring station that
witnessed the anomaly and determine if the cause
was an equipment problem
Notify the WUERM and begin the credibility
determination process
TOTAL ELAPSED TIME
Expected
Response Time
(minutes)
2
3
5
20
20
60
15
125
Range of
Response Times
(minutes)
1-5
2-10
2-10
10-30
10-30
30-120
5-20
60-225
Checklists
Four checklists, described in Table A-6, are used in the review of water quality monitoring triggers based
on this example. The checklists are included in Section A.7.
Table A-6. Example Checklists Used during Investigation of a Water Quality Monitoring Trigger
Reference
Checklist A-1
Checklist A-2
Checklist A-3
Checklist A-4
Checklist
Contamination Warning
System Trigger
Investigation
Distribution System Site
Investigation
Distribution System
Operations Review
Distribution System Work
Order Review
User
Water Quality
Supervisor
Water Quality
Field Technician
SCADA Operator
Distribution Work
Supervisor
Description
Checklist involves the review of water quality
data, plant operating conditions, event detection
system settings, and other information
potentially related to the trigger.
Checklist covers inspection of water quality
monitoring stations in support of the
investigation of a water quality trigger.
Checklist involves the review of distribution
system operations that may have influenced
water quality near the location of the trigger.
Checklist involves the review of distribution
system work orders that may have influenced
water quality near the location of the trigger.
September 2008
31
-------
Water Security Initiative: Operational Strategy Guidance
A.3: Sampling and Analysis Standard Operating Procedures
Component Description
In EPA's contamination warning system model, routine operation of the sampling and analysis
component involves routine monitoring at the treatment plants, distribution system locations where water
quality monitoring stations or enhanced security systems are installed, and other strategic distribution
system locations. The primary objective of routine monitoring as part of the contamination warning
system model is to maintain proficiency in the collection and analysis of samples for analytes that may be
of concern during a drinking water contamination incident, and to maintain a database of method
performance and contaminant occurrence (contaminants detected, levels detected, and frequency of
detection) throughout the distribution system. This phase of operation is called maintenance monitoring.
Data generated from baseline monitoring are updated through maintenance monitoring and may be
referenced during triggered sampling and analysis for determination of "possible" contamination events.
To facilitate routine monitoring as part of the contamination warning system a network of local
laboratories, listed in Table A-7, was established. Many of the laboratories may also play a role in
consequence management activities, depending on availability and circumstances surrounding the
suspected incident.
Table A-7. Local Laboratory Network and Sampling Frequency for Maintenance Monitoring
Laboratory
Utility
Contract
Laboratory
State Health
Department
Laboratory
Analysis
Gas Chromatography / Mass Spectrometry with
Purge and Trap Extraction
Gas Chromatography with Mass Spectrometry
Detection using liquid-solid extraction (LSE)
Cyanide - Colorimetric Analysis
Inductively Coupled Plasma - Mass Spectrometry
High Performance Liquid Chromatography with
fluorescence determination
Real-time PCR and Immunoassay (TRF) platforms
Alpha Beta Scintillation Sealer or Gas Flow Low-
Background Proportional Detector
High Purity Germanium Gamma Spectrometry
System
Analytes
VOCs
SVOCs
Free cyanide
Metals
Carbamates
BT Agents
Radiochemicals
In addition, the field methods identified in Table A-8 are also performed as part of routine monitoring on
a monthly basis at water quality monitoring stations, priority pump stations, reservoirs, and tanks in the
distribution system, with the objective of updating baseline data and maintaining emergency response
capabilities.
Table A-8. Field Methods for Safety and Contaminant Screening
Contaminants
Free cyanide
Free chlorine
pH, conductivity, and ORP
Turbidity
Chemical Warfare Agents (VX, sarin, etc.)
Radioactivity (alpha, beta, gamma)
VOCs and combustible gases
Toxicity
Field Test Kit
Portable Colorimeter
Portable Colorimeter
Portable electrochemical
detector
Portable Turbidimeter
Test kit
Hand-held device
Hand-held device
Test kit
September 2008
32
-------
Water Security Initiative: Operational Strategy Guidance
Roles and Responsibilities
As summarized in Table A-9, the sampling and analysis component involves internal and external
laboratory personnel, as well as personnel from water quality, distribution, and operations to assist in
investigation of triggers caused by analytical results outside of established baselines for routine samples.
Table A-9. Roles and Responsibilities for Routine Operation of Sampling and Analysis
Job Function
Water Quality Supervisor
Laboratory Supervisor
Laboratory Chemist
Laboratory Microbiologist
Water Quality Field
Technician
Distribution Work Supervisor
SCADA Operator
Water Utility Emergency
Response Manager
(WUERM)
State Health Department
Laboratory
Contract Laboratory
Role in Sampling and Analysis Component Operations
• Coordinate support from appropriate utility personnel during investigation of the
sampling and analysis trigger.
• Review water quality data and related information during the investigation of a
sampling and analysis trigger.
• Decide whether to initiate additional analysis of sample(s).
• Make the determination regarding whether or not the sampling and analysis
trigger is indicative of possible contamination.
• Notify the Water Utility Emergency Response Manager if the determination is
made that the sampling and analysis trigger is indicative of possible
contamination.
• Coordinate sample flow and laboratory analysis of routine samples.
• Perform data review and update baseline control charts.
• Assist with data interpretation and resolution of Quality Control issues for
analytical methods.
• Provide technical support regarding sample analysis.
• Perform routine and confirmatory analyses for chemical contaminants that are
analyzed in-house by the utility.
• Assist the Laboratory Supervisor with updating the Water Quality Database and
control charts.
• Provide technical support regarding sample analysis and data interpretation.
• Process samples for microbiological analysis using Laboratory Response
Network protocol.
• Assist the Laboratory Supervisor with updating the Water Quality Database and
control charts.
• Provide technical support regarding sample analysis and data interpretation.
• Collect samples from routine monitoring locations in the distribution system.
• Perform routine field screening.
• Review distribution system work orders to support the investigation of a
sampling and analysis trigger.
• Review operational data to support the investigation of a sampling and analysis
trigger.
• Review analytical and related information during the investigation of a sampling
and analysis trigger when the Water Quality Supervisor is unavailable.
• Implement the consequence management plan, as necessary.
• Perform screening and confirmatory analyses for select pathogens and toxins in
routine samples.
• Perform screening and confirmatory analyses for radiochemicals.
• Perform screening and confirmatory analyses for target analytes as specified in
the contract with the utility.
Process Flow
Samples collected through routine monitoring activities may not be analyzed as soon as they are received
by laboratories. However, in the event that a baseline sample exceeds a trigger level, the process flow in
Figure A-2 illustrates the steps that should be taken to investigate the trigger and determine whether or
not contamination is possible based on analytical results. The timeline for validation of a trigger from
routine sampling and analysis presented in Table A-10 reflects this potentially lengthy delay in
recognition of a trigger from this component.
September 2008
33
-------
Water Security Initiative: Operational Strategy Guidance
1: Water Quality Field Technicians collect samples.
Laboratory Supervisor coordinates receipt, processing,
and transport to appropriate laboratories.
2: Laboratories perform analysis using confirmatory and/
or screening methods
LEGEND
3: Laboratories review sample and method performance
data and notify utility by phone if any sample data exceed
predetermined utility notification levels or contain
non-target analytes
Start of Process
Action Performed
Decision Step
End of Decision Tree
4: Laboratory Supervisor reviews data
and updates control charts
NO
5: Are results in exceedence of
baseline control values?
YES
6: Utility resumes
routine monitoring
activities
7: Laboratory Supervisor notifies
Water Quality Supervisor
8: Water Quality Supervisor
coordinates the investigation of the
sampling and analysis trigger
9: Review distribution
system operations data
10: Review recent or
ongoing distribution
system work orders
11: Review water quality
and related data
NO-
12: Is baseline exceedence the result of
operational changes, distribution system
work, or other known and benign causes?
-YES
Contamination is 'possible'
13: The WUERM is notified of
'possible' contamination and begins
the credibility determination process
/
1
r
Contamination is unlikely
i
r
r14: Utility logs event and
esumes routine monitoring
activities
Figure A-2. Process Flow for Sampling and Analysis: Routine Monitoring and Initial Trigger
Validation
1. Water Quality Field Technicians collect samples. Laboratory Supervisor coordinates transfer
of sample to appropriate laboratories for analysis.
• Collect samples from designated contamination warning system sampling locations using
standard in-house procedures. As this is a routine sampling event, it is assumed that there is no
hazard present unless otherwise indicated.
• Receive samples from the field technicians and send samples to the appropriate laboratory for
analysis, per instruction of the Laboratory Supervisor.
September 2008
34
-------
Water Security Initiative: Operational Strategy Guidance
o Complete and maintain chain of custody forms in accordance with standard operating
procedures.
o For samples to be analyzed by contract laboratory, a local courier picks up samples from
the utility within 24 hours of collection.
o Contact courier to transport samples to State Health Department Laboratory for pathogen
and radiochemical analyses.
2. Designated laboratories perform analyses using confirmatory and/or screening methods.
• Analyze samples for VOCs by Gas Chromatography / Mass Spectrometry with Purge and Trap
Extraction, semivolatile organic compounds (SVOCs) by Gas Chromatography with Mass
Spectrometry Detection using liquid-solid extraction (LSE), and cyanide by a colorimetry test.
• Analyze samples for carbamates using High Performance Liquid Chromatography with
fluorescence determination and analyze samples for metals using Inductively Coupled Plasma -
Mass Spectrometry.
• Analyze samples for BT Agents using Real-time PCR and Immunoassay (TRF) platforms and
analyze radiochemicals by Gross Alpha and Gross Beta Scintillation and Gamma Spectroscopy.
3. Laboratories review sample and method performance data.
• Contact the utility Laboratory Supervisor by phone as soon as possible if any targeted
contaminant result exceeds the pre-determined utility notification level or any non-targeted
contaminants are detected. Any flags associated with the results should also be reported to aid in
the interpretation of the data.
o For chemicals and radiochemicals, each laboratory will be provided with utility
notification levels for targeted analytes. For BT agent analysis, the utility notification
levels are in accordance with the State Health Department and Laboratory Response
Network protocols.
• Proceed with standard data reporting. This includes delivery of an electronic file compliant with
the utility's Water Quality Database and a hardcopy deliverable summarizing results.
4. Laboratory Supervisor reviews data and updates control charts.
• Receive data from external laboratories by phone or electronically and work with the appropriate
laboratory personnel to review data and update control charts.
5. Are results in exceedence of baseline control values?
• If "NO," baseline control values are not exceeded. Proceed to Step 6.
• If "YES," baseline control values are exceeded. Proceed to Step 7.
6. Resume routine monitoring activities.
• Baseline control values are not exceeded and utility resumes routine monitoring activities.
7. Laboratory Supervisor notifies the Water Quality Supervisor and provides the following
information:
• Sample location.
• Sample date and time.
• Summary of analytical results and associated QA/QC data for both field and laboratory-based
methods.
• Summary interpretation of results.
8. Water Quality Supervisor coordinates the investigation of the sampling and analysis trigger.
September 2008 35
-------
Water Security Initiative: Operational Strategy Guidance
• Request the assistance of the appropriate utility personnel (i.e., SCADA Operator, Distribution
Work Supervisor, etc.) in the investigation of analytical result that exceeds baseline control
values. In the event that the Water Quality Supervisor is unavailable, a pre-assigned alternate
may assume the lead in the investigation.
• Review distribution system operations data, as described under Step 9.
• Review distribution system work orders, as described under Step 10.
• Review water quality data and related information, as described under Step 11.
9. Use Checklist A-3: Distribution System Operations Review to check for distribution system
operating conditions that could have influenced water quality at the sampling location,
including the following:
• Pump operation.
• Tank levels and fill/drain status.
• Valve open/close status.
• Relevant system alarms (e.g., control limit, loss of power or communications, intrusion, etc.).
• Unusual demands (e.g., due to fire flow).
• Pressure anomalies in the vicinity of the sampling location.
10. Use Checklist A-4: Distribution System Work Order Review to check ongoing or recent
distribution system work that could have influenced water quality at the sampling location,
including the following:
• Main breaks, repairs, and replacement.
• Flushing operations.
• Water outages.
• Power outages.
• Other distribution system work that could have impacted water quality in the vicinity of the
monitoring station(s).
11. Use Checklist A-l: Contamination Warning System Trigger Investigation to check water quality
data and related information, including the following:
• Water quality trend lines from online water quality monitoring location(s) hydraulically linked to
the location where the sample was collected, and based on date and time of sample collection. If
the sampling and analysis trigger is based on pathogen analyses, the investigation should include
data from chlorine sensors as well as heterotrophic plate counts and/or coliform data.
• Water quality trend lines for the finished water leaving the treatment plant that supplies the region
where the sample was collected. Approximate travel time from the plant to the sampling location
where the baseline exceedence occurred should be considered when selecting the time period for
display of data from the treatment plant.
• Recent treatment plant operating conditions, process water quality, or source water quality.
• A change in source water at the location and time of sample collection.
• Historic water quality trends, such as seasonal patterns, which may not be reflected in baseline
control charts.
• A log of previously observed water quality anomalies.
September 2008 36
-------
Water Security Initiative: Operational Strategy Guidance
12. Is the baseline exceedence a result of operational changes, distribution system work, or other
known and benign causes?
• If "NO," contamination is "possible". Goto Step 13.
• If "YES," contamination is considered unlikely, the investigation is closed and the trigger is
logged. Go to Step 14.
13. The Water Utility Emergency Response Manager is notified that contamination is "possible"
and begins the credibility determination process.
• If a baseline exceedence is identified and the contaminant was quantified using a confirmatory
method then there is a high degree of confidence in the analytical result. The fact that the
analytical result was from a confirmatory method is significant and should be given considerable
weight when proceeding with credibility determination.
• If a baseline exceedence is identified based on the results of a screening method (i.e., is from any
method that cannot provide both qualitative and quantitative information with accompanying
valid QC), additional analyses should be performed to confirm and quantify the analytical result.
When available, this information should be weighed against other information gathered through
the credibility determination process.
14. Utility logs event and resumes routine monitoring activities.
• Conduct additional sampling and analyses, if necessary, to better understand the cause of the
analytical result although contamination is considered unlikely based on available information.
• Document the investigation and any follow-up activities in the event that this information can aid
in the interpretation of future deviations from the baseline.
Table A-10. Example Timeline for Validation of a Sampling and Analysis Trigger in the Context of
an Operational Contamination Warning System
Process
Activity ID
Number
1-2
3
4
7
8
9-11
13
Process Activity Description
Collect and analyze samples1
Notify Laboratory Supervisor based on review of
sample and method performance data
Laboratory Supervisor reviews data and updates
control charts
Laboratory Supervisor notifies Water Quality
Supervisor that results are in exceedence of baseline
control values
Water Quality Supervisor initiates initial trigger
validation and coordinates review of operational,
work order, and other water quality data
9: Review distribution system operations data
10: Review distribution system work orders
1 1 : Review water quality and related data
Notify the WUERM and initiate credibility
determination process, along with additional
analyses as necessary2
TOTAL ELAPSED TIME3
Expected
Response Time
(minutes)
7 days
10
10
5
5
20
5
55
Range of
Response Times
(minutes)
1 - 14 days
5-20
5-20
1 -10
2-10
10-30
0-24 hours
23 minutes -25. 5
hours
Note that for routine monitoring there is no urgency during the sample collection, shipping, and analysis,
thus the lengthy duration for Steps 1 and 2. Also, the time for analysis of routinely collected samples is
dependent on the analysis schedules of the various laboratories in the network.
This step includes confirmatory analysis, which if necessary, could take 24 hours or longer.
This is the total elapsed time from Steps 3 through 13.
September 2008
37
-------
Water Security Initiative: Operational Strategy Guidance
Checklists
Three checklists, described in Table A-ll, are used in the review of sampling and analysis triggers based
on this example. The checklists are included in Section A.7.
Table A-11. Example Checklists Used during Investigation of a Sampling and Analysis Trigger
Reference
Checklist A-1
Checklist A-3
Checklist A-4
Checklist
Contamination Warning
System Trigger
Investigation
Distribution System
Operations Review
Distribution System Work
Order Review
User
Water Quality
Supervisor
SCADA Operator
Distribution Work
Supervisor
Description
Checklist involves the analysis of water quality
data and plant operating conditions in the
interpretation of analysis results.
Checklist involves the review of distribution
system operations that may have influenced
water quality at the sampling location.
Checklist involves the review of distribution
system work orders that may have influenced
water quality at the sampling location.
September 2008
38
-------
Water Security Initiative: Operational Strategy Guidance
A.4: Enhanced Security Monitoring Standard Operating
Procedures
Component Description
As a component of EPA's contamination warning system model, enhanced security monitoring may
provide an indication of contamination through detection of security breaches that could provide an
intruder with access to the drinking water supply. The enhanced security monitoring component is
comprised of security monitoring enhancements at priority pump stations, reservoirs, and tanks in the
distribution system. Security monitoring enhancements vary by facility, and are summarized in Table A-
12.
Table A-12. Summary of Enhanced Security Monitoring Equipment per Location
Facility Name
Poplar Grove Storage Tank
Golf Drive Storage Tank
North Service Reservoir
Hillcrest Reservoir
Fairview Pump Station
Mitchell Pump Station
Security Monitoring Device
Ladder Motion Sensor
Ladder Motion Sensor and Hatch Switch
Hatch Switches (3)
Level Switches (2) and Hatch Switches (5)
Indoor Fixed Mount Video Cameras (6)
Door Contact Switches (3) and Hatch Switches (4)
Indoor Motion Sensors (2)
Security and Video Panel Tamper Contact
Lighting Panel Tamper and Loss of Power Contact
Indoor Pan Tilt Zoom Video Cameras (4)
Outdoor Fixed Mounted Camera
Door Contact Switches (8)
Glass Break Sensors (4)
Security and Video Panel Tamper Contact
Lighting Panel Tamper and Loss of Power Contact
At elevated storage tanks, ladder motion sensors would signal an alarm if an intruder attempts to climb
the ladder in an effort to reach the tank hatches. The ladder motion sensor alarms provide an added level
of security as door contact switch alarms would also indicate whether someone had gained access to the
enclosures. Hatch switch alarms installed on top of storage tanks would signal an alarm if an intruder
were to tamper with a tank hatch opening. The combination of ladder motion sensor alarms and hatch
alarms provides redundancy and a more reliable indication of a tampering event.
Fixed mount and pan-tilt-zoom video cameras are installed inside and/or at entrances to pump stations
and are activated by door contact switches or internal motion sensor signals if an intruder were to attempt
to gain access. The video system stores continuous video data on local video recorders, and transmits
short duration event-based video clips in response to a detected security incident (e.g., door contact switch
or motion sensor alarm) for review by utility personnel.
The alarm and video data are sent to a SCADA system via a digital cellular network. A GUI provides
real-time alarm information (intrusion detection and video clips). In addition to the primary SCADA
workstation installed in the utility's control center, remote workstations are installed in the utility security
September 2008
39
-------
Water Security Initiative: Operational Strategy Guidance
office and guard station to provide utility security personnel with direct access to alarm information and
video data.
Table A-12 shows that different types of security information are available from different facilities in the
distribution system. Most notably, some facilities have video monitoring equipment, while others have
only contact switches and motion sensors. There is an important distinction in the investigation of
triggers from facilities with and without video monitoring equipment, as discussed below.
Roles and Responsibilities
Table A-13 shows that utility personnel in water quality, security, and distribution system operations
have a role in enhanced security monitoring. Utility security personnel have a lead role in the
investigation of enhanced security triggers, while the SCADA Operator has primary responsibility for
routine monitoring of alarms, including security alarms.
Table A-13. Roles and Responsibilities for Routine Operation of Enhanced Security Monitoring
Job Function
Utility Security Personnel
SCADA Operator
Distribution Work
Supervisor
Distribution Field Crews
Local Law Enforcement
Water Utility Emergency
Response Manager
(WUERM)
Role in Enhanced Security Monitoring Component Operations
• Lead the investigation of all enhanced security triggers, including: intrusions,
tampering incidents, witness accounts, and threats.
• Assess the legitimacy of witness accounts of possible intrusion.
• Notify local law enforcement if intrusion at a facility is suspected or a written or
verbal threat is received.
• Lead the on-site investigation of a security incident, with assistance from
distribution field crews and local law enforcement, as necessary.
• If an intrusion is confirmed, determine whether or not the intruder could have
accessed the water supply.
• Make the determination regarding whether or not a security incident is a
"possible" contamination threat.
• Monitor all SCADA alarms 24/7/365, including security alarms.
• Make the initial determination regarding whether or not the intrusion alarm has
detected an apparent intruder.
• Notify utility security personnel if an intrusion is suspected.
• Coordinate the site activities of field crews who may support utility security
personnel in the on-site investigation of a security incident.
• Review distribution system work activity to determine whether or not a security
alarm could have been inadvertently caused by utility personnel.
• Perform site activities to support utility security personnel in the on-site
investigation of a security breach.
• Conduct an investigation at the site of a security incident if warranted.
• Interview potential witnesses to a security incident.
• If an unlawful intrusion has been confirmed, establish a crime scene perimeter
and initiate a criminal investigation.
• Notify Water Quality Supervisor if contamination is possible.
• Implement the consequence management plan as necessary.
Process Flow
The process flow in Figure A-3 illustrates the steps taken during the investigation of an enhanced security
monitoring trigger. The process begins with recognition of a security alarm as displayed on the SCADA
GUI and ends with notification of the Water Utility Emergency Response Manager (if contamination is
deemed "possible"), or with logging the incident (if contamination is deemed "not possible"). The
anticipated timeline for validation of a trigger from enhanced security monitoring is presented in Table
A-14.
September 2008
40
-------
Water Security Initiative: Operational Strategy Guidance
1: SCADA Operator monitors for
security alarms 24/7/365
2: Security alarm displayed
on the SCADA GUI
3: Review alarm data using
SCADA GUI
4: Review recent or ongoing
distribution system work orders
, A: Security personnel receive
a witness account or threat of '
\
possible intrusion
r
I
X
I B: Utility security personnel •
I assess credibility of the
I witness or threat '
LEGEND
Start of Process
Action Performed
Decision Step
L \ End of Decision Tree
I
5: Was the alarm inadvertently
caused by authorized utility activity?
I
NO
6: Notify utility security personnel
I
': Is the location equipped
with video monitoring?
I
YES
8: Review video clips for signs
of intrusion
i
r
9: Can intrusion be ruled out?
I
NO
+• 10: Notify local law enforcement
11: Utility personnel and local law
enforcement investigate site of
alarm
12: Can the possibility of intruder
access to the water supply or
distribution system be ruled out?
I
NO
T
Contamination is 'possible'
113: Notify the WUERM and begin |
the credibility determination
process
\/co
^
r
Contamination is unlikely
14: Close investigation, log
incident, and return to
normal operation
T
r
Contamination is unlikely
|
15: Continue investigation
of security intrusion with
local law enforcement
Figure A-3. Process Flow for Enhanced Security Monitoring: Routine Monitoring and Initial
Trigger Validation
The process flow in Figure A-3 illustrates the primary process for investigating security alarms in Steps 1
through 15. A parallel process for investigating witness accounts and threat notifications is shown as
September 2008
41
-------
Water Security Initiative: Operational Strategy Guidance
Steps A and B, which merges with the primary flow at Step 9 of the process flow. This parallel path is
described directly below.
A. Security personnel receive a witness account or threat of possible intrusion/contamination.
• Utility security personnel may be alerted to a possible intrusion in the distribution system through
witness accounts from utility employees or the public, and potentially through a direct threat of
intrusion/contamination (e.g., threatening phone call). Unlike alarms, these alerts may occur at
any location within the system.
B. Utility security personnel assess credibility of the witness account or threat.
• Assess the credibility of a witness through interviews, possibly in collaboration with local law
enforcement.
• In the case of a direct threat to tamper with the water supply or utility property, local law
enforcement should be contacted to assist in the assessment.
• Go to Step 9 of the primary process flow to determine whether or not intrusion can be ruled out.
1. SCADA Operator monitors for security alarms 24/7/365.
• Monitor the security alarms 24/7/365 using the SCADA GUI.
• Routine monitoring for security alarms is integrated into the procedures typically used for
monitoring of other high priority SCADA alarms.
2. Security alarm displayed on the SCADA GUI.
• The SCADA GUI displays alarm status, date/time, location, and possibly video clips associated
with a security alarm.
• The alarm is acknowledged to turn off the audible alarm.
• Security alarms are also displayed to other users with responsibility for monitoring security
alarms, as listed in Table A-12, through remote workstations.
3. Review alarm data using the SCADA GUI for the following:
• Review available alarm information displayed on the SCADA GUI.
• Review the video clips from locations equipped with video cameras to assess whether or not the
alarm may be a result of legitimate utility activity. If the review of video data shows signs of
tampering that could have contaminated the water supply, utility security personnel should be
notified immediately. Go to Step 6.
• Review information from contact alarms and motion sensors (for locations without video
cameras) which may provide an indication of the location within the facility where activity is
occurring. This spatial information may help to assess the nature of the activity and the number
of individuals present.
• Contact the utility personnel thought to be at the site of the alarm for confirmation if legitimate
utility activities are being conducted.
4. Use Checklist A-4: Distribution System Work Order Review to check ongoing or recent
distribution system work that could have caused the security alarm.
• Notify the Distribution Work Supervisor about the security alarm and request support during the
initial investigation.
September 2008 42
-------
Water Security Initiative: Operational Strategy Guidance
• Review work orders to determine if utility personnel could have inadvertently caused the security
alarm.
• Contact the utility personnel thought to be at the site of the alarm for confirmation if legitimate
utility activities are being conducted.
5. Was the alarm inadvertently caused by authorized utility activity?
• If "YES," contamination is considered unlikely, the investigation is closed and the incident is
logged. Go to Step 14.
• If "NO," continue the investigation. Go to Step 6.
6. Notify utility security personnel.
• Notify utility security personnel of a suspected, unauthorized intrusion.
• Utility security personnel review available alarm information displayed on the SCADA GUI.
7. Is the location equipped with video monitoring?
• If "YES," go to Step 8.
• If "NO," go to Step 10.
8. Review video clips for signs of unauthorized intrusion, including:
• Visual confirmation of unauthorized personnel.
• Signs of forced entry, such as damaged doors or broken windows.
• Signs of tampering, such as damaged utility equipment.
• Presence of non-utility equipment, such as tanks, drums, and pumps.
• If the review of video data shows signs of tampering that could have contaminated the water
supply, contamination should immediately be considered "possible."
9. Can intrusion be ruled out?
• If "YES," contamination is considered unlikely. Go to Step 14.
• If "NO," continue the investigation. Go to Step 10.
10. Notify local law enforcement.
• At this point in the investigation, unauthorized intrusion is suspected.
• Contact local law enforcement from the jurisdiction where the facility is located to lead the
criminal investigation.
11. Using Checklist A-5: Security Incident Investigation, utility security personnel, distribution field
crews, and local law enforcement investigate the site of the alarm for the following:
• Visual confirmation of unauthorized personnel.
• Signs of forced entry, such as damaged doors or broken windows.
• Signs of tampering, such as damaged utility equipment.
• Presence of non-utility equipment, such as tanks, drums, pumps, containers, or unfamiliar
apparatus.
September 2008 43
-------
Water Security Initiative: Operational Strategy Guidance
• Signs of a security breach at any point that provides access to the drinking water supply.
• Precautions outlined in the Site Characterization Plan should be followed during the site
investigation.
12. Can the possibility of intruder access to the drinking water supply or distribution system be
ruled out?
• If "YES," contamination is considered unlikely. Go to Step 15.
• If "NO," contamination is considered "possible." Go to Step 13.
13. Notify the Water Utility Emergency Response Manager and initiate the credibility
determination process.
• Once an intrusion has been confirmed and intruder access to the water supply cannot be ruled out,
contamination is considered "possible."
• Utility security personnel notify the Water Utility Emergency Response Manager.
• The Water Utility Emergency Response Manager implements the credibility determination
process, including investigation of other contamination warning system components, as described
in the consequence management plan.
14. Close investigation, log alarm, and return to normal operation.
• At the conclusion of the investigation, if contamination can be ruled out, return to normal
operation.
• Utility security personnel document the investigation of the security incident.
15. Continue investigation of security intrusion.
• If intrusion is confirmed but contamination is ruled out, the security aspect of the investigation
should continue. The investigation should be a joint effort between utility security personnel and
local law enforcement.
• Utility security personnel document the investigation of the security breach.
Table A-14. Example Timeline for Validation of a Security Trigger in the Context of an Operational
Contamination Warning System
Process
Activity ID
Number
2
3-4
6
8
10
11-12
13
Process Activity Description
Security alarm displayed on SCADA GUI
3: Review ongoing distribution system activity
4: Review alarm data using the SCADA GUI
Notify utility security personnel
Review video clips if site is equipped with video
cameras
Notify local law enforcement
Investigate site of security alarm and assess possible
access to the drinking water supply
Notify the WUERM and begin the credibility
determination process
TOTAL ELAPSED TIME
Expected
Response Time
(minutes)
2
10
2
6
2
60
15
97
Range of
Response Times
(minutes)
1-5
5-15
1-5
4-10
1-5
30-120
5-20
47-180
September 2008
44
-------
Water Security Initiative: Operational Strategy Guidance
The time to trigger validation may be substantially reduced for sites with video monitoring equipment if
the video record shows clear evidence regarding unauthorized intrusion (i.e., the trigger may be resolved
at Step 4 in fewer than 10 minutes). Also, if the alert comes through a witness account or
intrusion/contamination threat warning, the time to complete the trigger validation process may be
reduced by 10 to 30 minutes.
Checklists
Two checklists, described in Table A-15, are used in the review of enhanced security triggers based on
this example. The checklists are included in Section A.7.
Table A-15. Example Checklists Used during Investigation of an Enhanced Security Trigger
Reference
Checklist A-4
Checklist A-5
Checklist
Distribution System Work
Order Review
Security Incident
Investigation
User
Distribution Work
Supervisor
Utility Security
Personnel
Description
Checklist involves the review of distribution
system work orders that may have caused a
security alarm.
Checklist covers activities during investigation of
a security breach.
September 2008
45
-------
Water Security Initiative: Operational Strategy Guidance
A.5: Consumer Complaint Surveillance Standard Operating
Procedures
Component Description
As a component of EPA's contamination warning system model, consumer complaint surveillance may
provide an indication of contamination through detection of unusual trends or characteristics in consumer
calls regarding water quality issues. The component design is based on the principles of funnel, filter, and
focus. Calls from multiple sources are funneled into the utility's call center. Next, calls are filtered by
customer service representatives in the call center to eliminate issues that do not involve unusual water
quality. Finally, focus is achieved through the collection of additional information about unusual water
quality concerns that may lead to detection of a water quality anomaly resulting in a consumer complaint
surveillance trigger.
The central element of the consumer complaint surveillance component is an event detection system that
uses algorithms to detect anomalies through analysis of the compiled data at various points through the
system. Table A-16 summarizes the data streams, information system and algorithms used by the event
detection system to analyze those data streams.
Table A-16. Summary of the Algorithms used in the Consumer Complaint Surveillance Event
Detection System
Data Stream
Interactive Voice
Response
Customer Service
Representative
Call Log
Water Quality
Work Orders
Detailed
Description of
Water Quality
Issues
Information System
Call Management
System
Call Management
System
Work Order System
Water Quality
Database
Event Detection System
Callers use a voice menu to select the reason for their call,
including an option to indicate a water quality concern. A
scan statistic is used to detect excursions above an
established base-state that is specific to both day of the week
and time of day.
Customer Service Representatives log each call within
predefined categories, one of which captures unusual water
quality concerns. A scan statistic is used to detect excursions
above an established base-state that is specific to both day of
the week and time of day. This is similar to the algorithm
used for the interactive voice response system, but the Call
Log provides a more accurate characterization of the nature
of the call and eliminates routine concerns, such as rusty or
cloudy water.
A water quality call may result in creation of a work order to
investigate the issue raised by the consumer. CIS is used to
analyze for unusual clusters of water quality work orders
relative to an established base-state.
Unusual water quality issues are referred to a designated
customer representative who interviews the caller to obtain
detailed information regarding the nature of the concern. An
anomaly detection algorithm analyzes the collective data for
anomalies in the characteristics, as well as the temporal
distribution, of reported water quality issues.
A dedicated server on the utility's local area network (LAN) hosts the consumer complaint surveillance
event detection system, comprised of the algorithms described in Table A-16. This server also hosts the
application that integrates information that resides in the three supporting IT systems: Call Management
System, Work Order System, and Water Quality Database, in addition to the city's GIS platform that
provides a means of spatial analysis and display of information. Because the server is located on the
LAN, the consumer complaint surveillance event detection system can be accessed from any workstation
connected to the network by users with the appropriate credentials.
September 2008
46
-------
Water Security Initiative: Operational Strategy Guidance
Roles and Responsibilities
Table A-17 shows that utility personnel in water quality, distribution system operation, and customer
service have a role in consumer complaint surveillance. While not explicitly shown, other city
departments and call centers may receive calls related to water quality issues. These partners have been
provided with a backdoor number to the utility to ensure that all such calls are funneled into the utility's
consumer complaint surveillance process.
Table A-17. Roles and Responsibilities for Routine Operation of Consumer Complaint
Surveillance
Job Function
Water Quality
Supervisor
Water Quality Customer
Service Representative
Water Quality Field
Technician
Customer Service
Representative
SCADA Operator
Distribution Work
Supervisor
Distribution Field Crews
Water Utility Emergency
Response Manager
(WUERM)
Role in Consumer Complaint Surveillance Component Operations
• Assume the lead in the investigation of a consumer complaint surveillance
alarm.
• Coordinate support from appropriate utility personnel during investigation of a
consumer complaint surveillance alarm.
• Review the collective data from the investigation and make the determination
regarding whether or not the consumer complaint surveillance trigger is a
"possible" contamination threat.
• Notify the Water Utility Emergency Response Manager if the determination is
made that contamination is "possible."
• Serve as a subject matter expert in the area of water quality and common
customer water quality concerns.
• Monitor for consumer complaint surveillance alarms during normal business
hours.
• Interview customers who contact the utility with questions or concerns regarding
water quality issues.
• Decide whether or not to create a work order to respond to a water quality
concern raised by a customer.
• Lead the field investigation of a consumer complaint in response to a water
quality work order.
• Receive all calls to the utility, including those dealing with water quality issues,
during normal business hours.
• Advise customers about water quality concerns related to typical distribution
system issues (e.g., rusty water, chlorine odor, etc.) without additional support,
unless requested by the customer.
• Identify calls that deal with unusual or complex water quality issues, and forward
those calls on to the Water Quality Customer Service Representative.
• Review distribution system operations to support the investigation of a
consumer complaint surveillance trigger.
• Monitor for consumer complaint surveillance alarms during non-business hours
when customer calls are re-routed to the Distribution Operations center.
• Receive all emergency calls to the utility during non-business hours, including
those related to water quality concerns.
• Advise customers about water quality concerns related to typical distribution
system issues (e.g., rusty water, chlorine odor, etc.).
• Decide whether or not to create a work order to respond to a water quality
concern raised by a customer during non-business hours.
• Assign distribution field crews to support the field investigation of a consumer
complaint.
• Review distribution system work orders to support the investigation of a
consumer complaint surveillance alarm.
• Support the water quality field technician during the field investigation of a
consumer complaint.
• Review water quality data and related information during the investigation of a
consumer complaint surveillance trigger when the Water Quality Supervisor (or
alternate) is unavailable.
• Implement the consequence management plan as necessary.
September 2008
47
-------
Water Security Initiative: Operational Strategy Guidance
Process Flow
The process flow for consumer complaint surveillance is shown in two diagrams for clarity. Figure A-4
presents the process flow for routine operation, illustrating how consumer calls are funneled, filtered, and
focused to efficiently identify water quality issues. Figure A-5 presents the process flow, beginning with
routine monitoring for consumer complaint surveillance triggers, the process for investigating a trigger,
and the determination regarding "possible" contamination. The anticipated timeline for validation of a
trigger from consumer complaint surveillance is presented in Table A-18.
September 2008 48
-------
Water Security Initiative: Operational Strategy Guidance
IVR selection stored in
Call Management System
/I: CSRs and the Distribution '
\Supervisor monitor utility calls 24/7/365/
i
2: Customer calls with a water quality
concern and selects an IVR option
:Call is coded in Call Log of
the Call Management System
3: Call routed to a Customer Service
Representative or Distribution Work
Supervisor
Additional data regarding
issue is entered into the
Water Quality Database
Work order captured in
the Work Order System
Results of site investigation
are entered into the Water
Quality Database
LEGEND
Start of Process
Action Performed
Decision Step
( ( Data Storage
L J End of Decision Tree
4: Is the call related to an
unusual water quality concern?
I
YES
5: Forward call to the Water Quality
Customer Service Representative
6: Water Quality Customer Service
Representative interviews caller
7: Does the call require a follow-up
site investigation?
YES
±
8: Create a water quality work order
9: Water Quality and Distribution Field
Crews conduct site investigation
I
10: Do the results of the site investigation
indicate possible contamination?
I
YES
Contamination is 'possible'
11: Notify the WUERM and begin
the credibility determination
v process .
NO-
Contamination is unlikely
12: Close investigation, log
incident, and return to
. routine operations .
Figure A-4. Process Flow for Consumer Complaint Surveillance: Routine Monitoring
1. Monitor consumer complaint calls to the utility 24/7/365.
• Customer service representatives (CSRs) handle all calls to the utility during normal business
hours.
September 2008
49
-------
Water Security Initiative: Operational Strategy Guidance
• The Distribution Work Supervisor handles emergency calls to the utility during non-business
hours, including calls from customers with concerns about water quality.
• Callers with non-emergency calls are directed to the interactive voice response (IVR) system and
encouraged to call back during normal business hours.
2. Customer calls with a water quality concern.
• Consumers are directed to call one number for any issue related to drinking water, including
water quality concerns. If consumer calls another local call center or department within the
utility's service area, the call recipient has been trained to route the call to the utility via a
backdoor number.
• Calls to the utility call center are processed through an interactive voice response system, which
presents callers with a voice menu with an option for "water quality questions or concerns."
• The caller's selection from the voice menu is stored in the Call Management System.
3. Call routed to a Customer Service Representative or Distribution Work Supervisor.
• Calls for which the "water quality questions or concerns" option is selected from the voice menu
are moved to the front of the queue during normal business hours or routed directly to the
Distribution Work Supervisor during non-business hours.
• Verify the location the customer is calling about and determines the type of water quality
concern.
• Typical water quality issues handled by the Customer Service Representative or Distribution
Work Supervisor include: rusty water, chlorine odor, and cloudy water due to dissolved air.
• Code all calls in Call Log once the nature of the water quality concern has been identified.
4. Is the call related to an unusual water quality concern?
• If "YES," consumer complaint tracking continues. Go to Step 5.
• If "NO," consumer complaint tracking stops and the call is handled according to routine utility
procedures. Go to Step 12.
5. Forward call to the Water Quality Customer Service Representative.
• If the customer water quality concern is received during non-business hours, the Distribution
Work Supervisor handles the call as described in Step 6.
6. Water Quality Customer Service Representative uses Checklist A-6: Water Quality Customer
Complaint Investigation to interview the caller.
• Collect additional information about the nature of the water quality issue from the caller. If the
caller has difficulty describing the issue, the categories listed in the checklist can be presented for
self-selection by the caller.
• Standardize and enter the information collected during the interview into the Water Quality
Database.
7. Does the call require a follow-up site investigation?
• If "YES," consumer complaint tracking continues. Go to Step 8.
• If "NO," consumer complaint tracking stops and the call is handled according to routine utility
procedures. Go to Step 12.
September 2008 50
-------
Water Security Initiative: Operational Strategy Guidance
8. Create a water quality work order.
• Create a work order in the Work Order System once it is determined that a site investigation is
necessary.
9. Conduct site investigation.
• Identify a field crew to support the site investigation.
• Determine whether or not utility personnel can be sent to the site to conduct the investigation. If
conditions are considered too hazardous for utility personnel, contamination is deemed
"possible." Go to Step 11 and seek additional support (e.g., a Hazmat responder) to investigate
the site.
• Dispatch field crew to the site of the reported water quality issue. Precautions outlined in a Site
Characterization Plan should be followed during the field investigation.
• Record results of the inspection in the Checklist A-2: Distribution System Site Investigation.
• Report the results to the Water Quality Customer Service Representative or Distribution Work
Supervisor from the field as they become available.
10. Do the results of the field investigation indicate contamination?
• If "YES," contamination is considered "possible." Go to Step 11.
• If "NO," consumer complaint tracking stops and the call is handled according to routine utility
procedures. Go to Step 12.
11. Notify the Water Utility Emergency Response Manager and initiate the credibility
determination process.
• If the field investigation yields signs of contamination or site hazards, contamination is
considered "possible" without going through the steps of the initial trigger validation process
flow in Figure A-5.
• The Water Quality Customer Service Representative or Distribution Work Supervisor notifies the
Water Utility Emergency Response Manager.
• The Water Utility Emergency Response Manager implements the credibility determination
process, including investigation of other contamination warning system components, as described
in the consequence management plan.
12. Close investigation, log alarm, and return to normal operation.
• Once sufficient information is collected to determine that a call is not related to an unusual water
quality issue, it is filtered out of the consumer complaint surveillance process.
• While the call may not relate to unusual water quality, it may still require follow-up by the utility.
Normal utility procedures are followed to resolve any calls filtered out of the consumer complaint
surveillance process.
Figure A-5, below, presents the process flow for initial trigger validation, showing the process for routine
monitoring of consumer complaint surveillance triggers, steps in the trigger investigation, and the
determination regarding "possible" contamination.
September 2008 51
-------
Water Security Initiative: Operational Strategy Guidance
1: Monitor consumer complaint
surveillance alarms
I
2: Consumer complaint surveillance
anomaly is detected and triggers alarm
3: The Water Quality Supervisor receives
notification of the consumer complaint
surveillance trigger
4: The Water Quality Supervisor coordinates the investigation
of the consumer complaint surveillance trigger
5: Review distribution
system operations data
6: Review re
distribution sy
tor ongoing
m work orders
7: Review water quality
and related data
LEGEND
C J Start of Process
| | Action Performed
< > Decision Step
I J End of Decision Tree
8: Is the consumer complaint trigger a
result of operational changes,
distribution system work, or other
known and benign causes?
I
NO
I
Contamination is 'possible'
9: Notify the WUERM and begin
the credibility determination
process
1
r
Contamination is unlikely
*
J10: Close investigation, log
ncident, and return to normal
operation
Figure A-5. Process Flow for Consumer Complaint Surveillance: Initial Trigger Validation
1. Monitor for consumer complaint surveillance alarms.
• The Water Quality Customer Service Representative monitors for alarms during normal business
hours.
• The Distribution Work Supervisor monitors for alarms during non-business hours.
2. Consumer complaint surveillance anomaly is detected and triggers alarm.
• The consumer complaint event detection system operates in real-time, continuously updating the
alarm status for each consumer complaint surveillance data stream listed in Table A-16. When a
consumer complaint anomaly is detected, the alarm status will change to alert the Water Quality
Customer Service Representative or Distribution Work Manager.
• The consumer complaint surveillance alarm includes the following information: dates and times
of complaints, locations of water quality complaints, and possibly annotated information about
the call.
September 2008
52
-------
Water Security Initiative: Operational Strategy Guidance
3. Water Quality Supervisor receives notification of the consumer complaint surveillance trigger.
• If the consumer complaint surveillance alarm occurs during non-business hours, a pre-assigned
alternate is notified.
4. The Water Quality Supervisor coordinates the investigation of the consumer complaint
surveillance trigger.
• Request the assistance of the appropriate utility personnel (i.e., SCADA Operator, Distribution
Work Supervisor, etc.) in the investigation of a consumer complaint surveillance alarm.
• Review distribution system operations data, as described under Step 5.
• Review recent and ongoing distribution system work, as described under Step 6.
• Review water quality data and related information, as described under Step 7.
5. Use Checklist A-3: Distribution System Operations Review to check for distribution system
operating conditions that could have influenced water quality aesthetics at the location of the
consumer complaint anomaly, including the following:
• Pump operation.
• Tank levels and fill/drain status.
• Valve open/close status.
• Relevant alarms (e.g., control limit, loss of power, loss of communications, intrusion, etc.)
• Large, unusual demands (e.g., due to fire flow).
• Pressure anomalies.
6. Use Checklist A-4: Distribution System Work Order Review to check ongoing or recent
distribution system work that could have influenced water quality aesthetics at the location of
the consumer complaint anomaly, including the following:
• Main breaks, repairs, and replacement.
• Flushing operations.
• Other distribution system work that could have impacted water quality aesthetics in the vicinity of
the sampling location(s) such as tank cleaning or painting, tank or pipe relining, etc.
7. Use Checklist A-l: Contamination Warning System Trigger Investigation to check water quality
data and other information potentially related to the trigger, including the following:
• Spatial representation of the data that produced the alarm to determine if there is a pattern or
cluster in the water quality calls.
• Characteristics of the reported water quality issues to determine if there are similarities in the
reported aesthetic qualities.
• Recent water quality data from water quality monitoring stations upstream or downstream from
the consumer complaint anomaly.
• Finished water quality data from the treatment plant that supplies the region in which the
consumer complaint occurred. Approximate travel time from the plant to the location of the
complaint should be considered when selecting the time period for analysis.
• Recent treatment plant operating conditions, process water quality, or source water quality.
September 2008 53
-------
Water Security Initiative: Operational Strategy Guidance
8. Is the consumer complaint surveillance trigger a result of operational changes, distribution
system work, or other known and benign causes?
• If "YES," contamination is considered unlikely. Go to Step 10.
• If "NO," contamination is considered "possible." Go to Step 9.
9. Notify the Water Utility Emergency Response Manager and initiate the credibility
determination process.
• Once all reasonable explanations for the consumer complaint surveillance trigger have been
assessed and ruled out contamination is considered "possible."
• The Water Quality Supervisor notifies the Water Utility Emergency Response Manager.
• The Water Utility Emergency Response Manager implements the credibility determination
process, including investigation of other contamination warning system components, as described
in the consequence management plan.
10. Close investigation, log alarm, and return to normal operation.
• At the conclusion of the investigation, if contamination can be ruled out, the system returns to
normal operation. However, some level of investigation may continue if the anomaly is
indicative of an operational or water quality problem.
• The Water Quality Supervisor documents the review and assessment of the consumer complaint
surveillance trigger by compiling the checklists used in the investigation.
Table A-18. Example Timeline for Validation of a Consumer Complaint Surveillance Trigger in the
Context of an Operational Contamination Warning System
Process
Activity ID
Number
2
3
4
5-7
8
9
Process Activity Description
Consumer complaint surveillance anomaly triggers
an alarm through analysis any of the data streams
listed in Table A-1 6
Notify Water Quality Supervisor
Initiate the trigger investigation and request support
from appropriate utility personnel
5: Review distribution system operations data
6: Review distribution system work orders
7: Review water quality and related data
Evaluate initial data and conduct determination
regarding "possible" contamination
Notify the WUERM and begin the credibility
determination process
TOTAL ELAPSED TIME
Expected
Response Time
(minutes)
2
3
5
20
20
15
65
Range of
Response Times
(minutes)
1-5
2-10
2-10
10-30
10-30
5-20
30-105
Checklists
Five checklists, described in Table A-19, are used in the review of consumer complaint surveillance
triggers based on this example. The checklists are included in Section A.7.
September 2008
54
-------
Water Security Initiative: Operational Strategy Guidance
Table A-19. Example Checklists Used during Investigation of a Consumer Complaint Surveillance
Trigger
Reference
Checklist A-1
Checklist A-2
Checklist A-3
Checklist A-4
Checklist A-6
Checklist
Contamination Warning
System Trigger
Investigation
Distribution System Site
Investigation
Distribution System
Operations Review
Distribution System Work
Order Review
Water Quality Consumer
Complaint Investigation
User
Water Quality
Supervisor
Water Quality
Field Technician
SCADA Operator
Distribution Work
Supervisor
Water Quality
Customer Service
Representative
Description
Checklist involves the review of water quality
data, plant operating conditions, spatial
distribution of consumer calls, and the nature of
the reported water quality concern to determine
if the problem is systematic.
Checklist is used to document observations
during field investigation of a consumer
complaint.
Checklist involves the review of distribution
system operating conditions that could have
influenced water quality aesthetics at the
location of the consumer complaint.
Checklist involves the review of distribution
system work orders that could have influenced
water quality aesthetics at the location of the
consumer complaint.
Checklist facilitates the collection of information
from callers reporting unusual water quality
concerns.
September 2008
55
-------
Water Security Initiative: Operational Strategy Guidance
A.6: Public Health Surveillance Standard Operating
Procedures
Component Description
Public health surveillance systems gather and analyze health-related data to identify anomalies, or triggers
that might indicate unusual incidence of disease. The role of public health surveillance in EPA's
contamination warning system model is to gather and analyze data for investigation that will augment
traditional epidemiological surveillance (which often relies on an astute clinician to notice and report
anomalies, or triggers) in order to determine whether a public health event could be attributable to
drinking water. The public health data streams discussed in Table A-20 may be used to detect chemical
or biological contaminants, as indicated.
Table A-20. Data Streams, Public Health Partners, and Detection Capabilities
Data Stream / Source
91 1 call data and EMS logs
Poison Control Center calls
Over-the-Counter drug sales
Infectious disease reports
Emergency room chief complaints
Public Health Partner
Fire Department
Regional Poison Control Center
Local Public Health Department
Local Public Health Department
Local Public Health Department
Target Contaminant Types Detected
Fast-Acting Chemicals
Fast-Acting Chemicals
Pathogens
Pathogens
Pathogens
Each public health data stream provides community level health information. The detail of this
information and the ability to share it is limited by the Health Insurance Portability and Accountability
Act (HIPAA). HIPAA aims to preserve the privacy of medical records by mandating protections on the
communication of medical data, and typically protects certain information that can be used to identify an
individual. Measures to remain compliant with HIPAA should be taken when using public health
surveillance data.
911 call data and EMS logs are gathered from the local fire department in as near real-time as possible
and analyzed using statistical algorithms, to identify anomalies that may be indicative of fast-acting
chemical contamination. An agreement with the local poison control center established analysis protocols
to detect fast-acting chemical contamination, as well as provide toxicological expertise in handling any
sort of public health event. Over-the-counter drug sales are monitored through the National Retail Data
Monitor to detect increases of sales that could be associated with public exposure to pathogens; likewise,
infectious disease records and emergency room chief complaints are collected and monitored to detect
changes in baseline for the identified syndromic categories using the Real-time Outbreak Disease
Surveillance (RODS) tool. Automated data gathering, analysis, and alert generation are emphasized in
order to maximize the potential for timely contamination detection and investigation.
Roles and Responsibilities
The role of public health experts in routine operation of the contamination warning system is to provide
information that might not otherwise be available to utilities. This allows for a coordinated investigation
and response to determine whether or not an association between public health events and water quality
anomalies is possible. As listed in Table A-21, water quality personnel, SCADA operators, and
distribution personnel within the utility play a critical role working in concert with local public health
partners as part of the operational strategy.
September 2008
56
-------
Water Security Initiative: Operational Strategy Guidance
Table A-21. Roles and Responsibilities for Routine Operation of Public Health Surveillance
Job Function
Fire Department
Poison Control Center
Local public health
agencies within utility
service area
Water Quality Supervisor
Laboratory Supervisor
SCADA Operator
Distribution Work
Supervisor
Water Utility Emergency
Response Manager
(WUERM)
Role in Routine Operation of
Public Health Surveillance
• Provide HIPAA-compliant EMS and/or 91 1 data.
• Ensure data meet mutually agreed upon quality control requirements.
• Provide HIPAA-compliant Poison Control data.
• Provide supplemental toxicological expertise.
• Ensure data meet mutually agreed upon quality control requirements.
• Investigate public health surveillance triggers.
• Make the determination whether or not public health surveillance trigger could
be related to drinking water.
• Notify the utility Water Quality Supervisor if the determination is made that the
public health surveillance trigger could be related to drinking water.
• Discuss data with the Water Quality Supervisor to help determine whether
public health surveillance trigger is a "possible" drinking water contamination
threat.
• Receive notification of public health surveillance triggers.
• Review pertinent water quality data and the status of other components to
assess spatial and/or temporal correlations to public health surveillance trigger.
• Discuss data with public health surveillance partners and make the
determination regarding whether or not the public health surveillance trigger is a
"possible" drinking water contamination threat.
• Notify the Water Utility Emergency Response Manager if the determination is
made that the public health surveillance trigger is a "possible" drinking water
contamination threat.
• Review analytical results to investigate a potential link between water and the
public health surveillance trigger.
• Review distribution system operations data to investigate a potential link
between water and the public health surveillance trigger.
• Review maintenance activities to investigate a potential link between water and
the public health surveillance trigger.
• Review water quality data and related information during the investigation of a
public health surveillance trigger when the Water Quality Supervisor (or
alternate) is unavailable.
• Implement the consequence management plan as necessary.
Process Flows
The process flow for public health surveillance in Figure A-6 shows investigation procedures for both the
public health officials and utility officials, and illustrates how they relate to one another. In EPA's
contamination warning system model, public health data are monitored and analyzed to identify triggers,
and local public health determines whether or not a trigger is considered valid and could be indicative of a
possible drinking water contamination event. The public health agency will then notify the utility Water
Quality Supervisor, who in turn investigates utility data to determine whether or not the public health
trigger may be related to drinking water quality. The investigation continues until it is determined either
that the trigger is related to drinking water contamination or that drinking water contamination can be
reliably ruled out. The anticipated timeline for validation of a trigger from public health surveillance is
presented in Table A-22.
September 2008
57
-------
Water Security Initiative: Operational Strategy Guidance
\
1: Analyze whether 911/EMS, NRDM, and/or
ER chief complaints indicate trigger
1
1
/ 3:
.:::::r_::::_:
2: Investigate 91 1/EMS, NRDM, I
and/or ER chief complaints trigger I
r
Is the 91 1/EMS, NRDM and/or ER chief X N
C ) Start of Process
| | Action Performed
< > Decision Step
Public health action
15: Poison Control Center |
| trigger received or LPH
| infectious disease
I trigger identified
YES
6: Could the public health surveillance
trigger be related to drinking water? /
r '
YES
8: Notify the utility Water Quality
Supervisor
1
9: Water Quality Supervisor investigates utility
data for suspected pathogen, chemical, or
unknown contamination
Review distribution
operations data
Review rec
distribution sy
4: Close investigation
I and log incident
NC
\ 7: Continue investigation of I
| trigger until source is I
\ determined and log incident /
or ongoing
m work orders
Review analytical results and
related water quality information
10: Is the public health surveillance trigger considered a
'possible' drinking water contamination event?
-Yes-
-No-
13: Notify WUERM of
'possible' contamination event
14: Notify LPH of 'possible'
contamination event
WUERM initiates credibility
determination process
(Consequence Management Plan)
I.
/""l1: Notify LPH of findings, logs"\
( incident, and returns to normal j
>v operation J
L
15. Continue investigation and
assist with credibility
determination
I 12: Continue investigation of .
| trigger until source is determined
k, and log incident '
Figure A-6. Process Flow for Public Health Surveillance: Routine Monitoring and Initial Trigger
Validation
1. Analyze data from the 911/EMS, National Retail Drug Monitor, and/or emergency room chief
complaint for potential triggers.
• Designated Epidemiology/Disease Investigation personnel from the Local Public Health agency
(or agencies) receive a trigger for the data stream through public health surveillance tools.
September 2008
58
-------
Water Security Initiative: Operational Strategy Guidance
2. Use Checklist A-7: Public Health Surveillance Trigger Investigation to support review of
911/EMS, National Retail Drug Monitor, and/or emergency room chief complaint data.
• Investigate the characteristics of the trigger:
o 911/EMS trigger: do the calls occur in one area, or "cluster?" What chief complaint or
syndrome differs from the expected health of the community?
o National Retail Drug Monitor trigger: what over-the-counter medicine category is being sold
at a rate higher than the established base-state?
o Emergency room chief complaint trigger: what chief complaint syndrome category differs
from the established base-state?
• Verify that underlying data are properly coded (e.g., does not contain an unusual amount of
missing data).
• Review the other public health surveillance data streams for corresponding trends.
• Develop a preliminary hypothesis regarding the cause of the trigger, such as whether the
causative agent is a pathogen or chemical and the potential source(s) of exposure, or if the trigger
is a false alarm.
3. Is the 911/EMS, National Retail Drug Monitor, and/or emergency room chief complaint trigger
considered valid?
• If no, proceed to Step 4. The trigger is not considered valid when it is not supported by other
available public health data.
• If yes, proceed to Step 6. The trigger is considered valid if other available public health data
streams or supplemental information indicate a public health episode. For example, an increase
in over-the-counter anti-nausea medications corresponds to an increase in emergency room chief
complaints for nausea and vomiting. There may be a time delay between when alarms may be
generated by these data streams; this should be considered in the investigation.
4. Close investigation and log alarm.
• Document the investigation and the reason(s) for determining the trigger to be a false alarm using
Checklist A-7: Public Health Surveillance Trigger Investigation.
5. Receipt of trigger from Poison Control Center or Local Public Health infectious disease
surveillance.
• Poison Control Center notifies Local Public Health via phone of a change from the established
base-state; notification includes the Poison Control Center's preliminary hypothesis regarding the
cause of the trigger, such as whether the causative agent is a pathogen or chemical, and the
potential source(s) of exposure.
• Local Public Health Epidemiologist/Disease Investigation personnel identify a change in
established base-state during the course of their routine infectious disease surveillance activities;
this trigger may include notification from an astute physician of a situation with an unusual
cluster of illnesses.
6. Could the public health surveillance trigger be related to drinking water?
• If no, proceed to Step 7. Trigger may be attributable to another known incident, or the symptoms
presented are not indicative of water-based illnesses.
• If yes, proceed to Step 8. Trigger could possibly be water related if symptoms or other patterns
presented relate to illnesses and conditions caused by pathogen or chemical water contamination.
September 2008 59
-------
Water Security Initiative: Operational Strategy Guidance
7. Continue investigation until source of trigger is identified and log alarm.
• Monitor relevant public health surveillance data streams until the source of the trigger is
identified or the trigger is determined to be a false alarm.
• Log trigger upon conclusion of investigation using Checklist A-7: Public Health Surveillance
Trigger Investigation.
8. Notify utility Water Quality Supervisor.
• Notify utility Water Quality Supervisor via telephone of the trigger and provide the following
information:
o Type of trigger (i.e., 91 I/EMS, National Retail Drug Monitor, emergency room chief
complaint, Poison Control Center, physician, other).
o Data element(s) causing trigger (i.e., provider impression, chief complaint, type of over-
the-counter).
o Additional indicators of a public health issue from other public health surveillance
streams, or the absence of such indicators.
o Time frame of cluster.
o Zip code(s).
o Hypothesis regarding the cause of health effects (i.e., pathogen, chemical, unknown) and
the potential source(s) of exposure.
o Plans for further investigation, including estimated timeline.
9. Water Quality Supervisor investigates utility data for suspect pathogen, chemical, or unknown
contamination.
• Review records from the previous 21 days (in cases of suspected pathogen), previous 24 hours (in
cases of suspected chemical), or both (for unknown contaminant) for other contamination
warning system triggers: water quality, routine sampling & analysis, consumer complaint
surveillance, and security alarms that could be spatially or temporally related to the public health
episode. Use Checklist A-1: Contamination Warning System Trigger Investigation.
• Contact Distribution Work Supervisor to review distribution system operations data, including
online water quality monitoring data for previous 21 days (in cases of suspected pathogen, 24
hours (in cases of suspected chemical), or both (for unknown contaminant). Use Checklist A-3:
Distribution System Operations Review.
• Contact Distribution Work Supervisor to review distribution system work activities for previous
21 days (in cases of suspected pathogen, 24 hours (in cases of suspected chemical), or both (for
unknown contaminant). Use Checklist A-4: Distribution System Work Order Review.
• Contact Laboratory Supervisor to review pertinent analytical results for sampling records not
included in contamination warning system triggers, such as coliform and/or HPC, for the previous
21 days (in cases of suspected pathogen, 24 hours (in cases of suspected chemical), or both (for
unknown contaminant), for all sample locations.
10. Is the trigger considered a "possible" drinking water contamination threat?
• If no, proceed to Step 11. The collective results of the utility investigation indicate normal
operations.
• If yes, proceed to Step 13. The collective results of the utility investigation indicate that the
public health event may be related to drinking water.
September 2008 60
-------
Water Security Initiative: Operational Strategy Guidance
11. Notify Local Public Health of findings and return to normal operations:
• Email Local Public Health Epidemiologist/Disease Investigator a copy of Checklist A-l:
Contamination Warning System Trigger Investigation which documents that no corresponding
contamination warning system triggers or abnormal water quality test results occurred during the
time period investigated.
• Return to normal operations.
12. Local Public Health continues investigation until source of trigger is identified and logs alarm.
• Monitor relevant public health surveillance data streams until the source of the trigger is
identified or the trigger is determined to be a false alarm. If at any time drinking water is
suspected as the source of the trigger, return to Step 8.
• Log trigger upon conclusion of investigation using Checklist A-7: Public Health Surveillance
Trigger Investigation.
• Email a copy of Checklist A-7: Public Health Surveillance Trigger Investigation to the utility
Water Quality Supervisor for the utility's records.
13. Notify the Water Utility Emergency Response Manager.
• Notify the Water Utility Emergency Response Manager of the "possible" drinking water
contamination threat and provide the details of the public health surveillance trigger and
corresponding information.
• Water Utility Emergency Response Manager initiates credibility determination process as
outlined in the consequence management plan.
14. The Water Utility Emergency Response Manager notifies Local Public Health of a "possible"
drinking water contamination event.
• During business hours, call the Local Public Health Epidemiologist/Disease Investigator who
reported the public health surveillance trigger; provide specific information about the
corresponding contamination warning system trigger and/or abnormal water quality test results
identified.
• After business hours or on the weekend, call appropriate "after hours" contact and request a call
back from Local Public Health agency that initially notified the utility of the public health
surveillance trigger; provide specific information about the corresponding trigger and/or
abnormal water quality test results identified to Local Public Health representative who returns
the call.
• Inform Local Public Health of the utility's plans for further investigation, including estimated
timeline.
15. Local Public Health continues investigation and assists with credibility determination.
• Continue to monitor relevant public health surveillance data streams until the source of the trigger
is identified or the trigger is determined to be a false alarm.
• Assist with credibility determination in coordination with the Water Utility Emergency Response
Manager.
• Document incident using Checklist A-7: Public Health Surveillance Trigger Investigation.
• Email a copy of Checklist A-7: Public Health Surveillance Trigger Investigation to the Water
Utility Emergency Response Manager for the utility's records.
September 2008 61
-------
Water Security Initiative: Operational Strategy Guidance
Table A-22. Example Timeline for Validation of a Public Health Surveillance Trigger in the Context
of an Operational Contamination Warning System
Process
Activity ID
Number
1 &5
2, 3&6
8
9&10
13&14
15
Process Activity Description
Receive notification of public health surveillance
trigger
Investigate and validate public health surveillance
trigger
Notify utility Water Quality Supervisor of public health
surveillance trigger
Water Quality Supervisor investigates utility data and
determines if there is a possible link to water
Notify the WUERM and Local Public Health agency
of "possible" drinking water contamination threat
Initiate credibility determination as defined in a
consequence management plan
TOTAL ELAPSED TIME
Expected
Response Time
(minutes)
15
60
10
45
10
10
145
Range of
Response Times
(minutes)
1-60
15-120
5-60
30-60
1-30
1-30
53-360
Checklists
Four checklists, described in Table A-23, are used in the review of public health surveillance triggers
based on this example. The checklists are included in Section A.7.
Table A-23. Example Checklists Used during Trigger Investigation for the Public Health
Surveillance Component
Reference
Checklist A-1
Checklist A-3
Checklist A-4
Checklist A-7
Checklist
Contamination Warning
System Trigger
Investigation
Distribution System
Operations Review
Distribution System Work
Order Review
Public Health
Surveillance Trigger
Investigation Checklist
User
Water Quality
Supervisor
SCADA Operator
Distribution Work
Supervisor
Local public
health agencies
Description
Checklist involves review of records of other
CWS triggers from previous 24 hours for
suspected chemical contamination or previous
21 days for suspected biological contamination.
Checklist involves the review of distribution
system operating conditions from previous 24
hours for suspected chemical contamination or
previous 21 days for suspected biological
contamination.
Checklist involves the review of distribution
system work orders from previous 24 hours for
suspected chemical contamination or previous
21 days for suspected biological contamination.
Checklist involves the review of 91 1/EMS,
National Retail Drug Monitor, and/or emergency
room chief complaint data.
September 2008
62
-------
Water Security Initiative: Operational Strategy Guidance
A.7: Examples of Contamination Warning System, Trigger
Investigation Checklists
This section contains examples of checklists that can be used during contamination warning system
trigger investigations, and are designed to facilitate implementation of the standard operating procedures
presented in Section A.2 through A.6. Each checklist is designed for a particular user, and identifies
specific activities that should be performed during investigation of a contamination warning system
trigger. Table A-24 provides a summary listing of the checklists, including a brief description, listing of
the primary user, and indication regarding which monitoring and surveillance SOPs the checklists are
designed to support.
Table A-24: Example Checklists Used during Investigation of Contamination Warning System
Triggers
Reference
Checklist A-1
Checklist A-2
Checklist A-3
Checklist A-4
Checklist A-5
Checklist A-6
Checklist A-7
Checklist
Contamination
Warning System
Trigger Investigation
Distribution System
Site Investigation
Distribution System
Operations Review
Distribution System
Work Order Review
Security Incident
Investigation
Water Quality
Consumer
Complaint
Investigation
Public Health
Surveillance Trigger
Investigation
Primary User
Water Quality
Supervisor
Water Quality
Field Technician
SCADA Operator
Distribution Work
Supervisor
Utility Security
Personnel
Water Quality
Customer Service
Representative
Local public
health agencies
Description
Checklist involves
the review of water
quality data and
plant operating
conditions.
Checklist covers site
investigation of a
consumer complaint
or a water quality
monitoring station.
Checklist involves
the review of
distribution system
operational data,
such as tank levels
and pump
operations.
Checklist involves
the review of
distribution system
work.
Checklist covers
activities during
investigation of a
security breach,
including the site
investigation.
Checklist facilitates
the collection of
information from
callers reporting
unusual water
quality concerns.
Checklist involves
the review of the
time, location, and
data elements that
characterize a public
health trigger.
Relevant CWS Components
Water Quality Monitoring;
Sampling and Analysis;
Consumer Complaint
Surveillance; Public Health
Surveillance
Water Quality Monitoring;
Consumer Complaint
Surveillance
Water Quality Monitoring;
Consumer Complaint
Surveillance; Sampling and
Analysis; Public Health
Surveillance
Water Quality Monitoring;
Sampling and Analysis;
Enhanced Security
Monitoring; Consumer
Complaint Surveillance
Enhanced Security
Monitoring
Consumer Complaint
Surveillance
Public Health Surveillance
September 2008
63
-------
Water Security Initiative: Operational Strategy Guidance
Checklist A-1: Contamination Warning System Trigger Investigation
Water Quality Supervisor or Alternate
Roles and Responsibilities
Assume the lead in the investigation of a water quality trigger when notified by SCADA Operator.
Assume the lead in the investigation of a consumer complaint surveillance trigger when a notification is received.
Coordinate support from SCADA Operator and the Distribution Work Supervisor during investigation of a trigger.
Review water quality data and related information during the investigation of a trigger.
Review public health surveillance data during investigation of a trigger.
Make the determination regarding whether or not a sampling and analysis trigger is valid, and decide whether to
initiate additional analysis of sample(s).
Make the determination regarding whether or not a water quality or consumer complaint surveillance trigger is a
"possible" contamination threat.
Notify the Water Utility Emergency Response Manager if the determination is made that contamination is "possible.
Information Systems used During a Trigger Investigation
• Water quality database.
• SCADA GUI.
• Water quality event detection system (EDS).
• Consumer complaint surveillance event detection system (EDS).
• Public health surveillance event detection system
Investigator Name
Investigator Role
Type of CWS Trigger
Water Quality Monitoring
Sampling and Analysis
Consumer Complaint Surveillance
Public Health Surveillance
Date of
CWS Trigger
Time of
CWS Trigger
Location of CWS Trigger
Sub-type of CWS Trigger
(component specific)
Water Quality Trigger Investigation Checklist
Activity
Completed
Time
Begin Trigger Investigation: Record time.
Water Quality Trend Analysis: Analyze water quality trend lines from the monitoring
location that detected the water quality anomaly.
Initiate Investigation: Following confirmation of the location and nature of the water
quality anomaly, contact the SCADA Operator and Distribution Supervisor to initiate the
EDS alarm investigation.
Historic Water Quality Data: Review historic water quality trends, such as seasonal or
weekly patterns that are not accounted for by the EDS tool.
Historic Water Quality Anomalies: Review records of previously observed water quality
anomalies. Consider patterns and causes of the previous anomalies.
Instrument Maintenance: Review the maintenance and calibration records for the water
quality monitoring station that detected the anomaly.
EDS Tool Configuration: Check the attributes, configuration, and settings of the EDS
tool that triggered the alarm.
Other Distribution System Locations: Analyze water quality trend lines from other water
quality monitoring stations in the distribution system. Consider the travel time between
monitoring stations when selecting time periods.
September 2008
64
-------
Water Security Initiative: Operational Strategy Guidance
Finished Water Quality: Analyze water quality trend lines from water quality monitorinq
stations at the treatment plants. Consider the travel time between monitoring stations
when selecting time periods.
Source Water Quality: Review online water quality monitorinq data for source water or
treatment plant process water.
Plant Operations: Check the recent treatment plant operatinq conditions.
a
a
a
Sampling and Analysis Trigger Investigation Checklist
Activity
Location: Confirm the location where the sample that produced the excursion was
collected.
Data and Time: Confirm the date and time of the sample.
Compare samplinq and analysis data to water quality results: Review samplinq data
and compare to water quality data for the sample time, including chlorine data and
heterotrophic plate counts and/or coliform data if trigger based on pathogen analysis.
Treatment plant operatinq conditions: Review conditions for treatment plant, with
assistance from the Treatment Plant Manager to interpret data.
Completed
a
a
a
a
Consumer Complaint Surveillance Trigger Investigation Checklist
Activity
Location: Plot customer complaint call and work orders on a GIS map to analyze for
spatial clusters.
Initiate Investiqation: Followinq confirmation of the location and nature of the customer
complaints, contact the SCADA Operator and Distribution Supervisor to initiate the
consumer complaint EDS alarm investigation.
Cluster Analysis: Analyze the nature of customer complaints to determine if there is a
commonality in the reported aesthetics.
Water Quality Trend Analysis: Check trend lines in water quality parameters from water
quality monitoring stations in the vicinity of complaints, if possible.
Finished Water Quality: Review recent finished water quality data for parameters that
may be indicative of aesthetic water quality problems in the finished water.
Source Water Quality: Review recent source water quality data for parameters that
may be indicative of aesthetic water quality problems in the finished water.
Plant Operations: Check for chanqes in treatment plant operatinq conditions that may
have an impact on water quality aesthetics.
a
a
a
a
a
a
a
Public Health Surveillance Trigger Investigation Checklist
Activity
Suspected Pathoqen Contamination: Check records from previous 21 days for other
CWS triggers: water quality monitoring, enhanced security monitoring, sampling and
analysis, or consumer complaint surveillance occurring within the zip codes(s) provided
by the public health agency. Check pertinent test results that are not included in the
other CWS triggers, such as coliform and/or HPC, for high values.
a
September 2008
65
-------
Water Security Initiative: Operational Strategy Guidance
Suspected Chemical Contamination: Check records from previous 24 hours for other
CWS triggers: water quality monitoring, enhanced security monitoring, sampling and
analysis, or consumer complaint surveillance occurring within the zip codes(s) provided
by local public health.
Suspected Contamination of Unknown Cause: Perform activities for both suspected
pathogen contamination and suspected chemical contamination.
"Possible" Determination
Activity
Completed
Time
Distribution System Operations: Evaluate information provided by SCADA Operator.
Distribution System Work: Evaluate information provided by Distribution Supervisor.
Consumer Call Classification: Evaluate information provided by Water Quality
Customer Service Representative.
Local Public Health Information: Evaluate information provided by local public health.
Historical anomalies: Review log of previously observed water quality anomalies in an
attempt to identify potential causes of the current trigger.
Is Contamination Possible?: When all reasonable explanations of the alarm have
been assessed and ruled out, consider water contamination as "possible." Otherwise,
reset the trigger and return to normal operations.
Q YES
a NO
WUERM Notification: If trigger cannot be explained by known, benign causes, notify
theWUERM.
Investigation Closed: Record time.
Cause of CWS Trigger
Briefly summarize the results of the investigation and document the suspected cause of trigger,
regardless of whether or not contamination was deemed "possible."
September 2008
66
-------
Water Security Initiative: Operational Strategy Guidance
Checklist A-2: Distribution System Site Investigation
Water Quality Field Technicians
Roles and Responsibilities
• Inspect online water quality monitoring stations.
• Lead the field investigation of a consumer complaint.
• Perform field water quality analyses at the site of the water quality monitoring station or consumer complaint.
• Collect samples from site of water quality monitoring station consumer complaint associated with a trigger.
Investigator Name Investigator Role
Type of CWS Trigger
Q Water Quality Monitoring LI Consumer Complaint Surveillance
trigger ?wl Trigger Location of CWS Trigger
Sub-type of CWS Trigger
(component specific)
Water Quality Monitoring Station Inspection Checklist
Activity
Location: Confirm the inspection location with the Water Quality Supervisor.
Notification: Notify the facility manager for inspections at non-utility locations.
Verify Power: Verify that all sensors on the monitoring station are powered.
Verify Pressure: Verify that pressure to the monitoring station is within
specifications.
Verify Flow: Verify flow to all sensors on the monitoring station.
Check Reagents: Where possible, verify the supply and flow of reagents, and
confirm that none are past expiration.
Check Carrier Gas Flow: Verify that the flow rate of a carrier gas is within
acceptable range for the TOC instruments.
Review Calibration Records: Check the last known calibration date and determine if
it was performed.
Field Verification of Sensor Reading: Compare monitoring station sensor readings
to field test result for a grab sample from the water quality monitoring station
sampling port and/or from a nearby fire hydrant.
Calibration Check: Perform a one-point calibration on the sensor that detected the
water quality anomaly.
Sample Collection: If sample was remotely collected, remove sample vessel for
transport to the water quality laboratory.
Sample Collection Vessel: Install a clean sample vessel with sodium thiosulfate
below the sample tap at the monitoring station.
Reset Station: Reset the remote sampling device as well as all local alarms.
Completed
a
a
a
a
a
a
a
a
a
a
a
a
a
Time
September 2008
67
-------
Water Security Initiative: Operational Strategy Guidance
Do the field inspection results confirm online water quality readings?
Reporting: Report results of the monitoring station investigation to the Water Quality
Supervisor from the field.
a YES
a NO
a
Consumer Complaint Site Investigation Checklist
Activity
Location: Confirm the location of the investigation with the Water Quality Customer
Service Representative.
Notification: Notify the customer when en route to the site.
Interview: Discuss the water quality issue with the customer. Verify all taps on the
premise where the water quality issue was observed.
Site Inspection: Investigate the premise for signs of recent work on the plumbing
system, as well as possible sources of contamination based on the characteristics of
the water quality issue.
Sample Collection - Premise: Collect samples from taps on the premise where the
water quality issue was observed. Perform field tests for basic water quality
parameters.
Sample Collection - Hydrants: Collect samples from hydrants upstream and
downstream from the premise. Perform field tests for basic water quality
parameters.
Reporting: Report results of the monitoring station investigation to the Water Quality
Supervisor or Distribution Work Supervisor from the field.
Completed
a
a
a
a
a
a
a
Time
Summary Findings of Site Inspection
September 2008
68
-------
Water Security Initiative: Operational Strategy Guidance
Checklist A-3: Distribution System Operations Review
SCADA Operator
Roles and Responsibilities
• Monitor all control room alarms, including water quality and physical security alarms, 24/7.
• Notify Water Quality Supervisor in the event of a water quality trigger.
• Notify Utility Security personnel in the event of a physical security trigger.
• Review distribution system operations data to support the investigation of a CWS trigger.
Information Systems used During a Trigger Investigation
• SCADA GUI.
Investigator Name
Investigator Role
Type of CWS Trigger
Water Quality Monitoring
Enhanced Security Monitoring
Sampling and Analysis
Consumer Complaint Surveillance
Public Health Surveillance
Date of
CWS Trigger
Time of
CWS Trigger
Location of CWS Trigger
Sub-type of CWS Trigger
(component specific)
Activity
Completed
Time
Notification: Notify Water Quality Supervisor in the event of a water quality trigger or
Utility Security personnel in the event of an enhanced security trigger.
SCADA Alarms: Review SCADA alarms that may be related to the CWS trigger, such
as water quality alarms, intrusion alarms, loss of power, and loss of communications.
System Operations: Review the impact of tank, reservoir, and pump operation on
water quality in the vicinity of the CWS trigger.
System Flows: Consider unusual flow conditions, such as fire flows, that may have
impacted water quality in the vicinity of the CWS trigger.
System Pressures: Consider unusual pressure conditions, such as surges or low
pressure events, that may have impacted water quality in the vicinity of the CWS
trigger.
Reporting: Report results of the CWS trigger investigation to Water Quality Supervisor
or Utility Security personnel as appropriate.
Summary Findings of Investigation
September 2008
69
-------
Water Security Initiative: Operational Strategy Guidance
Checklist A-4: Distribution System Work Order Review
Distribution Work Supervisor
Roles and Responsibilities
• Review distribution system work orders to support the investigation of a CWS trigger.
• Monitor for consumer complaint surveillance triggers during non-business hours.
Information Systems used During a Trigger Investigation
Work Order System.
Investigator Name
Investigator Role
Type of CWS Trigger
fj Water Quality Monitoring
fj Enhanced Security Monitoring
fj Sampling and Analysis
fj Consumer Complaint Surveillance
fj Public Health Surveillance
Date of
CWS Trigger
Time of
CWS Trigger
Location of CWS Trigger
Sub-type of CWS Trigger
(component specific)
Activity
Completed
Time
Active Work: Identify any current work in the distribution system involving utility
personnel or contractors that could be related to the CWS trigger.
Main Breaks: Identify any main breaks in the vicinity of the CWS trigger, and
investigate the potential relationship between the break and the CWS trigger.
Flushing Operations: Identify any flushing operations in the vicinity of the CWS trigger,
and investigate the potential relationship to the CWS trigger.
Water Outages: Identify any water outages or recent valve operations in the vicinity of
the CWS trigger, and investigate the potential relationship to the CWS trigger.
System Maintenance: Review system maintenance activities, such as tank cleaning,
and investigate the potential relationship to the CWS trigger.
Power Outages: Identify any power outages affecting system operations, and
investigate the potential relationship to the CWS trigger.
Reporting: Report results of the CWS trigger investigation to Water Quality Supervisor
or Utility Security personnel as appropriate.
Summary Findings of Investigation
September 2008
70
-------
Water Security Initiative: Operational Strategy Guidance
Checklist A-5: Security Incident Investigation
Utility Security Personnel
Roles and Responsibilities
Lead the investigation of all enhanced security triggers, including: intrusions, tampering incidents, witness accounts,
and threats.
Assess the legitimacy of witness accounts of possible intrusion through interviews.
Notify local law enforcement if intrusion at a facility is suspected or a written or verbal threat is received.
Lead the on-site investigation of a security incident, with assistance from Distribution System Field Crews and local
law enforcement as necessary.
If an intrusion is confirmed, determine whether or not the intruder could have accessed the water supply.
Information Systems used During a Trigger Investigation
SCADA GUI (to view video clips).
Investigator Name
Investigator Role
Date of Security
Trigger
Time of Security
Trigger
Location of Enhanced Security Trigger
Type of Enhanced Security Trigger
Verbal/Written threat
fj Security alarm with cameras
fj Security alarm w/o cameras
fj Witness Account
Verbal/Written Threat Review Checklist
Activity
Completed
Time
Document Threats Received by Phone: Record date, time, name, incoming phone
number, caller characteristics, background noises, type of possible malice, reason, etc.
Document Written Threats: Record date, time, name, mode of receipt (US mail, fax, email,
FedEx, etc.), return address/fax number, type of possible malice, reason, etc.
Notification: Notify local law enforcement.
Camera Video Clip Review Checklist
Activity
Completed
Time
Video Review Clip For:
• Visual confirmation of intruder at site.
• Signs of forced entry, such as cut fences, cut locks, damaged doors, etc.
• Signs of tampering, such as damaged utility equipment.
• Presence of non-utility equipment, such as tanks, drums, etc.
a
a
a
a
Notification: Unless video review confirms legitimate utility activity was the cause of the
alarm, notify local law enforcement.
Witness Account
Activity
Completed
Time
Receive witness account: If witness reports account directly to the utility, Utility Security
Personnel should collect incident information from the witness. If witness reports directly
to law enforcement, Utility Security Personnel should support the investigation.
September 2008
71
-------
Water Security Initiative: Operational Strategy Guidance
Is witness employed by the utility? If "yes", then account is considered a reliable threat
warning.
Document witness information: Include date/time of interview, name, full contact
information, and why the witness was in the vicinity of suspicious activity.
Document location: Verify location and type of facility where suspicious activity was
witnessed.
Document type of suspicious activity: Determine whether activity was trespassina,
vandalism, theft, tampering, surveillance, breaking and entering, or other suspicious
activity.
Document a description of the suspects: Include how many suspects were present, sex,
race, hair coloring, clothing, voice, or other unusual characteristics.
Document a description of any vehicles at the site: Include make, model, color, license
plate, or other unusual characteristics.
Document a description of any unusual equipment at the site: Equipment could include
explosives, firearms, tools, containers, hardware, pumps, PPE, lab equipment, or other
equipment.
Document a description of any unusual conditions at the site: Conditions could include
explosions or fires, dead/stressed vegetation, fogs or vapors, dead animals, unusual
odors, or unusual noises.
Consider reliability of the source: If witness is not employed by the utility, have they filed
false reports in the past? If "yes", then source is considered suspect.
a YES
a NO
a
a
a
a
a
a
a
a YES
a NO
Site Investigation (including investigation of alarms from sites w/o cameras)
Activity
Confirm location of possible security breach: Determine facility type, such as source
water, tank, treatment plant, distribution, water main, etc.
Check for unusual equipment: Such as discarded PPE (e.q., qloves, masks) tools,
hardware, lab equipment, empty containers, etc.
Check for unusual vehicles on the site: Such as non-utility trucks, cars, SUVs,
construction vehicle, etc.
Check for siqns oftamperinq: Such as cut locks, open access hatches, damaqed
gates/windows/doors, missing or damaged equipment, facility in disarray, etc.
Check for siqns of hazards: Such as unexplained or unusual odors, dead or distressed
vegetation, unexplained clouds or vapors, dead animals, unexplained liquids, etc.
Check for any siqn of security breach: Check all points of access for the facility.
Completed
a
a
a
a
a
a
Time
"Possible" Determination
Activity
Is Contamination Possible?: If the possibility of an intruder qaininq access to the water
supply cannot be ruled out, then contamination is "possible."
WUERM Notification: If contamination is considered "possible", notify the WUERM. If not,
continue investigation of intrusion with local law enforcement.
Completed
a YES
a NO
a
Time
September 2008
72
-------
Water Security Initiative: Operational Strategy Guidance
Investiqation
Closed.
Return
to
normal
operations.
a
Cause of the Enhanced Security Trigger
Briefly summarize the results of the investigation and document the suspected cause of trigger,
regardless of whether or not contamination was deemed "possible."
September 2008 73
-------
Water Security Initiative: Operational Strategy Guidance
Checklist A-6: Water Quality Consumer Complaint Investigation
Water Quality Customer Service Representative
Roles and Responsibilities
• Monitor for consumer complaint surveillance triggers during normal business hours.
• Interview customers who contact the utility with questions or concerns regarding water quality issues.
• Decide whether or not to create a work order to respond to a water quality concern raised by a customer.
Information Systems used During a Trigger Investigation
• Water Quality Database.
• Work Order System.
Investigator Name
Investigator Role
Date of Call
Time of Call
Location of Customer Calls
Consumer Complaint Investigation Checklist
Activity
Completed
Time
Address: Confirm the address where the customer observed the water quality issue.
Date & Time: Confirm the date and time when the customer first noticed the issue
and how long it persisted.
Location: Confirm the specific location within the premise where the customer
observed the water quality issue (e.g. bathroom, kitchen, etc.).
Water Quality Issue: Ask the caller to describe the nature of the water quality issue in
as much detail as possible. Record characteristics below.
Possible Causes: Ask the caller about possible causes of the issue, such as recent
work on the plumbing system at the premise.
Recent Calls: Review work orders and call logs to determine if other water quality
calls of a similar nature or in the same vicinity occurred recently.
Field Investigation: Does the nature of the water quality issue described by the
customer necessitate a field inspection?
Q YES
a NO
Water Quality Work Order: If "yes" to previous question, create water quality work
order to initiate field investigation.
Notifications: Notify the Water Utility Emergency Response Manager if the results of
the field investigation indicate "possible" contamination.
Description of Water Quality Issue
Q Odor
Taste
Appearance
Tactile
Illness
Musty
a Bitter
Cloudy
a Oily
Nausea
Chlorine
Sweet
Rusty/red
Soapy
Diarrhea
Sulfur/septic
a Metallic
Particulate
Abrasive
Rash
September 2008
74
-------
Water Security Initiative: Operational Strategy Guidance
Additional Description:
Summary Findings of Investigation
September 2008 75
-------
Water Security Initiative: Operational Strategy Guidance
Checklist A-7: Public Health Surveillance Trigger Investigation
Local Public Health Agencies
Roles and Responsibilities
Review time, location, and data trends of triggers from public health data streams.
Information Systems Unique to Public Health Surveillance Trigger Investigation
• Public health surveillance event detection system
• Real-time Outbreak Disease Surveillance System (RODS)
• National Retail Data Monitor (NRDM)
• Poison Control Center
Investigator Name
Investigator Role
Date of Public
Health Trigger
Time of Public
Health Trigger
Location of Public Health Surveillance Trigger
Type of Public Health Surveillance Trigger
a EMS
a RODS
Poison Control Center
a 911
a NRDM
Infectious Disease
Activity
Completed Time or Result
Verify location: Is the location of the trigger within the jurisdiction?
Q YES
a NO
Verify location: Does 911/EMS call data occur in a "cluster"?
Q YES
a NO
Verify data completeness: Ensure the underlying data are properly coded and
complete.
Determine initial cause of trigger:
911/EMS: What chief complaint or syndrome has increased?
RODS (ER Chief Complaints): What chief complaint has increased?
NRDM: What over-the-counter medicine is being sold at a higher rate?
Poison Control Center: What sort of symptoms and call types have increased?
Infectious Disease Reports: What disease incidence has increased?
Check other Public Health data streams: Do they show similar trends that support
initial trigger?
Q YES
a NO
Determine if Public Health Surveillance trigger could be related to drinking water: Do
symptoms or other patterns presented relate to illnesses and conditions caused by
pathogen or chemical water contamination? Determine whether symptoms are more
likely due to pathogen or chemical contamination.
Q YES
a NO
Notify Water Quality Supervisor: Public health agency notifies the utility Water Quality
Supervisor that public health trigger is suspected to be related to water, and provides
available information, such as location or possible causative agents.
Assist in investigation: Continue to monitor relevant public health data until source of
trigger is identified and log alarm.
September 2008
76
-------
Water Security Initiative: Operational Strategy Guidance
Email checklist: Email copy of the "Public Health Surveillance Trigger Investigation
Checklist" to Water Quality Supervisor for the utility's records.
Summary Findings of Investigation
September 2008
77
------- |