Water Security Initiative:
Interim Guidance on Planning for
Contamination Warning System
Deployment
..
Contamination
Warning System
-------
Office of Water
EPA817-R-07-002
May 2007
-------
Disclaimer
The Water Security Division, of the Office of Ground Water and Drinking Water, has reviewed and
approved this document for publication. Neither the United States Government nor any of its employees,
contractors, or their employees make any warranty, expressed or implied, or assume any legal liability or
responsibility for any third party's use of or the results of such use of any information, apparatus, product,
or process discussed in this report, or represents that its use by such party would not infringe on privately
owned rights. This document is not a substitute for applicable legal requirements, nor is it a regulation
itself.
Mention of trade names or commercial products does not constitute endorsement or recommendation for
use.
Questions concerning this document should be addressed to:
Jessica Pulz
U.S. EPA Water Security Division
26 West Martin Luther King Drive
Cincinnati, OH 45268-1320
(513)569-7918
Pulz.Jessica@epa.gov
or
Steve Allgeier
U.S. EPA Water Security Division
26 West Martin Luther King Drive
Cincinnati, OH 45268-1320
(513)569-7131
Allgeier. Steve @epa. gov
-------
Purpose of this Document
EPA intends this guidance manual to assist drinking water utilities with planning for contamination
warning system deployment based on the model developed under EPA's Water Security initiative
(formerly known as WaterSentinel). In particular, this manual may aid respondents to an upcoming EPA
Request for Applications (RFA). Under this RFA, the Agency would make financial awards for drinking
water utilities to demonstrate and evaluate contamination warning system pilots. EPA anticipates issuing
this RFA in June 2007.
Further, EPA plans to issue additional interim guidance on contamination warning system operation and
consequence management planning in late 2007. All these interim guidance manuals will then be revised
as needed based on findings of the demonstration pilots and public comment prior to being issued in final
form. Along with this guidance, the Agency intends to develop outreach program to promote adoption of
effective and sustainable drinking water contamination warning systems. The following is a summary of
these additional documents.
Request for Applications (Summer 2007) will solicit proposals for financial awards to assist
drinking water utilities with demonstrating and evaluating contamination warning system
pilots based on the Water Security initiative model.
Interim Concept of Operations Guidance (Fall 2007) will describe the process and
procedures involved in routine operation of a contamination warning system, including
process and information flows, roles and responsibilities, and the initial investigation and
validation of alarms. This document could be used by drinking water utilities to inform and
refine component-specific designs and support deployment and operation of their
contamination warning system.
Interim Consequence Management Plan Guidance (Fall 2007) will assist drinking water
utilities in the development and implementation of a utility-specific consequence management
plan for an existing or emerging contamination warning system. This guidance will also
address integration of the consequence management plan with existing plans, training and
exercise scenarios, and outreach to other local, state, and federal agencies.
Request for Comments
EPA is soliciting suggestions and recommendations to make this interim guidance manual more complete
and user-friendly. Commenters are encouraged to be as specific as possible and to provide references
where appropriate. Submit suggestions by e-mail to: watersecurity@epa.gov and indicate that the
message relates to the "Interim Guidance on Planning for Contamination Warning System Deployment."
-------
Planning for WS-CWS Deployment
Abbreviations and Acronyms
APHL Association of Public Health
Laboratories
AWWA American Water Works Association
CDC Centers for Disease Control and
Prevention
CFR Code of Federal Regulations
CID Criminal Investigation Division
CPTED Crime Prevention through
Environmental Design
CWA Clean Water Act
DHS Department of Homeland Security
EARS Early Aberration Reporting System
EDS Event Detection System
EMS Emergency Medical Service
EPA Environmental Protection Agency
ESSENCE Electronic Surveillance System for the
Early Notification of Community Based
Epidemics
ETV Environmental Technology Verification
FBI Federal Bureau of Investigation
FEMA Federal Emergency Management
Agency
GCWW Greater Cincinnati Water Works
CIS Geographic Information System
HAZWOPR Hazardous Waste Operations
HIPAA Heath Insurance Portability and
Accountability Act
ICS Incident Command System
IT Information Technology
LRN Laboratory Response Network
MRL Minimum Reporting Level
MOU Memorandum of Understanding
NACWA National Association of Clean Water
Agencies
NELAC National Environmental Laboratory
Accreditation Conference
NELAP National Environmental Laboratory
Accreditation Program
NEMI National Environmental Methods Index
NIMS National Incident Management System
NLTN National Laboratory Training Network
NRDM
NRP
NSF
ORP
OSHA
OTC
O&M
PCS
PIR
PLC
PPE
QA
QC
RAM-W
RFA
ROC
RODS
RPTB
RRU
SAM
SCADA
SDWA
SOP
TEVA
TEVA SPOT
TOC
TTEP
T-JM
USDA
VOC
VSAT1
WC IT
WS
WS-CWS
National Retail Data Monitor
National Response Plan
National Sanitation Foundation
Oxidation Reduction Potential
Occupation Safety and Health
Administration
Over-the-counter (drug sales)
Operation and Maintenance
Polychlorinated Biphenyl
Passive Infrared
Programmable Logic Controller
Personal Protective Equipment
Quality Assurance
Quality Control
Risk Assessment Methodology for
Water Utilities
Request for Applications
Receiver Operating Characteristic
Real-time Outbreak and Disease
Surveillance
Response Protocol Tool-box
Risk Reduction Units
Standardized Analytical Methods
Supervisory Control and Data
Acquisition
Safe Drinking Water Act
Standard Operating Procedure
Threat Ensemble Vulnerability
Assessment
TEVA Sensor Placement Optimization
Tool
Total Organic Carbon
Technology Testing and Evaluation
Program
United States Department of Agriculture
Volatile Organic Compound
Vulnerability Self Assessment Tool
Water Contaminant Information Tool
Water Security
WS Contamination Warning System
May 2007
-------
Planning for WS-CWS Deployment
Acknowledgements
EPA's Office of Ground Water and Drinking Water would like to recognize the following individuals and
organizations for their assistance and contributions in development of this document:
Technical Support Center
Eric Bissonette
Office of General Counsel
Peter Ford
Water Security Division
Steve Allgeier
Jeffrey Pencil
David Harvey
Elizabeth Hedrick
Mike Henrie
Nancy Muzzy
Brian Pickard
Jessica Pulz
Dan Schmelling
David Travers
National Homeland Security Research Center
Kathy Clayton
Contractor Support
Yildiz Chambers, CSC
John Chandler, CSC
Kevin Cornell, CSC
Mike Denison, CH2M Hill
Bill Desing, CH2M Hill
Jean Dupree, CSC
Todd Elliott, CH2M Hill
Katie Gavit, CSC
Darcy Gibbons, CSC
David Gnugnoli, CSC
Yakir Hassit, CH2M Hill
Gary Jacobson, CH2M Hill
Reese Johnson, CH2M Hill
Colm Kenny, CH2M Hill
Kim Morgan, CSC
Misty Pope, CSC
Curtis Robbins, CH2M Hill
Jerry Scott, CSC
Doron Shalvi, CSC
Scott Weinfeld, CSC
May 2007
-------
Planning for WS-CWS Deployment
Executive Summary
This manual provides an overview of design and implementation considerations to assist drinking water
utilities in planning for contamination warning system deployment. As a planning tool, this document
describes a general framework and process for implementation and identifies available tools and
resources to support implementation of a contamination warning system.
What is a contamination warning system?
A contamination warning system provides drinking water utilities with a systematic and comprehensive
approach for monitoring and surveillance of the distribution system. Through implementation of the
monitoring and surveillance strategies and a comprehensive consequence management plan, utilities can
improve their ability to detect intentional or unintentional distribution system contamination. In addition,
their increased ability to monitor and understand distribution system water quality may help to optimize
operations and improve the overall quality of the product delivered to customers. Monitoring and
surveillance components of the contamination warning system include the following:
Online water quality monitoring comprises stations located throughout the distribution system
that measure parameter such as chlorine, total organic carbon, conductivity, and pH among
others. Software analyzes the monitoring data to establish a water quality base state. Possible
contamination is indicated when a significant, unexplained deviation from the base state occurs.
Sampling and analysis is the collection of distribution system samples that are analyzed for
various contaminant classes as well as specific contaminants. Sampling is both routine to
establish a baseline and triggered to respond to an indication of possible contamination from
another component. Analyses are conducted for chemicals, radionuclides, pathogens, and toxins
using a laboratory network.
Enhanced security monitoring includes the equipment and procedures that detect and respond
to security breaches at distribution system facilities. Security equipment may include cameras,
motion activated lighting, door contact alarms, ladder and window motion detectors, area motion
detectors, and access hatch contact alarms.
Consumer complaint surveillance enhances the collection and automates the analysis of calls by
consumers for water quality problems indicative of possible contamination. Consumers may
detect contaminants with characteristics that impart an odor, taste, or visual change to the
drinking water.
Public health surveillance involves the analysis of health-related data to identify disease events
that may stem from drinking water contamination. Public health data may include over-the-
counter (OTC) drug sales, hospital admission reports, infectious disease surveillance, emergency
medical service (EMS) reports, 911 calls, and poison control center calls.
In addition to these monitoring and surveillance components, consequence management is a critical
aspect of the overall architecture for a contamination warning system. Consequence management refers
to the procedures and protocols for assessing credibility of a contamination incident and implementing
response actions.
Why deploy a contamination warning system?
Monitoring the distribution system is the primary focus of contamination warning systems. Through the
assessment of vulnerabilities to drinking water systems, water security experts have identified the
distribution system as one of the most vulnerable components in a drinking water utility, with respect to
contamination. Furthermore, intentional contamination, or even the threat of contamination, can have
significant impacts.
May 2007
-------
Planning for WS-CWS Deployment
Drinking water utilities occasionally receive threats or indications of possible contamination. These
contamination threat warnings can be a direct threat or an unusual observation or discovery that indicates
the potential for contamination and initiates actions to investigate and potentially respond. However,
these threat warnings are not standardized and are difficult to corroborate in the absence of an integrated
monitoring and surveillance system and close coordination with response partners.
Deployment of contamination warning systems for drinking water distribution systems provides a
mechanism by which drinking water utilities can detect and respond to contamination threats and
incidents. A contamination warning system is a proactive approach to managing threat warnings that uses
advanced monitoring technologies/strategies and enhanced surveillance activities to collect, integrate,
analyze, and communicate information to provide a timely warning of potential water contamination
incidents and initiate response actions to minimize public health and economic impacts.
In addition, contamination warning system implementation provides the opportunity for dual-use
applications beyond security that could help to promote sustainability of the system by optimizing utility
operations. Drinking water distribution systems may be accidentally contaminated through cross-
connections with non-potable water, permeation of contaminated water through leaking pipes in areas of
the distribution system subject to low pressures, or chemical reactions or microbial growth within the
distribution system pipes. Such unintentional events that result in degradation to distributed water quality
may occur with some regularity.
Potential dual-use benefits of a contamination warning system could include the following:
Detection of cross-connections and other distribution system water quality problems
Improved relationship with public health organizations, including mutual sharing of information
and alerts
Enhanced knowledge of distribution system water quality leading to improved operations (e.g.,
more consistent disinfection residual levels, improved corrosion control, early warning of
nitrification episodes, reduced disinfection byproduct levels, etc.)
Identification of problem valves (closed, partially closed, inoperable)
Improved coordination with local, state, and federal response organizations
Reduced occurrence of tampering and vandalism
Improved information technology systems and interoperability
Improved consumer complaint tracking and response
Improved laboratory capability and an established laboratory network
Consequence management plans applicable to any water quality emergency
What approach or framework should be applied for deployment of a contamination
warning system?
A contamination warning system is, by design, a systematic approach to monitoring and surveillance for
the timely detection of drinking water contamination. As such, deployment of a contamination warning
system relies on the application of system engineering principles to support coordination of technical and
management activities. Through system engineering, disciplines and specialty groups are integrated in a
team effort forming a structured development process that proceeds from design to implementation to
operation. From the beginning of the project, system engineering principles are critical to successful
planning and implementation. The primary application of system engineering for a contamination
warning system is to ensure that the system - monitoring and surveillance components and consequence
management - functions as an integrated whole. System engineering principles should be applied to
every aspect of contamination warning system implementation, including staffing. While routine
operation and maintenance of the contamination warning system should generally fall within the routine
job functions of utility staff, design and implementation may involve significant time and effort from
dedicated managers within the utility. Depending on utility organization and operational approach, these
May 2007 iv
-------
Planning for WS-CWS Deployment
activities may be managed by one individual, or more likely, a core, multi-disciplinary project
management team.
As discussed in Section 3, deployment of a contamination warning system at a water utility should follow
the typical programmatic approach in which proposed enhancements are planned, designed, implemented,
tested, maintained and refined. Table ES-1 provides a summary of the design and implementation
framework applied throughout the document.
Table ES-1. Overview of Design and Implementation Framework
Stage of Approach
Planning and pre-
design
Design
Implementation
Preliminary testing
Operation and
maintenance
Evaluation and
refinement
Description
Developing a core implementation team, defining design objectives to guide
implementation, and a preliminary assessment of existing capabilities relative to design
objectives.
Development of a preliminary concept of operations and development of a detailed work
plan and schedule to guide implementation.
Implementation of enhancements, installation of equipment, and training according to the
plan.
Operation of the contamination warning system for the purpose of collecting data
necessary to understand system performance and finalization of the concept of operations
to optimize system.
Operation of the contamination warning system for the purpose of monitoring for
contamination incidents and other water quality issues.
Analysis of data and information generated during full operation to refine and optimize the
system.
Who should be involved in contamination warning system deployment?
The drinking water utility is the operational hub of the system as the primary operator of the majority of
monitoring and surveillance components of the contamination warning system, with the exception of
public health surveillance. However, other partners may be involved in initial investigation of alarms
(trigger validation) and/or consequence management activities. Figure ES-1 provides an overview of
potential partners in contamination warning system implementation. As illustrated in this figure, the
number and scope of partners that can become involved in responding to a contamination event can be
significant. In planning for implementation of a contamination warning system, drinking water utilities
should identify and engage local partners early in the process, particularly those partners such as local
health departments and public health and environmental laboratories that may have a significant role in
routine operations. Specific responsibilities of partners and when they are engaged may vary by utility
and jurisdiction. Section 2 of the document provides additional details regarding the roles and
responsibilities of external partners.
May 2007
-------
Planning for WS-CWS Deployment
Federal Bureau of
Investigation
State Emergency
Management
and
Homeland Security
Agencies
State Law
Enforcement
Centers For Disease
Control and Prevention
EPA Regional Offices
Local Health
Department
Local Fire, EMS,
and HazMat
Local Emergency
Planning Committees
Local Wastewater
Utility
Host
Facilities
Local Law
Enforcement
Local Civil
Government
Public Health and
Environmental
Laboratories
State Drinking and Waste
Water Primacy Agencies
Neighboring Utilities
EPA Criminal
Investigation Division
EPA National Response
Center
State Emergency
Res ponders
State Government
Media
Figure ES-1. Potential Contamination Warning System Partners
What are key design considerations for contamination warning system deployment?
Using this document as a guide, the utility in collaboration with local partners where appropriate, should
define what it wants the system to do (defining the design basis), develop a preliminary model of how the
system would function (developing a preliminary concept of operations) and compare the model to the
utility's current capabilities in order to identify the gaps and inform the plan to achieve the desired goals
and objectives. Design decisions to support planning for implementation of a contamination warning
system are summarized in Table ES-2. It is important to apply system engineering principles to all
aspects of design and implementation, particularly as they relate to utility IT systems.
Table ES-2. Summary of Design Decisions to Support Planning for Contamination Warning
System Implementation
Component
Document
Section
Design Decisions
Online Water
Quality
Monitoring
Water quality parameters to be monitored
Use of a single monitoring station design or multiple designs in a tiered system
Specific sensors and instruments integrated into a water quality monitoring
station
Number of water quality monitoring stations to install
Methodology for determining the locations at which water quality monitoring
stations should be installed
Communication architecture to transmit data from monitoring locations to an
operations center
IT architecture used to manage and store water quality and related data
Event detection software deployed to detect anomalies
Sampling and
Analysis
List of target contaminants, including contaminant class; responsible laboratory;
analytical method; laboratory certification or accreditation for the method
Sampling plan that addresses the training that staff should receive; sampling
equipment that should be procured; sampling locations, and rationale; and
sampling frequency, and rationale
Procedures for triggered sampling and analysis
Site characterization procedures and responsibilities
May 2007
VI
-------
Planning for WS-CWS Deployment
Component
Document
Section
Design Decisions
Enhanced
Security
Monitoring
Preliminary facility list
Site assessment summaries
Facility risk ranking including a summary of physical security effectiveness,
probability of attack, and consequence of contamination criteria
Final facility list with a description of recommended improvements for each
facility
Communication architecture to transmit data from monitoring locations to an
operations center
Consumer
Complaint
Surveillance
The utility-specific model of the consumer complaint surveillance component
design
An assessment of the existing consumer complaint management system
An approach for enhancing the consumer complaint management system into a
consumer complaint surveillance system
IT architecture used to manage and store water quality complaint and related
data
Event detection software deployed to detect anomalies
Public Health
Surveillance
Identification of local public health partners
Identification and assessment of existing surveillance capability relative to
contamination warning system objectives
Improvements or additions to existing surveillance capabilities
Development of a framework for communication and notification
Consequence
Management
Objectives for consequence management plan
Utility self-assessment of existing plans and response capabilities
Identification and assessment of response partner capabilities
Level of integration with other response plans
Framework for development of consequence management plan that allows for
seamless transition from routine operations and initial trigger validation to
consequence management actions
May 2007
VII
-------
Planning for WS-CWS Deployment
Water Security Initiative:
Interim Guidance on Planning for Contamination Warning System
Deployment
ABBREVIATIONS AND ACRONYMS I
ACKNOWLEDGEMENTS II
EXECUTIVE SUMMARY Ill
SECTION 1.0: INTRODUCTION 1
1.1 CONTAMINATION WARNING SYSTEMS - AN OVERVIEW 2
1.2 DOCUMENT OVERVIEW 7
SECTION 2.0: PROJECT PLANNING AND MANAGEMENT 8
2.1 APPLICATION OF SYSTEM ENGINEERING PRINCIPLES 8
2.1.1 Development and Management of Work Plan and Schedule 8
2.1.2 Integrated Concept of Operations 9
2.1.3 IT System Engineering 10
2.2 UTILITY STAFFING 11
2.2.1 Project Management Team 11
2.2.2 Utility Staff 12
2.3 LOCAL PARTNERS 13
2.3.1 Identifying and Engaging Partners 14
2.3.2 Considerations for Formal Agreements with Local Partners 16
2.4 COSTS 17
SECTION 3.0: DESIGN AND IMPLEMENTATION FRAMEWORK 18
3.1 PLANNING AND PRE-DESIGN 18
3.1.1 Building the Team 18
3.1.2 Defining the Utility-Specific Design Basis and Design Objectives 18
3.1.3 Preliminary Assessment and Gap Analysis 20
3.2 DESIGN 20
3.2.1 Conceptualize System 20
3.2.2 Work Plan for Implementation 21
3.3 IMPLEMENTATION 21
3.4 PRELIMINARY TESTING 21
3.4.1 Baseline Operation 22
3.4.2 Finalization of Concept of Operations and Consequence Management Plan 22
3.5 OPERATION AND MAINTENANCE 23
3.5.1 Operation 23
3.5.2 Maintenance 23
3.6 EVALUATION AND REFINEMENT 23
3.6.1 Evaluation 24
3.6.2 Refinement 24
SECTION 4.0: ONLINE WATER QUALITY MONITORING 25
4.1 MONITORING NETWORK DESIGN 26
4.1.1 Pre-Design 27
4.1.2 Design and Implementation Approach 29
4.1.3 Available Tools and Resources 30
4.2 MONITORING STATION DESIGN AND INSTALLATION 31
4.2.1 Pre-Design 31
4.2.2 Design and Implementation Approach 33
4.2.3 Available Tools and Resources 38
4.3 COMMUNICATIONS ARCHITECTURE 38
4.3.1 Pre-Design 39
May 2007 viii
-------
Planning for WS-CWS Deployment
4.3.2 Design and Implementation Approach 39
4.3.3 Available Tools and Resources 40
4.4 DATA MANAGEMENT AND IT ARCHITECTURE 41
4.4.1 Pre-Design 41
4.4.2 Design and Implementation Approach 42
4.4.3 Available Tools and Resources 42
4.5 WATER QUALITY EVENT DETECTION 43
4.5.1 Planning 43
4.5.2 Implementation Approach 46
4.5.3 Available Tools and Resources 47
4.6 STAFFING AND COST CONSIDERATIONS 48
4.6.1 Staffing 48
4.6.2 Cost Considerations 49
SECTION 5.0: SAMPLING AND ANALYSIS 51
5.1 LABORATORY CAPABILITY AND CAPACITY 52
5.1.1 Pre-design 52
5.1.2 Design and Implementation Approach 55
5.1.3 Available Tools and Resources 56
5.2 SAMPLING AND ANALYSIS 56
5.2.7 Pre-design 57
5.2.2 Design and Implementation Approach 58
5.2.3 Available Tools and Resources 61
5.3 SITE CHARACTERIZATION AND FIELD SCREENING 61
5.3.7 Pre-design 61
5.3.2 Design and Implementation Approach 63
5.3.3 Available Tools and Resources 65
5.4 STAFFING AND COST CONSIDERATIONS 65
5.4.1 Staffing 65
5.4.2 Cost Considerations 66
SECTION 6.0: ENHANCED SECURITY MONITORING 67
6.1 PRE-DESIGN 69
6.2 DESIGN AND IMPLEMENTATION APPROACH 72
6.3 AVAILABLE TOOLS AND RESOURCES 75
6.4 STAFFING AND COST CONSIDERATIONS 76
6.4.1 Staffing 76
6.4.2 Cost Considerations 77
SECTION 7.0: CONSUMER COMPLAINT SURVEILLANCE 78
7.1 PRE-DESIGN 80
7.2 DESIGN AND IMPLEMENTATION APPROACH 81
7.3 AVAILABLE TOOLS AND RESOURCES 82
7.4 STAFFING AND COST CONSIDERATIONS 83
7.4.1 Staffing 83
7.4.2 Cost Considerations 84
SECTION 8.0: PUBLIC HEALTH SURVEILLANCE 85
8.1 PRE-DESIGN 86
8.2 DESIGN AND IMPLEMENTATION APPROACH 89
8.3 AVAILABLE TOOLS AND RESOURCES 90
8.4 STAFFING AND COST CONSIDERATIONS 91
8.4.1 Staffing 91
8.4.2 Cost Considerations 92
SECTION 9.0: CONSEQUENCE MANAGEMENT 94
9.1 PRE-DESIGN 94
9.2 DESIGN AND IMPLEMENTATION APPROACH 97
9.3 AVAILABLE TOOLS AND RESOURCES 98
May 2007 ix
-------
Planning for WS-CWS Deployment
9.4 STAFFING AND COST CONSIDERATIONS 100
9.4.1 Staffing 100
9.4.2 Cost Considerations 100
SECTION 10.0: REFERENCES 102
APPENDIX A: GLOSSARY 104
APPENDIX B: INFORMATION SECURITY CONSIDERATIONS 106
B.I ASSESSMENT OF MATERIAL SENSITIVITY, ACCESS, AND LAWS 106
B.2 SENSITIVE MATERIALS TRACKING AND STORAGE 107
B.3 SENSITIVE MATERIALS HANDLING PROTOCOL AND PROCEDURES 107
B.4 STAFF CERTIFICATION AND BACKGROUND CHECKS 108
B.5 COORDINATION AND COOPERATION WITH PARTNERS, CONTRACTORS AND SUBCONTRACTORS 109
List of Tables
TABLE ES-l. OVERVIEW OF DESIGN AND IMPLEMENTATION FRAMEWORK v
TABLE ES-2. SUMMARY OF DESIGN DECISIONS TO SUPPORT PLANNING FOR CONTAMINATION WARNING SYSTEM
IMPLEMENTATION vi
TABLE 1-1. CONTAMINANT DETECTION CLASSES AND POTENTIAL MEANS OF DETECTION 5
TABLE 1-2. DESIGN BASIS SUMMARY BY CONTAMINATION WARNING SYSTEM COMPONENT 6
TABLE 2-1. SUMMARY OF POTENTIAL CONTAMINATION WARNING SYSTEM PARTNERS 14
TABLE 3-1. OVERVIEW OF DESIGN AND IMPLEMENTATION FRAMEWORK 18
TABLE 3-2. DESIGN BASIS CONSIDERATIONS 19
TABLE 4-1. DESIGN BASIS CONSIDERATIONS FOR ONLINE WATER QUALITY MONITORING 25
TABLE 4-2. IMPACT OF CONTAMINANT DETECTION CLASSES ON WATER QUALITY PARAMETERS 31
TABLE 4-3. STANDARD MEASURES FOR EVALUATING EDS TOOL PERFORMANCE 44
TABLE 4-4. SAMPLE MEASURES FOR EVALUATING EDS SOFTWARE 45
TABLE 4-5. ONLINE WATER QUALITY MONITORING STAFFING CONSIDERATIONS 48
TABLE 5-1. DESIGN BASIS CONSIDERATIONS FOR SAMPLING AND ANALYSIS 51
TABLE 5-2. CONSIDERATIONS FOR ANALYTICAL APPROACH TO ESTABLISHING SAMPLING AND ANALYSIS
CAPABILITIES BY CONTAMINANT CLASS 52
TABLE 5-3. EXAMPLES OF BASELINE DATA SOURCES 59
TABLE 5-4. CONSIDERATIONS FOR CONTAMINANT COVERAGE FOR FIELD SCREENING 63
TABLE 5-5. SAMPLING AND ANALYSIS STAFFING CONSIDERATIONS 65
TABLE 6-1. DESIGN BASIS CONSIDERATIONS FOR ENHANCED SECURITY MONITORING AT SELECTED SITES 67
TABLE 6-2. EXAMPLE IMPROVEMENTS BY WATER UTILITY FACILITY TYPE 68
TABLE 6-3. ENHANCED SECURITY MONITORING STAFFING CONSIDERATIONS 76
TABLE 7-1. DESIGN BASIS CONSIDERATIONS FOR CONSUMER COMPLAINT SURVEILLANCE 78
TABLE 7-2. CONSUMER COMPLAINT SURVEILLANCE STAFFING CONSIDERATIONS 83
TABLE 8-1. DESIGN BASIS CONSIDERATIONS FOR PUBLIC HEALTH SURVEILLANCE 86
TABLE 8-2. PUBLIC HEALTH SURVEILLANCE STAFFING CONSIDERATIONS 92
TABLE 9-1. OVERVIEW OF RESPONSE PARTNER ROLES AND RESPONSIBILITIES FOR CONSEQUENCE MANAGEMENT ..96
List of Figures
FIGURE ES-l. POTENTIAL CONTAMINATION WARNING SYSTEM PARTNERS vi
FIGURE 1-1. OVERVIEW OF EPA's WATER SECURITY INITIATIVE 1
FIGURE 1-2. ARCHITECTURE OF THE WATER SECURITY CONTAMINATION WARNING SYSTEM 3
FIGURE 2-1. POTENTIAL CONTAMINATION WARNING SYSTEM PARTNERS 14
FIGURE 2-2. RECOMMENDED STRATEGY FOR ENGAGING CONTAMINATION WARNING SYSTEM PARTNERS 16
FIGURE 4-1. EXAMPLE MONITORING STATION TRADEOFF CURVE 28
FIGURE 4-2. EXAMPLE WATER QUALITY MONITORING STATION DESIGN USED IN THE INITIAL PILOT 35
FIGURE 4-3. EXAMPLE ROC CURVE 45
FIGURE 7-1. THE RECOMMENDED FILTER, FUNNEL, AND Focus APPROACH TO CUSTOMER FEEDBACK DATA
OPTIMIZATION FOR UTILITY-MANAGED CONSUMER CALLS 80
FIGURE 8-1 PUBLIC HEALTH DATA SOURCES AND DESIGN OBJECTIVES 87
May 2007
-------
Planning for WS-CWS Deployment
Section 1.0: Introduction
This document presents a basic framework to assist drinking water utilities with planning for
contamination warning system deployment based on the model developed under U.S. Environmental
Protection Agency's (EPA) Water Security initiative (formerly known as WaterSentinel).
Initiated in response to Homeland Security Presidential Directive 9, the overall goal of the Water Security
initiative is to design and deploy contamination warning systems for drinking water utilities through a
phased approach that includes conceptual design, implementation at an initial pilot utility, expansion to
additional pilot utilities, and ultimately development of guidance and tools to support implementation at
drinking water utilities across the nation. Figure 1-1 summarizes this process.
Phase
Approach
Scope
Design
Specificity
Funding
DESIGN
System Architecture
DEMONSTRATE
Initial Pilot
Additional Pilots
EXPAND
Voluntary National Adoption
^^~~[S Applied by^T/
Apply to single Evaluate multiple Evaluate
Conceptual , N Pilot utilitV ^^ U"eS ^^ C°nvert'°
design ^^ ^ ["] ^ H guidance for
AX . r/ AX r/
V 7 Refine !__£ \J Refine 1_J>
\/ anc| ^ and
enhance enhance
Not
applicable
Low
ll
High-
Applies to pilot utility only
& ml
ia s
iL^ A
High-
Appliesto each pilot
EPA Funds
iW
Medium -
Applies to range of utilities
Utility Funds
Figure 1-1. Overview of EPA's Water Security Initiative
Monitoring the distribution system is the primary focus of contamination warning systems. Through the
assessment of vulnerabilities to drinking water systems, water security experts have identified the
distribution system as one of the most vulnerable components in a drinking water utility, with respect to
contamination. Furthermore, intentional contamination, or even the threat of contamination can have
significant impacts.
Drinking water utilities occasionally receive threats or indications of possible contamination. These
contamination threat warnings can be a direct threat or an unusual observation or discovery that indicates
the potential for contamination and initiates actions to investigate and potentially respond. However,
these threat warnings are not standardized and are difficult to corroborate in the absence of an integrated
monitoring and surveillance system and close coordination with response partners including, but not
limited to public health, emergency responders, and law enforcement.
Deployment of contamination warning systems for drinking water distribution systems provides a
mechanism by which drinking water utilities can detect and respond to contamination threats and
incidents. A contamination warning system is a proactive approach to managing threat warnings that uses
advanced monitoring technologies/strategies and enhanced surveillance activities to collect, integrate,
analyze, and communicate information to provide a timely warning of potential water contamination
incidents and initiate response actions to minimize public health and economic impacts.
May 2007
-------
Planning for WS-CWS Deployment
In addition, deployment of a contamination warning system provides the opportunity for dual-use
applications beyond security that could help to promote sustainability of the system by optimizing utility
operations. Drinking water distribution systems may be accidentally contaminated through cross-
connections with non-potable water, permeation of contaminated water through leaking pipes in areas of
the distribution system subject to low pressures, or chemical reactions or microbial growth within the
distribution system pipes. Such unintentional events that result in degradation to distributed water quality
may occur with some regularity.
In 2005, EPA documented the conceptual design for contamination warning systems in WaterSentinel
System Architecture (USEPA, 2005a) and began implementation of the first WS contamination warning
system pilot in partnership with the City of Cincinnati at the Greater Cincinnati Water Works (GCWW).
Section 1.1 provides an overview of contamination warning systems and a summary of the design basis
and Section 1.2 provides an overview of how to use this document to support design and implementation
of contamination warning systems based on EPA's approach and lessons learned from the initial pilot.
1.1 Contamination Warning Systems - An Overview
A contamination warning system is not merely a collection of monitors and equipment placed throughout
a water system to alert of intrusion or contamination. Fundamentally, it is an exercise in information
acquisition and management. Different information streams are captured, managed, analyzed, and
interpreted to recognize potential contamination incidents in time to respond effectively. These data
sources, when used concurrently, should support and augment each other such that the chances of
detecting a contamination incident are better than using any one information source on its own. While the
contamination warning system should be designed by the utility, some data sources may be outside of the
utility; thus, cooperation with partners is an integral part to the success of a contamination warning
system. A complete contamination warning system consists of the following monitoring and surveillance
components:
Online water quality monitoring comprises stations located throughout the distribution system
that measure chlorine, total organic carbon, conductivity, and other parameters. Software
analyzes the monitoring data to establish a water quality base state. Possible contamination is
indicated when a significant, unexplained deviation from the base state occurs.
Sampling and analysis is the collection of distribution system samples that are analyzed for
various contaminant classes as well as specific contaminants. Sampling is both routine to
establish a baseline and triggered to respond to an indication of possible contamination from
another component. Analyses are conducted for chemicals, radionuclides, pathogens, and toxins
using a laboratory network.
Enhanced security monitoring includes the equipment and procedures that detect and respond
to security breaches at distribution system facilities. Security equipment may include cameras,
motion activated lighting, door contact alarms, ladder and window motion detectors, area motion
detectors, and access hatch contact alarms.
Consumer complaint surveillance enhances the collection and automates the analysis of calls by
consumers for water quality problems indicative of possible contamination. Consumers may
detect contaminants with characteristics that impart an odor, taste, or visual change to the
drinking water.
Public health surveillance involves the analysis of health-related data to identify disease events
that may stem from drinking water contamination. Public health data may include over-the-
counter (OTC) drug sales, hospital admission reports, infectious disease surveillance, emergency
medical service (EMS) reports, 911 calls, and poison control center calls.
As illustrated in Figure 1-2, consequence management is another key aspect of the contamination
warning system architecture. Consequence management governs response actions when contamination is
May 2007
-------
Planning for WS-CWS Deployment
determined to be possible and includes activities that involve drinking water utilities as well as external
partners.
Figure 1-2. Architecture of the Water Security Contamination Warning System
The basic process or conceptual model for contamination warning system operation is described as
follows, moving from left to right in the diagram.
Monitoring and Surveillance. As previously discussed, integration of information from a
variety of data sources internal and external to the drinking water utility is a critical aspect of
contamination warning system monitoring and surveillance activities. While the specific types of
information streams may vary, the basic components of online water quality monitoring,
sampling and analysis, enhanced security monitoring, consumer complaint surveillance, and
public health surveillance are necessary to meet the design objectives as discussed later in this
section. Monitoring and surveillance of these components and information streams occurs on a
routine basis, in near-real time until an anomaly or deviation from the baseline or base state is
detected.
Event detection and Possible Determination. Event detection is the process or mechanism by
which an anomaly or deviation from the baseline or base state is detected. This detection is
referred to as a trigger. How event detection is implemented and the tools that are utilized may
vary significantly from component to component and can include sophisticated algorithms,
simple business logic, etc. This process should be automated to the extent possible. Another
aspect of the contamination warning system that is tightly coupled with event detection is the
initial validation of triggers to assess possibility of drinking water contamination. As discussed in
greater detail throughout this document, many of the components provide non-specific indicators
of contamination and thus, when a trigger occurs, validation is necessary prior to determining if
contamination is "possible." If trigger validation indicates that contamination is "possible," then
the credibility determination step is initiated; otherwise, the contamination warning system
component returns to routine monitoring and surveillance.
Credible Determination. Credibility determination is a transition from routine operation to
consequence management. Credibility determination procedures are performed using information
from all contamination warning system components as well as external resources when available
and relevant. Through the credibility determination process, some preliminary response actions
may be initiated to limit or minimize impacts of suspected contamination. If contamination is
determined to be credible, additional confirmatory and response actions are initiated. If
May 2007 3
-------
Planning for WS-CWS Deployment
contamination can be ruled out based on additional information gathered through the credibility
determination process, the system returns to routine monitoring and surveillance activities.
Confirmed Determination. In this stage of consequence management, additional information is
gathered and assessed to confirm drinking water contamination. Response actions initiated
during credible determination are expanded and additional response activities may be
implemented.
Remediation and Recovery. Once contamination has been confirmed, and the immediate crisis
has been addressed through response, remediation and recovery actions defined in the
consequence management plan are performed to restore the system to normal operations.
The architecture presented in Figure 1-2 was derived initially through the conceptual design of the
contamination warning system and was refined based on lessons learned from design and implementation
of the initial Water Security initiative pilot.
The consequences associated with a particular contamination scenario are largely a function of the
contaminant type and concentration, the location of contaminant introduction, and the relative timing of
exposure, onset of symptoms, detection, and response. These results lead to a design basis for a
contamination warning system that considered four primary attributes: contaminant coverage, spatial
coverage, timing of detection, and reliability. Therefore, defining a design basis by the following
performance objectives should guide a utility through the design and implementation of an effective
contamination warning system. The system, as a whole, should be able to meet the following design
objectives:
Detection of a broad spectrum of contaminant classes. There are a large number of
contaminants that could cause serious harm if introduced into the drinking water distribution
system. As part of the contamination warning system design basis, contaminants were prioritized
and then binned into 12 detection classes (see additional information below). Use of the detection
classes to inform design provides more robust detection capability than analyzing for only a select
number of contaminants and also avoids the challenge associated with designing a system around
a list containing hundreds of potential contaminants.
Achieve spatial coverage of the entire distribution system. Spatial coverage can be considered
hydraulically and geographically. For components such as online water quality monitoring,
spatial coverage is a function of the number of online water quality monitors and their placement
throughout the distribution system. For other components such as consumer complaints or public
health surveillance, spatial coverage varies geographically based on population density,
population demographics (industrial vs. residential), and/or types of surveillance systems and
tools used within a jurisdiction.
Detect contamination in sufficient time for effective response. There are three periods
associated with the evaluation of the timeline of a contamination incident, including (1) the time
during which consequences (exposures, illnesses, fatalities, pipe contamination, etc.) are
experienced in the population, (2) the time of initial detection, and (3) the time of response
actions. A key aspect of a contamination warning system is to provide initial detection in a
timeframe that allows for the implementation of response actions that result in a significant
reduction in consequences.
Reliably indicate a contamination incident with a minimum number of false-positives.
Reliability can be considered from two perspectives. The first is operation, that is, factors such as
contamination warning system component capabilities and necessary maintenance. The second is
performance, defined as the ability of the system to provide information that leads decision
makers to successfully infer that contamination has or has not occurred.
Provide a sustainable architecture to monitor distribution system water quality. The
integration of multiple monitoring and surveillance strategies already in use at the utility and
public health department should improve acceptance of the system, and thus long-term
sustainability. A contamination warning system should be designed as a dual-use application to
May 2007
-------
Planning for WS-CWS Deployment
benefit the utility in day-to-day operations while also providing the capability to detect intentional
or accidental contamination incidents.
The first design objective described above introduced the concept of contaminant classes. Table 1-1
presents a summary of contaminant detection classes developed during the conceptual design phase of
EPA's Water Security initiative (USEPA, 2005a, USEPA, 2005b). This table shows the potential means
of detection for each contaminant class by three of the components: water quality monitoring, consumer
complaint surveillance, and public health surveillance (considered as two independent data streams).
Enhanced security monitoring could potentially detect any contamination event, while sampling and
analysis can detect contaminants within the group of target analytes for the methods employed. Thus,
collectively, the five components provide comprehensive contaminant coverage, and provide a means of
confirming a possible contamination incident through an independent data stream.
Table 1-1. Contaminant Detection Classes and Potential Means of Detection
Class
1
2
3
4
5
6
7
8
9
10
11
12
Description
Petroleum products
Pesticides (with odor or taste)
Inorganic compounds
Metals
Pesticides (odorless)
Chemical warfare agents
Radionuclides
Bacterial toxins
Plant toxins
Pathogens causing diseases with unique symptoms
Pathogens causing diseases with common
symptoms
Persistent chlorinated organic compounds
Water
Quality
X
X
X
X
X
X
X
X
X
X
X
X
Consumer
Complaints
X
X
X
X
X
911 calls
/EMS
X
X
X
X
X
X
Syndromic
Surveillance1
X
X
X
X
Collecting and analyzing nontraditional data to detect a change or trend in the health of a population using
categories of disease rather than formal diagnosis.
In designing a contamination warning system, each of the design objectives should be considered for the
system as a whole as well as for each component. Table 1-2 summarizes the design objectives as they
relate to each of the contamination warning system components. These design objectives are presented
again in Sections 4 through 8 along with a summary of design and implementation considerations for
each of the monitoring and surveillance components.
May 2007
-------
Planning for WS-CWS Deployment
Table 1-2. Design Basis Summary by Contamination Warning System Component
WS-CWS
Component
Online Water
Quality
Monitoring
Sampling and
Analysis
Enhanced
Security
Monitoring
Consumer
Complaint
Surveillance
Public Health
Surveillance
Capability
Can indicate the presence
of a contaminant that
significantly affects one or
more monitored
parameters.
Can positively identify the
presence of any
contaminant in the suite of
target analytes and above
a well-defined minimum
reporting level.
Can detect an intrusion
that may have provided the
opportunity for introduction
of any contaminant.
Can indicate the presence
of a contaminant that
significantly affects one or
more aesthetic qualities of
water.
Can detect the presence of
a symptom or illness in a
population which may be
the result of the presence
of a disease causing
agent. May be able to
identify the contaminant
through clinical diagnosis/
testing.
Contaminant
Coverage
High detection
potential for
classes 1, 2, 3, 5,
8, 9, 10, 11 and 12.
Moderate detection
potential for
classes 4, 6, and 7.
High detection
potential for
classes 1, 2, 3, 4,
7, and 12;
Moderate detection
potential for
classes 5, 6, 8, 9,
10, 11.
Covers all
contaminant
classes.
High detection
potential for
classes 1 and 2.
Moderate detection
potential for
classes 3, 4, and 5.
Covers
contaminant
classes 2 through
11; detection
potential varies with
type of
surveillance.
Spatial Coverage
Function of location,
number, and density
of monitoring
stations.
Function of location,
number, and density
of sampling stations,
as well as sample
type (composite vs.
grab).
Limited to those
elements of
infrastructure for
which physical
security can be
monitored.
Entire service area
for contaminants with
detectable taste,
color, or odor
characteristics.
Comprehensive
coverage of a
particular city or
county, which may
include all, or a large
portion of, the utility
service area.
Timeliness
Function of hydraulic
travel time from the
point of contaminant
introduction to the
sensor, and the
concentration of the
contaminant.
Function of sampling &
analysis frequency and
the total time to process
the sample and analyze
the results.
Function of the type of
security monitoring
system and the time to
evaluate a security
breach.
Function of the time
from exposures to
consumer reporting,
complaint
categorization,
assessment and
investigation.
Function of the time
from the initial
exposures, the onset of
symptoms, and the
point at which public
health officials
recognize the incident
as a potential water-
borne illness.
Reliability
Rate of false positive / negative
results in this application is largely
unknown at this time. May be
addressed through event detection
systems and consequence
management.
Function of the reliability of
sampling and analysis methods
(high for established techniques).
Baseline needed for reliable
interpretation of results.
Can be a reliable means of
identifying an intrusion, especially
when these breaches may involve
contamination, such as in storage
tanks and reservoirs.
A potentially reliable indicator for
contaminants with detectable
characteristics if a robust
complaint reporting and tracking
system is in place.
May be a reliable means of
identifying the incidence of illness
in a population, but timing of
communication between drinking
water and public health officials
should be optimized such that
appropriate response, actions
could be implemented in time to
reduce consequences.
Sustainability
Provides utility with a better
understanding of water
quality variability
throughout distribution
system and provides an
opportunity to optimize
distribution system
operation.
Provides utility with an
opportunity to exercise
sampling and laboratory
protocols and may; provide
information about
previously unknown
contaminants that occur in
the system.
Provides utility with
increased physical
infrastructure protection
and awareness. Reduces
the occurrence of nuisance
tampering.
Provides utility an
opportunity to manage
consumer information more
effectively and can serve
as a tool for enhanced
consumer confidence.
Provides an opportunity for
utility and local health
May 2007
-------
Planning for WS-CWS Deployment
1.2 Document Overview
This document describes planning considerations for contamination warning system deployment based on
the approach deployed at, and lessons learned from, the initial Water Security initiative pilot at GCWW.
The primary focus of the document is on planning and pre-design. Throughout the document available
resources and tools are highlighted to facilitate contamination warning system design and
implementation. The lists of resources are not intended to be exhaustive, but rather a starting point to
facilitate the planning process. Sections included in the document are as follows:
Section 2: Project Planning and Management. This section introduces the concept of system
engineering for contamination warning systems and discusses critical aspects of project planning
for design and implementation. Although the drinking water utility is the primary organization
involved with design and implementation, other key partners have a significant role as well and
should be included in the process. Subsections focus on system engineering, utility staffing,
coordination with local partners, and costing considerations.
Section 3: Design and Implementation Framework. This section describes the framework for
design and implementation of a contamination warning system including planning or pre-design,
design, implementation, preliminary testing (start-up and baselining of components), operation
and maintenance, and evaluation and refinement.
Section 4: Online Water Quality Monitoring. This section provides the design basis for online
water quality monitoring, identifies critical design decisions and describes design and
implementation aspects relative to monitoring network design (placement of monitoring stations),
monitoring station design, communications and information technology architecture, and water
quality event detection.
Section 5: Sampling and Analysis. This section describes the design basis for sampling and
analysis, identifies critical design decisions and describes design and implementation aspects
relative to laboratory capability and capacity, sampling and analysis (baseline, maintenance,
triggered), and. field screening and site characterization.
Section 6: Enhanced Security Monitoring. This section describes the design basis for enhanced
security monitoring and a systematic methodology for selecting sites for security improvements
and designing those enhancements.
Section 7: Consumer Complaint Surveillance. This section describes an approach to managing
customer calls and customer information. The document focuses on assessing existing call
management systems and protocols based on contamination warning system design objectives
and describes considerations for optimized call management, tracking, and analysis.
Section 8: Public Health Surveillance. This section describes the design basis for public health
surveillance and design considerations to assist with planning for implementation. The public
health surveillance component of a contamination warning system relies heavily on relationships
and partnerships with local health departments. In addition to describing the design basis for the
public health surveillance component, this section provides background information for drinking
water utilities on the types of public health surveillance that may be implemented by local or state
health departments. Considerations for enhancing or expanding on these existing systems or
approaches to include more timely data streams such as 911 calls or emergency medical service
(EMS) events for fast-acting contaminants are also addressed.
Section 9: Consequence Management. This section describes planning considerations for
development of a consequence management plan for the drinking water utility, including
identification and coordination with local response partners and organizations. Types of plans
and information that can be leveraged to support development of this plan - both internal and
external to the utility - are discussed. In addition, existing training programs and resources are
identified.
In addition to the sections described above, this document also includes a list of acronyms, references, a
glossary (Appendix A), and information security considerations (Appendix B).
May 2007 7
-------
Planning for WS-CWS Deployment
Section 2.0: Project Planning and Management
This section discusses critical aspects of project planning and management for deployment of a
contamination warning system. Although the drinking water utility is the primary organization involved
with deployment, other key partners also have significant roles and should be included in the process. As
discussed throughout this section, deployment of a contamination warning system is a significant
undertaking that impacts most departments or divisions within the utility at some stage. A commitment at
all levels of the organization, from senior managers and supervisors to staff supporting routine operations
within the utility or in the field, is essential.
2.1 Application of System Engineering Principles
A contamination warning system is, by design, a systematic approach to monitoring and surveillance for
the timely detection of drinking water contamination. As such, deployment of a contamination warning
system relies on system engineering principles to support coordination of technical and management
activities. System engineering is an interdisciplinary approach to design and implementation of systems.
Through system engineering, disciplines and specialty groups are integrated in a team effort forming a
structured development process that proceeds from design to implementation to operation. From the
beginning of the project, system engineering principles are critical to successful planning and
implementation. The primary application of system engineering for a contamination warning system is to
ensure that the system - monitoring and surveillance components and consequence management as
discussed in detail in Section 4 to Section 9 - functions as an integrated whole.
System engineering involves the integration of monitoring and surveillance components and consequence
management. Throughout planning and implementation, many activities can and should occur in parallel.
However, it is necessary to reconcile activities and key stages of deployment to optimize the function of
the system. For example, once a preliminary concept of operations (as discussed in Section 2.1.2) has
been defined for the system, component-specific designs and consequence management planning can
occur in parallel. However, it is important to reconcile the consequence management plan with the
various monitoring and surveillance components as they are designed and implemented to ensure
seamless transition from routine operations to consequence management.
As part of system engineering, coordination strategies should be applied throughout the deployment
process to ensure that there is a consistent vision and understanding of goals and objectives. To facilitate
coordination, meetings involving representatives from each of the components should be held early in the
planning process. It may be helpful to establish a core team with representatives for each of the
component teams or supporting divisions involved in design, implementation, or operation of the
contamination warning system. Regular meetings serve as a forum for component teams to share their
progress with the project management team, and as an opportunity to identify synergies among the
enhancements. Through discussions, critical cross-cutting issues may be identified and resolved
consistently across the project, making for more efficient problem solving and refinement. In addition,
these meetings could highlight dual-use applications of the system, reinforcing the value and improving
sustainability. As utilities work through system deployment, it may be necessary to prioritize certain
activities across components of the system based on criticality, resources, schedule, or other issues.
Addressing these issues through these routine coordination meetings may help to facilitate the
prioritization process.
2.1.1 Development and Management of Work Plan and Schedule
The development of detailed work plans for monitoring and surveillance components as well as
consequence management is critical. These work plans should outline the enhancements necessary to
progress from the existing state to the desired state and should be useful in coordinating timely
implementation activities. The work plan builds on the current state of the utility by identifying specific
May 2007 8
-------
Planning for WS-CWS Deployment
equipment, computer hardware or software, training and process modifications that are necessary to
achieve specified goals. Each step in the work plan should contain detailed action items and a defined
schedule to meet the goals specified by the design decisions. Coordination across components to
maximize efficiency is critical in work plan development to ensure that all necessary activities are
planned for, while eliminating redundancies.
Although detailed schedules and work plans should be developed to support component activities, it is
important to track high-level milestones across all components to ensure that activities are managed
efficiently and effectively and that there is communication and coordination on overarching or cross-
cutting issues. To facilitate this high-level planning, development of an integrated concept of operations
can serve as a project management tool as well as a definition of system operations.
2.1.2 Integrated Concept of Operations
The concept of operations is the description of the routine operation and initial trigger validation for each
component of the contamination warning system. While the terminology and approach for this
documentation may vary, it is important that the operational concepts be clearly documented and that
utility staff and local partners understand their roles and responsibilities in operating the system. The
concept of operations should be developed at the component level to present the day-to-day functioning
of components and management of information to support trigger validation and initial response actions
(or, in the case of consequence management, integrating information from multiple components and
initiating a response). However, the development of the component concept of operations should be
coordinated to ensure that information from multiple components can be integrated to better inform
response decisions. It is also important to integrate day-to-day operation of the contamination warning
system into routine job duties. Guidance on development of a concept of operations will be the focus of
future EPA efforts.
While the concept of operations for a contamination warning system is intended to guide day-to-day
operations and trigger validation, it also has an important design function at the system level. A
preliminary concept of operation for each component can be an effective means of determining utility-
derived requirements for the design and implementation of that component. An integrated concept of
operations encompasses the component-specific concept of operations by explaining how the components
inter-relate, how their data streams are combined and how the system as a whole meets the design basis.
It is recommended that a preliminary concept of operations be developed during the planning or pre-
design stage.
To develop a preliminary concept of operations, utilities should define design objectives as discussed in
Section 3 and leverage the component-specific concepts presented in Section 4 - Section 9. At the
conclusion of planning and pre-design, utilities should be able to develop a high-level concept of
operations that includes critical information point such as the users and key decision makers involved in
routine operation of a given component. As part of the system engineering approach to implementation, it
may be possible to develop and define teams and roles and responsibilities for utility staff across all
components, thereby presenting a cohesive, system-wide picture of what key utility personnel should do
during the day-to-day operation of the contamination warning system. An important feature of this
integrated view is its ability to illustrate how contamination warning system activities relate to normal job
functions and provide benefits to the utility and staff beyond contamination warning. In addition, a
preliminary timeline for initial trigger validation should be part of the preliminary concept of operations,
and is an important consideration in early stages of consequence management plan development. With
the initial draft of the concept of operations, these timelines should be projections based on desired goals
for trigger validation and decision making. It will likely be necessary to refine these projections based on
the design that is ultimately implemented. These timelines may also serve as a metric for evaluation of
the system and individual components as part of evaluation and refinement. Development of a
May 2007
-------
Planning for WS-CWS Deployment
preliminary concept of operations will also serve to highlight the significance of information technology
(IT) systems, data flows, and IT-based user requirements in contamination warning system deployment.
2.1.3 IT System Engineering
A contamination warning system is not merely a collection of monitors and equipment placed throughout
a water system to alert of intrusion or contamination. Fundamentally, it is an exercise in information
acquisition and management. Different information streams should be captured, managed, analyzed, and
interpreted in time to recognize potential contamination incidents and respond effectively. Information
from several different databases and information systems at a water utility should be integrated to enable
event detection, trigger validation, credibility determination, and management of response actions in a
timely manner. In addition, data and information from local partners including, but not limited to, fire
departments, public health, and law enforcement should be integrated. Thus, the success of the
contamination warning system implemented at a drinking water utility will depend heavily on effective
data and information management.
In planning for contamination warning system implementation and development of a preliminary concept
of operations, development of an inventory of the IT systems used in the operation of the components and
day-to-day operations may serve as a valuable tool. It is also useful to consider information flows and
how information is collected and managed throughout existing systems. An information flow diagram
describes how data and information flows between system elements and users during routine operations
of a contamination warning system component, including the information systems, databases, and user
interfaces used during activities related to routine monitoring and surveillance, event detection, trigger
validation, and credibility determination for a component. Development of a preliminary information
flow diagram could serve as a useful tool for facilitating design and implementation discussions and
decisions. Although each component data stream is unique, a common set of system elements (data
sources, event detection, alarm or trigger notification, and data storage) can be defined.
An assessment of the utility's existing capabilities relative to the design objectives of the contamination
warning system and the attributes of the specific component of the system is essential for successful
implementation. From an IT system engineering perspective, primary objectives for design and
implementation include the following:
Ability to leverage and integrate with existing utility IT systems and policies
Mechanisms for timely integration and analysis of contamination warning system data and the
ability to make data available to support timely decision-making
Approaches and mechanisms that can be readily adapted to meet changing needs and priorities
The basic objectives described above can be used to define the component-specific requirements for data
management. Other considerations include the following:
Electronic data management for all components. All information collected as part of the
contamination warning system should be managed in a database. For each component,
information should be tracked as the system transitions from routine operation to event detection,
credibility determination, consequence management, and finally return to normal operations. In
order to avoid duplication of effort, information should be accessible across multiple databases
and systems.
Data storage. The data collected should be stored in a reliable data store that has sufficient
capacity and performance to support routine operations, contamination situations, and system
evaluation. Historical data should be maintained at appropriate locations to enable engineers and
scientists to study past normal and event situations to evaluate and optimize system performance.
Automated and integrated data analysis. As data are captured in electronic format in a
consistent environment, algorithms operate on the data to integrate information and facilitate data
analysis. Automated anomaly detection algorithms indicate when the data may be indicative of a
contamination incident, signaling the need for human involvement in the assessment process.
May 2007 10
-------
Planning for WS-CWS Deployment
The type and level of sophistication of algorithms will likely vary by component, from
sophisticated algorithms for the analysis of online water quality and public health surveillance
data to more simplistic algorithms for consumer complaints. The use of tools for spatial and
temporal analysis of the indicators, such as Geographic Information Systems (GIS), should also
be considered.
User interface for operation and analysis. Although some contamination warning system
operations are automated, human operators and analysts will assess situations and make many of
the decisions necessary to manage a potential contamination incident. Thus, there is a need to
develop or utilize existing user interfaces to support data analysis and decision making. If a
utility's existing GIS is sufficiently robust and integrated, it can serve as an effective means for
display of information from any or all components of the contamination warning system.
Data exchange capability for transmission of information within the utility and between the
utility and contamination warning system partners. The data collected at the utility may need
to be shared with other local government agencies such as public health departments.
Availability and redundancy of systems. Information systems should be designed to have
system availability that is consistent with the significant importance of the contamination warning
system function. Backup and recovery plans and procedures should be defined and implemented.
Security, authorizations, and controls. Access to contamination warning system data and
applications should be restricted to authorized personnel as determined by the utility. In addition,
data encryption should be considered for information exchanged between the utility and other
partners when it passes over public networks.
Change management and maintenance. Throughout the life-cycle of IT software and hardware
it will be necessary to implement upgrades and perform routine maintenance. From a planning
perspective this should be considered both in terms of costs and the approach for implementing
upgrades and performing maintenance.
2.2 Utility Staffing
Approaches for staffing a contamination warning system will vary by utility, but should work within
existing organizational structures and routine job functions to the extent possible. As discussed in Section
2.1, application of system engineering principles plays a critical role in successful implementation of a
contamination warning system. This applies to staffing as well - planning, design, implementation,
operation, and ultimately evaluation will rely on effective communication and coordination across
divisions or departments within the utility. To the extent possible, generic titles and roles are provided to
organize the discussion in this section. Based on the description of activities, utilities should identify
individuals and departments within their own organization to fulfill these roles as appropriate,
independent of job title.
2.2.1 Project Management Team
While routine operation and maintenance of the contamination warning system should generally fall
within the routine job functions of utility staff, design and implementation will involve significant time
and effort from dedicated managers within the utility. Depending on utility organization and operational
paradigm, the activities described below may be managed by one individual, or more likely, a core project
management team. In addition, support from the utility director and/or board of directors, as well as all
senior supervisors and managers is critical to contamination warning system implementation. Project
management activities to be addressed by the management team generally include the following:
Development of design goals and objectives
Communication of the goals and objectives of the contamination warning system to all levels of
management, staff, and external partners
Prioritization of work activities and allocation of resources.
Networking and establishment of agreements with external partners
May 2007 11
-------
Planning for WS-CWS Deployment
Coordination of procurement, installation, and inspection of equipment, hardware, and software,
including licenses and maintenance agreements
Coordination of teams supporting design and implementation of monitoring and surveillance
components and consequence management plan
Verification of reporting protocols and coordination with primacy agency
Development of an overarching framework describing the project phases, goals, objectives,
schedule, and milestones
Coordination of IT and data management aspects of design, implementation, and operation across
all aspects of the contamination warning system to ensure consistency with existing protocols and
procedures and interoperability of systems
Integration of contamination warning system concept of operations with existing plans and
protocols that govern routine operations and activities within the utility
Integration of contamination warning system consequence management plan with existing
response protocols, plans, and procedures
Development of evaluation plan and identification of metrics for evaluation
Coordinate identification and documentation of lessons learned
2.2.2 Utility Staff
While the primary focus of a contamination warning system is monitoring water quality in the distribution
system, the design and implementation of a contamination warning system involves the collaboration of
the utility as a whole; however, each department has its own capabilities to contribute. The information
presented below is organized by general divisions or departments; actual departments and roles may vary
by utility. Component-specific activities are described in "Staffing and Costing Considerations"
subsections within Section 4 - Section 9.
Security/Risk
Evaluate physical vulnerabilities of facilities
Design security enhancements to facilities
Provide oversight during installation of security enhancements
Coordinate planning for and response to security breaches with law enforcement
Manage and coordinate response to security incidents utility-wide
Coordinate training and functional drills related to consequence management
Determine if security breaches provided an opportunity to contaminate water
Water Quality
Evaluate vulnerabilities of distribution system
Select water quality parameters and sensor hardware
Design/create monitoring network (placement of monitors around system)
Design and oversee the conduct of tracer studies if needed
Coordinate installation of monitoring stations
Select target analytes and lab methods for baseline and triggered sampling
Identify sampling locations and frequency for baseline sampling program
Develop and implement sampling protocols
Identify requirements for integration of data into event detection system (EDS) for water quality
monitoring and consumer complaint surveillance component
Coordinate laboratory capabilities with local laboratory network
Maintain proficiency for any in-house lab and field analyses
Establish water quality baselines for comparison with abnormal water quality results
Develop site characterization procedures, including selection of field test equipment
Coordinate with public health on surveillance activities
Coordinate training on operation of all contamination monitoring system components
May 2007 12
-------
Planning for WS-CWS Deployment
Investigate water quality and distribution issues in reference to observed triggers
Evaluate remediation options
Operations and/or Distribution
Coordinate with Supervisory Control and Data Acquisition (SCADA) or other communication
systems
Support implementation of distribution system tracer studies, if conducted
Investigate distribution system operations and maintenance in reference to observed triggers
Support field sampling activities
Plan and implement isolation and containment options
Support evaluation and implementation of remediation options
Information Technology
Identify data management, hardware, and software needs for utility
Design and implement changes
Coordinate the flow of all of the information/data streams
Engineering and Planning
Design and construct water quality monitors
Design and construct security improvements to facilities
Provide input for monitoring network design
Perform hydraulic and water quality modeling
Design remediation activities in event of confirmed contamination
Public/Customer Affairs
Identify improvements to call center to recognize and track water quality related calls
Implement hardware, software, and training changes
Interface with outside organizations for public health surveillance
Identify appropriate customer call information streams and how to integrate into Event Detection
System
Provide public outreach
Establish baseline levels for water-quality related customer complaints
Public notification of contamination and appropriate actions
Administration
Financial tracking
Project tracking
Procurement
Legal review of agreements, documents, etc.
Management of contracts, grants, agreements, etc.
2.3 Local Partners
Designing, implementing, and ultimately operating a contamination warning system is a complex task
that relies on the coordination and cooperation of many partners. In addition, the impacts of a drinking
water contamination event are not isolated within a utility, so local partners should be engaged in the
complex task of responding to a contamination event, intentional or accidental, that can involve criminal
activity, public health impacts, regulatory compliance, and hazardous materials response. Because it is
necessary for the utility to rely on the assistance of local partners to operate the system, those partners
should be involved as appropriate in the design and implementation of the system. As part of this
process, the utility should consider what formal agreements can be reached with other agencies, which
can vary depending on whether the utility is publicly or privately owned. Furthermore, if publicly owned,
May 2007 13
-------
Planning for WS-CWS Deployment
nuances of being an independent public agency, a single municipal department, or part of a public works
department should be considered.
2.3.1 Identifying and Engaging Partners
The utility is the operational hub of the system as the primary operator of the majority of monitoring and
surveillance components of the contamination warning system, with the exception of public health
surveillance. However, other partners may be involved in trigger validation and/or consequence
management activities. Figure 2-1 provides an overview of potential partners in contamination warning
system deployment.
Federal Bureau of
Investigation
State Emergency
Management
and
Homeland Security
Agencies
State Law
Enforcement
Stat
Wat
Centers For Disease EPA Regional Offices
Control and Prevention
Local Health Local Wastewater Local Law
Department Utility Enforcement
Local Fire, EMS, / Water \ Local Civil
and HazMat f Utilitv Government
.... ... Public Health and
Local Emergency Host ,. . . .
. . _ ... _ ..... Environmental
Planning Committees Facilities , . . .
Laboratories
e Drinking and Waste Neighboring Utilities
er Primacy Agencies
EPA Criminal
Investigation Division
EPA National Response
Center
State Emergency
Res ponders
State Government
Media
Figure 2-1. Potential Contamination Warning System Partners
During the early stages of the investigation of and response to a "possible" contamination incident, a
utility will likely rely on local partners for assistance, and as the credibility that a contamination incident
has occurred increases, the number of partners will increase. As illustrated in Figure 2-1, the number and
scope of partners that can become involved in responding to a contamination event can be significant. In
planning for contamination warning system deployment, drinking water utilities should identify and
engage local partners early in the process, particularly those partners such as local health departments and
public health and environmental laboratories that will have a significant role in routine operations.
Specific responsibilities of partners and when they are engaged will vary by utility and jurisdiction.
However, Table 2-1 provides a summary of possible contamination warning system partners and their
possible role in design, implementation, operation, and/or response.
Table 2-1. Summary of Potential Contamination Warning System Partners
Partner Organization
Host facilities
Local health department
Roles and Responsibilities
Provide facilities for placement of water quality and/or enhanced security monitoring
stations, including 24/7 access to utility staff for maintenance, trigger validation, and
response activities.
Monitor health of the population through public health surveillance as part of routine
operation of the system and may have some degree of analytical capability to
support sampling and analysis. Provide support during consequence management
including consultation and public notification. Serve as conduit to state and Federal
health departments and agencies.
May 2007
14
-------
Planning for WS-CWS Deployment
Partner Organization
Local law enforcement
Local civil government
Local emergency
planning committees and
emergency management
agencies
Local fire, EMS, and
Hazmat
Environmental and public
health laboratories
Local wastewater utility
Neighboring utilities
Media
State government
State emergency
responders
State drinking water and
wastewater primary
agencies
State emergency
management and
homeland security
agencies
State law enforcement
EPA Regional offices
and/or laboratories
Federal Bureau of
Investigation (FBI)
Centers for Disease
Control and Prevention
(CDC)
Roles and Responsibilities
May assist in routine operation of enhanced security monitoring; provide support
during consequence management through credibility determination and response.
May also serve as conduit to state and national law enforcement and intelligence
agencies.
Should be engaged early in the planning for implementation. In the event a utility's
service and/or wholesale areas span multiple jurisdictional entities, it may be
necessary to engage in formal agreements among different local governments and
their respective agencies to ensure cooperation and support and allocation of
funding. Also, should an event occur, the elected officials of different jurisdictions
should be appropriately informed of the state of the situation so that they can
effectively communicate with their constituencies.
Primarily support consequence management activities as a conduit to other
response agencies at the state and Federal level. Can support provision of alternate
water supplies, coordination, disaster declaration, and transition to National
Response Plan implementation.
Local fire and EMS organizations may have a role in routine operations for public
health surveillance as a provider of 91 1 and/or EMS data. These organizations, as
well as local Hazmat play a critical role in consequence management including site
characterization activities to support credibility determination.
Provide support to routine sampling and analysis activities to establish baseline and
maintain analytical proficiency. In addition, provide analytical support during
consequence management to assist in credibility determination as well as response
and remediation efforts. State public health laboratories provide access to CDC's
Laboratory Response Network.
May provide analytical support for routine sampling and analysis. Should be
consulted in the development and implementation of consequence management
plans due to the potential impact of contamination on wastewater operations.
May provide support in the event of a contamination incident through mutual aide,
assisting with provision of alternate water supplies, remediation, and recovery
activities.
Local media organizations may serve as a valuable resource in communicating
messages to the public in the event a contamination incident occurs.
May provide support to implementation activities in terms of gaining cooperation
from State organizations. May have a role in establishing formal agreements with
State partners or coordinating funding resources. For consequence management,
should be informed and engaged once contamination has been confirmed to assist
in coordination of resources and communication.
Provide support to consequence management phases if a contamination incident is
confirmed. Should be engaged in consequence management planning to ensure
efficient transition in the event a contamination incident escalates.
Primacy agencies can be public health agencies as well as separate state or local
environmental agencies, like state or regional water quality boards. If contamination
does occur, there may be regulatory ramifications related to use of contaminated
water, public notification, environmental concerns for discharged water, quality of
alternative supplies, and more. Additionally, the primacy agency, along with EPA,
should be consulted on any potential remediation and recovery plan.
Provide support to consequence management phases if a contamination incident is
confirmed. Should be engaged in consequence management planning to ensure
efficient transition in the event a contamination incident escalates.
Provide support to consequence management phases if a contamination incident is
confirmed. Should be engaged in consequence management planning to ensure
efficient transition in the event a contamination incident escalates.
May assist in coordination of Federal resources and may also assist by providing
analytical surge capacity during phases of consequence management.
May assist in site characterization and/or consequence management plan
development. Establishing a relationship with local FBI agents early in the
implementation process is critical to establish and understand roles and
responsibilities in the event contamination occurs.
Provide oversight to the Laboratory Response Network, a network of public health
laboratories with the ability to analyze for select agents based on established
analytical protocols. Ensure member laboratories have appropriate training,
equipment, reagents, and resources. Provide technical consultation during
credibility determination and other phases of consequence management.
May 2007
15
-------
Planning for WS-CWS Deployment
Partner Organization
EPA Criminal
Investigation Division
(CID)
EPA National Response
Center
Roles and Responsibilities
Provide support to consequence management phases if a contamination incident is
confirmed. Should be engaged in consequence management planning to ensure
efficient transition in the event a contamination incident escalates.
Provide support to consequence management phases if a contamination incident is
confirmed. Should be engaged in consequence management planning to ensure
efficient transition in the event a contamination incident escalates.
A recommended strategy for engaging partners is to first consider those partners who will have a role in
routine operations of the system or will be involved as "first responders" based on the consequence
management plan. Engaging the numerous partners involved in establishing a contamination warning
system is a daunting challenge on its own, without the myriad other tasks the utility implementation team
will be occupied with. It is not uncommon for the service area of a utility to span city limits and county
borders, and into the jurisdictions of numerous police, fire, and public health agencies, not to mention the
umbrella jurisdictions of hierarchical agencies like county and state emergency management, public
health, and homeland security agencies, to name just a few. Therefore, it is essential to take full
advantage of the existing groups and organizations in which these partners may already participate.
Figure 2-2 illustrates a recommended approach for engaging partners.
Engagement to form CWS
Implementation Team
Direct engagement
with key county,
state, and federal
partners
Regional Homeland
Security Programs
&
Existing Local
Partner Groups
Engagement
through existing
partner groups
Direct
engagement with
extended
partners
Figure 2-2. Recommended Strategy for Engaging Contamination Warning System Partners
Depending on the role or type of support contamination warning system partners play in implementation,
it may be necessary or desirable to establish formal agreements. Considerations for establishing these
agreements are discussed in Section 2.3.2.
2.3.2 Considerations for Formal Agreements with Local Partners
Inter-agency agreements, memorandums of understanding, memorandums of agreement, mutual aid
agreements, and other agreements are becoming common practice in most jurisdictions. The documents
contain language that is mutually agreed upon by all stakeholder agencies and generally define
collaborative efforts that involve action items, equipment resources, or regional governance. When
engaging local, county, state and federal partners in implementation activities, the utility should address
the subject of inter-agency agreements early in the process. Addressing formal agreements early in the
implementation process is extremely important, as they commit partner agencies to specific roles and
actions. Without them, implementation can be stalled by inter-agency disagreements or
misunderstandings, or an agency may be left responsible for costs they believed would be covered by
another.
May 2007
16
-------
Planning for WS-CWS Deployment
The utility should first identify its own protocols for establishing formal agreements with external
agencies, organizations, and partners. This includes identifying who holds the authority to enter the
utility into these types agreements (who signs the document), any procedural details, like minimum or
maximum review periods, paperwork routing procedures, restrictions on the types of agencies or groups
the utility may enter into agreements with (public and private), or limits of commitment (monetary or
other). The utility should also develop a clear understanding of the same types of information from the
agencies it intends to engage. Subjects of the agreements extend beyond simply who pays for equipment;
commitments should be made to provide man-power both for the implementation and operation of the
contamination warning system; allocation of resources; etc. If funding is from an external source all
applicable standards and regulations for establishing formal agreements should be followed.
2.4 Costs
Cost of contamination warning system implementation can vary widely from utility to utility, based on a
variety of factors, and perhaps most significantly based on existing capabilities across components.
Factors to consider in estimating costs for specific components are embedded throughout Section 4 -
Section 9. From a project planning perspective, considerations should be given to long-term issues
including operation and maintenance and sustainability prior to finalizing a design. Contamination
warning system deployment should be considered a significant program or initiative that will involve
long-term involvement and large investments by the utility. Substantial equipment and construction may
be necessary, as well as additional maintenance needs. These maintenance needs should be tracked on a
system-wide basis and the labor and equipment expenditures should be included in budgets for the life of
the system. All costs should be monitored and compared to budgetary constraints. Additionally, this
project may involve sensitive information which should be tracked, and personnel who use or handle the
information will have to be screened. This places an additional responsibility on the administrative
personnel to adequately document and supervise information security issues. Considerations for
information security are discussed in Appendix B.
The potential cost of contamination warning system deployment should include both the tangible
(equipment, installation, etc.) as well as the intangible (staff perception, morale, motivation, etc.)
elements. The tangible costs should be estimated using a life-cycle approach to capture the management
and coordination effort, the capital costs of the equipment, initial training, startup, testing, and calibration,
and the long term operations and maintenance costs (training, calibration, spare equipment and
components, maintenance contracts, chemical testing supplies, monthly communications fees, etc.). The
intangible costs should be identified and recognized and a management plan developed to address any
issues. Examples of intangible costs could include a staff person's perception that the operation of the
system adds more work to his/her day with no additional compensation or the fear that the system will be
advertised to the public as enhancing safety and security while not actually delivering on the promise.
In addition to component-specific cost factors discussed throughout the remaining sections of this
document, the following factors should be considered as part of project management and system
engineering costs:
Project Management and Coordination. Development of agreements, schedule, work plan,
communication products; administration and financial tracking; routine coordination and strategy
meetings.
Development of Integrated Concept of Operations. Coordination of concept of operations
development; analysis of component-specific concept of operations for consistency; development
of integrated concept of operations documentation and procedures.
IT System Engineering. Assessment of existing systems; procurement and installation of
hardware and software; implementation, operation, and maintenance of systems.
Evaluation and Refinement. Development and implementation of evaluation plan;
identification and documentation of lessons learned; identification of refinements to optimize
system performance.
May 2007 17
-------
Planning for WS-CWS Deployment
Section 3.0: Design and Implementation Framework
Deployment of a contamination warning system should follow the typical programmatic approach in
which proposed enhancements are planned, designed, implemented, tested, maintained and refined. This
section provides a comprehensive framework for design and implementation. Alternate approaches for
design and implementation can be considered, however the concepts described below should be
addressed. Table 3-1 summarizes a recommended approach for contamination warning system design
and implementation based on lessons learned from the initial pilot. While there may be some deviations
in terms of how each stage is applied for a given component, all components should address planning and
pre-design, design, implementation, preliminary testing, operation and maintenance, and evaluation and
refinement.
Table 3-1. Overview of Design and Implementation Framework
Stage of Approach
Planning and pre-
design
Design
Implementation
Preliminary testing
Operation and
maintenance
Evaluation and
refinement
Description
Developing a core implementation team, defining design objectives to guide
implementation, and a preliminary assessment of existing capabilities relative to design
objectives.
Development of a preliminary concept of operations and development of a detailed work
plan and schedule to guide implementation.
Implementation of enhancements, installation of equipment, and training according to the
plan.
Operation of the contamination warning system for the purpose of collecting data
necessary to understand system performance and finalization of the concept of operations
to optimize system.
Operation of the contamination warning system for the purpose of monitoring for
contamination incidents and other water quality issues.
Analysis of data and information generated during full operation to refine and optimize the
system.
Additional detail on the application of this framework to contamination warning system monitoring and
surveillance components and consequence management is presented in Section 4 - Section 9.
3.1 Planning and Pre-design
The initial steps in developing a contamination warning system are included in the planning and pre-
design stage. The utility should develop a team to support design and implementation and define what it
wants the system or component to do.
3.1.1 Building the Team
As emphasized in earlier sections of the document, deployment of a contamination warning system
involves an integrated team within the utility that also extends to external partners for certain aspects of
monitoring, surveillance, and response. It is important that the implementation team apply the system
engineering principles discussed in Section 2.1 throughout all phases of contamination warning system
implementation. Utility departments and divisions should work together as an integrated team to leverage
existing infrastructure, systems, protocols, and procedures to ensure effective operation of the system.
3.1.2 Defining the Utility-Specific Design Basis and Design Objectives
As introduced in Section 1.1, the contamination warning system should be able to meet the following
design objectives:
Detection of a broad spectrum of contaminant classes
Spatial coverage of the entire distribution system
Detection of contamination in sufficient time for effective response
May 2007 18
-------
Planning for WS-CWS Deployment
Reliable indication of a contamination incident with a minimum number of false-positives
A sustainable architecture to monitor distribution system water quality
These concepts should be considered during each step of the pre-design process, for the components and
the system as a whole. Failing to consider each design objective may result in the design of
contamination warning system that accomplishes some of its goals, but as a collective system fails to
meet its ultimate objective. Table 3-2 summarizes overarching design basis considerations for the
contamination warning system.
Table 3-2. Design Basis Considerations
Design
Objectives
Description
Design and Implementation Considerations
Capability
Ability to detect contamination of the
distribution system through contamination
warning system components.
Most components provide an indirect measure of
contamination; thus it is necessary to have a
process for validation of triggers and coordination
across components.
Contaminant
Coverage
Contaminant classes that can be detected
by the system; actual contaminants may
vary from system to system depending on
the manner in which components are
implemented.
May be influenced by several factors for each
component such as type of public health
surveillance or disinfectant residual. Design
objectives should target coverage of as many
contaminant classes as possible. Expanding to
additional contaminant classes could be an objective
to consider as part of system evaluation.
Spatial
Coverage
Amount of distribution system covered by
one or more of the contamination warning
system components.
May vary by component pending jurisdictions,
number of sensors, sampling routes, security sites,
etc. While the integrated contamination warning
system should cover the entire distribution system,
the degree of coverage by each component will
vary. An aspect of system engineering should be to
maximize coverage to the extent possible across all
components in a way that optimizes protection
through the entire system.
Timeliness
Function of ability to detect anomalies and
conduct initial trigger validation to
determine possible contamination.
Development of procedures for routine operation of
the system and initial trigger validation will inform
design and operation of the system and ultimately
impact timeliness.
Reliability
At the system level, reliability is
characterized in terms of the rate of false
alarms and occurrence of undetected
contamination.
Reliability of the system is influenced by design
decisions made at the component level. However,
system reliability can be improved by integration of
information and coordination across components to
maximize confidence in a system alarm.
Sustainability
Ability to provide drinking water utility with
an understanding of the distribution system
in terms of water quality and variability and
information to optimize distribution system
operation.
Contamination warning system activities and
procedures should be designed for incorporation into
routine job functions to maintain the system and
support dual use application. Security should not be
the only consideration in developing design
objectives.
In the planning and pre-design stages of contamination warning system deployment, drinking water
utilities should evaluate these design objectives relative to their specific needs and objectives and
customize and adapt as appropriate. It is important to consider objectives beyond the security aspects of
contamination warning systems, particularly dual use applications that could help to promote
Sustainability of the system by optimizing utility operations. Potential dual-use benefits of a
contamination warning system could include the following:
Detection of cross-connections and other distribution system water quality problems
Improved relationship with public health organizations, including mutual sharing of information
and alerts
Enhanced knowledge of distribution system water quality leading to improved operations (e.g.,
more consistent disinfection residual levels, improved corrosion control, early warning of
nitrification episodes, reduced disinfection byproduct levels, etc.)
May 2007
19
-------
Planning for WS-CWS Deployment
Identification of problem valves (closed, partially closed, inoperable)
Improved coordination with local, state, and federal response organizations
Reduced occurrence of tampering and vandalism
Improved information technology systems and interoperability
Improved consumer complaint tracking and response
Improved laboratory capability and an established laboratory network
Consequence management plans applicable to any water quality emergency
3.1.3 Preliminary Assessment and Gap Analysis
Understanding the starting point and existing resources that will form the foundation for the system is
critical in the early stages of planning and pre-design. Before progress towards the desired end-state of
the contamination warning system can be made or measured, a utility should fully understand the
capabilities of their existing systems, procedures, and other resources.
The recommended approach for conducting a thorough analysis of the utility's existing capabilities is to
use personnel with varying degrees of experience with the systems being evaluated. Experience at the
pilot utility demonstrated that including staff who do not routinely use a particular system can provide
additional insight into what is a routine process for an experienced user. This can be accomplished by
using a mix of utility personnel and/or outside consultants. The intent is to gain a complete understanding
of the current state of the utility, and the benefit of using a diverse team is that it results in a more realistic
plan to reach the desired capabilities.
At the conclusion of this self-assessment, the utility's design and implementation team will have
developed a solid understanding of their existing systems. Using the preliminary concept of operations as
a benchmark, the utility can examine its existing operations by process or by component and measure the
gap between current and desired capabilities. The result is a gap analysis that clearly outlines the progress
that a utility should make to achieve the design objectives. For example, the analysis of the customer call
center at the pilot utility discovered that the staff was already collecting very useful complaint data, and
an excellent complaint management system was already in place. Only minor modifications that had very
minimal impact on existing staff and processes were necessary to transition the existing system into a one
that would feed the critical water quality complaints into the system. This gap analysis may be further
refined following development of a preliminary concept of operations as discussed in Section 3.2.
3.2 Design
The design stage of the programmatic approach for a contamination warning system encompasses the
development of the plans and specifications for each component and a consequence management plan. It
is critical that the performance objectives of the design basis are considered throughout this process, and
the design stage is no exception. At the system level, design should go beyond information integration
and event detection. It should also consider how all resources - staff, IT, communications, equipment,
etc. can be leveraged across the entire project. Further, any resources at the disposal of the utility and its
partners, including but not limited to staff, communications, equipment, and training opportunities, should
be evaluated for possible use within the project. This will not only result in an efficient use of time and
materials, but will also highlight the dual-use benefits possible through implementation of the
contamination warning system (e.g., radio communication equipment purchased for the use of equipment
maintenance becomes more valuable when it is also used as part of the consequence management plan).
3.2.1 Conceptualize System
Before beginning physical design, consider a contamination warning system at a conceptual level, and the
manner in which this conceptual model translates to physical systems and implementation. Firm
understanding of the basis for design decisions should inform all aspects of the design process. This
May 2007 20
-------
Planning for WS-CWS Deployment
understanding can be developed, in part, though the development of a preliminary concept of operations
for the entire contamination warning system.
At this early stage of planning, many of the details of routine operation are unknown. However, a
preliminary concept of operations can still be developed at this stage to identify the general capabilities
that the fully implemented system should possess and how that capability relates to the existing resources
and capabilities within the organization. The preliminary concept of operations will also establish
potential roles and responsibilities and user requirements for IT-related systems. In doing so, this
document outlines the characteristics of the finished system and provides the framework for the design
and implementation of the individual components. Its development is a critical step towards successful
implementation of a contamination warning system because it provides an initial benchmark against
which to measure a utility's existing capabilities and a means by which to plan for enhancements. Note
that additional details will be added to the concept of operations as the contamination warning system is
designed and implemented, including a very detailed concept of operations for each component. Thus
development of the concept of operations should be viewed as an iterative part of the design and
implementation process, with the preliminary concept of operations serving as a starting point for the
design process.
3.2.2 Work Plan for Implementation
Prior to initiating implementation activities for any component, a detailed implementation plan, or work
plan, should be developed that clearly identifies priorities, schedule, milestones, and resources. It may be
necessary to revisit the assessment and gap analysis in order to prioritize some of the identified
enhancements to maximize time and resources. This should be considered at the system level, in advance
of implementation using a system engineering approach as discussed in Section 2.1.
3.3 Implementation
Implementation of a contamination warning system begins with consensus on the approach identified in
the component-specific work plan and completion of any associated design work (i.e., hardened access
points as part of the physical security enhancements). This stage can involve significant coordination
with outside consultants and contractors, depending on the capabilities and availabilities of in-house staff.
Again, since a contamination warning system relies heavily on effective data management, the
involvement of the information technology staff is critical. Each component-specific work plan
developed during the design stage should be implemented concurrently, to the extent possible. The
timelines should be carefully monitored to ensure delays do not create problems in other components'
deployments. In addition, any training specified in the component-specific work plan should be
conducted during the implementation stage. During this stage of the project, enhancements are
implemented and installed, concept of operations and consequence management plans are reviewed,
revised, and reconciled, and training for routine operation, maintenance, and consequence management
should be conducted.
3.4 Preliminary Testing
Once components begin to come on line, the contamination warning system transitions to the preliminary
testing stage of implementation. During this period, enhancements are in place and the system is
technically operational, however it is being operated in a "test" mode. Meanwhile, the concept of
operations and consequence management plan can be finalized, taking into consideration the additional
information and insights gained during the design and implementation stages, and reflecting the "as-built"
system.
May 2007 21
-------
Planning for WS-CWS Deployment
3.4.1 Baseline Operation
The automated analytical capabilities of a contamination warning system depend on having a historical
baseline or base state against which to match current operational conditions. Anomalies from the baseline
create a trigger, which is then analyzed and validated, or dismissed, as a possible contamination event.
Depending on the seasonal variability in the operating conditions for a specific water utility, the
preliminary testing stage could last up to a year, or even longer. If the utility utilizes multiple water
sources, blends sources at varying ratios, or uses an alternative source exclusively during different
portions of the year, operations under these conditions should be experienced by the system to gain a
complete picture of the normal variability in water quality parameters. Similarly, changes in source water
quality, such as episodic taste and odor events, and operational changes, such as periodic changes in
distribution system residual, should be incorporated into the base state to the extent possible. The more
"normal" conditions that the system can be exposed to during the preliminary testing stage, the more
detailed the baseline will be and the more successful the contamination warning system will be in meeting
is design objectives, particularly in regards to its accuracy and reliability.
During this period of preliminary testing, the system is operating at full capability. All field equipment
are installed, communication and IT enhancements are in place, data streams are being transmitted to the
utility and personnel have been trained to collect the information pertinent to the contamination warning
system. However, the system is being operated for the purpose of collecting data necessary to understand
system performance. This does not include responding to the alarms generated by the system, except in
the capacity of testing procedures implemented in response to a trigger. Because a baseline has yet to be
established, it is difficult to identify whether an alarm is indicative of a water quality problem, or merely
an aspect of the base state that had not been previously observed by the system.
This mode of operation can place the utility in a vulnerable position. Information indicating possible
contamination events will be received by the utility, likely more frequently than before, without knowing
the reliability of this information. It is critical, therefore, that the utility remains vigilant in its use of
existing, established procedures of possible water quality problems or security concerns during this stage.
An example at the pilot utility involved its consumer complaint surveillance component. Prior to
implementation of their system, the pilot utility had a threshold value for the number of water quality
complaints that were manually logged in a certain database within a set time period. If that number was
exceeded, utility personnel began certain protocols to investigate a potential contamination event. During
baseline operations, the pilot utility began identifying that a larger number of their consumer calls related
to water quality in some way. However, until a baseline value for this new indicator of possible
contamination events was developed, response and remediation steps were not initiated for each call or
even when the old threshold value was reached. Instead, the utility continued to monitor the number of
complaints that were of the same severity as those previously entered into their particular database
(although now that process was more automated). When the historically validated threshold was reached,
the utility initiated its established response protocol.
3.4.2 Finalization of Concept of Operations and Consequence Management Plan
As discussed earlier, the concept of operations is the description of the routine operation and initial trigger
validation for each component of the contamination warning system. While the terminology and
approach for this documentation may vary, it is important that the operational concepts be clearly
documented and that utility staff and local partners understand their roles and responsibilities in operating
the system.
At this stage, the system-wide concept of operations should be revised to reflect information and insights
gained through design and implementation. The concept of operations should now include details of the
capabilities of the components, the data streams that will be mined and the IT system infrastructure to
collect and analyze the results. Roles and responsibilities for specific job descriptions or critical
May 2007 22
-------
Planning for WS-CWS Deployment
personnel should be defined and job-specific checklists developed to facilitate the performance of new
tasks and new processes implemented as a part of system enhancements. It is critical that the
development of the component concept of operations are coordinated to ensure that resources are applied
in a consistent and compatible manner across components, and integrated to ensure that data from
multiple components can be integrated to better inform response decisions.
Recommendations regarding the development of the consequence management plan, and the steps to
finalize this document are covered in separate guidance published by the EPA. However, completion of
the Consequence Management Plan should be informed by final Concept of Operations to ensure the
former is capable of integrating information from the multiple components and affecting a response.
3.5 Operation and Maintenance
This stage represents the remaining life of the system. During this period the utility should be prepared to
maintain the components to continue to meet the design basis for the life of the system. This stage should
not begin until a baseline for the system is established and the consequence management plan is in place.
3.5.1 Operation
With the establishment of a robust baseline and the completion of the final consequence management
plan, the contamination warning system can be placed into full operation. At this stage the system is
operating at full capacity and actively monitoring the water distribution system for contamination
incidents and other water quality problems. Each component is feeding its data via established
communication protocols into the event detection system(s) at the utility for analysis. When component-
specific threshold values are reached, triggers are initiated to indicate a departure from the baseline. If
that trigger can be validated by the processes and procedures detailed in the concept of operations, a water
contamination event is deemed possible and the Consequence Management Plan is implemented to affect
a response. If the event is further confirmed, the Consequence Management Plan guides the remainder of
the response and remediation actions. If the components have been well designed and the system as a
whole been automated, the day-to-day operations do not look significantly different for the majority of
utility personnel involved.
3.5.2 Maintenance
Maintenance of the components of the contamination warning system is the most influential determinant
of the long-term success of the contamination warning system in meeting its design basis. Maintenance,
in the context of a contamination warning system, refers to the activities to maintain the intended
capabilities of the system. This includes the physical maintenance of equipment, upgrades of software,
and the continual training of personnel. For example, "orphaned" water quality monitors are of little use
in a contamination incident and it is vital that personnel responsible for their upkeep and calibration
understand their importance. Specifics regarding the maintenance of particular equipment and elements
of each component are included in Section 4 - Section 9. However, from a system-wide perspective, if
the component designs were completed with an awareness of the need for sustainability, then they would
ideally specify equipment, procedures or other enhancements that have dual-use applications. The use of
the security enhancements for routine operations of a water utility will ensure that the contamination
warning capabilities are fully functional in the remote chance an event were to occur.
3.6 Evaluation and Refinement
Evaluation and refinement should be considered in planning for contamination warning system design
and implementation. Time and resources should be built into the schedule for carrying out evaluation
activities and implementing refinements to optimize system performance.
May 2007 23
-------
Planning for WS-CWS Deployment
3.6.1 Evaluation
The primary function of contamination warning system evaluation is to gauge the effectiveness,
reliability, usability, and sustainability of the system; adjust and streamline the system and approach; and
adapt to and incorporate advances in technologies, methods, and protocols that occur over time.
Operation. Refers, in general, to all aspects and degrees of functionality, usability and utility.
Ideally, the chosen equipment and technologies will not require an increase in skill level or
manpower to operate and maintain, and will be robust enough to work under real-world
conditions. More likely, however, a balance should be struck between the ideal and what is
feasible, affordable, and available.
Performance. Measures of performance are equally important to those of operation.
Performance of the tools, components, and system refers, in general, to their ability to
consistently provide accurate data in a timely manner consistent with intentions of the system
design. A system that is easy to operate will not be useful unless it performs as intended to meet
the overall system goals. To evaluate all system aspects fully, the performance should be
measured in several different ways including the range of contaminants and contaminant classes
that may be detected; the accuracy of the data produced; the ability to discern whether a data
anomaly is indicative of contamination or caused by something benign and the ability to reliably
detect an actual contamination event. Timeliness is an important measure of performance. A
system that can detect contamination consistently but requires long periods for detection is of
limited use. All aspects of the timeliness of a component's data collection, analysis and event
response should be considered.
Sustainability. Sustainability will attempt to measure the likelihood that the contamination
warning system will become viewed as a sufficiently viable, valuable, cost-effective system
resulting in relatively wide spread adoption by the drinking water utility industry. Utilities will
view the system as sustainable if it provides benefits that are worth the system life cycle costs.
For the benefits to be seen as sufficient, for all except a few utilities it is recognized that the
contamination warning system should provide benefits other than warning of intentional
contamination. Intentional contamination, although it would have high adverse consequences is
viewed as a low probability event by most utilities. Therefore, part of the technical evaluation
will focus on identifying and evaluating the level and degree of these dual-use benefits. Life
cycle costs include costs to design, install, maintain, and operate the system. These costs will
include funding for engineering and utility staff labor, equipment, consumables and spare parts.
Evaluating the level and degree of these dual-use benefits will be the second key part of
evaluating sustainability.
In general, methods for contamination warning system evaluation can be divided into two categories:
field evaluation and data analysis. Field evaluation includes activities such as drills and exercises, direct
observations and performance tests, interviews, and documentation of lessons learned. Data analysis may
involve simulations and/or analysis and integration of data. These methods, along with the objectives and
metrics considered during evaluation will be expanded on in future guidance based on lessons learned
from the initial Water Security initiative pilot.
3.6.2 Refinement
As with any system, refinements to the contamination warning system are likely to be identified through
routine operation and maintenance or evaluation. In order to optimize system performance and ensure
that the design objectives continue to be met, it will be necessary to make some refinements to the system.
Refinements could include modifications to protocols and procedures or may be more extensive such as
replacement or modification of equipment. It is also important to consider that technologies may evolve
over time, goals and objectives may change, or other events could occur that necessitate refinement to the
system.
May 2007 24
-------
Planning for WS-CWS Deployment
Section 4.0: Online Water Quality Monitoring
The online water quality monitoring component consists of multiple water quality monitoring stations
installed at key locations throughout the distribution system with the goal of establishing the base state for
water quality and using sophisticated event detection systems to monitor for water quality anomalies that
could be indicative of contamination.
Online water quality monitoring is included as a component of the contamination warning system due to
its demonstrated potential to rapidly detect contamination through changes in several commonly
measured water quality parameters (Hall, 2007a, EPA 2006). These changes may result from the aqueous
chemistry of the contaminant (e.g., dissolution of an organic compound may result in an increase in the
TOC concentration) or from reactions with the disinfectant residual (e.g., oxidation of a reactive
contaminant consumes the free chlorine residual). While there are limited empirical data regarding the
impact of many contaminants of concern on conventional water quality parameters, there has been a
substantial amount of research over the past few years demonstrating that many contaminants of concern
can produce measurable changes in conventional water quality parameters (Hall, 2007b). Furthermore,
many of these contaminants have been shown to impact water quality at concentrations well below
reported lethal dose concentrations (Hall, 2007c).
Table 4-1 describes the manner in which each of the design objectives presented in Section 1.1 is defined
with respect to the online water quality monitoring component, and considerations regarding how these
objectives impact design and implementation.
Table 4-1. Design Basis Considerations for Online Water Quality Monitoring
Design
Objective
Description
Design and Implementation Considerations
Capability
Can indicate the presence of a contaminant
that significantly affects one or more monitored
parameters.
Detection of water quality changes is an indirect
measure of contamination; thus operation of this
component should include a process to
investigate the possible cause of the anomaly.
Contaminant
Coverage
High detection potential for classes 1, 2, 3, 5,
8, 9, 10, 11 and 12. Moderate detection
potential for classes 4, 6, and 7.
The specific parameters monitored will determine
the contaminant coverage. Disinfectant residual
type also has an impact on contaminant coverage
(Szabo, 2006).
Spatial
Coverage
Function of location, number, and density of
monitoring stations.
Several tools are available to design a water
quality monitoring network in a manner that
optimizes one or more design objectives, such as
minimizing consequences over a large number of
contamination scenarios.
Timeliness
Function of hydraulic travel time from the point
of contaminant introduction to the sensor, and
the concentration of the contaminant.
Time to detection can be considered as a primary
or secondary objective in the monitoring network
design. In either case, a well calibrated
distribution system model is necessary to perform
this analysis.
Reliability
Rate of false positive / negative results in this
application is largely unknown at this time.
May be addressed through event detection
systems and consequence management.
The design elements with the greatest impact on
reliability are the event detection system and the
water quality monitoring stations. If reliable
sensors are used and properly maintained, the
capabilities of the event detection system will
dominate reliability as it is defined here.
Sustainability
Provides utility with a better understanding of
water quality variability throughout distribution
system and provides an opportunity to
optimize distribution system operation.
The selection of parameters and monitoring
locations will have a direct influence on dual-use
applications that will improve sustainability, and
thus should be considered in the design of the
water quality monitoring system.
May 2007
25
-------
Planning for WS-CWS Deployment
The primary objective of this section is to describe considerations during planning for the implementation
of a water quality monitoring network in a drinking water distribution system as a component of a
contamination warning system. These considerations were derived from EPA's experience designing the
system at the initial pilot for this program. In planning for implementation of a water quality monitoring
network several key design decisions should be made, including the following:
Water quality parameters to be monitored
Use of a single monitoring station design or multiple designs in a tiered system
Specific sensors and instruments integrated into a water quality monitoring station
Number of water quality monitoring stations to install
Methodology for determining the locations at which water quality monitoring stations will be
installed
Comprehensive concept of operations to guide routine operations and trigger validation
Communication architecture to transmit data from monitoring locations to an operations center
IT architecture used to manage and store water quality and related data
Event detection software deployed to detect anomalies
Staffing available for monitoring station equipment operation and maintenance
A key objective of Section 4 is to provide information that will enable the reader to work through these
design decisions in a systematic and integrated fashion.
While the overall design of an online water quality monitoring system should be developed in an
integrated fashion such that all elements are compatible and serve a specific function in the overall
system, Section 4 considers five major design elements to facilitate the presentation of material. These
elements are:
Monitoring Network (Section 4.1): The spatial plan for deployment of water quality monitoring
stations throughout a drinking water distribution system. The monitoring network design
specifies the number and precise location of each water quality monitoring station.
Monitoring Stations (Section 4.2): The specific instruments, probes, or other equipment used to
monitor a water quality parameter, as configured into a monitoring station that contains all
ancillary equipment (e.g., plumbing, electric, communications, etc.).
Communication Systems (Section 4.3): All equipment, software, and services needed to transfer
data from each water quality monitoring station to a central location (typically a SCADA control
center).
Data Management Systems (Section 4.4): All hardware, software, and protocols necessary to
manage and store water quality and related data for event detection. The utility SCADA system
will typically serve as the foundation of the data management system for the water quality
monitoring network.
Event Detection Systems (Section 4.5): Software or algorithms designed to analyze real-time
water quality data in order to detect anomalous conditions that might be indicative of
contamination.
Each of these design elements is discussed in a dedicated subsection of Section 4, and is presented in the
phases of pre-design, design and implementation, and available tools and resources. Section 4 concludes
with a discussion of staffing and cost considerations.
4.1 Monitoring Network Design
Monitoring network design is a systematic process for determining the location and number of monitoring
stations deployed in a contamination warning system. The design will directly impact two important
aspects of system performance: the time of detection and the spatial coverage of the system. Section 4.1
presents information useful to the design and implementation of a water quality monitoring network
design, with an emphasis on activities related to planning and pre-design of the monitoring network.
When applicable, references to additional resources are included and summarized in Section 4.1.3.
May 2007 26
-------
Planning for WS-CWS Deployment
4.1.1 Pre-Design
Prior to designing the water quality monitoring network a number of key decisions should be made that
may involve some level of investigation and analysis. These decisions include the following:
The objectives of the monitoring network design
The level of validation needed for the distribution system model
The practical upper limit on the number of monitoring stations to be installed
Use of a tiered approach to monitoring network design
Types of facilities that will be considered candidate locations for monitoring station installation
Methodology used to design monitoring network
The remainder of this section describes considerations relating to each of these design decisions and,
where appropriate, references tools or resources that may be useful in this process. The pre-design phase
culminates in selection of a methodology and overall framework for monitoring network design that is
based on these key decisions.
Network design objectives. A water quality monitoring network can be designed around a
number of different objectives, some of which may be complementary while others are in some
degree of conflict. Furthermore, the various network design tools available may be able to
optimize towards some objectives but not others. Thus, an important step in the pre-design phase
is to decide on the primary objective that the network design will attempt to optimize. Examples
of design objectives include:
o Minimizing the consequences to the population
o Minimizing the extent of contamination
o Minimizing the time to detection
o Maximizing spatial coverage of the distribution system
o Maximizing the number of contamination events detected
o Maximizing protection of key facilities or populations
Given available tools, a monitoring network design can be truly optimized to only one of these
objectives; however, it is instructive to evaluate the performance of the monitoring network
design with respect to the other objectives, and to consider the trade-offs involved in the selection
of a primary objective.
Distribution system model validation. Many approaches to monitoring network design utilize
distribution system models, and thus the accuracy of the design is dependent on the accuracy of
the model. For this reason, distribution system model validation is an important part of
monitoring network pre-design. There are numerous approaches to model validation that vary
widely with respect to complexity, cost, and resulting degree of model confidence. Approaches
to validation of hydraulic and/or water quality portions of a distribution model include:
o Desktop analysis to verify that hydraulic behavior matches actual system operations
o Pressure studies to validate the hydraulic model
o Chlorine decay studies to evaluate the water quality model
o Tracer studies to validate the water quality model
An up-to-date, accurate network model is useful not only for sensor network design, but also for
emergency response planning, and potentially for identifying sampling locations and populations
at risk following a contamination incident. Potential resources that may be useful in
characterizing distribution system model performance are listed in Section 4.1.3.
Maximum number of monitoring stations. In order to proceed with the design of a monitoring
network, it will be necessary to establish an upper bound on the total number of monitoring
stations that could be installed. This is a function of the total budget for the monitoring network
May 2007 27
-------
Planning for WS-CWS Deployment
and the unit cost for each monitoring station. It may also include an analysis of incremental
benefit of each additional monitoring station added to the system, such as the example shown in
Figure 4-1.
Illustrative Example
o
10 20 30 40
Number of Monitoring Stations
50
Figure 4-1. Example Monitoring Station Tradeoff Curve
Tiered approach to monitoring stations. A potential means of increasing the total number of
monitoring locations within a given budget is to use a tiered approach in which two or more water
quality monitoring station designs, with different costs and capabilities, are deployed. Under this
tiered approach, the design of the water quality monitoring network will involve a tradeoff
between the total number of monitoring stations installed in the network and the unit cost (and
presumably capability) of the monitoring stations, as described in Section 4.2. In the context of
the design basis, this is a trade-off between spatial coverage and contaminant coverage. While a
tiered design may allow more monitoring stations to be deployed, the stations that monitor for
fewer parameters will have reduced contaminant coverage. Therefore, some contamination
events that would have been detected by the more complex stations will go undetected even if
they pass locations with simpler monitoring stations. Thus, the applicability of a tiered design
should be considered at a system level to determine the optimal trade-off between spatial and
contaminant coverage, and some of the monitoring network design tools can provide a means of
evaluating such a tradeoff. Additional discussion of tiered monitoring stations is included in
Section 4.2.
Candidate facilities for installing monitoring stations. Prior to designing a water quality
monitoring network, it is necessary to identify categories of feasible installation locations, such
as: utility facilities, fire stations, police stations, post offices, government buildings, etc. During
this initial phase of investigation, a set of general requirements can be provided to facility
managers to determine if they would be willing and able to host a water quality monitoring
station. These general requirements may include: security, 24/7/365 access to facility, and space
for a monitoring station. At this stage it is unlikely that specific facilities will be considered, but
rather categories of facilities (e.g., all fire and police stations within a given jurisdiction).
Specific locations will not be investigated until a monitoring network design has been completed
as described in Section 4.1.2. The purpose of identifying categories of feasible locations at this
point is to determine how the monitoring network design will be constrained, and for this reason
the categories of feasible locations should be as comprehensive as possible.
May 2007
28
-------
Planning for WS-CWS Deployment
Methodology for designing the network. The final stage in the pre-design of the monitoring
network is selection of methodology for designing the monitoring network. As mentioned
previously, most require use of a distribution system model including PipelineNet, Threat
Ensemble Vulnerability Assessment - Sensor Placement Optimization Tool (TEVA - SPOT), and
various tools built into hydraulic modeling software applications. Section 4.1.3 provides a partial
listing of available tools. It is necessary for the utility to have a calibrated distribution system
model in order to use these software applications. In selecting a monitoring network design tool,
it is important to ensure that tool is compatible with the platform for the utility's distribution
system model, and that it will support the overall approach to design. In particular, it is important
to ensure that the tool can optimize the design to the desired objective(s). Furthermore, if a tiered
design is used, it would be beneficial if the tool could optimize a design that incorporates
monitoring stations with differing arrays of sensors. Other factors to consider in selection of a
monitoring network design tool include:
o Transparency and rigor of the optimization methodology
o Applicability of the design and optimization methodology to detection of intentional
contamination incidents
o Time required to produce a design
o Features that facilitate comparison of multiple designs
o Visualization tools and compatibility with GIS
o Usability
If a distribution system model is not available, it will be difficult to optimize a sensor network design to a
specific objective, and it may be impossible to systematically evaluate trade-offs for various design
options. Expert-based designs can be developed without a model, but have been shown to perform poorly
compared to optimization methods that utilize distribution system models (Ostfeld, 2006).
The constraints on possible monitoring locations are another important design consideration. In general,
designs that place monitoring stations only at utility-owned sites will not be able to perform as well as
designs that allow for a large number of potential sites for locating monitoring stations.
4.1.2 Design and Implementation Approach
The activities described in Section 4.1.1 describe a process for pre-design of a water quality monitoring
network, culminating in documentation of an overall framework and selection of a methodology for
designing the monitoring network. The approach for design and implementation of the monitoring
network based on the pre-design may include the technical considerations and specifications described
below.
Design:
Develop a comprehensive list of physical addresses for potential installation locations. This list
should be based solely on consideration of the general categories identified during pre-design.
GIS can be an effective tool for compiling and visualizing this information.
Identify nodes in the distribution system model that correspond to each physical address.
Develop a suite of design constraints in terms of number of stations, potential installation
locations, and type of water quality monitoring station if tiered water quality monitoring network
designs will be considered.
Use a sensor placement tool, such as TEVA or PipelineNet, to develop a monitoring network
design for each set of constraints.
Compare the various monitoring network designs (e.g., through a tradeoff analysis, cost benefit
analysis, regret analysis, etc.).
Select a monitoring network design that specifies number and location of water quality
monitoring stations. If a tiered design is used, also specify the type of water quality monitoring
station at each location.
May 2007 29
-------
Planning for WS-CWS Deployment
Field verify each installation location in the design to ensure:
o Access to electrical power to run the equipment
Certain equipment may require higher voltage or current than may be found
through common wall sockets
o Access to water to run samples from the distribution system directly to the sensor station
o Drainage for the discharge stream from the monitoring station.
Verify the discharge option complies with applicable regulations.
o Size of area that the station is being installed
There should be enough room to contain the sensor station and there should be
enough space for utility personnel to access the station for maintenance
o Security of the location
Limit access to those without a need to maintain the equipment
o Accessibility, if location is not a utility owned facility
Utility personnel should have 24/7/365 access to the equipment at all times in
order to maintain the station or respond to an alarm event
o Safety
Health and safety should be addressed according to each site's safety procedures.
In all cases, however, the minimum safety considerations should meet
Occupation Safety and Health Administration (OSHA) requirements.
As field verification finds some locations to be unsuitable, iterate through modifications to the
monitoring network design and field verification until acceptable physical locations for all water
quality monitoring stations have been identified.
Obtain any required reviews or approvals on the design.
Implementation:
Develop agreements with facility owners who will host water quality monitoring stations.
o Facility access
o Contacts at facility and utility
o Water and sewer credits
Evaluation and Refinement:
Through simulations, evaluate ability of the "as-built" monitoring network design to detect an
ensemble of contamination incidents or other water quality anomalies.
Upon revision to the distribution system model (e.g., due to system growth, changing demand
patterns, recent calibration activities, etc.), evaluate potential modifications to the monitoring
network design.
Periodic evaluation of potential benefits of the addition of more water quality monitoring stations
or the relocation of existing stations.
4.1.3 A valiable Tools and Resources
The following tools and resources are available to support the design, implementation, and evaluation of a
water quality monitoring network as a component of a contamination warning system:
Ostfeld, et. al. "Battle of the Water Sensor Networks", in proceedings of the ASCE/EWRI Water
Distribution System Analysis Symposium, August 27-30, 2006. Cincinnati, OH.
Berry, J., Fleischer, L., Hart, W.E., Phillips, C.A., and Watson, J.P. 2005, "Sensor Placement in
Municipal Water Networks," J. Water Resources Planning and Management, 131 (3): 237-243
(2005).
Boccelli, D. L., Shang, F., Uber, J. G. and Wang, J. "Tracer Studies and Water Quality
Monitoring for Evaluating Network Model Confidence." 4th International Conference on
Watershed Management and Urban Water Supply, Shenzhen, China. 2004.
May 2007 30
-------
Planning for WS-CWS Deployment
Murray, R., Janke, R., Uber, J., Published in the Proceedings of the ASCE/EWRI Congress, Salt
Lake City, UT. The Threat Ensemble Vulnerability Assessment (TEVA) Program for Drinking
Water Distribution System Security. 2004.
Watson, Jean-Paul, et al, "A Multiple-Objective Analysis of Sensor Placement Optimization in
Water Networks", Proceedings of the ASCE/EWRI Congress, June 2004
PipelineNet: http://eh2o.saic.com/iwqss/.
Hart, W. E., J. Berry, R. Murray, C. A., Phillips, L. A., Riesen, J. P., Watson, 2007. "SPOT: A
Sensor Placement Optimization Toolkit for Drinking Water Contaminant Warning System
Design," Proceedings of the World Environmental and Water Resources Congress, Tampa,
Florida, 2007.
Murray, R., W. E. Hart, and J. Berry. "Sensor Network Design for Contamination Warning
Systems: Tools and Applications", Proceedings of the AWWA Water Security Congress, 2006.;
TEVA: http://www.epa.gov/nhsrc/water/dw/teva.html;
Info Water Sensor Location Manager (SLM):
http://www.mwhsoft.com/page/pjroduct/infowaterSLM/infowaterslm feature.htm (based on the
PipelineNet tool)
USEPA. 2005. Water Distribution System Analysis: Field Studies, Modeling and Management: A
Reference Guide for Utilities. EPA/600/R-06/028.
4.2 Monitoring Station Design and Installation
Monitoring station design requires careful consideration of the selected water quality parameters to be
monitored, as well as of the resources and conditions existing at the intended installation sites. Each
individual location will impose unique constraints on the installation, design and operation of a
monitoring station, and potentially constrain the parameters or instruments which may be used. Section
4.1 presented general guidelines regarding site constraints for installation and operation of monitoring
stations, while Section 4.2 deals with the design of the actual water quality monitoring stations. Section
4.2.1 describes activities related to planning and pre-design of the monitoring stations, while Section 4.2.2
describes the remainder of the design and implementation process in summary fashion. When applicable,
references to additional resources are included and summarized in Section 4.2.3.
4.2.1 Pre-Design
The most significant decision made prior to designing water quality monitoring stations is selection of the
parameters to be monitored. These parameters will define the capabilities of the water quality monitoring
system for detection of contamination events as well as operation for dual-use applications. It will also
have a significant impact on the cost and design of each water quality monitoring system. This section
describes factors to consider in the selection of water quality parameters.
One of the primary considerations in selecting water quality parameters for use in a contamination
warning system is the potential for parameters to change in response to the presence of a contaminant at
concentrations which pose a threat to public health, infrastructure, or acceptability of the water to
consumers. A substantial body of research has been conducted which demonstrates that the vast majority
of contaminants of concern from a security perspective do alter water quality in a detectable manner.
Section 4.2.3 includes references to some of this research, and Table 4-2 summarizes the impact of the
different contaminant detection classes, as presented in Table 1-1, on the water quality parameters.
Table 4-2. Impact of Contaminant Detection Classes on Water Quality Parameters
Class
1
2
3
4
5
Description
Petroleum products
Pesticides (reactive)
Inorganic compounds
Metals
Pesticides (non-reactive)
Example Contaminant
Diesel
Aldicarb
Arsenite salts
Mercuric salts
Fluoroacetate
TOC
X
X
X
CI21
X
X
COND
X
X
May 2007
31
-------
Planning for WS-CWS Deployment
Class
6
7
8
9
10,11
12
Description
Chemical warfare agents
Radionuclides
Bacterial toxins2
Plant toxins
Pathogens2
Persistent chlorinated organic
compounds
Example Contaminant
VX
Cesium-137
Botulinum toxin
Ricin
Vibrio cholerae
PCBs
TOC
X
X
X
X
X
CI21
X
X
X
COND
X
Acronyms: TOC - total organic carbon, Cb - chlorine residual, COND - conductivity
1) Indicated contaminant classes have been shown to consume free chlorine residual. Results are not applicable to
chloramine residual.
2) These contaminants are chlorine sensitive, thus it would be necessary to neutralize the chlorine residual in order
to maintain potency. Many neutralizing agents would also increase TOC.
In general, the results of reported research illustrate that free chlorine is the most sensitive indicator of
contamination, showing significant changes from baseline values at concentrations often one to two
orders of magnitude below lethal concentrations. Specifically, many contaminants were detected at
concentrations around 1 mg/L, while the corresponding lethal concentration might range from 10 to 100
mg/L. In the case of pathogens and bacterial toxins, the active contaminant would generally not produce
a detectable change in free chlorine residual; however, co-contaminants would often be present that
would reduce free chlorine residual (a condition necessary to maintain viability of these contaminants).
These results are applicable to systems using a free chlorine residual, but not to chloramines which were
found to be stable in the presence of all contaminants tested, and thus chloramine (or combined chlorine)
residual does not appear to provide a reliable means of contaminant detection.
These studies also indicate that total organic carbon (TOC) is a particularly useful parameter for detecting
the presence of many organic compounds, with a sensitivity ranging from -0.5 mg/L to more than 1
mg/L, depending on baseline levels and variability. Even at the upper end of this range, most organic
contaminants should trigger a change in TOC concentration at concentrations well below the lethal
concentration.
Conductivity was observed to respond slightly to some inorganic contaminants, including some metals,
but the response was not as strong as that observed for free chlorine residual and TOC. Nonetheless,
conductivity has demonstrated the potential for detection of some contaminants that do not alter chlorine
or TOC. Generally, higher concentrations of contaminants are needed to trigger a response from
conductivity sensors.
Beyond free chlorine residual, TOC, and conductivity, other water quality parameters may provide
supporting information about potential contamination. Oxidation reduction potential (ORP) will
generally behave similarly to chlorine residual, and can be used to corroborate an observed change in the
chlorine residual. ORP may also serve a more prominent role in systems that use a chloramine
disinfectant residual as certain oxidation reactions can occur without reacting with chloramines. pH is
important to aqueous chemistry and may be useful in understanding observed changes in other
parameters, such as free chlorine residual. Studies have generally shown that turbidity is an erratic and
unreliable primary indictor of contamination; however, as with pH and ORP, it may be useful in
understanding changes in other measured parameters.
Other parameters, beyond the basic parameters described above, can be considered in the design of the
water quality monitoring system. While it is of primary importance that selected parameters relate to the
primary objective of the system - detection of contamination, other factors to consider in selection of
water quality parameters include dual-use applications and sustainability.
The collected water quality monitoring data may provide useful information and benefits to ongoing
operation of the distribution system and water treatment processes. Existing distribution system water
quality data should be reviewed in terms of occurrence, average concentrations, and variability to identify
May 2007
32
-------
Planning for WS-CWS Deployment
other parameters that may provide additional information useful to other objectives relating to water
quality or system operations.
The sustainability of monitoring for different parameters should be considered. The utility's existing
online water quality monitoring program should be reviewed in terms of equipment, performance, and
maintenance requirements. Research and industry literature, evaluations, performance studies, etc. should
be reviewed to gather information on the performance and maintenance requirements of various
technologies. Conduct a preliminary assessment regarding the sustainability of monitoring station
designs which incorporate various sensors and technologies. Only those technologies which can be easily
and affordably maintained in an acceptable operating condition will be useful in a monitoring system over
a long time period.
Based on identified monitoring parameters, a preliminary selection of monitoring instruments and
analyzers can be made. The selection may be based on the utility's direct experience with various
analyzers, consultation with other utilities regarding their experience with equipment, and a review of
manufacture technical literature and third party technology evaluations (Environmental Technology
Verification [ETV], Technology Testing and Evaluation Program [TTEP], National Sanitation Foundation
[NSF], etc.).
The parameters selected to be included in the system need not be limited to traditional water quality
parameters, or those parameters used in previously published contamination warning system pilot studies.
However, the selected parameters should relate to the primary objective of contamination monitoring as
discussed under "system design considerations."
Depending upon the identified needs, costs and constraints, it may be beneficial to use a tiered system in
which two or more monitoring station designs, each with different costs and capabilities, are deployed. A
tiered design may allow for more monitoring locations, but at the cost of reduced contaminant coverage at
locations with the simplified design. A clear understanding of the contaminant detection potential of each
system, and the resulting detection compromises throughout the distribution system is necessary to most
effectively deploy a tiered system.
4.2.2 Design and Implementation Approach
The activities described in Section 4.2.1 describe a process for pre-design of water quality monitoring
stations, culminating in selection of parameters to be monitored as well as the potential to use tiered
designs with different levels of detection capability. The approach for design, installation, and operation
of water quality monitoring stations based on the pre-design may include the technical considerations and
specifications described below.
Design:
The first phase of monitoring station design should include a visit to each candidate field installation
location. An evaluation of each site should consider:
Size constraints - Confirm sufficient access and space at the site for installation/fabrication and
regular maintenance of analyzers
Environmental conditions - Temperature, humidity, vibration, and air quality should be
considered as they pertain to the monitoring system as well as the maintenance personnel
Site requirements - Ensure availability of sample water, drain, electric power, and the ability to
route conduit for data communications. Depending upon site conditions, it may be necessary to
condition or regulate the supply water, pump the sample drain, upgrade or modify the site
electrical equipment, or locate communications equipment remote from the monitoring
equipment. It is also important to verify that the flow of water through the monitoring station is
adequate to produce detection times on the order of two hours or less. In some cases, this may
require water to be bypassed to drain in order to reduce residence time.
May 2007 33
-------
Planning for WS-CWS Deployment
Security and accessibility - Monitoring station level of security from tampering and disruption at
each location, and degree of accessibility for utility personnel at any time. At non-utility owned
sites, it may be necessary to enclose the monitoring system in a locked enclosure. Ensure that
water utility personnel will have access to the monitoring station at any time in event of a
contamination occurrence.
Based on the site surveys and the analyzers identified during pre-design, the list of selected sensors and
ancillary equipment should be finalized for each site and each tier of monitoring system. Other
considerations to be addressed include the following:
Remote sampling equipment details and capabilities
Control and data systems including Programmable Logic Controllers (PLCs) and local data
loggers
Communication methods, such as radio, telephone, internet and associated equipment for the
selected method
Power assurance, such as an emergency generator or other uninterrupted power supply, that meets
reliability and duration criteria required for the system
Physical design of the water quality monitoring station (or multiple stations if tiered approach is used) can
proceed in the following areas:
Frame or panel design - Depending upon whether the monitoring stations will be permanently or
temporarily placed in the assigned locations, different types of supporting structures may be
required. Specific requirements particular to the surveyed installation sites should also be
considered. Placement of analyzers and other components should be considered to result in a
system of the required overall size.
Electrical system design - The electrical power supply as well as the control, data and
communication components should be adequately routed, enclosed and configured.
Plumbing system design - Provide sample water to the instruments at a pressure which is suitable
for the instruments. Use of a pressure regulator may be required to reduce the distribution system
pressure to the required range. Plumbing components which will not corrode, plug or easily
break are recommended. The ability to monitor and regulate flow to individual instrument or
groups of instruments is useful, however rotometers or other regulating devices should be
carefully selected to ensure proper operation. Use of a coarse strainer at the system inlet will
prevent small particles from entering the monitoring instruments or components and impacting
operation.
Sampling system design - An automated system should be included to collect a water sample
when potential contamination is detected. Design factors to consider include the quantity of
sample to be collected to meet the requirements for laboratory analysis; materials of construction
of the sample container and components to preclude contamination; method of activation -
automatic or operator initiated; personnel protection from potentially contaminated sample water.
Obtain any required reviews or approvals of the design.
To illustrate the concepts discussed in this section, a schematic of the water quality monitoring station
design used in the initial Water Security initiative pilot is shown in Figure 4-2.
May 2007 34
-------
Planning for WS-CWS Deployment
PLC/
Electrical
Panel
Radio
Panel
1.J.J
roc
analyzer
a ;a
Figure 4-2. Example Water Quality Monitoring Station Design used in the Initial Pilot
This design is based, in part, on a requirement for a mobile, easily relocated station. This example design
includes a side-mounted TOC, PLC, and radio panel to facilitate installation at sites where space is
limited or where it is necessary to move the system through small doorways or hatches to maneuver it
into place. Some of the additional hydraulic and mechanical features incorporated in this design include:
Supply and Drain Hoses:
o For the ease of installation and relocation of the station, flexible hoses were used.
o A high quality hose that complies with American Water Works Association (AWWA)
standards was used to prevent crimping.
Supply and Drain Pipes:
o Brass supply piping and fittings are used for durability, ease of installation, and most
importantly because they do not leach organic materials that could confound analysis.
Sturdy CPVC piping and fittings may also be used, however careful attention to structural
stability is required, as well as a flush-out period to purge the system of trace plastic and
adhesive chemicals prior to placing the system on line.
o Flexible tubing routes sample flow from the supply manifold and flow regulating
rotometers to the individual instruments.
o CPVC drain piping is used, and generously sized to ensure free flowing gravity drains.
o A manual sample supply shut-off valve provides easy isolation of the system in the event
of a substantial leak.
o A 40-mesh Y-strainer prevents entrained particles from the distribution system from
entering sensitive analyzer components.
o An actuated solenoid valve routes flow to an appropriately sized container to collect a
sample for laboratory analysis when possible water contamination is sensed (not shown).
Flow Regulation:
o A downstream pressure regulating valve provides sample water at the proper pressure to
the monitoring instruments.
o Flow regulating rotometers are provided to control and confirm sample flow to each
instrument or group of instruments
May 2007
35
-------
Planning for WS-CWS Deployment
o A manual bypass valve is provided at the downstream end of the sample supply manifold.
The outlet of this valve is routed directly to the system drain. The valve provides the
ability to easily flush the supply pipe. For monitoring systems which are located a
significant distance from the distribution system main, the bypass may be left partially
open during normal system operation to reduce the travel time from the main to the
station, thus reducing the time to detection of a water quality anomaly.
Electrical Panel:
o All necessary electrical feeds and monitoring/control signals are routed through an
electrical panel to simplify the design and allow for ease of installation.
Radio Panel
o For a system which communicates via radio, it may be beneficial to locate the radio
equipment in a dedicated panel, rather than including them in the electrical panel. Doing
so facilitates establishment of a radio link, which may require that the antenna be located
a distance away from the monitoring station.
o The data communications and analysis aspects of a contamination warning system are
further discussed in Section 4.3
The design depicted in Figure 4-2 serves as a reference only and is not to be considered a preferred
design. The ultimate design of monitoring stations should be determined by the installing utility and
tailored to the unique requirements of each application.
Fabrication and Testing:
A contract should be established with a qualified fabricator who is familiar not only with
electrical and instrumented systems, but also with standards and requirements for piped and tubed
hydraulic systems. A sufficiently detailed fabrication specification should be developed and
included as part of the contract.
Instruments and other major components may be purchased by the utility or by the fabricator.
Some cost savings may be realized if purchased by the utility, as well as better assurance and
control over delivery schedules.
As with any instrumentation and control system, a factory acceptance test should be conducted to
ensure that the specified components have been used, that the wiring is to specification and
correctly installed, and that all items function as necessary. Inspection and testing of hydraulic
components is also necessary to ensure proper operation and leak-free assembly. Identification
and correction of errors or problems at the factory can be accomplished with less impact to cost
and schedule than those found during site installation. Manufacturer specifications for some
sophisticated analyzers recommend that power not be applied until they are installed in the field
and connected to the water source. For these types of instruments, be sure to include an
appropriate warning in the fabrication specification, and not simply rely on warnings included in
the analyzer manual or packing documents.
Installation:
Depending upon the nature of the designed systems and the installation location, it may be
necessary to contract with a separate entity for installation at the monitoring sites. For example,
systems which are fully factory fabricated, but which are to be installed in facilities with difficult
access may require the services of a qualified mechanical contractor who can rig the units into
place. Easy access locations may be installed or fabricated in place by the instrument contractor.
Other locations may require the skills of both types of service providers.
Installation of the monitoring stations will involve the coordination of a number of entities
involved in activities ranging from delivery of the fabricated units to the installation site to final
inspection and approval of the installed equipment. A separate installation specification may be
beneficial so that the installer is aware of all site conditions and expectations. The specification
should clearly define the exact location for installation of the unit as well as details for
connections to power, water, communications, and drain. A comprehensive installation and
May 2007 36
-------
Planning for WS-CWS Deployment
inspection schedule can be developed to ensure that contractors and inspectors are available at
key points of the project. Finally, it is important to clearly identify all required inspections for the
installed systems so they can be scheduled into the installation timeline.
Start-up and Baseline:
Adequate time and resources should be allocated so that the monitoring station can be commissioned and
brought to a level of operations which is supported by a high level of confidence from the operating and
maintenance personnel.
Typical commissioning activities include:
Initial configuration and calibration of instruments. Acquire calibration and operating solutions
and reagents from the instrument vendor or other sources. Note that some reagents have limited
shelf life from the date of manufacture, so delayed delivery may be desired.
Configuration and testing of communications from the monitoring station to the central control
facility
Signal testing and verification at each monitoring station and at the central control facility
Validation of water quality monitoring station performance (e.g., by routine comparison of sensor
measurements with grab samples analyzed with an accepted, independent method).
Operation and Maintenance:
Operation and maintenance activities include the following:
Documentation including, as-built specifications, Operations and Maintenance (O&M) manual,
etc.: Purchased instruments and controls equipment are typically delivered by the manufacturer
with operating and maintenance instructions. The monitoring station fabricator should also
provide catalog and O&M documents for all components provided for the system. One complete
set of O&M documents should be made available at the site of each monitoring station, or one set
available at a central location.
System design drawings should be marked-up and modified to reflect the as-build condition of
each monitoring station. This is especially true for electrical design drawings. Accurate drawings
should be kept with the system O&M documents and readily available to maintenance personnel
and operations technicians.
In order to maintain the high level of availability that is necessary for contaminant monitoring
systems, a detailed maintenance and calibration plan should be developed and followed.
Most instrument technicians are able to perform routine maintenance checks such as confirming
sample flow rates and refilling reagent supplies. Some complex analyzers require annual or semi-
annual service which may be best left to factory service personnel. A factory service contract,
including the cost of replacement consumables may be beneficial.
A stock of replacement parts or maintenance items should be kept in supply to enable quick repair
in the event of unexpected component failure. For components that have a limited shelf life, a
limited quantity of such parts should be maintained, and they should be cycled into field use prior
to expiration.
For monitoring stations that are installed in facilities not owned by the water utility, it is
important to maintain contact with the facility host and adhere to the hosting terms of agreement.
O&M schedules delineating field actions for various maintenance and calibration intervals along
with record keeping requirements (i.e., log book documentation)
Evaluation and Refinement:
Maintenance records can provide valuable information regarding the performance and usefulness
of selected monitoring instrumentation. Equipment that does not provide useful data in a cost
effective manner should be evaluated for replacement with another technology or manufacturer.
Periodic comparison of sensor measurements with laboratory measurements can help to ensure
that received data values are an accurate representation of the actual water quality. Erroneous
readings can result in false positive warnings or missed suspicious events.
May 2007 37
-------
Planning for WS-CWS Deployment
The state of available sensor technologies and changing contaminant monitoring trends and
priorities should be followed. As monitoring equipment and recommended operating strategies
evolve, utilities should evaluate whether changes should be made to their monitoring stations.
4.2.3 Available Tools and Resources
The following tools and resources are available to support the design, installation, operation, and
evaluation of online water quality monitoring stations for the online water quality monitoring component
of a contamination warning system:
American Society of Civil Engineers. Interim Voluntary Guidelines for Designing an Online
Contaminant Monitoring System. US. EPA Cooperative Research and Development Agreement,
X-83128301-0, December 9 2004.
Grayman, Walter M., et al., Design of Early Warning and Predictive Source-Water Monitoring
Systems. AWWA Research Foundation and American Waterworks Association. 2001.
Hall, J. et al., "Online Water Quality Parameters as Indicators of Distribution System
Contamination," mJAWWA, 99:1:66. January, 2007.
Hall, J. et al., "Contaminant Minimum-Dose Threshold Concentrations for Water Quality
Sensors." USEPA Report, December, 2007. Available only through WaterlSAC.
USEPA, "Water Quality Sensor Response to Potential Chemical Threats in a Pilot-Scale Water
Distribution System." USEPA Report, January, 2006. Available only through WaterlSAC.
Szabo, J. et al., "Water Quality Sensor Response to Contamination in a Single Pass Water
Distribution System Simulator." USEPA Report, January, 2007. Available only through
WaterlSAC
Szabo, J.G., Hall, J.S. and Meiners, G.C. "Water quality sensor responses to injected
contaminants in a chloraminated pipe loop ". American Water Works Association (AWWA)
Water Security Congress, Technical Session TUE6: Technology Forum B, Washington, DC,
September 10-12, 2006
Hargesheimer, Erika, et al, eds. Online Monitoring for Drinking Water Utilities. AWWA
Research Foundation and American Water Works Association. 2002.
International Standard ISO 15839. Water quality On-line sensors/analysing equipment for
water Specifications and performance tests. 2003
ETV and TTEP evaluation reports of online monitoring equipment. Available at epa.gov/etv.
4.3 Communications Architecture
The objective of this task is to provide a long-term, cost effective communications system for relaying
monitoring station data to the SCADA system software and ultimately to the Event Detection System
(EDS) in a timely manner.
There are numerous technologies available today to relay SCADA information from remote sites. These
include radio (e.g., spreadspectrum), copper (e.g., cable, telephone), cellular, and fiber to name a few.
Each technology has advantages and disadvantages. It is usually desirable for a utility to choose a
communications system that is easy to support and is cost effective.
Below is a list of activities and functions that may be required to provide a communication system for
implementation and operation the online water quality monitoring component of a contamination warning
system.
May 2007 38
-------
Planning for WS-CWS Deployment
4.3.1 Pre-Design
Prior to design, numerous activities are necessary to produce a successful communications project.
Staffing the design and implementation team. It is vital that all the stakeholders are represented
on the design team to provide consensus on all decisions. Otherwise, the project may be riddled
with delays, lack of trust, and poor quality work.
o Suggested Team Members include:
Public Works or Utility:
SCADA Manager
IT Manager
IT Network Engineer
Operations and Maintenance Manager
Engineering
End users such as Water Quality Staff and System Operators
Consultants including representatives from other project groups in reviews.
Determine the schedule for the project. Allow for long delays in dealing with communications
studies, right of way issues, communication providers, and contracting of sub-contractors. It will
be vital to have a project manager with aggressive communication skills to implement this task.
Define the requirements of the communication system needed to support the proposed network of
online water quality monitoring stations. Consider communication requirements for the enhanced
security systems deployed as part of the contamination warning system. Consider the rate and
quantity of data to be transmitted.
Evaluate existing communication system architecture used to transmit data and commands
between remote facilities and the utility operations center. Assess ability of existing system to
accommodate the proposed water quality monitoring network. Take into consideration the need
for future growth and changing technologies.
If existing communications systems are unable to meet the requirements evaluate alternatives.
Identify constraints on communication alternatives, e.g., hilly terrain may make radio
communications cost prohibitive. Consider an alternative communication system for just the
contamination warning system or replacing the entire communications system.
4.3.2 Design and Implementation Approach
The activities described in Section 4.3.1 describe a process for pre-design of a communications
architecture for a water quality monitoring network, culminating in documentation of initial requirements
and a preliminary assessment of the utility's existing communication architecture. The approach for
design, installation, and operation of a communications system may include the technical considerations
and specifications described below:
Design:
Select communication technology (e.g., radio, digital, phone lines, fiber, etc.). Multiple
technologies may be desirable or necessary in some applications. Determine the communications
protocol (e.g., Ethernet). Evaluate for integrity, loss of service, reliability, and security.
Develop overall communication architecture. Include all stakeholders identified in the pre-design
phase in the development of the communications architecture.
Obtain necessary reviews and approvals on the proposed architecture. Schedule frequent review
opportunities that include all stakeholders, and provide a high level of consensus along the way.
May 2007 39
-------
Planning for WS-CWS Deployment
Implementation:
Select service provider (if applicable), and establish any necessary contractual relationships.
Select an installer, and establish any necessary contractual relationships. Installer's experience
with all the technologies needed to complete the work should be heavily weighted in their
selection.
Identify roles and responsibilities for procurement, installation, and testing of various components
of the communications architecture.
Work with service provider to get components installed and configured.
Test communication pathways between water quality monitoring sites and operations center.
Develop location-specific installation specifications.
Procure system components that will not be provided by the service provider.
Develop installation and inspection schedule. Coordinate this schedule with installation schedule
for water quality monitoring stations.
Install remote communications systems at water quality monitoring stations.
Program PLCs at each monitoring location to record data from sensors at specified polling
interval.
Verify that the installed communication system at each water quality monitoring site is
transmitting data back to the operations center.
Start-up and Baseline:
Verify data processing and storage at each water quality monitoring location.
Verify data integrity during transmission from water quality monitoring locations to operations
center.
Stress test for communications requirements, including total bandwidth available.
Operation and Maintenance:
Verify that firmware and software is current with respect to vendor provided updates, patches,
etc.
Document all communications related settings. This includes all devices that create, push, or
receive data within the architecture. This is important for calibration and equipment replacement
during routine maintenance.
Collect all operation and maintenance documentation.
Annual inspection and maintenance of communications hardware (or follow the utility's existing
O&M plan for communication systems).
Maintain terms of agreement with communications service provider.
Evaluation and Refinement:
Annual assessment of communication system performance relative to performance specifications
documented in the final design. This should include communications integrity and security.
4.3.3 Available Tools and Resources
The following tools and resources are available to support the design and implementation of a
communication architecture as an element of the online water quality monitoring component of a
contamination warning system:
National Institute of Standards and Technology, Guide to Supervisory Control and Data
Acquisition and Industrial Control (SP800-82). September 2006
Instrumentation, Systems, and Automation Society, Manufacturing and Control System Security,
November 2005.
National Institute of Standards and Technology, Information Security (SP800-53). February
2005
May 2007 40
-------
Planning for WS-CWS Deployment
Telecommunications Industry Association, Data Service Options for Spread Spectrum Systems
(TIA/EIA/IS-2000-A), March 2001
4.4 Data Management and IT Architecture
The objective of this task to provide support for the processing of data needed to implement the online
water quality monitoring component of a contamination warning system. This includes protocols for
receiving data from the monitoring stations, incorporating the data into a SCADA system, delivering it to
event detection tools, providing warnings of events, providing security of data flow, ensuring backup of
systems and data, and passing information to other related systems or entities.
This task is a key function to the success of moving information from sensors to SCADA systems to EDS
tools. The task lead should be incorporated into all other project groups to insure a complete working
system. Section 4.4.1 provides a list of activities and functions to develop a data management and IT
architecture for the online water quality monitoring component of a contamination warning system.
4.4.1 Pre-Design
Prior to design, numerous activities are necessary to produce a successful data management and IT
architecture:
Staffing the design and implementation team. It is vital that all the stakeholders are represented
on the design team to provide consensus throughout the project. Otherwise, the project will be
riddled with delays, lack of trust, and poor quality of work.
o Suggested Team Members are:
Public Works or Utility:
SCADA Manager
IT Manager
IT Network Engineer
Operations and Maintenance Manager
Engineering
End users such as Water Quality Staff and System Operators
Consultants including representatives from other project groups in reviews.
Determine the schedule for this task. Consider incorporating this schedule into all project tasks.
Nearly all the contamination warning system project tasks rely on some form of data
management. Members of this group should be flexible to project changes and good
communicators. Change management skills are critical.
Define the requirements of the data management system needed to support the proposed network
of online water quality monitoring stations. Two primary requirements are to manage data
collected from the online water quality monitoring stations and support EDS tools. Other
requirements might include, data storage, backups, remote connectivity, protocol standards,
software management tools, and data transfer speeds.
Evaluate potential options for managing data from online water quality monitoring stations.
Consider options for collection by the SCADA system and transfer of data to the EDS tools.
Evaluate potential deployment options for EDS tools considering factors such as:
o Proximity to source data
o Reliability
o Security
o System compatibility
o Computing and monitoring resources
Assess the ability of the existing SCADA system to serve these data management functions,
including hosting the EDS tool(s). For most installations, this will be the ideal architecture.
Consider implications to the health and welfare of the SCADA system.
May 2007 41
-------
Planning for WS-CWS Deployment
4.4.2 Design and Implementation Approach
The activities described in Section 4.4.1 describe a process for pre-design of a data management
architecture for a water quality monitoring network, culminating in documentation of initial requirements
and a preliminary assessment of the utility's existing SCADA and IT architecture. The approach for
design installation, and operation of a communications system may include the technical considerations
and specifications described below.
Design:
Develop an overall IT architecture. The architecture should include:
o Network diagrams including hardware and software resources
o Flow diagrams
o Personnel interaction
Obtain necessary reviews and approvals on the proposed architecture from all stakeholders.
Stakeholders may include project members from other teams.
Specify all hardware and software to be purchased that meets utility or facility standards.
Implementation:
Procure required hardware and software
Coordinate installation and startup plan with project task leaders and stakeholders
Install and test hardware/software according to the approved architecture
Program SCADA system with tags for all water quality monitoring locations. Build screens for
monitoring and EDS Tool warnings.
Install and configure data migration utilities if needed
Design and configure data historian and backup system
Start-up and Baseline:
Verify data transfer among the system components
Verify data integrity at SCADA system relative to data collected at the PLC
Verify data storage and archive
Verify operations of the EDS system within the overall IT architecture
Operation and Maintenance:
Verify that software is current with respect to vendor provided updates, patches, etc.
Document all configurations and settings
Collect all O&M documentation
Annual inspection and maintenance of IT hardware (or follow the utility's existing O&M plan for
IT systems)
Evaluation and Refinement:
Annual assessment of IT system performance relative to performance specifications documented
in the final design.
4.4.3 Available Tools and Resources
The following tools and resources are available to support the design and implementation of a data
management architecture as an element of the online water quality monitoring component of a
contamination warning system:
National Institute of Standards and Technology, Guide to Supervisory Control and Data
Acquisition and Industrial Control (SP800-82). September 2006
Instrumentation, Systems, and Automation Society (ISA), Manufacturing and Control System
Security, November 2005.
May 2007 42
-------
Planning for WS-CWS Deployment
Inter National Committee for Information Technology Standards, Information Technology -
Security Techniques (17799-2005)
National Institute of Standards and Technology, Information Security (SP800-53). February
2005
4.5 Water Quality Event Detection
Water quality monitoring stations only provide information about general water quality conditions at a
specific location and time, and in isolation are not well suited to detect contamination. This is
compounded by the fact that water quality at a given location can be highly variable, with several factors
affecting water quality at a given time including pump and tank operations, system demand, and source
water quality. Sophisticated algorithms incorporated into EDS tools can efficiently mine the large
amount of water quality data produced by these monitoring stations and detect anomalies that may be
indicative of contamination or other water quality problems that relate to dual-use application of the water
quality monitoring system. Thus, the EDS tool implemented as part of the water quality monitoring
component of a contamination warning system is critical to the performance of the system and has a
significant influence on the overall reliability of the system. Likewise, EDS tool performance impacts
sustainability because a system plagued by false alarms will be quickly ignored and ultimately forgotten
unless the performance can be improved through tuning of the EDS tool or inclusion of more data. The
EDS tool may also impact contaminant coverage and timing of detection. Given that EDS performance
impacts four aspects of the design basis (reliability, sustainability, timeliness, and contaminant coverage),
selection of the tool is of paramount importance to the water quality monitoring component of a
contamination warning system. Section 4.5 presents information useful to the selection and
implementation of an EDS tool for water quality monitoring, with an emphasis on activities related to
planning. When applicable, references to additional resources are included and summarized in Section
4.5.3.
4.5.1 Planning
Unlike the previous four design elements of the water quality monitoring system, EDS tools will
commonly be off-the-shelf products, and thus do not require the same level of design and pre-design as
the other elements. However, given that the application of EDS to water quality is relatively new and that
there are few published, third-party evaluations of these tools, careful consideration should be given to the
basis for selection of an EDS tool. Thus much of the effort associated with the planning stage of water
quality event detection is collection of information that will lay the ground work for design and
implementation of a selection study. Suggested information collection efforts include:
A review of utility water quality and operational data
A preliminary assessment of metrics against which EDS tool performance can be evaluated
A market survey to identify potential EDS tools
A summary of EDS tool specifications for candidate tools
An assessment of potential deployment environments for the EDS tools
The remainder of this section describes considerations relating to each of these planning activities and,
where appropriate, references tools or resources that may be useful in this process. The planning phase
produces a list of candidate EDS tools and a summary of specifications, performance study results, and
other available information for each tool.
Part of planning for an EDS selection study is conducting a thorough review of all utility data that may be
useful for water quality event detection, such as water quality and operational data. Once water quality
parameters have been selected, per the pre-design activities described in Section 4.2.1, a review of
existing distribution system water quality data may provide insight regarding typical values and expected
variability. While the degree of variability in water quality parameters, especially free chlorine residual,
May 2007 43
-------
Planning for WS-CWS Deployment
is highly dependant on location, a general sense of distribution system variability may be useful in the
design of the selection study and may even impact monitoring network design (see Section 4.1.1).
Operational data is important because system operations can have a dramatic and predictable impact on
water quality. Incorporation of operational data into water quality event detection can improve the
performance of the tools as it allows for incorporation of cause and effect relationships that result in water
quality changes. However, in order to use this data, the utility should have a firm understanding of the
available data and knowledge regarding how operational changes impact water quality throughout the
system. As with water quality variability, these relationships are location-specific. At the planning phase
it is sufficient to characterize the operational data that is available so that it can be used in either the
selection study or during implementation.
Also important in the planning process is a precise characterization of the requirements for an EDS tool.
Each utility's needs, priorities, and available resources are unique, so a ranking of the important aspects
of an EDS tool is key in tool selection. A sample ranking of objectives could be:
1. An EDS tool should be compatible with current IT infrastructure and be easily integrated with
existing SCADA system or other applications
2. The tool should detect a high percentage of potential contamination events (e.g., 99.99%)
3. The tool should have a minimal number of false alarms (e.g., a maximum of one per week)
4. The tool should have minimal maintenance requirements and should be supported by the tool
developer
5. The tool should have dual-use functionality and be able to detect water quality anomalies that
could arise from variety of causes that may be of concern to the utility (e.g., cross-connections)
6. Minimal personnel training should be necessary to implement the tool
7. The tool should be able to handle noisy and imperfect data
8. The tool should have minimal cost, both initial and ongoing, including any necessary software
and hardware
The tables below describe some standard performance measures that can be used to evaluate and compare
EDS tools. Table 4-3 describes some performance measures that quantify the detection capability of an
EDS tool, and in this table an event is defined as a continuous period of time during which water quality
is anomalous at the monitoring locations.
Table 4-3. Standard Measures for Evaluating EDS tool Performance
Performance
Measure
Specificity
Sensitivity
False Alarm Rate
Average False
Alarm Length
Median Detection
Time
Description
The percent of time for which the EDS
tool correctly does not alarm
The percent of events that the EDS
correctly identifies
The frequency of false alarms
The average length of a false alarm
The median time it takes the EDS tool
to detect an event relative to the time
the event reaches the monitoring
station
Example
The tool correctly does not alarm 99.5% of the time
The tool detects 98% of events
On average, the tool produces one alarm per week
The duration of a false alarm ranges from 2 minutes
to 3 hours, with an average of 15 minutes
The delay between the time the event is at the
monitoring station and the EDS tool alarms ranges
from 2 minutes to 84 minutes, with an average of 6
minutes
The performance of the tool, as characterized by metrics such as those in Table 4-3, should guide the
selection of a tool, other factors relating to the operation, usability, and maintenance of the tool are
important considerations as well. Table 4-4 describes other factors that may be considerations in the
selection of an EDS tool.
May 2007
44
-------
Planning for WS-CWS Deployment
Table 4-4. Sample Measures for Evaluating EDS Software
EDS Software
Performance Measures
Initial Cost
Recurring Cost
Calibration or Training
Compatibility
Customization
Usability
User Interface
Efficiency
Reliability
Developer Support
Description
Fee for purchase and installation of EDS tool
Costs incurred for labor materials necessary to support the operation and
maintenance of EDS tool
Procedures for training EDS tools, ability of staff to learn and implement training
procedures, and requirements for re-training
Compatibility with existing hardware, software, schemas, and other aspects of the
planned deployment environment
Ability to customize EDS tool to a particular utility environment, including:
monitoring locations, water quality parameters, operational data, alarm threshold,
and information displays. Ability to analyze WQ data from multiple locations in near
real-time.
Level of skill required to train, operate, and maintain the EDS tool
Capability of the user interface to navigate the major features of the tool and to
display information in a usable format.
Speed and memory requirements for software
Percentage of time EDS operates as designed
Availability of adequate, long-term support for the operation of the EDS tool
Generally, two preparatory processes are carried out by an EDS software developer before an EDS tool is
deployed at a water utility in real-time operation. First, the tool is trained on a dataset from each location
that will be monitored in real-time. This training dataset should be free of anomalies or events so that it
can be used by the EDS tool to "learn" typical water quality patterns. For example, a 0.2 mg/L change in
chlorine at one location might be normal under certain conditions, whereas it would be extremely rare
(and considered anomalous) at another location. Once trained, the tool would respond differently to such
a change at the two locations (it would likely alarm for the second location but not the first).
Once training is complete, the EDS tool developer will typically work with the utility to configure the
tool, or adjust variables within the tool to maximize event detection performance (sensitivity) while
maintaining a false alarm rate that is acceptable for the utility. A receiver operating characteristic (ROC)
curve, which plots 1-specifity vs. sensitivity, can be helpful in quantifying the tradeoff between detection
capability and false alarm rates for a given tool. Ideally, the curve should hug the top left corner of the
graph, as this represents a low number of false alarms with a high probability of detection. Figure 4-3
provides an example ROC curve.
False Alarm Rate (FAR)
Figure 4-3. Example ROC Curve
In order to develop a candidate pool of EDS tools for consideration in the selection study, a market survey
or literature review is proposed. This information gathering exercise should consider the performance
metrics and EDS tool attributes discussed above, such as what information is readily available. Potential
sources of information include technical and scientific literature, vendor information and websites, and
information from utilities currently using or considering using EDS tools. Note that there are a limited
May 2007
45
-------
Planning for WS-CWS Deployment
number of tools marketed specifically for event detection in water distribution systems. However, there
are many general tools for event detection or anomaly detection that could potentially be adapted to the
water quality domain. Adaptation of general event detection tools to the water quality domain may
require additional tool customization and/or integration efforts. The Overview of Event Detection Systems
(USEPA, 2005c) describes some available tools, and also includes examples of how tools have been used
at water utilities. Additional information sources are included in Section 4.5.3.
The results of this market survey can be used to develop a list of candidate EDS tools for consideration in
the selection study. More detailed information and product specifications should be collected for each of
the candidate tools as this information will be critical to the design and conduct of the selection study as
discussed in Section 4.5.2.
4.5.2 Implementation Approach
The activities described in Section 4.5.1 describe a process for performing a preliminary assessment of
EDS tools, culminating in a list of candidate tools and a summary of performance specifications. The
approach for the selection, implementation, and operation of an EDS tool may include the technical
considerations and specifications described below. Prior to selecting an EDS tool, a preliminary concept
of operations as described in Section 3.2.1 should be developed. This exercise will help to define
requirements for how data can be provided to the tool for analysis and may place some constraints on tool
selection. Development of the concept of operations should also be tightly coupled with communications
and data management activities.
Planning-Selection Study:
Define requirements for EDS tool performance, compatibility, usability, features, support, etc.
Precisely define the metrics and attributes that will be considered in the EDS tool selection study,
and document how each will be assessed in either a quantitative or qualitative manner.
Perform initial analysis of candidate EDS tools against requirements to reduce the pool of
candidate technologies to those that meet the most critical qualitative attributes.
From the remaining list of candidate tools, develop quantitative basis for selection of an EDS tool
for deployment. Options for a selection study include:
o Utility-specific EDS tool evaluation using data collected from each water quality
monitoring location after the monitoring stations have been found to be producing valid
data. (Guidance on the design of an EDS evaluation study is currently under
development by USEPA based on experience at the initial Water Security initiative pilot.)
o Third party technology evaluations, either through established programs such as ETV,
TTEP, NSF, or through research programs.
o EDS tool performance documented in peer-reviewed literature and vendor-supplied
information.
o Experience and data from other utilities using EDS tools.
Select EDS tool(s) for deployment at the utility based on the results of the selection study.
Implementation:
Verify that the utility's IT system architecture can accommodate the selected tool. Modify the
architecture if necessary.
Collect water quality data from all installed water quality monitoring stations to support initial
training of the EDS tools. Use only data collected after it has been verified that the water quality
monitoring stations are producing reliable data.
Identify operational data that is expected to enhance EDS tool performance and the associated
operational logic that relates water quality at a specific monitoring location to distribution system
operations.
Collect relevant distribution system operational data to support training of the EDS tools.
May 2007 46
-------
Planning for WS-CWS Deployment
Train and tune the EDS tool(s) to each water quality monitoring location using the data described
above. If a utility-specific EDS tool evaluation was performed, data from the evaluation may be
useful in training the tool(s) for deployment.
Install and test EDS tool(s) according to the architecture developed under "data management."
Modify the concept of operations as appropriate to reflect the as-built design.
Start-up and Baseline:
Bring the EDS online and establish connectivity with SCADA or other automated water quality
data source.
Train utility staff on EDS tool operation and maintenance, as well as the concept of operations for
the water quality monitoring component of the contamination warning system.
Monitor EDS tool performance over the first several weeks of operation, and make adjustments
as necessary.
Operation and Maintenance:
Monitor and respond to EDS alarms in accordance with the concept of operations.
Log any anomalies detected by the EDS tool(s).
Update the EDS tool event library, if applicable (some tools keep a library of water quality types
observed at a given monitoring station so that a repeat occurrence of the water quality type can be
classified as normal).
Retrain the tool if new sensors are added, water quality changes significantly, or performance is
found to be unsatisfactory.
Verify that EDS tools and supporting software are current with respect to vendor-provided
updates, patches, etc.
Evaluation and Refinement:
Continuous documentation of performance during operation, e.g., false alarms, detection of true
water quality anomalies, and dual-use benefits.
Through simulated events, periodically evaluate the overall performance of the installed EDS
tool(s). If necessary, update EDS tool configuration to optimize performance.
Track and evaluate the development of new EDS tools.
4.5.3 Available Tools and Resources
The following tools and resources are available to support the selection, evaluation, and implementation
of EDS tools as part of the online water quality monitoring component of a contamination warning
system:
U. S. Environmental Protection Agency, 2005. "Overview of Event Detection Systems for
WaterSentinel."
http://www.epa.gov/safewater/watersecuritv/pubs/watersentinel event detection.pdf
U. S. Environmental Protection Agency, 2006. "Framework for the Evaluation of Event
Detection Software for Drinking Water Contamination Warning Systems."
Kroll, D., King, K. (2006). Real World Operational Testing and Deployment of an On-line Water
Security Monitoring Station. In Proceedings ofWDSA 2006 Symposium. Cincinnati.
Klise, K., McKenna, S. (2006). Multivariate Application for Detecting Anomalous Water Quality.
In Proceedings ofWDSA 2006 Symposium. Cincinnati.
Jarrett, R., Robinson, G., O'Halloran, R. (2006). On-line Monitoring of Water Distribution
Systems: Data Processing and Anomaly Detection. In Proceedings ofWDSA 2006 Symposium.
Cincinnati.
Umberg, K., Uber, J., Murray, R. (2006). Performance Evaluation of Real-time Event Detection
Algorithms. In Proceedings ofWDSA 2006 Symposium. Cincinnati.
May 2007 47
-------
Planning for WS-CWS Deployment
Hart, D., S. A. McKenna, K. Klise, V. Cruz, and M. Wilson, 2007. "CANARY: A Water
Quality Event Detection Algorithm," Proceedings of the ASCE World Environmental and Water
Resources (EWRI) Congress, Tampa, Florida, 2007
4.6 Staffing and Cost Considerations
Planning for the implementation of the water quality monitoring component of a contamination warning
system requires involvement of a wide array of utility personnel and potentially contractor staff. Costs
will be highly dependent on the utility's capabilities and intended enhancements. Therefore, the
remainder of Section 4.6 illustrates the staffing considerations and cost factors that are recommended for
consideration during project planning and pre-design. This section provides a summary of previously
discussed information that should be considered when developing preliminary staffing plans and cost
estimates. Cost considerations represent some unique aspects of implementation based on lessons learned
from the initial pilot.
4.6.1 Staffing
As mentioned above, staffing considerations are critical to the successful implementation of a
contamination warning system. Table 4-5 offers a quick overview of which staff may be necessary to
design, implement, and operate an online water quality monitoring program as part of a contamination
warning system and during which phases of implementation these personnel may be needed.
Table 4-5. Online Water Quality Monitoring Staffing Considerations
Division or Department
Water Quality
Information Technology
Engineering and Planning
Operations and/or
Distribution
Host Facilities
Implementation Stage
PD
X
X
X
X
D
X
X
X
X
I
X
X
X
X
X
PT
X
X
X
X
O&M
X
X
X
X
E&R
X
X
X
Comments
System end user. Defines system requirements.
Lead in selection of parameters and instruments.
Lead in monitoring network design. Lead in
operations and maintenance.
Designs, implements, and manages all IT systems
used to support online water quality monitoring.
Review and approval of designs. Responsible for
installation oversight and inspections. Support
monitoring network design.
Provides domain knowledge relating to system
operations. Responsible for monitoring water quality
alarms 24/7/365. Support installation and
maintenance of monitoring stations.
Provide access to facilities during installation and
maintenance activities. Contact utility in the event of
a problem that could impact the monitoring station.
PD = Pre-design; D = Design; I = Implementation; PT = Preliminary Testing; O&M = Operations and
Maintenance; E&R = Evaluation and Refinement
Building the team to implement this component of a contamination warning system will involve all
divisions within the utility. Therefore, it is important to have senior leadership involved and invested
during each stage to facilitate the resolution of cross-division issues. Several key personnel, like the IT
manager and the Water Quality division manager, would ideally be members of this and all other
component teams to facilitate the application of the system engineering principles outlined in Section 2.1.
Such involvement can be a significant commitment of time and resources for these individuals, but the
utility can reap substantial benefit in the long-term success of the system. Other team members'
participation will be less demanding, but still critical, and will vary depending on the utility-specific gap
between the initial conditions and the final planned capabilities of the contamination warning system.
Furthermore, it may be useful to obtain the help of consultants and contractors to aid the utility in the
implementation of an online water quality monitoring system, especially given that the level of effort
required to implement this component would go beyond the human resources at many utilities.
Consultants and contractors can be critical partners during each stage of the process by providing
May 2007
48
-------
Planning for WS-CWS Deployment
component-specific technical knowledge and installation experience, and by delivering training on the
new equipment, procedures and processes.
4.6.2 Cost Considerations
This section presents a summary of the design and implementation considerations discussed above that
may influence costs. This list also may include other factors that were encountered during
implementation of the initial Water Security initiative pilot and could be overlooked during cost
estimation in the absence of this experience. Although this list of cost considerations may not be
exhaustive, these factors, at a minimum, should be considered when planning.
Pre-Design:
Development of design objectives for each element of the online water quality monitoring
component
Assessment of existing communications, SCADA and IT systems
Design:
Development of preliminary concept of operations for the water quality monitoring component
Monitoring Network: assessment and calibration of hydraulic model used in the development of
the monitoring network design, followed by field verification of potential monitoring station
locations and development of installation specifications for selected locations
Monitoring Station: layout and design of multi-sensor instrument rack (or racks, if a tiered
network approach is implemented)
Communications and IT Architecture: design of communications and data management
architecture; testing of communication pathways and conduct radio survey if necessary
Water Quality Event Detection System: design of the event detection system architecture; design
and implementation of event detection system tool evaluation study
Implementation:
Monitoring Network: coordination of installation at non utility-owned monitoring station
locations
Monitoring Station: equipment procurement and rack fabrication; transportation and installation
of monitoring stations
Communications and IT Architecture: establishment of contract with communication service
provider(s); procurement, installation and configuration of communications and data
management hardware and software
Water Quality Event Detection System: procurement, installation, and testing of event detection
system hardware and software; training and configuration of event detection system tools
Preliminary Testing:
Monitoring Station: initial calibration and support; equipment shakedown and training on
operation, maintenance and alarm response
Communications and IT Architecture: test of installed data management architecture and
complete communication system
Water Quality Event Detection System: preliminary evaluation of event detection system
performance, and fine-tuning the configuration as necessary
Operations and Maintenance:
Monitoring Network: water and sewer credits to non-utility hosts of monitoring stations
Monitoring Station: development of written documentation; scheduled and unscheduled sensor
maintenance, repair and upgrades
May 2007 49
-------
Planning for WS-CWS Deployment
Communications and IT Architecture: monthly service fees for communications service
provider(s); development of written documentation; scheduled and unscheduled communication
equipment maintenance, repair and upgrades
Water Quality Event Detection System: scheduled and unscheduled EDS equipment
maintenance, repair and upgrades
Evaluation and Refinement:
Monitoring Network: ongoing hydraulic model update and recalibration; potential relocation or
addition of monitoring stations
Monitoring Station: equipment upgrades due to improvements in sensing technology
Communications and IT Architecture: equipment upgrades due to improvements in
communication technology
Water Quality Event Detection System: drills and exercises, including simulations; data analysis;
equipment upgrades due to improvements in event detection technology
Based on experience at the pilot utility, the most significant issues will likely be the capability of the
existing communication system and the upfront capital cost associated with the fabrication of the
monitoring stations. Unfortunately, it may not be possible to receive an accurate estimation of the costs
associated with each monitoring station until a prototype is created and unit costs can be calculated. The
use of a tiered monitoring station design may reduce the eventual equipment costs, but additional
prototypes would be necessary, maintenance and operation may be more complicated, and there will be a
reduced contaminant warning capabilities at locations with a simplified design. The end result should be
a balance between capability of the entire system and cost. Finally, the cost to operate and maintain even
a modest water quality monitoring network can be significant, and should be considered early in the
planning stage to ensure that the system built can be sustained.
May 2007 50
-------
Planning for WS-CWS Deployment
Section 5.0: Sampling and Analysis
Although a critical aspect of contamination warning systems, sampling and analysis is not considered an
early detection strategy. Rather, sampling and analysis serves the following three functions:
Baseline monitoring. Data from samples collected and analyzed during the initial stages of
implementation are used to create a "baseline" profile of contaminant occurrence in the
distribution system, as well as characterize possible matrix effects on method performance.
Maintenance monitoring. Ongoing sampling analysis activities maintain laboratory proficiency
for techniques that may otherwise only be used in response to a triggered event and are used to
continually update baseline data seasonally.
Triggered sampling and analysis. Sample collection and analysis in response to contamination
indicators by other contamination warning system components is part of the credibility
determination and consequence management process. Triggered analyses may be specific, based
on information available from other components, or may involve a broad screen to potentially
detect unknown contaminants.
Table 5-1 summarizes design and implementation considerations for the sampling and analy
component of a contamination warning system.
Table 5-1. Design Basis Considerations for Sampling and Analysis
sis
Design
Objective
Description
Design and Implementation Considerations
Capability
Can positively identify the presence of
any contaminant in the suite of target
analytes and above a well-defined
minimum reporting level.
Assess existing laboratory capability and capacity and
identify enhancements or establish laboratory networks.
Contaminant
Coverage
High detection potential for classes 1, 2,
3, 4, 7, and 12; Moderate detection
potential for classes 5, 6, 8, 9, 10, 11.
Methods and analytical approaches should be identified
for as many of the contaminant classes as possible.
Methods should be validated for use in drinking water.
Spatial
Coverage
Function of location, number, and
density of sampling stations, as well as
sample type (composite vs. grab).
Consider sampling at locations identified as priority
sites through sensor network design (Section 4.1) as
these could be the source of triggered sampling events.
Also consider aspects of the distribution system such
as source water, water age, and pipe material that
could result in variability or method interferences.
Timeliness
Function of sampling & analysis
frequency and the total time to process
the sample and analyze the results.
Baseline monitoring should occur at a frequency
sufficient to support data quality objectives, inform the
design of maintenance monitoring and provide an
understanding of method performance and variability in
the distribution system prior to full operation of the
contamination warning system. Maintenance
monitoring should occur at a frequency sufficient to
maintain laboratory capabilities, update baseline data to
account for seasonal variability, and support dual-use
applications.
Reliability
Function of the reliability of sampling and
analysis methods (high for established
techniques). Baseline needed for
reliable interpretation of results.
Methods utilized for baseline, maintenance, and
triggered sampling and analysis should be validated for
use in drinking water. In some cases where analytical
methods have not been fully validated for certain
classes of contaminants, procedures to demonstrate
initial and ongoing proficiency should be implemented
to support interpretation of results.
Sustainability
Provides utility with an opportunity to
exercise sampling and laboratory
protocols and may; provide information
about previously unknown contaminants
that occur in the system.
Assess whether or not to enhance in-house laboratory
expertise or rely on outside laboratories for support.
This decision will influence level of effort and costs. As
indicated previously, dual use applications should also
be considered in terms of sustainability of the program.
May 2007
51
-------
Planning for WS-CWS Deployment
The remainder of this section is organized according to the following design elements described:
Laboratory capability and capacity (Section 5.1). This includes consideration of the
contaminants to monitor, analytical laboratories that would be able to support both routine and
non-routine analyses, and analytical methods and data quality objectives.
Sampling and analysis activities (Section 5.2). This includes consideration of sampling
locations, sampling frequency, and ongoing sampling and analysis procedures.
Site characterization and field screening (Section 5.3). This includes consideration of the roles
that will be played by the utility and others investigating an incident, and consideration of the
field testing capabilities that will be needed to conduct preliminary assessments of contaminants.
Discussion of these design elements is presented in the phases of pre-design, design and implementation,
and available tools and resources. Section 5 concludes with a discussion of staffing and cost
considerations.
5.1 Laboratory Capability and Capacity
Establishing and maintaining adequate laboratory capability to address a range of potential contaminants
is a fundamental aspect of contamination warning system design and implementation. Although some of
capabilities may be met with an existing or expanded utility laboratory, analyses for at least some
contaminant classes should be performed by other laboratories and coordinated by the utility. In addition,
sufficient capacity for these analyses should be considered to ensure that a large number of samples does
not overwhelm the laboratory performing the analyses.
5.1.1 Pre-design
During the pre-design phase for laboratory capability and capacity, the utility's design objectives should
consider the following:
Identifying potential contaminants of concern
Identifying qualified laboratories that can support (and will commit to supporting) sample
analysis needs that cannot be met through existing or expanded utility laboratory capabilities
Identifying the analytical methods that will be used for each of the targeted analytes
Evaluating the capabilities and credentials of each laboratory with respect to the identified
methods to identify the most effective assignment of laboratory roles
Identifying Potential Contaminants of Concern
The sampling and analysis component should address the contaminant classes presented in Table 5-2. As
noted in this table, consideration should be given for an analytical approach for each contaminant class.
Although a goal of implementation is to maximize the number of analytes monitored in each class, there
are inherent limitations in monitoring capabilities for some contaminant classes due to the need for
specially equipped laboratories and the proximity of, or access to, these laboratories by the utility.
Table 5-2. Considerations for Analytical Approach to Establishing Sampling and Analysis
Capabilities by Contaminant Class
Class
1
2
Description
Petroleum products
Pesticides (with odor or taste)
Example
Contaminants
Diesel
Aldicarb,
fenamiphos,
cyanide salts
Considerations for Analytical Approach
Screening for volatile and semivolatile organic
compounds1
Various methods may be applicable, depending
on targeted contaminants, including: liquid
chromatography, gas chromatography, and
spectrophotometric methods
May 2007
52
-------
Planning for WS-CWS Deployment
Class
3
4
5
6
7
8
9
10,11
10,11
12
Description
Inorganic compounds
Metals
Pesticides (odorless)
Chemical warfare agents
Radionuclides
Bacterial toxins
Plant toxins
Pathogens (select agents)
Pathogens (non-select agents)
Persistent chlorinated organic
compounds
Example
Contaminants
Arsenite salts,
strychnine
Mercuric chloride
Sodium
fluoroacetate
VX
Cesium-137
Botulinum
Ricin
Bacillus anthracis
Vibrio cholerae
PCBs
Considerations for Analytical Approach
Various methods may be applicable, depending
on targeted contaminants, including inductively
coupled plasma/mass spectrometry (ICP/MS)
and chromatography with different detectors,
Screening for heavy metals using ICP/MS
Ion chromatography with conductivity detection
(method is currently being validated)
Analysis by a surety laboratory with access to
restricted standards
Screening for alpha, beta, and gamma emitters
Analysis of the toxins and pathogens currently
addressed by the state public health laboratory
participating in the Laboratory Response
Network (LRN)
Analysis for a minimum of two non-select agents
by procedures recommended in EPA SAM2
Gas chromatography/mass spectrometry
methods
Data quality objectives for the chemical analyses should include detection at the parts per million level for specific
chemicals (e.g., aldicarb, rather than total organic carbon)
2 EPA Standardized Analytical Methods for Environmental Restoration following Homeland Security Events (SAM)
A candidate list of target contaminants can be developed from the information in Table 5-2 and the
additional details on specific contaminants and methods from the references listed in Section 5.1.3. Of
these references, EPA's Water Contaminant Information Tool (WCIT) and EPA's Standardized Analytical
Methods for Environmental Restoration following Homeland Security Events (SAM) are particularly
useful for identifying contaminants of concern in drinking water.
Identifying Laboratories to Support the Contamination Warning System
It is unlikely that a water utility laboratory will have the capability to analyze for all of the target
contaminants on the list that is developed. To meet the design objectives of the sampling and analysis
component, the utility should establish relationships with multiple laboratories, potentially including
commercial laboratories, municipal laboratories, and state laboratories.
Ideally, these relationships will be in the form of contracts or purchase orders. This is the likely option
for relationships with commercial laboratories, but also may be an appropriate vehicle for accessing local
or state laboratory capabilities. Alternatively, an interagency agreement or memorandum of
understanding may be needed. Note that developing and executing these contracts or vehicles is
addressed during design and implementation (Section 5.1.2). During the pre-design phase, the goal is
simply to identify the laboratories that will be part of the analysis network, work with the laboratories to
establish roles, and agree upon the vehicle that will be used to access their capabilities.
To ensure timely results, utilities should identify labs within close proximity to the utility, when available.
The utility also should ensure that laboratories are qualified to perform analyses when these analyses are
monitored under laboratory oversight programs. For regulated drinking water contaminants, laboratories
should be certified through the EPA Drinking Water Laboratory Certification Program or the National
Environmental Laboratory Accreditation Program (NELAP). For analyses of contaminants addressed by
other programs, such as select agents under the Centers for Disease Control (CDC) Laboratory Response
Network (LRN), the laboratory should be approved under these programs. Laboratories should provide
documentation of certifications and accreditations. Some commercial laboratories may hold NELAP
accreditation for non-regulated contaminants of concern.
Determining the analytical support that each laboratory would provide can be approached using the
following steps:
May 2007
53
-------
Planning for WS-CWS Deployment
1. Determine which contaminants can be analyzed using the utility's current in-house laboratory
capabilities and capacity
2. For the remaining list, determine which contaminants can be analyzed by the in-house laboratory
by expanding capabilities (acquiring instrumentation, or certifications) or capacity (additional
staff or training)
3. For the remaining list, determine which contaminants can be contracted to one or more
commercial laboratories
4. For the remaining list, determine which contaminants can only be addressed through support from
local or state public health or environmental laboratories
Due to security restrictions, handling requirements, facility containment requirements, or instrumentation,
only a limited community of laboratories will have the capability to perform analyses for toxins, select
agents, radiochemicals, and chemical warfare agents. It may be difficult to identify a support laboratory
in close proximity of the utility. This gap in analytical capabilities should be considered as part of the
overall contamination warning system. It may be desirable to enhance credibility determination capability
through other components.
Identify Analytical Methods
After establishing the target list of contaminants and laboratories, analytical methods should be selected
for contaminants for which multiple method options are available. The utility should consider using the
following steps to identify the most appropriate method to use:
1. Determine if there is an approved EPA method for measurement of the targeted analyte(s) in
drinking water
2. If there is no approved EPA method, consult SAM as a resource to identify methods (Section
5.1.3)
3. If SAM does not recommend a method, consider other methods for measurement of the analyte of
interest that have been validated for use in drinking water (potentially validated in other matrices
if there are no drinking water methods available). In some cases, a validated analytical method
may not be available or appropriate for use in routine monitoring for a given contaminant or
class. For these instances utilities should identify opportunities to participate in method
validation studies organized by EPA or other organizations to address this gap.
Regardless of the validation status of a method, the most critical factor in ensuring that data are of known
and documented quality is ongoing assessment of method performance in the laboratory performing these
analyses for the contamination warning system. This assessment depends on the contaminant and
technique, and may include initial and ongoing spiked samples and blanks to assess bias, precision,
sensitivity, and contamination.
The utility also should consider selecting methods that can be used to simultaneously measure several of
the targeted analytes to improve analytical efficiency. Analytical techniques using chromatography
and/or mass spectrometry are examples.
For both routine and triggered method selection, both screening capability and confirmation capability of
candidate methods should be taken into account. For instances where the likely contaminant has not been
identified, specificity and reliability of data may not be as critical as broad detection and rapid results.
For instances where a specific contaminant is suspected, specificity and reliability is paramount.
Establishing Laboratory Roles
When multiple laboratories are capable of performing a method, the following factors should be
considered in determining which laboratory should be assigned responsibility for the analysis:
Is the laboratory certified/accredited/approved for the method in drinking water?
Does the laboratory's quality assurance (QA) program for the method include the following:
May 2007 54
-------
Planning for WS-CWS Deployment
o Sample receipt, storage and tracking protocols to chain of custody
o Initial and ongoing proficiency testing and quality control (QC) analyses for all analytes
and methods of interest
o Data review procedures
o Data storage and transfer
o Documentation of personnel qualifications and training
o Ability and willingness to analyze samples potentially containing unknown contaminants
If no certification/accreditation/approval is available for the method, is the method currently
performed routinely by the laboratory?
If the method is not routinely used by the laboratory, is the method a modification of a method
the laboratory routinely uses?
5.1.2 Design and Implementation Approach
Design and implementation of the approach determined through the contaminant and laboratory selection
process should include the technical considerations and specifications described below.
Design:
Develop contractual agreements and/or memoranda of understanding (MOUs) with laboratories
external to the water utility
o Number and frequency of samples, by method
o Initial and ongoing QC requirements
o Data reporting requirements (including data elements to report, data reporting forms or
transfer protocol, and results turnaround time)
Address logistics with off-site laboratories
o Sample transport approach (i.e., lab personnel, FedEx, contracted courier)
o Establish a point-of-contact at the utility and at all support laboratories
Implementation:
Procure instrumentation, reagents, and supplies for any expanded in-house capabilities
Sign contracts and/or MOUs with external laboratories
Develop SOPs for new activities
Conduct initial demonstration of capabilities for new methods used at utility laboratory and/or
external laboratories
Implement QA plan (addressed in Section 5.2.2)
Start-up and Baseline Sampling and Analysis:
Identify method performance problems or matrix interference/inhibition issues. For example,
high levels of background organisms may impact pathogen method performance at a given
location.
Resolve startup issues at laboratories that will likely arise (QC issues, documentation
completeness issues, sample transfer and receipt procedures and integrity issues, sample and data
flow)
Operation and Maintenance:
Verify ongoing acceptable performance by laboratories using proficiency testing (PT) samples
analyzed under established certification/accreditation/approval programs
Track availability of new, validated methods for contaminants of interest
Evaluation and Refinement:
Assess ongoing acceptable method and laboratory performance on drinking water samples from
the utility based on QC samples and PT samples
May 2007 55
-------
Planning for WS-CWS Deployment
Consider implementation of expanded analytical capability based on availability of new
analytical methods
5.1.3 A vailable Tools and Resources
The following tools and resources are available to support design and implementation of laboratory
capability and capacity as part of a contamination warning system:
American Association for Laboratory Accreditation. Contains a list of NELAC approved PT
sample providers, http://www.a2la.org/dirsearchnew/ptproviders.cfm
Association of Public Health Laboratories (APHL). Information provided to improve the
capacity and capability of public health laboratories in their response to biological, chemical, and
radiological threats, as well as other public health emergencies.
http ://www .aphl .org/programs/emergency_preparedness
CDC-Select Agent Program. Centers for Disease Control and Prevention regulates the
possession, use, and transfer of select agents, available at: http://www.bt.cdc.gov
EPA Analytical Methods for Drinking Water. Information on sources of methods as well as
links to the various organizations which distribute them. Environmental Protection Agency,
November 2006. http://www.epa.gov/OGWDW/methods/methods.html or
http://www.epa.gov/waterscience/methods
EPA Drinking Water Certification Program, http://www.epa.gov/safewater/labcert/index.html
EPA Laboratory Compendium. Published laboratory analytical methods that are used by
industries and municipalities to analyze the chemical and biological components of wastewater,
drinking water, sediment, and other environmental samples that are required by regulations under
the authority of the Clean Water Act (CWA) and the Safe Drinking Water Act (SDWA). Almost
all of these methods are published as regulations at Title 40 of the Code of Federal Regulations
(CFR).
EPA Standardized Analytical Methods [SAM] for Environmental Restoration following
Homeland Security Events REVISION 3.0, EPA/600/R-07/015
http: //www .epa. gov/nhsrc/
Laboratory Response Network (LRN). use this tool to identify laboratories close enough in
proximity that could serve as contract laboratories to analyze samples potentially containing
contaminants of interest, available at:
http://www.bt.cdc.gov/lrn
NELAC Institute. Contains a listing of NELAP accredited labs, http://www.nelac-institute.org/
National Environmental Methods Index (NEMI). Use NEMI to compare and contrast the
performance and relative cost of analytical, text, and sampling methods for environmental
monitoring. US Geological Survey, Environmental Protection Agency
http: //www .nemi. gov/
Water Contaminant Information Tool (WCIT). Secure (password protected), on-line database
that provides current, reliable information on chemical, biological, and radiological contaminants
of concern for water security. Environmental Protection Agency, December 2006
http://www.epa.gov/wcit
5.2 Sampling and Analysis
After target contaminants and support laboratories have been identified, the utility should develop and
implement an approach for baseline, maintenance, and triggered sampling and analysis. This ultimately
should be documented in a comprehensive sampling and analysis plan, as indicated in Section 5.2.2.
However, this process can begin with consideration of the factors that should be documented in this plan,
as discussed in Section 5.2.1.
May 2007 56
-------
Planning for WS-CWS Deployment
5.2.1 Pre-design
During the pre-design phase for sampling and analysis, the utility's design objectives should consider, but
not be limited to:
Sampling locations
Sampling frequency
Sampling procedures
Sampling Location Considerations
In addition to adapting the approach used for locating sensor stations (Section 4.1) to identify sampling
points for baseline and maintenance monitoring, the following factors should be considered in selecting
potential sampling locations:
Proximity of location to utility (a sampling location situated further from the utility may have
greater vulnerability, such as potentially decreased chlorine residual)
Accessibility of location (ease of access for samplers to collect large samples and to gain access
to sample collection location)
Percentage of output from each plant in system (consider increased sampling of key locations that
receive finished water from plants with a higher relative output)
Age of the water being sampled (extremely aged water may potentially have a decreased chlorine
residual, and/or increased potential for the presence of biofilms compared to water that is 1 day
old)
Age and composition of piping that finished water has passed through prior to being sampled
Open source of finished water (an open reservoir of finished water could prove to be more
vulnerable than an underground holding tank)
Locations where backflow into the system could pose a threat
Other key locations, such as fire stations, elevated tanks, and pump stations
Other key locations as identified through assessments for online water quality (Section 4.1)
and/or enhanced security monitoring (Section 6.1)
Because triggered samples may come from anywhere in the distribution system, a primary goal of
location selection for baseline monitoring should be to collect water from locations that is representative
of as large a region of the distribution system as possible. The first set of analyses of a triggered sample
will likely include the same methods and procedures that are used for baseline monitoring
The utility also should plan to sample from key locations throughout the distribution system so that data
can be compared to finished water data collected at the treatment plants. The utility may consider using
the each source water treatment plant as a control by which to compare all data from baseline and
triggered sampling events. The purpose of baseline sampling from multiple and diverse locations is to
determine if the water is homogeneous with respect to contaminants detected, levels detected, frequency
of detections and method performance. Data should only be pooled from multiple locations if it is
scientifically justifiable to do so. This determination will involve statistical analysis of data.
Sampling Frequency Considerations
To support development of representative and complete baseline data, sampling frequency should
consider the following:
Size of the utility's distribution system and service community
Flow rates through various parts of distribution system
High pressure points within the system
Seasonal affects (i.e., increased intake of surface water into plant due to rainy season or snow/ice
melts)
Changes in water source (if different sources are used during different times of the year)
May 2007 57
-------
Planning for WS-CWS Deployment
Frequency of customer complaints (i.e., increased customer complaints during stagnant, hot
seasons)
The utility may choose to collect samples more regularly while establishing a baseline for the potential
contaminants, and then adjust the sampling schedule to sample sufficiently to maintain sampling and
analysis response capabilities, continually update baseline data and to address seasonal or other issues that
may influence change in baseline levels of contaminants that already are present in the system.
Considerations for New or Modified Sampling and Analysis Procedures
Some aspects of sampling and analysis for the contamination warning system will be different from those
used routinely for compliance monitoring. The utility should consider the following factors when
planning for these new procedures:
Procedures, training, and equipment to concentrate large bulk samples for select pathogen and
toxin analysis for ease of transport to the state public health/LRN laboratory (volumes up to 100
L may need to be collected at each sampling location).
Additional safety equipment that may be needed to protect samplers from exposure to potential
contaminants (e.g., goggles, gloves, face shields, laboratory coats).
Procedures for ongoing field methods QC to aid in confirmation of site characterization
determinations; this should include corrective actions for QC failures
Procedures and training on proper chain of custody and evidentiary sample handling training to
ensure that sample documentation is addressed properly from sample collection, shipping to
method support lab, receipt at method support lab and throughout handling at the method support
lab during analysis
Training on collection, packaging, and transport/shipping procedures for drinking water samples
that may contain disease causing agents or materials considered to be hazardous by commercial
shippers
Training of utility laboratory staff on new analytical capabilities.
Procedures for ongoing analytical QC for new methods to assess data quality; this should include
corrective actions for QC failures
Procedures for data review for each new method (both those added to in-house laboratory
capabilities as well as those performed by external laboratories)
A response protocol in the event of a positive result
5.2.2 Design and Implementation Approach
Design and implementation of the sampling and analysis activities determined through the pre-design
process should include the technical considerations and specifications described below.
Design:
Develop a preliminary concept of operations that describes the process flow for routine sampling
and analysis and establishes roles and responsibilities
Establish sampling locations and a baseline monitoring sampling schedule based on pre-design
considerations. Sampling design should strive to collect data to address both spatial differences
("snapshot" differences from samples collected over a short period of time) and longer term
trends. Snapshot differences may be differences between sampling locations and treatment plants
and between sampling locations when collected over a short interval (such as 1 month). Longer
term trends (sampling location-specific trends and regional aggregate trends) is useful only when
sampling continues over a period of time in which a trend may be anticipated. For water utilities,
that period of time may be as long as 1 year to capture seasonal and operational changes.
Analyze historical data for any contaminants of concern based on contamination warning system
objectives to inform sampling and analysis plan
Develop standard operating procedures for sampling that address the following:
May 2007 58
-------
Planning for WS-CWS Deployment
o Sample collection procedures, containers, and preservatives
o Safety equipment use
o Sample packaging and transport
Develop or modify existing chain-of-custody, data reporting forms, or other utility forms needed
to meet new sampling and analysis needs unique to the contamination warning system
Identify field sampling equipment that should be procured
Identify laboratory instruments that should be procured
Develop list of field sampling supplies that should be procured and stocked for ongoing sampling
Develop list of laboratory supplies that should be procured and stocked for ongoing analyses
Develop a QA Project Plan that addresses the following:
o Sampling QA/QC (e.g., type and frequency of field QC to include in sampling activities)
o Analytical QA/QC
o Data review procedures, and corrective actions in the event of QC failures.
Implementation:
Procure field sampling equipment
Procure laboratory instruments and establish service agreements
Procure field sampling supplies that should be procured and stocked for ongoing sampling
Develop list of laboratory supplies that should be procured and stocked for ongoing analyses
Conduct training on sampling procedures
o Specialized procedures for sample collection
o Safety considerations
o Disease causing and hazardous materials packaging and shipping procedures
Establish "notification levels" for each contaminant; further action will be needed if a
contaminant is detected above this level. Until a baseline level of contaminants is established,
these notification levels should be based on relevant data from studies conducted by the utility, as
well as health advisory and other levels available through the sources listed in Section 5.2.3. After
baseline sampling analysis is conducted, these notification levels should be adjusted.
Consider the sources of data that can help establish a baseline of contaminant levels. Table 5-3
provides examples of sources to consider.
Revise concept of operations based on design and implementation activities
Table 5-3. Examples of Baseline Data Sources
Baseline Data
Historical data for specific contaminants and water quality from the following:
1. the point of entry to distribution system
2. within the distribution system
2
Method performance data at the treatment plant and throughout the distribution system for specific
contaminants (spike recoveries, interferences, matrix effects).
3
Targeted contaminants detected at treatment plants and from individual sampling locations within the
distribution system
4
Non-targeted contaminants that are detected at the treatment plants and within the distribution system
5
Tentatively identified compounds (TICs) that are detected at the treatment plants and within the distribution
system.
6
Levels of contaminants detected (quantified and semi-quantified for targeted, non-targeted and TICs) at the
treatment plant and individual locations
5
Frequency of detections of targeted and non-targeted contaminants, as well as tentatively identified
compounds in treatment plant water, individual locations and multiple locations overtime
7
Contaminant specific control charts
Trend charts
Start-up and Baseline Sampling and Analysis:
In general, a baseline monitoring program should proceed through phases of activity, culminating
in the development of a maintenance monitoring program. Those phases are described below.
May 2007 59
-------
Planning for WS-CWS Deployment
o Phase 1: SOPs and necessary resource document development for critical activities related
to baseline monitoring should be developed. Initial demonstrations of capability (IDC) and
minimum reporting limits (MRLs) for each method and contaminant should be established.
Data reporting requirements and protocols should be established.
o Phase 2: Following the development of SOPs and completion and review of IDC data,
finished water at the treatment plants is analyzed with respect to contaminant occurrence
(contaminants detected, levels detected and frequency of detections) and method
performance. Finished water from the treatment plant serves as a benchmark for comparison
of contaminant occurrence and method performance of water from the distribution system.
All future baseline and triggered sampling events may include the source treatment plant
water as a control.
o Phase 3: Regular surveillance monitoring of strategic/priority locations should be initiated
and conducted at regular intervals to establish baseline for these locations and to determine if
there are seasonal or regional trends.
o Phase 4: A survey study should be performed to determine contaminant occurrence and
method performance in the distribution system. Sample collection locations should be
selected with the goal of achieving spatial coverage of the distribution system and to capture
a wide range of conditions in water age, pressure zones and pipe material. This phase of
study is designed to survey the distribution system for contaminant occurrence and method
performance. The results from this survey study may result in the design of Phase 5 (focused
studies).
o Phase 5: Based on the results of Phases 2-4, short-term, focused studies may be conducted
to look more closely at possible differences in contaminant occurrence or method
performance within the distribution system. If significant differences are found, these
findings may influence the selection locations for maintenance monitoring.
o Phase 6: The final phase should be the analysis of results from Phases 1 - 5 to establish the
management, interpretation and use of baseline data and to establish a maintenance
monitoring program. Emphasis should be placed on access and interpretation of baseline data
during a triggered sampling and analysis event.
Document progress of sampling and maintain communication with laboratories to ensure
contractual obligations are being met
Consideration for baseline data should include storing the data in a manner that allows easy
retrieval and use for interpretation of data from triggered sampling events.
Consider adjusting sampling schedule after evaluating results from the first year of sampling and
analysis
Consider a shift from the baseline monitoring phase to the maintenance monitoring phase after a
year of baseline sampling and analysis is complete
Finalize concept of operations document to reflect results of baseline sampling and analysis and
transition to maintenance monitoring
Operation and Maintenance:
Development of a maintenance monitoring program should consider results and data evaluation
from the baseline monitoring period as well as cost considerations and dual benefit. For
contaminants that fall under drinking water regulation, less frequent maintenance monitoring may
be warranted, whereas, contaminants that may not otherwise be monitored under any program
may be given priority for more frequent maintenance monitoring. Maintenance monitoring
should strive to maintain capabilities and serve to collect data that may be used to update baseline
data (Table 5-3).
Maintain baseline data (i.e., tabular data, control charts, method performance by location)
Perform periodic maintenance of sampling equipment, as necessary to maintain response
capabilities
Restock sampling equipment as necessary (bottles, cubitainers) or laboratory supplies (reagents or
consumables)
May 2007 60
-------
Planning for WS-CWS Deployment
Evaluation and Refinement:
Analyze baseline data to assist in determining when contaminant levels in triggered samples
exceed baseline levels
Evaluate ease of use of accessibility of baseline data during a triggered event
Review target contaminant list and consider if new methods for additional contaminants are
available and could be implemented
Provide additional training to sampling teams and/or laboratory staff to address deficiencies
observed during baseline/maintenance sample analyses (i.e., contaminated blanks, high false
positives/negatives)
5.2.3 Available Tools and Resources
The following tools and resources are available to support design and implementation of sampling and
analysis as part of a contamination warning system:
EPA Health Advisory Levels: http://www.epa.gov/waterscience/criteria/drinking
EPA Region 3 Risk Based Concentrations:
http://www.epa.gov/reg3hwmd/risk/human/index.html
EPA Region 6 Human Health Medium-Specific Screening Levels (HHMSSLs):
http: //www .epa. gov/earth 1 r6/6pd/rcra_c/pd-n/screen .htm
EPA Region 9 Preliminary Remediation Goals (PRGs):
http://www.epa.gov/region9/waste/sfund/prg/faq.htm
EPA Response Protocol Toolbox (RPTB), Module 3 and 4, EPA-817-D-03-003,
http://cfpub.epa.gov/safewater/watersecurity/publications.cfm
EPA Water Training Opportunities, Workshops/Training, November 2006
http://www.epa.gov/water/training.html
National Laboratory Training Network (NLTN). Dedicated to improving laboratory practice
of public health significance through quality continuing education.
http: //www .phppo .cdc. gov/nltn/
Training resources for packing and shipping etiologic agent samples and hazardous
materials. Examples include:
o http://www.ercweb.com/classes/
o http: //saf-t-pak .com/
The Water Quality Data Elements User Guide, http://acwi.gov/methods/
5.3 Site Characterization and Field Screening
Sections 5.1 and 5.2 address activities associated with laboratory-based analyses performed on samples
transported from the field. These activities should be addressed both on a routine basis and during a
response. Additional activitiessite characterization and field screeningshould be addressed in the
field specifically for response.
Site characterization is the process of collecting information at a site to support the evaluation of a
drinking water contamination threat. This process may include site evaluation, sample collection, and
field screening. Field screening involves rapid sample testing in the field to evaluate any potential safety,
chemical, biological or radiochemical hazards present at the site and to provide the laboratory with
preliminary information that may help focus their analytical activities.
5.3.1 Pre-design
During the pre-design phase for site characterization and field screening, the utility's design objectives
should consider, but not be limited to:
May 2007 61
-------
Planning for WS-CWS Deployment
The role that the utility will play in site characterization activities, versus roles by other
organizations, such as law enforcement and hazardous materials response units
Current site characterization expertise among these organizations, and coordination across these
organizations, versus the expertise and coordination needed to fulfill each organization's role to
fully and effectively evaluate a potential contaminant threat site
Current field screening equipment and expertise, versus the equipment and expertise needed to
rapidly test for contaminants in the field
Site Characterization
Details on the site characterization process for a potential water system contamination incident are
provided in the EPA Response Protocol Toolbox (RPTB). Site characterization is intended to provide
important information to guide activities not only of water utility managers and staff, but also external
first responders (such as local law enforcement and HazMat teams) and other government agencies that
may be involved (such as the FBI and the EPA's CID. Information gathered during site characterization
is combined with other information to perform a threat evaluation, the results of which may feed back into
additional site characterization activities.
During pre-design, the utility should consider the following steps for addressing site characterization:
Define the potential scope of site characterization activities, based on the RPTB, the utility's
own emergency response planning materials, or other resources (see Section 5.3.3).
Identify the organizations that should be involved in site characterization. Although this will
depend on the scope of the incident, the objective of pre-design is to identify a comprehensive
list of these organizations.
Work with these organizations to map roles for each site characterization activity to the
responsible organization. The same activity also may involve different organizations, depending
on the contaminant (such as a toxic industrial chemical versus a chemical warfare agent).
Work with these organizations to assess the level of training and expertise of each to fulfill their
role and identify shortfalls that should be addressed.
Through the consequence management process (see Section 9), identify exercises and other
opportunities to test and maintain a high level of coordination among the organizations that may
be involved in site characterization.
Field Screening
Although field screening is one of many potential site characterization activities, it merits some specific
pre-design consideration because it relies on appropriate equipment and training to be effective. Two
types of rapid testing should be considered:
Testing of materials other than the water (safety screening) to determine whether the
environment around the potential contamination site can be safely accessed (or accessed with
appropriate personal protective equipment [PPE])
Testing of the water (water screening) or other media (e.g., contents of discarded containers or
suspicious residues) to determine whether samples can be safely handled or transported to
laboratory(ies) for analysis and to provide the laboratory with preliminary information on
potential contamination to help focus subsequent laboratory analyses.
Field screening capabilities should address the target parameters presented in Table 5-4. Although one
goal of pre-design for field screening is to maximize the number or type of contaminants that can be
tested in the field, there are practical considerations that may limit this, including equipment cost and
initial and ongoing training time and cost.
May 2007 62
-------
Planning for WS-CWS Deployment
Table 5-4. Considerations for Contaminant Coverage for Field Screening
Screening
Type
Safety
Safety
Water
Water
Water
Water
Water
Water
Target
Parameter
Radioactivity
(alpha, beta,
and gamma)
VOC(PID),
LEL, CO,
H2S,O2
Cyanide
Chlorine
residual
pH/
conductivity/
ORP
Turbidity
Chemical
Warfare
Agents (VX,
sarin, etc.)1
Toxicity1
Considerations for Field
Testing Equipment
Geiger counters and
scintillators, equipment
that can distinguish
gamma/beta from
alpha/beta emissions
Multi-meters used for
confined space entry
(photoionization detector
with other meters)
In-field cyanide detector
(e.g., a colorimeter or
spectrophotometer)
In-field cyanide detector
(e.g., a colorimeter or
spectrophotometer)
Electrode detector
Turbidimeter; most
measure light scattering
M272 Water Testing Kit or
similar commercial
versions
Commercial toxicity test
kit
Considerations for Training
The level of sophistication and
expense vary widely. Vendor
training typically is required.
Some HazMat units may be
able to provide training
Instrument manual and
training videos can be
sufficient; training from an
experienced user is ideal
Instrument manual and
periodic QC samples are
usually sufficient
Instrument manual and QC
samples are usually sufficient
Instrument manual. None
beyond normal utility
procedures for these
measurements
Instrument manual and QC
samples are usually sufficient
Instrument manual and QC
samples are usually sufficient
Vendor training sometimes
required
Comments
May be expanded
to water testing
with a special
probe
Detects chemicals
in air
Tests water for
cyanide ion, but
not combined
forms
Absence of
residual chlorine
may indicate a
problem
Abnormal pH or
conductivity may
indicate a problem
High turbidity may
indicate a problem
May also detect
some pesticides
and common
chemicals
Should establish a
baseline
Note: M272 and toxicity testing kits are time-consuming to use. These kits are generally only used if initial
screening is inconclusive, or if the situation indicates that these screening tests may be relevant.
A candidate list of target contaminants can be developed from the information in Table 5-4 and the
additional details on specific field screening equipment and capabilities available from the RPTB.
5.3.2 Design and Implementation Approach
This section addresses considerations for design and implementation of site characterization and field
screening capabilities.
Develop a customized site characterization plan based on the circumstances of the threat warning
that integrates with the concept of operations and consequence management plan (Section 2.1 and
Section 9, respectively). This customized plan may be adapted from a generic site
characterization plan or as part of a response to a specific contamination threat. It is impossible
to predict every possible scenario, so it is best to specify example scenarios that each warrants a
different level of response. The site characterization team uses the customized plan as the basis
for reporting their observations/data at the investigation site.
Develop a health and safety plan to address any concerns that may arise in the field
o Appropriate PPE
o Emergency call list of agencies and individuals that should be notified in an emergency
(e.g., hospitals, HazMat, MEDTox, fire and police)
May 2007
63
-------
Planning for WS-CWS Deployment
o 40-hour OSHA training
Select Site Characterization Team and Team Leader
o Experience
o Training
o Availability
o Anticipated level of response. Note that outside agency intervention may be necessary
(e.g., HazMat).
Develop basis for personal protective equipment and field screening equipment
o Preliminary information from monitoring station, personnel, or customer complaint
o Direct experience and availability of various equipment types (utility vs. HazMat team)
o Suspected contaminant
Select primary components of the sampling kit
o Bottle types and preservatives
o Number of containers needed
o Labels, Chain-of-Custody
o Shipping materials
o Name and address of receiving laboratory
Assess resources available for expanded efforts
Implementation:
Ensure equipment for use in the field investigation is available and ready
o Instrumentation is pre-calibrated
o Communication devices and power supplies are tested and operational
o Equipment for sampling (e.g., coolers, bottles, preservatives)
o Appropriate documentation (field logbooks, chain of custody, health and safety plans,
standard operating procedures, etc.)
Prepare lists of field trained individuals within the organization
o Prepare call lists for expanded situations (police, fire, HAZMat, etc.)
o Schedule training at required intervals
Health & Safety
Use of equipment, instrumentation, sampling procedures
Shipping hazardous materials
Procure necessary instrumentation
o Schedule routine maintenance and calibration
o Secure back-up parts, and consumables
Prepare site investigation kits
o Personnel protective equipment
o Screening instrumentation
o Sampling kits
o Documentation
Develop SOPs as necessary
Develop working relationships with local fire, police and HAZMat personnel
o Meet with other organizations (Police, Fire, HazMat, etc.) to agree upon each parties
roles and responsibilities in an expanded situation
Perform background Site Hazard Assessment at representative collection sites
Start-up and Baseline:
Site inspections to observe normal surroundings
Establish initial and ongoing QC requirements
Establish background levels of contaminants
May 2007 64
-------
Planning for WS-CWS Deployment
Operation and Maintenance:
Maintenance and calibration of screening equipment as per manufacturer's requirements or
specifications
Periodic testing of instrumentation with independent measurements
Evaluation and Refinement:
Track and evaluate the development of new field equipment technologies
Provide additional training to address deficiencies observed during site characterization
5.3.3 Available Tools and Resources
The following tools and resources are available to support design and implementation of site
characterization and field screening as part of a contamination warning system:
EPA Response Protocol Toolbox (RPTB), Module 3, EPA-817-D-03-003,
http://cfpub.epa.gov/safewater/watersecurity/publications.cfm
Resources for Strategic Site Investigation and Monitoring, United States Office of Solid
Waste and EPA 542-F-01-030b, Environmental Protection Emergency Response September 2001
Agency (5102G) http: //www. epa. gov/tio/
Improving Sampling, Analysis, and Data Management for Site Investigation and Cleanup
United States Office of Solid Waste and EPA-542-F-01-030a Environmental Protection
Emergency Response April 2001Agency (5102G) http://www.epa.gov/tio/
EPA Water and Wastewater Security Product Guide, September 2005,
http://cfpub.epa.gov/safewater/watersecuritv/guide/index.cfm
5.4 Staffing and Cost Considerations
Planning for the implementation of the sampling and analysis component of a contamination warning
system will likely involve a more limited range of utility staff than other components, but consideration
should be given to both routine (baseline and maintenance) and triggered sampling events, the latter of
which may involve a wider array of utility staff. Cost factors will be driven by the analyses the utility
laboratory will perform, the role of commercial laboratories, and the cost basis for support by state or
local public health or other laboratories.
5.4.1 Staffing
Table 5-5 offers a quick overview of which staff may be necessary to design and implement an expanded,
multi-laboratory sampling and analysis program as part of a contamination warning system and during
which phases of implementation these personnel may be needed.
Table 5-5. Sampling and Analysis Staffing Considerations
Division or Department
Water Quality
Information Technology
Security/Risk
Administration
Implementation Stage
PD
X
D
X
X
X
X
I
X
X
X
X
PT
X
O&M
X
X
ER
X
Comments
Involvement in all aspects of sampling and
analysis activities
Support to data transfer and management
aspects of laboratory results
Site characterization support
Contracting vehicles or MOUs needed to access
external laboratory assets
May 2007
65
-------
Planning for WS-CWS Deployment
5.4.2 Cost Considerations
This section presents a summary of the design and implementation considerations discussed above that
may influence costs. This list also may include other factors that were encountered during
implementation of the initial Water Security initiative pilot and could be overlooked during cost
estimation in the absence of this experience. Although this list of cost considerations may not be
exhaustive, these factors, at a minimum, should be considered when planning.
Cost considerations for activities performed by the utility:
Additional staff to support sampling or analysis activities
Field screening equipment
Laboratory instruments
Service and preventative maintenance contracts for new instrumentation
Sample collection containers
Additional laboratory reagents, standards, and disposables
Safety equipment (e.g., face shields, respirators, flammable cabinets)
Additional costs for hazardous and/or biological waste disposable (e.g., procure an autoclave,
contract waste disposal)
Training:
o Disease causing agent and hazardous materials shipping
o Site safety (e.g., OSHA HAZWOPR)
o New laboratory instruments
o Field screening equipment
Proficiency testing to maintain certification/accreditation or demonstrate proficiency for methods
that are not covered under the certification program
Laboratory information management system changes
Initial and ongoing costs of special permits (e.g., CDC, USDA) that may be required
Cost considerations for use of external laboratories:
Costs that may be incurred based on a minimum vs. maximum number of samples analyzed (e.g.,
cost per sample for 10 samples vs. 100 samples)
Additional costs that may be charged if samples are received after hours, exceed daily capacity,
require additional analyses
Additional costs for sample processing (e.g., concentration for pathogen samples) at the
laboratory, rather than in the field
Transport costs (e.g., courier, shipping costs) based on proximity to the support laboratory
In the process of determining costs the utility should determine whether expanding in-house analytical
capability is cost effective and sustainable. This will not be an option for some contaminants (such as
analysis of select agents, which is restricted to CDC LRN laboratories). However, expanding in-house
capabilities to analyze others may be more cost effective if this capability can not only address
contamination warning system needs, but also enable the utility to shift some current compliance
monitoring sample analyses from a commercial laboratory to the utility's in-house laboratory.
May 2007 66
-------
Planning for WS-CWS Deployment
Section 6.0: Enhanced Security Monitoring
Enhanced security monitoring includes the systems, equipment, and procedures that detect and respond to
security breaches at distribution system facilities such as pump stations, reservoirs and storage vessels
that are vulnerable to contamination. The monitoring strategy includes detection by physical security
systems such as alarms and cameras, witness accounts, notifications by perpetrators, media, and law
enforcement, as well as associated response methods. A security breach is an unauthorized intrusion into
a secured facility that may be discovered through direct observation, an alarm trigger, or signs of intrusion
(cut locks, open doors, cut fences). Security alarms are a common threat warning for a utility but are
often unintentionally caused by routine operation and maintenance activities. Actual security breaches
usually are the result of criminal activity such as trespassing, vandalism, and theft, rather than attempts to
contaminate the water. Under the contamination warning system model, enhanced security monitoring
should be designed to help discriminate between security breach alarms and notifications that may be
related to a contamination incident and those resulting from other activities. Table 6-1 summarizes
design basis considerations for enhanced security monitoring.
Table 6-1. Design Basis Considerations for Enhanced Security Monitoring at Selected Sites
Design
Objective
Capability
Contaminant
Coverage
Spatial
Coverage
Timeliness
Reliability
Sustainability
Description
Can detect an intrusion that may
have provided the opportunity for
introduction of any contaminant.
Covers all contaminant classes.
Limited to those elements of
infrastructure for which physical
security can be monitored.
Function of the type of security
monitoring system and the time to
evaluate a security breach.
Can be a reliable means of
identifying an intrusion, especially
when these breaches may involve
contamination, such as in storage
tanks and reservoirs.
Provides utility with increased
physical infrastructure protection
and awareness. Reduces the
occurrence of nuisance tampering.
Design and Implementation Considerations
System should be capable of detecting and assessing
breaches that could provide access to the water supply
at distribution system facilities where contaminant
addition would impact a significant number of
customers.
System should detect and assess breaches at all
possible entry points, at facilities selected for security
enhancements, which could provide access to the
water supply, regardless of contaminant quantity, type
or method of injection.
Improvements should be focused on distribution system
facilities such as pump stations, wells, reservoirs, and
storage tanks where a large volume of water could be
contaminated and impact a significant number of users.
Service connections and hydrants should generally not
be considered due to the high number of nodes and the
low benefit/cost ratio of hardening those
appurtenances.
System should be designed such that alarms produced
allow responders to quickly make an assessment and
generate the proper response. For many facilities, use
of video to assess alarms may be critical.
System should be designed such that video images
and proper response procedures are used to minimize
false alarms.
System should utilize equipment that is robust, does
not have substantial maintenance requirements, and
does not produce frequent false alarms.
The overall objective of this section is to describe considerations and a process for design and
implementation of the enhanced security monitoring component of a contamination warning system to
guide planning activities. In planning for implementation of enhanced security monitoring, several key
design decisions should be made including the following:
Facilities at which to install security enhancements
Type of security enhancements for consideration (e.g., alarms, motion sensors, video)
Prioritization framework for ranking sites and security enhancements
May 2007
67
-------
Planning for WS-CWS Deployment
Communications architecture for transmission of data and alarms
Approach for implementation
Approach for operation and maintenance
A key objective of this section is to provide information to enable the reader to consider these decisions in
a systematic process. The primary design element for enhanced security monitoring is physical security
enhancements, along with integrated communications architecture and data management. Physical
security enhancements should focus more heavily on detection and assessment of a potential
contamination event at a facility and less on preventing the event from occurring due to challenges with
and feasibility of implementing improvements to prevent an adversary from gaining access to a facility.
The physical security enhancements should be designed in conjunction with the design of the
communications architecture and data management to ensure the physical security system design
objectives are met. Communications architecture for enhanced security monitoring should be closely
coordinated with the same communications systems for online water quality monitoring since they often
may share common systems. Unique considerations for enhanced security monitoring, particularly with
regard to the transmission and storage of video data should be identified and considered in the
development of a comprehensive communications and data management architecture for these
components.
The initial assessment process for enhanced security monitoring is one of the most critical aspects of this
component of the contamination warning system. The process is similar to the approach taken by most
water utilities in conducting their vulnerability assessments in response to the Public Health Security and
Bioterrorism Response Act of 2002 as described in Instructions to Assist Community Water Systems in
Complying with the Public Health Security and Bioterrorism Preparedness and Response Act of 2002
(USEPA, 2003). In contrast to the vulnerability assessment process which evaluated all possible threats
to the entire water utility, a distribution system physical risk assessment focuses only on intentional
distribution system water contamination. Findings from the previously completed vulnerability
assessments may be utilized to identify the critical facilities within the distribution system. For some
contamination warning system components, the dual benefit of detecting accidental or naturally occurring
contamination events is possible. In the case of enhanced security monitoring, the dual benefit is the
ability to detect all types of intrusions including those involving intentional contamination.
Table 6-2 provides example preliminary recommendations for security improvements for typical water
utility facilities. The actual recommendations should be utility-specific. The process for selecting the
improvements is described in subsequent sections of this document.
Table 6-2. Example Improvements by Water Utility Facility Type
Facility Type
Typical Recommended Improvements
Construct a structure over vents to prevent addition of contaminants
Contact switches and alarms for hatches and access points
Finished Water
Reservoirs and
Ground Level , Harden hatches and covers for which alarms can not be feasibly added
^ - Cover or install barriers for overflow pipes that are vulnerable to contamination
Develop written procedures for isolating tanks and reservoirs in the event of suspected
contamination and provide training for these procedures
May 2007 68
-------
Planning for WS-CWS Deployment
Facility Type
Pump Stations
Elevated
Storage
Tanks
Typical Recommended Improvements
Interior motion detectors to detect intruders entering through windows and vents in areas
that provide access to water pumps and pipes
Contact switches for doors to detect intruders entering areas that provide access to
water pumps and pipes
Camera(s) that are activated by motion detectors or contact switches
Card access reader
Video and communication interfaces
Lighting improvements for camera systems where needed
Develop written procedures for isolating and turning off pumps in the event of suspected
contamination and provide training for these procedures
Interior motion detectors
Audible alarms at facility
Develop written procedures for isolating tanks and reservoirs in the event
contamination and provide training for these procedures
of suspected
6.1 Pre-design
Pre-design and planning for enhanced security monitoring involves the following:
Determine design basis threat
Develop preliminary facility list
Site assessments
Perform risk ranking to assess risk before improvements
To evaluate how well existing and future proposed security systems and procedures protect facilities from
contamination, it is important to define the specific potential threats to those facilities. Before
determining the effectiveness of protection systems, agreement should be reached defining types and
capabilities of the adversaries who may attempt to contaminate the system. This is important because the
effectiveness of a protection system can vary greatly depending upon the adversary. For example, a
standard steel door with hinge protection should delay or defeat a vandal attempting to enter a facility, but
a sophisticated, trained adversary such as a terrorist armed with the proper tools and equipment could
defeat the door easily and quickly. The same variance in effectiveness often applies to detection and
alarm systems. Of equal importance is defining the types and quantities of contaminants that may be used.
In order to evaluate the contamination risk of each facility, the design basis threat should be defined. This
includes identifying the capabilities of adversaries as well as the quantity and type of contaminants that
could be used to contaminate the water supply.
The recommended design basis threat is a highly sophisticated adversary or group of adversaries with the
resources and ability to access all contaminants represented by the classes listed in Table 1-1. This type
of adversary has the ability to gain expertise in the areas of distribution system design, operation,
hydraulics, drinking water treatment, chemistry, microbiology, etc.
The next step during pre-design and planning is to preliminarily identify which utility facilities may be at
the highest risk for contamination based on factors including: consequence of contamination, site
location, access, and visibility. Selection of these preliminary sites can be facilitated through review of
previously conducted vulnerability assessments, which may have characterized relevant attributes of each
facility even if the focus of the assessment was not on contamination. If a distribution system model is
available, it could be used to estimate the consequences of contamination at various distribution system
facilities. The process for estimating consequences using a distribution system model is described below.
The preliminary list of facilities may be evaluated further through site assessments for potential enhanced
security monitoring upgrades.
May 2007 69
-------
Planning for WS-CWS Deployment
After the preliminary facilities list has been developed, the next step is to conduct detailed site
assessments of each facility on the list with a focus on existing physical security systems, communication
capabilities (i.e., transmission of alarms, video, etc.), proximity to the public, terrain, adjacent land uses,
site access, site lighting, alarm and detection systems, and physical barriers such as fencing and hardened
structures. Using the observations made during the site assessments, an evaluation should be performed
regarding the possible modes of entry for contaminants (i.e., dumping contaminants directly into
reservoir, injecting into pipe tap, etc.) and volumes of contaminants that could practically be delivered to
each of the facilities and put into the water without arousing suspicion. For example, the site
configuration of some facilities may not allow a heavy truck to get close enough to the facility to deliver
1,000's of gallons of the contaminant, while other facilities may not be accessible at all by a vehicle in
which an adversary may be limited to quantities that could be transported by hand.
The contamination risk of each facility can be evaluated after facility assessments have been conducted
and the design basis threat has been defined. The risk assessment provides the basis for prioritizing which
facilities should be considered for security improvements and which security improvements would be
most cost-effective in terms of ability to reduce risk of contamination. The contamination risk at a facility
is a function of three primary parameters:
Effectiveness of the facility's existing physical security system
Probability that the facility may be targeted by an adversary
Consequences of a contamination event at the facility
There are several methods available to calculate risk. The method most commonly used during the 2002
vulnerability assessment process was the Risk Assessment Methodology for Water Utilities (RAM-WSM)
developed by Sandia National Laboratory. A method very similar to RAM-WSM can be used to compare
the risk of intentional contamination at distribution system facilities. Other risk assessment methods are
available including VSAT (Vulnerability Self-Assessment Tool) developed by the National Association
of Clean Water Agencies (NACWA).
The effectiveness of a facility's existing physical security system can be assessed during the detailed site
assessment. The considerations used to estimate the effectiveness of a security system include detection,
delay, and response. Traditionally, physical security systems (i.e., door alarms, motion detectors, etc.),
have been designed to detect a security breach early enough to provide adequate delay and allow
sufficient time for law enforcement to respond and prevent the adversary from completing their intended
act. However, for the case of potential contamination, the consequences can in some cases be quickly
eliminated or significantly reduced through an operational response triggered by a security alarm or
notification (i.e., isolating a finished water storage tank, shutting down pumps, etc.).
Estimating the relative probability that an adversary may attack one of a utility's facilities compared to
another may be somewhat difficult to assess. However, it is likely that some facilities may be more
attractive targets compared to others. Some aspects of the facility that could be used to estimate the
probability of attack include:
RecognizabilityHow easy would it be to recognize a facility as a water utility facility that
provides access to drinking water?
Visibility to Surrounding PublicIf a facility is visible to the public living and working near the
facility, an attack on that facility may be deterred due to the increased probability of the public
witnessing the attack. However, a sophisticated adversary may not be deterred significantly from
attacking a visible facility.
Access to and Ability to Deliver ContaminantThis estimates how difficult it would be to get a
contaminant to a facility and add it to the system. This is a function of the possible modes of
entry for contaminants (i.e., dumping contaminants directly into reservoir, injecting into pipe tap),
and volumes of contaminants that could practically be delivered to each of the facilities and put
into the water without arousing suspicion. Also, some utilities may not provide vehicle access
which would limit the amount of contaminant that could be delivered.
May 2007 70
-------
Planning for WS-CWS Deployment
Effectiveness of AttackThis estimates how effective a potential attack would be on a facility
based upon the ability to add a sufficient amount of a contaminant to reach a lethal concentration
in the distribution system.
The consequences of a contamination event occurring at a facility ideally should be estimated using the
utility's distribution system model and GIS to simulate how a contaminant would spread and how many
people would be affected by the incident. The following parameters should be considered in a modeling
analysis to estimate the resulting consequences of contamination at any distribution system facility:
Volume of contaminant added (depends on contaminant and site characteristics)
Concentration of contaminant added (depends on contaminant availability)
Toxicity of contaminant (depends on contaminant type)
Duration of contaminant addition (depends on site characteristics)
Duration of model simulation (recommend minimum of 24 hours)
Type of storage tank mixing modeled (i.e., completely mixed, plug flow, etc.)
The results of the site assessment, described previously, can be used to define and constrain the
contamination scenarios that are modeled (e.g., the volume of contaminant that can be delivered to a
location, the duration of contaminant addition, etc.). Furthermore, additional understanding may be
gained by varying parameters such as the time of day that the contaminant is introduced at the facility
because as demand patterns vary over the course of a day, the system hydraulics may change, which may
impact the total number of exposures significantly. The risk of contamination for each facility is
calculated utilizing the selected risk assessment equation and the estimated values for physical security
effectiveness, probability of an attack, and consequences of an attack. Once calculated, the facilities
should be sorted in order of highest to lowest risk. Section 6.2 provides additional detail for designing
security systems for the selected facilities.
In addition, most of the physical security improvements have communication and data management
requirements that should be considered during pre-design. A robust, reliable, and secure architecture
should be developed. The same system developed in Section 4.3 for online water quality monitoring may
in some cases be used for enhanced security monitoring. Utilities may decide, however, to have a
dedicated architecture for enhanced security monitoring. Pre-design considerations unique to enhanced
security monitoring include the following:
Determination of requirements of the communication system needed to support the proposed
network of physical security improvements. Consider communication requirements for the online
water quality monitoring stations deployed as part of the contamination warning system, and
evaluate the feasibility of using a single architecture to support both.
Assessment of existing communication system architecture used to transmit data and commands
between remote facilities and the utility central control location. Assess ability of existing system
to accommodate the proposed enhanced security monitoring systems. Evaluate use of the
existing data recording systems (e.g., SCAD A) for managing data from cameras, contact alarms
and other remote devices. If existing communications systems and data recording systems are
unable to meet the requirements, the utility desires to separate security data from process control
SCADA systems, or the utility desires to transmit security data to locations (e.g., security guard
station) outside the SCADA network, evaluate alternatives. Identify constraints on
communication alternatives such as hilly terrain that may make radio communications cost
prohibitive. Alternative communication methods include SCADA, Tl lines, digital cellular
services, and private radio network.
A potentially significant data management challenge for enhanced security monitoring could be
the management of video data from remote sites. The ability to transmit video for intrusion alarm
assessment can be the most demanding communications network requirement of the physical
security improvements. The video transmission option that provides the best resolution with the
quickest response is full streaming video over fiber optic lines. The installation of fiber optic
lines to remote facilities is often necessary, making this option very costly. The other video
May 2007 71
-------
Planning for WS-CWS Deployment
options are various technologies to compress and package video clips for transmission.
Transmission options such as Tl line and digital cellular services were described above.
6.2 Design and Implementation Approach
Once ranked, the facilities are evaluated further to identify methods for reducing risk of contamination to
the facilities. To reduce risk, physical security system effectiveness should be increased or the
consequences or probability of attack reduced. To determine how security system effectiveness could be
increased, conceptual design and associated cost estimates of security improvements should be
completed. Table 6-2 showed a list of typically recommended improvements. These recommendations
may, however, vary from utility to utility and may be dependent upon several factors that are unique to
each utility.
Methods to reduce risk by increasing physical security system effectiveness include increasing detection
and assessment capabilities, improving delay, and improving responses to an alarm. Means to improve
the utility's security systems should include capital improvements, such as surveillance and monitoring
equipment and alarms, facility structural improvements, and procedure modifications. In addition,
response can be improved by developing written procedures for isolating each facility in the event that
contamination is suspected. It is recommended that these written procedures be developed and the
procedures incorporated into standard operating procedures and consequence management training and
drills.
In addition to increasing the effectiveness of the physical protection systems, contamination risk can be
reduced by decreasing the consequences of a contamination event. If cameras are installed, the video
images can confirm that an alarm was caused by an intruder and the operator can take action to mitigate a
potential contamination event, including disabling pumps, closing valves, or changing the hydraulic grade
line to prevent the spread of a potential contaminant. It is recommended that detailed procedures like this be
developed for each facility. In addition, a timely "do not drink" or "do not use" order given to the public
would in many cases reduces the consequences of a contamination event. Procedures and guidance for
issuing "do not drink" and "do not use" orders should be developed as part of the consequence
management plan.
Decreasing the probability of an attack, especially for a sophisticated adversary, may be difficult to
achieve but some options are available. Removing or covering signs that identify the facilities as a utility
facility could be considered. However, since the adversary may be sufficiently sophisticated to identify all
facilities, removing or covering signs is not recommended. For most facilities, it likely will not be possible
to make them more visible to the surrounding public to deter attacks. However there may be facilities where
clearing brush or small trees or other Crime Prevention through Environmental Design (CPTED) strategies,
as discussed in Interim Voluntary Security Guideancefor Water Utilities (AWWA, 2004), may be
beneficial.
Barriers could be added at some facilities to prevent vehicles that could be used to carry large quantities of
contaminants from entering the site. However, these facilities would still be subject to attack with smaller
quantities of the most potent contaminants. The effectiveness of barriers should be reviewed further during
design, although the cost of an effective barrier at even one facility may limit the budget available to
enhance security at other locations. To help prioritize improvements, the cost of improving security
systems at each site can be evaluated in the context of the associated risk reduction that would result from
making the recommended improvements. Based on the preliminary design, planning level capital costs
should be calculated for each facility. Methods for calculating planning level capital costs are described
in detail in Section 6.3.2.
For each facility, the risk score following improvements is estimated and the difference between this
score and the original risk score is calculated and the difference is expressed as risk reduction units
(RRU). The costs to benefits of for each facility are expressed in terms of the capital costs of
May 2007 72
-------
Planning for WS-CWS Deployment
improvements per RRU ($/RRU). After developing the preliminary design for security improvements at
each facility, the risk of contamination for each facility should be recalculated.
The values of RRU are utilized in the facility prioritization process. If any modifications to the
preliminary design concepts are made as a result of the cost analysis or facility prioritization process, the
RRU should be recalculated accordingly. Once calculated, the list of facilities and associated cost benefit
ratios should be sorted in order of increasing $/RRU. From this list, the facilities that may receive
enhanced security monitoring improvements can be selected based on the ranking and available budget
for enhanced security monitoring.
Design:
Establish goals for detection, delay and response that should help either prevent the adversary from
successfully contaminating the water or mitigate the consequences of a contamination event by
successfully detecting the adversary and minimizing the response time. Security systems should be
selected that will allow these goals to be met.
Coordinate system design with the concept of operations as described in Section 2.1 to help establish
the procedure aspects of the security systems and define the requirements for equipment and data
systems.
Select, specify, and locate equipment type and required equipment features based on factors such as
building configuration and design criteria. For example, for camera systems, the available line of site,
required resolution and speed of data transmission must be considered. For motion detectors, the
range of detection, sensitivity and potential facility obstructions must be considered.
Once the design criteria and functional goals are defined, methods can be evaluated to achieve them. The
following are features and equipment that may be included in an enhanced security monitoring system.
Hardening - The focus of physical security improvements is on detection and assessment rather
than delay. Consequently, significant hardening of building structures such as doors and windows
is generally not recommended, but expenditures on facility alarms and cameras are
recommended. Hardening of exterior reservoir access hatches and vents, however, is
recommended where installing cameras or hatch/vent contact alarms is difficult or cost
prohibitive. Concrete or metal enclosures around hatches or vents can physically hide and/or
make it very difficult to add contaminates reservoirs through vents.
o Vents - design to make contaminant addition difficult while still allowing ventilation
o Hatches- add contact alarms if routing electrical conduit feasible or bury/hide
Facility access control - Access control systems allow a means for employees to automatically
disable alarms upon entering a facility and provide records of employee egress.
o Numeric keypads
o Electronic locks - replace lock cylinders on doors and controls access on a per-key basis
using programmable keys
o Coded credentials - proximity cards
o Biometric devices - fingerprint and iris scanners
Contact switches - Contact switches are recommended for all exterior doors or for interior doors
providing access to areas such as pump rooms where an intruder could contaminate water.
Exterior reservoir hatches and valve vault hatches that provide access to water piping should also
have contact switches. Contact alarms may be discrete for specific entry location identification or
daisy-chained to provide a general facility intrusion alarm.
Cameras - Contact alarms, motion detectors, or changes in camera image fields may be used to
activate cameras to assess the alarm. Pan-tilt-zoom cameras can cover more area than stationary
cameras but should be able to quickly turn and focus to area of concern. Camera field of view
and distance from object impact the ability to identify intruders. Adequate lighting is necessary
for assessment. Other factors to consider are resolution and lenses.
o Stationary
o Pan-tilt-zoom
May 2007 73
-------
Planning for WS-CWS Deployment
o "Smart" cameras that recognize suspicious behavior and/or motion in a limited part of the
view
Lighting - Adequate lighting is extremely important when using cameras for identification
purposes. Lamps should quickly illuminate to full brightness.
Motion detectors - Motion detectors are used to detect intruders gaining access through windows,
and storage tank ladders and standpipes. Interferences from shadows and other causes should be
considered.
o Microwave
o Passive infrared (PIR)
o Microwave - PIR - Dual technology minimizes false alarms on ladders.
Glass break sensors - Glass break sensors may be used in lieu of or in combination with motion
detectors.
o Acoustic
o Shock
o Acoustic - Shock - Dual technology greatly reduces false alarms from background
noises.
Audible alarms - Audible alarms are mainly for deterrence. Impacts on neighbors may be a
concern.
Heavy duty conduit and integrity monitoring - Heavy duty conduit and integrity monitoring
protect power and communications wiring.
Obtain any required code reviews or approvals on the design especially relating to egress.
Video system types - Full streaming video provides real time capabilities but is often not a viable
option due to the cost of installing the communications network. Two video packaging
technologies include compressed video clips and flash memory. Compressed video clips systems
are sent in packages over SCADA systems and typically includes digital video recorders installed
in the monitored facilities to store images for potential criminal investigation. Flash memory
systems include a video server to convert analog video signals to digital images and a processor
to provide intrusion event video packaging and transmission. Flash memory may capture
intrusion events but lacks the large video storage capabilities of compressed video clips.
Video transmission - Good video resolution and transmission speed are the key requirements for
rapid assessment.
o Tl lines - Determine capability and interface requirements. Firewalls may be an issue.
o Private radio networks - Perform path measurements to locate repeater sites.
o Digital cellular service - Digital cellular service is a relatively recent technology
development and is not available in all areas of the United States. It requires coordination
with the local wireless carrier to determine the best way to link data from remote
facilities to the command center.
Multiple technologies may be desirable or necessary in some applications. For example, existing
Tl lines may be available for some facilities and would provide excellent video transmission and
response. Since Tl lines are costly to install and may require relatively high monthly
communication fees, alternative video transmission options should be considered for areas
without Tl lines.
Implementation:
Delivery Options: Traditional, Design-build (develop 30 to 50% design documents and
contractor completes design); hardware and equipment could be procured by the utility or
supplied by the contractor as part of the construction contract.
Pre-selection of eligible contractors
Bid and award of contract
Construction - inspection, contractor oversight, contract modifications
Document security issues
Select communications service provider (if applicable), and establish any necessary contractual
relationships.
May 2007 74
-------
Planning for WS-CWS Deployment
Select an installer, and establish any necessary contractual relationships. Installer's experience
with all the technologies needed to complete the work should be heavily weighted in their
selection.
Identify roles and responsibilities for procurement, installation, and testing of various components
of the communications architecture.
Work with service provider to get components installed and configured.
Test communication pathways between enhanced security monitoring sites and the operations
center.
Develop location-specific installation specifications.
Procure system components that may not be provided by the service provider.
Install remote communications systems at selected sites.
Refine concept of operations based on as-built design
Training of security personnel regarding the new monitoring systems, how they should interface
with them, and how they should respond to an alarm
Preliminary Testing:
Initial configuration and calibration of enhanced security monitoring equipment, including
verification of the following:
o Contact switches and glass break sensors activation and transmission
o Cameras work in conjunction with contact alarms
o Lighting works in conjunction with contact alarms and is adequate for video resolution
o Video transmission provides the speed and resolution specified for a full assessment by
bench testing
Troubleshooting
Contractor functional and performance testing
Refine concept of operations, if necessary, based on any modifications
Operation and Maintenance:
In-house versus contracted O&M services
Documentation including, as-built specifications, O&M manual, etc.
Periodic maintenance and calibration, including consumables.
Annual inspection and maintenance of contact switches, motion sensors, glass break sensors,
cameras and other physical security equipment.
Unexpected maintenance events
Amortized replacement costs
Evaluation and Refinement:
Evaluate frequency and cause of false alarms
Conduct drills and exercises to identify refinements to concept of operations and to assess and
improve equipment performance and alarm response
6.3 Available Tools and Resources
The following tools and resources are available to support the design, installation, operation, and
evaluation of physical security improvement for the enhanced security monitoring component of a
contamination warning system:
AWWA. Interim Voluntary Security Guidance for Water Utilities, 2004.
AWWA. Guidelines for the Physical Security of Water Utilities, 2006.
Garcia, Mary Lynn. The Design and Evaluation of Physical Protection Systems, 2001.
The Integrated Physical Security Handbook. Philpot and Einstein, Homeland Defense Journal.
2006. http://www.phvsicalsecuritvhandbook.org
May 2007 75
-------
Planning for WS-CWS Deployment
USEPA Water and Wastewater Security Product Guide.
http://crpub.epa.gov/safewater/watersecurity/guide/
Sandia Corporation. Risk Assessment Methodology for Water Utilities (RAM-W), 2002.
National Association of Clean Water Agencies (NACWA). Vulnerability Self-Assessment
Toolfor Water & Wastewater Utilities (Version 3.2 Update), 2005.
http://nacwa.org/pugs/index.cfm
USEPA. 2003. Instructions to Assist Community Water Systems in Complying with the Public
Health Security andBioterrorism Preparedness Response Act of 2002 (EPA 810-B-02-001).
6.4 Staffing and Cost Considerations
Planning for the implementation of the enhanced security monitoring component of a contamination
warning system requires involvement of a wide array of utility personnel and potentially contractor staff.
Costs may be highly dependent on the utility's capabilities and intended enhancements. Therefore, the
remainder of Section 6.3 illustrates the staffing considerations and cost factors that are recommended for
consideration during project planning and pre-design.
6.4.1 Staffing
As mentioned above, staffing considerations are critical to the successful implementation of a
contamination warning system. Table 6-3 offers a quick overview of which staff may be necessary to
design, implement, and operate an online water quality monitoring program as part of a contamination
warning system and during which phases of implementation these personnel may be needed.
Table 6-3. Enhanced Security Monitoring Staffing Considerations
Division or Department
Water Quality
Information Technology
Engineering and Planning
Operations and/or
Distribution
Security/Administrative
IT
Implementation Stage
PD
X
X
X
X
X
X
D
X
X
X
X
X
X
I
X
X
X
X
X
X
PT
X
X
X
X
X
O&M
X
X
X
X
E&R
X
X
X
X
X
X
Comments
Helps define monitoring system requirements
Designs, implements, and manages all IT systems
used to support communications and data
management needs of enhanced security monitoring
Review and approval of designs. Responsible for
installation oversight and inspections. Support
security system design.
Provide input during design. Supports selection of
distribution system security equipment and
improvements. Lead in operations and maintenance
of equipment. Support installation of security
systems.
Lead in selection of distribution system security
equipment and improvements. May be responsible
for monitoring and responding to alarms.
Designs, implements, and manages all IT systems
used to support enhanced security monitoring.
PD = Pre-design; D = Design; I = Implementation;
Maintenance; E&R = Evaluation and Refinement
PT = Preliminary Testing; O&M = Operations and
Building the team to implement this component of a contamination warning system should involve all
divisions within the utility. Therefore, it is important to have senior leadership involved and invested
during each stage to facilitate the resolution of cross-division issues. Several key personnel, like the IT
manager and the Water Quality division manager, would ideally be members of this and all other
component teams to facilitate the application of the system engineering principles outlined in Section 2.1.
Such involvement can be a significant commitment of time and resources for these individuals, but the
utility can reap substantial benefit in the long-term success of the system. Other team members'
participation may be less demanding, but still critical, and may vary depending on the utility-specific gap
between the initial conditions and the final planned capabilities of the contamination warning system.
Furthermore, it may be useful to obtain the help of consultants and contractors to aid the utility in the
May 2007
76
-------
Planning for WS-CWS Deployment
implementation of an enhanced security monitoring system, especially given that the level of effort
required to implement this component would go beyond the human resources at many utilities.
Consultants and contractors can be critical partners during each stage of the process by providing
component-specific technical knowledge and installation experience, and by delivering training on the
new equipment, procedures and processes.
6.4.2 Cost Considerations
This section presents a summary of the design and implementation considerations discussed above that
may influence costs. This list also may include other factors that were encountered during
implementation of the initial Water Security initiative pilot and could be overlooked during cost
estimation in the absence of this experience. Although this list of cost considerations may not be
exhaustive, these factors, at a minimum, should be considered when planning.
Development of design objectives for both elements of the enhanced security monitoring
component
Assessment of existing security, communications, SCADA and IT systems
Development of preliminary concept of operations for the enhanced security monitoring
component, including decisions regarding design basis threat, facilities to be protected,
monitoring equipment to be deployed and the approach for communicating and responding to
alarms.
Field verification of preliminary locations and development of installation specifications for
selected locations
Layout and design of security equipment and communications system components (for each site).
Even if the enhancements are handled as a design-build, significant design work may be
necessary to produce the preliminary drawings.
Design of communications and data management architecture; testing of communication
pathways and conducting radio survey if necessary
Installation of security and communications equipment at each site
Establishment of contract with communication service provider(s); procurement, installation and
configuration of communications and data management hardware and software; depending on
existing infrastructure and topography, laying fiber optic cable or erecting antennas may add
significant cost.
Initial equipment shakedown and training on operation, maintenance and alarm response
Communications and IT Architecture: test of installed data management architecture and
complete communication system
Development of written documentation; scheduled and unscheduled security monitoring
equipment maintenance, repair and upgrades
Service fees for communications service provider(s); development of written documentation
related to communications architecture; scheduled and unscheduled communication equipment
maintenance, repair and upgrades
Equipment upgrades due to improvements in sensing and or communication technology
Drills and exercises
Based on experience at the pilot utility, the most significant issues may likely be the capability of the
existing communication system and the capital costs associated with the installation of the security
monitoring equipment. Finally, the cost to maintain and upgrade security monitoring equipment can be
significant, and should be considered early in the planning stage to ensure that the system implemented
can be sustained.
May 2007 77
-------
Planning for WS-CWS Deployment
Section 7.0: Consumer Complaint Surveillance
Located throughout a utility's distribution network, consumers can provide near real-time input regarding
changes in water characteristics discernable through the senses. Consumers may detect contaminants
with characteristics that impart an odor, taste, or visual change to the drinking water. Complaints from
residential, commercial and industrial consumers are routinely reported to water utilities on a very timely
basis. As such, consumer complaints may provide one of the earliest warnings of a possible
contamination incident for contaminants in classes 1 through 5, if an effective system is in place to detect
anomalous trends in complaints and quickly respond to them. Generally, utilities document reports of
unusual water characteristics and use them to identify and address water quality problems. The
procedures and systems used to handle these reports are commonly referred to as the utility's consumer
complaint management system.
As part of a contamination warning system, the complaint management system should extend beyond just
managing complaints - the system should monitor the complaint handling process and identify when
conditions could be indicative of a water quality problem. This can be achieved by identifying
information within the consumer complaint management system that can serve as an indicator of a
possible contamination event when monitored and compared against a threshold value. If the collective
information indicates an anomalous pattern in water quality calls, the contamination warning system
triggers an alarm, followed by further investigation. By expanding on existing systems that manage
consumer complaints, consumer complaint surveillance provides dual-use benefits to the utility by
enhancing its customer service as it captures early signs of a potential water quality issue. Table 7-1
summarizes the design basis for consumer complaint surveillance and provides considerations as to how
these objectives impact design and implementation.
Table 7-1. Design Basis Considerations for Consumer Complaint Surveillance
Design
Objective
Capability
Contaminant
Coverage
Spatial
Coverage
Timeliness
Reliability
Sustainability
Description
Can indicate the presence of a
contaminant that significantly affects one
or more aesthetic qualities of water.
High detection potential for classes 1 and
2. Moderate detection potential for classes
3, 4, and 5.
Entire service area for contaminants with
detectable taste, color, or odor
characteristics.
Function of the time from exposures to
consumer reporting, complaint
categorization, assessment and
investigation.
A potentially reliable indicator for
contaminants with detectable
characteristics if a robust complaint
reporting and tracking system is in place.
Provides utility an opportunity to manage
consumer information more effectively
and can serve as a tool for enhanced
consumer confidence.
Design and Implementation Considerations
Detection of aesthetic changes is an indirect measure
of contamination.
Contaminant coverage may also be a function of
consumer education. In addition, population diversity
may impact the ability to detect an aesthetic change in
water quality.
While consumer complaint surveillance can
encompass the entire service area, it is important to
consider how calls are handled outside the utility if
multiple jurisdictions or municipalities are served.
Time to detection is a function of the system and
procedures used to detect and investigate an anomaly
in the number, type, or distribution of water quality
complaints.
Reliability is a function of educated consumers and
their ability to notice aesthetic changes and ultimately
report these to the utility in a timely manner. It is also
a function of the system used to detect anomalies.
Enhancements to the consumer complaint
management system should be integrated with routine
job functions and should improve customer
service/satisfaction and potentially day-to-day
operations in a call center.
The objective of this section is to assist in the planning for implementation of consumer complaint
surveillance as part of a contamination warning system. Although many utilities currently implement
some consumer complaint monitoring and management activities, typically these activities are not
integrated in a manner to support contamination warning system objectives, such as the timely
May 2007 78
-------
Planning for WS-CWS Deployment
recognition of possible contamination. The consumer complaint surveillance component should provide
utilities with a mechanism to enhance their current call management system and incorporate consumer
call information with other contamination warning system monitoring and surveillance data. Design
decisions to consider include determining how to:
Educate consumers. Utility consumers who are aware of indicators of potential contamination,
including suspicious activity and unusual water characteristics, and who are educated on how to
report such indicators, are an invaluable tool in detection of a potential water quality incident.
Capture all complaints. A "funnel" for collecting all water quality complaints into the
consumer complaint surveillance system should exist. For example, a unified call center with a
widely publicized telephone number in place to capture the largest percentage of potential
complaints. In addition, procedures should be in place to capture complaints that are directed to
other points inside the utility that are initially received by other agencies. In cases where
customer calls are managed outside of the utility, priority should be given to tunneling water
quality related calls to the utility in a timely and efficient manner.
Electronically manage data. All water quality complaints should be entered into an electronic
database as they are received and categorized by type. A complaint record is carried through the
process with information being added to it as it is received or investigations are conducted, and
duplicate data entry is minimized or eliminated. Each complaint should be tracked from receipt
to closure and retained in a historical database.
Automate and integrate data analysis. Once all of the information (water quality complaint
data and location) is collected for a call, it can be evaluated based on utility known parameters.
This can be accomplished using manual procedures already in place for review of complaint data
or through an automated event detection system. An event detection system is an automated
software and/or hardware system that analyzes data in near "real-time" for information that may
be indicative of a contamination incident based on a utility pre-determined threshold. For
example, the frequency or locations of water quality complaints could provide indication of
possible contamination
Establish procedures and protocols. Written standard operating procedures (SOPs) for every
step in the water quality complaint handling process is an essential attribute of consumer
complaint surveillance. These SOPs should facilitate effective and timely communications,
including clear guidance regarding the decision process to determine appropriate response
actions, such as field investigations and sampling and analysis. All personnel potentially involved
in the consumer complaint management system should be trained in these procedures.
Address additional training of personnel. Trained and dedicated personnel may be crucial to
implementing a successful consumer complaint surveillance process. These people are the front
line contacts who interact with the consumers and process their input. The experience and
professionalism of the utility personnel are critical to timely and accurate recording of data, and
their judgmentin conjunction with automated and integrated data analysismay be vital in
assessing when a possible contamination incident has occurred.
Achieve continuous system improvement. The consumer complaint management system
should be evaluated routinely (at least every five years) for two reasons: 1) to gauge how well it is
meeting the intended goals of consumer complaint surveillance as well as other benefits derived
from enhancements to the consumer complaints management system and 2) is to identify
technological innovations or procedural modifications that warrant changes to the system.
Throughout the enhancement process, the utility should also be mindful that the efficacy of the consumer
complaint surveillance component is contingent upon the occurrence of data collection, analysis and
notification steps in a timeframe that allows for effective response actions to mitigate a water
contamination incident.
May 2007 79
-------
Planning for WS-CWS Deployment
7.1 Pre-Design
Pre-design for the consumer complaint surveillance component of a contamination warning system
involves completing an assessment and gap analysis of a utility's call center procedures and call
management system. The purpose of the assessment and gap analysis is to determine to what extent the
current consumer complaint management system procedures and data systems can be modified and
recommend enhancements to consumer complaint management systems to meet the attributes of an
effective consumer complaint surveillance component
Consumer Complaint Surveillance Model
The design of a consumer complaint surveillance component of a contamination warning system is based
on a model of "Funnel, Filter and Focus" as described below:
Funnel. All customer calls should be directed to one-point of contact within the utility
Filter. Utility employees who routinely handle calls should be able to respond to billing,
metering reading and general water quality concerns. Calls concerning more specific water
quality concerns should be forwarded to other appropriate personnel
Focus. Personnel with training and experience in water quality should gather in-depth
information for the consumer and make a determination of the need for field sampling.
Figure 7-1 illustrates this model. In cases where consumer calls are managed outside of the utility, a
similar approach should be applied to manage water quality related calls.
LJJ
CC.
LJJ
Primary Source:
Water Utility Call Center
Secondary Source: ^ I ^ Secondary
Other Agencies ^^^ I ^f Source:
^ * r Other Utility
Departments
Non Water Quality
Related Calls
Non Water Quality
Related Calls
(As determined by
Call Center Staff)
Water Quality Calls
Related to System
Operations
(Main Breaks,
^~~ ^
Operations
Staff
/
o
o
Water Quality Specialist
analyzes remaining
complaints for indications
of possible contamination
Figure 7-1. The Recommended Filter, Funnel, and Focus Approach to Customer Feedback Data
Optimization for Utility-Managed Consumer Calls
May 2007
80
-------
Planning for WS-CWS Deployment
Assessment and Gap Analysis of Existing Consumer Complaint Management Activities
Based on experience at the pilot utility, an assessment and gap analysis is an effective tool to assess the
current consumer complaint management activities and identify enhancements that should achieve the
objectives of a contamination warning system. Such an assessment involves a thorough investigation of
the utility call center and its call handling procedures. The goal is for the consumer complaint
surveillance team to develop an understanding of the entire call management system.
During the assessment, the consumer complaint surveillance team gathers an accurate picture of the
utility's call management system. Superimposing this picture onto the model of a consumer complaint
surveillance component outlined in Figure 7-1, identifies the gaps where enhancements could bring the
existing complaint management system into alignment with the model. Key considerations in the analysis
include:
Identification of consumer complaint surveillance data streams. The utility should identify
existing data streams related to water quality calls and the systems currently used to process these
calls (e.g., call and work management systems, if available). As the utility assesses the data upon
which a consumer complaint surveillance system is to be built, it is important to consider the
sources from which that data is to be derived. It may be necessary to consider how data is to be
aggregated and integrated from such disparate sources as third-party call management system
hosts, other organizations within the city/jurisdiction, or other sources.
Establishing a trigger and process for event detection. Once the existing data steams are
identified, the next step is to establish a trigger level(s) based on background consumer complaint
call volumes at a particular time. It should be recognized that consumer complaint call volume
may vary based on the day of the week or time of day. Depending on how complaints are
captured the data analysis can be completed by breaking the data into complaint categories by day
of the week and/or by month. This process should be a method for counting the frequency and
spatial distribution of water quality related consumer complaints over time and notifying
responsible stakeholders once the characteristics of these data surpass the established level.
Ultimately the goal is to answer the question, when does the complaint frequency rise to a level
that indicates a significant change in water quality has occurred. Automation can occur through
simple frequency counts based on time and location(s) of water quality complaints or could be a
more sophisticated algorithm, such as a programmed EDS which operates behind the scenes of
the existing consumer complaints management system hardware and software data systems.
Alarm notification and user interfaces. Once a consumer complaint surveillance trigger level
is surpassed, the user should be notified of the event. The notification process should attempt to
utilize existing hardware and software systems available to the extent possible, such as e-mail
and/or text messaging. Multiple notification procedures may be necessary to ensure coverage for
both business and non-business hours. Other user interfaces to consider are graphical information
systems for display of alarm data.
Linkage of data systems. It is critical to recognize during the assessment process that an
effective call management system for consumer complaint surveillance should be able to link
across associated data bases such as: call tracking, customer information and the utility's asset
management systems. This ability should allow the utility to quickly identify the location of the
complaint and facilitate geographical analysis of the complaint data. It would also provide the
utility a significant dual-use benefit by facilitating tracking of all maintenance related issues from
call receipt until work-order close-out.
7.2 Design and Implementation Approach
The activities described in Section 7.1 describe a process for pre-design of a consumer complaint
surveillance system, culminating in documentation of an overall framework and selection of a
methodology for designing the system. The design and implementation of the consumer complaint
May 2007 81
-------
Planning for WS-CWS Deployment
surveillance component should be based on the outcomes of pre-design and may include the
considerations described below.
Design:
Perform additional reviews of historical water quality consumer call data analysis (if available)
Clarify requirements of software/hardware needs (including event detection system)
Develop a detailed method for automating data collection and analysis
Refine notification protocols (e.g., text message, e-mail, pager)
Develop end-to-end test protocols to verify system functionality
Develop a preliminary concept of operations to describe routine operation of the component and
additional user-specific standard operating procedures as needed
Create a work plan and schedule to meet the design and time requirements of the project
Implementation:
Install and test necessary hardware/software or other equipment for data gathering and
automation
Install and configure EDS tools
Revise concept of operations as appropriate to reflect the routine operation of the component as-
built
Conduct staff training on consumer complaint surveillance tools, systems, and procedures
Preliminary Testing:
Integrate the consumer complaint surveillance concept of operations into routine operations
Collect and evaluate additional baseline data used in event detection
Refine EDS tool configuration
Operation & Maintenance:
Monitor and respond to EDS alarms in accordance with the concept of operations.
Log any anomalies detected by the EDS tool(s).
Reconfigure the EDS tool if performance is found to be unsatisfactory.
Verify that EDS tools and supporting software are current with respect to vendor-provided
updates, patches, etc.
Conduct periodic staff training on concept of operations
Evaluation & Refinement:
Continuous documentation of performance during operation, e.g., false alarms, detection of true
water quality anomalies, and dual-use benefits
Through simulated events, periodically evaluate the overall performance of the operational
consumer complaint surveillance component. If necessary, update procedures or EDS tools to
optimize performance.
Track and evaluate the development of new EDS tools
7.3 Available Tools and Resources
The following tools and resources are available to support design and implementation of consumer
complaint surveillance as part of a contamination warning system:
Whelton, A., Dietrich, A.M., Gallagher, D.L, Roberson, A. "Using Customer Feedback for
Improved Water Quality and Infrastructure Monitoring," submitted to Journal of the American
Waterworks Association. (Dec. 2006).
Dietrich, A.M. "Aesthetic issues for drinking water," Journal Water and Health, Volume 4,
Supplemental 1 (2006).
May 2007 82
-------
Planning for WS-CWS Deployment
Whelton, A., Dietrich, A.M., Burlingame, G.A., Cooney, M.F., "Detecting Contaminated
Drinking Water: Harnessing Consumer Complaints," submitted to J. Amer. Water Works Assoc.
(Dec. 2006).
Lauer, William C. "Water Quality Complaint Investigator's Field Guide." American Water
Works Association (2004).
7.4 Staffing and Cost Considerations
Planning for the implementation of the consumer complaint surveillance component of a contamination
warning system requires involvement of a wide array of utility personnel and potentially contractor staff.
Costs may vary based on the utility's existing capabilities and the extent of enhancements. Therefore, the
remainder of Section 7.4 illustrates the staffing considerations and cost factors that are recommended for
consideration during project planning.
7.4.1 Staffing
Owing to the potential complexity of creating or modifying call management systems to accommodate the
objectives of a contamination warning system, a variety of staff, both internal and external to the utility
should be considered. Depending on in-house expertise, it may be necessary to consider engaging third-
party contractors to modify existing systems. Table 7-2 provides an overview of the staff and resources
that may be engaged to design, implement, and operate a consumer complaint surveillance component as
part of a contamination warning system.
Table 7-2. Consumer Complaint Surveillance Staffing Considerations
Division / Department
Water Quality
IT
Engineering
Supply
Distribution
Administration (internal
Call Center
Outsourced Call Center
City Call Center
Phase
PD
X
X
X
X
X
D
X
X
X
X
X
I
X
X
X
X
X
PT
X
X
X
X
X
X
X
O&M
X
X
X
X
X
X
X
ER
X
X
X
X
X
X
X
Comments
Provide water quality information during
investigations of water quality complaints
Designs, implements, and manages all IT systems
used to support call management system
N/A
Provide investigation support to water quality calls
Provide investigation support to water quality calls
Recognize water quality related complaints and pass
on to Water Quality Personnel
Provide call center support
Provide call center support
PD = Pre-design; D = Design; I = Implementation;
Maintenance; E&R = Evaluation and Refinement
PT = Preliminary Testing; O&M = Operations and
Building the team to implement this component of a contamination warning system may involve man
divisions within the utility. Therefore, it is important to have senior leadership involved and invested
during each stage to facilitate the resolution of cross-division issues. Several key personnel, such as the
IT manager and the water quality division manager or equivalent, would ideally be members of this and
all other component teams to facilitate the application of the system engineering principles outlined in
Section 2.1. Such involvement can be a significant commitment of time and resources for these
individuals, but the utility can reap substantial benefit in the long-term success of the system. Other team
members' participation may be less demanding, but still critical, and may vary depending on the utility-
specific gap between the initial conditions and the final planned capabilities of the contamination warning
system. If the call center is outsourced, the utility should incorporate the requirements of the consumer
complaint surveillance component of the contamination warning system into the contract with the call
center contractor to the extent possible.
May 2007
83
-------
Planning for WS-CWS Deployment
7.4.2 Cost Considerations
This section presents a summary of the design and implementation considerations discussed above that
may influence costs. This list also may include other factors that were encountered during
implementation of the initial Water Security initiative pilot and could be overlooked during cost
estimation in the absence of this experience. Although this list of cost considerations may not be
exhaustive, these factors, at a minimum, should be considered when planning.
Baseline assessment costs may include inspection and formal assessment report development by a
project team to document the utility's existing call management system hardware, software and
business management practices and to assess current capabilities to meet the contamination
warning system objectives.
Concept of operations costs include document development to specify the utility process and
information systems to be used during routine operation and initial trigger validation of the
consumer complaint surveillance component.
Modification of existing call management software or the assessment and procurement of new
call management software to meet the contamination warning system objectives for this
component.
Assessment of existing consumer complaint data streams to determine an approximation of event
trigger level and applicability of statistical algorithms. The cost may also include coding utility-
specific automated analysis tools along with the consumer complaints management system data
extraction and transformation for data processing.
Design and testing of consumer complaint surveillance alarm notification. This process may
require the procurement of new hardware and/or software to ensure the timely display of alarms.
Additional cost may also be incurred to incorporate and test GIS display of possible consumer
complaint surveillance water contamination alarms, if this capability is desired.
Deployment and testing of automated analysis tools along with the consumer complaints
management system data extraction and transformation for data processing.
Training on enhancements and concept of operations.
Testing of existing call management software or new call management software to verify
operation is consistent with the design.
Software and hardware upgrades for call management system, event detection tools and alarm
notification system
Periodic evaluation and recalibration of consumer complaint trigger values, based on a review of
historic data. Refinement efforts could also focus on a review of the effectiveness of the concept
of operations.
Conducting a consumer complaint surveillance evaluation drill. Consider one drill per year to
include all personnel identified within the concept of operations for this component.
May 2007 84
-------
Planning for WS-CWS Deployment
Section 8.0: Public Health Surveillance
Public health surveillance systems gather and analyze health-related data to identify anomalies that might
indicate unusual incidence of disease. The role of public health surveillance in a contamination warning
system is to gather and analyze data for investigation that will augment traditional epidemiological
surveillance (which often relies on an astute clinician to notice and report anomalies). When anomalies
are detected by public health analysis, coordinating with the utility will assist in determining whether the
anomaly is related to water. The involvement of public health experts in a contamination warning system
also adds a unique area of expertise that can not only support this component, but also consequence
management.
Some public health data, such as OTC drug sales and emergency room visits, may be suited to detecting
biological contaminants found in contaminant classes 10 and 11 (Table 1-1). Other data, such as EMS
records, 911 call data, and Poison Control Center data may be better at detecting fast-acting chemicals,
like those found in contaminant classes 1-9.
Public health surveillance is performed by many entities, including local health departments, 911 call
centers, Poison Control Centers, and nationwide surveillance systems, such as the National Retail Data
Monitor, which gathers and analyzes OTC data. Depending on the utility's area of service, there could be
numerous local health departments (the locality could include city, county, or regional departments). Each
of these entities is responsible for different areas, and some parts of the utility service area may not be
covered by all of the public health partners. For example, a county health department may cover the
entire utility service area, but 911 call data may be limited to one designated public service answering
point (such as one city within the service area). Because of this, coordination with as many entities as
feasible is needed to cover as much of the service area population as possible.
Public health surveillance systems that would be appropriate for a contamination warning system can be
grouped into two categories:
Traditional surveillance systems include hospital disease reporting and laboratory reports.
These are generally collected by the health department(s) and analysis is performed using
morbidity rates (e.g., the number of measles cases per county per year). However, collection of
these data can be relatively slow, with a lag time of days or weeks. Therefore, these data may
be more useful for follow up investigation and false-alarm evaluation, but other more timely
approaches to data collection and analysis should be considered.
Syndromic surveillance aims to use any data available in as near real-time as possible to detect
possible outbreaks based on statistical analysis of "syndromes," or categories of disease. This
surveillance approach may be more useful for quick detection of the contaminants of concern.
Syndromic surveillance can also involve performing "fused analysis," whereby information
from many sources is analyzed together to detect possible anomalies. Some examples of
syndromic surveillance programs currently being used by health departments include BioSense,
the National Retail Data Monitor (NRDM), RODS and ESSENCE (see Section 8.3 for
additional detail).
The overall objective of this section is to describe options and a process to develop a design and
implementation strategy for the public health surveillance component of a contamination warning system,
based on existing capabilities in the area where the utility is located. This strategy will help local public
health identify routine health changes more quickly and aid in cooperation with the utility to determine
whether a possible water contamination is the cause of the health anomaly. Design basis considerations
to achieve successful public health surveillance are summarized in Table 8-1.
May 2007 85
-------
Planning for WS-CWS Deployment
Table 8-1. Design Basis Considerations for Public Health Surveillance
Design
Objective
Capability
Contaminant
Coverage
Spatial
Coverage
Timeliness
Reliability
Sustainability
Description
Can detect the presence of a symptom or
illness in a population which may be the
result of the presence of a disease causing
agent. May be able to identify the
contaminant through clinical diagnosis/
testing.
Covers contaminant classes 2 through 1 1 ;
detection potential varies with type of
surveillance.
Comprehensive coverage of a particular city
or county, which may include all, or a large
portion of, the utility service area.
Function of the time from the initial
exposures, the onset of symptoms, and the
point at which public health officials
recognize the incident as a potential water-
borne illness.
May be a reliable means of identifying the
incidence of illness in a population, but
timing of communication between drinking
water and public health officials should be
optimized such that appropriate response,
actions could be implemented in time to
reduce consequences.
Provides an opportunity for collaboration
between utility and local health
department(s).
Design and Implementation
Considerations
Consider appropriate algorithms and the
concept of fused analysis to look at many
results concurrently (Burkam, et. al).
Include public health data streams for both
fast-acting chemicals and biological
contaminants.
Should include participation of numerous
agencies and partners; any one type of
data may not cover the entire service area.
Where possible, automation of data
collection and analysis as well as alerts will
increase timeliness. New data streams
such as 91 1 or EMS data should be
considered.
Develop a well defined communications
protocol and Concept of Operations,
including role descriptions and a firm
commitment to investigation.
Should be able to be performed without
compromising other roles of public health
agencies.
8.1 Pre-design
Design of the public health surveillance component depends primarily on existing public health
surveillance systems within the utility's service area and will vary greatly between utilities. Integration of
public health surveillance in the contamination warning system will involve relationship building as much
as analytical implementations. Appropriate roles, limits of information sharing, and how investigations
will proceed should be addressed, with the end goal of communicating and responding to possible events
more effectively.
The pre-design process for implementation of public health surveillance in a contamination warning
system should consider the following:
Identification of local public health partners
Identification and assessment of existing capability
Development of a framework for communication and notification
Identification of Local Public Health Partners
The first step in the pre-design process is to identify current relationships between the utility and public
health partners. If there is already a high degree of cooperation and coordination between the utility and
local health partners, an expansion of these relationships should be considered to include a contamination
warning system. If no relationship exists, points of contact should be identified for those who can
effectively share information and participate in joint investigations of possible contamination events. A
point of contact in the local health department(s) is particularly important. In addition to local public
health departments, it is also important to engage the local or regional poison control center, local fire
departments, dispatch centers, and in some cases State health departments as well. Because the
contamination warning system will engage agencies that may, under other circumstances, have little need
to interact, an effective way of communicating information should be established early in the planning
process.
May 2007
86
-------
Planning for WS-CWS Deployment
Identification and Assessment of Existing Surveillance Capability
Data streams that could be available for public health surveillance should be identified with the help of
local health departments, fire departments, Poison Control Centers, and other partners. These data streams
may include EMS, 911, laboratory tests, hospital data, Poison Control Center calls, and OTC data.
Surveillance capability may vary from jurisdiction to jurisdiction in terms of degree of automation, type
of surveillance, and area covered. It is important to assess the capability of the existing surveillance
systems relative to the contamination warning system objectives of contaminant coverage, spatial
coverage, timeliness (degree of automation), and reliability. The relationships between possible public
health data and the design objectives of spatial coverage, timeliness, and contaminant coverage are shown
in Figure 8-1.
It is important to note that, while automation of data analysis is a desired goal of a contamination warning
system, this may not always be the most cost-effective means of improving surveillance capabilities.
Response times and communication protocols also may be improved significantly by evaluating and
optimizing manual processes and procedures. The costs, organizational feasibility, and enhanced
surveillance capabilities yielded by these improvements should be weighed against the costs and benefits
of automated data analysis options. Consideration should also be given to the ability to detect both fast
acting contaminants and contaminants with a long latency period.
SPATIAL TIMELINES OF
DATA SOURCE COVERAGE DATA LOCATION COLLECTION
f5Tj5fBil City and/or County Near Real-Time
TO^-lmr3 Department
EMS
911 City and/or County ^^ NeaMTme
JWl ""S'0"81 Poison Control Near Real-Time
^53s>i Center
^Poison
Wdfc Civ/County/
w*w Regional
t^v*-,
City/County/
^ Regional LPH Days or Weeks
Hosp tats
INITIAL CONTAMINANT
DETECTED
Fast-Acting Chemicals
(Classes 1-9)
Fast-Acting Chemicals
(Classes 1-9)
Fast-Acting Chemicals
(Classes 1-9)
Pathogens
(Classes 10-11)
Pathogens
(Classes 10-11)
Figure 8-1 Public Health Data Sources and Design Objectives
At a minimum, data should be applicable to the detection of biological and chemical contaminants though
either formal diagnosis or results (i.e., a laboratory test) or syndromic surveillance (i.e., OTC data
profile). Especially important to consider is whether the data collection process can be automated, which
can decrease time and effort of collection and analysis and thereby increase sustainability. Not only will
this be helpful as part of the contamination warning system, but also will benefit the local health
department in performing faster epidemiological analyses.
In the absence of data automation, other approaches for achieving the design objectives should be
considered. These approaches could include the following:
May 2007
87
-------
Planning for WS-CWS Deployment
Increased manual surveillance activities through increased staff to increase frequency of data
monitoring
Increased breadth and scope of manual surveillance activities
Working cooperatively with local health departments to fund and add staff positions to their
operations dedicated to surveillance activities focused on waterborne contamination
Because the data streams should be complementary to each other and to data available at the utility,
possible anomalies may be quickly verified or discounted. For example, a high volume of 911 calls could
be compared with EMS run volume and run location to verify whether the high volume of 911 calls
indicates a real event, or is just due to chance. Data should also be representative of the entire utility
service area; if a utility serves multiple counties, then data from each county should be utilized. Public
health data under consideration also should be compared with utility data to see where they might
supplement each other. An example would be mapping 911 call data against consumer complaints calls
to provide an investigative starting point.
Development of a Framework for Communication and Notification
Utility and public health partners should evaluate any current notification protocols. A call list, phone
tree, or other established contact order may already exist. If a protocol exists, the organizations should
determine whether the current protocol is appropriate for a contamination warning system, and then
modified and/or updated into a preliminary concept of operations, as discussed in Section 2.0. If there is
no communication protocol available, it should be created.
The communication protocol should involve personnel who understand the concepts of a contamination
warning system, can interpret data presented to them in relation to possible contamination events, and will
know how to proceed with the investigation. Where possible, automation of alerts, such as e-mail alerts
generated by an analysis program and sent to predefined recipients, should be considered. Roles also may
need further refinement to clearly define who will be doing what at each stage. This will ensure not only
that every investigative duty is performed, but also that efforts are not duplicated. Identifying who does
what also will help effectively use the expertise of public health officials. As part of the assignment of
roles and responsibilities, a strong commitment to follow-through on these roles and responsibilities
should be emphasized.
When developing a communication protocol, issues related to the Health Insurance Portability and
Accountability Act (HIPAA) should be considered. This law recognizes that advances in electronic
technology could erode the privacy of health information and mandates privacy protections for
individually identifiable health information. Health officials have access to data that, due to HIPAA
limitations or other constraints, are not usually available to utilities.
Information typically protected under HIPAA includes any information that can be used to identify an
individual, and how this is interpreted may vary by jurisdiction. For example, public health officials may
determine that age, sex, and address, data cannot all be included, as this information could identify a
patient even though the "Patient ID" field is not provided. However, communication between the utility
and local public health agency(ies) is essential, and data sharing should be conducted to effectively
investigate anomalies. For example, based on discussions among the utility and local health departments
participating in EPA's initial contamination warning system pilot, the following data elements were used
for analysis:
Patient location (zip code or address for EMS and 911 data, respectively)
Event date and time
Chief complaint (used to categorize into syndromes).
May 2007 88
-------
Planning for WS-CWS Deployment
Care should be taken with access to data presented in a User Interface to ensure they are compliant with
HIPAA. More information on HIPAA and compliance with can be found at the Department of Health
and Human Services website (http://www.hhs.gov/ocr/hipaa/)
At the conclusion of the pre-design stage, the utility should have an understanding of the public health
surveillance tools within the utility's service area and approach for engaging and coordinating with public
health partners to leverage or adapt these tools to meet the contamination warning system design
objectives. This approach should stress cooperation among public health organizations and the utility to
improve existing data surveillance, event detection, and communication practices without compromising
current public health or utility services.
8.2 Design and Implementation Approach
The activities discussed in Section 8.1 describe design factors that should be considered when designing
and implementing a public health surveillance component within a contamination warning system.
Design:
Develop a preliminary concept of operations to describe the process flow, data streams, and roles
and responsibilities
Establish how the data should be efficiently gathered
o Assess methods for automating data and/or adding data streams, including data that may not
have been electronically captured previously (e.g., 911 and/or EMS data)
o If automation is not feasible and/or cost effective, consider alternative methods of providing
public health support to the contamination warning system (e.g., increased frequency or
scope of manual surveillance activities or staff support to local health departments to focus
on waterborne contamination)
o Consider participation in BioSense, RODS, ESSENCE, or another syndromic surveillance
program to improve data collection capabilities, and determine if the area is covered by the
NRDM
In collaboration with local health departments, research and identify appropriate analysis tools
o Consider methods that can analyze data by time (i.e., counts per day) and by location (i.e.,
counts in a spatial cluster) in as near real-time as possible
o Research available literature to determine which algorithm(s) are appropriate for the data.
Possible analysis tools include regression models, multivariate models (e.g., CUSUM), and
spatial analysis
o Consider a fused analysis approach
Establish a means for displaying data in a HIPAA-compliant way in one user-friendly location,
such as a User Interface on a centrally-accessible website.
Implementation:
Optimize data collection and analysis procedures and protocols through implementation of
refined procedures or installation of software and equipment to support automation
o IT mapping of data on servers
o Computer code for automating and optimizing datasets
Install analysis tools decided upon in design stage
o Software to run statistical algorithms
o Computer code for automating analysis
o Display of results on a user-friendly interface
Hire additional staff that may have been identified in the design stage
Conduct training on analysis tools and interpretation of results, particularly for new analysis tools
Hold meetings between utility and public health agency(ies) to educate on roles and
responsibilities
Participate in table-top and other communication exercises
Implement data agreements, where necessary (e.g., a contract with the Poison Control Center)
May 2007 89
-------
Planning for WS-CWS Deployment
Preliminary Testing:
Based on analysis of historical data, determine when an anomaly requires investigation relative to
contamination warning system objectives. This "alert level" may be revised based on results
from data generated during the preliminary testing stage.
Determine acceptable false-alarm levels (How often is it acceptable to investigate a result that is
not a true anomaly?)
Adjust roles and communication protocols based on results from table-top exercises and other
exercises
Revise concept of operations as appropriate based on design and implementation activities and
results from preliminary testing activities
Operation and Maintenance:
Use communication protocol for investigation
o Contact appropriate people during anomaly investigation
o Share information agreed upon during pre-design stages
o Update contact lists as necessary
Continue data gathering and analysis
Perform data maintenance activities as necessary.
o Software upgrades
o Archive old records
Evaluation and Refinement:
Evaluate ease of interpretation and usefulness of data, including sensitivity (false-alarm)
measurements and data quality issues.
Determine how well communication protocol is working, and consider methods of improvement.
o Consider further automation of alerts, such as email or text message
o Consider improvements to any manual data-gathering methods
o Consider information shared and if it is useful
Adjust alarm thresholds as necessary, based on sensitivity measurements.
Adjust data gathering frequency as necessary. This could also include automation of data not
previously automated.
8.3 Available Tools and Resources
The following resources are available to assist in design and implementation of the public health
surveillance component:
Early Aberration Reporting System (EARS). EARS is a software program developed by the CDC for
the purpose of syndromic surveillance. Versions of the program can be run using either SAS or EARS.
Public health data can be run through EARS to detect possible outbreaks or bioterrorism events. Analysis
capabilities include categorization of symptoms into syndromes using text search string functions,
aberration detection using CUSUM methods (Cl-Mild, C2-Medium, and C3-Ultra), and graphic
representation of the analysis using graphs and maps, http://www.bt.cdc.gov/surveillance/ears/
Electronic Surveillance System for the Early Notification of Community-Based Epidemics
(ESSENCE). Developed by the Department of Defense and Johns Hopkins University Applied Physics
Laboratory, ESSENCE aims to collect and analyze a variety of data sources for the early recognition of
abnormal community disease patterns that could result from natural causes or terrorist activities.
ESSENCE uses data from military and civilian databases of patient visits, OTC sales, chief complaint
data from Emergency Rooms, 911 calls, Poison Control Center Calls, laboratory records, as well as
weather and community events to perform a fused analysis using spatial and temporal algorithms.
http://www.geis.fhp.osd.mil/GEIS/SurveillanceActivities/ESSENCE/ESSENCE.asp
May 2007 90
-------
Planning for WS-CWS Deployment
Real-time Outbreak and Disease Surveillance (RODS). RODS was originally developed by the
University of Pittsburgh in collaboration with Carnegie Mellon to provide a computer-based public health
surveillance system for the early detection of disease outbreaks. RODS looks at emergency room and
OTC data into one user interface; it collects hospital data in near real-time and analyzes it using Recursive
Least-Square and What's Strange About Recent Events algorithms. It has been implemented at health
departments in numerous states, http://rods.health.pitt.edu/
National Retail Data Monitor (NRDM). The NRDM was also developed by the University of
Pittsburgh in collaboration with Carnegie Mellon to gather and monitor information on OTC drug sales
for possible outbreak detection. This data is collected for use in the RODS analysis tool. The data is
gathered from over 20,000 retail stores throughout the country, and is available to public health
departments free of charge. http://rods.health.pitt.edu/NRDM.htm
BioSense. BioSense is the national program designed to improve the nation's capabilities for real-time
bio-surveillance and situational awareness headed by the CDC. It operates using an Internet-accessible
system that allows users to visualize information about public health trends from early detection data
sources, using advanced algorithms to analyze the data and provide a nationwide, real-time picture.
BioSense is also discussed in Section 1.0 of this document, www.cdc.gov/biosense/
FirstWatch. FirstWatch is a software program designed to integrate and analyze data from various
health and public safety sources (i.e., 911 call, police dispatches, etc.) FirstWatch displays many data
streams on one user interface and performs analysis using predetermined thresholds. Automation for
analysis is already built into the program. Its use has been growing, and is now used in some capacity by
over 50 cities in the U.S. and Canada, www.firstwatch.net
SaTScan. _SatScan is a statistical program developed for the purpose of cluster analysis using both spatial
and temporal methods. It includes models for case-control (Bernoulli model), Poisson based population,
space-time permutation, ordinal, exponential, and normal distribution analysis. Information about the
significant clusters can be output to files for plotting using a separate GIS system. SatScan has been used
extensively in recent years for syndromic surveillance purposes, www.satscan.org
HIPAA. The Health Insurance Portability and Accountability Act was passed in 1996 to protect the
privacy of patient medical data by limiting who can have access to individually identifiable medical data.
http: //www .hhs. gov/ocr/hipaa/
8.4 Staffing and Cost Considerations
In planning the public health surveillance component of a contamination warning system, it is essential to
incorporate staff from a wide range of partners, some of which may have no previous experience with
public health data collection and investigation. Cost factors will vary substantially, depending on how
many partners participate, and their capabilities.
8.4.1 Staffing
During planning, input from numerous entities should be gathered, including local health department
epidemiologists, fire departments/EMS crew, 911 dispatch operators, Poison Control Centers, as well as
the utility. Each of these can provide unique input on advantages and disadvantages of proposed public
health surveillance systems. These agencies should be engaged early in the planning process, such that
their effort on a day to day basis (i.e., during operation and maintenance) is minimized. At later stages of
operation and maintenance, there should at a minimum be adequate staff to interpret any alert generated
by the public health event detection component and successfully communicate details of the alert to
appropriate utility staff for investigation. Where appropriate, additional staff dedicated to manual
surveillance activities could be considered in place of automated data streams. Table 8-2 provides a
summary of staffing considerations for the public health surveillance component.
May 2007 91
-------
Planning for WS-CWS Deployment
Table 8-2. Public Health Surveillance Staffing Considerations
Division or Department
Implementation Phase
PD
D
I
PT
O&M
E&R
Comments
Utility
Project Manager
Water Quality
X
X
X
X
X
X
Provide investigation support to public health alerts
Provide investigation support to public health alerts
Public Health Partners
Epidemiologist
Fire /EMS
911
Poison Control Center
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
May include one or more epidemiologist(s); will be
the main person performing surveillance of data
streams
PD and D activities include identification of possible
data streams; O&M activities will mainly be
maintenance
PD &D activities include identification of possible
data streams; O&M activities will mainly be
maintenance
Roles may be based on service agreement with the
poison control center; may include data sharing and
toxicological expertise
Other Support Partners
IT Support
X
X
X
X
X
X
May be managed at the city or regional level
PD = Pre-design; D = Design; I = Implementation; PT = Preliminary Testing; O&M = Operations and Maintenance;
E&R = Evaluation and Refinement
8.4.2 Cost Considerations
This section presents a summary of the design and implementation considerations discussed above that
may influence costs. This list also may include other factors that were encountered during
implementation of the initial Water Security initiative pilot and could be overlooked during cost
estimation in the absence of this experience. Although this list of cost considerations may not be
exhaustive, these factors, at a minimum, should be considered when planning.
Costs associated with the incorporation of public health surveillance in a contamination warning system
will be based heavily on the current status of public health data systems and existing relationships
between local health departments and the utility. Estimating costs will only be possible after the basic
decisions presented above are considered. Substantial costs could be incurred with the purchase of new
software, development of code to automate systems integration, and initial training. However, investment
in these resources now could prevent even higher labor expenses during operation and maintenance.
In making decisions on how to collect data, automating data streams may be more expensive during
installation due to equipment and software procurement; however, this initial cost may be offset in the
long run by reduced effort needed by staff to continue data collection. Key cost considerations should
include the following:
Determining contacts and developing a Concept of Operations. Effort will be needed to
generate these contacts and documents
Computer hardware or software for data collection and analysis. It may be necessary to
procure computer hardware or software and/or develop custom software for the purposes of data
gathering, analysis, and display. Where possible, available and experienced resources should be
leveraged to reduce these costs.
Determining and testing alert levels. It will take some effort to determine and test appropriate
alert levels to maximize effective investigation while minimizing false positives.
Training on utilizing new analysis tools. These costs will depend on how many new tools are
implemented, and the overall number of people who need training
Table-top and communication exercises. Resources will be necessary to plan, support, and
participate in table-top and other response exercises.
May 2007
92
-------
Planning for WS-CWS Deployment
In general, the more agencies that are involved with public health surveillance, the more expensive it will
be in terms of training costs, coordination, and level of effort. However, involving more agencies also
means increasing potential resources in terms of software, equipment and expertise. Leveraging existing
systems could offset the cost of increased coordination by reducing the need for developing new systems.
May 2007 93
-------
Planning for WS-CWS Deployment
Section 9.0: Consequence Management
As discussed and illustrated in Section 1, consequence management plays a critical role in a
contamination warning system. When triggers from one or more of the monitoring and surveillance
components discussed in Section 4 - Section 8 have been validated, consequence management governs
the response, remediation, and recovery actions. A consequence management plan that successfully
guides these actions is a cornerstone of an effective contamination warning system, and it is essential to
have the plan in place and tested prior to operation of any contamination warning system components.
While development of a consequence management plan can occur in parallel with design and
implementation of monitoring and surveillance components of the contamination warning system, it is
important to reconcile the consequence management plan with routine operations as the system evolves
through the application of system engineering principles as discussed in Section 2.1.
In planning for contamination warning system deployment, it is important to recognize that while the
integrated, routine, and active approach for monitoring and surveillance of public health and water quality
in the distribution system may be a new concept, drinking water utilities should have an existing
emergency response plan that can serve as a starting point for consequence management. In response to
the terrorist attacks of 2001, Congress passed the Public Health Security and Bioterrorism Preparedness
and Response Act of 2002 (the Bioterrorism Act) which required drinking water utilities to prepare or
revise, where necessary, an emergency response plan that incorporates the results of vulnerability
assessments.
Consequence management plans developed in support of contamination warning systems should build on
the utility's existing emergency response plan, focusing specifically on the contamination threat to the
distribution system and should integrate response and decision-making with the routine operation of the
system as defined in the concept of operations. EPA previously provided guidance on response to
drinking water contamination in a suite of six modules that composed the Response Protocol Toolbox and
companion Response Guidelines (USEPA, 2004a-h). Many of the concepts presented in the Response
Protocol Toolbox are applicable to development of a consequence management plan for contamination
warning systems.
This section provides high-level considerations to assist utilities and local partners in planning for
development of a consequence management plan to support contamination warning system
implementation.
9.1 Pre-design
Designing a consequence management plan is a critical task as part of contamination warning system
deployment. Pre-design activities should be carefully conducted, such that subsequent steps can be
effectively executed, with the result being a complete utility-specific consequence management plan that
presents a comprehensive framework for response to validated contamination warning system triggers.
Roles and responsibilities should be defined and assessments conducted, both at the utility and with local
partners. These preliminary assessments can help identify existing plans, both within the utility and
partner organizations, and training opportunities that can be integrated as part of the effort to develop the
consequence management plan for the contamination warning system.
Pre-design activities for development of a consequence management plan include the following:
Identification of objectives for the consequence management plan
Self-assessment
Identification of partners
Development of a preliminary work plan and path forward
May 2007 94
-------
Planning for WS-CWS Deployment
Consequence Management Plan Objectives
Prior to development of a consequence management plan to support contamination warning system
implementation, drinking water utilities should define the objectives of the plan. Most utilities have
detailed emergency response plans and/or action plans that may address contamination of the distribution
system. For the purposes of contamination warning system deployment, objectives of the consequence
management plan may include the following:
Clearly defined roles and responsibilities for utility and response partners in all stages of
consequence management
Comprehensive decision-making framework to support timely response to a trigger from one or
more contamination warning system components
Guidance on use of specific response procedures
Seamless transition from routine operations to consequence management activities
Integration with local and regional response plans
Determining objectives of the consequence management plan early in the process should enable the utility
to effectively use information from their emergency response plans to create a comprehensive
consequence management plan applicable to a contamination warning system.
Utility Self-Assessment
A self assessment of the utility's existing emergency response plans and overall preparedness is the first
step in developing a consequence management plan to support contamination warning system
implementation. The purpose of the self assessment is to identify what procedures are already in place
regarding planning, preparedness, and response and assess these procedures relative to the objectives of
the contamination warning system consequence management plan. There are two primary aspects of this
assessment: existing plans and response resources and capabilities. The utility should first review their
current response procedures and related documents for different events to determine what elements of a
consequence management plan they may already have. Examples of the types of plans that should be
considered include the following:
Plans for responding to a water contamination or water quality event, cross connections, chemical
spills near source water, intentional contamination of the water system
Plans for responding to increased or overwhelming consumer complaint calls, or calls reporting
illness from the water
Plans for responding to facility alarms, reports of suspicious persons near utility facilities, or
threats made to the system, both directly to the utility and though third parties (police, media,
etc.)
Operational plans that address issues such as depressurization or power outage
Severe weather response plans
Civil disorder response plans
Mutual aid agreements with other utilities
Issuing of water-use restrictions
Risk communication and public notification plans
As plans are reviewed, the situation addressed, utility divisions are involved, and outside agencies
involved should be captured to identify gaps that should be addressed through consequence management
planning activities.
In addition to an assessment of existing operational plans, the utility should conduct an assessment of
response resources and capabilities. This involves identifying assets (e.g., staff, equipment) as well as
training needs that are required to implement the existing plans and operations. Throughout the
development of the consequence management plan, the utility should maintain a list of items or resources
May 2007 95
-------
Planning for WS-CWS Deployment
that should be acquired, enhanced, or improved. During design and implementation, the list can be
revisited and shortfalls in training, equipment, and other resources can be resolved.
An aspect of the utility self-assessment related to response resources and capabilities includes assessing
Incident Command System (ICS) training and National Incident Management System (NIMS)
compliance. ICS is a flexible command and control system designed to manage any magnitude of
emergency. ICS is a key component of NIMS, and NIMS is a key component of the National Response
Plan (NRP) which directs, among other things, how command over an incident is escalated from the local
to state to federal level, and back down again. More information on ICS and NIMS can be found at the
Federal Emergency Management Agency's (FEMA) website
(http://www.fema.gov/emergency/nims/index.shtm). There are many reasons to have staff trained in ICS
and NIMS. ICS has been used effectively since the 1970s and is a proven system. The response partners
engaged will most likely be well versed in ICS and NIMS, and should expect your staff to be as well.
Additionally, as directed by Homeland Security Presidential Directive 5, full NIMS compliance is a
requirement for receiving federal preparedness funds. The utility should also consult their state
emergency management and/or homeland security agency, as many states have NIMS requirements that
are more stringent than the federal requirements.
Identification of Response Partners
Identification of partners to support contamination warning system design and implementation is
discussed in Section 2.0. For the purposes of development of the consequence management plan, specific
contacts should be identified within each of those organizations to discuss roles and responsibilities
specifically related to response, remediation, and recovery actions. Table 9-1 provides an overview of
the roles and responsibilities response partners may play in implementing the consequence management
plan, and thus, when they should be engaged in the planning process.
Table 9-1. Overview of Response Partner Roles and Responsibilities for Consequence
Management
Response Partner
Drinking water utility
Local health department
Local law enforcement
Local civil government
Local emergency planning committees and
emergency management agencies
Local fire, EMS, and Hazmat
Environmental and public health laboratories
Local wastewater utility
Neighboring utilities (water and/or wastewater)
Media
State government
State emergency responders
State drinking water and wastewater primary
agencies
State emergency management and homeland
security agencies
State law enforcement
EPA Regional offices and/or laboratories
Federal Bureau of Investigation
Centers for Disease Control and Prevention
Operational
Response
Y
Y
Y
Y
Public Health
Response
Y
Y
Y
Y
Y
Site
Characterization
Y
Y
Y
Y
Y
Y
Criminal
Investigation
Y
Y
Y
Y
Y
Y
Y
Y
Expanded
Sampling
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Laboratory
Analysis
Y
Y
Y
Y
Y
Y
Y
Risk
Communication
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Remediation and
Recovery
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
May 2007
96
-------
Planning for WS-CWS Deployment
Response Partner
EPA Criminal Investigation Division
EPA National Response Center
ns n)
C (ft
0 C
+= O
ns Q.
<-
ns to
ns O
P
Q)
o:
Y
Y
As response partners are identified, it may also be necessary to work with them to assess their existing
plans and capabilities to ensure there should be a seamless transition as an event escalates through the
stages of response, remediation, and recovery. Different response partners may have different areas of
expertise, and likewise might have different levels of response preparedness. These discrepancies should
be identified in the pre-design phase, and so that resolving them can be accomplished through design and
implementation activities.
Work Plan to Guide Consequence Management Plan Development
Following the utility self-assessment and identification of response partners, the next step in the pre-
design process is to develop a work plan to guide development of the consequence management plan.
The work plan should identify a framework or process for plan development, including engagement of
local partners, communications and roll out, and closing of any gaps related to training or capability. In
addition, major goals and milestones may be established in the work plan, such that the objectives defined
at the start of pre-design become practical, concrete targets. The work plan should aim at directing steps
to meet these targets and staying on schedule. Completion of a well-defined work plan should aid in the
success of developing a consequence management plan as the utility moves forward through design and
implementation activities.
9.2 Design and Implementation Approach
Design and implementation activities associated with consequence management plan development
include the following:
Consequence management plan development. Develop a framework and approach for credibility
determination, confirmation, and remediation and recovery. The development process may
include the following:
o Defining an ICS structure for the utility
o Engagement of local partners and alignment with existing plans at the local, regional, and
state levels
o Reconciliation with the concept of operations as defined for the monitoring and
surveillance components
o Integration of site characterization plan
Communication strategy
o Develop a strategy and framework for communications within the utility, between the
utility and external partners, and customers.
o Develop a risk communication strategy and message maps for communicating
information during a suspected or confirmed contamination incident.
Implementation:
Procure equipment needed to augment utility or local partner capabilities. Equipment may
include sampling equipment, personal protective equipment, field screening equipment,
communications equipment, etc. See Section 5.3 for a discussion of site characterization
equipment, coordination, and planning.
May 2007
97
-------
Planning for WS-CWS Deployment
Training
o Develop training materials that address all roles and responsibilities
o Conduct training on the plan. The scope of this training should include all levels of staff
within the utility that have a role in consequence management as well as local partners.
o Ensure appropriate staff are trained and certified in ICS and NIMS.
Revise the consequence management plan as necessary based on feedback from training and
reconciliation with routine operations and initial trigger validation for each of the components.
Preliminary Testing:
Conduct additional training on the consequence management plan as necessary.
Design and implement drills and exercises to test the consequence management plan that involve
drinking water utility staff as well as local response partners.
Refine and finalize the consequence management plan, including reconciliation with the concept
of operations to ensure a smooth transition from routine operations, initial trigger validation, and
consequence management across all components of the system.
Operations and Maintenance:
Deploy plan as necessary in response to validated triggers from one or more of the contamination
warning system components.
Conduct ongoing training to ensure that new staff are familiar with the consequence management
plan.
Update the plan as necessary based on enhancements or modifications to monitoring and
surveillance aspects of the system or local response partner capabilities.
Evaluation and Refinement:
Conduct routine drills and exercises to evaluate the operation and performance of various aspects
of the consequence management plan
Refine the plan as appropriate based on lessons learned through drills and exercises
9.3 Available Tools and Resources
The following tools and resources are available to support development of a consequence management
plan for a contamination warning system:
Response Protocol Toolbox Developed by the EPA, this series of six documents covers topics
such as communications and notifications, threat evaluation, site characterization, sample
analysis, and response actions to help the water sector prepare for and respond to contamination
threats and incidents.
o Overview (EPA-817-D-03-007)
http://www.epa.gov/safewater/watersecurity/pubs/guide response overview.pdf
o Water Utility Planning Guide - Module 1 (EPA-817-D-03-001)
http://www.epa.gov/safewater/watersecuritv/pubs/guide response modulel.pdf
o Contamination Threat Management Guide - Module 2 (EPA-817-D-03-002)
http://www.epa.gov/safewater/watersecurity/pubs/guide response module2.pdf
o Site Characterization and Sampling Guide - Module 3 (EPA-817-D-03-003)
http://www.epa.gov/safewater/watersecurity/pubs/guide_response_module3.pdf
o Analytical Guide - Module 4 (EPA-817-D-03-004)
http://www.epa.gov/safewater/watersecuritv/pubs/guide response module4.pdf
o Public Health Response Guide - Module 5 (EPA-817-D-03-005)
http://www.epa.gov/safewater/watersecurity/pubs/guide response module5.pdf
o Remediation and Recovery Guide - Module 6 (EPA-817-D-03-006)
http://www.epa.gov/safewater/watersecuritv/pubs/guide response module6.pdf
May 2007 98
-------
Planning for WS-CWS Deployment
Response Protocol Toolbox: Response Guidelines. An action oriented document to assist
drinking water utilities, laboratories, emergency responders, state drinking water programs,
technical assistance providers, and public health and law enforcement officials during the
management of an ongoing contamination threat or incident. The Response Guidelines are not
intended to replace to Response Protocol Toolbox and they do not contain the detailed
information contained within the six complete modules. The Response Guidelines are to be
viewed as the application of the same principles contained in the Response Protocol Toolbox
during an actual incident. The Response Guidelines have been developed to provide an easy to
use document for field and crisis conditions. Finally, users are encouraged to adapt the Response
Guidelines as necessary to meet their own needs and objectives.
http://www.epa.gov/safewater/watersecurity/pubs/rptb response guidelines.pdf
Incident Command System (ICS) Training and National Incident Management System
(NIMS) Compliance ICS is a command and control system designed to grow and contract to
manage any magnitude of emergency. ICS is a key component of NIMS, and NIMS is a key
component of the National Response Plan (NRP) which directs how command over an incident is
escalated from the local to state to federal level. Additionally, as directed by Homeland Security
Presidential Directive (HSPD) 5, full NIMS compliance is a requirement for receiving federal
preparedness funds More information on ICS and NIMS can be found at FEMA's website:
http://www.fema.gov/emergencv/nims/index.shtm
Federal Emergency Management Agency (FEMA) FEMA is the federal agency responsible for
responding to and aiding the recovery from natural or man-made disasters. The have developed
the ICS and NIMS training program, as well as the National Response Plan (NRP) to establish an
all-hazards response to emergencies for communities throughout the United States.
www.fema.gov
Department of Homeland Security (DHS) DHS offers many resources relating to water
security, included training and research under the areas of awareness, prevention, protection,
response and recovery. The website is also contains information relating to bioterrorism laws,
regulations and policies, as well as background information about the Homeland Security
Presidential Directives, www.dhs.gov
Centers for Disease Control and Prevention (CDC) The CDC offers response plans for agents,
diseases, and other events through its Emergency Preparedness and Response branch. Specific
training opportunities offered include those for Bioterrorism, Chemical, and Radiation
Emergencies, www.bt.cdc.gov
Federal Bureau of Investigation (FBI) The FBI is a major partner in investigating terrorist
activities, and may be one of the responders to a contamination event. The FBI can also support
local and state enforcement agencies, www.fbi.gov
Water Information Sharing and Analysis Center (WaterlSAC) Online database containing
information, expert analysis, and government alerts. Provides tools for water security and links to
other agencies, such as homeland security, law enforcement, and public health.
www.waterisac.org
The Association of State Drinking Water Administrators (ASDWA) A professional
organization that supports states in their efforts to assure quality drinking water, and encourages
coordination between state drinking water agencies. Provides tools and technical materials for
area wide optimization programs, data management and security, www.asdwa.org
May 2007 99
-------
Planning for WS-CWS Deployment
American Water Works Association (AWWA) AWWA is a professional organization
dedicated to improving water quality and supply. They provide numerous resources and training
tools for use by utilities, including Water 101: Security Planning and Partnership for Safe Water
online courses, www.awwa.org
Utilities Helping Utilities: An Action Plan For Mutual Aid and Assistance Networks For
Water and Wastewater Utilities A document developed by AWWA to help utilities develop
Water and Wastewater Agency Response Networks (WARNs), a mutual aid and assistance
program to be used after a utility has sustain damage from man-made or natural disasters.
http://www.awwa.org/Advocacy/govtaff/
Effective Risk and Crisis Communication during Water Security Emergencies. This report
summarizes results from three water security risk communication message mapping workshops
conducted by U.S. EPA's National Homeland Security Research Center during 2005/2006. It
provides information about effective message development and delivery that could be useful to
water sector organizations as they develop their respective risk communication plans. Message
mapping is a process by which users can predict 95 percent of questions likely to be asked by the
media and others following an incident, prepare clear and concise answers to the questions along
with supporting information ahead of time, and practice effective message delivery before a crisis
occurs. nttp://www.epa.gov/nhsrc/pubs/reportCrisisCom040207.pdf
9.4 Staffing and Cost Considerations
Staffing and cost considerations for development and implementation of a consequence management plan
as part of a contamination warning system may vary significantly from utility to utility based on existing
capabilities and the approach used to develop the consequence management plan. Section 9.4.1 and 9.4.2
provide general considerations for staffing and costs, respectively.
9.4.1 Staffing
Response partners involved in development and implementation of the consequence management plan are
discussed in Section 9.1. The response partners needed to support consequence management activities
may vary from utility to utility; however, once identified, they should be engaged from the pre-design
through the evaluation and refinement process. For the drinking water utility implementing a
contamination warning system, staffing considerations may vary based on the approach taken to develop
the plan. Throughout development and implementation of the plan, it may be necessary to engage
representatives from each utility department or division. Ultimately, the departments or divisions
engaged may also vary based on response actions needed. While senior managers may have a more
significant role in development of the plan, all levels of staff may need to be engaged in some aspect of
training, participation in drills and exercises, and plan implementation.
9.4.2 Cost Considerations
Costs associated with development and implementation of a consequence management plan for
contamination warning systems may vary based on existing plans and capabilities within the utility and
the jurisdictions included in the utility's service area. Most of the costs associated with development of
the consequence management plan are related to staff time. In planning for contamination warning
system implementation, utilities should also consider costs associated with training, equipment (e.g.,
communications), and development and implementation of drills and exercises to evaluate the plan. Like
the development of the consequence management plan, most of the costs associated with training and
development of drills may be labor costs, and may vary based on how many people are involved.
May 2007 100
-------
Planning for WS-CWS Deployment
When identifying cost considerations for developing the consequence management plan, it should be
considered that some of the equipment and training necessary to address gaps here my already be planned
for implementation under one of the other contamination warning system components. As such, these
costs should be discussed amongst the project management team. Identifying instances such as this may
not only succeed in reducing overall costs, but strengthens the concept that a contamination warning
system (and in particular a consequence management plan) requires the components to function together.
May 2007 101
-------
Planning for WS-CWS Deployment
Section 10.0: References
AWWA. 2004. Interim Voluntary Security Guidance for Water Utilities.
Hall, J. Zaffiro, A. D., Marx, R. B., Kefauver, P C., Krishnan, E. R, Haught, R C., Herrmann, J. G.
2007a. Online Water Quality Parameters as Indicators of Distribution System Contamination. Journal of
the American Waterworks Association, 99 (l):66-77.
Hall, J., Szabo, J.G. 2007b. Evaluation of Water Quality Monitoring Technologies to Respond to Changes
in Drinking Water Quality. Presented at GWRC On Line Monitoring Workshop; KIWA Facility,
Nieuwegein, NETHERLANDS, March 21-22.
Hall, J.S., Haught, R., Rahman, M., Richardson-Coy, R. and Piao, H. 2007c. The bench-scale minimum
dose threshold experiment. EPA/600/R-07/002. Water Information Sharing and Analysis Center
(WaterlSAC) (www.waterisac.org).
Szabo, J.G., Hall, J.S. and Meiners, G.C. 2006. Water quality sensor responses to injected contaminants
in a chloraminatedpipe loop. American Water Works Association (AWWA) Water Security Congress,
Technical Session TUE6: Technology Forum B, Washington, DC, September 10-12, 2006
Szabo, J. et al. 2007. Water Quality Sensor Response to Contamination in a Single Pass Water
Distribution System Simulator. EPA-600-R-07-001. Available only through WaterlSAC
USEPA. 2003. Instructions to Assist Community Water Systems in Complying with the Public Health
Security and Bioterrorism Preparedness and Response Act of 2002. EPA-810-B-02-001.
USEPA. 2004a. Response Protocol Toolbox Overview. (EPA-817-D-03-007).
USEPA. 2004b. Response Protocol Toolbox: Water Utility Planning Guide -Module I. (EPA-817-D-03-
001).
USEPA. 2004c. Response Protocol Toolbox: Contamination Threat Management Guide -Module 2.
(EPA-817-D-03-002).
USEPA. 2004d. Response Protocol Toolbox: Site Characterization and Sampling Guide -Module 3.
(EPA-817-D-03-003).
USEPA. 2004e. Response Protocol Toolbox: Analytical Guide - Module 4. (EPA-817-D-03-004).
USEPA. 2004f Response Protocol Toolbox: Public Health Response Guide -Module 5. (EPA-817-D-
03-005).
USEPA. 2004g. Response Protocol Toolbox: Remediation and Recovery Guide -Module 6. (EPA-817-
D-03-006).
USEPA. 2004h. Response Protocol Toolbox: Planning for and Responding to Drinking Water
Contamination Threats and Incidents Response Guidelines.
USEPA. 2005a. WaterSentinel System Architecture. EPA-817-D-05-003.
USEPA. 2005b. WaterSentinel Contaminant Selection. Sensitive Information, Limited Distribution, For
Official Use Only.
May 2007 102
-------
Planning for WS-CWS Deployment
USEPA. 2005c. Overview of Event Detection Systems. EPA-817-D-05-001.
May 2007 103
-------
Planning for WS-CWS Deployment
Appendix A: Glossary
Anomaly. Deviations from an established baseline or base state. Specifically, a water quality anomaly is
a deviation from an established water quality base state at a specific location.
Base State. Normal conditions that result from typical system operation. The base state includes
predictable fluctuations in measured parameters that result from known changes to the system. For
example a water quality base state includes the effects of draining and filling tanks, pump operation, and
pipe flushing, all of which may alter water quality in a somewhat predictable fashion.
Baseline Data. Baseline data is all available chemical, radiochemical, pathogen and toxin analytical data
relative to a baseline sample that may be used to determine "possible" or "credible" contamination.
Baseline data may be contaminant and location-specific control charts and tabulated contaminant data.
Concept of Operations (Con Ops). A process for routine operation of a drinking water contamination
warning system, which establishes specific roles and responsibilities, process and information flows, and
procedural activities. The Con Ops includes the process for validation of a contamination warning system
trigger and determining whether or not contamination is "possible."
Consequence Management Plan. Provides a decision-making framework that governs when, how,
what, and who will be involved in making decisions in response to a "possible" contamination incident in
order to minimize the response timeline and implement operational or public health response actions
appropriately.
"Credible." In the context of the credibility determination process, water contamination is characterized
as 'credible' if information collected during the investigation of "possible" contamination corroborates
information from the validated contamination warning system trigger.
Credibility Determination. Contamination warning system triggers will be investigated to determine
whether or not they are indicative of "possible" contamination. Credibility determination is the
subsequent investigation to determine whether or not additional information, including data from other
monitoring and surveillance components, corroborates the information from the validated trigger. If the
additional information corroborates the trigger, contamination is considered 'credible.'
Event Detection System (EDS). A system designed specifically to detect anomalies from the various
monitoring and surveillance components of a contamination warning system. An EDS may take a variety
of forms, ranging from complex set of computer algorithms to a simple set of heuristics that are manually
implemented. In essence, an EDS is a data mining tool that supports the efficient analysis of large
amounts of monitoring and surveillance data to pick out possible anomalies while at the same time
minimizing false alarms.
EDS Alarm. A notification from the EDS tool that an anomaly has been detected. Contamination
warning system alarms may be visible and/or audible, and may initiate automatic notifications such as
pager or e-mail alerts. Most EDS alarms require some degree of validation before they are considered
indicative of "possible" contamination.
Field Screening. Performing a series of tests to evaluate any potential chemical, biological or
radiochemical dangers present at the site.
Job Function. A description of the duties and responsibilities of a specific job within an organization.
May 2007 104
-------
Planning for WS-CWS Deployment
Monitoring and Surveillance. Element of a contamination warning system that provide a standardized
set of information streams used in the detection of potential contamination incidents.
"Possible." In the context of the contamination warning system concept of operations, water
contamination is characterized as "possible" if the cause of a trigger cannot be identified and/or
determined to be benign.
Risk Reduction Units (RRUs). The difference between the calculated risk before security improvements
versus after security improvements. A measure of enhanced security. RRU = (Risk Before
improvements) - (Risk After Improvements)
Security Breach. An unauthorized intrusion into a secured facility that may be discovered through direct
observation, an alarm trigger, or signs of intrusion (cut locks, open doors, cut fences).
Site characterization. The process of collecting information from an investigation site to support the
evaluation of a drinking water contamination threat
Standard Operating Procedure (SOP). A step-by-step list of actions that guide the user in the
implementation of a specific task.
Syndromic surveillance. Collecting and analyzing nontraditional data to detect a change or trend in the
health of a population using categories of disease rather than formal diagnosis.
Target Contaminant. A contaminant that has been identified by the EPA for monitoring under the Water
Security Sentinel Initiative. Target contaminants are monitored using drinking water confirmatory
methods. Reported results are qualitative and quantitative
Threat Warning. An unusual occurrence, observation or discovery that indicates a potential
contamination incident and initiates actions to address this concern.
Trigger. Information from a monitoring and surveillance component that an anomaly has been detected.
Trigger Validation. The process of investigating potential causes of a contamination warning system
trigger to either rule out contamination or determine that contamination is "possible."
May 2007 105
-------
Planning for WS-CWS Deployment
Appendix B: Information Security Considerations
Because the Water Security initiative is first and foremost a homeland security program with a counter-
terrorism focus, information security is extremely important. Certain utility materials and materials
developed in support of the program can potentially be exploited by adversaries to defeat the system.
Therefore, it is necessary for partner utilities to develop formal procedures and protocols for the
identification, handling, tracking, and overall security of any sensitive documents involved with the
development, implementation, and operation of their contamination warning system. Training on and
abiding by these procedures and protocols should be considered for any and all staff who may access
sensitive materials as part of their job; this includes utility, partner, contractor and subcontractor
personnel.
Some key elements and steps in developing and implementing an information security strategy include:
Assessment of Material Sensitivity, Access, and Law
Sensitive Materials Tracking and Storage
Sensitive Materials Handling Protocols and Procedures
Staff Certification and Background Checks
Coordination and Cooperation with Partners, Contractors and Subcontractors
Additionally, it is recommended that utility program management designate a team responsible for the
development, operation, and maintenance of an information security strategy, and reconciliation with
partner agencies sensitive information programs. This team should be comprised of members with
backgrounds in general and/or information security, information technology, emergency response, and
law and law enforcement.
B.1 Assessment of Material Sensitivity, Access, and Laws
The first step to implementing an information security program is to identify potentially sensitive
materials. The information security team should consider existing utility and partner materials, as well as
materials that will be developed. Examples include a utility's sensitive facilities list, security procedures
and emergency response procedures, hydraulic models, facility maps and blueprints, locations and types
of contamination warning system enhancements and certain contact lists and notification procedures. As
materials are reviewed, the team can begin developing a tiering system for different levels of sensitivity.
It is recommended that the specific terminology used in this tiering system draw on accepted
nomenclature (e.g., terms such as "For Official Use Only", "Sensitive", "Confidential", and
"Proprietary"). However, the terms chosen and the level of sensitivity they reflect should be clearly
defined in the information strategy document, and whether they truly afford any legal protection.
Also as part of the assessment, local, state, and federal laws and regulations should be reviewed to
determine what legal protections may be afforded to sensitive materials. Because local, state, and federal
governments are involved in contamination warning system implementation, the federal Freedom of
Information Act (FOIA), as well as similar laws and regulations passed in other jurisdictions may be
applied to certain types of information, especially if the utility itself is part of a government entity. The
FOIA legislation, implemented in 1967, governs the disclosure of documents and information controlled
by the U.S. government (this includes documents submitted to the government by an outside government
agency or private entity). In general, most information held by the government should be made available
to the public or other entities if requested, provided said information is not covered by one of nine
exemptions (such as certain types of confidential business information, or CBI, and National Security
Information, which has been designated exempt to protect the security of the Nation). Since the
legislation went into effect, and particularly in the aftermath of 9/11, certain information, particularly
relating to critical infrastructure, like water utilities, has been determined to be covered by one or more of
the exemptions. Additionally, many other jurisdictions have exempted sensitive security information
May 2007 106
-------
Planning for WS-CWS Deployment
related to critical infrastructure from there own regulations that are similar to FOIA. Local, state, and
federal laws related to disclosure of government held information should be reviewed to determine what
disclosure requirements and exemptions might apply to the information being assessed. It is
recommended that both the utility's legal counsel and that of the respective governing entity be engaged
in this process.
It is important to remember that sensitivity assessment of materials will remain an ongoing process
throughout the life of the contamination warning system; some materials may have their sensitivity
determination reconsidered while new materials will have to be placed in the ranking system. It is also
important to remember that new documents produced from sensitive materials should also be assessed for
their sensitivity, and possibly included in the sensitive materials program.
B.2 Sensitive Materials Tracking and Storage
Maintaining an effective tracking and storage system for sensitive materials is an important aspect of
drinking water security. Additionally, it will be necessary to exchange information with partner agencies
as part of the development and implementation of a contamination warning system. Their sensitive data
should be protected as thoroughly as the utility's own documents, both so that the partners maintain
confidence in their partnership with the utility, and so that the utility does not become the conduit for
exploitable information falling into the wrong hands.
As materials are being assessed, a tracking and storage system should be developed. Materials can be
assigned tracking numbers which identify the date of receipt or creation, creating or owning agency, form
of the material (electronic, paper, etc.) kind of material (map, document, blueprint, etc.), whether the
material is a duplicate or is a number in a series of duplicates, and other identifying pieces of information.
As part of the tracking system, a system for logging materials in and out should be implemented. This
could be as simple as a paper or electronic log sheet maintained by a responsible person or group within
the utility which lists the date, tracking number, and signature, initial, or other mark of the person
obtaining the material. More complex systems can involve confirmatory phone calls or e-mails for
materials that are mailed or sent electronically, or by using the receipt and tracking services of the U.S.
Postal Service and other carriers.
Storage refers to where the materials are stored when they are checked-in. The utility should decide
whether a central storage location will be used, or whether different divisions will have storage areas for
the materials they primarily use for their implementation efforts. In deciding on storage procedures and
location, a utility should consider such factors as whether the location is physically secure from break-in,
whether a severe rain storm or other natural event might damage it, fire protection, and whether electronic
materials are protected from power failure.
B.3 Sensitive Materials Handling Protocol and Procedures
Proper handling of materials is important for the same reasons that storage and tracking are important. It
is a wasted effort to expend resources protecting documents while in storage, only to have them stolen
from a car, lost on the bus, left in a printer tray, or emailed to "reply to all". Sensitive materials handling
in this context refers to how those materials should be protected when checked-out of central storage.
Before or while materials are being assessed for sensitivity, handling procedures and protocols should be
developed. Handling protocols are fairly common anywhere sensitive information is used, from
government agencies to corporations intent on keeping proprietary or other sensitive information a secret,
and they can range from common sense handling procedures, like locking them in a drawer when not in
use, to intensive procedures used for classified information. The sensitive material handling procedures
should be specific to different forms of media and transmitting media, like e-mailing electronic
May 2007 107
-------
Planning for WS-CWS Deployment
attachments. Some resources that can be found online include EPA's Office of Science and Technology's
Confidential Business Information plan and the Toxic Substances Control Act Confidential Business
Information Protection Manual. However, aside from minimum protections required by EPA, it will be
up to the utility and its partners to determine how conservative their sensitive materials plan is. Some
common precautions include:
Shredding extra or damaged paper documents
Locking documents up at night or when not in use for extended periods of time, even when in
secure facilities
Employing cover sheets for covering sensitive material when leaving one's workspace for a short
period of time, or when a person not authorized to view the material enters the office. For
electronic files, this would mean enabling the computer screen saver or locking the computer.
Utilizing encryption software for electronic transmittal of materials.
Only using company or agency email addresses (for example, not gmail, AOL, or hotmail
accounts).
Downloading attachments immediately and deleting the transmitting email.
Prohibiting saving of sensitive materials on an unprotected Local Area Network (LAN).
Double wrapping materials when transporting them outside secure facilities. Double wrapping
consists of wrapping documents in an inner envelope or packaging that contains the "please
return to" information, as well as clear markings that the packaging contains sensitive or
proprietary information. An outer envelope or packaging should contain the same "please return
to" information but not the sensitive or proprietary information label, so as not to arouse the
curiosity of whoever finds it.
Requiring the same level of protection for electronic media, like thumb drives, as are required for
paper documents.
Calling the recipient before faxing a document, and requiring a confirmatory phone call back.
Not leaving sensitive materials unattended in cars, even if the car is alarmed or locked.
Utilizing hotel room safes
These types of precautions should also be observed whenever the sensitivity of a document or material is
in question, including for materials produced from sensitive materials, but have yet to be assessed for
their sensitivity.
Responsible handling of sensitive materials extends to protecting the data they contain as well.
Precautions should be taken to ensure that discussions of sensitive materials, or documents produced from
sensitive materials (and hence might contain sensitive materials themselves), are afforded equal
precautions. Care should be taken during conference calls to ensure all participants are cleared to discuss
sensitive information, and that others cannot join the call without the host's knowledge. Similarly, using
speaker-phone when discussing sensitive materials should be avoided. Sensitive materials should not be
discussed in public places. To help this effort, persons cleared for access to sensitive materials should be
provided updated lists regularly of who else is cleared to view sensitive materials.
B.4 Staff Certification and Background Checks
Part of ensuring the security of sensitive materials is ensuring the integrity of those who will be handling
the materials. Background checks in general, and specific training on the handling of sensitive materials,
will help achieve this goal.
If not already done, background checks should be performed on all staff involved in the design,
implementation, and operation of the major components of the contamination warning system: this
includes utility, partner, contractor and subcontractor staff. While it may not be necessary to perform a
background check on a member of a maintenance crew whose day-to-day responsibilities will be largely
unaffected by the program, it will be necessary to perform them for staff who, say, have access to the
locations of monitoring stations. The utility should assess its existing rules regarding which personnel
May 2007 108
-------
Planning for WS-CWS Deployment
should have background checks, how extensive those checks are, and how they should adjust them based
on new roles and responsibilities resulting from implementing the contamination warning system. It is
strongly recommended that any staff who will be handling sensitive materials receive background checks.
In addition to background checks, a certification or training process should be implemented for staff who
will be handling sensitive materials. At a minimum, staff should certify that they have read and
understand relevant materials related to the handling of sensitive materials, and lists should be kept
showing who has received such training and when they are due for recertification; signed nondisclosure
agreements can be utilized as part of this certification process.
B.5 Coordination and Cooperation with Partners, Contractors and Subcontractors
As mentioned, it may be necessary to provide certain sensitive materials to (or receive from) response
partners and contractor personnel. While it is up to the utility to decide what level of precaution is
appropriate for their internal sensitive materials, they should be cognizant that their partners may have
more or less restrictive measures. Prior to the exchange of any sensitive materials, the utility should
provide their sensitive materials handling guidance to the partner agency and request the partner agency's
equivalent guidance. This way, any conflicts between the two can be settled before materials are
exchanged. Additionally, formal non-disclosure agreements should be provided by contractors and
subcontractors to the utility, and the utility should check to see if any similar agreements partners have
with their contractors and subcontractors are consistent with the utility's procedures.
To ease this process, it is recommended that the utility seek guidance in developing their own sensitive
materials program from such partners as local counter-terrorism and intelligence groups, such as the FBI
local field office, or local Joint Terrorism Task Force (JTTF). Other local partners, such as public health
agencies, who have experience maintaining data with special handling requirements, can also be
consulted.
May 2007 109
------- |