Water Security Initiative:
Interim Guidance on Planning for
Contamination Warning System
Deployment

      ..
  Contamination
 Warning System


-------
Office of Water
EPA817-R-07-002
May 2007

-------
                                       Disclaimer

The Water Security Division, of the Office of Ground Water and Drinking Water, has reviewed and
approved this document for publication. Neither the United States Government nor any of its employees,
contractors, or their employees make any warranty, expressed or implied, or assume any legal liability or
responsibility for any third party's use of or the results of such use of any information, apparatus, product,
or process discussed in this report, or represents that its use by such party would not infringe on privately
owned rights. This document is not a substitute for applicable legal requirements, nor is it a regulation
itself.

Mention of trade names or commercial products does not constitute endorsement or recommendation for
use.

Questions concerning this document should be addressed to:

Jessica Pulz
U.S. EPA Water Security Division
26 West Martin Luther King Drive
Cincinnati, OH 45268-1320
(513)569-7918
Pulz.Jessica@epa.gov

or

Steve Allgeier
U.S. EPA Water Security Division
26 West Martin Luther King Drive
Cincinnati, OH 45268-1320
(513)569-7131
Allgeier. Steve @epa. gov

-------
                           Purpose of this Document
EPA intends this guidance manual to assist drinking water utilities with planning for contamination
warning system deployment based on the model developed under EPA's Water Security initiative
(formerly known as WaterSentinel).  In particular, this manual may aid respondents to an upcoming EPA
Request for Applications (RFA).  Under this RFA, the Agency would make financial awards for drinking
water utilities to demonstrate and evaluate contamination warning system pilots.  EPA anticipates issuing
this RFA in June 2007.

Further, EPA plans to issue additional interim guidance on contamination warning system operation and
consequence management planning in late 2007. All these interim guidance manuals will then be revised
as needed based on findings of the demonstration pilots and public comment prior to being issued in final
form. Along with this guidance, the Agency intends to develop outreach program to promote adoption of
effective and sustainable drinking water contamination warning systems. The following is a summary of
these additional documents.
  Request for Applications (Summer 2007) will solicit proposals for financial awards to assist
  drinking water utilities with demonstrating and evaluating contamination warning system
  pilots based on the Water Security initiative model.
  Interim Concept of Operations Guidance (Fall 2007) will describe the process and
  procedures involved in routine operation of a contamination warning system, including
  process and information flows, roles and responsibilities, and the initial investigation and
  validation of alarms.  This document could be used by drinking water utilities to inform and
  refine component-specific designs and support deployment and operation of their
  contamination warning system.
  Interim Consequence Management Plan Guidance (Fall 2007) will assist drinking water
  utilities in the development and implementation of a utility-specific consequence management
  plan for an existing or emerging contamination warning system. This guidance will also
  address integration of the consequence management plan with existing plans, training and
  exercise scenarios, and outreach to other local, state, and federal agencies.
                             Request for Comments

EPA is soliciting suggestions and recommendations to make this interim guidance manual more complete
and user-friendly.  Commenters are encouraged to be as specific as possible and to provide references
where appropriate. Submit suggestions by e-mail to: watersecurity@epa.gov and indicate that the
message relates to the "Interim Guidance on Planning for Contamination Warning System Deployment."

-------
                                Planning for WS-CWS Deployment
                         Abbreviations  and Acronyms
APHL      Association of Public Health
           Laboratories
AWWA     American Water Works Association
CDC       Centers for Disease Control and
           Prevention
CFR       Code of Federal Regulations
CID        Criminal Investigation Division
CPTED     Crime Prevention through
           Environmental Design
CWA       Clean Water Act
DHS       Department of Homeland Security
EARS      Early Aberration Reporting System
EDS       Event Detection System
EMS       Emergency Medical Service
EPA       Environmental Protection Agency
ESSENCE  Electronic Surveillance System for the
           Early Notification of Community Based
           Epidemics
ETV       Environmental Technology Verification
FBI        Federal Bureau  of Investigation
FEMA      Federal Emergency Management
           Agency
GCWW     Greater Cincinnati Water Works
CIS        Geographic Information System
HAZWOPR  Hazardous Waste Operations
HIPAA      Heath Insurance Portability and
           Accountability Act
ICS        Incident Command System
IT         Information Technology
LRN       Laboratory Response Network
MRL       Minimum Reporting Level
MOU       Memorandum of Understanding
NACWA    National Association of Clean Water
           Agencies
NELAC     National Environmental Laboratory
           Accreditation Conference
NELAP     National Environmental Laboratory
           Accreditation Program
NEMI       National Environmental Methods Index
NIMS       National Incident Management System
NLTN      National Laboratory Training Network
NRDM
NRP
NSF
ORP
OSHA

OTC
O&M
PCS
PIR
PLC
PPE
QA
QC
RAM-W™

RFA
ROC
RODS

RPTB
RRU
SAM
SCADA

SDWA
SOP
TEVA

TEVA SPOT

TOC
TTEP
    T-JM
USDA
VOC
VSAT1
WC IT
WS
WS-CWS
National Retail Data Monitor
National Response Plan
National Sanitation Foundation
Oxidation  Reduction Potential
Occupation Safety and Health
Administration
Over-the-counter (drug sales)
Operation and Maintenance
Polychlorinated Biphenyl
Passive Infrared
Programmable Logic Controller
Personal Protective Equipment
Quality Assurance
Quality Control
Risk Assessment Methodology for
Water Utilities
Request for Applications
Receiver Operating Characteristic
Real-time Outbreak and Disease
Surveillance
Response Protocol Tool-box
Risk Reduction Units
Standardized Analytical Methods
Supervisory Control and Data
Acquisition
Safe Drinking Water Act
Standard Operating Procedure
Threat Ensemble Vulnerability
Assessment
TEVA Sensor Placement Optimization
Tool
Total Organic Carbon
Technology Testing and Evaluation
Program
United States Department of Agriculture
Volatile Organic Compound
Vulnerability Self Assessment Tool
Water Contaminant Information Tool
Water Security
WS Contamination Warning System
May 2007

-------
                             Planning for WS-CWS Deployment
                              Acknowledgements

EPA's Office of Ground Water and Drinking Water would like to recognize the following individuals and
organizations for their assistance and contributions in development of this document:

Technical Support Center
   •   Eric Bissonette

Office of General Counsel
   •   Peter Ford

Water Security Division
   •   Steve Allgeier
   •   Jeffrey Pencil
   •   David Harvey
   •   Elizabeth Hedrick
   •   Mike Henrie
   •   Nancy Muzzy
   •   Brian Pickard
   •   Jessica Pulz
   •   Dan Schmelling
   •   David Travers

National Homeland Security Research Center
   •   Kathy Clayton
                                   Contractor Support
       Yildiz Chambers, CSC
       John Chandler, CSC
       Kevin Cornell, CSC
       Mike Denison, CH2M Hill
       Bill Desing, CH2M Hill
       Jean Dupree, CSC
       Todd Elliott, CH2M Hill
       Katie Gavit, CSC
       Darcy Gibbons, CSC
       David Gnugnoli, CSC
Yakir Hassit, CH2M Hill
Gary Jacobson, CH2M Hill
Reese Johnson, CH2M Hill
Colm Kenny, CH2M Hill
Kim Morgan, CSC
Misty Pope, CSC
Curtis Robbins, CH2M Hill
Jerry Scott, CSC
Doron Shalvi, CSC
Scott Weinfeld, CSC
May 2007

-------
                                Planning for WS-CWS Deployment


                                Executive Summary

This manual provides an overview of design and implementation considerations to assist drinking water
utilities in planning for contamination warning system deployment.  As a planning tool, this document
describes a general framework and process for implementation and identifies available tools and
resources to  support implementation of a contamination warning system.

What is a contamination warning system?

A contamination warning system provides drinking water utilities with a systematic and comprehensive
approach for monitoring and surveillance of the distribution system. Through implementation of the
monitoring and surveillance strategies and a comprehensive consequence management plan, utilities can
improve their ability to detect intentional or unintentional distribution system contamination. In addition,
their increased ability to monitor and understand distribution system water quality may help to optimize
operations and improve the overall quality of the product delivered to customers.  Monitoring and
surveillance  components of the contamination warning system include the following:
    •  Online water quality monitoring comprises stations located throughout the distribution system
       that measure parameter such as chlorine, total organic carbon, conductivity, and pH among
       others. Software analyzes the monitoring data to establish a water quality base state. Possible
       contamination is indicated when a significant, unexplained deviation from the base state occurs.
    •  Sampling and analysis is the collection of distribution system samples that are analyzed for
       various contaminant classes as well as specific contaminants.  Sampling is both routine to
       establish a baseline and triggered to respond to an indication of possible contamination from
       another component. Analyses are conducted for chemicals, radionuclides, pathogens, and toxins
       using a laboratory network.
    •  Enhanced security monitoring includes the equipment and procedures that detect and respond
       to security breaches at distribution  system facilities. Security equipment may include cameras,
       motion activated lighting, door contact alarms, ladder and window motion detectors, area motion
       detectors,  and access hatch contact alarms.
    •  Consumer complaint surveillance enhances the collection and automates the analysis of calls by
       consumers for water quality problems indicative of possible contamination. Consumers may
       detect contaminants with characteristics that impart an odor, taste, or visual change to the
       drinking water.
    •  Public health surveillance involves the analysis of health-related data to identify disease events
       that may stem from drinking water contamination.  Public health data may include over-the-
       counter (OTC) drug sales, hospital admission reports, infectious disease surveillance, emergency
       medical service (EMS) reports, 911 calls, and poison control center calls.

In addition to these monitoring and surveillance components, consequence management is a critical
aspect of the overall architecture  for a contamination warning system.  Consequence management refers
to the procedures and protocols for assessing credibility of a contamination incident and implementing
response actions.

Why deploy a contamination warning system?

Monitoring the distribution system is the primary focus of contamination warning systems. Through the
assessment of vulnerabilities to drinking water systems, water security experts have identified the
distribution system as one of the most vulnerable components in a drinking water utility, with respect to
contamination. Furthermore, intentional contamination,  or even the threat of contamination,  can have
significant impacts.
May 2007

-------
                                Planning for WS-CWS Deployment

Drinking water utilities occasionally receive threats or indications of possible contamination.  These
contamination threat warnings can be a direct threat or an unusual observation or discovery that indicates
the potential for contamination and initiates actions to investigate and potentially respond. However,
these threat warnings are not standardized and are difficult to corroborate in the absence of an integrated
monitoring and surveillance system and close coordination with response partners.

Deployment of contamination warning systems for drinking water distribution systems provides a
mechanism by which drinking water utilities can detect and  respond to contamination threats and
incidents. A contamination warning system is a proactive approach to managing threat warnings that uses
advanced monitoring technologies/strategies and enhanced surveillance activities to collect, integrate,
analyze, and communicate information to provide a timely warning of potential water contamination
incidents and initiate response actions to minimize public health and economic impacts.

In addition, contamination warning system implementation provides the opportunity for dual-use
applications beyond security that could help to promote sustainability of the system by optimizing utility
operations. Drinking water distribution  systems may be accidentally contaminated through cross-
connections with non-potable water, permeation of contaminated water through leaking pipes in areas of
the distribution system subject to low pressures, or chemical reactions or microbial growth within the
distribution system pipes.  Such unintentional events that result in degradation to distributed water quality
may occur with some regularity.

Potential dual-use benefits of a contamination warning system could include the following:
    •  Detection of cross-connections and other distribution system water quality problems
    •  Improved relationship with public health organizations, including mutual sharing of information
       and alerts
    •  Enhanced knowledge of distribution system water quality leading to improved operations (e.g.,
       more consistent disinfection residual levels, improved corrosion control, early warning of
       nitrification episodes, reduced disinfection byproduct levels, etc.)
    •  Identification of problem valves (closed, partially closed, inoperable)
    •  Improved coordination with local, state, and federal response organizations
    •  Reduced occurrence  of tampering and vandalism
    •  Improved information technology systems and interoperability
    •  Improved consumer complaint tracking and response
    •  Improved laboratory capability and an established laboratory network
    •  Consequence management plans applicable to any water quality emergency

What approach or framework should be applied for deployment of a contamination
warning system?

A contamination warning system is, by design, a systematic approach to monitoring and surveillance for
the timely detection of drinking water contamination. As such, deployment of a contamination warning
system relies  on the application of system engineering principles to support coordination of technical and
management activities. Through system engineering, disciplines and specialty groups are  integrated in a
team effort forming a structured development process that proceeds from design to implementation to
operation. From the beginning of the project, system engineering principles are critical to successful
planning and implementation. The primary application of system engineering for a contamination
warning system is to ensure that the system - monitoring and surveillance components and consequence
management - functions as an integrated whole.  System engineering principles should be applied to
every aspect of contamination warning system implementation, including staffing. While  routine
operation and maintenance of the contamination warning system should generally fall within the routine
job functions of utility staff, design and  implementation may involve significant time and effort from
dedicated managers within the utility. Depending on utility  organization and operational approach, these
May 2007                                                                                   iv

-------
                                Planning for WS-CWS Deployment

activities may be managed by one individual, or more likely, a core, multi-disciplinary project
management team.

As discussed in Section 3, deployment of a contamination warning system at a water utility should follow
the typical programmatic approach in which proposed enhancements are planned, designed, implemented,
tested, maintained and refined. Table ES-1 provides a summary of the design and implementation
framework applied throughout the document.

Table ES-1. Overview of Design and Implementation Framework
Stage of Approach
Planning and pre-
design
Design
Implementation
Preliminary testing
Operation and
maintenance
Evaluation and
refinement
Description
Developing a core implementation team, defining design objectives to guide
implementation, and a preliminary assessment of existing capabilities relative to design
objectives.
Development of a preliminary concept of operations and development of a detailed work
plan and schedule to guide implementation.
Implementation of enhancements, installation of equipment, and training according to the
plan.
Operation of the contamination warning system for the purpose of collecting data
necessary to understand system performance and finalization of the concept of operations
to optimize system.
Operation of the contamination warning system for the purpose of monitoring for
contamination incidents and other water quality issues.
Analysis of data and information generated during full operation to refine and optimize the
system.
Who should be involved in contamination warning system deployment?

The drinking water utility is the operational hub of the system as the primary operator of the majority of
monitoring and surveillance components of the contamination warning system, with the exception of
public health surveillance. However, other partners may be involved in initial investigation of alarms
(trigger validation) and/or consequence management activities.  Figure ES-1 provides an overview of
potential partners in contamination warning system implementation. As illustrated in this figure, the
number and scope of partners that can become involved in responding to a contamination event can be
significant. In planning for implementation of a contamination warning system, drinking water utilities
should identify and engage local partners early in the process, particularly those partners such as local
health departments and public health and environmental laboratories that may have a significant role in
routine operations. Specific responsibilities of partners and when they are engaged may vary by utility
and jurisdiction.  Section 2 of the document provides additional details regarding the roles and
responsibilities of external partners.
May 2007

-------
                                 Planning for WS-CWS Deployment
     Federal Bureau of
       Investigation
    State Emergency
      Management
          and
    Homeland Security
       Agencies
       State Law
      Enforcement
           Centers For Disease
          Control and Prevention
                        EPA Regional Offices
           Local Health
           Department
        Local Fire, EMS,
          and HazMat
         Local Emergency
       Planning Committees
              Local Wastewater
                   Utility
                    Host
                  Facilities
                                                               Local Law
                                                              Enforcement
                                   Local Civil
                                   Government
Public Health and
 Environmental
  Laboratories
                   State Drinking and Waste
                   Water Primacy Agencies
                                    Neighboring Utilities
                      EPA Criminal
                   Investigation Division
                  EPA National Response
                         Center
                     State Emergency
                       Res ponders
State Government
                                                        Media
Figure ES-1.  Potential Contamination Warning System Partners
What are key design considerations for contamination warning system deployment?

Using this document as a guide, the utility in collaboration with local partners where appropriate, should
define what it wants the system to do (defining the design basis), develop a preliminary model of how the
system would function (developing a preliminary concept of operations) and compare the model to the
utility's current capabilities in order to identify the gaps and inform the plan to achieve the desired goals
and objectives. Design decisions to support planning for implementation of a contamination warning
system are summarized in Table ES-2. It is important to apply system engineering principles to all
aspects of design and implementation,  particularly as they relate to utility IT systems.

Table ES-2.  Summary of Design Decisions to Support Planning for Contamination Warning
     System Implementation	
Component
Document
 Section
Design Decisions
Online Water
Quality
Monitoring
               Water quality parameters to be monitored
               Use of a single monitoring station design or multiple designs in a tiered system
               Specific sensors and instruments integrated into a water quality monitoring
               station
               Number of water quality monitoring stations to install
               Methodology for determining the locations at which water quality monitoring
               stations should be installed
               Communication architecture to transmit data from monitoring locations to an
               operations center
               IT architecture used to manage and store water quality and related data
               Event detection software deployed to detect anomalies	
Sampling and
Analysis
               List of target contaminants, including contaminant class; responsible laboratory;
               analytical method; laboratory certification or accreditation for the method
               Sampling plan that addresses the training that staff should  receive; sampling
               equipment that should be procured;  sampling locations, and rationale; and
               sampling frequency, and rationale
               Procedures for triggered sampling and analysis
               Site characterization procedures and responsibilities	
May 2007
                                                                                VI

-------
                                  Planning for WS-CWS Deployment
Component
Document
 Section
Design Decisions
Enhanced
Security
Monitoring
               Preliminary facility list
               Site assessment summaries
               Facility risk ranking including a summary of physical security effectiveness,
               probability of attack, and consequence of contamination criteria
               Final facility list with a description of recommended improvements for each
               facility
               Communication architecture to transmit data from monitoring locations to an
               operations center	
Consumer
Complaint
Surveillance
               The utility-specific model of the consumer complaint surveillance component
               design
               An assessment of the existing consumer complaint management system
               An approach for enhancing the consumer complaint management system into a
               consumer complaint surveillance system
               IT architecture used to manage and store water quality complaint and related
               data
               Event detection software deployed to detect anomalies	
Public Health
Surveillance
               Identification of local public health partners
               Identification and assessment of existing surveillance capability relative to
               contamination warning system objectives
               Improvements  or additions to existing surveillance capabilities
               Development of a framework for communication and notification	
Consequence
Management
               Objectives for consequence management plan
               Utility self-assessment of existing plans and response capabilities
               Identification and assessment of response partner capabilities
               Level of integration with other response plans
               Framework for development of consequence management plan that allows for
               seamless transition from routine operations and initial trigger validation to
               consequence management actions	
May 2007
                                                                                  VII

-------
                              Planning for WS-CWS Deployment
                             Water Security Initiative:
   Interim Guidance on Planning for Contamination Warning System
                                      Deployment
ABBREVIATIONS AND ACRONYMS	I

ACKNOWLEDGEMENTS	II

EXECUTIVE SUMMARY	Ill

SECTION 1.0: INTRODUCTION	1

  1.1     CONTAMINATION WARNING SYSTEMS - AN OVERVIEW	2
  1.2     DOCUMENT OVERVIEW	7

SECTION 2.0: PROJECT PLANNING AND MANAGEMENT	8

  2.1     APPLICATION OF SYSTEM ENGINEERING PRINCIPLES	8
    2.1.1   Development and Management of Work Plan and Schedule	8
    2.1.2   Integrated Concept of Operations	9
    2.1.3   IT System Engineering	10
  2.2     UTILITY STAFFING	11
    2.2.1   Project Management Team	11
    2.2.2   Utility Staff	12
  2.3     LOCAL PARTNERS	13
    2.3.1   Identifying and Engaging Partners	14
    2.3.2   Considerations for Formal Agreements with Local Partners	16
  2.4     COSTS	17

SECTION 3.0: DESIGN AND IMPLEMENTATION FRAMEWORK	18

  3.1     PLANNING AND PRE-DESIGN	18
    3.1.1   Building the Team	18
    3.1.2   Defining the Utility-Specific Design Basis and Design Objectives	18
    3.1.3   Preliminary Assessment and Gap Analysis	20
  3.2     DESIGN	20
    3.2.1   Conceptualize System	20
    3.2.2   Work Plan for Implementation	21
  3.3     IMPLEMENTATION	21
  3.4     PRELIMINARY TESTING	21
    3.4.1   Baseline Operation	22
    3.4.2   Finalization of Concept of Operations and Consequence Management Plan	22
  3.5     OPERATION AND MAINTENANCE	23
    3.5.1   Operation	23
    3.5.2   Maintenance	23
  3.6     EVALUATION AND REFINEMENT	23
    3.6.1   Evaluation	24
    3.6.2   Refinement	24

SECTION 4.0: ONLINE WATER QUALITY MONITORING	25

  4.1     MONITORING NETWORK DESIGN	26
    4.1.1   Pre-Design	27
    4.1.2   Design and Implementation Approach	29
    4.1.3   Available Tools and Resources	30
  4.2     MONITORING STATION DESIGN AND INSTALLATION	31
    4.2.1   Pre-Design	31
    4.2.2   Design and Implementation Approach	33
    4.2.3   Available Tools and Resources	38
  4.3     COMMUNICATIONS ARCHITECTURE	38
    4.3.1   Pre-Design	39

May 2007                                                                              viii

-------
                               Planning for WS-CWS Deployment

    4.3.2   Design and Implementation Approach	39
    4.3.3   Available Tools and Resources	40
  4.4     DATA MANAGEMENT AND IT ARCHITECTURE	41
    4.4.1   Pre-Design	41
    4.4.2   Design and Implementation Approach	42
    4.4.3   Available Tools and Resources	42
  4.5     WATER QUALITY EVENT DETECTION	43
    4.5.1   Planning	43
    4.5.2   Implementation Approach	46
    4.5.3   Available Tools and Resources	47
  4.6     STAFFING AND COST CONSIDERATIONS	48
    4.6.1   Staffing	48
    4.6.2   Cost Considerations	49

SECTION 5.0:  SAMPLING AND ANALYSIS	51

  5.1     LABORATORY CAPABILITY AND CAPACITY	52
    5.1.1   Pre-design	52
    5.1.2   Design and Implementation Approach	55
    5.1.3   Available Tools and Resources	56
  5.2     SAMPLING AND ANALYSIS	56
    5.2.7   Pre-design	57
    5.2.2   Design and Implementation Approach	58
    5.2.3   Available Tools and Resources	61
  5.3     SITE CHARACTERIZATION AND FIELD SCREENING	61
    5.3.7   Pre-design	61
    5.3.2   Design and Implementation Approach	63
    5.3.3   Available Tools and Resources	65
  5.4     STAFFING AND COST CONSIDERATIONS	65
    5.4.1   Staffing	65
    5.4.2   Cost Considerations	66

SECTION 6.0:  ENHANCED SECURITY MONITORING	67

  6.1     PRE-DESIGN	69
  6.2     DESIGN AND IMPLEMENTATION APPROACH	72
  6.3     AVAILABLE TOOLS AND RESOURCES	75
  6.4     STAFFING AND COST CONSIDERATIONS	76
    6.4.1   Staffing	76
    6.4.2   Cost Considerations	77

SECTION 7.0:  CONSUMER COMPLAINT SURVEILLANCE	78

  7.1     PRE-DESIGN	80
  7.2     DESIGN AND IMPLEMENTATION APPROACH	81
  7.3     AVAILABLE TOOLS AND RESOURCES	82
  7.4     STAFFING AND COST CONSIDERATIONS	83
    7.4.1   Staffing	83
    7.4.2   Cost Considerations	84

SECTION 8.0:  PUBLIC HEALTH SURVEILLANCE	85

  8.1     PRE-DESIGN	86
  8.2     DESIGN AND IMPLEMENTATION APPROACH	89
  8.3     AVAILABLE TOOLS AND RESOURCES	90
  8.4     STAFFING AND COST CONSIDERATIONS	91
    8.4.1   Staffing	91
    8.4.2   Cost Considerations	92

SECTION 9.0:  CONSEQUENCE MANAGEMENT	94

  9.1     PRE-DESIGN	94
  9.2     DESIGN AND IMPLEMENTATION APPROACH	97
  9.3     AVAILABLE TOOLS AND RESOURCES	98

May 2007                                                                                 ix

-------
                              Planning for WS-CWS Deployment

  9.4    STAFFING AND COST CONSIDERATIONS	100
    9.4.1  Staffing	100
    9.4.2  Cost Considerations	100

SECTION 10.0: REFERENCES	102

APPENDIX A: GLOSSARY	104

APPENDIX B: INFORMATION SECURITY CONSIDERATIONS	106

  B.I    ASSESSMENT OF MATERIAL SENSITIVITY, ACCESS, AND LAWS	106
  B.2    SENSITIVE MATERIALS TRACKING AND STORAGE	107
  B.3    SENSITIVE MATERIALS HANDLING PROTOCOL AND PROCEDURES	107
  B.4    STAFF CERTIFICATION AND BACKGROUND CHECKS	108
  B.5    COORDINATION AND COOPERATION WITH PARTNERS, CONTRACTORS AND SUBCONTRACTORS	109

List  of Tables
TABLE ES-l. OVERVIEW OF DESIGN AND IMPLEMENTATION FRAMEWORK	v
TABLE ES-2. SUMMARY OF DESIGN DECISIONS TO SUPPORT PLANNING FOR CONTAMINATION WARNING SYSTEM
    IMPLEMENTATION	vi
TABLE 1-1. CONTAMINANT DETECTION CLASSES AND POTENTIAL MEANS OF DETECTION	5
TABLE 1-2. DESIGN BASIS SUMMARY BY CONTAMINATION WARNING SYSTEM COMPONENT	6
TABLE 2-1. SUMMARY OF POTENTIAL CONTAMINATION WARNING SYSTEM PARTNERS	14
TABLE 3-1. OVERVIEW OF DESIGN AND IMPLEMENTATION FRAMEWORK	18
TABLE 3-2. DESIGN BASIS CONSIDERATIONS	19
TABLE 4-1. DESIGN BASIS CONSIDERATIONS FOR ONLINE WATER QUALITY MONITORING	25
TABLE 4-2. IMPACT OF CONTAMINANT DETECTION CLASSES ON WATER QUALITY PARAMETERS	31
TABLE 4-3. STANDARD MEASURES FOR EVALUATING EDS TOOL PERFORMANCE	44
TABLE 4-4. SAMPLE MEASURES FOR EVALUATING EDS SOFTWARE	45
TABLE 4-5. ONLINE WATER QUALITY MONITORING STAFFING CONSIDERATIONS	48
TABLE 5-1. DESIGN BASIS CONSIDERATIONS FOR SAMPLING AND ANALYSIS	51
TABLE 5-2. CONSIDERATIONS FOR ANALYTICAL APPROACH TO ESTABLISHING SAMPLING AND ANALYSIS
    CAPABILITIES BY CONTAMINANT CLASS	52
TABLE 5-3. EXAMPLES OF BASELINE DATA SOURCES	59
TABLE 5-4. CONSIDERATIONS FOR CONTAMINANT COVERAGE FOR FIELD SCREENING	63
TABLE 5-5. SAMPLING AND ANALYSIS STAFFING CONSIDERATIONS	65
TABLE 6-1. DESIGN BASIS CONSIDERATIONS FOR ENHANCED SECURITY MONITORING AT SELECTED SITES	67
TABLE 6-2. EXAMPLE IMPROVEMENTS BY WATER UTILITY FACILITY TYPE	68
TABLE 6-3. ENHANCED SECURITY MONITORING STAFFING CONSIDERATIONS	76
TABLE 7-1. DESIGN BASIS CONSIDERATIONS FOR CONSUMER COMPLAINT SURVEILLANCE	78
TABLE 7-2. CONSUMER COMPLAINT SURVEILLANCE STAFFING CONSIDERATIONS	83
TABLE 8-1. DESIGN BASIS CONSIDERATIONS FOR PUBLIC HEALTH SURVEILLANCE	86
TABLE 8-2. PUBLIC HEALTH SURVEILLANCE STAFFING CONSIDERATIONS	92
TABLE 9-1. OVERVIEW OF RESPONSE PARTNER ROLES AND RESPONSIBILITIES FOR CONSEQUENCE MANAGEMENT ..96

List  of Figures
FIGURE ES-l. POTENTIAL CONTAMINATION WARNING SYSTEM PARTNERS	vi
FIGURE 1-1.  OVERVIEW OF EPA's WATER SECURITY INITIATIVE	1
FIGURE 1-2.  ARCHITECTURE OF THE WATER SECURITY CONTAMINATION WARNING SYSTEM	3
FIGURE 2-1.  POTENTIAL CONTAMINATION WARNING SYSTEM PARTNERS	14
FIGURE 2-2.  RECOMMENDED STRATEGY FOR ENGAGING CONTAMINATION WARNING SYSTEM PARTNERS	16
FIGURE 4-1.  EXAMPLE MONITORING STATION TRADEOFF CURVE	28
FIGURE 4-2.  EXAMPLE WATER QUALITY MONITORING STATION DESIGN USED IN THE INITIAL PILOT	35
FIGURE 4-3.  EXAMPLE ROC CURVE	45
FIGURE 7-1.  THE RECOMMENDED FILTER, FUNNEL, AND Focus APPROACH TO CUSTOMER FEEDBACK DATA
    OPTIMIZATION FOR UTILITY-MANAGED CONSUMER CALLS	80
FIGURE 8-1 PUBLIC HEALTH DATA SOURCES AND DESIGN OBJECTIVES	87
May 2007

-------
                                Planning for WS-CWS Deployment


                            Section 1.0:  Introduction

This document presents a basic framework to assist drinking water utilities with planning for
contamination warning system deployment based on the model developed under U.S. Environmental
Protection Agency's (EPA) Water Security initiative (formerly known as WaterSentinel).

Initiated in response to Homeland Security Presidential Directive 9, the overall goal of the Water Security
initiative is to design and deploy contamination warning systems for drinking water utilities through a
phased approach that includes conceptual design, implementation at an initial pilot utility, expansion to
additional pilot utilities, and ultimately development of guidance and tools to support implementation at
drinking water utilities across the nation.  Figure 1-1 summarizes this process.
Phase
Approach
Scope
Design
Specificity
Funding
DESIGN
System Architecture
DEMONSTRATE
Initial Pilot
Additional Pilots
EXPAND
Voluntary National Adoption
^^~~[S Applied by^T/
Apply to single Evaluate multiple Evaluate
Conceptual , 	 N Pilot utilitV ^^ U"eS ^^ C°nvert'°
design ^^ ^ ["] ^ H guidance for
AX . r/ AX • r/
V 7 Refine !__£ \J Refine 1_J>
\/ anc| ^ and
enhance enhance
Not
applicable
Low
ll
High-
Applies to pilot utility only
•& ml
ia s
iL^ A
High-
Appliesto each pilot
EPA Funds
iW
Medium -
Applies to range of utilities
Utility Funds
Figure 1-1.  Overview of EPA's Water Security Initiative

Monitoring the distribution system is the primary focus of contamination warning systems. Through the
assessment of vulnerabilities to drinking water systems, water security experts have identified the
distribution system as one of the most vulnerable components in a drinking water utility, with respect to
contamination. Furthermore, intentional contamination, or even the threat of contamination can have
significant impacts.

Drinking water utilities occasionally receive threats or indications of possible contamination.  These
contamination threat warnings can be a direct threat or an unusual observation or discovery that indicates
the potential for contamination and initiates actions to investigate and potentially respond.  However,
these threat warnings are not standardized and are difficult to corroborate in the absence of an integrated
monitoring and surveillance system and close coordination with response partners including, but not
limited to public health, emergency responders, and law enforcement.

Deployment of contamination warning  systems for drinking water distribution systems provides a
mechanism by which drinking water utilities can detect and respond to contamination threats  and
incidents. A contamination warning system is a proactive approach to managing threat warnings that uses
advanced monitoring technologies/strategies and enhanced surveillance activities to collect, integrate,
analyze, and communicate information  to provide a timely warning of potential water contamination
incidents and initiate response actions to minimize public health and economic impacts.
May 2007

-------
                                Planning for WS-CWS Deployment

In addition, deployment of a contamination warning system provides the opportunity for dual-use
applications beyond security that could help to promote sustainability of the system by optimizing utility
operations. Drinking water distribution systems may be accidentally contaminated through cross-
connections with non-potable water, permeation of contaminated water through leaking pipes in areas of
the distribution system subject to low pressures, or chemical reactions or microbial growth within the
distribution system pipes. Such unintentional events that result in degradation to distributed water quality
may occur with some regularity.

In 2005, EPA documented the conceptual design for contamination warning systems in WaterSentinel
System Architecture (USEPA, 2005a) and began implementation of the first WS contamination warning
system pilot in partnership with the City of Cincinnati at the Greater Cincinnati Water Works (GCWW).
Section 1.1 provides an overview of contamination warning systems and a summary of the design basis
and Section 1.2 provides an overview of how to use this document to support design and implementation
of contamination warning systems  based on EPA's approach and lessons learned from the initial pilot.

1.1    Contamination Warning Systems - An Overview

A contamination warning system is not merely a collection of monitors and equipment placed throughout
a water system to alert of intrusion or contamination. Fundamentally, it is an exercise in information
acquisition and management. Different information streams are captured, managed, analyzed, and
interpreted to recognize potential contamination incidents in time to respond effectively. These data
sources, when used concurrently, should support and augment each other such that the chances of
detecting a contamination incident are better than using any one information source on its own. While the
contamination warning system should be designed by the utility, some data sources may be outside of the
utility; thus, cooperation with partners is an integral part to the success of a contamination warning
system. A complete contamination warning system consists of the following monitoring and surveillance
components:

    •  Online water quality monitoring comprises stations located throughout the distribution system
       that measure chlorine, total organic carbon, conductivity, and other parameters. Software
       analyzes the monitoring data to establish a water quality base state.  Possible contamination is
       indicated when a significant, unexplained deviation from the base state  occurs.
    •  Sampling and analysis is  the collection of distribution system samples that are analyzed for
       various contaminant classes as well as specific contaminants.  Sampling is both routine to
       establish a baseline and triggered to respond to an indication of possible contamination from
       another component. Analyses are conducted for chemicals, radionuclides, pathogens, and toxins
       using a laboratory network.
    •  Enhanced security monitoring includes the equipment and procedures that detect and respond
       to security breaches at distribution system facilities. Security equipment may include cameras,
       motion activated lighting, door contact alarms, ladder and window motion detectors, area motion
       detectors, and access hatch contact alarms.
    •  Consumer complaint surveillance enhances the collection and automates the analysis of calls by
       consumers for water quality problems indicative of possible contamination. Consumers may
       detect contaminants with characteristics that impart an odor, taste, or visual change to the
       drinking  water.
    •  Public health surveillance involves the analysis of health-related data to identify disease events
       that may stem from drinking water contamination.  Public health data may include over-the-
       counter (OTC) drug sales,  hospital admission reports, infectious disease surveillance, emergency
       medical service (EMS) reports, 911 calls, and poison control center calls.

As illustrated in Figure 1-2, consequence management is another key aspect of the contamination
warning system architecture.  Consequence management governs response actions when contamination is
May 2007

-------
                                Planning for WS-CWS Deployment
determined to be possible and includes activities that involve drinking water utilities as well as external
partners.
Figure 1-2. Architecture of the Water Security Contamination Warning System
The basic process or conceptual model for contamination warning system operation is described as
follows, moving from left to right in the diagram.
    •  Monitoring and Surveillance. As previously discussed, integration of information from a
       variety of data sources internal and external to the drinking water utility is a critical aspect of
       contamination warning system monitoring and surveillance activities. While the specific types of
       information streams may vary, the basic components of online water quality monitoring,
       sampling  and analysis, enhanced security monitoring, consumer complaint surveillance, and
       public health surveillance are necessary to meet the design objectives as discussed later in this
       section. Monitoring and surveillance of these components and information streams occurs on a
       routine basis, in near-real time until an anomaly or deviation from the baseline or base state is
       detected.
    •  Event detection and Possible Determination. Event detection is the process or mechanism by
       which an  anomaly or deviation from the baseline or base state is detected. This detection is
       referred to as a trigger. How event detection is implemented and the tools that are utilized may
       vary significantly from component to component and can include sophisticated algorithms,
       simple business logic, etc. This process should be automated to the extent possible.  Another
       aspect of the  contamination warning system that is tightly coupled with event detection is the
       initial validation of triggers to assess possibility of drinking water contamination. As discussed in
       greater detail throughout this document, many of the components provide non-specific indicators
       of contamination and thus, when a trigger occurs, validation  is necessary prior to determining if
       contamination is "possible." If trigger validation indicates that contamination is "possible," then
       the credibility determination step is initiated; otherwise, the contamination warning system
       component returns to routine monitoring and surveillance.
    •  Credible  Determination.  Credibility determination is a transition from routine operation to
       consequence  management.  Credibility determination procedures are performed using information
       from  all contamination warning system components as well as external resources when available
       and relevant.  Through the credibility determination process, some preliminary response actions
       may be initiated to limit or minimize impacts of suspected contamination. If contamination is
       determined to be credible, additional confirmatory and response actions are initiated. If

May 2007                                                                                   3

-------
                                Planning for WS-CWS Deployment

       contamination can be ruled out based on additional information gathered through the credibility
       determination process, the system returns to routine monitoring and surveillance activities.
    •  Confirmed Determination.  In this stage of consequence management, additional information is
       gathered and assessed to confirm drinking water contamination.  Response actions initiated
       during credible determination are expanded and additional response activities may be
       implemented.
    •  Remediation and Recovery. Once contamination has been confirmed, and the immediate crisis
       has been addressed through response, remediation and recovery actions defined in the
       consequence management plan are  performed to restore the system to normal operations.

The architecture presented in Figure  1-2 was derived initially through the conceptual design of the
contamination warning system and was refined based on lessons learned from design and implementation
of the initial Water Security initiative pilot.

The consequences associated with a particular contamination scenario are largely a function of the
contaminant type and concentration,  the location of contaminant introduction, and the relative timing of
exposure, onset of symptoms, detection, and response. These results lead to a design basis for a
contamination warning system that considered four primary attributes: contaminant coverage, spatial
coverage, timing of detection, and reliability. Therefore, defining a design basis by the following
performance objectives should guide a utility through the design and implementation of an effective
contamination warning system. The  system, as a whole, should be able to meet the following design
objectives:
    •  Detection of a broad spectrum of contaminant classes. There are a large number of
       contaminants that could cause serious harm if introduced into the drinking water distribution
       system. As part of the contamination warning system design basis, contaminants were prioritized
       and then binned into 12 detection classes (see additional information below).  Use of the detection
       classes to inform design provides more robust detection capability than analyzing  for only a select
       number of contaminants and also avoids the challenge associated with designing a system around
       a list containing hundreds of potential contaminants.
    •  Achieve spatial coverage of the entire distribution system. Spatial coverage can be considered
       hydraulically and geographically. For components such as online water quality monitoring,
       spatial coverage is a function of the number of online water quality monitors and their placement
       throughout the distribution system.  For other components such as consumer complaints or public
       health surveillance, spatial coverage varies geographically based on population density,
       population demographics (industrial vs. residential), and/or types of surveillance systems and
       tools used within a jurisdiction.
    •  Detect contamination in sufficient time for effective response. There are three  periods
       associated with the evaluation of the timeline of a contamination incident, including (1) the time
       during which consequences (exposures, illnesses, fatalities, pipe contamination, etc.) are
       experienced in the population, (2) the time of initial detection, and (3) the time of response
       actions. A key aspect of a contamination warning system is to provide initial detection  in a
       timeframe that allows for the implementation of response actions that result in a significant
       reduction in consequences.
    •  Reliably indicate  a contamination incident with a minimum number of false-positives.
       Reliability can be considered from two perspectives. The first is operation, that is, factors such as
       contamination warning system component capabilities and necessary maintenance. The second is
       performance, defined as the ability  of the system to  provide information that leads decision
       makers to successfully infer that contamination has  or has not occurred.
    •  Provide a sustainable architecture to monitor distribution system water quality. The
       integration of multiple monitoring and surveillance  strategies already in use at the utility and
       public health department should improve acceptance of the system, and thus long-term
       sustainability. A contamination warning system should be designed as a dual-use  application to
May 2007

-------
                                Planning for WS-CWS Deployment

       benefit the utility in day-to-day operations while also providing the capability to detect intentional
       or accidental contamination incidents.

The first design objective described above introduced the concept of contaminant classes.  Table 1-1
presents a summary of contaminant detection classes developed during the conceptual design phase of
EPA's Water Security initiative (USEPA, 2005a, USEPA, 2005b).  This table shows the potential means
of detection for each contaminant class by three of the components: water quality monitoring, consumer
complaint surveillance, and public health surveillance (considered as two independent data streams).
Enhanced security monitoring could potentially detect any contamination event, while sampling and
analysis can detect contaminants within the group of target analytes for the methods employed. Thus,
collectively, the five components provide comprehensive contaminant coverage,  and provide a means of
confirming a possible contamination incident through an independent data stream.

Table 1-1.  Contaminant Detection Classes and Potential Means of Detection
Class
1
2
3
4
5
6
7
8
9
10
11
12
Description
Petroleum products
Pesticides (with odor or taste)
Inorganic compounds
Metals
Pesticides (odorless)
Chemical warfare agents
Radionuclides
Bacterial toxins
Plant toxins
Pathogens causing diseases with unique symptoms
Pathogens causing diseases with common
symptoms
Persistent chlorinated organic compounds
Water
Quality
X
X
X
X
X
X
X
X
X
X
X
X
Consumer
Complaints
X
X
X
X
X







911 calls
/EMS

X
X
X
X
X
X





Syndromic
Surveillance1







X
X
X
X

 Collecting and analyzing nontraditional data to detect a change or trend in the health of a population using
categories of disease rather than formal diagnosis.

In designing a contamination warning system, each of the design objectives should be considered for the
system as a whole as well as for each component.  Table 1-2 summarizes the design objectives as they
relate to each of the contamination warning system components. These design objectives are presented
again in Sections 4 through 8 along with a summary of design and implementation considerations for
each of the monitoring and surveillance components.
May 2007

-------
                                               Planning for WS-CWS Deployment
Table 1-2. Design Basis Summary by Contamination Warning System Component
WS-CWS
Component
Online Water
Quality
Monitoring


Sampling and
Analysis

Enhanced
Security
Monitoring

Consumer
Complaint
Surveillance


Public Health
Surveillance



Capability
Can indicate the presence
of a contaminant that
significantly affects one or
more monitored
parameters.


Can positively identify the
presence of any
contaminant in the suite of
target analytes and above
a well-defined minimum
reporting level.

Can detect an intrusion
that may have provided the
opportunity for introduction
of any contaminant.

Can indicate the presence
of a contaminant that
significantly affects one or
more aesthetic qualities of
water.
Can detect the presence of
a symptom or illness in a
population which may be
the result of the presence
of a disease causing
agent. May be able to
identify the contaminant
through clinical diagnosis/
testing.
Contaminant
Coverage
High detection
potential for
classes 1, 2, 3, 5,
8, 9, 10, 11 and 12.
Moderate detection
potential for
classes 4, 6, and 7.

High detection
potential for
classes 1, 2, 3, 4,
7, and 12;
Moderate detection
potential for
classes 5, 6, 8, 9,
10, 11.
Covers all
contaminant
classes.

High detection
potential for
classes 1 and 2.
Moderate detection
potential for
classes 3, 4, and 5.

Covers
contaminant
classes 2 through
11; detection
potential varies with
type of
surveillance.

Spatial Coverage
Function of location,
number, and density
of monitoring
stations.


Function of location,
number, and density
of sampling stations,
as well as sample
type (composite vs.
grab).

Limited to those
elements of
infrastructure for
which physical
security can be
monitored.
Entire service area
for contaminants with
detectable taste,
color, or odor
characteristics.

Comprehensive
coverage of a
particular city or
county, which may
include all, or a large
portion of, the utility
service area.

Timeliness
Function of hydraulic
travel time from the
point of contaminant
introduction to the
sensor, and the
concentration of the
contaminant.


Function of sampling &
analysis frequency and
the total time to process
the sample and analyze
the results.

Function of the type of
security monitoring
system and the time to
evaluate a security
breach.

Function of the time
from exposures to
consumer reporting,
complaint
categorization,
assessment and
investigation.
Function of the time
from the initial
exposures, the onset of
symptoms, and the
point at which public
health officials
recognize the incident
as a potential water-
borne illness.
Reliability
Rate of false positive / negative
results in this application is largely
unknown at this time. May be
addressed through event detection
systems and consequence
management.


Function of the reliability of
sampling and analysis methods
(high for established techniques).
Baseline needed for reliable
interpretation of results.

Can be a reliable means of
identifying an intrusion, especially
when these breaches may involve
contamination, such as in storage
tanks and reservoirs.

A potentially reliable indicator for
contaminants with detectable
characteristics if a robust
complaint reporting and tracking
system is in place.
May be a reliable means of
identifying the incidence of illness
in a population, but timing of
communication between drinking
water and public health officials
should be optimized such that
appropriate response, actions
could be implemented in time to
reduce consequences.
Sustainability
Provides utility with a better
understanding of water
quality variability
throughout distribution
system and provides an
opportunity to optimize
distribution system
operation.
Provides utility with an
opportunity to exercise
sampling and laboratory
protocols and may; provide
information about
previously unknown
contaminants that occur in
the system.
Provides utility with
increased physical
infrastructure protection
and awareness. Reduces
the occurrence of nuisance
tampering.
Provides utility an
opportunity to manage
consumer information more
effectively and can serve
as a tool for enhanced
consumer confidence.


Provides an opportunity for
utility and local health



May 2007

-------
                                Planning for WS-CWS Deployment
1.2     Document Overview
This document describes planning considerations for contamination warning system deployment based on
the approach deployed at, and lessons learned from, the initial Water Security initiative pilot at GCWW.
The primary focus of the document is on planning and pre-design.  Throughout the document available
resources and tools are highlighted to facilitate contamination warning system design and
implementation.  The lists of resources are not intended to be exhaustive, but rather a starting point to
facilitate the planning process.  Sections included in the document are as follows:
    •   Section 2: Project Planning and Management.  This section introduces the concept of system
        engineering for contamination warning systems and discusses critical aspects of project planning
        for design and implementation.  Although the drinking water utility is the primary organization
        involved with design and implementation, other key partners have a significant role as well and
        should be included in the process.  Subsections focus on system engineering, utility staffing,
        coordination with local partners, and costing considerations.
    •   Section 3: Design and Implementation Framework.  This section describes the framework for
        design and implementation of a contamination warning system including planning or pre-design,
        design, implementation, preliminary testing (start-up and baselining of components), operation
        and maintenance, and evaluation and refinement.
    •   Section 4: Online Water Quality Monitoring. This section provides the design basis for online
        water quality monitoring, identifies critical design decisions and describes design and
        implementation aspects relative to monitoring network design (placement of monitoring stations),
        monitoring station design, communications and information technology architecture, and water
        quality event detection.
    •   Section 5: Sampling and Analysis. This section describes the design basis for sampling and
        analysis, identifies critical design decisions and describes design and implementation aspects
        relative to laboratory capability and capacity, sampling  and analysis (baseline, maintenance,
        triggered), and. field screening and site characterization.
    •   Section 6: Enhanced Security Monitoring.  This section describes the design basis for enhanced
        security monitoring and a systematic methodology for selecting sites for security improvements
        and designing those enhancements.
    •   Section 7: Consumer Complaint Surveillance. This section describes an approach to managing
        customer calls and customer information.  The document focuses on assessing existing call
        management systems and protocols based on contamination warning system design objectives
        and describes considerations for optimized call management, tracking, and analysis.
    •   Section 8: Public Health Surveillance. This section describes the design basis for public health
        surveillance and design considerations to assist with planning for implementation. The public
        health surveillance component of a contamination warning system relies heavily on relationships
        and partnerships with local health departments.  In addition to describing the design basis for the
        public health surveillance component, this section provides background information for drinking
        water utilities on the types of public health surveillance that may be implemented by local or state
        health departments.  Considerations for enhancing or expanding on these existing systems or
        approaches to include more timely data streams such as 911 calls or emergency medical service
        (EMS) events for fast-acting contaminants are also addressed.
    •   Section 9: Consequence Management.  This section describes planning considerations for
        development of a consequence management plan for the drinking water utility, including
        identification and coordination with local response partners and organizations.  Types of plans
        and information that can be leveraged to support development of this plan - both internal and
        external to the utility - are discussed.  In addition, existing training programs and resources are
        identified.

In addition to the sections described above, this document also includes a list of acronyms, references, a
glossary (Appendix A), and information security considerations (Appendix B).

May 2007                                                                                   7

-------
                               Planning for WS-CWS Deployment

           Section 2.0:   Project  Planning and Management

This section discusses critical aspects of project planning and management for deployment of a
contamination warning system. Although the drinking water utility is the primary organization involved
with deployment, other key partners also have significant roles and should be included in the process. As
discussed throughout this section, deployment of a contamination warning system is a significant
undertaking that impacts most departments or divisions within the utility at some stage. A commitment at
all levels of the organization, from senior managers and supervisors to staff supporting routine operations
within the utility or in the field, is essential.

2.1    Application of System Engineering  Principles
A contamination warning system is, by design, a systematic approach to monitoring and surveillance for
the timely detection of drinking water contamination. As such, deployment of a contamination warning
system relies on system engineering principles to support coordination of technical and management
activities. System engineering is an interdisciplinary approach to design and implementation of systems.
Through system engineering, disciplines and specialty groups are integrated in a team effort forming a
structured development process that proceeds from design to implementation to operation. From the
beginning of the project, system engineering principles are critical to successful planning and
implementation. The primary application of system  engineering for a contamination warning system is to
ensure that the system - monitoring and surveillance components and consequence management as
discussed in detail in Section 4 to Section 9 - functions as an integrated whole.

System engineering involves the  integration of monitoring and surveillance components and consequence
management.  Throughout planning and implementation, many activities can and should occur in parallel.
However, it is necessary to reconcile activities and key stages of deployment to optimize the function of
the system.  For example, once a preliminary concept of operations (as discussed in Section 2.1.2) has
been defined for the system, component-specific designs and consequence management planning can
occur in parallel.  However, it is important to reconcile the consequence management plan with the
various monitoring and surveillance components as they are designed and implemented to ensure
seamless transition from routine operations to consequence management.

As part of system engineering, coordination strategies should be applied throughout the deployment
process to ensure that there is a consistent vision and understanding of goals and objectives. To facilitate
coordination, meetings involving representatives from each of the components should be held early in the
planning process.  It may be helpful to establish a core team with representatives for each  of the
component  teams or supporting divisions involved in design, implementation, or operation of the
contamination warning system. Regular meetings serve as a forum for component teams to share their
progress with the project management team, and as an opportunity to identify synergies among the
enhancements. Through discussions,  critical cross-cutting issues may be identified and resolved
consistently across the project, making for more efficient problem solving and refinement. In addition,
these meetings could highlight dual-use  applications of the system, reinforcing the value and improving
sustainability. As  utilities work through system deployment, it may be necessary to prioritize certain
activities across components of the system based on  criticality, resources, schedule, or other issues.
Addressing these issues through these routine coordination meetings may help to facilitate the
prioritization process.

2.1.1  Development and Management of Work Plan and Schedule

The development of detailed work plans for monitoring and surveillance components as well as
consequence management is critical.  These work plans should outline the enhancements necessary to
progress from the existing state to the desired state and should be useful in coordinating timely
implementation activities.  The work plan builds on the current state of the utility by identifying specific

May 2007                                                                                 8

-------
                                 Planning for WS-CWS Deployment

equipment, computer hardware or software, training and process modifications that are necessary to
achieve specified goals. Each step in the work plan should contain detailed action items and a defined
schedule to meet the goals specified by the design decisions. Coordination across components to
maximize efficiency is critical in work plan development to ensure that all necessary activities are
planned for, while eliminating redundancies.

Although detailed schedules and work plans should be developed to support component activities, it is
important to track high-level milestones across all components to ensure that activities are managed
efficiently and effectively and that there is communication and coordination on overarching or cross-
cutting issues. To facilitate this high-level planning, development of an integrated concept of operations
can serve as a project management tool as well as a definition of system operations.

2.1.2   Integrated Concept of Operations

The concept of operations is the description of the routine operation and initial trigger validation for each
component of the contamination warning system. While the terminology and approach for this
documentation may vary, it is important that the  operational concepts be clearly documented and that
utility  staff and local partners understand their roles and responsibilities in operating the system. The
concept of operations should be developed at the component level to present the day-to-day functioning
of components and management of information to support trigger validation and initial response actions
(or, in  the case of consequence management,  integrating information from multiple components and
initiating a response). However, the development of the component concept of operations should be
coordinated to ensure that information from multiple components can be integrated to better inform
response decisions.  It is also important to integrate day-to-day operation of the contamination warning
system into routine job duties.  Guidance on development of a concept of operations will be the focus of
future  EPA efforts.

While  the concept of operations for a contamination warning system is intended to guide day-to-day
operations and trigger validation, it also has an important design function at the system level. A
preliminary concept of operation for each component can be an effective means of determining utility-
derived requirements for the design and implementation of that component. An integrated concept of
operations encompasses the component-specific concept of operations by explaining how the components
inter-relate, how their data streams are combined and how the system as a whole meets the design basis.
It is recommended that a preliminary concept of operations be developed during the planning or pre-
design stage.

To develop a preliminary concept of operations, utilities should define design objectives as discussed in
Section 3 and leverage the component-specific concepts presented in Section 4 - Section 9.  At the
conclusion of planning and pre-design, utilities should be able to develop a high-level concept of
operations that includes critical information point such as the users and key decision makers involved in
routine operation of a given component. As part of the system engineering approach to implementation, it
may be possible to develop and define teams  and roles and responsibilities for utility staff across all
components, thereby presenting a cohesive, system-wide picture of what key utility personnel should do
during the day-to-day operation of the  contamination warning system. An important feature of this
integrated view is its ability to illustrate how  contamination warning system activities relate to normal job
functions and provide benefits to the utility and staff beyond contamination warning.  In addition, a
preliminary timeline for initial trigger validation should be part of the preliminary concept of operations,
and is  an important  consideration in early stages  of consequence management plan development. With
the initial draft of the concept of operations, these timelines should be projections based on desired goals
for trigger validation and decision making. It will likely be necessary to refine these projections based on
the design that is ultimately implemented. These timelines may also serve as a metric for evaluation of
the system and individual components as part of evaluation and refinement. Development of a
 May 2007

-------
                                Planning for WS-CWS Deployment

preliminary concept of operations will also serve to highlight the significance of information technology
(IT) systems, data flows, and IT-based user requirements in contamination warning system deployment.

2.1.3  IT System Engineering

A contamination warning system is not merely a collection of monitors and equipment placed throughout
a water system to alert of intrusion or contamination.  Fundamentally, it is an exercise in information
acquisition and management. Different information streams should be captured, managed, analyzed, and
interpreted in time to recognize potential contamination incidents and respond effectively.  Information
from several different databases and information systems at a water utility should be integrated to enable
event detection, trigger validation, credibility determination, and management of response actions in a
timely  manner.  In addition, data and information from local partners including, but not limited to, fire
departments, public health, and law enforcement should be integrated. Thus, the success of the
contamination warning system implemented at a drinking water utility will depend heavily on effective
data and information management.

In planning for contamination warning  system implementation and development of a preliminary concept
of operations, development of an inventory of the IT systems used in the operation of the components and
day-to-day operations may serve as a valuable tool. It is also useful to consider information flows and
how information is collected and managed throughout existing systems.  An information flow diagram
describes how data and information flows between system elements and users during routine operations
of a contamination warning system component, including the information systems, databases, and user
interfaces used during activities related to routine monitoring and surveillance, event detection, trigger
validation, and credibility determination for a component. Development of a preliminary information
flow diagram could serve as a useful tool for facilitating design and implementation discussions and
decisions. Although each component data stream is unique,  a common set of system  elements (data
sources, event detection, alarm or trigger notification, and data storage) can be defined.

An  assessment of the utility's existing capabilities relative to the design objectives of the contamination
warning system and the attributes of the specific component of the system is essential for successful
implementation.  From an IT system engineering perspective, primary objectives for design and
implementation include the following:
    •   Ability to leverage and integrate with existing utility IT systems  and policies
    •   Mechanisms for timely integration and analysis of contamination warning system data and the
       ability to make data available to support timely decision-making
    •   Approaches and mechanisms that can be readily adapted to meet changing needs and priorities

The basic objectives described above can be used to define the component-specific requirements for data
management. Other considerations include the following:
    •   Electronic data management  for all components.  All information collected as part of the
       contamination warning system  should be managed in a database. For each component,
       information should be tracked as the system transitions from routine operation to event detection,
       credibility determination, consequence management, and finally return to normal operations. In
       order to avoid  duplication of effort, information should be accessible across multiple databases
       and systems.
    •   Data storage.  The data collected should be stored in a reliable data store that has sufficient
       capacity and performance to support routine operations, contamination situations, and system
       evaluation. Historical data should be maintained at appropriate locations to enable engineers and
       scientists to study past normal and event situations to evaluate and optimize system performance.
    •   Automated and integrated data analysis. As data are captured in electronic format in a
       consistent environment, algorithms operate on the data to integrate information and facilitate data
       analysis. Automated anomaly detection algorithms indicate when the data may be indicative of a
       contamination incident, signaling the need for human involvement in the assessment process.

May 2007                                                                                  10

-------
                                Planning for WS-CWS Deployment

       The type and level of sophistication of algorithms will likely vary by component, from
       sophisticated algorithms for the analysis of online water quality and public health surveillance
       data to more simplistic algorithms for consumer complaints. The use of tools for spatial and
       temporal analysis of the indicators, such as Geographic Information Systems (GIS), should also
       be considered.
    •  User  interface for operation and analysis. Although some contamination warning system
       operations are automated, human operators and analysts will assess situations and make many of
       the decisions necessary to manage a potential contamination incident. Thus, there is a need to
       develop or utilize existing user interfaces to support data analysis and decision making. If a
       utility's existing GIS is sufficiently robust and integrated, it can serve as an effective means for
       display of information from any or all components of the contamination warning system.
    •  Data exchange capability for transmission of information within the utility and between the
       utility and contamination warning system partners.  The data collected at the utility may need
       to be  shared with other local government agencies such as public health departments.
    •  Availability and redundancy of systems.  Information systems should be designed to have
       system availability that is consistent with the significant importance of the contamination warning
       system function.  Backup and recovery plans and procedures should be defined and implemented.
    •  Security, authorizations, and controls. Access to contamination warning system data and
       applications should be restricted to authorized personnel as determined by the utility.  In addition,
       data encryption should be considered for information exchanged between the utility and other
       partners when it passes over public networks.
    •  Change management  and maintenance. Throughout the life-cycle of IT software and hardware
       it will be necessary to implement upgrades and perform routine maintenance. From a planning
       perspective this should be considered both in terms of costs and the approach for implementing
       upgrades and performing maintenance.

2.2    Utility Staffing
Approaches for staffing a contamination warning system will vary by utility, but should work within
existing organizational structures and routine job functions to the extent possible.  As discussed in Section
2.1, application of system engineering principles plays a critical role in successful  implementation of a
contamination warning system. This applies to staffing as well - planning, design, implementation,
operation, and ultimately evaluation will rely on effective communication and coordination across
divisions or departments within the utility. To the extent possible, generic titles and roles are provided to
organize the discussion in this section.  Based on the description of activities, utilities should identify
individuals and departments within their own organization to fulfill these roles as appropriate,
independent of job title.

2.2.1  Project Management Team

While routine operation and maintenance of the contamination warning system should generally fall
within the routine job functions of utility staff, design and implementation will involve significant time
and effort from dedicated managers within the utility. Depending on utility organization and operational
paradigm, the activities described below may be managed by one individual, or more likely, a core project
management team. In addition, support from the utility director and/or board of directors, as well as all
senior supervisors and managers is critical to contamination warning system implementation.  Project
management activities to be addressed by the management team generally include  the following:
    •  Development of design goals and objectives
    •  Communication of the  goals and objectives of the contamination warning system to all levels of
       management, staff, and external partners
    •  Prioritization of work activities and allocation of resources.
    •  Networking and establishment of agreements with external partners
May 2007                                                                                   11

-------
                                Planning for WS-CWS Deployment

    •  Coordination of procurement, installation, and inspection of equipment, hardware, and software,
       including licenses and maintenance agreements
    •  Coordination of teams supporting design and implementation of monitoring and surveillance
       components and consequence management plan
    •  Verification of reporting protocols and coordination with primacy agency
    •  Development of an overarching framework describing the project phases, goals, objectives,
       schedule, and milestones
    •  Coordination of IT and data management aspects of design, implementation, and operation across
       all aspects of the contamination warning system to ensure consistency with existing protocols and
       procedures and interoperability of systems
    •  Integration of contamination warning system concept of operations with existing plans and
       protocols that govern routine operations and activities within the utility
    •  Integration of contamination warning system consequence management plan with existing
       response protocols, plans, and procedures
    •  Development of evaluation plan and identification of metrics for evaluation
    •  Coordinate identification and documentation of lessons learned

2.2.2  Utility Staff

While the primary focus of a contamination warning system is monitoring water quality in the distribution
system, the design and implementation of a contamination warning system involves the collaboration of
the utility as a whole; however, each department has its own capabilities to contribute.  The information
presented below is organized by general divisions or departments; actual departments and roles may vary
by utility. Component-specific activities are described in "Staffing and Costing Considerations"
subsections within Section 4 - Section 9.

Security/Risk
    •  Evaluate physical vulnerabilities of facilities
    •  Design security enhancements to facilities
    •  Provide oversight during installation of security enhancements
    •  Coordinate planning for and response to security breaches with law enforcement
    •  Manage and coordinate response to security incidents utility-wide
    •  Coordinate training and functional drills related to consequence management
    •  Determine if security breaches provided an opportunity to contaminate water

Water Quality
    •  Evaluate vulnerabilities of distribution system
    •  Select water quality parameters and sensor hardware
    •  Design/create monitoring network (placement of monitors around system)
    •  Design and oversee the conduct of tracer studies if needed
    •  Coordinate installation of monitoring stations
    •  Select target analytes and lab methods for baseline and triggered sampling
    •  Identify sampling locations and frequency for baseline sampling program
    •  Develop and implement sampling protocols
    •  Identify requirements for integration of data into event detection system (EDS)  for water quality
       monitoring and consumer complaint surveillance component
    •  Coordinate laboratory capabilities with local laboratory network
    •  Maintain proficiency for any in-house lab and field analyses
    •  Establish water quality baselines for comparison with abnormal water quality results
    •  Develop site characterization procedures, including selection of field test equipment
    •  Coordinate with public health on surveillance activities
    •  Coordinate training on operation of all contamination monitoring system components
May 2007                                                                                   12

-------
                                Planning for WS-CWS Deployment
    •  Investigate water quality and distribution issues in reference to observed triggers
    •  Evaluate remediation options

Operations and/or Distribution
    •  Coordinate with Supervisory Control and Data Acquisition (SCADA) or other communication
       systems
    •  Support implementation of distribution system tracer studies, if conducted
    •  Investigate distribution system operations and maintenance in reference to observed triggers
    •  Support field sampling activities
    •  Plan and implement isolation and containment options
    •  Support evaluation and implementation of remediation options

Information Technology
    •  Identify data management, hardware, and software needs for utility
    •  Design and implement changes
    •  Coordinate the flow of all of the information/data streams

Engineering and Planning
    •  Design and construct water quality monitors
    •  Design and construct security improvements to facilities
    •  Provide input for monitoring network design
    •  Perform hydraulic and water quality modeling
    •  Design remediation activities in event of confirmed contamination

Public/Customer Affairs
    •  Identify improvements to call center to recognize and track water quality related calls
    •  Implement hardware, software, and training changes
    •  Interface with outside organizations for public health surveillance
    •  Identify appropriate customer call information streams and how to integrate into Event Detection
       System
    •  Provide public outreach
    •  Establish baseline levels for water-quality related customer complaints
    •  Public notification of contamination and appropriate actions

Administration
    •  Financial tracking
    •  Project tracking
    •  Procurement
    •  Legal review of agreements, documents, etc.
    •  Management of contracts, grants, agreements, etc.

2.3    Local Partners
Designing, implementing, and ultimately operating a contamination warning system is a complex task
that relies on the coordination and cooperation of many partners. In addition, the impacts of a drinking
water contamination event are not isolated within a utility, so local partners should be engaged in the
complex task of responding to a contamination event, intentional or accidental, that can involve criminal
activity, public health impacts, regulatory compliance, and hazardous materials response. Because it is
necessary for the utility to rely on the assistance of local partners to operate the system, those partners
should be involved as appropriate in the design and implementation of the system.  As part of this
process, the utility should consider what formal agreements can be reached with other agencies, which
can vary depending on whether the utility is publicly or privately owned.  Furthermore, if publicly owned,

May 2007                                                                                   13

-------
                                Planning for WS-CWS Deployment

nuances of being an independent public agency, a single municipal department, or part of a public works
department should be considered.

2.3.1  Identifying and Engaging Partners

The utility is the operational hub of the system as the primary operator of the majority of monitoring and
surveillance components of the contamination warning system, with the exception of public health
surveillance. However, other partners may be involved in trigger validation and/or consequence
management activities. Figure 2-1 provides an overview of potential partners in contamination warning
system deployment.
Federal Bureau of
Investigation
State Emergency
Management
and
Homeland Security
Agencies
State Law
Enforcement
Stat
Wat
Centers For Disease EPA Regional Offices
Control and Prevention
Local Health Local Wastewater Local Law
Department Utility Enforcement
Local Fire, EMS, / Water \ Local Civil
and HazMat f Utilitv Government
.... ... Public Health and
Local Emergency Host ,. . . .
„. . _ ... _ ..... Environmental
Planning Committees Facilities , . . .
Laboratories
e Drinking and Waste Neighboring Utilities
er Primacy Agencies
EPA Criminal
Investigation Division
EPA National Response
Center
State Emergency
Res ponders
State Government
Media
Figure 2-1.  Potential Contamination Warning System Partners

During the early stages of the investigation of and response to a "possible" contamination incident, a
utility will likely rely on local partners for assistance, and as the credibility that a contamination incident
has occurred increases, the number of partners will increase.  As illustrated in Figure 2-1, the number and
scope of partners that can become involved in responding to a contamination event can be significant.  In
planning for contamination warning system deployment, drinking water utilities should identify and
engage local partners early in the process, particularly those partners such as local health departments and
public health and environmental laboratories that will have a significant role in routine operations.
Specific responsibilities of partners and when they are engaged will vary by utility and jurisdiction.
However, Table 2-1 provides a summary of possible contamination warning system partners and their
possible role in design, implementation, operation, and/or response.

Table 2-1. Summary of Potential Contamination Warning System Partners
Partner Organization
Host facilities
Local health department
Roles and Responsibilities
Provide facilities for placement of water quality and/or enhanced security monitoring
stations, including 24/7 access to utility staff for maintenance, trigger validation, and
response activities.
Monitor health of the population through public health surveillance as part of routine
operation of the system and may have some degree of analytical capability to
support sampling and analysis. Provide support during consequence management
including consultation and public notification. Serve as conduit to state and Federal
health departments and agencies.
May 2007
14

-------
                            Planning for WS-CWS Deployment
Partner Organization
Local law enforcement
Local civil government
Local emergency
planning committees and
emergency management
agencies
Local fire, EMS, and
Hazmat
Environmental and public
health laboratories
Local wastewater utility
Neighboring utilities
Media
State government
State emergency
responders
State drinking water and
wastewater primary
agencies
State emergency
management and
homeland security
agencies
State law enforcement
EPA Regional offices
and/or laboratories
Federal Bureau of
Investigation (FBI)
Centers for Disease
Control and Prevention
(CDC)
Roles and Responsibilities
May assist in routine operation of enhanced security monitoring; provide support
during consequence management through credibility determination and response.
May also serve as conduit to state and national law enforcement and intelligence
agencies.
Should be engaged early in the planning for implementation. In the event a utility's
service and/or wholesale areas span multiple jurisdictional entities, it may be
necessary to engage in formal agreements among different local governments and
their respective agencies to ensure cooperation and support and allocation of
funding. Also, should an event occur, the elected officials of different jurisdictions
should be appropriately informed of the state of the situation so that they can
effectively communicate with their constituencies.
Primarily support consequence management activities as a conduit to other
response agencies at the state and Federal level. Can support provision of alternate
water supplies, coordination, disaster declaration, and transition to National
Response Plan implementation.
Local fire and EMS organizations may have a role in routine operations for public
health surveillance as a provider of 91 1 and/or EMS data. These organizations, as
well as local Hazmat play a critical role in consequence management including site
characterization activities to support credibility determination.
Provide support to routine sampling and analysis activities to establish baseline and
maintain analytical proficiency. In addition, provide analytical support during
consequence management to assist in credibility determination as well as response
and remediation efforts. State public health laboratories provide access to CDC's
Laboratory Response Network.
May provide analytical support for routine sampling and analysis. Should be
consulted in the development and implementation of consequence management
plans due to the potential impact of contamination on wastewater operations.
May provide support in the event of a contamination incident through mutual aide,
assisting with provision of alternate water supplies, remediation, and recovery
activities.
Local media organizations may serve as a valuable resource in communicating
messages to the public in the event a contamination incident occurs.
May provide support to implementation activities in terms of gaining cooperation
from State organizations. May have a role in establishing formal agreements with
State partners or coordinating funding resources. For consequence management,
should be informed and engaged once contamination has been confirmed to assist
in coordination of resources and communication.
Provide support to consequence management phases if a contamination incident is
confirmed. Should be engaged in consequence management planning to ensure
efficient transition in the event a contamination incident escalates.
Primacy agencies can be public health agencies as well as separate state or local
environmental agencies, like state or regional water quality boards. If contamination
does occur, there may be regulatory ramifications related to use of contaminated
water, public notification, environmental concerns for discharged water, quality of
alternative supplies, and more. Additionally, the primacy agency, along with EPA,
should be consulted on any potential remediation and recovery plan.
Provide support to consequence management phases if a contamination incident is
confirmed. Should be engaged in consequence management planning to ensure
efficient transition in the event a contamination incident escalates.
Provide support to consequence management phases if a contamination incident is
confirmed. Should be engaged in consequence management planning to ensure
efficient transition in the event a contamination incident escalates.
May assist in coordination of Federal resources and may also assist by providing
analytical surge capacity during phases of consequence management.
May assist in site characterization and/or consequence management plan
development. Establishing a relationship with local FBI agents early in the
implementation process is critical to establish and understand roles and
responsibilities in the event contamination occurs.
Provide oversight to the Laboratory Response Network, a network of public health
laboratories with the ability to analyze for select agents based on established
analytical protocols. Ensure member laboratories have appropriate training,
equipment, reagents, and resources. Provide technical consultation during
credibility determination and other phases of consequence management.
May 2007
15

-------
                                Planning for WS-CWS Deployment
Partner Organization
EPA Criminal
Investigation Division
(CID)
EPA National Response
Center
Roles and Responsibilities
Provide support to consequence management phases if a contamination incident is
confirmed. Should be engaged in consequence management planning to ensure
efficient transition in the event a contamination incident escalates.
Provide support to consequence management phases if a contamination incident is
confirmed. Should be engaged in consequence management planning to ensure
efficient transition in the event a contamination incident escalates.
A recommended strategy for engaging partners is to first consider those partners who will have a role in
routine operations of the system or will be involved as "first responders" based on the consequence
management plan.  Engaging the numerous partners involved in establishing a contamination warning
system is a daunting challenge on its own, without the myriad other tasks the utility implementation team
will be occupied with. It is not uncommon for the service area of a utility to span city limits and county
borders, and into the jurisdictions of numerous police, fire, and public health agencies, not to mention the
umbrella jurisdictions of hierarchical agencies like county and state emergency management, public
health, and homeland security agencies, to name just a few. Therefore, it is essential to take full
advantage of the existing groups and organizations in which these partners may already participate.
Figure 2-2 illustrates a recommended approach for engaging partners.
          Engagement to form CWS
           Implementation Team
                                                    Direct engagement
                                                    with key county,
                                                    state, and federal
                                                       partners
Regional Homeland
Security Programs
     &
  Existing Local
 Partner Groups
 Engagement
through existing
partner groups
                                                       Direct
                                                    engagement with
                                                      extended
                                                      partners
Figure 2-2.  Recommended Strategy for Engaging Contamination Warning System Partners

Depending on the role or type of support contamination warning system partners play in implementation,
it may be necessary or desirable to establish formal agreements. Considerations for establishing these
agreements are discussed in Section 2.3.2.

2.3.2   Considerations for Formal Agreements with Local Partners

Inter-agency agreements, memorandums of understanding, memorandums of agreement, mutual aid
agreements, and other agreements are becoming common practice in most jurisdictions.  The documents
contain  language that is mutually agreed upon by all stakeholder agencies and generally define
collaborative efforts that involve action items, equipment resources, or regional governance. When
engaging local, county, state and federal  partners in implementation activities, the utility should address
the subject of inter-agency agreements early in the process.  Addressing formal  agreements early in the
implementation process is extremely important, as they commit partner agencies to specific roles and
actions. Without them, implementation can be stalled by inter-agency disagreements or
misunderstandings, or an agency may be left responsible for costs they believed would be covered by
another.
May 2007
                                           16

-------
                                Planning for WS-CWS Deployment

The utility should first identify its own protocols for establishing formal agreements with external
agencies, organizations, and partners.  This includes identifying who holds the authority to enter the
utility into these types agreements (who signs the document), any procedural details, like minimum or
maximum review periods, paperwork routing procedures, restrictions on the types of agencies or groups
the utility may enter into agreements with (public and private), or limits of commitment (monetary or
other). The utility should also develop a clear understanding of the same types of information from the
agencies it intends to engage.  Subjects of the agreements extend beyond simply who pays for equipment;
commitments should be made to provide man-power both for the implementation and operation of the
contamination warning system; allocation of resources; etc.  If funding is from an external source all
applicable standards and regulations for establishing formal agreements should be followed.

2.4     Costs
Cost of contamination warning system implementation can vary widely from utility to utility, based on a
variety of factors, and perhaps most significantly based on existing capabilities across components.
Factors to consider in estimating costs for specific components are embedded throughout Section 4 -
Section 9. From a project planning perspective, considerations should be given to long-term issues
including operation and maintenance and sustainability prior to finalizing a design. Contamination
warning system deployment should be considered a significant program or initiative that will involve
long-term involvement and large investments by the utility.  Substantial equipment and construction may
be necessary, as well as additional maintenance needs. These maintenance needs should be tracked on a
system-wide basis and the labor and equipment expenditures should be  included in budgets for the life of
the system. All costs should be monitored and compared to budgetary constraints. Additionally, this
project may involve sensitive information which should be tracked, and personnel who use or handle the
information will have to be screened.  This places  an additional responsibility on the administrative
personnel to adequately document and supervise information security issues. Considerations for
information security are discussed in Appendix B.

The potential cost of contamination warning system deployment should include both the tangible
(equipment, installation, etc.) as well as the intangible (staff perception, morale, motivation, etc.)
elements.  The tangible costs should be estimated using a life-cycle approach to capture the management
and coordination effort, the capital costs of the equipment, initial training, startup, testing, and calibration,
and the long term operations and maintenance costs (training, calibration, spare equipment and
components, maintenance contracts, chemical testing supplies, monthly communications fees, etc.). The
intangible costs should be identified and recognized and a management plan developed to address any
issues. Examples of intangible costs could include a staff person's perception that the operation of the
system adds more work to his/her day with no additional compensation  or the fear that the system will be
advertised to the public as enhancing safety and security while not actually delivering on the promise.

In addition to component-specific cost factors discussed throughout the  remaining sections of this
document, the following factors should be considered as part of project  management and system
engineering costs:
    •   Project Management and Coordination. Development of agreements, schedule, work plan,
        communication products; administration and financial tracking; routine coordination and strategy
        meetings.
    •   Development of Integrated Concept of Operations. Coordination of concept of operations
        development; analysis of component-specific concept of operations for consistency;  development
        of integrated concept of operations documentation and procedures.
    •   IT System Engineering. Assessment of existing systems; procurement and installation of
        hardware and software; implementation, operation, and maintenance of systems.
    •   Evaluation and Refinement. Development and implementation of evaluation plan;
        identification and documentation of lessons learned; identification of refinements to optimize
        system performance.

May 2007                                                                                  17

-------
                              Planning for WS-CWS Deployment


       Section 3.0:  Design and Implementation  Framework

Deployment of a contamination warning system should follow the typical programmatic approach in
which proposed enhancements are planned, designed, implemented, tested, maintained and refined.  This
section provides a comprehensive framework for design and implementation. Alternate approaches for
design and implementation can be considered, however the concepts described below should be
addressed. Table 3-1 summarizes a recommended approach for contamination warning system design
and implementation based on lessons learned from the initial pilot.  While there may be some deviations
in terms of how each stage is applied for a given component, all components should address planning and
pre-design, design, implementation, preliminary testing, operation and maintenance, and evaluation and
refinement.

Table 3-1. Overview of Design and Implementation Framework
Stage of Approach
Planning and pre-
design
Design
Implementation
Preliminary testing
Operation and
maintenance
Evaluation and
refinement
Description
Developing a core implementation team, defining design objectives to guide
implementation, and a preliminary assessment of existing capabilities relative to design
objectives.
Development of a preliminary concept of operations and development of a detailed work
plan and schedule to guide implementation.
Implementation of enhancements, installation of equipment, and training according to the
plan.
Operation of the contamination warning system for the purpose of collecting data
necessary to understand system performance and finalization of the concept of operations
to optimize system.
Operation of the contamination warning system for the purpose of monitoring for
contamination incidents and other water quality issues.
Analysis of data and information generated during full operation to refine and optimize the
system.
Additional detail on the application of this framework to contamination warning system monitoring and
surveillance components and consequence management is presented in Section 4 - Section 9.

3.1    Planning and Pre-design
The initial steps in developing a contamination warning system are included in the planning and pre-
design stage. The utility should develop a team to support design and implementation and define what it
wants the system or component to do.

3.1.1  Building the Team

As emphasized in earlier sections of the document, deployment of a contamination warning system
involves an integrated team within the utility that also extends to external partners for certain aspects of
monitoring, surveillance, and response. It is important that the implementation team apply the system
engineering principles discussed in Section 2.1 throughout all phases of contamination warning system
implementation. Utility departments and divisions should work together as an integrated team to leverage
existing infrastructure, systems, protocols, and procedures to ensure effective operation of the system.

3.1.2  Defining the Utility-Specific Design Basis and Design Objectives

As introduced in Section 1.1, the contamination warning system should be able to meet the following
design objectives:
    •   Detection of a broad spectrum of contaminant classes
    •   Spatial coverage of the entire distribution system
    •   Detection of contamination in sufficient time for effective response

May 2007                                                                               18

-------
                                  Planning for WS-CWS Deployment

    •   Reliable indication of a contamination incident with a minimum number of false-positives
    •   A sustainable architecture to monitor distribution system water quality

These concepts should be considered during each step of the pre-design process, for the components and
the system as a whole.  Failing to consider each design objective may result in the design of
contamination warning system that accomplishes some of its goals, but as a collective system fails to
meet its ultimate objective. Table 3-2 summarizes overarching design basis considerations for the
contamination warning system.

Table 3-2. Design Basis Considerations	
    Design
  Objectives
              Description
   Design and Implementation Considerations
Capability
Ability to detect contamination of the
distribution system through contamination
warning system components.
Most components provide an indirect measure of
contamination; thus it is necessary to have a
process for validation of triggers and coordination
across components.	
Contaminant
Coverage
Contaminant classes that can be detected
by the system; actual contaminants may
vary from system to system depending on
the manner in which components are
implemented.
May be influenced by several factors for each
component such as type of public health
surveillance or disinfectant residual. Design
objectives should target coverage of as many
contaminant classes as possible.  Expanding to
additional contaminant classes could be an objective
to consider as part of system evaluation.	
Spatial
Coverage
Amount of distribution system covered by
one or more of the contamination warning
system components.
May vary by component pending jurisdictions,
number of sensors, sampling routes, security sites,
etc. While the integrated contamination warning
system should cover the entire distribution system,
the degree of coverage by each component will
vary. An  aspect of system engineering should  be to
maximize coverage to the extent possible across all
components in a way that optimizes protection
through the entire system.	
Timeliness
Function of ability to detect anomalies and
conduct initial trigger validation to
determine  possible contamination.
Development of procedures for routine operation of
the system and initial trigger validation will inform
design and operation of the system and ultimately
impact timeliness.	
Reliability
At the system level, reliability is
characterized in terms of the rate of false
alarms and occurrence of undetected
contamination.
Reliability of the system is influenced by design
decisions made at the component level. However,
system reliability can be improved by integration of
information and coordination across components to
maximize confidence in a system alarm.	
Sustainability
Ability to provide drinking water utility with
an understanding of the distribution system
in terms of water quality and variability and
information to optimize distribution system
operation.
Contamination warning system activities and
procedures should be designed for incorporation into
routine job functions to maintain the system and
support dual use application. Security should not be
the only consideration in developing design
objectives.	
In the planning and pre-design stages of contamination warning system deployment, drinking water
utilities should evaluate these design objectives relative to their specific needs and objectives and
customize and adapt as appropriate. It is important to consider objectives beyond the security aspects of
contamination warning systems, particularly dual use applications that could help to promote
Sustainability of the system by optimizing utility operations.  Potential dual-use benefits of a
contamination warning system could include the following:
    •   Detection of cross-connections and other distribution system water quality problems
    •   Improved relationship with public health organizations, including mutual sharing of information
        and alerts
    •   Enhanced knowledge of distribution system water quality leading to improved operations (e.g.,
        more consistent disinfection residual levels, improved corrosion control, early warning of
        nitrification episodes, reduced disinfection byproduct levels, etc.)
May 2007
                                                                                  19

-------
                                Planning for WS-CWS Deployment

    •  Identification of problem valves (closed, partially closed, inoperable)
    •  Improved coordination with local, state, and federal response organizations
    •  Reduced occurrence of tampering and vandalism
    •  Improved information technology systems and interoperability
    •  Improved consumer complaint tracking and response
    •  Improved laboratory capability and an established laboratory network
    •  Consequence management plans  applicable to any water quality emergency

3.1.3  Preliminary Assessment and Gap Analysis

Understanding the starting point and existing resources that will form the foundation for the system is
critical in the early stages of planning and pre-design. Before progress towards the desired end-state of
the contamination warning system can be made or measured, a utility should fully understand the
capabilities of their existing systems, procedures, and other resources.

The recommended approach for conducting a thorough analysis of the utility's existing capabilities is to
use personnel with varying degrees of experience with the systems being evaluated.  Experience at the
pilot utility demonstrated that including staff who do not routinely use a particular system can provide
additional insight into what is a routine process for an experienced user. This can be accomplished by
using a mix of utility personnel and/or outside consultants. The intent is to gain a complete understanding
of the current state of the utility, and the benefit of using a diverse team is that it results  in a more realistic
plan to reach the desired capabilities.

At the conclusion of this self-assessment, the utility's design and implementation team will have
developed a solid understanding of their existing systems. Using the preliminary concept of operations as
a benchmark, the utility can examine its existing operations by process or by component and measure the
gap between current and desired capabilities. The result is a gap analysis that clearly outlines the progress
that a utility should make to achieve the design objectives.  For example, the analysis of the customer call
center at the pilot utility discovered that the staff was already collecting very useful complaint data, and
an excellent complaint management system was already in place.  Only minor modifications that had very
minimal impact on existing staff and processes were necessary to transition the existing system into a one
that would feed the critical water quality  complaints into the system. This gap analysis may be  further
refined following development of a preliminary concept of operations as discussed in Section 3.2.

3.2    Design
The design stage of the programmatic approach for a contamination warning system encompasses the
development of the plans and specifications for each component and a consequence management plan.  It
is critical that the performance objectives of the design basis are considered throughout this process, and
the design stage is no exception.  At the system level, design should go beyond information integration
and event detection.  It should also consider how all resources - staff,  IT, communications, equipment,
etc. can be leveraged across the entire project. Further, any resources  at the disposal of the utility and its
partners,  including but not limited to staff, communications, equipment, and training opportunities, should
be evaluated for possible use within the project. This will not only result in an efficient use of time and
materials, but will also highlight the dual-use benefits possible through implementation  of the
contamination warning system  (e.g., radio communication equipment purchased for the  use of equipment
maintenance becomes more valuable when it is also used as part of the consequence management plan).

3.2.1  Conceptualize System

Before beginning physical design, consider a contamination warning system at a conceptual level, and the
manner in which this conceptual model translates to physical systems and implementation. Firm
understanding of the basis for design decisions should inform all aspects of the design process.  This

May 2007                                                                                   20

-------
                                Planning for WS-CWS Deployment

understanding can be developed, in part, though the development of a preliminary concept of operations
for the entire contamination warning system.

At this early stage of planning, many of the details of routine operation are unknown. However, a
preliminary concept of operations can still be developed at this stage to identify the general capabilities
that the fully implemented system should possess and how that capability relates to the existing resources
and capabilities within the organization. The preliminary concept of operations will also establish
potential roles and responsibilities and user requirements for IT-related systems. In doing so, this
document outlines the characteristics of the finished system and provides the framework for the design
and implementation of the individual components. Its development is a critical step towards successful
implementation of a contamination warning system because it provides an initial benchmark against
which to measure a utility's existing capabilities and a means by which to plan for enhancements. Note
that additional details will be added to the concept of operations as the contamination warning system is
designed and implemented, including a very detailed concept of operations for each component. Thus
development of the concept of operations should be viewed as an iterative part of the design and
implementation process, with the preliminary concept of operations serving as a starting point for the
design process.

3.2.2   Work Plan for Implementation

Prior to initiating implementation activities for any component, a detailed implementation plan, or work
plan, should be developed that clearly identifies priorities, schedule, milestones, and resources.  It may be
necessary to revisit the assessment and gap analysis in order to prioritize some of the identified
enhancements to maximize time and resources. This should be considered at the system level, in advance
of implementation using a system engineering approach as discussed in  Section 2.1.

3.3    Implementation
Implementation of a contamination warning system begins with consensus on the approach identified in
the component-specific work plan and completion of any associated design work (i.e., hardened access
points as part of the physical  security enhancements). This stage can involve significant coordination
with outside consultants and contractors, depending on the capabilities and availabilities of in-house staff.
Again, since a contamination warning system relies heavily on effective data management, the
involvement of the information technology staff is critical. Each component-specific work plan
developed during the design stage should be implemented concurrently, to the extent possible. The
timelines should be carefully monitored to ensure delays do not create problems in other components'
deployments. In addition, any training specified in the  component-specific work plan should be
conducted during the implementation stage. During this stage of the project, enhancements are
implemented and installed, concept of operations and consequence management plans are reviewed,
revised, and reconciled, and training for routine operation, maintenance, and consequence management
should be conducted.

3.4    Preliminary Testing
Once components begin to come on line, the contamination warning system transitions to the preliminary
testing stage of implementation. During this period, enhancements are in place and the  system is
technically operational, however it is being operated in a "test" mode. Meanwhile, the concept of
operations and consequence management plan can be finalized, taking into consideration the additional
information and insights gained during the design and implementation stages, and reflecting the "as-built"
system.
May 2007                                                                                  21

-------
                                 Planning for WS-CWS Deployment

3.4.1   Baseline Operation

The automated analytical capabilities of a contamination warning system depend on having a historical
baseline or base state against which to match current operational conditions. Anomalies from the baseline
create a trigger, which is then analyzed and validated, or dismissed, as a possible contamination event.
Depending on the seasonal variability in the operating conditions for a specific water utility, the
preliminary testing stage could last up to a year, or even longer.  If the utility utilizes multiple water
sources, blends sources at varying ratios, or uses an alternative source exclusively during different
portions of the year, operations under these conditions should be experienced by the system to gain a
complete picture of the normal variability in water quality parameters. Similarly, changes in source water
quality, such as episodic taste and odor events, and operational changes, such as periodic changes in
distribution system residual, should be incorporated into the base state to the extent possible. The more
"normal" conditions that the system can be exposed to during the preliminary testing stage, the more
detailed the baseline will be and the more successful the contamination warning system will  be in meeting
is design objectives, particularly in regards to its accuracy and reliability.

During this period of preliminary testing, the system is operating at full capability. All field equipment
are installed, communication and IT enhancements are in place, data streams are being transmitted to the
utility and personnel have been trained to collect the information pertinent to the contamination warning
system. However, the system is being operated for the purpose of collecting data necessary to understand
system performance. This does not include responding to the alarms generated by the system, except in
the capacity of testing procedures implemented in response to a trigger.  Because a baseline has yet to be
established, it is difficult to identify whether an alarm is indicative of a water quality problem, or merely
an aspect of the base state that had not been previously observed by the  system.

This mode of operation can place the  utility in a vulnerable position. Information indicating possible
contamination events will be received by the utility, likely more frequently than before, without knowing
the reliability of this information.  It is critical, therefore, that the utility remains vigilant in its use of
existing, established procedures of possible water quality problems or security concerns during this stage.

An example at the pilot utility involved  its consumer complaint surveillance component.  Prior to
implementation of their system, the pilot utility had a threshold value for the number of water quality
complaints that were manually logged in a certain database within a set time period. If that number was
exceeded, utility personnel began certain protocols to investigate a potential contamination event.  During
baseline operations, the pilot utility began identifying that a larger number of their consumer calls related
to water quality in some way. However, until a baseline value for this new indicator of possible
contamination events was developed, response and remediation steps were not initiated for each call or
even when the old threshold value was reached.  Instead, the utility continued to monitor the number of
complaints that were of the same severity as those previously entered into their particular database
(although now that process was more automated). When the historically validated threshold was reached,
the utility initiated its established response protocol.

3.4.2   Finalization of Concept of Operations and Consequence Management Plan

As discussed earlier, the concept of operations is the description of the routine operation and initial trigger
validation for each component of the  contamination warning system.  While the terminology and
approach for this documentation may vary, it is important that the operational concepts be clearly
documented and that utility staff and local partners understand their roles and responsibilities in operating
the system.

At this stage, the system-wide concept of operations should be revised to reflect information and insights
gained through design and implementation.  The concept of operations should now include details of the
capabilities of the components, the data streams that will be mined and the IT system infrastructure to
collect and analyze the results. Roles and responsibilities for specific job descriptions or critical

May 2007                                                                                    22

-------
                                Planning for WS-CWS Deployment

personnel should be defined and job-specific checklists developed to facilitate the performance of new
tasks and new processes implemented as a part of system enhancements.  It is critical that the
development of the component concept of operations are coordinated to ensure that resources are applied
in a consistent and compatible manner across components, and integrated to ensure that data from
multiple components can be integrated to better inform response decisions.

Recommendations regarding the development of the consequence management plan, and the steps to
finalize this document are covered in separate  guidance published by the EPA.  However, completion of
the Consequence Management Plan should be informed by final Concept of Operations to ensure the
former is capable of integrating information from the multiple components and affecting a response.

3.5    Operation and Maintenance
This stage represents the remaining life of the  system. During this period the utility should be prepared to
maintain the components to continue to meet the design basis for the life of the system. This stage should
not begin until a baseline for the system is established and the consequence management plan is in place.

3.5.1  Operation

With the establishment of a robust baseline and the completion of the final consequence management
plan, the contamination warning system can be placed into full operation. At this stage the system is
operating at full capacity and actively monitoring the water distribution system for contamination
incidents and other water quality problems.  Each component is feeding its data via established
communication protocols into the  event detection system(s) at the utility for analysis. When component-
specific threshold values are reached, triggers  are initiated to indicate a departure from the baseline. If
that trigger can be validated by the processes and procedures detailed in the concept of operations, a water
contamination event is deemed possible and the Consequence  Management Plan is implemented to affect
a response. If the event is further  confirmed, the Consequence Management Plan guides the remainder of
the response and remediation actions.  If the components have been well designed and the system as a
whole been automated, the day-to-day operations do not look significantly different for the majority of
utility personnel involved.

3.5.2  Maintenance

Maintenance of the components of the contamination warning system is the most influential determinant
of the long-term success of the contamination  warning system in meeting its design basis. Maintenance,
in the context of a contamination warning system, refers to the activities to maintain the intended
capabilities of the system.  This includes the physical maintenance of equipment, upgrades of software,
and the continual training of personnel. For example, "orphaned" water quality monitors are of little use
in a contamination incident and it  is vital that personnel responsible for their upkeep and calibration
understand their importance.  Specifics regarding the maintenance of particular equipment and elements
of each component are included in Section 4 - Section 9. However, from a system-wide perspective,  if
the component designs were completed with an awareness of the need for sustainability, then they would
ideally specify equipment, procedures or other enhancements that have dual-use applications. The use of
the security enhancements for routine operations of a water utility will ensure that the contamination
warning capabilities are fully functional in the remote chance an event were to occur.

3.6    Evaluation and Refinement
Evaluation and refinement should be  considered  in planning for contamination warning system design
and implementation. Time and resources should be built into the schedule for carrying out evaluation
activities and implementing refinements to optimize system performance.
May 2007                                                                                  23

-------
                                 Planning for WS-CWS Deployment

3.6.1   Evaluation

The primary function of contamination warning system evaluation is to gauge the effectiveness,
reliability, usability, and sustainability of the system; adjust and streamline the system and approach; and
adapt to and incorporate advances in technologies, methods, and protocols that occur over time.
        Operation. Refers, in general, to all aspects and degrees of functionality, usability and utility.
        Ideally, the chosen equipment and technologies will not require an increase in skill level or
        manpower to operate and maintain, and will be robust enough to work under real-world
        conditions. More likely, however, a balance should be struck between the ideal and what is
        feasible, affordable, and available.
    •   Performance.  Measures of performance are equally  important to those of operation.
        Performance of the tools, components, and system refers, in general, to their ability to
        consistently provide accurate data in a timely manner consistent with intentions of the system
        design. A system that is  easy to operate will not be useful unless  it performs as intended to meet
        the overall system goals. To evaluate all system aspects fully, the performance  should be
        measured in several different ways including the range of contaminants and contaminant classes
        that may be detected; the accuracy of the data produced; the ability to discern whether a data
        anomaly is indicative of contamination or caused by something benign and the ability to reliably
        detect an actual contamination event. Timeliness is an important measure of performance. A
        system that can detect contamination consistently but requires long periods for detection is of
        limited use. All aspects of the timeliness of a component's data collection, analysis and event
        response should be considered.
    •   Sustainability. Sustainability will attempt to measure the likelihood that the contamination
        warning system will become viewed as a sufficiently  viable, valuable, cost-effective system
        resulting in relatively wide spread adoption by the drinking water utility industry. Utilities will
        view the system as sustainable if it provides benefits that are worth the system life cycle costs.
        For the benefits to be seen as sufficient, for all except a few utilities it is recognized that the
        contamination warning system should provide benefits other than warning of intentional
        contamination. Intentional contamination, although it would have high adverse consequences is
        viewed as a low probability event by most utilities. Therefore, part of the technical evaluation
        will focus on identifying and evaluating the level and degree of these dual-use benefits. Life
        cycle costs include costs  to design, install, maintain, and operate the system. These costs will
        include funding for engineering and utility staff labor, equipment, consumables and spare parts.
        Evaluating the  level and degree of these dual-use benefits will be the second key part of
        evaluating sustainability.

In general, methods for contamination warning system evaluation can be divided into two categories:
field evaluation and data analysis. Field  evaluation includes activities such as drills and exercises, direct
observations and performance tests, interviews, and documentation of lessons learned.  Data analysis may
involve simulations and/or analysis and integration of data. These methods, along with the objectives and
metrics considered during evaluation will be expanded on in future guidance based on lessons learned
from the initial Water Security initiative pilot.

3.6.2   Refinement

As with any system, refinements  to the contamination warning system are likely to be identified through
routine operation and maintenance or evaluation.  In order to optimize system performance and ensure
that the design objectives continue to be met, it will be necessary to make some refinements to the system.
Refinements could include modifications to protocols and procedures or may be more extensive such as
replacement or modification of equipment. It is also important to consider that technologies may evolve
over time, goals and objectives may change, or other events could occur that necessitate refinement to the
system.
May 2007                                                                                    24

-------
                                 Planning for WS-CWS Deployment


             Section 4.0:  Online Water Quality Monitoring


The online water quality monitoring component consists of multiple water quality monitoring stations
installed at key locations throughout the distribution system with the goal of establishing the base state for
water quality and using sophisticated event detection systems to monitor for water quality anomalies that
could be indicative of contamination.

Online water quality monitoring is included as a component of the contamination warning system due to
its demonstrated potential to rapidly detect contamination through changes in several commonly
measured water quality parameters (Hall, 2007a, EPA 2006).  These changes may result from the aqueous
chemistry of the contaminant (e.g., dissolution of an organic compound may result in an increase in the
TOC concentration) or from reactions with the disinfectant residual (e.g., oxidation of a reactive
contaminant consumes the free chlorine residual). While there are limited empirical data regarding the
impact of many contaminants of concern on conventional water quality parameters, there has been a
substantial amount of research over the past few years demonstrating that many contaminants of concern
can produce measurable changes in conventional water quality parameters (Hall, 2007b).  Furthermore,
many of these contaminants have been shown to impact water quality at concentrations well below
reported lethal dose concentrations (Hall, 2007c).

Table 4-1 describes the manner in which each of the design objectives presented in Section 1.1 is defined
with respect to the online water quality monitoring component, and considerations regarding how these
objectives impact design and implementation.
Table 4-1. Design Basis Considerations for Online Water Quality Monitoring
   Design
  Objective
               Description
  Design and Implementation Considerations
Capability
Can indicate the presence of a contaminant
that significantly affects one or more monitored
parameters.
Detection of water quality changes is an indirect
measure of contamination; thus operation of this
component should include a process to
investigate the possible cause of the anomaly.
Contaminant
Coverage
High detection potential for classes 1, 2, 3, 5,
8, 9, 10, 11 and 12. Moderate detection
potential for classes 4, 6, and 7.
The specific parameters monitored will determine
the contaminant coverage.  Disinfectant residual
type also has an impact on contaminant coverage
(Szabo, 2006).	
Spatial
Coverage
Function of location, number, and density of
monitoring stations.
Several tools are available to design a water
quality monitoring network in a manner that
optimizes one or more design objectives, such as
minimizing consequences over a large number of
contamination scenarios.
Timeliness
Function of hydraulic travel time from the point
of contaminant introduction to the sensor, and
the concentration of the contaminant.
Time to detection can be considered as a primary
or secondary objective in the monitoring network
design. In either case, a well calibrated
distribution system model is necessary to perform
this analysis.	
Reliability
Rate of false positive / negative results in this
application is largely unknown at this time.
May be addressed through event detection
systems and consequence management.
The design elements with the greatest impact on
reliability are the event detection system and the
water quality monitoring stations.  If reliable
sensors are used and properly maintained, the
capabilities of the event detection system will
dominate reliability as it is defined here.	
Sustainability
Provides utility with a better understanding of
water quality variability throughout distribution
system and provides an opportunity to
optimize distribution system operation.
The selection of parameters and monitoring
locations will have a direct influence on dual-use
applications that will improve sustainability, and
thus should be considered in the design of the
water quality monitoring system.	
May 2007
                                                                                25

-------
                                Planning for WS-CWS Deployment

The primary objective of this section is to describe considerations during planning for the implementation
of a water quality monitoring network in a drinking water distribution system as a component of a
contamination warning system. These considerations were derived from EPA's experience designing the
system at the initial pilot for this program. In planning for implementation of a water quality monitoring
network several key design decisions should be made, including the following:
    •  Water quality parameters to be monitored
    •  Use of a single monitoring station design or multiple designs in a tiered system
    •  Specific sensors and instruments integrated into a water quality monitoring station
    •  Number of water quality monitoring stations to install
    •  Methodology for determining the locations at which water quality monitoring stations will be
       installed
    •  Comprehensive concept of operations to guide routine operations and trigger validation
    •  Communication architecture to transmit data from monitoring locations to an operations center
    •  IT architecture used to manage and store water quality and related data
    •  Event detection software deployed to detect anomalies
    •  Staffing available for monitoring station equipment operation and maintenance

A key objective of Section 4 is to provide information that will enable the reader to work through these
design decisions in a systematic and integrated fashion.

While the  overall design of an online water quality monitoring system should be developed in an
integrated fashion such that all elements are compatible and serve a specific function in the overall
system, Section 4 considers five major design elements to facilitate the presentation of material. These
elements are:
    •  Monitoring Network (Section 4.1): The spatial plan for deployment of water quality monitoring
       stations throughout a drinking water distribution system. The monitoring network design
       specifies the number and precise location of each water quality monitoring station.
    •  Monitoring Stations (Section 4.2):  The specific instruments, probes, or other equipment used to
       monitor a water quality parameter, as configured into a monitoring station that contains all
       ancillary equipment (e.g., plumbing, electric,  communications, etc.).
    •  Communication Systems  (Section 4.3): All equipment, software, and  services needed to transfer
       data from each water quality monitoring station to a central location (typically a SCADA control
       center).
    •  Data Management Systems (Section 4.4):  All hardware, software, and protocols necessary to
       manage and store water quality and related data for event detection. The utility SCADA system
       will typically serve as the foundation of the data management system for the water quality
       monitoring network.
    •  Event Detection Systems (Section 4.5): Software or algorithms designed to analyze real-time
       water quality data in order to detect anomalous conditions that might be indicative of
       contamination.

Each of these design elements is discussed in a dedicated subsection of Section 4, and is presented in the
phases of pre-design, design and implementation, and available tools and resources. Section 4 concludes
with a discussion of staffing and cost considerations.

4.1    Monitoring Network Design
Monitoring network design is a systematic process for determining the location and number of monitoring
stations deployed in a contamination warning system. The  design will directly impact two important
aspects of system performance: the time of detection and the spatial coverage of the system. Section 4.1
presents information useful to the design and implementation of a water quality monitoring network
design, with an emphasis on activities related to planning and pre-design of the monitoring network.
When applicable, references to additional resources are included and summarized in Section 4.1.3.

May 2007                                                                                   26

-------
                                Planning for WS-CWS Deployment

4.1.1  Pre-Design

Prior to designing the water quality monitoring network a number of key decisions should be made that
may involve some level of investigation and analysis. These decisions include the following:
    •  The objectives of the monitoring network design
    •  The level of validation needed for the distribution system model
    •  The practical upper limit on the number of monitoring stations to be installed
    •  Use of a tiered approach to monitoring network design
    •  Types of facilities that will be considered candidate locations for monitoring station installation
    •  Methodology used to design monitoring network

The remainder of this section describes considerations relating to each of these design decisions and,
where appropriate, references tools or resources that may be useful in this process. The pre-design phase
culminates in selection of a methodology and overall framework for monitoring network design that is
based on these key decisions.

    •  Network design objectives. A water quality monitoring network can be designed around a
       number of different objectives, some of which may be complementary while others are in some
       degree of conflict.  Furthermore, the various network design tools available may be able to
       optimize towards some objectives but not others. Thus, an important step in the pre-design phase
       is to decide on the primary objective that the network design will attempt to optimize. Examples
       of design  objectives include:
           o   Minimizing the consequences to  the population
           o   Minimizing the extent of contamination
           o   Minimizing the time to detection
           o   Maximizing spatial coverage of the distribution system
           o   Maximizing the number of contamination events detected
           o   Maximizing protection of key facilities or populations

       Given available tools, a monitoring network design can be truly optimized to only one of these
       objectives; however, it is instructive to evaluate the performance of the monitoring network
       design with respect to the other objectives, and to consider the trade-offs involved in the selection
       of a primary objective.

    •  Distribution system model validation.  Many approaches to monitoring network design utilize
       distribution system models, and thus the accuracy of the design is dependent on the accuracy of
       the model. For this  reason, distribution system model validation is an important part of
       monitoring network pre-design.  There are numerous  approaches to model validation that vary
       widely with respect to complexity, cost, and resulting degree of model confidence.  Approaches
       to validation of hydraulic and/or water quality portions of a distribution model include:
           o   Desktop analysis to verify that hydraulic behavior matches actual system operations
           o   Pressure studies to validate the hydraulic model
           o   Chlorine decay studies to evaluate the water quality model
           o   Tracer studies to validate the water quality model

       An up-to-date, accurate network model is useful not only for sensor network design, but also for
       emergency response planning, and potentially for identifying sampling locations and populations
       at risk following a contamination incident.  Potential resources that may be useful in
       characterizing distribution system model  performance are listed in Section 4.1.3.

    •  Maximum number of monitoring stations.  In order to proceed with the  design of a monitoring
       network, it will be necessary to establish  an upper bound on the total number of monitoring
       stations that could be installed. This is a  function of the total budget for the monitoring network

May 2007                                                                                   27

-------
                                Planning for WS-CWS Deployment

       and the unit cost for each monitoring station.  It may also include an analysis of incremental
       benefit of each additional monitoring station added to the system, such as the example shown in
       Figure 4-1.
                                   Illustrative  Example
            o
10            20            30           40
   Number of Monitoring Stations
50
Figure 4-1. Example Monitoring Station Tradeoff Curve


    •   Tiered approach to monitoring stations.  A potential means of increasing the total number of
       monitoring locations within a given budget is to use a tiered approach in which two or more water
       quality monitoring station designs, with different costs and capabilities, are deployed. Under this
       tiered approach, the design of the water quality monitoring network will involve a tradeoff
       between the total number of monitoring stations installed in the network and the unit cost (and
       presumably capability) of the monitoring stations, as described in Section 4.2. In the context of
       the design basis, this is a trade-off between spatial coverage and contaminant coverage.  While a
       tiered design may allow more monitoring stations to be deployed, the stations that monitor for
       fewer parameters will have reduced contaminant coverage. Therefore, some contamination
       events that would have been detected by the more complex stations will go undetected even if
       they pass locations with simpler monitoring stations. Thus, the applicability of a tiered design
       should be considered at a system level to determine the optimal trade-off between spatial and
       contaminant coverage, and some of the monitoring network design tools can provide a means of
       evaluating such a tradeoff. Additional discussion of tiered monitoring stations is included in
       Section 4.2.

    •   Candidate facilities for installing monitoring stations.  Prior to designing a water quality
       monitoring network, it is necessary to identify categories of feasible installation locations, such
       as: utility facilities, fire stations, police stations, post offices, government buildings, etc. During
       this initial phase of investigation, a set of general requirements can be provided to facility
       managers to determine if they would be willing and able to host a water quality monitoring
       station.  These general requirements may include:  security, 24/7/365 access to facility, and space
       for a monitoring station. At this stage it is unlikely that specific facilities will be considered, but
       rather categories of facilities (e.g., all fire and police stations within a given jurisdiction).
       Specific locations will not be investigated until a monitoring network design has been completed
       as described in Section 4.1.2. The purpose of identifying categories of feasible locations at this
       point is to determine how the monitoring network design will be constrained, and for this reason
       the categories of feasible locations should be as comprehensive as possible.
May 2007
                                                                  28

-------
                                Planning for WS-CWS Deployment

    •  Methodology for designing the network. The final stage in the pre-design of the monitoring
       network is selection of methodology for designing the monitoring network. As mentioned
       previously, most require use of a distribution system model including PipelineNet, Threat
       Ensemble Vulnerability Assessment - Sensor Placement Optimization Tool (TEVA - SPOT), and
       various tools built into hydraulic modeling software applications. Section 4.1.3 provides a partial
       listing of available tools.  It is necessary for the utility to have a calibrated distribution system
       model in order to use these software applications. In selecting a monitoring network design tool,
       it is important to ensure that tool is compatible with the platform for the utility's distribution
       system model, and that it will support the overall approach to design.  In particular, it is important
       to ensure that the tool can optimize the design to the desired objective(s).  Furthermore, if a tiered
       design is used, it would be beneficial if the tool could optimize a design that incorporates
       monitoring stations with differing  arrays of sensors. Other factors to consider in selection of a
       monitoring network design tool include:
           o   Transparency and rigor of the optimization methodology
           o   Applicability of the design and optimization methodology to detection of intentional
               contamination incidents
           o   Time required to produce  a design
           o   Features that facilitate comparison of multiple designs
           o   Visualization tools and compatibility with GIS
           o   Usability

If a distribution system model is not available, it will be difficult to optimize a sensor network design to a
specific objective, and it may be impossible to systematically evaluate trade-offs for various design
options.  Expert-based designs can be developed without a model, but have been shown to perform poorly
compared to optimization methods that utilize distribution system models (Ostfeld, 2006).

The constraints on possible monitoring locations are another important design consideration. In general,
designs that place monitoring stations only at utility-owned sites will not be able to perform as well as
designs that allow for a large number of potential sites for locating monitoring stations.

4.1.2  Design and Implementation Approach

The activities described in  Section 4.1.1 describe a process for pre-design of a water quality monitoring
network, culminating in documentation of an overall framework and selection of a methodology for
designing the monitoring network. The approach for design and implementation of the monitoring
network based on the pre-design may include the technical considerations and specifications described
below.

Design:
    •  Develop a comprehensive list of physical addresses  for potential installation locations. This list
       should be based solely on consideration of the general categories identified during pre-design.
       GIS can be an effective tool for compiling  and visualizing this information.
    •  Identify nodes in the distribution system model that correspond to each physical address.
    •  Develop a suite of design constraints in terms of number of stations, potential installation
       locations, and type of water quality monitoring station if tiered water quality monitoring network
       designs will be considered.
    •  Use a sensor placement tool, such  as TEVA or PipelineNet, to develop a monitoring network
       design for each set of constraints.
    •  Compare the various monitoring network designs (e.g., through a tradeoff analysis, cost benefit
       analysis, regret analysis, etc.).
    •  Select a monitoring network design that specifies number and location of water quality
       monitoring stations. If a tiered design is used, also specify the type of water quality monitoring
       station at each location.

May 2007                                                                                   29

-------
                                Planning for WS-CWS Deployment

    •  Field verify each installation location in the design to ensure:
           o   Access to electrical power to run the equipment
                   •   Certain equipment may require higher voltage or current than may be found
                      through common wall sockets
           o   Access to water to run samples from the distribution system directly to the sensor station
           o   Drainage for the discharge stream from the monitoring station.
                   •   Verify the discharge option complies with applicable regulations.
           o   Size of area that the station is being installed
                   •   There should be enough room to contain the sensor station and there should be
                      enough space for utility personnel to access the station for maintenance
           o   Security of the location
                   •   Limit access to those without a need to maintain the equipment
           o   Accessibility, if location is not a utility owned facility
                   •   Utility personnel should have 24/7/365 access to the equipment at all times in
                      order to maintain the station or respond to an alarm event
           o   Safety
                   •   Health and safety should be addressed according to each site's safety procedures.
                      In all cases, however, the minimum safety considerations should meet
                      Occupation Safety and Health Administration (OSHA) requirements.
    •  As field verification finds some locations to be unsuitable, iterate through modifications to the
       monitoring network design and field verification until  acceptable physical locations for all water
       quality monitoring stations have been identified.
    •  Obtain any required reviews or approvals on the design.

Implementation:
    •  Develop agreements with facility owners who will host water quality monitoring stations.
           o   Facility access
           o   Contacts at facility and utility
           o   Water and sewer credits

Evaluation and Refinement:
    •  Through simulations, evaluate ability of the "as-built" monitoring network design to detect an
       ensemble of contamination incidents or other water quality anomalies.
    •  Upon revision to the distribution system model (e.g., due to system growth, changing demand
       patterns, recent calibration activities, etc.), evaluate potential modifications to the monitoring
       network design.
    •  Periodic evaluation of potential benefits of the addition of more water quality monitoring stations
       or the relocation of existing stations.

4.1.3  A valiable Tools and Resources

The following tools and resources are available to support the design, implementation, and evaluation of a
water quality monitoring network as a component of a contamination warning system:
    •  Ostfeld, et. al. "Battle of the Water Sensor Networks", in proceedings of the  ASCE/EWRI Water
       Distribution System Analysis Symposium, August 27-30, 2006. Cincinnati,  OH.
    •  Berry, J., Fleischer, L., Hart, W.E., Phillips, C.A., and Watson, J.P. 2005, "Sensor Placement in
       Municipal Water Networks," J. Water Resources Planning and Management, 131  (3): 237-243
       (2005).
    •  Boccelli, D. L., Shang, F., Uber, J. G. and Wang, J. "Tracer Studies and Water Quality
       Monitoring for Evaluating Network Model Confidence." 4th International Conference on
       Watershed Management and Urban  Water Supply, Shenzhen,  China. 2004.
May 2007                                                                                   30

-------
                                Planning for WS-CWS Deployment

    •  Murray, R., Janke, R., Uber, J., Published in the Proceedings of the ASCE/EWRI Congress, Salt
       Lake City, UT. The Threat Ensemble Vulnerability Assessment (TEVA) Program for Drinking
       Water Distribution System Security. 2004.
    •  Watson, Jean-Paul, et al, "A Multiple-Objective Analysis of Sensor Placement Optimization in
       Water Networks", Proceedings of the ASCE/EWRI Congress, June 2004
    •  PipelineNet:  http://eh2o.saic.com/iwqss/.
    •  Hart, W. E., J. Berry, R. Murray, C. A., Phillips, L. A., Riesen, J. P., Watson, 2007. "SPOT: A
       Sensor Placement Optimization Toolkit for Drinking Water Contaminant Warning System
       Design," Proceedings of the World Environmental and Water Resources Congress, Tampa,
       Florida, 2007.
    •  Murray, R., W. E. Hart, and J. Berry. "Sensor Network Design for Contamination Warning
       Systems: Tools and Applications", Proceedings of the AWWA Water Security Congress, 2006.;
    •  TEVA:  http://www.epa.gov/nhsrc/water/dw/teva.html;
    •  Info Water Sensor Location Manager (SLM):
       http://www.mwhsoft.com/page/pjroduct/infowaterSLM/infowaterslm  feature.htm (based on the
       PipelineNet tool)
    •  USEPA. 2005.  Water Distribution System Analysis: Field Studies, Modeling and Management: A
       Reference Guide for Utilities. EPA/600/R-06/028.

4.2    Monitoring Station Design and Installation
Monitoring station design requires careful consideration of the selected water quality parameters to be
monitored, as well as of the resources and conditions existing at the intended installation sites. Each
individual location will impose unique constraints on the installation, design and operation of a
monitoring station, and potentially constrain the parameters or instruments which may be used. Section
4.1 presented general guidelines regarding site  constraints for installation and operation of monitoring
stations, while Section 4.2 deals with the  design of the actual water quality monitoring stations. Section
4.2.1 describes activities related to planning and pre-design of the monitoring stations, while Section 4.2.2
describes the remainder of the design and implementation process in summary fashion.  When applicable,
references to additional resources are included  and summarized in Section 4.2.3.
4.2.1  Pre-Design

The most significant decision made prior to designing water quality monitoring stations is selection of the
parameters to be monitored. These parameters will define the capabilities of the water quality monitoring
system for detection of contamination events as well as operation for dual-use applications. It will also
have a significant impact on the cost and design of each water quality monitoring system.  This section
describes factors to consider in the selection of water quality parameters.

One of the primary considerations in selecting water quality parameters for use in a contamination
warning system is the potential for parameters to change in response to the presence of a contaminant at
concentrations which pose a threat to public health, infrastructure, or acceptability of the water to
consumers. A substantial body of research has been conducted which demonstrates that the vast majority
of contaminants of concern from a security perspective do alter water quality in a detectable manner.
Section 4.2.3 includes references to some of this research, and Table 4-2 summarizes the impact of the
different contaminant detection classes, as presented in Table 1-1, on the water quality parameters.
Table 4-2. Impact of Contaminant Detection Classes on Water Quality Parameters
Class
1
2
3
4
5
Description
Petroleum products
Pesticides (reactive)
Inorganic compounds
Metals
Pesticides (non-reactive)
Example Contaminant
Diesel
Aldicarb
Arsenite salts
Mercuric salts
Fluoroacetate
TOC
X
X


X
CI21

X
X


COND


X
X

May 2007
31

-------
                                Planning for WS-CWS Deployment
Class
6
7
8
9
10,11
12
Description
Chemical warfare agents
Radionuclides
Bacterial toxins2
Plant toxins
Pathogens2
Persistent chlorinated organic
compounds
Example Contaminant
VX
Cesium-137
Botulinum toxin
Ricin
Vibrio cholerae
PCBs
TOC
X

X
X
X
X
CI21
X

X

X

COND

X




Acronyms: TOC - total organic carbon, Cb - chlorine residual, COND - conductivity
1)  Indicated contaminant classes have been shown to consume free chlorine residual. Results are not applicable to
    chloramine residual.
2)  These contaminants are chlorine sensitive, thus it would be necessary to neutralize the chlorine residual in order
    to maintain potency.  Many neutralizing agents would also increase TOC.

In general, the results of reported research illustrate that free chlorine is the most sensitive indicator of
contamination, showing significant changes from baseline values at concentrations often one to two
orders of magnitude below lethal concentrations.  Specifically, many contaminants were detected at
concentrations around 1 mg/L, while the  corresponding lethal concentration might range from 10 to 100
mg/L.  In the case of pathogens and bacterial toxins, the active contaminant would generally not produce
a detectable change in free chlorine residual; however, co-contaminants would often be present that
would reduce free chlorine residual (a condition necessary to maintain viability of these contaminants).
These results are applicable to systems using a free chlorine residual, but not to chloramines which were
found to be stable in the presence of all contaminants tested, and thus chloramine (or combined chlorine)
residual does not appear to provide a reliable means of contaminant detection.

These studies also indicate that total organic carbon (TOC) is a particularly useful parameter for detecting
the presence of many organic compounds, with a  sensitivity ranging from -0.5 mg/L to more than 1
mg/L, depending on baseline levels and variability.  Even at the upper  end of this range, most organic
contaminants should trigger a change in TOC concentration at concentrations well below the lethal
concentration.

Conductivity was observed to respond slightly to  some inorganic contaminants, including some metals,
but the response was not as strong as that observed for free chlorine residual and TOC. Nonetheless,
conductivity has demonstrated the potential for detection of some contaminants that do not alter chlorine
or TOC. Generally, higher concentrations of contaminants are needed to trigger a response from
conductivity sensors.

Beyond free chlorine residual, TOC, and conductivity, other water quality parameters may provide
supporting information about potential contamination.  Oxidation reduction potential (ORP) will
generally behave similarly to chlorine residual, and can be used to corroborate an observed change in the
chlorine residual.  ORP may also serve a more prominent role in systems that use a chloramine
disinfectant residual as certain oxidation  reactions can occur without reacting with chloramines. pH is
important to aqueous chemistry and may be useful in understanding observed changes in  other
parameters, such as free chlorine residual. Studies have generally shown that turbidity is an erratic and
unreliable primary indictor of contamination; however, as with pH and ORP, it may be useful in
understanding changes in other measured parameters.

Other parameters, beyond the basic parameters described above, can be considered in the design of the
water quality monitoring system. While  it is of primary importance that selected parameters relate to the
primary objective of the system - detection of contamination, other factors to consider in selection of
water quality parameters include dual-use applications and sustainability.

The collected water quality monitoring data may provide useful information and benefits to ongoing
operation of the distribution system and water  treatment processes.  Existing distribution system water
quality data should be reviewed in terms  of occurrence, average concentrations, and variability to identify
May 2007
32

-------
                                Planning for WS-CWS Deployment

other parameters that may provide additional information useful to other objectives relating to water
quality or system operations.
The sustainability of monitoring for different parameters should be considered.  The utility's existing
online water quality monitoring program should be reviewed in terms of equipment, performance, and
maintenance requirements.  Research and industry literature, evaluations, performance studies, etc. should
be reviewed to gather information on the performance and maintenance requirements of various
technologies. Conduct a preliminary assessment regarding the sustainability of monitoring station
designs which incorporate various sensors and technologies. Only those technologies which can be easily
and affordably maintained in an acceptable operating condition will be useful in a monitoring system over
a long time period.

Based on identified monitoring parameters, a preliminary selection of monitoring instruments and
analyzers can be made. The selection may be based on the utility's direct experience with various
analyzers, consultation with other utilities regarding their experience with equipment, and a review of
manufacture technical literature and third party technology evaluations (Environmental Technology
Verification [ETV], Technology Testing and Evaluation Program [TTEP], National Sanitation Foundation
[NSF], etc.).

The parameters selected to be included in the system need not be limited to traditional water quality
parameters, or those parameters used in previously published contamination warning system pilot studies.
However, the selected parameters should relate to the primary objective of contamination monitoring as
discussed under "system design considerations."

Depending upon the identified needs, costs and constraints, it may be beneficial to use a tiered system in
which two or more monitoring  station designs, each with different costs and capabilities, are deployed.  A
tiered design may allow for more monitoring locations, but at the cost of reduced contaminant coverage at
locations with the simplified design.  A clear understanding of the contaminant detection potential of each
system, and the resulting detection compromises throughout the distribution system is necessary to most
effectively deploy a tiered system.

4.2.2  Design and Implementation Approach

The activities described in Section 4.2.1 describe a process for pre-design of water quality monitoring
stations, culminating in selection of parameters to be monitored as well as the potential to use tiered
designs with different levels of detection capability. The approach for design, installation, and operation
of water quality monitoring  stations based on the pre-design may include  the technical considerations and
specifications described below.

Design:
The first phase of monitoring station design should include a visit to  each candidate field installation
location. An evaluation of each site should consider:
    •  Size constraints - Confirm sufficient access and space at the site for installation/fabrication and
       regular maintenance of analyzers
    •  Environmental conditions - Temperature, humidity, vibration, and air quality should be
       considered as they pertain to the monitoring system as well as the maintenance personnel
    •  Site requirements - Ensure availability of sample water, drain, electric power, and the ability to
       route conduit for data communications.  Depending upon site conditions, it may be necessary to
       condition or regulate the  supply water, pump the sample drain, upgrade  or modify the site
       electrical equipment, or locate communications equipment remote from the monitoring
       equipment.  It is also important to verify that the flow of water through the monitoring station is
       adequate to produce detection times on the order of two hours or less. In some cases, this may
       require  water to be bypassed to drain in order to reduce residence time.
May 2007                                                                                   33

-------
                                Planning for WS-CWS Deployment

    •   Security and accessibility - Monitoring station level of security from tampering and disruption at
        each location, and degree of accessibility for utility personnel at any time. At non-utility owned
        sites, it may be necessary to enclose the monitoring system in a locked enclosure. Ensure that
        water utility personnel will have access to the monitoring station at any time in event of a
        contamination occurrence.

Based on the site surveys and the analyzers identified during pre-design, the list of selected sensors and
ancillary equipment should be finalized for each site and each tier of monitoring system. Other
considerations to be addressed include the following:
    •   Remote sampling equipment details and capabilities
    •   Control and data systems including Programmable Logic Controllers (PLCs)  and local data
        loggers
    •   Communication methods, such as radio, telephone, internet and associated equipment for the
        selected method
    •   Power assurance, such as an emergency generator or other uninterrupted power supply, that meets
        reliability and duration criteria required for the system

Physical design of the water quality monitoring station (or multiple stations if tiered approach is used) can
proceed in the  following areas:
    •   Frame or panel design - Depending upon whether the monitoring stations will be permanently or
        temporarily placed in the assigned locations, different types of supporting structures may be
        required.  Specific requirements particular to the surveyed installation sites should also be
        considered. Placement of analyzers and other components should be considered to result in a
        system of the required overall size.
    •   Electrical system design - The electrical power supply as well as the control,  data and
        communication components should be adequately routed, enclosed and configured.
    •   Plumbing system design - Provide sample water to the  instruments at a pressure which is suitable
        for the instruments. Use of a pressure regulator may be required to reduce the distribution system
        pressure to the required range. Plumbing components which will not corrode, plug or easily
        break are recommended.  The ability to monitor and regulate flow to individual instrument or
        groups of instruments is useful, however rotometers or other regulating devices should be
        carefully selected to ensure proper operation. Use of a coarse strainer at the system inlet will
        prevent small particles from entering the monitoring instruments or components and impacting
        operation.
    •   Sampling system design - An automated system should be included to collect a water sample
        when potential contamination is detected.  Design factors to consider include  the quantity of
        sample to be collected to meet the requirements for laboratory analysis; materials of construction
        of the sample container and components to preclude contamination; method of activation -
        automatic or operator initiated; personnel protection from potentially contaminated sample water.
    •   Obtain any required reviews or approvals of the design.

To illustrate the concepts discussed in this section, a schematic of the water quality monitoring station
design used in the initial Water Security initiative pilot is shown in Figure 4-2.
May 2007                                                                                   34

-------
                                Planning for WS-CWS Deployment
                          PLC/
                         Electrical
                          Panel
                          Radio
                          Panel
                                                                  1.J.J
 roc
analyzer
                                  a             ;a
Figure 4-2.  Example Water Quality Monitoring Station Design used in the Initial Pilot
This design is based, in part, on a requirement for a mobile, easily relocated station.  This example design
includes a side-mounted TOC, PLC, and radio panel to facilitate installation at sites where space is
limited or where it is necessary to move the system through small doorways or hatches to maneuver it
into place. Some of the additional hydraulic and mechanical features incorporated in this design include:

    •  Supply and Drain Hoses:
           o   For the ease of installation and relocation of the station, flexible hoses were used.
           o   A high quality hose that complies with American Water Works Association (AWWA)
               standards was used to prevent crimping.
    •  Supply and Drain Pipes:
           o   Brass supply piping and fittings are used for durability, ease of installation, and most
               importantly because they do not leach organic materials that could confound analysis.
               Sturdy CPVC piping and fittings may also be used, however careful attention to structural
               stability is required, as well as a flush-out period to purge the  system of trace plastic and
               adhesive chemicals prior to placing the system on line.
           o   Flexible tubing routes sample flow from the supply manifold  and flow regulating
               rotometers to the individual instruments.
           o   CPVC drain piping is used, and generously sized to ensure free flowing gravity drains.
           o   A manual sample supply shut-off valve provides easy isolation of the system in the event
               of a substantial leak.
           o   A 40-mesh Y-strainer prevents entrained particles from the distribution system from
               entering sensitive analyzer components.
           o   An actuated solenoid valve routes flow to an appropriately sized container to collect a
               sample for laboratory analysis when possible water contamination is sensed (not shown).
    •  Flow Regulation:
           o   A downstream pressure regulating valve provides sample water at the proper pressure to
               the monitoring instruments.
           o   Flow regulating rotometers are provided to control and confirm sample flow to each
               instrument or group of instruments
May 2007
                             35

-------
                                 Planning for WS-CWS Deployment

           o   A manual bypass valve is provided at the downstream end of the sample supply manifold.
               The outlet of this valve is routed directly to the system drain. The valve provides the
               ability to easily flush the supply pipe.  For monitoring systems which are located a
               significant distance from the distribution system main, the bypass may be left partially
               open during normal system operation to reduce the travel time from the main to the
               station, thus reducing the time to detection of a water quality anomaly.
    •   Electrical Panel:
           o   All necessary electrical feeds and monitoring/control signals are routed through an
               electrical panel to simplify the  design and allow for ease of installation.
    •   Radio Panel
           o   For a system which communicates via radio, it may be beneficial to locate the radio
               equipment in a dedicated panel, rather than including them in the electrical panel.  Doing
               so facilitates establishment of a radio link, which may require that the antenna be located
               a distance away from the monitoring station.
           o   The data communications and analysis aspects of a contamination warning system are
               further discussed in Section 4.3

The design depicted in Figure 4-2 serves as a reference only and is not to be  considered a preferred
design.  The ultimate design of monitoring stations should be determined by the installing utility and
tailored to the unique requirements of each application.

Fabrication and Testing:
    •   A contract should be established with a qualified fabricator who is familiar not only with
        electrical and instrumented systems, but also with standards and requirements for piped and tubed
        hydraulic systems. A sufficiently detailed fabrication specification should be developed and
        included as part of the  contract.
    •   Instruments and other major components may be purchased by the utility or by the fabricator.
        Some cost savings may be realized if purchased by the utility, as well as better assurance and
        control over delivery schedules.
    •   As with any instrumentation and control system, a factory acceptance test should be conducted to
        ensure that the specified components have been used, that the wiring is to specification and
        correctly installed, and that all items function as necessary.  Inspection and testing of hydraulic
        components is also necessary to ensure proper operation and leak-free assembly. Identification
        and correction of errors or problems at the factory can be accomplished with less impact to cost
        and schedule than those found during site installation. Manufacturer specifications for some
        sophisticated analyzers recommend that power not be applied until they are installed in the field
        and connected to the water source. For these types of instruments, be sure to include an
        appropriate warning in the fabrication specification, and not simply rely on warnings included in
        the analyzer manual or packing documents.

Installation:
    •   Depending upon the nature of the designed  systems and the  installation location, it may be
        necessary to contract with a separate  entity for installation at the monitoring sites. For example,
        systems which are fully factory fabricated, but which are to be installed in facilities with difficult
        access may require the services of a qualified mechanical contractor who can rig the units into
        place.  Easy access locations may be  installed or fabricated in place by the instrument contractor.
        Other locations may require the skills of both types of service providers.
    •   Installation of the  monitoring stations will involve the coordination of a number of entities
        involved in activities ranging from delivery of the fabricated units to the installation site to final
        inspection and approval of the installed equipment. A separate installation specification may be
        beneficial so that the installer is aware of all site conditions and expectations. The specification
        should clearly define the exact location for installation of the unit as well as details for
        connections to power,  water, communications, and drain. A comprehensive installation and

May 2007                                                                                     36

-------
                                Planning for WS-CWS Deployment

       inspection schedule can be developed to ensure that contractors and inspectors are available at
       key points of the project. Finally, it is important to clearly identify all required inspections for the
       installed systems so they can be scheduled into the installation timeline.

Start-up and Baseline:
Adequate time and resources should be allocated so that the monitoring station can be commissioned and
brought to a level of operations which is supported by a high level of confidence from the operating and
maintenance personnel.

Typical commissioning activities include:
    •  Initial configuration and calibration of instruments.  Acquire calibration and operating solutions
       and reagents from the instrument vendor or other sources. Note that some reagents have limited
       shelf life from the date of manufacture, so delayed delivery may be desired.
    •  Configuration and testing of communications from the monitoring station to the central control
       facility
    •  Signal testing and verification at each monitoring station and at the central control facility
    •  Validation of water quality monitoring station performance (e.g., by routine comparison of sensor
       measurements with grab samples analyzed with an accepted, independent method).

Operation and Maintenance:
Operation and maintenance activities include the following:
    •  Documentation including, as-built specifications, Operations and Maintenance (O&M) manual,
       etc.: Purchased instruments and controls equipment are typically delivered by the manufacturer
       with operating and maintenance instructions. The monitoring station fabricator should also
       provide catalog and O&M documents for all components provided for the system. One complete
       set of O&M documents should be made available at the  site of each monitoring station, or one set
       available at a central location.
    •  System design drawings should be marked-up and modified to reflect the as-build condition of
       each monitoring station. This is especially true for electrical design drawings. Accurate drawings
       should be kept with the system O&M documents and readily available to maintenance personnel
       and operations technicians.
    •  In order to maintain the high level of availability that is necessary for contaminant monitoring
       systems, a detailed maintenance and calibration plan should be developed and followed.
    •  Most instrument technicians are able to perform routine  maintenance checks such as confirming
       sample flow rates and refilling reagent supplies. Some complex analyzers require annual or semi-
       annual service which may be best left to factory service  personnel. A factory service contract,
       including the cost of replacement consumables may be beneficial.
    •  A stock of replacement parts or maintenance items should be kept in supply to enable quick repair
       in the event of unexpected component failure. For components that have a limited shelf life, a
       limited quantity of such parts should be maintained, and they should be cycled into field use prior
       to expiration.
    •  For monitoring stations that are installed in facilities not owned by the water utility, it is
       important to maintain contact with the facility host and adhere to the hosting terms of agreement.
    •  O&M schedules delineating field actions for various maintenance and calibration intervals along
       with record keeping requirements (i.e., log book documentation)

Evaluation and Refinement:
    •  Maintenance records can provide valuable information regarding the performance and usefulness
       of selected monitoring instrumentation. Equipment that does not provide useful data in a cost
       effective manner should be evaluated for replacement with another technology or manufacturer.
    •  Periodic comparison of sensor measurements with laboratory measurements can help to ensure
       that received data values are an accurate representation of the actual water quality. Erroneous
       readings can result in false positive warnings or missed suspicious events.

May 2007                                                                                    37

-------
                                Planning for WS-CWS Deployment

    •   The state of available sensor technologies and changing contaminant monitoring trends and
       priorities should be followed. As monitoring equipment and recommended operating strategies
       evolve, utilities should evaluate whether changes should be made to their monitoring stations.

4.2.3  Available Tools and Resources

The following tools and resources are available to support the design, installation, operation, and
evaluation of online water quality monitoring stations for the online  water quality monitoring component
of a contamination warning system:
    •   American Society of Civil Engineers. Interim Voluntary Guidelines for Designing an Online
       Contaminant Monitoring System. US. EPA Cooperative Research and Development Agreement,
       X-83128301-0, December 9 2004.
    •   Grayman, Walter M., et al., Design of Early Warning and Predictive Source-Water Monitoring
       Systems. AWWA Research Foundation and American Waterworks Association. 2001.
    •   Hall, J. et al., "Online Water Quality Parameters as Indicators of Distribution System
       Contamination," mJAWWA, 99:1:66.  January, 2007.
    •   Hall, J. et al., "Contaminant Minimum-Dose Threshold Concentrations for Water Quality
       Sensors." USEPA Report, December, 2007.  Available only through WaterlSAC.
    •   USEPA, "Water Quality Sensor Response to Potential Chemical Threats in a Pilot-Scale Water
       Distribution System."  USEPA Report, January, 2006. Available only through WaterlSAC.
    •   Szabo, J. et al., "Water Quality  Sensor Response to Contamination in a Single Pass Water
       Distribution System Simulator." USEPA Report,  January, 2007. Available only through
       WaterlSAC
    •   Szabo, J.G., Hall, J.S. and Meiners, G.C. "Water quality sensor responses to injected
       contaminants in a chloraminated pipe loop ". American Water Works Association (AWWA)
       Water  Security Congress, Technical Session TUE6: Technology Forum B, Washington, DC,
       September 10-12, 2006
    •   Hargesheimer, Erika, et al, eds.  Online Monitoring for Drinking Water Utilities.  AWWA
       Research Foundation and American Water Works Association.  2002.
    •   International Standard ISO 15839. Water quality — On-line sensors/analysing equipment for
       water — Specifications and performance tests. 2003
    •   ETV and TTEP evaluation reports of online monitoring equipment. Available at epa.gov/etv.

4.3    Communications Architecture

The objective of this task is to provide a long-term, cost effective communications  system for relaying
monitoring station data to the SCADA system software and ultimately to the Event Detection System
(EDS) in a timely manner.

There are numerous technologies available today to relay SCADA information from remote sites. These
include radio (e.g., spreadspectrum), copper (e.g., cable, telephone), cellular, and fiber to name a few.
Each technology has advantages and disadvantages.  It is usually desirable  for a utility to choose a
communications system that is easy to support and is cost  effective.

Below is a list of activities and functions that may be required to provide a communication system for
implementation and operation the online water quality monitoring component of a contamination warning
system.
May 2007                                                                                 38

-------
                                Planning for WS-CWS Deployment

4.3.1  Pre-Design

Prior to design, numerous activities are necessary to produce a successful communications project.

    •  Staffing the design and implementation team.  It is vital that all the stakeholders are represented
       on the design team to provide consensus on all decisions. Otherwise, the project may be riddled
       with delays, lack of trust, and poor quality work.
           o   Suggested Team Members include:
                   •   Public Works or Utility:
                          •   SCADA Manager
                          •   IT Manager
                          •   IT Network Engineer
                          •   Operations and Maintenance Manager
                          •   Engineering
                          •   End users such as Water Quality Staff and System Operators
                   •   Consultants including representatives from other project groups in reviews.
    •  Determine the schedule for the project.  Allow for long delays in dealing with communications
       studies, right of way  issues, communication providers, and contracting of sub-contractors. It will
       be vital to have a project manager with aggressive communication skills to implement this task.
    •  Define the requirements of the communication system needed to support the proposed network of
       online water quality monitoring stations. Consider communication requirements for the enhanced
       security systems deployed as part of the contamination warning  system. Consider the rate and
       quantity of data to be transmitted.
    •  Evaluate existing communication system architecture used to transmit data and commands
       between remote facilities and the utility operations center.  Assess ability of existing system to
       accommodate the proposed water quality monitoring network. Take into consideration the need
       for future growth and changing technologies.
    •  If existing communications systems are unable to meet the requirements evaluate alternatives.
       Identify constraints on communication alternatives,  e.g., hilly terrain may make radio
       communications cost prohibitive.  Consider an alternative communication system for just the
       contamination warning system or replacing the entire communications system.

4.3.2  Design and Implementation Approach

The activities described in Section 4.3.1 describe a  process for pre-design of a communications
architecture for a water quality monitoring network, culminating in documentation of initial requirements
and a preliminary assessment of the utility's existing communication architecture. The approach for
design, installation, and operation of a communications system may include the technical considerations
and specifications described below:

Design:
    •  Select communication technology (e.g., radio, digital, phone lines, fiber, etc.).  Multiple
       technologies may be  desirable or necessary in some applications. Determine the communications
       protocol (e.g., Ethernet).  Evaluate for integrity, loss of service, reliability, and security.
    •  Develop overall communication architecture.  Include all stakeholders identified in the pre-design
       phase in the development of the communications architecture.
    •  Obtain necessary reviews and approvals on the proposed architecture.  Schedule frequent review
       opportunities that include all stakeholders,  and provide a high level of consensus along the way.
May 2007                                                                                   39

-------
                                Planning for WS-CWS Deployment
Implementation:
    •  Select service provider (if applicable), and establish any necessary contractual relationships.
    •  Select an installer, and establish any necessary contractual relationships.  Installer's experience
       with all the technologies needed to complete the work should be heavily weighted in their
       selection.
    •  Identify roles and responsibilities for procurement, installation, and testing of various components
       of the communications architecture.
    •  Work with service provider to get components installed and configured.
    •  Test communication pathways between water quality monitoring sites and operations center.
    •  Develop location-specific installation specifications.
    •  Procure system components that will not be provided by the service provider.
    •  Develop installation and inspection schedule. Coordinate this schedule with installation schedule
       for water quality monitoring stations.
    •  Install remote communications systems at water quality monitoring stations.
    •  Program PLCs at each monitoring location to record data from sensors at specified polling
       interval.
    •  Verify that the installed communication system at each water quality monitoring site is
       transmitting data back to the operations center.

Start-up and Baseline:
    •  Verify data processing and storage at each water quality monitoring location.
    •  Verify data integrity during transmission from water quality monitoring locations to operations
       center.
    •  Stress test for communications requirements, including total bandwidth available.

Operation and Maintenance:
    •  Verify that firmware and software is current with respect to vendor provided updates, patches,
       etc.
    •  Document all communications related settings.  This includes all devices that create, push, or
       receive  data within the architecture.  This is important for calibration and equipment replacement
       during routine maintenance.
    •  Collect  all operation and maintenance documentation.
    •  Annual  inspection and maintenance of communications hardware (or follow the utility's existing
       O&M plan for communication systems).
    •  Maintain terms of agreement with communications service provider.

Evaluation and Refinement:
    •  Annual  assessment of communication system performance relative to performance specifications
       documented in the final design.  This should include communications integrity and security.

4.3.3  Available Tools and Resources

The following tools and resources are available to support the design and implementation of a
communication  architecture as an element of the online water quality monitoring  component of a
contamination warning  system:
    •  National Institute of Standards and Technology, Guide to Supervisory Control and Data
       Acquisition and Industrial Control (SP800-82).  September 2006
    •  Instrumentation, Systems, and Automation Society, Manufacturing and Control System Security,
       November 2005.
    •  National Institute of Standards and Technology, Information Security (SP800-53). February
       2005
May 2007                                                                                   40

-------
                                Planning for WS-CWS Deployment

    •  Telecommunications Industry Association, Data Service Options for Spread Spectrum Systems
       (TIA/EIA/IS-2000-A),  March 2001

4.4    Data Management and IT Architecture

The objective of this task to provide support for the processing of data needed to implement the online
water quality monitoring component of a contamination warning system. This includes protocols for
receiving data from the monitoring stations, incorporating the data into a SCADA system, delivering it to
event detection tools, providing warnings of events, providing security of data flow, ensuring backup of
systems and data, and passing information to other related systems or entities.

This task is a key function to the success of moving information from sensors to SCADA systems to EDS
tools. The task lead should be incorporated into all other project groups to insure a complete working
system. Section 4.4.1 provides a list of activities and functions to develop a data management and IT
architecture for the online water quality monitoring component of a contamination warning system.

4.4.1  Pre-Design

Prior to design, numerous activities are necessary to produce a successful data management and IT
architecture:
    •  Staffing the design and  implementation team. It is vital that all the stakeholders are represented
       on the design team to provide consensus throughout the project. Otherwise, the project will be
       riddled with delays, lack of trust, and poor quality of work.
           o   Suggested Team Members are:
                  •   Public Works or Utility:
                          •   SCADA Manager
                          •   IT Manager
                          •   IT Network Engineer
                          •   Operations and Maintenance Manager
                          •   Engineering
                          •   End users  such as Water Quality Staff and System Operators
                  •   Consultants including representatives from other project groups in reviews.
    •  Determine the schedule for this task.  Consider incorporating this schedule into all project tasks.
       Nearly all the contamination warning system project tasks rely on some form of data
       management.  Members of this group should be flexible to project changes and good
       communicators. Change management skills are critical.
    •  Define the requirements of the data management system needed to support the proposed network
       of online water quality monitoring  stations.  Two primary requirements are to manage data
       collected from the online water quality monitoring stations and support EDS tools. Other
       requirements might include, data storage, backups, remote connectivity, protocol standards,
       software management tools, and data transfer speeds.
    •  Evaluate potential options for managing data from online water quality monitoring stations.
       Consider options for collection by the SCADA system and transfer of data to the EDS tools.
    •  Evaluate potential deployment options for EDS tools considering factors such as:
           o   Proximity to source data
           o   Reliability
           o   Security
           o   System compatibility
           o   Computing and monitoring resources
    •  Assess the ability of the existing SCADA system to serve these data management functions,
       including hosting the EDS tool(s).  For most installations, this will be the ideal architecture.
       Consider implications to the health and welfare of the SCADA system.


May 2007                                                                                  41

-------
                                Planning for WS-CWS Deployment
4.4.2  Design and Implementation Approach

The activities described in Section 4.4.1 describe a process for pre-design of a data management
architecture for a water quality monitoring network, culminating in documentation of initial requirements
and a preliminary assessment of the utility's existing SCADA and IT architecture.  The approach for
design installation, and operation of a communications system may include the technical considerations
and specifications described below.

Design:
    •  Develop an overall IT architecture.  The architecture should include:
           o   Network diagrams including hardware and software resources
           o   Flow diagrams
           o   Personnel interaction
    •  Obtain necessary reviews and approvals on the proposed architecture from all stakeholders.
       Stakeholders may include project members from other teams.
    •  Specify all hardware and software to be purchased that meets utility or facility standards.

Implementation:
    •  Procure required hardware and software
    •  Coordinate installation and startup plan with project task leaders and stakeholders
    •  Install and test hardware/software according to the approved architecture
    •  Program SCADA system with tags for all water quality monitoring locations. Build screens for
       monitoring and EDS Tool warnings.
    •  Install and configure data migration utilities if needed
    •  Design and configure data historian and backup system

Start-up and Baseline:
    •  Verify data transfer among the system components
    •  Verify data integrity at SCADA system relative to data collected at the PLC
    •  Verify data storage and archive
    •  Verify operations of the EDS system within the overall IT architecture

Operation and Maintenance:
    •  Verify that software is current with respect to vendor provided updates, patches, etc.
    •  Document all configurations and settings
    •  Collect all O&M documentation
    •  Annual inspection and maintenance of IT hardware (or follow the utility's existing O&M plan for
       IT systems)

Evaluation and Refinement:
    •  Annual assessment of IT system performance relative to performance specifications documented
       in the final design.

4.4.3  Available Tools and Resources

The following tools and resources are available to support the design and implementation of a data
management architecture as an element of the online water quality monitoring component of a
contamination warning system:
    •  National Institute of Standards and Technology, Guide to Supervisory Control and Data
       Acquisition and Industrial Control (SP800-82).  September 2006
    •  Instrumentation, Systems, and Automation Society (ISA), Manufacturing and Control System
       Security, November 2005.

May 2007                                                                                  42

-------
                                Planning for WS-CWS Deployment

    •  Inter National Committee for Information Technology Standards, Information Technology -
       Security Techniques (17799-2005)
    •  National Institute of Standards and Technology, Information Security (SP800-53). February
       2005

4.5    Water Quality Event Detection
Water quality monitoring stations only provide information about general water quality conditions at a
specific location and time, and in isolation are not well suited to detect contamination. This is
compounded by the fact that water quality at a given location can be highly variable, with several factors
affecting water quality at a given time including pump and tank operations, system demand, and source
water quality.  Sophisticated algorithms incorporated into EDS tools can efficiently mine the large
amount of water quality data produced by these monitoring stations and detect anomalies that may be
indicative of contamination  or other water quality problems that relate to dual-use application of the water
quality monitoring system.  Thus, the EDS tool implemented as part of the water quality monitoring
component of a contamination warning system is critical to the performance of the system and has a
significant influence on the overall reliability of the system.  Likewise, EDS tool performance impacts
sustainability because a system plagued by false alarms will be quickly ignored and ultimately forgotten
unless the performance can be improved through tuning of the EDS tool or inclusion of more data. The
EDS tool may also impact contaminant coverage and timing of detection. Given that EDS performance
impacts four aspects of the design basis (reliability, sustainability, timeliness, and contaminant coverage),
selection of the tool is of paramount importance to the water quality monitoring component of a
contamination warning system. Section 4.5 presents information useful to the selection and
implementation of an EDS tool for water quality monitoring, with an emphasis on activities related to
planning. When applicable, references to additional resources are included and summarized in Section
4.5.3.

4.5.1  Planning

Unlike the previous four design elements of the water quality monitoring system, EDS tools will
commonly be off-the-shelf products, and  thus do not require the same level of design and pre-design as
the other elements.  However, given that the application of EDS to  water quality is relatively new and that
there are  few published, third-party evaluations of these tools, careful consideration should be given to the
basis for selection of an EDS tool.  Thus much of the effort associated with the planning stage of water
quality event detection is collection of information that will lay the ground work for design and
implementation of a selection study.  Suggested information collection efforts include:
    •  A review of utility water quality and operational data
    •  A preliminary assessment of metrics against which EDS tool performance can be evaluated
    •  A market survey to  identify potential EDS tools
    •  A summary of EDS tool specifications for candidate tools
    •  An assessment of potential deployment environments for the EDS tools

The remainder of this section describes considerations relating to each of these planning activities and,
where appropriate, references tools or resources that may be useful in this process. The planning phase
produces a list of candidate EDS tools and a summary of specifications, performance study results, and
other available information for each tool.

Part of planning for an EDS selection study is conducting a thorough review of all utility data that may be
useful for water quality event detection, such as water quality and operational data. Once water quality
parameters have been selected, per the pre-design activities described in Section 4.2.1, a review of
existing distribution system  water quality data may provide insight regarding typical values and expected
variability.  While the degree of variability in water quality parameters, especially free chlorine residual,
May 2007                                                                                   43

-------
                                Planning for WS-CWS Deployment

is highly dependant on location, a general sense of distribution system variability may be useful in the
design of the selection study and may even impact monitoring network design (see Section 4.1.1).

Operational data is important because system operations can have a dramatic and predictable impact on
water quality. Incorporation of operational data into water quality event detection can improve the
performance of the tools as it allows for incorporation of cause and effect relationships that result in water
quality changes. However, in order to use this data, the utility should have a firm understanding of the
available data and knowledge regarding how operational changes impact water quality throughout the
system.  As with water quality variability, these relationships are location-specific. At the planning phase
it is sufficient to characterize the operational data that is available so that it can be used in either the
selection study or during implementation.

Also important in the planning process  is a precise characterization of the requirements for an EDS tool.
Each utility's needs, priorities, and available resources are unique, so a ranking of the important aspects
of an EDS tool is key in tool selection.  A sample ranking of objectives could be:
    1.  An EDS tool should be compatible with current IT infrastructure and be  easily integrated with
       existing SCADA system or other applications
    2.  The tool should detect a high percentage of potential contamination events (e.g., 99.99%)
    3.  The tool should have a minimal number of false alarms (e.g., a maximum of one per week)
    4.  The tool should have minimal maintenance requirements and should be supported by the tool
       developer
    5.  The tool should have dual-use functionality and be able to detect water quality anomalies that
       could arise from variety of causes that may be of concern to the utility (e.g., cross-connections)
    6.  Minimal personnel training should be necessary to implement the tool
    7.  The tool should be able to handle noisy and imperfect data
    8.  The tool should have minimal cost, both initial and ongoing, including any necessary software
       and hardware

The tables below describe some standard performance measures that can be used to evaluate and compare
EDS tools.  Table 4-3 describes some performance measures that quantify the detection capability of an
EDS tool, and in this table an event is defined as a continuous period of time during which water quality
is anomalous at the monitoring locations.

Table 4-3. Standard Measures for Evaluating EDS tool Performance
Performance
Measure
Specificity
Sensitivity
False Alarm Rate
Average False
Alarm Length
Median Detection
Time
Description
The percent of time for which the EDS
tool correctly does not alarm
The percent of events that the EDS
correctly identifies
The frequency of false alarms
The average length of a false alarm
The median time it takes the EDS tool
to detect an event relative to the time
the event reaches the monitoring
station
Example
The tool correctly does not alarm 99.5% of the time
The tool detects 98% of events
On average, the tool produces one alarm per week
The duration of a false alarm ranges from 2 minutes
to 3 hours, with an average of 15 minutes
The delay between the time the event is at the
monitoring station and the EDS tool alarms ranges
from 2 minutes to 84 minutes, with an average of 6
minutes
The performance of the tool, as characterized by metrics such as those in Table 4-3, should guide the
selection of a tool, other factors relating to the operation, usability, and maintenance of the tool are
important considerations as well.  Table 4-4 describes other factors that may be considerations in the
selection of an EDS tool.
May 2007
44

-------
                                Planning for WS-CWS Deployment
Table 4-4.  Sample Measures for Evaluating EDS Software
EDS Software
Performance Measures
Initial Cost
Recurring Cost
Calibration or Training
Compatibility
Customization
Usability
User Interface
Efficiency
Reliability
Developer Support
Description
Fee for purchase and installation of EDS tool
Costs incurred for labor materials necessary to support the operation and
maintenance of EDS tool
Procedures for training EDS tools, ability of staff to learn and implement training
procedures, and requirements for re-training
Compatibility with existing hardware, software, schemas, and other aspects of the
planned deployment environment
Ability to customize EDS tool to a particular utility environment, including:
monitoring locations, water quality parameters, operational data, alarm threshold,
and information displays. Ability to analyze WQ data from multiple locations in near
real-time.
Level of skill required to train, operate, and maintain the EDS tool
Capability of the user interface to navigate the major features of the tool and to
display information in a usable format.
Speed and memory requirements for software
Percentage of time EDS operates as designed
Availability of adequate, long-term support for the operation of the EDS tool
Generally, two preparatory processes are carried out by an EDS software developer before an EDS tool is
deployed at a water utility in real-time operation.  First, the tool is trained on a dataset from each location
that will be monitored in real-time.  This training dataset should be free of anomalies or events so that it
can be used by the EDS tool to "learn" typical water quality patterns.  For example, a 0.2 mg/L change in
chlorine at one location might be normal under certain conditions, whereas it would be extremely rare
(and considered anomalous) at another location. Once trained, the tool would respond differently to such
a change at the two locations (it would likely alarm for the second location but not the first).

Once training is complete, the EDS tool developer will typically work with the utility to configure the
tool, or adjust variables within the tool to maximize event detection performance (sensitivity) while
maintaining a false alarm rate that is acceptable for the utility.  A receiver operating characteristic (ROC)
curve, which plots 1-specifity vs. sensitivity, can be helpful  in quantifying the tradeoff between detection
capability and false alarm rates for a given tool. Ideally, the curve should hug the top left corner of the
graph, as this represents a low number of false alarms with a high probability of detection. Figure 4-3
provides an example ROC curve.
                                       False Alarm Rate (FAR)
                              Figure 4-3. Example ROC Curve

In order to develop a candidate pool of EDS tools for consideration in the selection study, a market survey
or literature review is proposed. This information gathering exercise should consider the performance
metrics and EDS tool attributes discussed above, such  as what information is readily available. Potential
sources of information include technical and scientific literature, vendor information and websites, and
information from utilities currently using or considering using EDS tools. Note that there are a limited
May 2007
45

-------
                                Planning for WS-CWS Deployment

number of tools marketed specifically for event detection in water distribution systems. However, there
are many general tools for event detection or anomaly detection that could potentially be adapted to the
water quality domain. Adaptation of general event detection tools to the water quality domain may
require additional tool customization and/or integration efforts. The Overview of Event Detection Systems
(USEPA, 2005c) describes some available tools, and also includes examples of how tools have been used
at water utilities. Additional information sources are included in Section 4.5.3.

The results of this market survey can be used to develop a list of candidate EDS tools for consideration in
the selection study.  More detailed information and product specifications should be collected for each of
the candidate tools as this information will be critical to the design and conduct of the selection study as
discussed in Section 4.5.2.

4.5.2   Implementation Approach

The activities described in Section 4.5.1 describe a process for performing a preliminary assessment of
EDS tools, culminating in a list of candidate tools and a summary of performance specifications. The
approach for the selection, implementation, and operation of an EDS tool may include the technical
considerations and specifications described below. Prior to selecting an EDS tool, a preliminary concept
of operations as described in Section 3.2.1 should be developed. This exercise will help to  define
requirements for how data can be provided to the tool for analysis and may place some constraints on tool
selection.  Development of the concept of operations should also be tightly coupled with communications
and data management activities.

Planning-Selection Study:
    •   Define requirements for EDS tool performance, compatibility, usability, features, support, etc.
    •   Precisely define the metrics and attributes that will be considered in the EDS tool selection study,
        and document how each will be assessed in either a quantitative or qualitative manner.
    •   Perform initial analysis of candidate EDS tools against requirements to reduce the pool of
        candidate technologies to those that meet the most critical qualitative attributes.
    •   From the remaining list of candidate tools, develop quantitative basis for selection of an EDS tool
        for deployment. Options for a selection study include:
           o   Utility-specific EDS tool evaluation using data collected from each water quality
               monitoring location after the monitoring stations have been found to be producing valid
               data. (Guidance on the design of an EDS evaluation study is currently under
               development by USEPA based on experience at the initial Water Security initiative pilot.)
           o   Third party technology evaluations, either through established programs such as ETV,
               TTEP, NSF, or through research programs.
           o   EDS tool performance documented in peer-reviewed literature and vendor-supplied
               information.
           o   Experience and data from other utilities using EDS tools.
    •   Select EDS tool(s) for deployment at the utility based on the results of the selection study.

Implementation:
    •   Verify that the utility's IT system architecture can accommodate the selected tool.  Modify the
        architecture if necessary.
    •   Collect water quality data from all installed water quality monitoring stations to support initial
        training of the EDS tools.  Use only data collected after it has been verified that the water quality
        monitoring stations are producing reliable data.
    •   Identify operational data that is expected to enhance EDS tool performance and the associated
        operational logic that relates water quality at a specific monitoring location to distribution system
        operations.
    •   Collect relevant distribution system operational data to support training of the EDS tools.


May 2007                                                                                   46

-------
                                Planning for WS-CWS Deployment

    •   Train and tune the EDS tool(s) to each water quality monitoring location using the data described
       above. If a utility-specific EDS tool evaluation was performed, data from the evaluation may be
       useful in training the tool(s) for deployment.
    •   Install and test EDS tool(s) according to the architecture developed under "data management."
    •   Modify the concept of operations as appropriate to reflect the as-built design.

Start-up and Baseline:
    •   Bring the EDS online and establish connectivity with SCADA or other automated water quality
       data source.
    •   Train utility staff on EDS tool operation and maintenance, as well as the concept of operations for
       the water quality monitoring component of the contamination warning system.
    •   Monitor EDS tool performance over the first several weeks of operation, and make adjustments
       as necessary.

Operation and Maintenance:
    •   Monitor and respond to EDS alarms in accordance with the concept of operations.
    •   Log any anomalies detected by the EDS tool(s).
    •   Update the EDS tool event library, if applicable (some tools keep a library of water quality types
       observed at a given monitoring station so that a repeat occurrence of the water quality type can be
       classified as normal).
    •   Retrain the tool if new sensors are added, water quality changes significantly, or performance is
       found to be unsatisfactory.
    •   Verify that EDS tools and supporting software are current with respect to vendor-provided
       updates, patches, etc.

Evaluation and Refinement:
    •   Continuous documentation of performance during operation, e.g., false alarms, detection of true
       water quality anomalies, and dual-use benefits.
    •   Through simulated events, periodically evaluate the overall performance of the installed EDS
       tool(s).  If necessary, update EDS tool configuration to optimize performance.
    •   Track and evaluate the development of new EDS tools.

4.5.3  Available Tools and Resources

The following tools and resources are available to support the selection, evaluation, and implementation
of EDS tools as part of the online water quality monitoring component of a contamination warning
system:
    •   U. S. Environmental Protection Agency, 2005.  "Overview of Event Detection Systems for
       WaterSentinel."
       http://www.epa.gov/safewater/watersecuritv/pubs/watersentinel event detection.pdf
    •   U. S. Environmental Protection Agency, 2006.  "Framework for the Evaluation of Event
       Detection Software for Drinking Water Contamination Warning Systems."
    •   Kroll, D., King, K. (2006). Real World Operational Testing and Deployment of an On-line Water
       Security Monitoring Station.  In Proceedings ofWDSA 2006 Symposium. Cincinnati.
    •   Klise, K., McKenna, S. (2006). Multivariate Application for Detecting Anomalous Water Quality.
       In Proceedings ofWDSA 2006 Symposium. Cincinnati.
    •   Jarrett, R., Robinson, G., O'Halloran, R. (2006). On-line Monitoring of Water Distribution
       Systems: Data Processing and Anomaly Detection. In Proceedings ofWDSA 2006 Symposium.
       Cincinnati.
    •   Umberg, K., Uber, J., Murray, R. (2006).  Performance Evaluation of Real-time Event Detection
       Algorithms. In Proceedings ofWDSA 2006 Symposium. Cincinnati.
May 2007                                                                                 47

-------
                                Planning for WS-CWS Deployment

    •  Hart, D., S. A. McKenna, K. Klise, V. Cruz, and M. Wilson, 2007.  "CANARY: A Water
       Quality Event Detection Algorithm," Proceedings of the ASCE World Environmental and Water
       Resources (EWRI) Congress, Tampa, Florida, 2007

4.6    Staffing and Cost Considerations
Planning for the implementation of the water quality monitoring component of a contamination warning
system requires involvement of a wide array of utility personnel and potentially  contractor staff.  Costs
will be highly dependent on the utility's capabilities and intended enhancements. Therefore, the
remainder of Section 4.6 illustrates the staffing considerations and cost factors that are recommended for
consideration during project planning and pre-design. This section provides a summary of previously
discussed information that should be considered when developing preliminary staffing plans and cost
estimates. Cost considerations represent some unique aspects of implementation based on lessons learned
from the initial pilot.

4.6.1  Staffing

As mentioned above, staffing considerations are critical to the successful implementation of a
contamination warning system. Table 4-5 offers a quick overview of which staff may be necessary to
design, implement, and operate an online water quality monitoring program as part of a contamination
warning system and during which phases of implementation these personnel may be needed.

Table 4-5.  Online Water Quality Monitoring Staffing Considerations
Division or Department
Water Quality
Information Technology
Engineering and Planning
Operations and/or
Distribution
Host Facilities
Implementation Stage
PD
X
X
X
X

D
X
X
X
X

I
X
X
X
X
X
PT
X
X

X
X
O&M
X
X

X
X
E&R
X
X
X


Comments
System end user. Defines system requirements.
Lead in selection of parameters and instruments.
Lead in monitoring network design. Lead in
operations and maintenance.
Designs, implements, and manages all IT systems
used to support online water quality monitoring.
Review and approval of designs. Responsible for
installation oversight and inspections. Support
monitoring network design.
Provides domain knowledge relating to system
operations. Responsible for monitoring water quality
alarms 24/7/365. Support installation and
maintenance of monitoring stations.
Provide access to facilities during installation and
maintenance activities. Contact utility in the event of
a problem that could impact the monitoring station.
PD = Pre-design; D = Design; I = Implementation; PT = Preliminary Testing; O&M = Operations and
Maintenance; E&R = Evaluation and Refinement

Building the team to implement this component of a contamination warning system will involve all
divisions within the utility. Therefore, it is important to have senior leadership involved and invested
during each stage to facilitate the resolution of cross-division issues. Several key personnel, like the IT
manager and the Water Quality division manager, would ideally be members of this and all other
component teams to facilitate the application of the system engineering principles outlined in Section 2.1.
Such involvement can be a significant commitment of time and resources for these individuals, but the
utility can reap substantial benefit in the long-term success of the system. Other team members'
participation will be less demanding, but still critical, and will vary depending on the utility-specific gap
between the  initial conditions and the final planned capabilities of the contamination warning system.
Furthermore, it may be useful to obtain the help of consultants and contractors to aid the utility in the
implementation of an online water quality monitoring system, especially given that the level of effort
required to implement this component would go beyond the human resources at many utilities.
Consultants and contractors can be critical partners during each stage of the process by providing
May 2007
48

-------
                                Planning for WS-CWS Deployment

component-specific technical knowledge and installation experience, and by delivering training on the
new equipment, procedures and processes.

4.6.2  Cost Considerations

This section presents a summary of the design and implementation considerations discussed above that
may influence costs. This list also may include other factors that were encountered during
implementation of the initial Water Security initiative pilot and could be overlooked during cost
estimation in the absence of this experience. Although this list of cost considerations may not be
exhaustive, these factors, at a minimum, should be considered when planning.

Pre-Design:
    •   Development of design objectives for each element of the online water quality monitoring
       component
    •   Assessment of existing communications, SCADA and IT systems

Design:
    •   Development of preliminary concept of operations for the water quality monitoring component
    •   Monitoring Network:  assessment and calibration of hydraulic model used in the development of
       the monitoring network design, followed by field verification of potential monitoring station
       locations and development of installation specifications for selected locations
    •   Monitoring Station:  layout and design of multi-sensor instrument rack (or racks, if a tiered
       network approach is implemented)
    •   Communications and IT Architecture: design of communications and data management
       architecture; testing of communication pathways and conduct radio survey if necessary
    •   Water Quality Event Detection System:  design of the event detection system architecture; design
       and implementation of event detection system tool evaluation study

Implementation:
    •   Monitoring Network:  coordination of installation at non utility-owned monitoring station
       locations
    •   Monitoring Station:  equipment procurement and rack fabrication; transportation and installation
       of monitoring stations
    •   Communications and IT Architecture: establishment of contract with communication service
       provider(s); procurement, installation and configuration of communications and data
       management hardware and software
    •   Water Quality Event Detection System:  procurement, installation, and testing of event detection
       system hardware and software; training and configuration of event detection system tools

Preliminary Testing:
    •   Monitoring Station:  initial calibration and support; equipment shakedown and training on
       operation, maintenance and alarm response
    •   Communications and IT Architecture: test of installed data management architecture and
       complete communication system
    •   Water Quality Event Detection System:  preliminary evaluation of event detection system
       performance, and fine-tuning the configuration as necessary

Operations and Maintenance:
    •   Monitoring Network: water and sewer credits to non-utility hosts of monitoring stations
    •   Monitoring Station:  development of written documentation; scheduled and unscheduled sensor
       maintenance, repair and upgrades
May 2007                                                                                  49

-------
                                Planning for WS-CWS Deployment

    •  Communications and IT Architecture:  monthly service fees for communications service
       provider(s); development of written documentation; scheduled and unscheduled communication
       equipment maintenance, repair and upgrades
    •  Water Quality Event Detection System: scheduled and unscheduled EDS equipment
       maintenance, repair and upgrades

Evaluation and Refinement:
    •  Monitoring Network:  ongoing hydraulic model update and recalibration; potential relocation or
       addition of monitoring stations
    •  Monitoring Station: equipment upgrades due to improvements in sensing technology
    •  Communications and IT Architecture:  equipment upgrades due to improvements in
       communication technology
    •  Water Quality Event Detection System: drills and exercises, including simulations; data analysis;
       equipment upgrades due to improvements in event detection technology

Based on experience at the pilot utility, the most significant issues will likely be the capability of the
existing communication system and the upfront capital cost associated with the fabrication of the
monitoring stations. Unfortunately, it may not be possible to receive an accurate estimation of the costs
associated with each monitoring station until a prototype is created and unit costs can be calculated. The
use of a tiered monitoring station design may reduce the eventual equipment costs, but additional
prototypes would be necessary, maintenance and operation may be more complicated, and there will be a
reduced contaminant warning capabilities at locations with a simplified design. The end result should be
a balance between capability of the entire system and cost. Finally, the cost to operate and maintain even
a modest water quality monitoring network can be significant, and should be considered early in the
planning  stage to ensure that the system built can be sustained.
May 2007                                                                                  50

-------
                                  Planning for WS-CWS Deployment
                     Section 5.0:  Sampling  and Analysis
Although a critical aspect of contamination warning systems, sampling and analysis is not considered an
early detection strategy.  Rather, sampling and analysis serves the following three functions:
    •   Baseline monitoring.  Data from samples collected and analyzed during the initial stages of
        implementation are used to create a "baseline" profile of contaminant occurrence in the
        distribution system, as well as characterize possible matrix effects on method performance.
    •   Maintenance monitoring. Ongoing sampling analysis activities maintain laboratory proficiency
        for techniques that may otherwise only be used in response to a triggered event and are used to
        continually update baseline data seasonally.
    •   Triggered sampling and analysis.  Sample collection and analysis in response to contamination
        indicators by other contamination warning system components is part of the credibility
        determination and consequence management process.  Triggered analyses may be specific, based
        on information available from other components, or may involve a broad screen to potentially
        detect unknown contaminants.
Table 5-1 summarizes design and implementation considerations for the sampling and analy
component of a contamination warning system.

Table 5-1. Design Basis Considerations for Sampling and Analysis	
                                                                        sis
   Design
  Objective
            Description
    Design and Implementation Considerations
Capability
Can positively identify the presence of
any contaminant in the suite of target
analytes and above a well-defined
minimum reporting level.	
Assess existing laboratory capability and capacity and
identify enhancements or establish laboratory networks.
Contaminant
Coverage
High detection potential for classes 1, 2,
3, 4, 7, and 12; Moderate detection
potential for classes 5, 6, 8, 9, 10, 11.
Methods and analytical approaches should be identified
for as many of the contaminant classes as possible.
Methods should be validated for use in drinking water.
Spatial
Coverage
Function of location, number, and
density of sampling stations, as well as
sample type (composite vs. grab).
Consider sampling at locations identified as priority
sites through sensor network design (Section 4.1) as
these could be the source of triggered sampling events.
Also consider aspects of the distribution system such
as source water, water age, and pipe material that
could result in variability or method interferences.	
Timeliness
Function of sampling & analysis
frequency and the total time to process
the sample and analyze the results.
Baseline monitoring should occur at a frequency
sufficient to support data quality objectives, inform the
design of maintenance monitoring and provide an
understanding of method performance and variability in
the distribution system prior to full  operation of the
contamination warning system.  Maintenance
monitoring should occur at a frequency sufficient to
maintain laboratory capabilities, update baseline data to
account for seasonal variability, and support dual-use
applications.	
Reliability
Function of the reliability of sampling and
analysis methods (high for established
techniques).  Baseline needed for
reliable interpretation of results.
Methods utilized for baseline, maintenance, and
triggered sampling and analysis should be validated for
use in drinking water.  In some cases where analytical
methods have not been fully validated for certain
classes of contaminants, procedures to demonstrate
initial and ongoing proficiency should be implemented
to support interpretation of results.	
Sustainability
Provides utility with an opportunity to
exercise sampling and laboratory
protocols and may; provide information
about previously unknown contaminants
that occur in the system.	
Assess whether or not to enhance in-house laboratory
expertise or rely on outside laboratories for support.
This decision will influence level of effort and costs.  As
indicated previously, dual use applications should also
be considered in terms of sustainability of the program.
May 2007
                                                                                 51

-------
                                Planning for WS-CWS Deployment

The remainder of this section is organized according to the following design elements described:
    •   Laboratory capability and capacity (Section 5.1).  This includes consideration of the
       contaminants to monitor, analytical laboratories that would be able to support both routine and
       non-routine analyses, and analytical methods and data quality objectives.
    •   Sampling and analysis activities (Section 5.2). This includes consideration of sampling
       locations, sampling frequency, and ongoing sampling and analysis procedures.
    •   Site characterization and field screening (Section 5.3). This includes consideration of the roles
       that will be played by the utility and others investigating an incident, and consideration of the
       field testing capabilities that will be needed to conduct preliminary assessments of contaminants.

Discussion of these design elements is presented in the phases of pre-design, design and implementation,
and available tools and resources. Section 5 concludes with a discussion of staffing and cost
considerations.


5.1    Laboratory Capability and Capacity

Establishing and maintaining adequate laboratory capability to address a range of potential contaminants
is a fundamental aspect of contamination warning system design and implementation. Although some of
capabilities may be met with an existing or expanded utility laboratory, analyses for at least some
contaminant classes should be performed by other laboratories and coordinated by the utility. In addition,
sufficient capacity for these analyses should be considered to ensure that a large number of samples  does
not overwhelm the laboratory performing the analyses.
5.1.1  Pre-design

During the pre-design phase for laboratory capability and capacity, the utility's design objectives should
consider the following:
    •  Identifying potential contaminants of concern
    •  Identifying qualified laboratories that can support (and will commit to supporting) sample
       analysis needs that cannot be met through existing or expanded utility laboratory capabilities
    •  Identifying the analytical methods that will be used for each of the targeted analytes
    •  Evaluating the capabilities and credentials of each laboratory with respect to the identified
       methods to identify the most effective assignment of laboratory roles

Identifying Potential Contaminants of Concern

The sampling and analysis component should address the contaminant classes presented in Table 5-2. As
noted in this table, consideration should be given for an analytical approach for each contaminant class.
Although a goal of implementation is to maximize the number of analytes monitored in each class, there
are inherent limitations in monitoring capabilities for some contaminant classes due to the need for
specially equipped laboratories and the proximity of, or access to, these laboratories by the utility.

Table 5-2.  Considerations for Analytical  Approach to Establishing Sampling and Analysis
     Capabilities by Contaminant Class
Class
1
2
Description
Petroleum products
Pesticides (with odor or taste)
Example
Contaminants
Diesel
Aldicarb,
fenamiphos,
cyanide salts
Considerations for Analytical Approach
Screening for volatile and semivolatile organic
compounds1
Various methods may be applicable, depending
on targeted contaminants, including: liquid
chromatography, gas chromatography, and
spectrophotometric methods
May 2007
52

-------
                                 Planning for WS-CWS Deployment
Class
3
4
5
6
7
8
9
10,11
10,11
12
Description
Inorganic compounds
Metals
Pesticides (odorless)
Chemical warfare agents
Radionuclides
Bacterial toxins
Plant toxins
Pathogens (select agents)
Pathogens (non-select agents)
Persistent chlorinated organic
compounds
Example
Contaminants
Arsenite salts,
strychnine
Mercuric chloride
Sodium
fluoroacetate
VX
Cesium-137
Botulinum
Ricin
Bacillus anthracis
Vibrio cholerae
PCBs
Considerations for Analytical Approach
Various methods may be applicable, depending
on targeted contaminants, including inductively
coupled plasma/mass spectrometry (ICP/MS)
and chromatography with different detectors,
Screening for heavy metals using ICP/MS
Ion chromatography with conductivity detection
(method is currently being validated)
Analysis by a surety laboratory with access to
restricted standards
Screening for alpha, beta, and gamma emitters
Analysis of the toxins and pathogens currently
addressed by the state public health laboratory
participating in the Laboratory Response
Network (LRN)
Analysis for a minimum of two non-select agents
by procedures recommended in EPA SAM2
Gas chromatography/mass spectrometry
methods
  Data quality objectives for the chemical analyses should include detection at the parts per million level for specific
  chemicals (e.g., aldicarb, rather than total organic carbon)
2 EPA Standardized Analytical Methods for Environmental Restoration following Homeland Security Events (SAM)

A candidate list of target contaminants can be developed from the information in Table 5-2 and the
additional details on specific contaminants and methods from the references listed in Section 5.1.3. Of
these references, EPA's Water Contaminant Information Tool (WCIT) and EPA's Standardized Analytical
Methods for Environmental Restoration following Homeland Security Events (SAM) are particularly
useful for identifying contaminants of concern in drinking water.

Identifying Laboratories to Support the Contamination Warning System

It is unlikely that a water utility laboratory will have the capability to analyze for all of the target
contaminants on the list that is developed. To meet the design objectives of the sampling and analysis
component, the utility should establish relationships with multiple laboratories, potentially including
commercial  laboratories, municipal laboratories, and state laboratories.

Ideally, these relationships will be in the form of contracts or purchase orders. This is the likely option
for relationships with commercial laboratories, but also may be an appropriate vehicle for accessing local
or state  laboratory capabilities. Alternatively, an interagency agreement or memorandum of
understanding may be needed.  Note that developing and executing these contracts or vehicles is
addressed during design and implementation (Section 5.1.2).  During the pre-design phase, the goal is
simply to identify the laboratories that will be part of the analysis network, work with the laboratories to
establish roles, and agree upon the vehicle that will be used to access their capabilities.

To  ensure timely results, utilities should  identify labs within close proximity to the utility, when available.
The utility also should ensure that laboratories are qualified to perform analyses when these analyses  are
monitored under laboratory oversight programs. For regulated drinking water contaminants, laboratories
should be certified through the EPA Drinking Water Laboratory Certification Program or the National
Environmental Laboratory Accreditation Program (NELAP). For analyses of contaminants addressed by
other programs, such as select agents under the Centers for Disease Control (CDC) Laboratory Response
Network (LRN), the laboratory should be approved under these programs. Laboratories should provide
documentation of certifications and accreditations. Some commercial laboratories may hold NELAP
accreditation for non-regulated contaminants of concern.

Determining the analytical support that each laboratory would provide can be approached using the
following steps:
May 2007
53

-------
                                Planning for WS-CWS Deployment

    1.   Determine which contaminants can be analyzed using the utility's current in-house laboratory
        capabilities and capacity
    2.   For the remaining list, determine which contaminants can be analyzed by the in-house laboratory
        by expanding capabilities (acquiring instrumentation, or certifications) or capacity (additional
        staff or training)
    3.   For the remaining list, determine which contaminants can be contracted to one or more
        commercial laboratories
    4.   For the remaining list, determine which contaminants can only be addressed through support from
        local or state public health or environmental laboratories

Due to security restrictions, handling requirements, facility containment requirements, or instrumentation,
only a limited community of laboratories will have the capability to perform analyses for toxins, select
agents, radiochemicals, and chemical warfare agents. It may be difficult to identify a support laboratory
in close proximity of the utility. This gap in analytical capabilities should be considered as part of the
overall contamination warning system. It may be desirable to enhance credibility determination capability
through other components.

Identify Analytical Methods

After establishing the target list of contaminants and laboratories, analytical methods should be selected
for contaminants for which multiple  method options are available. The utility should consider using the
following steps to identify the most appropriate method to use:
    1.   Determine if there is an approved EPA method for measurement of the targeted analyte(s) in
        drinking water
    2.   If there is no approved EPA method, consult SAM as a resource to identify  methods (Section
        5.1.3)
    3.   If SAM does not recommend a method, consider other methods for measurement of the analyte of
        interest that have been validated for use in drinking water (potentially validated in other matrices
        if there are no drinking water methods available). In some cases, a validated analytical method
        may not be available or appropriate for use in routine monitoring for a given contaminant or
        class.  For these instances utilities should identify opportunities to participate in method
        validation studies organized by EPA or other organizations to address this gap.

Regardless of the validation status of a method, the most critical factor in ensuring that data are of known
and documented quality is ongoing assessment of method performance in the laboratory performing these
analyses for the contamination warning system. This assessment depends on the contaminant and
technique, and may include initial and ongoing spiked samples and blanks to assess bias, precision,
sensitivity, and contamination.

The utility also should consider selecting methods that can be used to simultaneously measure several of
the targeted analytes to improve analytical efficiency.  Analytical techniques  using chromatography
and/or mass spectrometry are examples.

For both routine and triggered method selection, both screening capability and confirmation capability of
candidate methods should be taken into account. For instances where the likely contaminant has not been
identified, specificity and reliability  of data may not be as critical as broad detection and rapid results.
For instances where a specific contaminant is suspected, specificity and reliability is paramount.

Establishing Laboratory Roles

When multiple laboratories are capable of performing a method, the following factors should be
considered in determining which laboratory should be assigned responsibility for the analysis:
    •   Is the laboratory certified/accredited/approved for the method in drinking water?
    •   Does the laboratory's quality assurance (QA) program for the method include the following:

May 2007                                                                                    54

-------
                                Planning for WS-CWS Deployment

           o   Sample receipt, storage and tracking protocols to chain of custody
           o   Initial and ongoing proficiency testing and quality control (QC) analyses for all analytes
               and methods of interest
           o   Data review procedures
           o   Data storage and transfer
           o   Documentation of personnel qualifications and training
           o   Ability and willingness to analyze samples potentially containing unknown contaminants
    •  If no certification/accreditation/approval is available for the method, is the method currently
       performed routinely by the laboratory?
    •  If the method is not routinely used by the laboratory, is the method a modification of a method
       the laboratory routinely uses?

5.1.2  Design and Implementation Approach

Design and implementation of the approach determined through the contaminant and laboratory selection
process should include the technical considerations and specifications described below.

Design:
    •  Develop contractual agreements and/or memoranda of understanding (MOUs) with laboratories
       external to the water utility
           o   Number and frequency of samples, by method
           o   Initial and ongoing QC requirements
           o   Data reporting requirements (including data elements to report, data reporting forms or
               transfer protocol, and results turnaround time)
    •  Address logistics with off-site laboratories
           o   Sample transport approach  (i.e., lab personnel, FedEx, contracted courier)
           o   Establish a point-of-contact at the utility and at all support laboratories

Implementation:
    •  Procure instrumentation, reagents, and  supplies for any expanded in-house capabilities
    •  Sign contracts and/or MOUs with external laboratories
    •  Develop SOPs for new activities
    •  Conduct initial demonstration of capabilities for new methods used at utility laboratory and/or
       external laboratories
    •  Implement QA plan (addressed in Section 5.2.2)

Start-up and Baseline Sampling and Analysis:
    •  Identify method performance problems or matrix interference/inhibition issues.  For example,
       high levels of background  organisms may impact pathogen method performance at a given
       location.
    •  Resolve startup issues at laboratories that will likely arise (QC issues, documentation
       completeness issues, sample transfer and receipt procedures and integrity issues, sample and data
       flow)

Operation and Maintenance:
    •  Verify ongoing acceptable performance by laboratories using proficiency testing (PT) samples
       analyzed under established certification/accreditation/approval programs
    •  Track availability of new, validated methods for contaminants of interest

Evaluation and Refinement:
    •  Assess ongoing acceptable method and laboratory performance on drinking water samples from
       the utility based on QC samples and PT samples


May 2007                                                                                   55

-------
                                Planning for WS-CWS Deployment

    •   Consider implementation of expanded analytical capability based on availability of new
       analytical methods

5.1.3  A vailable Tools and Resources

The following tools and resources are available to support design and implementation of laboratory
capability and capacity as part of a contamination warning system:
    •   American Association for Laboratory Accreditation.  Contains a list of NELAC approved PT
       sample providers, http://www.a2la.org/dirsearchnew/ptproviders.cfm
    •   Association of Public Health Laboratories (APHL). Information provided to improve the
       capacity and capability of public health laboratories in their response to biological, chemical, and
       radiological threats, as well as other public health emergencies.
       http ://www .aphl .org/programs/emergency_preparedness
    •   CDC-Select Agent Program. Centers for Disease Control and Prevention regulates the
       possession, use, and transfer of select agents, available at: http://www.bt.cdc.gov
       EPA Analytical Methods for Drinking Water. Information on sources of methods as well as
       links to  the various organizations which distribute them.  Environmental Protection Agency,
       November 2006. http://www.epa.gov/OGWDW/methods/methods.html or
       http://www.epa.gov/waterscience/methods
    •   EPA Drinking Water Certification Program,  http://www.epa.gov/safewater/labcert/index.html
    •   EPA Laboratory Compendium. Published laboratory analytical methods that are used by
       industries and municipalities to analyze the chemical and biological components of wastewater,
       drinking water, sediment, and other environmental samples that are required by regulations under
       the authority of the Clean Water Act (CWA) and the  Safe Drinking Water Act (SDWA). Almost
       all of these methods are published as regulations at Title 40 of the Code of Federal Regulations
       (CFR).
    •   EPA Standardized Analytical Methods [SAM] for Environmental Restoration following
       Homeland Security Events REVISION 3.0, EPA/600/R-07/015
       http: //www .epa. gov/nhsrc/
    •   Laboratory Response Network (LRN).  use this tool to identify laboratories close enough in
       proximity that could serve as  contract laboratories to  analyze samples potentially containing
       contaminants of interest, available at:
       http://www.bt.cdc.gov/lrn
    •   NELAC Institute. Contains a listing of NELAP accredited labs, http://www.nelac-institute.org/
    •   National Environmental Methods Index (NEMI). Use NEMI to compare and contrast the
       performance and relative cost of analytical, text, and  sampling methods for environmental
       monitoring.  US Geological Survey, Environmental Protection Agency
       http: //www .nemi. gov/
    •   Water Contaminant Information Tool (WCIT). Secure (password protected), on-line database
       that provides current, reliable information on chemical, biological, and radiological contaminants
       of concern for water security. Environmental Protection Agency, December 2006
       http://www.epa.gov/wcit

5.2    Sampling and Analysis
After target contaminants and support laboratories have been identified, the utility should develop and
implement an approach for baseline, maintenance, and triggered sampling and analysis.  This ultimately
should be documented in a comprehensive sampling and analysis plan, as indicated in Section 5.2.2.
However, this process can begin with consideration of the factors that should be documented in this plan,
as discussed in Section 5.2.1.
May 2007                                                                                 56

-------
                                Planning for WS-CWS Deployment

5.2.1   Pre-design

During the pre-design phase for sampling and analysis, the utility's design objectives should consider, but
not be limited to:
    •   Sampling locations
    •   Sampling frequency
    •   Sampling procedures

Sampling Location Considerations
In addition to adapting the approach used for locating sensor stations (Section 4.1) to identify sampling
points for baseline and maintenance monitoring, the following factors should be considered in selecting
potential sampling locations:
    •   Proximity of location to utility (a sampling location situated further from the utility may have
        greater vulnerability, such as potentially decreased chlorine residual)
    •   Accessibility of location (ease of access for samplers to collect large samples and to gain access
        to sample collection location)
    •   Percentage of output from each plant in system (consider increased sampling of key locations that
        receive finished water from plants with a higher relative output)
    •   Age of the water being sampled (extremely aged water may potentially have a decreased chlorine
        residual, and/or increased potential for the presence of biofilms compared to water that is 1 day
        old)
    •   Age and composition of piping that finished water has passed through prior to being sampled
    •   Open source of finished water (an open reservoir of finished water could prove to be more
        vulnerable than an underground holding tank)
    •   Locations where backflow into the system could pose a threat
    •   Other key locations, such as fire stations, elevated tanks, and pump stations
    •   Other key locations as identified through assessments for online water quality (Section 4.1)
        and/or enhanced security monitoring (Section 6.1)

Because triggered samples may come from anywhere in the distribution system, a primary goal of
location selection for baseline monitoring should be to collect water from locations that is representative
of as large a region of the distribution system as possible. The first set of analyses of a triggered sample
will likely include the same methods and procedures that are used for baseline monitoring

The utility also should plan to sample from key locations throughout the distribution system so that data
can be compared to finished water data collected at the treatment plants. The utility may consider using
the each source water treatment plant as a control by which to compare all data from baseline and
triggered sampling events. The purpose of baseline sampling from multiple and diverse locations is to
determine if the water is homogeneous with respect to contaminants detected, levels detected, frequency
of detections and method performance. Data should only be pooled from multiple locations if it is
scientifically justifiable to do so. This determination will involve statistical analysis of data.

Sampling Frequency Considerations

To support development of representative and complete baseline data, sampling frequency  should
consider the following:
    •   Size of the utility's distribution system and service community
    •   Flow rates through various parts of distribution system
    •   High pressure points within the system
    •   Seasonal affects (i.e., increased intake of surface water into plant due to rainy season or snow/ice
        melts)
    •   Changes in water source (if different sources are used during different times of the  year)


May 2007                                                                                   57

-------
                                Planning for WS-CWS Deployment

    •  Frequency of customer complaints (i.e., increased customer complaints during stagnant, hot
       seasons)

The utility may choose to collect samples more regularly while establishing a baseline for the potential
contaminants, and then adjust the sampling schedule to sample sufficiently to maintain sampling and
analysis response capabilities, continually update baseline data and to address seasonal or other issues that
may influence change  in baseline levels of contaminants that already are present in the system.

Considerations for New or Modified Sampling and Analysis Procedures
Some aspects of sampling and analysis for the contamination warning  system will be different from those
used routinely for compliance monitoring. The utility should consider the following factors when
planning for these new procedures:
    •  Procedures, training, and equipment to concentrate large bulk samples for select pathogen and
       toxin analysis  for ease of transport to the  state public health/LRN laboratory (volumes up to 100
       L may need to be collected at each sampling location).
    •  Additional safety equipment that may be  needed to protect samplers from exposure to potential
       contaminants (e.g., goggles, gloves, face  shields, laboratory coats).
    •  Procedures for ongoing field methods QC to aid in confirmation of site characterization
       determinations; this should include corrective actions for QC failures
    •  Procedures and training on proper chain of custody and evidentiary sample handling training to
       ensure that sample documentation is addressed properly from sample collection, shipping to
       method support lab, receipt at method support lab and throughout handling at the method support
       lab during analysis
    •  Training on collection, packaging, and transport/shipping procedures for drinking water samples
       that may contain disease causing agents or materials considered to be hazardous by commercial
       shippers
    •  Training of utility laboratory staff on new analytical capabilities.
    •  Procedures for ongoing analytical QC for new methods to assess data quality; this should include
       corrective actions for QC failures
    •  Procedures for data review for each new method (both those added to in-house laboratory
       capabilities as well as those performed by external laboratories)
    •  A response protocol in the event of a positive result

5.2.2  Design and  Implementation Approach

Design and implementation of the sampling and analysis activities determined through the pre-design
process should include the technical considerations and  specifications described below.

Design:
    •  Develop a preliminary concept of operations that describes the process flow for routine sampling
       and analysis and establishes roles and responsibilities
    •  Establish sampling locations and a baseline monitoring sampling schedule based on pre-design
       considerations. Sampling design should strive to collect data to address both spatial differences
       ("snapshot" differences from samples collected  over a short period of time) and longer term
       trends.  Snapshot differences may be differences between sampling locations and treatment plants
       and between sampling locations when collected over a short interval (such as 1 month). Longer
       term trends (sampling location-specific trends and regional aggregate trends) is useful only when
       sampling continues over a period of time in which a trend may be anticipated. For water utilities,
       that period of time may be as long as 1 year to capture seasonal and operational changes.
    •  Analyze historical data for any contaminants of concern based on contamination warning system
       objectives to inform sampling and analysis plan
    •  Develop standard operating procedures for sampling that address the following:

May 2007                                                                                   58

-------
                                 Planning for WS-CWS Deployment

           o  Sample collection procedures, containers, and preservatives
           o  Safety equipment use
           o  Sample packaging and transport
    •   Develop or modify existing chain-of-custody, data reporting forms, or other utility forms needed
        to meet new sampling and analysis needs unique to the contamination warning system
    •   Identify field sampling equipment that should be procured
    •   Identify laboratory instruments that should be procured
    •   Develop list of field sampling supplies that should be procured and stocked for ongoing sampling
    •   Develop list of laboratory supplies that should be procured and stocked for ongoing analyses
    •   Develop a QA Project Plan that addresses the following:
           o  Sampling QA/QC (e.g., type and frequency of field QC to include in sampling activities)
           o  Analytical QA/QC
           o  Data review procedures, and corrective actions in the event of QC failures.

Implementation:
    •   Procure field sampling equipment
    •   Procure laboratory instruments and establish service agreements
    •   Procure field sampling supplies that should be procured and stocked for ongoing sampling
    •   Develop list of laboratory supplies that should be procured and stocked for ongoing analyses
    •   Conduct training on sampling procedures
           o  Specialized procedures for sample collection
           o  Safety considerations
           o  Disease causing and hazardous materials packaging and shipping procedures
    •   Establish "notification levels" for each contaminant; further action will be needed if a
        contaminant is detected above this level. Until a baseline level of contaminants is established,
        these notification levels should be based on relevant data from studies conducted by the utility, as
        well as health advisory and other levels available through the sources listed in Section 5.2.3. After
        baseline sampling analysis is conducted, these notification levels should be adjusted.
    •   Consider the sources of data that can help establish a baseline of contaminant levels. Table 5-3
        provides examples of sources to consider.
    •   Revise concept of operations based on design and implementation activities

Table 5-3.  Examples  of Baseline Data Sources	
                                           Baseline Data
      Historical data for specific contaminants and water quality from the following:
       1. the point of entry to distribution system
       2. within the distribution system	
 2
Method performance data at the treatment plant and throughout the distribution system for specific
contaminants (spike recoveries, interferences, matrix effects).	
 3
Targeted contaminants detected at treatment plants and from individual sampling locations within the
distribution system	
 4
Non-targeted contaminants that are detected at the treatment plants and within the distribution system
 5
Tentatively identified compounds (TICs) that are detected at the treatment plants and within the distribution
system.	
 6
Levels of contaminants detected (quantified and semi-quantified for targeted, non-targeted and TICs) at the
treatment plant and individual locations	
 5
Frequency of detections of targeted and non-targeted contaminants, as well as tentatively identified
compounds in treatment plant water, individual locations and multiple locations overtime	
 7
Contaminant specific control charts
      Trend charts
Start-up and Baseline Sampling and Analysis:
    •  In general, a baseline monitoring program should proceed through phases of activity, culminating
       in the development of a maintenance monitoring program.  Those phases are described below.
May 2007                                                                                      59

-------
                                Planning for WS-CWS Deployment

       o   Phase 1: SOPs and necessary resource document development for critical activities related
           to baseline monitoring should be developed.  Initial demonstrations of capability (IDC) and
           minimum reporting limits (MRLs) for each method and contaminant should be established.
           Data reporting requirements and protocols should be established.
       o   Phase 2: Following the development of SOPs and completion and review of IDC data,
           finished water at the treatment plants is analyzed with respect to contaminant occurrence
           (contaminants detected, levels detected and frequency of detections) and method
           performance. Finished water from the treatment plant serves as a benchmark for comparison
           of contaminant occurrence and method performance of water from the distribution system.
           All future baseline and triggered sampling events may include the  source treatment plant
           water as a control.
       o   Phase 3: Regular surveillance monitoring of strategic/priority locations should be initiated
           and conducted at regular intervals to establish baseline for these locations and to determine if
           there are seasonal or regional trends.
       o   Phase 4: A survey study should be performed to determine contaminant occurrence and
           method performance in the distribution system. Sample collection locations should be
           selected with the goal of achieving spatial coverage of the distribution system and to capture
           a wide range of conditions in water age, pressure zones and pipe material.  This phase of
           study is  designed to survey the distribution system for contaminant occurrence and method
           performance. The results from this survey study may result in the  design of Phase 5 (focused
           studies).
       o   Phase 5: Based on the results of Phases 2-4, short-term, focused studies may be conducted
           to look more closely at possible differences in contaminant occurrence or method
           performance within the distribution system.  If significant differences are found, these
           findings may influence the selection locations for maintenance monitoring.
       o   Phase 6: The final phase  should be the analysis of results from Phases 1 - 5 to establish the
           management, interpretation and use of baseline data and to establish a maintenance
           monitoring  program.  Emphasis should be placed on access and interpretation of baseline data
           during a triggered sampling and analysis event.
    •   Document progress of sampling and maintain communication with laboratories to ensure
       contractual obligations are being met
    •   Consideration for baseline data should include storing the data in a manner that allows easy
       retrieval and use for interpretation of data from triggered sampling  events.
    •   Consider adjusting sampling schedule after evaluating results from the first year of sampling and
       analysis
    •   Consider a shift from the baseline monitoring phase to the maintenance monitoring phase after a
       year of baseline sampling and analysis is  complete
    •   Finalize concept of operations document to reflect results of baseline sampling and analysis and
       transition to  maintenance monitoring

Operation and Maintenance:
    •   Development of a maintenance monitoring program should consider results and data evaluation
       from the baseline monitoring period as well as cost considerations and dual benefit. For
       contaminants that fall under drinking water regulation, less frequent maintenance monitoring may
       be warranted, whereas, contaminants that may not otherwise be monitored under any program
       may be given priority for more frequent maintenance monitoring.  Maintenance monitoring
       should strive to maintain capabilities and serve to collect data that may be used to update baseline
       data (Table 5-3).
    •   Maintain baseline data (i.e., tabular data,  control charts, method performance by location)
    •   Perform periodic maintenance of sampling equipment, as necessary to maintain response
       capabilities
    •   Restock sampling equipment as necessary (bottles, cubitainers) or laboratory supplies (reagents or
       consumables)

May 2007                                                                                  60

-------
                                Planning for WS-CWS Deployment


Evaluation and Refinement:
    •  Analyze baseline data to assist in determining when contaminant levels in triggered samples
       exceed baseline levels
    •  Evaluate ease of use of accessibility of baseline data during a triggered event
    •  Review target contaminant list and consider if new methods for additional contaminants are
       available and could be implemented
    •  Provide additional training to sampling teams and/or laboratory staff to address deficiencies
       observed during baseline/maintenance sample analyses (i.e., contaminated blanks, high false
       positives/negatives)

5.2.3  Available Tools and Resources

The following tools and resources are available to support design and implementation of sampling and
analysis as part of a contamination warning system:
    •  EPA Health Advisory Levels: http://www.epa.gov/waterscience/criteria/drinking
    •  EPA Region 3 Risk Based Concentrations:
       http://www.epa.gov/reg3hwmd/risk/human/index.html
    •  EPA Region 6 Human Health Medium-Specific Screening Levels (HHMSSLs):
       http: //www .epa. gov/earth 1 r6/6pd/rcra_c/pd-n/screen .htm
    •  EPA Region 9 Preliminary Remediation Goals (PRGs):
       http://www.epa.gov/region9/waste/sfund/prg/faq.htm
    •  EPA Response Protocol Toolbox (RPTB), Module 3 and 4, EPA-817-D-03-003,
       http://cfpub.epa.gov/safewater/watersecurity/publications.cfm
    •  EPA Water Training Opportunities, Workshops/Training, November 2006
       http://www.epa.gov/water/training.html
    •  National Laboratory Training Network (NLTN). Dedicated to improving laboratory practice
       of public health significance through quality continuing education.
       http: //www .phppo .cdc. gov/nltn/
    •  Training resources for packing and shipping etiologic agent samples and hazardous
       materials. Examples include:
       o   http://www.ercweb.com/classes/
       o   http: //saf-t-pak .com/
    •  The Water Quality Data Elements User Guide, http://acwi.gov/methods/

5.3    Site Characterization and Field Screening

Sections 5.1 and 5.2 address activities  associated with laboratory-based analyses performed on samples
transported from the field.  These activities should be addressed both on a routine basis and during a
response. Additional activities—site characterization and field screening—should be addressed in the
field specifically for response.

Site characterization is the process of collecting information at a site to support the evaluation of a
drinking water contamination threat. This process may include site evaluation, sample collection, and
field screening.  Field screening involves rapid sample testing in the field to evaluate any potential safety,
chemical, biological or radiochemical hazards present at the site and to provide the laboratory with
preliminary information that may help focus their analytical activities.

5.3.1  Pre-design

During the pre-design phase for site characterization and field screening, the utility's design objectives
should consider, but not be limited to:
May 2007                                                                                  61

-------
                                 Planning for WS-CWS Deployment

     •   The role that the utility will play in site characterization activities, versus roles by other
         organizations, such as law enforcement and hazardous materials response units
     •   Current site characterization expertise among these organizations, and coordination across these
         organizations, versus the expertise and coordination needed to fulfill each organization's role to
         fully and effectively evaluate a potential contaminant threat site
     •   Current field screening equipment and expertise, versus the equipment and expertise needed to
         rapidly test for contaminants in the field

Site Characterization
Details on the site characterization process for a potential water system contamination incident are
provided in the EPA Response Protocol Toolbox (RPTB). Site characterization is intended to provide
important information to guide activities not only of water utility managers and staff, but also external
first responders (such as local law enforcement and HazMat teams) and other government agencies that
may be involved (such as the FBI and the EPA's CID.  Information gathered during site characterization
is combined with other information to perform  a threat evaluation,  the results of which may feed back into
additional site characterization activities.

During pre-design, the utility should consider the following steps for addressing site characterization:
     •   Define the potential scope of site  characterization activities, based on the RPTB, the utility's
         own emergency response planning materials, or other resources (see Section 5.3.3).
     •   Identify the organizations that should be involved in site characterization. Although this will
         depend on the scope of the incident, the objective of pre-design is to identify a comprehensive
         list of these organizations.
     •   Work with these organizations to map roles for each site characterization activity to the
         responsible organization. The same activity also may involve different organizations, depending
         on the contaminant (such as a toxic industrial chemical versus a chemical warfare agent).
     •   Work with these organizations to assess the level of training and expertise of each to fulfill their
         role and identify shortfalls that should be addressed.
     •   Through the consequence management process (see Section  9), identify exercises and other
         opportunities to test and maintain a high level of coordination among the organizations that may
         be involved in site characterization.

Field Screening
Although field screening is one of many potential site characterization activities, it merits some specific
pre-design consideration because it relies on appropriate equipment and training to be effective.  Two
types of rapid testing should be considered:
     •   Testing of materials other than the water (safety screening) to determine whether the
         environment around the potential contamination site can be safely accessed (or accessed with
         appropriate personal protective equipment [PPE])
     •   Testing of the water (water screening) or other media (e.g., contents of discarded containers or
         suspicious residues) to determine whether samples can be safely handled or transported to
         laboratory(ies) for analysis and to provide the laboratory with preliminary information on
         potential contamination to help focus  subsequent laboratory analyses.

Field screening capabilities should address the target parameters presented in Table 5-4. Although one
goal of pre-design for field screening is to maximize the number or type of contaminants that can be
tested in the field, there are practical considerations that may limit this, including equipment cost and
initial and ongoing training time and cost.
May 2007                                                                                     62

-------
                                Planning for WS-CWS Deployment
Table 5-4.  Considerations for Contaminant Coverage for Field Screening
Screening
Type
Safety
Safety
Water
Water
Water
Water
Water
Water
Target
Parameter
Radioactivity
(alpha, beta,
and gamma)
VOC(PID),
LEL, CO,
H2S,O2
Cyanide
Chlorine
residual
pH/
conductivity/
ORP
Turbidity
Chemical
Warfare
Agents (VX,
sarin, etc.)1
Toxicity1
Considerations for Field
Testing Equipment
Geiger counters and
scintillators, equipment
that can distinguish
gamma/beta from
alpha/beta emissions
Multi-meters used for
confined space entry
(photoionization detector
with other meters)
In-field cyanide detector
(e.g., a colorimeter or
spectrophotometer)
In-field cyanide detector
(e.g., a colorimeter or
spectrophotometer)
Electrode detector
Turbidimeter; most
measure light scattering
M272 Water Testing Kit or
similar commercial
versions
Commercial toxicity test
kit
Considerations for Training
The level of sophistication and
expense vary widely. Vendor
training typically is required.
Some HazMat units may be
able to provide training
Instrument manual and
training videos can be
sufficient; training from an
experienced user is ideal
Instrument manual and
periodic QC samples are
usually sufficient
Instrument manual and QC
samples are usually sufficient
Instrument manual. None
beyond normal utility
procedures for these
measurements
Instrument manual and QC
samples are usually sufficient
Instrument manual and QC
samples are usually sufficient
Vendor training sometimes
required
Comments
May be expanded
to water testing
with a special
probe
Detects chemicals
in air
Tests water for
cyanide ion, but
not combined
forms
Absence of
residual chlorine
may indicate a
problem
Abnormal pH or
conductivity may
indicate a problem
High turbidity may
indicate a problem
May also detect
some pesticides
and common
chemicals
Should establish a
baseline
 Note: M272 and toxicity testing kits are time-consuming to use.  These kits are generally only used if initial
 screening is inconclusive, or if the situation indicates that these screening tests may be relevant.

A candidate list of target contaminants can be developed from the information in Table 5-4 and the
additional details on specific field screening equipment and capabilities available from the RPTB.

5.3.2  Design and Implementation Approach

This section addresses considerations  for design and implementation of site characterization and field
screening capabilities.
       Develop a customized site characterization plan based on the circumstances of the threat warning
       that integrates with the concept of operations and consequence management plan (Section 2.1 and
       Section 9, respectively). This customized plan may be adapted from a generic site
       characterization plan or as part of a response to a specific contamination threat.  It is impossible
       to predict every possible scenario, so it is best to specify example scenarios that each warrants a
       different level of response. The site characterization team uses the customized plan as the basis
       for reporting their observations/data at the investigation site.
       Develop a health and safety plan to address any concerns that may arise in the field
           o   Appropriate PPE
           o   Emergency call list of agencies and individuals that should be notified in an emergency
               (e.g., hospitals, HazMat, MEDTox, fire and police)
May 2007
63

-------
                                Planning for WS-CWS Deployment

           o   40-hour OSHA training
    •   Select Site Characterization Team and Team Leader
           o   Experience
           o   Training
           o   Availability
           o   Anticipated level of response. Note that outside agency intervention may be necessary
               (e.g., HazMat).
    •   Develop basis for personal protective equipment and field screening equipment
           o   Preliminary information from monitoring station, personnel, or customer complaint
           o   Direct experience and availability of various equipment types (utility vs. HazMat team)
           o   Suspected contaminant
    •   Select primary components of the sampling kit
           o   Bottle types and preservatives
           o   Number of containers needed
           o   Labels, Chain-of-Custody
           o   Shipping materials
           o   Name and address of receiving laboratory
    •   Assess resources available for expanded efforts

Implementation:
    •   Ensure equipment for use in the field investigation is available and ready
           o   Instrumentation is pre-calibrated
           o   Communication devices and power supplies are tested and operational
           o   Equipment for sampling (e.g., coolers, bottles, preservatives)
           o   Appropriate documentation (field logbooks, chain of custody, health and safety plans,
               standard operating procedures, etc.)
    •   Prepare lists of field trained individuals within the organization
           o   Prepare call lists for expanded situations (police, fire,  HAZMat, etc.)
           o   Schedule training at required intervals
                  •   Health & Safety
                  •   Use of equipment, instrumentation, sampling  procedures
                  •   Shipping hazardous materials
    •   Procure necessary instrumentation
           o   Schedule  routine maintenance and calibration
           o   Secure back-up parts, and consumables
    •   Prepare site investigation kits
           o   Personnel protective equipment
           o   Screening instrumentation
           o   Sampling kits
           o   Documentation
    •   Develop SOPs as necessary
    •   Develop working relationships with local fire, police and HAZMat personnel
           o   Meet with other organizations (Police, Fire, HazMat, etc.) to agree upon each parties
               roles and responsibilities in an expanded situation
    •   Perform background Site Hazard Assessment at representative collection sites

Start-up and Baseline:
    •   Site inspections to observe normal surroundings
    •   Establish initial and ongoing QC  requirements
    •   Establish background levels of contaminants
May 2007                                                                                   64

-------
                               Planning for WS-CWS Deployment

Operation and Maintenance:
    •   Maintenance and calibration of screening equipment as per manufacturer's requirements or
       specifications
    •   Periodic testing of instrumentation with independent measurements

Evaluation and Refinement:
    •   Track and evaluate the development of new field equipment technologies
    •   Provide additional training to address deficiencies observed during site characterization

5.3.3  Available Tools and Resources

The following tools and resources are available to support design and implementation of site
characterization and field screening as part of a contamination warning system:
    •   EPA Response Protocol Toolbox (RPTB), Module 3, EPA-817-D-03-003,
       http://cfpub.epa.gov/safewater/watersecurity/publications.cfm
    •   Resources for Strategic Site Investigation and Monitoring, United States Office of Solid
       Waste and EPA 542-F-01-030b, Environmental Protection Emergency Response September 2001
       Agency (5102G) http: //www. epa. gov/tio/
    •   Improving Sampling, Analysis, and Data Management for Site Investigation and Cleanup
       United States Office of Solid Waste and EPA-542-F-01-030a Environmental Protection
       Emergency Response April 2001Agency (5102G) http://www.epa.gov/tio/
    •   EPA Water and Wastewater Security Product Guide, September 2005,
       http://cfpub.epa.gov/safewater/watersecuritv/guide/index.cfm

5.4    Staffing and Cost Considerations
Planning for the implementation of the sampling and analysis component of a contamination warning
system will likely involve a more limited range of utility staff than other components, but consideration
should be given to both routine (baseline and maintenance) and triggered sampling events, the latter of
which may involve a wider array of utility staff. Cost factors will be driven by the analyses the utility
laboratory will perform, the role of commercial laboratories, and the cost basis for support by state or
local public health or other laboratories.
5.4.1  Staffing

Table 5-5 offers a quick overview of which staff may be necessary to design and implement an expanded,
multi-laboratory sampling and analysis program as part of a contamination warning system and during
which phases of implementation these personnel may be needed.

Table 5-5. Sampling and Analysis Staffing Considerations
Division or Department
Water Quality
Information Technology
Security/Risk
Administration
Implementation Stage
PD
X



D
X
X
X
X
I
X
X
X
X
PT
X



O&M
X
X


ER
X



Comments
Involvement in all aspects of sampling and
analysis activities
Support to data transfer and management
aspects of laboratory results
Site characterization support
Contracting vehicles or MOUs needed to access
external laboratory assets
May 2007
65

-------
                                Planning for WS-CWS Deployment

5.4.2   Cost Considerations

This section presents a summary of the design and implementation considerations discussed above that
may influence costs.  This list also may include other factors that were encountered during
implementation of the initial Water Security initiative pilot and could be overlooked during cost
estimation in the absence of this experience. Although this list of cost considerations may not be
exhaustive, these factors, at a minimum, should be considered when planning.

Cost considerations for activities performed by the utility:
    •   Additional staff to support sampling or analysis activities
    •   Field screening equipment
    •   Laboratory instruments
    •   Service and preventative maintenance contracts for new instrumentation
    •   Sample collection containers
    •   Additional laboratory reagents, standards, and disposables
    •   Safety equipment (e.g., face shields, respirators, flammable cabinets)
    •   Additional costs for hazardous and/or biological waste disposable (e.g., procure an autoclave,
        contract waste disposal)
    •   Training:
           o   Disease  causing agent and hazardous materials shipping
           o   Site safety (e.g., OSHA HAZWOPR)
           o   New laboratory instruments
           o   Field screening equipment
    •   Proficiency testing to  maintain certification/accreditation or demonstrate proficiency for methods
        that are not covered under the certification program
    •   Laboratory information management system changes
    •   Initial and ongoing  costs of special permits (e.g., CDC, USDA) that may be required

Cost considerations for use  of external laboratories:
    •   Costs that may be incurred based on a minimum vs. maximum number of samples analyzed (e.g.,
        cost per sample for  10 samples vs. 100 samples)
    •   Additional costs that may be charged if samples are received after hours, exceed daily capacity,
        require additional analyses
    •   Additional costs for sample processing (e.g., concentration for pathogen samples) at the
        laboratory, rather than in the field
    •   Transport costs (e.g., courier, shipping costs) based on proximity to the support laboratory

In the process of determining costs the utility should determine whether expanding in-house analytical
capability is cost effective and sustainable. This will not be an option for some contaminants (such as
analysis of select agents, which is restricted to CDC LRN laboratories).  However, expanding in-house
capabilities to analyze others may be more cost effective if this capability can not only address
contamination warning system needs, but also enable the utility to shift some current compliance
monitoring sample analyses from a commercial laboratory to the utility's in-house laboratory.
May 2007                                                                                   66

-------
                               Planning for WS-CWS Deployment
              Section 6.0:   Enhanced Security Monitoring

Enhanced security monitoring includes the systems, equipment, and procedures that detect and respond to
security breaches at distribution system facilities such as pump stations, reservoirs and storage vessels
that are vulnerable to contamination.  The monitoring strategy includes detection by physical security
systems such as alarms and cameras, witness accounts, notifications by perpetrators, media, and law
enforcement, as well as associated response methods. A security breach is an unauthorized intrusion into
a secured facility that may be discovered through direct observation, an alarm trigger, or signs of intrusion
(cut locks, open doors, cut fences).  Security alarms are a common threat warning for a utility but are
often unintentionally caused by routine operation and maintenance activities.  Actual security breaches
usually are the result of criminal activity such as trespassing, vandalism, and theft, rather than attempts to
contaminate the water. Under the contamination warning system model, enhanced security monitoring
should be designed to help discriminate between security breach alarms and notifications that may be
related to a contamination incident and those resulting from other activities. Table 6-1 summarizes
design basis considerations for enhanced security monitoring.

Table 6-1.  Design Basis Considerations for Enhanced Security Monitoring at Selected Sites
Design
Objective
Capability
Contaminant
Coverage
Spatial
Coverage
Timeliness
Reliability
Sustainability
Description
Can detect an intrusion that may
have provided the opportunity for
introduction of any contaminant.
Covers all contaminant classes.
Limited to those elements of
infrastructure for which physical
security can be monitored.
Function of the type of security
monitoring system and the time to
evaluate a security breach.
Can be a reliable means of
identifying an intrusion, especially
when these breaches may involve
contamination, such as in storage
tanks and reservoirs.
Provides utility with increased
physical infrastructure protection
and awareness. Reduces the
occurrence of nuisance tampering.
Design and Implementation Considerations
System should be capable of detecting and assessing
breaches that could provide access to the water supply
at distribution system facilities where contaminant
addition would impact a significant number of
customers.
System should detect and assess breaches at all
possible entry points, at facilities selected for security
enhancements, which could provide access to the
water supply, regardless of contaminant quantity, type
or method of injection.
Improvements should be focused on distribution system
facilities such as pump stations, wells, reservoirs, and
storage tanks where a large volume of water could be
contaminated and impact a significant number of users.
Service connections and hydrants should generally not
be considered due to the high number of nodes and the
low benefit/cost ratio of hardening those
appurtenances.
System should be designed such that alarms produced
allow responders to quickly make an assessment and
generate the proper response. For many facilities, use
of video to assess alarms may be critical.
System should be designed such that video images
and proper response procedures are used to minimize
false alarms.
System should utilize equipment that is robust, does
not have substantial maintenance requirements, and
does not produce frequent false alarms.
The overall objective of this section is to describe considerations and a process for design and
implementation of the enhanced security monitoring component of a contamination warning system to
guide planning activities. In planning for implementation of enhanced security monitoring, several key
design decisions should be made including the following:
    •  Facilities at which to install security enhancements
    •  Type of security enhancements for consideration (e.g., alarms, motion sensors, video)
    •  Prioritization framework for ranking sites and security enhancements
May 2007
67

-------
                                Planning for WS-CWS Deployment

    •   Communications architecture for transmission of data and alarms
    •   Approach for implementation
    •   Approach for operation and maintenance

A key objective of this section is to provide information to enable the reader to consider these decisions in
a systematic process. The primary design element for enhanced security monitoring is physical security
enhancements, along with integrated communications architecture and data management. Physical
security enhancements should focus more heavily on detection and assessment of a potential
contamination event at a facility and less on preventing the event from occurring due to challenges with
and feasibility of implementing improvements to prevent an adversary from gaining access to a facility.
The physical security enhancements  should be designed in conjunction with the design of the
communications architecture and data management to ensure the physical security system design
objectives are met. Communications architecture for enhanced security monitoring should be closely
coordinated with the  same communications systems for online water quality monitoring since they often
may share common systems. Unique considerations for enhanced security monitoring, particularly with
regard to the transmission and storage of video data should be identified and considered in the
development of a comprehensive  communications and data management architecture for these
components.


The initial assessment process for enhanced security monitoring is one of the most critical aspects of this
component of the contamination warning system.  The process is similar to the approach taken by most
water utilities in conducting their vulnerability assessments in response to the Public Health Security and
Bioterrorism Response Act of 2002 as described in Instructions to Assist Community Water Systems in
Complying with the Public Health Security and Bioterrorism Preparedness and Response Act of 2002
(USEPA, 2003). In contrast to the vulnerability assessment process which evaluated all possible threats
to the entire water utility, a distribution system physical risk assessment focuses only on intentional
distribution system water contamination. Findings from the previously completed vulnerability
assessments may be utilized to identify the critical facilities within the distribution system.  For some
contamination warning system components, the dual benefit of detecting accidental or naturally occurring
contamination events is possible.  In the case of enhanced security monitoring, the dual benefit is the
ability to detect all types of intrusions including those involving intentional contamination.

Table 6-2 provides example preliminary recommendations for security improvements for typical water
utility facilities. The actual recommendations should be utility-specific.  The process for selecting the
improvements is described in subsequent sections of this document.

Table 6-2.  Example Improvements by Water Utility Facility Type	
 Facility Type
                                    Typical Recommended Improvements
                   Construct a structure over vents to prevent addition of contaminants
                   Contact switches and alarms for hatches and access points
Finished Water
Reservoirs and
Ground Level     ,  Harden hatches and covers for which alarms can not be feasibly added
    ^           -  Cover or install barriers for overflow pipes that are vulnerable to contamination
                   Develop written procedures for isolating tanks and reservoirs in the event of suspected
                   contamination and provide training for these procedures
May 2007                                                                                   68

-------
                                 Planning for WS-CWS Deployment
Facility Type
Pump Stations
Elevated
Storage
Tanks
Typical Recommended Improvements
• Interior motion detectors to detect intruders entering through windows and vents in areas
that provide access to water pumps and pipes
• Contact switches for doors to detect intruders entering areas that provide access to
water pumps and pipes
• Camera(s) that are activated by motion detectors or contact switches
• Card access reader
• Video and communication interfaces
• Lighting improvements for camera systems where needed
• Develop written procedures for isolating and turning off pumps in the event of suspected
contamination and provide training for these procedures
• Interior motion detectors
• Audible alarms at facility
• Develop written procedures for isolating tanks and reservoirs in the event
contamination and provide training for these procedures
of suspected
6.1     Pre-design
Pre-design and planning for enhanced security monitoring involves the following:

    •   Determine design basis threat
    •   Develop preliminary facility list
    •   Site assessments
    •   Perform risk ranking to assess risk before improvements

To evaluate  how well existing and future proposed security systems and procedures protect facilities from
contamination, it is important to define the specific potential threats to those facilities. Before
determining the effectiveness of protection systems, agreement should be reached defining types and
capabilities of the adversaries who may attempt to contaminate the system.  This is important because the
effectiveness of a protection system can vary greatly depending upon the adversary.  For example, a
standard steel door with hinge protection should delay or defeat a vandal attempting to enter a facility, but
a sophisticated, trained adversary such as a terrorist armed with the proper tools and equipment could
defeat the door easily and quickly.  The same variance in effectiveness often applies to detection and
alarm systems. Of equal importance is defining the types and quantities of contaminants that may be used.
In order to evaluate the contamination risk of each facility, the design basis  threat should be defined. This
includes identifying the capabilities  of adversaries as well as the quantity and type of contaminants that
could be used to contaminate the water supply.

The recommended design basis threat is a highly sophisticated adversary or group of adversaries with the
resources and ability to access all contaminants represented by the classes listed in Table 1-1.  This type
of adversary has the ability to gain expertise in the areas of distribution system design, operation,
hydraulics, drinking water treatment, chemistry, microbiology, etc.

The next step during pre-design and planning is to preliminarily identify which utility facilities may be at
the highest risk for contamination based on factors including: consequence of contamination, site
location, access, and visibility. Selection of these preliminary sites can be facilitated through review of
previously conducted vulnerability assessments, which may have  characterized relevant attributes of each
facility even if the focus of the assessment was not on contamination.  If a distribution system model is
available, it  could be used to estimate the consequences of contamination at various distribution system
facilities.  The process for estimating consequences using a distribution system model is described below.
The preliminary list of facilities may be evaluated further through site assessments for potential enhanced
security monitoring upgrades.
May 2007                                                                                     69

-------
                                 Planning for WS-CWS Deployment

After the preliminary facilities list has been developed, the next step is to conduct detailed site
assessments of each facility on the list with a focus on existing physical security systems, communication
capabilities (i.e., transmission of alarms, video, etc.), proximity to the public, terrain, adjacent land uses,
site access, site lighting, alarm and detection systems, and physical barriers such as fencing and hardened
structures.  Using the observations made during the site assessments, an evaluation should be performed
regarding the possible modes of entry for contaminants (i.e., dumping contaminants directly into
reservoir, injecting into pipe tap, etc.) and volumes of contaminants that could practically be delivered to
each of the facilities and put into the water without arousing suspicion. For example, the site
configuration of some facilities may not allow a heavy truck to get close enough to the facility to deliver
1,000's of gallons of the contaminant, while other facilities may not be accessible at all by a vehicle in
which an adversary may be limited to quantities that could be transported by hand.

The contamination risk of each facility can be evaluated after facility assessments have been conducted
and the design basis threat has been defined.  The risk assessment provides the basis for prioritizing which
facilities should be considered for security improvements and which security improvements would be
most cost-effective in terms of ability to reduce risk of contamination. The contamination risk at a facility
is a function of three primary parameters:
    •   Effectiveness of the facility's existing physical security system
    •   Probability that the facility may be targeted by an adversary
    •   Consequences of a contamination event at the facility

There are several methods available to calculate risk.  The method most commonly used during the 2002
vulnerability assessment process was the Risk Assessment Methodology for Water Utilities (RAM-WSM)
developed by Sandia National Laboratory. A method very similar to RAM-WSM can be used to compare
the risk of intentional contamination at distribution system facilities.  Other risk assessment methods are
available including VSAT™ (Vulnerability Self-Assessment Tool) developed by the National Association
of Clean Water Agencies (NACWA).

The effectiveness of a facility's existing physical  security system can be assessed during the detailed site
assessment. The considerations used to estimate the effectiveness of a security system include detection,
delay, and response.  Traditionally, physical security systems (i.e., door alarms, motion detectors, etc.),
have been designed to detect a security breach early enough to provide adequate delay and allow
sufficient time for law enforcement to respond and prevent the adversary from completing their intended
act. However, for the case of potential contamination, the consequences can in some cases be quickly
eliminated or significantly reduced through an operational response triggered by a security alarm or
notification (i.e., isolating a finished water storage tank, shutting down pumps, etc.).

Estimating the relative probability that an adversary may attack one of a utility's facilities compared to
another may be somewhat difficult to  assess. However, it is likely that some facilities may be more
attractive targets  compared to others. Some aspects of the facility that could be used to estimate the
probability of attack include:
    •   Recognizability—How easy would it be to recognize a facility as a water utility facility that
       provides access to drinking water?
    •   Visibility to Surrounding Public—If a facility is visible to the public living and working near the
       facility, an attack on that facility may be deterred due to the increased probability of the public
       witnessing the attack.  However, a sophisticated adversary may not be deterred significantly from
       attacking a visible facility.
    •   Access to and Ability to Deliver Contaminant—This estimates how difficult it would be to get a
       contaminant to a facility and add it to the system.  This is  a function of the possible modes of
       entry for contaminants (i.e., dumping contaminants directly into reservoir, injecting into pipe tap),
       and volumes of contaminants that could practically be delivered to each of the facilities and put
       into the water without arousing suspicion. Also, some utilities may not provide vehicle access
       which would limit the amount of contaminant that could be delivered.


May 2007                                                                                     70

-------
                                Planning for WS-CWS Deployment

    •  Effectiveness of Attack—This estimates how effective a potential attack would be on a facility
       based upon the ability to add a sufficient amount of a contaminant to reach a lethal concentration
       in the distribution system.

The consequences of a contamination event occurring at a facility ideally should be estimated using the
utility's distribution system model and GIS to simulate how a contaminant would spread and how many
people would be affected by the incident. The following parameters should be considered in a modeling
analysis to estimate the resulting consequences of contamination at any distribution system facility:
    •  Volume of contaminant added (depends on contaminant and site characteristics)
    •  Concentration of contaminant added (depends on contaminant availability)
    •  Toxicity of contaminant (depends on contaminant type)
    •  Duration of contaminant addition (depends on site characteristics)
    •  Duration of model simulation (recommend minimum of 24 hours)
    •  Type of storage tank mixing modeled (i.e.,  completely mixed, plug flow, etc.)

The results of the site assessment, described previously, can be used to define and constrain the
contamination scenarios that are modeled (e.g., the  volume of contaminant that can be delivered to a
location, the duration of contaminant addition, etc.). Furthermore, additional understanding may be
gained by varying parameters such as the time of day that the contaminant is introduced at the facility
because as demand patterns vary over the course of a day, the system hydraulics may change, which may
impact the total number of exposures significantly.  The risk of contamination for each facility is
calculated utilizing the selected risk assessment equation and the estimated values for physical security
effectiveness, probability of an attack, and consequences of an attack.  Once calculated, the facilities
should be  sorted in order of highest to lowest risk.  Section 6.2 provides additional detail for designing
security systems for the selected facilities.

In addition, most of the physical security improvements have communication and data management
requirements that should be considered during pre-design. A robust, reliable, and secure architecture
should be  developed. The same system developed in Section 4.3 for online water quality monitoring may
in some cases be used for enhanced security monitoring. Utilities may decide, however, to have a
dedicated  architecture for enhanced security monitoring. Pre-design considerations unique to enhanced
security monitoring include the following:
    •  Determination of requirements of the communication system needed to support the proposed
       network of physical security improvements. Consider communication requirements for the online
       water quality monitoring stations deployed as part of the contamination warning system, and
       evaluate the feasibility of using a single architecture to support both.
    •  Assessment of existing communication  system architecture used to transmit data and commands
       between remote facilities and the utility central control location. Assess ability of existing system
       to accommodate the proposed enhanced security monitoring systems. Evaluate use of the
       existing data recording systems (e.g., SCAD A) for managing data from cameras, contact alarms
       and other remote devices. If existing communications systems and data recording systems are
       unable to meet the requirements, the utility desires to separate security data from process control
       SCADA systems, or the utility desires to transmit security data to locations (e.g., security guard
       station) outside the SCADA network, evaluate alternatives.  Identify constraints on
       communication alternatives such as hilly terrain that may make radio communications cost
       prohibitive. Alternative communication methods include SCADA, Tl lines, digital cellular
       services, and private radio network.
    •  A potentially significant data management  challenge for enhanced security monitoring could be
       the management of video data from remote sites. The ability to transmit video for intrusion alarm
       assessment can be the most demanding communications network requirement of the physical
       security improvements. The video transmission option that provides the best resolution with the
       quickest response is full streaming video over fiber optic lines. The installation of fiber optic
       lines to remote facilities is often necessary, making this option very costly.  The other video

May 2007                                                                                   71

-------
                                Planning for WS-CWS Deployment

       options are various technologies to compress and package video clips for transmission.
       Transmission options such as Tl line and digital cellular services were described above.

6.2    Design and Implementation Approach
Once ranked, the facilities are evaluated further to identify methods for reducing risk of contamination to
the facilities. To reduce risk, physical security system effectiveness should be increased or the
consequences or probability of attack reduced. To determine how security system effectiveness could be
increased, conceptual design and associated cost estimates of security improvements should be
completed. Table 6-2 showed a list of typically recommended improvements.  These recommendations
may, however, vary from utility to utility and may be dependent upon several factors that are unique to
each utility.

Methods to reduce risk by increasing physical security system effectiveness include increasing detection
and assessment capabilities, improving delay, and improving responses to an alarm.  Means to improve
the utility's security systems should include capital improvements, such as surveillance and monitoring
equipment and alarms, facility structural improvements, and procedure modifications. In addition,
response can be improved by developing written procedures for isolating each facility in the event that
contamination is suspected. It is recommended that these written procedures be developed  and the
procedures incorporated into standard operating procedures and consequence management training and
drills.

In addition to increasing the effectiveness of the physical protection systems, contamination risk can be
reduced by decreasing the consequences of a contamination event. If cameras are installed, the video
images can confirm that an alarm was caused by an intruder and the operator can take action to mitigate a
potential contamination event, including disabling pumps, closing valves, or changing the hydraulic grade
line to prevent the spread of a potential contaminant.  It is recommended that detailed procedures like this be
developed for each facility. In addition,  a timely "do not drink" or "do not use" order given to the public
would in many cases reduces the consequences of a contamination event.  Procedures and guidance for
issuing "do not drink" and "do not use" orders should be developed as part of the consequence
management plan.

Decreasing the probability of an attack, especially for a sophisticated adversary, may be difficult to
achieve but some options are available.  Removing or covering signs that identify the facilities  as a utility
facility could be considered. However, since the adversary may be sufficiently sophisticated to identify all
facilities, removing or covering signs is not recommended. For most facilities, it likely will not be possible
to make them more visible to the surrounding public to deter  attacks. However there may be facilities where
clearing brush or small trees or other Crime Prevention through Environmental Design (CPTED) strategies,
as discussed in Interim Voluntary Security Guideancefor Water Utilities (AWWA, 2004), may be
beneficial.

Barriers could be added at some facilities to prevent vehicles that could be used to  carry large quantities of
contaminants from entering the site. However, these facilities would still be subject to attack with smaller
quantities of the most potent contaminants. The effectiveness of barriers should be reviewed further during
design, although the cost of an effective barrier at even one facility may limit the budget available to
enhance security at other locations. To help prioritize improvements, the cost of improving security
systems at each site can be evaluated in the context of the associated risk reduction that would result from
making the recommended improvements.  Based on the preliminary design, planning level capital costs
should be  calculated for  each facility. Methods for calculating planning level capital costs are described
in detail in Section 6.3.2.

For each facility, the risk score following improvements is estimated and the difference between this
score and the original risk score  is calculated and the difference is expressed as risk reduction units
(RRU). The costs to benefits of for each facility are expressed in terms  of the capital costs of

May 2007                                                                                    72

-------
                                Planning for WS-CWS Deployment

improvements per RRU ($/RRU).  After developing the preliminary design for security improvements at
each facility, the risk of contamination for each facility should be recalculated.

The values of RRU are utilized in the facility prioritization process. If any modifications to the
preliminary design concepts are made as a result of the cost analysis or facility prioritization process, the
RRU should be recalculated accordingly. Once calculated, the list of facilities and associated cost benefit
ratios should be sorted in order of increasing $/RRU. From this list, the facilities that may receive
enhanced security monitoring improvements can be  selected based on the ranking and available budget
for enhanced security monitoring.

Design:
•   Establish goals for detection, delay and response that should help either prevent the adversary from
    successfully contaminating the water or mitigate the  consequences of a contamination event by
    successfully detecting the adversary and minimizing the response time. Security systems should be
    selected that will allow these goals to be met.
•   Coordinate  system design with the concept of operations as described in Section 2.1 to help establish
    the procedure aspects of the security systems and define the requirements for equipment and data
    systems.
•   Select, specify, and locate equipment type  and required equipment features based on factors such as
    building configuration and design criteria.  For example, for camera systems, the available line of site,
    required resolution and speed of data transmission must be considered.  For motion detectors, the
    range of detection, sensitivity and potential facility obstructions must be considered.

Once the design criteria and functional goals are defined, methods can be evaluated to achieve them. The
following are features and equipment that may be included in an enhanced security monitoring  system.
    •  Hardening - The focus of physical security improvements is on detection and assessment rather
       than delay. Consequently,  significant hardening of building structures such as doors and windows
       is generally not recommended, but expenditures  on facility alarms and cameras are
       recommended. Hardening of exterior reservoir access hatches and vents, however, is
       recommended where installing cameras or hatch/vent contact alarms is difficult or cost
       prohibitive.  Concrete or metal enclosures around hatches or vents can physically hide and/or
       make it very difficult to add  contaminates reservoirs through vents.
           o   Vents - design to make  contaminant addition difficult while still allowing ventilation
           o   Hatches- add contact alarms if routing electrical conduit feasible or bury/hide
    •  Facility access control - Access control systems  allow a means for employees to automatically
       disable  alarms upon entering a facility and provide records of employee egress.
           o   Numeric keypads
           o   Electronic locks - replace lock cylinders on doors and controls access on a per-key basis
               using programmable keys
           o   Coded credentials -  proximity cards
           o   Biometric devices -  fingerprint and iris scanners
    •  Contact switches - Contact switches are recommended for all exterior doors or for interior doors
       providing access to areas such as pump rooms where an  intruder could contaminate water.
       Exterior reservoir hatches  and valve vault hatches that provide access to water piping should also
       have contact switches.  Contact  alarms may be discrete for specific entry location identification or
       daisy-chained to provide a general facility intrusion alarm.
    •  Cameras - Contact alarms, motion detectors, or changes in camera image fields may be used to
       activate cameras to assess  the alarm. Pan-tilt-zoom cameras can cover more area than stationary
       cameras but should be able to quickly turn and focus to area of concern.  Camera field of view
       and distance from object impact the ability to identify intruders. Adequate lighting is necessary
       for assessment. Other factors to consider are resolution and lenses.
           o   Stationary
           o   Pan-tilt-zoom

May 2007                                                                                    73

-------
                                Planning for WS-CWS Deployment

           o   "Smart" cameras that recognize suspicious behavior and/or motion in a limited part of the
               view
    •  Lighting - Adequate lighting is extremely important when using cameras for identification
       purposes. Lamps should quickly illuminate to full brightness.
    •  Motion detectors - Motion detectors are used to detect intruders gaining access through windows,
       and storage tank ladders and standpipes. Interferences from shadows and other causes should be
       considered.
           o   Microwave
           o   Passive infrared (PIR)
           o   Microwave - PIR - Dual technology minimizes false alarms on ladders.
    •  Glass break sensors - Glass break sensors may be used in lieu of or in combination with motion
       detectors.
           o   Acoustic
           o   Shock
           o   Acoustic - Shock - Dual technology greatly reduces false alarms from background
               noises.
    •  Audible alarms - Audible alarms are mainly for deterrence. Impacts on neighbors may be a
       concern.
    •  Heavy duty conduit and integrity monitoring - Heavy duty conduit and integrity monitoring
       protect power and communications wiring.
    •  Obtain any required code reviews or approvals on the  design especially relating to egress.
    •  Video system types - Full streaming video provides real time capabilities but is often not a viable
       option due to the cost of installing the communications network. Two video packaging
       technologies include compressed video clips and flash memory. Compressed video clips systems
       are sent in packages over SCADA systems and typically includes digital video recorders installed
       in the monitored facilities to store images for potential criminal investigation. Flash memory
       systems include a video server to convert analog video signals to digital images and a processor
       to provide intrusion event video packaging and transmission. Flash memory may capture
       intrusion events but lacks the large video storage capabilities of compressed video clips.
    •  Video transmission - Good video resolution and transmission speed are the key requirements for
       rapid assessment.
           o   Tl lines - Determine capability and interface  requirements. Firewalls may be an issue.
           o   Private radio networks - Perform path measurements to locate repeater sites.
           o   Digital cellular service - Digital cellular service is a relatively recent technology
               development and is not available in all areas of the United States. It requires coordination
               with the local wireless carrier to determine the best way to link data from remote
               facilities to the command center.
    •  Multiple technologies may be desirable or necessary in some applications. For example, existing
       Tl lines may be  available for some facilities and would provide excellent video transmission and
       response. Since Tl lines are costly to install and may require relatively high monthly
       communication fees, alternative video transmission options should be considered for areas
       without Tl lines.

Implementation:
    •  Delivery Options:  Traditional, Design-build (develop 30 to 50% design documents and
       contractor completes design); hardware and equipment could be procured by the utility or
       supplied by the contractor as part of the construction contract.
    •  Pre-selection of eligible contractors
    •  Bid and award of contract
    •  Construction - inspection, contractor oversight, contract modifications
    •  Document security issues
    •  Select communications service provider (if applicable), and establish any necessary contractual
       relationships.

May 2007                                                                                   74

-------
                                Planning for WS-CWS Deployment
    •   Select an installer, and establish any necessary contractual relationships. Installer's experience
       with all the technologies needed to complete the work should be heavily weighted in their
       selection.
    •   Identify roles and responsibilities for procurement, installation, and testing of various components
       of the communications architecture.
    •   Work with service provider to get components installed and configured.
    •   Test communication pathways between enhanced security monitoring sites and the operations
       center.
    •   Develop location-specific installation specifications.
    •   Procure system components that may not be provided by the service provider.
    •   Install remote communications systems at selected sites.
    •   Refine concept of operations based on as-built design
    •   Training of security personnel regarding the new monitoring systems, how they should interface
       with them, and how they should respond to an alarm

Preliminary Testing:
    •   Initial configuration and calibration of enhanced security monitoring equipment, including
       verification of the following:
           o   Contact switches and glass break sensors activation and transmission
           o   Cameras work in conjunction with contact alarms
           o   Lighting works in conjunction with contact alarms and is adequate for video resolution
           o   Video transmission provides the speed and resolution specified for a full assessment by
               bench testing
    •   Troubleshooting
    •   Contractor functional and performance testing
    •   Refine concept of operations, if necessary, based on any modifications

Operation and Maintenance:
    •   In-house versus contracted O&M services
    •   Documentation including, as-built specifications, O&M manual, etc.
    •   Periodic maintenance and calibration, including consumables.
    •   Annual inspection and maintenance of contact switches, motion sensors, glass break sensors,
       cameras and other physical  security equipment.
    •   Unexpected maintenance events
    •   Amortized replacement costs

Evaluation and Refinement:
    •   Evaluate frequency and cause of false alarms
    •   Conduct drills and exercises to identify refinements to concept of operations and to assess and
       improve equipment performance and alarm response

6.3    Available Tools and Resources
The following tools and resources are available to support the design, installation, operation, and
evaluation of physical security improvement for the enhanced security monitoring component of a
contamination warning system:

    •   AWWA.  Interim Voluntary Security Guidance for Water Utilities, 2004.
    •   AWWA.  Guidelines for the Physical Security of Water Utilities, 2006.
    •   Garcia, Mary Lynn. The Design and Evaluation of Physical Protection Systems, 2001.
    •   The Integrated Physical Security Handbook. Philpot and Einstein, Homeland Defense Journal.
       2006. http://www.phvsicalsecuritvhandbook.org

May 2007                                                                                  75

-------
                                Planning for WS-CWS Deployment

    •   USEPA Water and Wastewater Security Product Guide.
       http://crpub.epa.gov/safewater/watersecurity/guide/
    •   Sandia Corporation. Risk Assessment Methodology for Water Utilities (RAM-W™), 2002.
    •   National Association of Clean Water Agencies (NACWA). Vulnerability Self-Assessment
       Tool™for Water & Wastewater Utilities (Version 3.2 Update), 2005.
       http://nacwa.org/pugs/index.cfm
    •   USEPA. 2003. Instructions to Assist Community Water Systems in Complying with the Public
       Health Security andBioterrorism Preparedness Response Act of 2002 (EPA 810-B-02-001).

6.4    Staffing and Cost  Considerations
Planning for the implementation of the enhanced security monitoring component of a contamination
warning system requires involvement of a wide array of utility personnel and potentially contractor staff.
Costs may be highly dependent on the utility's capabilities and intended enhancements. Therefore, the
remainder of Section 6.3 illustrates the staffing considerations and cost factors that are  recommended for
consideration during project planning and pre-design.

6.4.1  Staffing

As mentioned above, staffing considerations are critical to the successful implementation of a
contamination warning system. Table 6-3 offers a quick overview of which staff may  be necessary to
design, implement, and operate an online water quality monitoring program as part of a contamination
warning system and during which phases of implementation these personnel may be needed.

Table 6-3.  Enhanced Security Monitoring Staffing Considerations
Division or Department
Water Quality
Information Technology
Engineering and Planning
Operations and/or
Distribution
Security/Administrative
IT
Implementation Stage
PD
X
X
X
X
X
X
D
X
X
X
X
X
X
I
X
X
X
X
X
X
PT
X
X

X
X
X
O&M

X

X
X
X
E&R
X
X
X
X
X
X
Comments
Helps define monitoring system requirements
Designs, implements, and manages all IT systems
used to support communications and data
management needs of enhanced security monitoring
Review and approval of designs. Responsible for
installation oversight and inspections. Support
security system design.
Provide input during design. Supports selection of
distribution system security equipment and
improvements. Lead in operations and maintenance
of equipment. Support installation of security
systems.
Lead in selection of distribution system security
equipment and improvements. May be responsible
for monitoring and responding to alarms.
Designs, implements, and manages all IT systems
used to support enhanced security monitoring.
PD = Pre-design; D = Design; I = Implementation;
Maintenance; E&R = Evaluation and Refinement
PT = Preliminary Testing; O&M = Operations and
Building the team to implement this component of a contamination warning system should involve all
divisions within the utility. Therefore, it is important to have senior leadership involved and invested
during each stage to facilitate the resolution of cross-division issues.  Several key personnel, like the IT
manager and the Water Quality division manager, would ideally be members of this and all other
component teams to facilitate the application of the system engineering principles outlined in Section 2.1.
Such involvement can be a significant commitment of time and resources for these individuals, but the
utility can reap substantial benefit in the long-term success of the system. Other team members'
participation may be less demanding, but still critical, and may vary depending on the utility-specific gap
between the  initial conditions and the final planned capabilities of the contamination warning system.
Furthermore, it may be useful to obtain the help of consultants and contractors to aid the utility in the
May 2007
                                              76

-------
                                Planning for WS-CWS Deployment

implementation of an enhanced security monitoring system, especially given that the level of effort
required to implement this component would go beyond the human resources at many utilities.
Consultants and contractors can be critical partners during each stage of the process by providing
component-specific technical knowledge and installation experience, and by delivering training on the
new equipment, procedures and processes.

6.4.2  Cost Considerations

This section presents a summary of the design and implementation considerations discussed above that
may influence costs. This list also may include other factors that were encountered during
implementation of the initial Water Security initiative pilot and could be overlooked during cost
estimation in the absence of this experience. Although this list of cost considerations may not be
exhaustive, these factors, at a minimum, should be considered when planning.
    •   Development of design objectives for both elements of the enhanced security monitoring
       component
    •   Assessment of existing security, communications, SCADA and IT systems
    •   Development of preliminary concept of operations for the enhanced security monitoring
       component,  including decisions regarding design basis threat, facilities to be protected,
       monitoring equipment to be deployed and the approach for communicating and responding to
       alarms.
    •   Field verification of preliminary locations and development of installation specifications for
       selected locations
    •   Layout and design of security equipment and communications system components (for each site).
       Even if the enhancements are handled as a design-build, significant design work may be
       necessary to produce the preliminary drawings.
    •   Design of communications and data management architecture; testing of communication
       pathways and conducting  radio survey if necessary
    •   Installation of security and communications equipment at each site
    •   Establishment of contract with communication service provider(s);  procurement, installation and
       configuration of communications and data management hardware and software;  depending on
       existing infrastructure and topography, laying fiber optic cable or erecting antennas may add
       significant cost.
    •   Initial equipment shakedown and training on  operation, maintenance and alarm response
    •   Communications and IT Architecture: test of installed data management architecture and
       complete communication  system
    •   Development of written documentation; scheduled and unscheduled security monitoring
       equipment maintenance, repair and upgrades
    •   Service fees for communications service provider(s); development of written documentation
       related to communications architecture; scheduled and unscheduled communication equipment
       maintenance, repair and upgrades
    •   Equipment upgrades due to improvements in sensing  and or communication technology
    •   Drills and exercises

Based on experience at the pilot utility, the most significant issues may likely be the capability of the
existing communication system and the capital costs associated with the installation of the security
monitoring equipment. Finally, the cost to maintain and upgrade security monitoring equipment can be
significant, and should be considered early in the planning stage to ensure that the system implemented
can be sustained.
May 2007                                                                                  77

-------
                               Planning for WS-CWS Deployment
           Section 7.0:  Consumer Complaint Surveillance

Located throughout a utility's distribution network, consumers can provide near real-time input regarding
changes in water characteristics discernable through the senses. Consumers may detect contaminants
with characteristics that impart an odor, taste, or visual change to the drinking water.  Complaints from
residential, commercial and industrial consumers are routinely reported to water utilities on a very timely
basis. As such, consumer complaints may provide one of the earliest warnings of a possible
contamination incident for contaminants in classes 1 through 5, if an effective system is in place to detect
anomalous trends in complaints and quickly respond to them. Generally, utilities document reports of
unusual water characteristics and use them to identify and address water quality problems.  The
procedures and systems used to handle these reports are commonly referred to as the utility's consumer
complaint management system.

As part of a contamination warning system, the complaint management system should extend beyond just
managing complaints - the system should monitor the complaint handling process and identify when
conditions could be indicative of a water quality problem. This can be achieved by identifying
information within the consumer complaint management system that can serve as an indicator of a
possible contamination event when monitored and compared against a threshold value. If the collective
information indicates an anomalous pattern in water quality calls, the contamination warning system
triggers an alarm, followed by further investigation.  By expanding on existing systems that manage
consumer complaints, consumer complaint surveillance provides dual-use benefits to the utility by
enhancing its customer service as it captures early signs of a potential water quality issue.  Table 7-1
summarizes the design basis for consumer complaint surveillance and provides considerations as to how
these objectives impact design and implementation.

Table 7-1. Design Basis Considerations for Consumer Complaint Surveillance
Design
Objective
Capability
Contaminant
Coverage
Spatial
Coverage
Timeliness
Reliability
Sustainability
Description
Can indicate the presence of a
contaminant that significantly affects one
or more aesthetic qualities of water.
High detection potential for classes 1 and
2. Moderate detection potential for classes
3, 4, and 5.
Entire service area for contaminants with
detectable taste, color, or odor
characteristics.
Function of the time from exposures to
consumer reporting, complaint
categorization, assessment and
investigation.
A potentially reliable indicator for
contaminants with detectable
characteristics if a robust complaint
reporting and tracking system is in place.
Provides utility an opportunity to manage
consumer information more effectively
and can serve as a tool for enhanced
consumer confidence.
Design and Implementation Considerations
Detection of aesthetic changes is an indirect measure
of contamination.
Contaminant coverage may also be a function of
consumer education. In addition, population diversity
may impact the ability to detect an aesthetic change in
water quality.
While consumer complaint surveillance can
encompass the entire service area, it is important to
consider how calls are handled outside the utility if
multiple jurisdictions or municipalities are served.
Time to detection is a function of the system and
procedures used to detect and investigate an anomaly
in the number, type, or distribution of water quality
complaints.
Reliability is a function of educated consumers and
their ability to notice aesthetic changes and ultimately
report these to the utility in a timely manner. It is also
a function of the system used to detect anomalies.
Enhancements to the consumer complaint
management system should be integrated with routine
job functions and should improve customer
service/satisfaction and potentially day-to-day
operations in a call center.
The objective of this section is to assist in the planning for implementation of consumer complaint
surveillance as part of a contamination warning system. Although many utilities currently implement
some consumer complaint monitoring and management activities, typically these activities are not
integrated in a manner to support contamination warning system objectives, such as the timely
May 2007                                                                                78

-------
                                Planning for WS-CWS Deployment

recognition of possible contamination.  The consumer complaint surveillance component should provide
utilities with a mechanism to enhance their current call management system and incorporate consumer
call information with other contamination warning system monitoring and surveillance data. Design
decisions to consider include determining how to:
    •  Educate consumers.  Utility consumers who are aware of indicators of potential contamination,
       including suspicious activity and unusual water characteristics, and who are educated on how to
       report such indicators, are an invaluable tool in detection of a potential water quality incident.
    •  Capture all complaints. A  "funnel" for collecting all water quality complaints into the
       consumer complaint surveillance system should exist. For example, a unified call center with a
       widely publicized telephone  number in place to capture the largest percentage of potential
       complaints. In addition, procedures should be in place to capture complaints that are directed to
       other points inside the utility that are initially received by other agencies. In cases where
       customer calls are managed outside of the utility, priority should be given to tunneling water
       quality related calls to the utility in a timely and efficient manner.
    •  Electronically manage data. All water quality complaints should be entered into an electronic
       database as they are received and categorized by type. A complaint record is carried through the
       process with  information being added to it as  it is received or investigations are conducted, and
       duplicate data entry is minimized or eliminated. Each complaint should be tracked from receipt
       to closure and retained in a historical database.
    •  Automate and integrate data analysis.  Once all of the information (water quality complaint
       data and location) is collected for a call, it can be evaluated based on utility known parameters.
       This can be accomplished using manual procedures already in place for review of complaint data
       or through an automated event detection system. An event detection system is an automated
       software and/or hardware system that analyzes data in near "real-time" for information that may
       be indicative  of a contamination incident based on a utility pre-determined threshold. For
       example, the  frequency or locations of water quality complaints could provide indication of
       possible  contamination
    •  Establish procedures and protocols.  Written standard operating procedures (SOPs) for every
       step in the water quality complaint handling process is an essential attribute of consumer
       complaint surveillance.  These SOPs should facilitate effective and timely communications,
       including clear guidance regarding the decision process to determine appropriate response
       actions, such as field investigations and sampling and analysis. All personnel potentially involved
       in the consumer complaint management system should be trained in these procedures.
    •  Address additional training of personnel.  Trained and dedicated personnel may be crucial to
       implementing a successful consumer complaint surveillance process. These people are the front
       line contacts  who interact with the consumers and process their input. The experience and
       professionalism of the utility personnel are critical to timely and accurate recording of data, and
       their judgment—in conjunction with automated and integrated data analysis—may be vital in
       assessing when a possible contamination incident has occurred.
    •  Achieve continuous system improvement.  The consumer complaint management system
       should be evaluated routinely (at least every five years) for two reasons: 1) to gauge how well it is
       meeting the intended goals of consumer complaint surveillance as well as other benefits derived
       from enhancements to the consumer complaints management system and 2) is to identify
       technological innovations or procedural modifications that warrant changes to the system.

Throughout the enhancement process, the utility should also be mindful that the efficacy of the consumer
complaint surveillance component is contingent upon the occurrence  of data collection, analysis and
notification steps in a timeframe that allows for effective response actions to mitigate a water
contamination incident.
May 2007                                                                                   79

-------
                                Planning for WS-CWS Deployment

7.1    Pre-Design

Pre-design for the consumer complaint surveillance component of a contamination warning system
involves completing an assessment and gap analysis of a utility's call center procedures and call
management system. The purpose of the assessment and gap analysis is to determine to what extent the
current consumer complaint management system procedures and data systems can be modified and
recommend enhancements to consumer complaint management systems to meet the attributes of an
effective consumer complaint surveillance component

Consumer Complaint Surveillance Model

The design of a consumer complaint surveillance component of a contamination warning system is based
on a model of "Funnel, Filter and Focus" as described below:
    •   Funnel.  All customer calls should be directed to one-point of contact within the utility
    •   Filter. Utility employees who routinely handle calls should be able to respond to billing,
       metering reading and general water quality concerns.  Calls concerning more specific water
       quality concerns should be forwarded to other appropriate personnel
    •   Focus. Personnel with training and experience in water quality should gather in-depth
       information for the consumer and make a determination of the need for field sampling.

Figure 7-1 illustrates this model.  In cases where consumer calls are managed outside of the utility, a
similar approach should be applied to manage water quality related calls.
 LJJ
 CC.
 LJJ
                     Primary Source:
                  Water Utility Call Center
     Secondary Source: ^     I      ^ Secondary
       Other Agencies  ^^^   I   ^f  Source:
                      ^  •*•  r   Other Utility
                                   Departments
            Non Water Quality
              Related Calls
      Non Water Quality
       Related Calls
            (As determined by
            Call Center Staff)
Water Quality Calls
Related to System
Operations
(Main Breaks,

^——~~ ^
Operations
Staff
/
 o
 o
Water Quality Specialist
  analyzes remaining
complaints for indications
of possible contamination
Figure 7-1. The Recommended Filter, Funnel, and Focus Approach to Customer Feedback Data
Optimization for Utility-Managed Consumer Calls
May 2007
                                                                                    80

-------
                                Planning for WS-CWS Deployment

Assessment and Gap Analysis of Existing Consumer Complaint Management Activities

Based on experience at the pilot utility, an assessment and gap analysis is an effective tool to assess the
current consumer complaint management activities and identify enhancements that should achieve the
objectives of a contamination warning system. Such an assessment involves a thorough investigation of
the utility call center and its  call handling procedures. The goal is for the consumer complaint
surveillance team to develop an understanding of the entire call management system.

During the assessment, the consumer complaint surveillance team gathers an accurate picture of the
utility's call management system. Superimposing this picture onto the model of a consumer complaint
surveillance component outlined in Figure 7-1, identifies the gaps where enhancements could bring the
existing complaint management system into alignment with the model. Key considerations in the analysis
include:
    •   Identification of consumer complaint surveillance data streams. The utility should identify
        existing data streams related to water quality calls and the systems currently used to process these
        calls (e.g., call and work management systems, if available). As the utility assesses the data upon
        which a consumer complaint surveillance system is to be built, it is important to consider the
        sources from which  that data is to be derived. It may be necessary to consider how data is to be
        aggregated and integrated from such disparate sources as third-party call management system
        hosts, other organizations within the city/jurisdiction, or other sources.
    •   Establishing a trigger and process for event detection. Once the existing data steams  are
        identified, the next step is to establish a trigger level(s) based  on background consumer complaint
        call volumes at a particular time.  It should be recognized that consumer complaint call volume
        may vary based on the day of the week or time of day.  Depending on how complaints are
        captured the data analysis can  be completed by breaking the data into complaint categories by day
        of the week and/or by month.  This process should be a method for counting the frequency and
        spatial distribution of water quality related consumer complaints over time and notifying
        responsible stakeholders once  the characteristics of these data surpass the established level.
        Ultimately the goal is to answer the question, when does the complaint frequency rise to  a level
        that indicates a significant change in water quality has occurred.  Automation can occur through
        simple frequency counts based on time and location(s) of water quality complaints or could be a
        more sophisticated algorithm,  such as a programmed EDS which operates behind the scenes of
        the existing consumer complaints management system hardware and software data systems.
    •   Alarm notification  and user  interfaces. Once a consumer complaint surveillance trigger level
        is surpassed, the user should be notified of the event. The notification process should attempt to
        utilize existing hardware and software systems available to the extent possible, such as e-mail
        and/or text messaging. Multiple notification procedures may  be necessary to ensure coverage for
        both business and non-business hours. Other user interfaces to consider are graphical information
        systems for display of alarm data.
    •   Linkage of data systems. It is critical to recognize during the assessment process that an
        effective call management system for consumer complaint surveillance should be able to link
        across associated data bases such as: call tracking, customer information and the utility's asset
        management systems. This ability should allow the utility to quickly identify the location of the
        complaint and facilitate geographical analysis of the complaint data. It would also provide the
        utility a significant dual-use benefit by facilitating tracking of all maintenance related issues from
        call receipt until work-order close-out.

7.2     Design  and Implementation Approach
The activities described in Section 7.1  describe a process for pre-design of a consumer complaint
surveillance system, culminating in documentation of an overall framework and selection of a
methodology for designing the system. The design and implementation of the consumer complaint
May 2007                                                                                  81

-------
                                Planning for WS-CWS Deployment

surveillance component should be based on the outcomes of pre-design and may include the
considerations described below.

Design:
    •   Perform additional reviews of historical water quality consumer call data analysis (if available)
    •   Clarify requirements of software/hardware needs (including event detection system)
    •   Develop a detailed method for automating data collection and analysis
    •   Refine notification protocols (e.g., text message, e-mail, pager)
    •   Develop end-to-end test protocols to verify system functionality
    •   Develop a preliminary concept of operations to describe routine operation of the component and
       additional user-specific standard operating procedures as needed
    •   Create a work plan and schedule to meet the design and time requirements of the project

Implementation:
    •   Install and test necessary hardware/software or other equipment for data gathering and
       automation
    •   Install and configure EDS tools
    •   Revise concept of operations as appropriate to reflect the routine operation of the component as-
       built
    •   Conduct staff training on consumer complaint surveillance tools, systems, and procedures

Preliminary Testing:
    •   Integrate the consumer complaint surveillance concept of operations into routine operations
    •   Collect and evaluate additional baseline data used in event detection
    •   Refine EDS tool configuration

Operation & Maintenance:
    •   Monitor and respond to EDS alarms in accordance with the concept of operations.
    •   Log any anomalies detected by the EDS tool(s).
    •   Reconfigure the EDS tool if performance is found to be unsatisfactory.
    •   Verify that EDS tools and supporting software are current with respect to vendor-provided
       updates, patches, etc.
    •   Conduct periodic staff training on concept of operations

Evaluation  & Refinement:
    •   Continuous documentation of performance during operation, e.g., false alarms, detection of true
       water quality anomalies, and dual-use benefits
    •   Through simulated events, periodically evaluate the overall performance of the operational
       consumer complaint surveillance component.  If necessary, update procedures or EDS tools to
       optimize performance.
    •   Track and evaluate the development of new EDS tools

7.3    Available Tools and Resources
The following tools and resources are available to support design and implementation of consumer
complaint surveillance as part of a contamination warning system:
    •   Whelton, A.,  Dietrich, A.M., Gallagher, D.L, Roberson, A. "Using Customer Feedback for
       Improved Water Quality and Infrastructure Monitoring," submitted to Journal of the American
       Waterworks Association. (Dec. 2006).
    •   Dietrich, A.M. "Aesthetic issues for drinking water," Journal Water and Health, Volume 4,
       Supplemental 1 (2006).
May 2007                                                                                  82

-------
                                Planning for WS-CWS Deployment

    •   Whelton, A., Dietrich, A.M., Burlingame, G.A., Cooney, M.F., "Detecting Contaminated
       Drinking Water: Harnessing Consumer Complaints," submitted to J. Amer. Water Works Assoc.
       (Dec. 2006).
    •   Lauer, William C.  "Water Quality Complaint Investigator's Field Guide." American Water
       Works Association (2004).

7.4    Staffing and Cost Considerations
Planning for the implementation of the consumer complaint surveillance component of a contamination
warning system requires involvement of a wide array of utility personnel and potentially contractor staff.
Costs may vary based on the utility's existing capabilities and the extent of enhancements. Therefore, the
remainder of Section 7.4 illustrates the staffing considerations and cost factors that are recommended for
consideration during project planning.

7.4.1  Staffing

Owing to the potential complexity of creating or modifying call management systems to accommodate the
objectives of a contamination warning system, a variety of staff, both internal and external to the utility
should be considered.  Depending on in-house expertise, it may be necessary to consider engaging third-
party contractors to modify existing systems. Table 7-2 provides an overview of the staff and resources
that may be engaged to design, implement, and operate a consumer complaint surveillance component as
part of a contamination warning system.

Table 7-2. Consumer Complaint Surveillance Staffing Considerations
Division / Department
Water Quality
IT
Engineering
Supply
Distribution
Administration (internal
Call Center
Outsourced Call Center
City Call Center
Phase
PD
X
X



X
X
X
D
X
X



X
X
X
I
X
X



X
X
X
PT
X
X

X
X
X
X
X
O&M
X
X

X
X
X
X
X
ER
X
X

X
X
X
X
X
Comments
Provide water quality information during
investigations of water quality complaints
Designs, implements, and manages all IT systems
used to support call management system
N/A
Provide investigation support to water quality calls
Provide investigation support to water quality calls
Recognize water quality related complaints and pass
on to Water Quality Personnel
Provide call center support
Provide call center support
PD = Pre-design; D = Design; I = Implementation;
Maintenance; E&R = Evaluation and Refinement
PT = Preliminary Testing; O&M = Operations and
Building the team to implement this component of a contamination warning system may involve man
divisions within the utility. Therefore, it is important to have senior leadership involved and invested
during each stage to facilitate the resolution of cross-division issues.  Several key personnel, such as the
IT manager and the water quality division manager or equivalent, would ideally be members of this and
all other component teams to facilitate the application of the system engineering principles outlined in
Section  2.1.  Such involvement can be a significant commitment of time and resources for these
individuals, but the utility can reap substantial benefit in the long-term success of the system. Other team
members' participation may be less demanding, but still critical, and may vary depending on the utility-
specific gap between the initial conditions and the final planned capabilities of the contamination warning
system.  If the call center is outsourced, the utility should incorporate the requirements of the consumer
complaint surveillance component of the contamination warning system into the contract with the call
center contractor to the extent possible.
May 2007
                                              83

-------
                                Planning for WS-CWS Deployment

7.4.2  Cost Considerations

This section presents a summary of the design and implementation considerations discussed above that
may influence costs. This list also may include other factors that were encountered during
implementation of the initial Water Security initiative pilot and could be overlooked during cost
estimation in the absence of this experience. Although this list of cost considerations may not be
exhaustive, these factors, at a minimum, should be considered when planning.
    •  Baseline assessment costs may include inspection and formal assessment report development by a
       project team to document the utility's existing call management system hardware, software and
       business management practices and to assess current capabilities to meet the contamination
       warning system objectives.
    •  Concept of operations costs include document development to specify the utility process and
       information systems to be used during routine operation and initial trigger validation of the
       consumer complaint surveillance component.
    •  Modification of existing call management software or the assessment and procurement of new
       call management software to meet the contamination warning system objectives for this
       component.
    •  Assessment of existing consumer complaint data streams to determine an approximation of event
       trigger level and applicability of statistical algorithms.  The cost may also include coding utility-
       specific automated analysis tools along with the consumer complaints management system data
       extraction and transformation for data processing.
    •  Design and testing of consumer complaint surveillance alarm notification. This process may
       require the procurement of new hardware and/or software to ensure the timely display of alarms.
    •  Additional cost may also be incurred to incorporate and test GIS display of possible consumer
       complaint surveillance water contamination alarms, if this capability is desired.
    •  Deployment and testing of automated analysis tools along with the consumer complaints
       management system data extraction and transformation for data processing.
    •  Training on enhancements and concept of operations.
    •  Testing of existing call management software or new call management software to verify
       operation is consistent with the design.
    •  Software and hardware upgrades for call management system, event detection tools and alarm
       notification system
    •  Periodic evaluation and recalibration of consumer complaint trigger values, based on a review of
       historic data. Refinement efforts could also focus on a review of the effectiveness of the concept
       of operations.
    •  Conducting a consumer complaint surveillance evaluation drill. Consider one drill per year to
       include all personnel identified within the concept of operations for this component.
May 2007                                                                                  84

-------
                                Planning for WS-CWS Deployment


                 Section 8.0:   Public Health Surveillance

Public health surveillance systems gather and analyze health-related data to identify anomalies that might
indicate unusual incidence of disease. The role of public health surveillance in a contamination warning
system is to gather and analyze data for investigation that will augment traditional epidemiological
surveillance (which often relies on an astute clinician to notice and report anomalies). When anomalies
are detected by public health analysis, coordinating with the utility will assist in determining whether the
anomaly is related to water. The involvement of public health experts in a contamination warning system
also adds a unique area of expertise that can not only support this component, but also consequence
management.

Some public health data, such as OTC drug sales and emergency room visits, may be suited to detecting
biological contaminants found in contaminant classes 10 and 11 (Table 1-1). Other data, such as EMS
records, 911 call data, and Poison Control Center data may be better at detecting fast-acting chemicals,
like those found in contaminant classes 1-9.

Public health surveillance is performed by many entities, including local health departments, 911 call
centers, Poison Control Centers, and nationwide surveillance systems, such as the National Retail Data
Monitor, which gathers and analyzes OTC data. Depending on the utility's area of service, there could be
numerous local health departments (the locality could include city, county, or regional departments). Each
of these entities is responsible for different areas, and some parts of the utility service area may not be
covered by all  of the public health partners. For example, a county health department may cover the
entire utility service area, but 911 call data may be limited to one designated public service answering
point (such as one city within the service area).  Because of this, coordination with as many entities as
feasible is needed to cover as much of the service area population as possible.

Public health surveillance systems that would be appropriate for a contamination warning system can be
grouped into two categories:

     •  Traditional surveillance systems include hospital disease reporting and laboratory reports.
        These are generally collected by the health department(s) and analysis is performed using
        morbidity rates (e.g., the number of measles cases per county per year). However, collection of
        these data can be relatively slow, with a lag time of days or weeks. Therefore, these data may
        be more  useful  for follow up investigation and false-alarm evaluation, but other more timely
        approaches to data collection and analysis should be considered.

     •  Syndromic surveillance aims to use any  data available in as near real-time as possible to detect
        possible  outbreaks based on statistical analysis of "syndromes," or categories of disease. This
        surveillance approach may  be more useful for quick detection of the  contaminants of concern.
        Syndromic surveillance can also involve performing "fused analysis," whereby information
        from many  sources is analyzed together to detect possible anomalies. Some examples of
        syndromic surveillance programs currently being used by health departments  include BioSense,
        the National Retail Data Monitor (NRDM), RODS and ESSENCE (see Section 8.3 for
        additional detail).

The overall objective of this section is to describe options and a process to develop a design and
implementation strategy  for the public health surveillance component of a contamination warning system,
based on existing capabilities in the area where the  utility is located. This strategy will help local public
health identify routine health changes more quickly and aid in cooperation with the utility to determine
whether a possible water contamination is the cause of the health anomaly.  Design basis considerations
to achieve successful public health surveillance are summarized in Table 8-1.
May 2007                                                                                   85

-------
                                 Planning for WS-CWS Deployment
Table 8-1.  Design Basis Considerations for Public Health Surveillance
Design
Objective
Capability
Contaminant
Coverage
Spatial
Coverage
Timeliness
Reliability
Sustainability
Description
Can detect the presence of a symptom or
illness in a population which may be the
result of the presence of a disease causing
agent. May be able to identify the
contaminant through clinical diagnosis/
testing.
Covers contaminant classes 2 through 1 1 ;
detection potential varies with type of
surveillance.
Comprehensive coverage of a particular city
or county, which may include all, or a large
portion of, the utility service area.
Function of the time from the initial
exposures, the onset of symptoms, and the
point at which public health officials
recognize the incident as a potential water-
borne illness.
May be a reliable means of identifying the
incidence of illness in a population, but
timing of communication between drinking
water and public health officials should be
optimized such that appropriate response,
actions could be implemented in time to
reduce consequences.
Provides an opportunity for collaboration
between utility and local health
department(s).
Design and Implementation
Considerations
Consider appropriate algorithms and the
concept of fused analysis to look at many
results concurrently (Burkam, et. al).
Include public health data streams for both
fast-acting chemicals and biological
contaminants.
Should include participation of numerous
agencies and partners; any one type of
data may not cover the entire service area.
Where possible, automation of data
collection and analysis as well as alerts will
increase timeliness. New data streams
such as 91 1 or EMS data should be
considered.
Develop a well defined communications
protocol and Concept of Operations,
including role descriptions and a firm
commitment to investigation.
Should be able to be performed without
compromising other roles of public health
agencies.
8.1     Pre-design
Design of the public health surveillance component depends primarily on existing public health
surveillance systems within the utility's service area and will vary greatly between utilities. Integration of
public health surveillance in the contamination warning system will involve relationship building as much
as analytical implementations. Appropriate roles, limits of information sharing, and how investigations
will proceed should be addressed, with the end goal of communicating and responding to possible events
more effectively.

The pre-design process for implementation of public health surveillance in a contamination warning
system should consider the following:
    •   Identification of local public health partners
    •   Identification and assessment of existing capability
    •   Development of a framework for communication and notification

Identification of Local Public Health Partners

The first step in the pre-design process is to identify current relationships between the utility and public
health partners. If there is already a high degree of cooperation and coordination between the utility and
local health partners, an expansion of these relationships should be considered to include a contamination
warning system. If no relationship exists, points of contact should be identified for those who can
effectively  share information and participate in joint investigations of possible contamination events. A
point of contact in the local health department(s) is particularly important. In addition to local public
health departments, it is also important to engage the local or regional poison control center, local fire
departments, dispatch centers, and in some cases State health departments as well. Because the
contamination warning system will engage agencies that may, under other circumstances, have little need
to interact,  an effective way of communicating information should be established early in the planning
process.
May 2007
86

-------
                                Planning for WS-CWS Deployment


Identification and Assessment of Existing Surveillance Capability

Data streams that could be available for public health surveillance should be identified with the help of
local health departments, fire departments, Poison Control Centers, and other partners. These data streams
may include EMS, 911, laboratory tests, hospital data, Poison Control Center calls, and OTC data.
Surveillance capability may vary from jurisdiction to jurisdiction in terms of degree of automation, type
of surveillance, and area covered.  It is important to assess the capability of the existing surveillance
systems relative to the contamination warning system objectives of contaminant coverage, spatial
coverage, timeliness (degree of automation), and reliability. The relationships between possible public
health data and the design objectives of spatial coverage, timeliness, and contaminant coverage are shown
in Figure 8-1.

It is important to note that, while automation of data analysis is a desired goal of a contamination warning
system, this may not always be the most cost-effective means of improving surveillance capabilities.
Response times and communication protocols also may be improved significantly by evaluating and
optimizing manual processes and procedures. The costs, organizational feasibility, and enhanced
surveillance capabilities yielded by these improvements should be weighed against the costs and benefits
of automated data analysis options. Consideration should also be given to the ability to detect both fast
acting contaminants and contaminants with a long latency period.
SPATIAL TIMELINES OF
DATA SOURCE COVERAGE DATA LOCATION COLLECTION
f5T™j5fBil City and/or County Near Real-Time
TO^-lmr3 Department
EMS
911 City and/or County ^^ NeaMTme
JWl ""S'0"81 Poison Control Near Real-Time
•^53s>i Center
^Poison
Wdfc Civ/County/
w*w Regional
t^v*-,
City/County/
^ Regional LPH Days or Weeks
Hosp tats
INITIAL CONTAMINANT
DETECTED
Fast-Acting Chemicals
(Classes 1-9)
Fast-Acting Chemicals
(Classes 1-9)
Fast-Acting Chemicals
(Classes 1-9)
Pathogens
(Classes 10-11)
Pathogens
(Classes 10-11)
Figure 8-1 Public Health Data Sources and Design Objectives


At a minimum, data should be applicable to the detection of biological and chemical contaminants though
either formal diagnosis or results (i.e., a laboratory test) or syndromic surveillance (i.e., OTC data
profile).  Especially important to consider is whether the data collection process can be automated, which
can decrease time and effort of collection and analysis and thereby increase sustainability. Not only will
this be helpful as part of the contamination warning  system, but also will benefit the local health
department in performing faster epidemiological analyses.
In the absence of data automation, other approaches for achieving the design objectives should be
considered.  These approaches could include the following:
May 2007
87

-------
                                 Planning for WS-CWS Deployment

     •   Increased manual surveillance activities through increased staff to increase frequency of data
         monitoring
     •   Increased breadth and scope of manual surveillance activities
     •   Working cooperatively with local health departments to fund and add staff positions to their
         operations dedicated to surveillance activities focused on waterborne contamination

Because the data streams should be complementary to each other and to data available at the utility,
possible anomalies may be quickly verified or discounted. For example, a high volume of 911 calls could
be compared with EMS run volume and run location to verify whether the high volume of 911 calls
indicates a real event, or is just due to chance.  Data should also be representative of the entire utility
service area; if a utility serves multiple counties, then data from each county should be utilized.  Public
health data under consideration also should be compared with utility data to see where they might
supplement each other. An example would be mapping 911 call data against consumer complaints calls
to provide an investigative starting point.

Development of a Framework for Communication and Notification

Utility and public health partners should evaluate any current notification protocols. A call list, phone
tree, or other established contact order may already exist. If a protocol exists, the organizations should
determine whether the current protocol is appropriate for a contamination warning system, and then
modified and/or updated into a preliminary concept of operations, as discussed in Section 2.0. If there is
no communication protocol available, it should be created.

The communication protocol should involve personnel who understand the concepts of a contamination
warning system, can interpret data presented to them in relation to possible contamination events, and will
know how to proceed with the investigation. Where possible, automation of alerts, such as e-mail alerts
generated by an analysis program  and sent to predefined recipients, should be considered. Roles also may
need further refinement to  clearly  define who will be doing what at each stage.  This will ensure not only
that every investigative duty is performed, but also that efforts are not duplicated.  Identifying who does
what also will  help effectively use the expertise of public health officials. As part of the assignment of
roles and responsibilities, a strong commitment to follow-through on these roles and responsibilities
should be emphasized.

When developing a communication protocol, issues related to the Health Insurance Portability and
Accountability Act (HIPAA) should be considered. This law recognizes that advances in electronic
technology could erode the privacy of health information and mandates privacy protections for
individually identifiable health information. Health officials have access to data that, due to HIPAA
limitations or other constraints, are not usually available to utilities.

Information typically protected under HIPAA  includes any information that can be used to identify an
individual, and how this is interpreted may vary by jurisdiction. For example, public health officials may
determine that age, sex, and address, data cannot all be included, as this information could identify  a
patient even though the "Patient ID" field is not provided. However, communication between the utility
and local public health agency(ies) is essential, and data sharing should be conducted to effectively
investigate anomalies.  For example, based on discussions among the utility and local health departments
participating in EPA's initial contamination warning system pilot, the following data elements were used
for analysis:
    •  Patient location (zip code  or address for EMS  and 911 data, respectively)
    •  Event  date and time
    •  Chief complaint (used to categorize into syndromes).
May 2007                                                                                    88

-------
                                Planning for WS-CWS Deployment

Care should be taken with access to data presented in a User Interface to ensure they are compliant with
HIPAA.  More information on HIPAA and compliance with can be found at the Department of Health
and Human Services website (http://www.hhs.gov/ocr/hipaa/)

At the conclusion of the pre-design stage, the utility should have an understanding of the public health
surveillance tools within the utility's service area and approach for engaging and coordinating with public
health partners to leverage or adapt these tools to meet the contamination warning system design
objectives. This approach should stress cooperation among public health organizations and the utility to
improve existing data surveillance, event detection, and communication practices without compromising
current public health or utility services.

8.2    Design and Implementation Approach
The activities discussed in Section 8.1 describe design factors that should be considered when designing
and implementing a public health surveillance component within a contamination warning system.

Design:
    •   Develop a preliminary concept of operations to describe the process flow, data streams, and roles
       and responsibilities
    •   Establish how the data should be efficiently gathered
         o  Assess methods for automating data and/or adding data streams, including data that may not
            have been electronically captured previously (e.g., 911 and/or EMS data)
         o  If automation is not feasible and/or cost effective, consider alternative methods of providing
            public health support to the contamination warning system (e.g., increased frequency or
            scope of manual surveillance  activities or staff support to local health departments to focus
            on waterborne contamination)
         o  Consider participation in BioSense, RODS,  ESSENCE, or another syndromic surveillance
            program to improve data collection capabilities, and determine if the area is covered by the
            NRDM
    •   In collaboration with local health departments, research and identify appropriate analysis tools
         o  Consider methods that can analyze data by time (i.e., counts per day) and by location (i.e.,
            counts in a spatial cluster) in as near real-time as possible
         o  Research available literature to determine which algorithm(s) are appropriate for the data.
            Possible analysis tools include regression models, multivariate models (e.g., CUSUM), and
            spatial analysis
         o  Consider a fused analysis approach
    •   Establish a means for displaying data in  a HIPAA-compliant way in one user-friendly location,
       such as a User Interface on a centrally-accessible  website.

Implementation:
    •   Optimize data collection and analysis procedures  and protocols through implementation of
       refined procedures or installation of software and equipment to support automation
         o  IT mapping of data on servers
         o  Computer code for automating and optimizing datasets
    •   Install analysis tools decided upon  in design stage
         o  Software to run statistical algorithms
         o  Computer code for automating analysis
         o  Display of results on a user-friendly interface
    •   Hire additional staff that may have  been identified in the design stage
    •   Conduct training  on analysis tools and interpretation of results, particularly for new analysis tools
    •   Hold meetings between utility  and  public health agency(ies) to educate on roles and
       responsibilities
    •   Participate in table-top and other communication  exercises
    •   Implement data agreements, where necessary (e.g., a contract with the Poison  Control Center)

May 2007                                                                                    89

-------
                                Planning for WS-CWS Deployment

Preliminary Testing:
    •  Based on analysis of historical data, determine when an anomaly requires investigation relative to
       contamination warning system objectives. This "alert level" may be revised based on results
       from data generated during the preliminary testing stage.
    •  Determine acceptable false-alarm levels (How often is it acceptable to investigate a result that is
       not a true anomaly?)
    •  Adjust roles and communication protocols based on results from table-top exercises and other
       exercises
    •  Revise concept of operations as appropriate based on design and implementation activities and
       results from preliminary testing activities

Operation and Maintenance:
    •  Use communication protocol for investigation
         o  Contact appropriate people during anomaly investigation
         o  Share information agreed upon during pre-design stages
         o  Update contact lists as necessary
    •  Continue data gathering and analysis
    •  Perform data maintenance activities as necessary.
         o  Software upgrades
         o  Archive old records

Evaluation and Refinement:
    •  Evaluate ease of interpretation and usefulness of data, including sensitivity (false-alarm)
       measurements and data quality issues.
    •  Determine how well communication protocol is working, and consider methods of improvement.
           o   Consider further automation of alerts, such as email or text message
           o   Consider improvements to any manual data-gathering methods
           o   Consider information shared and if it is useful
    •  Adjust alarm thresholds as necessary, based on sensitivity measurements.
    •  Adjust data gathering frequency as necessary. This  could also include automation of data not
       previously automated.

8.3    Available Tools  and Resources

The following resources are available to assist in design and implementation of the public health
surveillance component:

Early Aberration Reporting System (EARS). EARS is a software program developed by the CDC for
the purpose of syndromic surveillance.  Versions of the program can be run using either SAS or EARS.
Public health data can be run through EARS to detect possible outbreaks or bioterrorism events. Analysis
capabilities include categorization of symptoms into syndromes using text search string functions,
aberration detection using CUSUM methods (Cl-Mild, C2-Medium, and C3-Ultra), and graphic
representation of the analysis using graphs and maps, http://www.bt.cdc.gov/surveillance/ears/

Electronic Surveillance System for the Early Notification of Community-Based Epidemics
(ESSENCE). Developed by the Department of Defense and Johns Hopkins University Applied Physics
Laboratory, ESSENCE aims to collect and analyze a variety of data sources for the early recognition of
abnormal community disease patterns that could result from natural causes or terrorist activities.
ESSENCE uses data from military and civilian databases of patient visits, OTC sales, chief complaint
data from Emergency Rooms, 911 calls, Poison Control Center Calls, laboratory  records, as well as
weather and community events to perform a fused analysis using spatial and temporal algorithms.
http://www.geis.fhp.osd.mil/GEIS/SurveillanceActivities/ESSENCE/ESSENCE.asp
May 2007                                                                                  90

-------
                                Planning for WS-CWS Deployment

Real-time Outbreak and Disease Surveillance (RODS). RODS was originally developed by the
University of Pittsburgh in collaboration with Carnegie Mellon to provide a computer-based public health
surveillance system for the early detection of disease outbreaks. RODS looks at emergency room and
OTC data into one user interface; it collects hospital data in near real-time and analyzes it using Recursive
Least-Square and What's Strange About Recent Events algorithms. It has been implemented at health
departments in numerous states, http://rods.health.pitt.edu/

National Retail Data Monitor (NRDM).  The NRDM was also developed by the University of
Pittsburgh in collaboration with Carnegie Mellon to gather and monitor information on OTC drug sales
for possible outbreak detection. This data is collected for use in the RODS analysis tool.  The data is
gathered from  over 20,000 retail stores throughout the country, and is available to public health
departments free of charge. http://rods.health.pitt.edu/NRDM.htm

BioSense.  BioSense is the national program designed to improve the nation's capabilities for real-time
bio-surveillance and situational awareness headed by the CDC. It operates using an Internet-accessible
system that allows users to visualize information about public health trends from early detection data
sources, using  advanced algorithms to analyze the data and provide a nationwide, real-time picture.
BioSense is also discussed in Section 1.0 of this document, www.cdc.gov/biosense/

FirstWatch. FirstWatch is a software program designed to integrate and analyze data from various
health and public safety sources (i.e., 911 call, police dispatches, etc.) FirstWatch displays many data
streams on  one user interface and performs analysis using predetermined thresholds. Automation for
analysis is already built into the program.  Its use has been growing, and is now used in some capacity by
over 50 cities in the U.S. and Canada, www.firstwatch.net

SaTScan. _SatScan is a statistical program  developed for the purpose of cluster analysis using both spatial
and temporal methods. It includes models for case-control (Bernoulli model), Poisson based population,
space-time  permutation, ordinal, exponential, and normal distribution analysis. Information about the
significant clusters can be output to files for plotting using a separate GIS system. SatScan has been used
extensively in  recent years for syndromic surveillance purposes, www.satscan.org

HIPAA. The Health Insurance Portability  and Accountability Act was passed in 1996 to protect the
privacy of patient medical data by limiting  who can have access to individually identifiable medical data.
http: //www .hhs. gov/ocr/hipaa/

8.4     Staffing and Cost Considerations
In planning the public health surveillance component of a contamination warning system, it is essential to
incorporate staff from a wide range of partners, some of which may have no previous experience with
public health data collection and investigation. Cost  factors will vary substantially, depending on how
many partners  participate, and their capabilities.

8.4.1   Staffing

During planning, input from numerous entities should be gathered,  including  local health department
epidemiologists, fire departments/EMS crew, 911 dispatch operators, Poison Control Centers, as well as
the utility. Each of these can provide unique input on advantages and disadvantages of proposed public
health surveillance systems. These agencies should be engaged early in the planning process, such that
their effort  on  a day to day basis (i.e., during operation and maintenance) is minimized. At later stages of
operation and maintenance, there should at a minimum be adequate staff to interpret any alert generated
by the public health event detection component and successfully communicate details of the alert to
appropriate utility staff for investigation. Where appropriate, additional staff dedicated to manual
surveillance activities could be considered in place of automated data streams. Table 8-2 provides a
summary of staffing considerations for the  public health surveillance component.
May 2007                                                                                   91

-------
                                Planning for WS-CWS Deployment
Table 8-2.  Public Health Surveillance Staffing Considerations
Division or Department
Implementation Phase
PD
D
I
PT
O&M
E&R
Comments
Utility
Project Manager
Water Quality
X

X





X
X
X
X
Provide investigation support to public health alerts
Provide investigation support to public health alerts
Public Health Partners
Epidemiologist
Fire /EMS
911
Poison Control Center
X
X
X
X
X
X
X
X
X
X
X
X
X



X
X
X
X
X


X
May include one or more epidemiologist(s); will be
the main person performing surveillance of data
streams
PD and D activities include identification of possible
data streams; O&M activities will mainly be
maintenance
PD &D activities include identification of possible
data streams; O&M activities will mainly be
maintenance
Roles may be based on service agreement with the
poison control center; may include data sharing and
toxicological expertise
Other Support Partners
IT Support
X
X
X
X
X
X
May be managed at the city or regional level
PD = Pre-design; D = Design; I = Implementation; PT = Preliminary Testing; O&M = Operations and Maintenance;
E&R = Evaluation and Refinement
8.4.2   Cost Considerations

This section presents a summary of the design and implementation considerations discussed above that
may influence costs. This list also may include other factors that were encountered during
implementation of the initial Water Security initiative pilot and could be overlooked during cost
estimation in the absence of this experience. Although this list of cost considerations may not be
exhaustive, these factors, at a minimum, should be considered when planning.

Costs associated with the incorporation of public health surveillance in a contamination warning system
will be based heavily on the current status of public health data systems and existing relationships
between local health departments and the utility.  Estimating costs will only be possible after the basic
decisions presented above are considered. Substantial costs could be incurred with the purchase of new
software, development of code to automate systems integration, and initial training. However, investment
in these resources now could prevent even higher labor expenses during operation and maintenance.

In making decisions on how to collect data, automating data streams may be more expensive during
installation due to equipment and software procurement; however, this initial cost may be offset in the
long run by reduced effort needed by staff to continue data collection.  Key cost considerations should
include the following:
    •   Determining contacts  and developing a Concept of Operations. Effort will be needed to
        generate these contacts and documents
    •   Computer hardware or software for data collection and analysis. It may be necessary to
        procure computer hardware or software and/or develop custom software for the purposes of data
        gathering, analysis, and display. Where possible, available and experienced resources should be
        leveraged to reduce these costs.
    •   Determining and testing alert levels. It will take some effort to determine and test appropriate
        alert levels to maximize effective investigation while minimizing false positives.
    •   Training on utilizing new analysis tools. These costs will depend on how many new tools are
        implemented, and the overall number of people who need training
    •   Table-top and communication exercises. Resources will be necessary to plan, support, and
        participate in table-top and other response exercises.
May 2007
92

-------
                                Planning for WS-CWS Deployment

In general, the more agencies that are involved with public health surveillance, the more expensive it will
be in terms of training costs, coordination, and level of effort. However, involving more agencies also
means increasing potential resources in terms of software, equipment and expertise. Leveraging existing
systems could offset the cost of increased coordination by reducing the need for developing new systems.
May 2007                                                                                   93

-------
                               Planning for WS-CWS Deployment


                Section 9.0:  Consequence Management

As discussed and illustrated in Section 1, consequence management plays a critical role in a
contamination warning system. When triggers from one or more of the monitoring and surveillance
components discussed in Section 4 - Section 8 have been validated, consequence management governs
the response, remediation, and recovery actions.  A consequence management plan that successfully
guides these actions is a cornerstone of an effective contamination warning system, and it is essential to
have the plan  in place and tested prior to operation of any contamination warning system components.
While development of a consequence management plan can occur in parallel with design and
implementation of monitoring and surveillance components of the contamination warning system, it is
important to reconcile the consequence management plan with routine operations as the system evolves
through the application of system engineering principles as discussed in Section 2.1.

In planning for contamination warning system deployment, it is important to recognize that while the
integrated, routine,  and active approach for monitoring and surveillance of public health and water quality
in the distribution system may be a new concept, drinking water utilities should have an existing
emergency response plan that can serve as a starting point for consequence management. In response to
the terrorist attacks of 2001, Congress passed the Public Health  Security and Bioterrorism Preparedness
and Response Act of 2002 (the Bioterrorism Act) which required drinking water utilities to prepare or
revise, where  necessary, an emergency response plan that incorporates the results of vulnerability
assessments.

Consequence  management plans developed in support of contamination warning systems should  build on
the utility's existing emergency response plan, focusing specifically on the contamination threat to the
distribution system and should integrate response and decision-making with the routine operation of the
system as defined in the concept of operations. EPA previously provided guidance on response to
drinking water contamination in a suite of six modules that composed the Response Protocol Toolbox and
companion Response Guidelines (USEPA, 2004a-h). Many of the concepts presented in the Response
Protocol Toolbox are applicable to development of a consequence management plan for contamination
warning systems.

This section provides high-level considerations to assist utilities and local partners in planning for
development of a consequence management plan to support contamination warning system
implementation.

9.1     Pre-design
Designing a consequence management plan is a critical task as part of contamination warning system
deployment. Pre-design activities should be carefully conducted, such that subsequent steps can  be
effectively executed, with the result being a complete utility-specific consequence management plan that
presents a comprehensive framework for response to validated contamination warning system triggers.
Roles and responsibilities should be defined and assessments conducted, both at the utility and with local
partners.  These preliminary assessments can help identify existing plans, both within the utility and
partner organizations, and training opportunities that can be integrated as part of the effort to develop the
consequence management plan for the contamination warning system.

Pre-design activities for development of a consequence management plan include the following:
    •  Identification of objectives for the consequence management plan
    •  Self-assessment
    •  Identification of partners
    •  Development of a preliminary work plan and path forward
May 2007                                                                                 94

-------
                                Planning for WS-CWS Deployment


Consequence Management Plan Objectives

Prior to development of a consequence management plan to support contamination warning system
implementation, drinking water utilities should define the objectives of the plan. Most utilities have
detailed emergency response plans and/or action plans that may address contamination of the distribution
system. For the purposes of contamination warning  system deployment, objectives of the consequence
management plan may include the following:
    •  Clearly defined roles and responsibilities for utility and response partners in all stages of
       consequence management
    •  Comprehensive decision-making framework to support timely response to a trigger from one or
       more contamination warning system components
    •  Guidance on use of specific response procedures
    •  Seamless transition from routine operations to consequence management activities
    •  Integration with local  and regional response  plans

Determining objectives of the  consequence management plan early in the process should enable the utility
to effectively use information  from their emergency  response plans to create a comprehensive
consequence management plan applicable to a contamination warning system.

Utility Self-Assessment

A self assessment of the utility's existing emergency response plans and overall preparedness is the first
step in developing a consequence management plan to support contamination warning system
implementation. The purpose of the self assessment is to identify what procedures are already in place
regarding planning, preparedness, and response and assess these procedures relative to the objectives of
the  contamination warning system consequence management plan.  There are two primary aspects of this
assessment: existing plans and response resources and capabilities.  The utility should first review their
current response procedures and related documents for different events to determine what elements of a
consequence management plan they may already have. Examples of the types of plans that  should be
considered include the following:
    •  Plans for responding to a water contamination or water quality event, cross connections, chemical
       spills near source water, intentional contamination of the water system
    •  Plans for responding to increased or overwhelming consumer complaint calls, or calls reporting
       illness from the water
    •  Plans for responding to facility alarms, reports of suspicious persons near utility facilities, or
       threats made to the system, both directly to the utility and though third parties (police, media,
       etc.)
    •  Operational plans that address issues such as depressurization or power outage
    •  Severe weather response plans
    •  Civil disorder response plans
    •  Mutual aid agreements with other utilities
    •  Issuing of water-use restrictions
    •  Risk communication and public notification  plans

As plans  are reviewed, the situation addressed, utility divisions are involved, and outside agencies
involved  should be captured to identify gaps that should be addressed through consequence  management
planning  activities.

In addition to an assessment of existing operational plans, the utility should conduct an assessment of
response  resources and capabilities. This involves identifying assets (e.g., staff, equipment) as well as
training needs that are required to implement the existing plans  and operations. Throughout the
development of the consequence management plan, the utility should maintain a list of items or resources

May 2007                                                                                  95

-------
                               Planning for WS-CWS Deployment

that should be acquired, enhanced, or improved. During design and implementation, the list can be
revisited and shortfalls in training, equipment, and other resources can be resolved.

An aspect of the utility self-assessment related to response resources and capabilities includes assessing
Incident Command System (ICS) training and National Incident Management System (NIMS)
compliance.  ICS is a flexible command and control system designed to manage any magnitude of
emergency. ICS is a key component of NIMS, and NIMS is a key component of the National Response
Plan (NRP) which directs, among other things, how command over an incident is escalated from the local
to state to federal level, and back down again. More information on ICS and NIMS can be found at the
Federal Emergency Management Agency's (FEMA) website
(http://www.fema.gov/emergency/nims/index.shtm). There are many reasons to have staff trained in ICS
and NIMS. ICS has been used effectively since the 1970s and is a proven system. The response partners
engaged will most likely be well versed in ICS and NIMS, and should expect your staff to be as well.
Additionally, as directed by Homeland Security Presidential Directive 5, full NIMS compliance is a
requirement for receiving federal preparedness funds. The utility should also consult their state
emergency management and/or homeland security agency, as many states have NIMS requirements that
are more stringent than the federal requirements.

Identification of Response Partners

Identification of partners to support contamination warning system design and implementation is
discussed in Section 2.0. For the purposes of development of the consequence management plan, specific
contacts should be identified within each of those organizations to discuss roles and responsibilities
specifically related to response, remediation, and recovery actions. Table 9-1 provides an overview of
the roles and responsibilities response partners may play in implementing the consequence management
plan, and thus, when they should be engaged in the planning process.

Table 9-1. Overview of Response Partner Roles and Responsibilities for Consequence
     Management
Response Partner
Drinking water utility
Local health department
Local law enforcement
Local civil government
Local emergency planning committees and
emergency management agencies
Local fire, EMS, and Hazmat
Environmental and public health laboratories
Local wastewater utility
Neighboring utilities (water and/or wastewater)
Media
State government
State emergency responders
State drinking water and wastewater primary
agencies
State emergency management and homeland
security agencies
State law enforcement
EPA Regional offices and/or laboratories
Federal Bureau of Investigation
Centers for Disease Control and Prevention
Operational
Response
Y


Y



Y
Y









Public Health
Response
Y
Y

Y

Y











Y
Site
Characterization
Y
Y

Y
Y
Y
Y











Criminal
Investigation
Y

Y
Y
Y









Y
Y
Y
Y
Expanded
Sampling
Y


Y
Y
Y

Y
Y


Y


Y
Y
Y
Y
Laboratory
Analysis
Y
Y

Y


Y








Y
Y
Y
Risk
Communication
Y
Y
Y
Y
Y
Y

Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Remediation and
Recovery
Y
Y
Y
Y
Y
Y
Y
Y
Y

Y
Y
Y
Y
Y
Y
Y
Y
May 2007
96

-------
                                Planning for WS-CWS Deployment




Response Partner


EPA Criminal Investigation Division
EPA National Response Center


ns n)
C (ft
0 C
+= O
ns Q.





<-
ns to

ns O
P
Q)
o:
Y
Y
As response partners are identified, it may also be necessary to work with them to assess their existing
plans and capabilities to ensure there should be a seamless transition as an event escalates through the
stages of response, remediation, and recovery. Different response partners may have different areas of
expertise, and likewise might have different levels of response preparedness. These discrepancies should
be identified in the pre-design phase, and so that resolving them can be accomplished through design and
implementation activities.

Work Plan to Guide Consequence Management Plan Development

Following the utility self-assessment and identification of response partners, the next step in the pre-
design process is to develop a work plan to guide development of the consequence management plan.
The work plan should identify a framework or process for plan development, including engagement of
local partners, communications and roll out, and closing of any gaps related to training or capability. In
addition, major goals and milestones may be established in the work plan, such that the objectives defined
at the start of pre-design become practical, concrete targets. The work plan should aim at directing steps
to meet these targets and staying on schedule.  Completion of a well-defined work plan should aid in the
success of developing a consequence management plan as the utility moves forward through design and
implementation activities.

9.2    Design and Implementation Approach

Design and implementation activities associated with consequence management plan development
include the following:
    •  Consequence management plan development. Develop a framework and approach for credibility
       determination, confirmation, and remediation and recovery.  The development process may
       include the following:
           o  Defining an ICS structure for the utility
           o  Engagement of local partners and alignment with existing plans at the local, regional, and
              state levels
           o  Reconciliation with the concept of operations as defined for the monitoring and
              surveillance components
           o  Integration of site characterization plan
    •  Communication strategy
           o  Develop a strategy and framework for communications within the utility, between the
              utility and external partners, and customers.
           o  Develop a risk communication strategy and message maps for communicating
              information during a suspected or confirmed contamination incident.
Implementation:
    •  Procure equipment needed to augment utility or local partner capabilities.  Equipment may
       include sampling equipment, personal protective equipment, field screening equipment,
       communications equipment, etc. See Section 5.3 for a discussion of site characterization
       equipment, coordination, and planning.
May 2007
97

-------
                                Planning for WS-CWS Deployment

    •   Training
           o   Develop training materials that address all roles and responsibilities
           o   Conduct training on the plan.  The scope of this training should include all levels of staff
               within the utility that have a role in consequence management as well as local partners.
           o   Ensure appropriate staff are trained and certified in ICS and NIMS.
    •   Revise the consequence management plan as necessary based on feedback from training and
       reconciliation with routine operations and initial trigger validation for each of the components.

Preliminary Testing:
    •   Conduct additional training on the consequence management plan as necessary.
    •   Design and implement drills and exercises to test the consequence management plan that involve
       drinking water utility staff as well as local response partners.
    •   Refine and finalize the consequence management plan, including reconciliation with the concept
       of operations to ensure a smooth transition from routine operations, initial trigger validation, and
       consequence management across all components of the system.

Operations and Maintenance:
    •   Deploy plan as necessary in response to validated triggers from one or more of the contamination
       warning system components.
    •   Conduct ongoing training to ensure that new staff are familiar with the consequence management
       plan.
    •   Update the plan as necessary based on enhancements or modifications to monitoring and
       surveillance aspects of the system or local response partner capabilities.

Evaluation and Refinement:
    •   Conduct routine drills and exercises to evaluate the operation and performance of various aspects
       of the consequence management plan
    •   Refine the plan as appropriate based on lessons learned through drills and exercises

9.3    Available Tools and Resources
The following tools and resources are available to support development of a consequence management
plan for a contamination warning system:

    •   Response Protocol Toolbox Developed by the EPA, this series of six documents covers topics
       such as communications and notifications, threat evaluation, site characterization, sample
       analysis, and response actions to help the water sector prepare for and respond to contamination
       threats and incidents.
           o   Overview (EPA-817-D-03-007)
               http://www.epa.gov/safewater/watersecurity/pubs/guide response overview.pdf
           o  Water Utility Planning Guide - Module 1 (EPA-817-D-03-001)
               http://www.epa.gov/safewater/watersecuritv/pubs/guide response modulel.pdf
           o   Contamination Threat Management Guide - Module 2 (EPA-817-D-03-002)
               http://www.epa.gov/safewater/watersecurity/pubs/guide response module2.pdf
           o   Site Characterization and Sampling Guide - Module 3 (EPA-817-D-03-003)
               http://www.epa.gov/safewater/watersecurity/pubs/guide_response_module3.pdf
           o   Analytical Guide - Module 4 (EPA-817-D-03-004)
               http://www.epa.gov/safewater/watersecuritv/pubs/guide response module4.pdf
           o   Public Health Response Guide - Module 5 (EPA-817-D-03-005)
               http://www.epa.gov/safewater/watersecurity/pubs/guide response module5.pdf
           o   Remediation and Recovery Guide - Module 6 (EPA-817-D-03-006)
              http://www.epa.gov/safewater/watersecuritv/pubs/guide response module6.pdf
May 2007                                                                                  98

-------
                               Planning for WS-CWS Deployment

    •   Response Protocol Toolbox: Response Guidelines. An action oriented document to assist
       drinking water utilities, laboratories, emergency responders, state drinking water programs,
       technical assistance providers, and public health and law enforcement officials during the
       management of an ongoing contamination threat or incident. The Response Guidelines are not
       intended to replace to Response Protocol Toolbox and they do not contain the detailed
       information contained within the six complete modules. The Response Guidelines are to be
       viewed as the application of the same principles contained in the Response Protocol Toolbox
       during an actual incident. The Response Guidelines have been developed to provide an easy to
       use document for field and crisis conditions. Finally, users are encouraged to adapt the Response
       Guidelines as necessary to meet their own needs and objectives.
       http://www.epa.gov/safewater/watersecurity/pubs/rptb response guidelines.pdf
    •   Incident Command System (ICS) Training and National Incident Management System
       (NIMS) Compliance ICS is a command and control system designed to grow and contract to
       manage any magnitude of emergency.  ICS is a key component of NIMS, and NIMS is a key
       component of the National Response Plan (NRP) which directs how command over an incident is
       escalated from the local to state to federal level. Additionally, as directed by Homeland Security
       Presidential Directive (HSPD) 5, full NIMS compliance is a requirement for receiving federal
       preparedness funds More information on ICS and NIMS can be found at FEMA's website:
       http://www.fema.gov/emergencv/nims/index.shtm

    •   Federal Emergency Management Agency (FEMA) FEMA is the federal agency responsible for
       responding to and aiding the recovery from natural or man-made disasters.  The have developed
       the ICS and NIMS training program, as well as the National Response Plan (NRP) to establish an
       all-hazards response to emergencies for communities throughout the United States.
       www.fema.gov

    •   Department of Homeland Security (DHS) DHS offers many resources relating to water
       security, included training and research under the areas of awareness, prevention, protection,
       response  and recovery. The website is also contains information relating to bioterrorism laws,
       regulations and policies, as well as background information about the Homeland Security
       Presidential Directives, www.dhs.gov

    •   Centers for Disease Control and Prevention (CDC) The  CDC offers response plans for agents,
       diseases,  and other events through its Emergency Preparedness and Response branch. Specific
       training opportunities offered include those for Bioterrorism, Chemical, and Radiation
       Emergencies, www.bt.cdc.gov

    •   Federal Bureau of Investigation (FBI) The FBI is a major partner in investigating terrorist
       activities, and may be one of the responders to a contamination event. The FBI can also support
       local and state enforcement agencies, www.fbi.gov

    •   Water Information Sharing and Analysis Center (WaterlSAC) Online database containing
       information, expert analysis, and government alerts.  Provides tools for water security and links to
       other agencies, such as homeland security, law enforcement, and public health.
       www.waterisac.org


    •   The Association of State Drinking Water Administrators (ASDWA) A professional
       organization that supports states in their efforts to assure quality drinking water, and encourages
       coordination between state drinking water agencies. Provides tools and technical materials for
       area wide optimization programs, data management and security, www.asdwa.org


May 2007                                                                                 99

-------
                                Planning for WS-CWS Deployment

    •  American Water Works Association (AWWA) AWWA is a professional organization
       dedicated to improving water quality and supply. They provide numerous resources and training
       tools for use by utilities, including Water 101: Security Planning and Partnership for Safe Water
       online courses, www.awwa.org

    •  Utilities Helping Utilities: An Action Plan For Mutual Aid and Assistance Networks For
       Water and Wastewater Utilities A document developed by AWWA to help utilities develop
       Water and Wastewater Agency Response Networks (WARNs), a mutual  aid and assistance
       program to be used after a utility has sustain damage from man-made or natural disasters.
       http://www.awwa.org/Advocacy/govtaff/

    •  Effective Risk and Crisis Communication during Water Security Emergencies. This report
       summarizes results from three water security risk communication message mapping workshops
       conducted by U.S. EPA's National Homeland Security Research Center during 2005/2006. It
       provides information about effective message development and delivery that could be useful to
       water sector organizations as they develop their respective risk communication plans. Message
       mapping is a process by which users can predict 95 percent of questions likely to be asked by the
       media and others following an incident, prepare clear and concise answers to the questions  along
       with supporting information ahead of time, and practice effective message delivery before a crisis
       occurs.  nttp://www.epa.gov/nhsrc/pubs/reportCrisisCom040207.pdf

9.4    Staffing  and Cost Considerations
Staffing and cost considerations for development and implementation of a consequence management plan
as part of a contamination warning system may vary significantly from utility to utility based  on existing
capabilities and the approach used to develop the consequence management plan. Section 9.4.1 and 9.4.2
provide general considerations for staffing and costs, respectively.

9.4.1  Staffing

Response partners involved in development and implementation of the consequence management plan are
discussed in Section 9.1. The response partners needed to support consequence management  activities
may vary from utility to utility; however, once identified, they should be engaged from the pre-design
through the evaluation and refinement process. For the drinking water utility implementing a
contamination warning system, staffing considerations may vary based on the approach taken to develop
the plan.  Throughout development and implementation of the plan, it may be necessary to engage
representatives from each utility department or division. Ultimately, the departments or divisions
engaged may also vary based on response actions needed. While senior managers may have a more
significant role in development of the plan, all levels of staff may need to be engaged in some aspect of
training, participation in drills and exercises, and plan implementation.

9.4.2  Cost Considerations

Costs associated with development and implementation of a consequence management plan for
contamination warning systems may vary based on existing plans and capabilities within the utility and
the jurisdictions included in the utility's service area.  Most of the costs associated with development of
the consequence management plan are related to staff time.  In planning for contamination warning
system implementation, utilities should also consider costs associated with training, equipment (e.g.,
communications), and development and implementation of drills and exercises to evaluate the plan. Like
the development of the consequence management plan, most of the costs associated with training and
development of drills may be labor costs, and may vary based on how many people are involved.
May 2007                                                                                  100

-------
                               Planning for WS-CWS Deployment

When identifying cost considerations for developing the consequence management plan, it should be
considered that some of the equipment and training necessary to address gaps here my already be planned
for implementation under one of the  other contamination warning system components. As such, these
costs should be discussed amongst the project management team. Identifying instances such as this may
not only succeed in reducing overall  costs, but strengthens the concept that a contamination warning
system (and in particular a consequence management plan) requires the components to function together.
May 2007                                                                                 101

-------
                              Planning for WS-CWS Deployment


                           Section 10.0:   References

AWWA. 2004. Interim Voluntary Security Guidance for Water Utilities.

Hall, J. Zaffiro, A. D., Marx, R. B., Kefauver, P C., Krishnan, E. R, Haught, R C., Herrmann, J. G.
2007a. Online Water Quality Parameters as Indicators of Distribution System Contamination. Journal of
the American Waterworks Association, 99 (l):66-77.

Hall, J., Szabo, J.G. 2007b. Evaluation of Water Quality Monitoring Technologies to Respond to Changes
in Drinking Water Quality. Presented at GWRC On Line Monitoring Workshop; KIWA Facility,
Nieuwegein, NETHERLANDS, March 21-22.

Hall, J.S., Haught, R., Rahman, M., Richardson-Coy, R. and Piao, H. 2007c.  The bench-scale minimum
dose threshold experiment. EPA/600/R-07/002. Water Information Sharing and Analysis Center
(WaterlSAC) (www.waterisac.org).

Szabo, J.G., Hall, J.S. and Meiners, G.C.  2006. Water quality sensor responses to injected contaminants
in a chloraminatedpipe loop. American Water Works Association (AWWA) Water Security Congress,
Technical Session TUE6: Technology Forum B, Washington, DC, September 10-12, 2006

Szabo, J. et al. 2007. Water Quality Sensor Response to Contamination in a Single Pass Water
Distribution System Simulator. EPA-600-R-07-001. Available only through WaterlSAC

USEPA. 2003. Instructions to Assist Community Water Systems in Complying with the Public Health
Security and Bioterrorism Preparedness and Response Act of 2002.  EPA-810-B-02-001.

USEPA. 2004a.  Response Protocol Toolbox Overview. (EPA-817-D-03-007).

USEPA. 2004b.  Response Protocol Toolbox: Water Utility Planning Guide -Module I. (EPA-817-D-03-
001).

USEPA. 2004c.  Response Protocol Toolbox:  Contamination Threat Management Guide -Module 2.
(EPA-817-D-03-002).

USEPA. 2004d.  Response Protocol Toolbox:  Site Characterization and Sampling Guide -Module 3.
(EPA-817-D-03-003).

USEPA. 2004e.  Response Protocol Toolbox: Analytical Guide - Module 4.  (EPA-817-D-03-004).

USEPA. 2004f  Response Protocol Toolbox: Public Health Response Guide -Module 5. (EPA-817-D-
03-005).

USEPA. 2004g.  Response Protocol Toolbox:  Remediation and Recovery Guide -Module 6.  (EPA-817-
D-03-006).

USEPA. 2004h.  Response Protocol Toolbox:  Planning for and Responding to Drinking Water
Contamination Threats and Incidents Response Guidelines.

USEPA. 2005a.  WaterSentinel System Architecture.  EPA-817-D-05-003.

USEPA. 2005b. WaterSentinel Contaminant Selection. Sensitive Information, Limited Distribution, For
Official Use  Only.

May 2007                                                                               102

-------
                             Planning for WS-CWS Deployment





USEPA. 2005c. Overview of Event Detection Systems. EPA-817-D-05-001.
May 2007                                                                           103

-------
                                Planning for WS-CWS Deployment


                              Appendix A:  Glossary

Anomaly. Deviations from an established baseline or base state. Specifically, a water quality anomaly is
a deviation from an established water quality base state at a specific location.

Base State. Normal conditions that result from typical system operation. The base state includes
predictable fluctuations in measured parameters that result from known changes to the system. For
example a water quality base state includes the effects of draining and filling tanks, pump operation, and
pipe flushing, all of which may alter water quality in a somewhat predictable fashion.

Baseline Data. Baseline data is all available chemical, radiochemical, pathogen and toxin analytical data
relative to a baseline sample that may be used to determine "possible" or "credible" contamination.
Baseline data may be contaminant and location-specific control charts and tabulated contaminant data.

Concept of Operations (Con Ops). A process for routine operation of a drinking water contamination
warning system, which establishes  specific roles and responsibilities, process and information flows, and
procedural activities.  The Con Ops includes the process for validation of a contamination warning system
trigger and determining whether or not contamination is "possible."

Consequence Management Plan. Provides a decision-making framework that governs when, how,
what, and who will be involved in making decisions in response to a "possible" contamination incident in
order to minimize the response timeline and implement operational or public health response actions
appropriately.

"Credible." In the context of the credibility determination process, water contamination is characterized
as 'credible' if information collected during the investigation of "possible" contamination corroborates
information from the validated contamination warning system trigger.

Credibility Determination. Contamination warning system triggers will be investigated to determine
whether or not they are indicative of "possible" contamination. Credibility determination is the
subsequent investigation to determine whether or not additional information, including data from other
monitoring and surveillance components, corroborates the information from the validated trigger.  If the
additional information corroborates the trigger, contamination is considered 'credible.'

Event Detection System (EDS). A system designed specifically to detect anomalies from the various
monitoring and surveillance components of a contamination warning system. An EDS may take a variety
of forms, ranging from complex set of computer algorithms to a simple set of heuristics that are manually
implemented.  In essence, an EDS is a data mining tool that supports the efficient analysis of large
amounts of monitoring and surveillance data to pick out possible anomalies while at the same time
minimizing false alarms.

EDS Alarm.  A notification from the EDS tool that an anomaly has been detected.  Contamination
warning system alarms may be visible and/or audible, and may initiate automatic notifications such as
pager or e-mail alerts.  Most EDS alarms require some degree of validation before they are considered
indicative  of "possible" contamination.

Field Screening. Performing a series of tests to evaluate any potential chemical, biological or
radiochemical dangers present at the site.

Job Function. A description of the duties and responsibilities of a specific job within an organization.
May 2007                                                                                  104

-------
                                Planning for WS-CWS Deployment

Monitoring and Surveillance. Element of a contamination warning system that provide a standardized
set of information streams used in the detection of potential contamination incidents.

"Possible."  In the context of the contamination warning system concept of operations, water
contamination is characterized as "possible" if the cause of a trigger cannot be identified and/or
determined to be benign.

Risk Reduction Units (RRUs). The difference between the calculated risk before security improvements
versus after security improvements. A measure of enhanced security. RRU = (Risk Before
improvements) - (Risk After Improvements)

Security Breach. An unauthorized intrusion into a secured facility that may be discovered through direct
observation, an alarm trigger, or signs of intrusion (cut locks, open doors, cut fences).

Site characterization. The process of collecting information from an investigation site to support the
evaluation of a drinking water contamination threat

Standard Operating Procedure (SOP). A step-by-step list of actions that guide the user in the
implementation of a specific task.

Syndromic surveillance.  Collecting and analyzing nontraditional data to detect a change or trend in the
health of a population using categories of disease rather than formal diagnosis.

Target Contaminant. A contaminant that has been identified by the EPA for monitoring under the Water
Security Sentinel Initiative. Target contaminants are monitored using drinking water confirmatory
methods. Reported results are qualitative and quantitative

Threat Warning.  An unusual occurrence, observation or discovery that indicates a potential
contamination incident and initiates actions to address this concern.

Trigger. Information from a monitoring and surveillance component that an anomaly has been detected.

Trigger Validation. The process of investigating potential causes of a contamination warning system
trigger to either rule out contamination or determine that contamination is "possible."
May 2007                                                                                   105

-------
                                Planning for WS-CWS Deployment


         Appendix  B:   Information Security  Considerations

Because the Water Security initiative is first and foremost a homeland security program with a counter-
terrorism focus, information security is extremely important.  Certain utility materials and materials
developed in support of the program can potentially be exploited by adversaries to defeat the system.
Therefore, it is necessary for partner utilities to develop formal procedures and protocols for the
identification, handling, tracking, and overall security of any sensitive documents involved with the
development, implementation, and operation of their contamination warning system.  Training on and
abiding by these procedures and protocols should be considered for any and all staff who may access
sensitive materials as part of their job; this includes utility, partner, contractor and subcontractor
personnel.

Some key elements and steps in developing and implementing an information security strategy include:
   •  Assessment of Material Sensitivity, Access, and Law
   •  Sensitive Materials Tracking and Storage
   •  Sensitive Materials Handling Protocols and Procedures
   •  Staff Certification and Background Checks
   •  Coordination and Cooperation with Partners, Contractors and Subcontractors

Additionally, it is recommended that utility program management designate a team responsible for the
development, operation, and maintenance of an information security strategy, and reconciliation with
partner agencies sensitive information programs. This team should be comprised of members with
backgrounds in general and/or information security, information technology, emergency response, and
law and law enforcement.

B.1    Assessment of Material Sensitivity, Access, and Laws

The first step to implementing an information security program is to identify potentially sensitive
materials.  The information security team should consider existing utility and partner materials, as well as
materials that will be developed.  Examples include a utility's sensitive facilities list, security procedures
and emergency response procedures, hydraulic models,  facility maps and blueprints, locations and types
of contamination warning system enhancements and certain contact lists and notification procedures. As
materials are reviewed, the  team can begin developing a tiering system for different levels of sensitivity.
It is recommended that the specific terminology used in this tiering system draw on accepted
nomenclature (e.g., terms such as "For Official Use Only", "Sensitive", "Confidential", and
"Proprietary"). However, the terms chosen and the level of sensitivity they reflect should be clearly
defined in the information strategy document, and whether they truly afford any legal protection.

Also as part of the assessment, local, state, and federal laws and regulations should be reviewed to
determine what legal protections may be afforded to sensitive materials. Because local, state, and federal
governments are involved in contamination warning system implementation, the federal Freedom of
Information Act (FOIA), as well as similar laws and regulations passed  in other jurisdictions may be
applied to certain types of information, especially if the  utility itself is part of a government entity. The
FOIA legislation, implemented in 1967, governs the disclosure of documents and information controlled
by the U.S. government (this includes documents submitted to the government by an outside government
agency or private entity). In general, most information held by the government should be made available
to the public or other entities if requested, provided said information is not covered by one of nine
exemptions (such as certain types of confidential business information, or CBI, and National Security
Information, which has been designated exempt to protect the security of the Nation).  Since the
legislation went into effect, and particularly in the aftermath of 9/11, certain information, particularly
relating to critical infrastructure, like water utilities, has been determined to be covered by one or more of
the exemptions. Additionally, many other jurisdictions have exempted sensitive security information

May 2007                                                                                 106

-------
                                Planning for WS-CWS Deployment

related to critical infrastructure from there own regulations that are similar to FOIA. Local, state, and
federal laws related to disclosure of government held information should be reviewed to determine what
disclosure requirements and exemptions might apply to the information being assessed. It is
recommended that both the utility's legal counsel and that of the respective governing entity be engaged
in this process.

It is important to remember that sensitivity assessment of materials will remain an ongoing process
throughout the life of the contamination warning system; some materials may have their sensitivity
determination reconsidered while new materials will have to be placed in the ranking system. It is also
important to remember that new documents produced from sensitive materials should also be assessed for
their sensitivity, and possibly included in the sensitive materials program.

B.2     Sensitive Materials Tracking and Storage


Maintaining an effective tracking and storage system for sensitive materials is an important aspect of
drinking water security. Additionally, it will be necessary to exchange information with partner agencies
as part of the development and implementation of a contamination warning system.  Their sensitive data
should be protected as thoroughly as the utility's own documents, both so that the partners maintain
confidence in their partnership with the utility, and so that the utility does not become the conduit for
exploitable information falling into the wrong hands.

As materials are being assessed, a tracking and storage system should be developed. Materials can be
assigned tracking numbers which identify the date of receipt or creation, creating or owning agency, form
of the material (electronic, paper, etc.) kind of material (map, document, blueprint, etc.), whether the
material is a duplicate or is a number in a series  of duplicates, and other identifying pieces of information.

As part of the tracking system, a system for logging materials in and out should be implemented. This
could be as simple as a paper or electronic log sheet maintained by a responsible person or group within
the utility which lists the date, tracking number, and signature, initial, or other mark of the person
obtaining the material. More complex systems can involve confirmatory phone calls or e-mails for
materials that are mailed or sent electronically, or by using the receipt and tracking services of the U.S.
Postal Service and other carriers.

Storage refers to where the materials are stored when they are checked-in.  The utility should decide
whether a central storage location will be used, or whether different divisions will have storage areas  for
the materials they primarily use for their implementation efforts. In deciding on storage procedures and
location, a utility should consider such factors as whether the location is  physically secure from break-in,
whether a severe rain storm or other natural event might damage it, fire protection, and whether electronic
materials are protected from power failure.

B.3     Sensitive Materials Handling Protocol and Procedures

Proper handling of materials is important for the same reasons that storage and tracking are important. It
is a wasted effort to expend resources protecting documents while in storage, only to have them stolen
from a car, lost on the bus, left in a printer tray, or emailed to "reply to all". Sensitive materials handling
in this context refers to how those materials should be protected when checked-out of central storage.

Before or while materials are being assessed for sensitivity, handling procedures and protocols should be
developed. Handling protocols are fairly common anywhere sensitive information is used, from
government agencies to corporations intent on keeping proprietary or other sensitive information a secret,
and they can range from common sense  handling procedures, like locking them in a drawer when not in
use, to intensive procedures used for classified information.  The sensitive material handling procedures
should be specific to different forms of media and transmitting media, like e-mailing electronic

May 2007                                                                                   107

-------
                                Planning for WS-CWS Deployment

attachments.  Some resources that can be found online include EPA's Office of Science and Technology's
Confidential Business Information plan and the Toxic Substances Control Act Confidential Business
Information Protection Manual.  However, aside from minimum protections required by EPA, it will be
up to the utility and its partners to determine how conservative their sensitive materials plan is. Some
common precautions include:
    •    Shredding extra or damaged paper documents
    •    Locking documents up at night or when not in use for extended periods of time, even when in
        secure facilities
    •    Employing cover sheets for covering sensitive material when leaving one's workspace for a short
        period of time, or when a person not authorized to view the material enters the office. For
        electronic files, this would mean enabling the computer screen saver or locking the computer.
    •    Utilizing encryption software for electronic transmittal of materials.
    •    Only using company or agency email addresses (for example, not gmail, AOL, or hotmail
        accounts).
    •    Downloading attachments immediately and deleting the transmitting email.
    •    Prohibiting saving of sensitive materials on an unprotected Local Area Network (LAN).
    •    Double wrapping materials when transporting them outside secure facilities. Double wrapping
        consists of wrapping documents in an inner envelope or packaging that contains the "please
        return to" information, as well as clear markings that the packaging contains sensitive or
        proprietary information. An outer envelope or packaging should contain the same "please return
        to" information but not the sensitive or proprietary information label, so as not to arouse the
        curiosity of whoever finds it.
    •    Requiring the same level of protection for electronic media, like thumb drives, as are required for
        paper documents.
    •    Calling the recipient before faxing a document,  and requiring a confirmatory phone call back.
    •    Not leaving sensitive materials unattended in cars, even if the car is alarmed or locked.
    •    Utilizing hotel room safes

These types of precautions should also be observed whenever the sensitivity of a document or material is
in question, including for materials produced from sensitive materials, but have yet to be assessed for
their sensitivity.

Responsible handling of sensitive materials extends to protecting the data they contain as well.
Precautions should be taken to ensure that discussions of sensitive materials, or documents produced from
sensitive materials (and hence might contain sensitive materials themselves), are afforded equal
precautions. Care should be taken during conference calls to ensure all participants are cleared to discuss
sensitive information, and that others cannot join the call without the host's knowledge. Similarly, using
speaker-phone when discussing sensitive materials  should be avoided. Sensitive materials should not be
discussed in public places. To help this effort, persons cleared for access to sensitive materials should be
provided updated lists regularly of who else is cleared to view sensitive materials.

B.4    Staff Certification and Background Checks
Part of ensuring the security of sensitive materials is ensuring the integrity of those who will be handling
the materials. Background checks in general, and specific training on the handling of sensitive materials,
will help achieve this goal.

If not already done, background  checks should be performed on all staff involved in the design,
implementation, and operation of the major components of the contamination warning system: this
includes utility, partner, contractor and subcontractor staff. While it may not be necessary to perform a
background check on a member  of a maintenance crew whose day-to-day responsibilities will be largely
unaffected by the program, it will be necessary to perform them for staff who, say, have access to the
locations of monitoring stations.  The utility should assess its existing rules regarding which personnel

May 2007                                                                                  108

-------
                                Planning for WS-CWS Deployment

should have background checks, how extensive those checks are, and how they should adjust them based
on new roles and responsibilities resulting from implementing the contamination warning system. It is
strongly recommended that any staff who will be handling sensitive materials receive background checks.

In addition to background checks, a certification or training process should be implemented for staff who
will be handling sensitive materials.  At a minimum, staff should certify that they have read and
understand relevant materials related to the handling of sensitive materials, and lists should be kept
showing who has received such training and when they are due for recertification; signed nondisclosure
agreements can be utilized as part of this certification process.

B.5    Coordination and Cooperation with Partners, Contractors and Subcontractors
As mentioned, it may be necessary to provide certain sensitive materials to (or receive from) response
partners and contractor personnel.  While it is up to the utility to decide what level of precaution is
appropriate for their internal sensitive materials, they should be cognizant that their partners may have
more or less restrictive measures. Prior to the exchange of any sensitive materials, the utility should
provide their sensitive materials handling guidance to the partner agency and request the partner agency's
equivalent guidance.  This way, any conflicts between the two can be settled before materials are
exchanged. Additionally, formal non-disclosure agreements should be provided by contractors and
subcontractors to the utility, and the utility should check to see if any similar agreements partners have
with their contractors and subcontractors are consistent with the utility's procedures.

To ease this process, it is recommended that the utility seek guidance in developing their own sensitive
materials program from such partners as local counter-terrorism and intelligence groups, such as the FBI
local field office, or local Joint Terrorism Task Force (JTTF). Other local partners, such as public health
agencies, who have experience maintaining data with special handling requirements, can also be
consulted.
May 2007                                                                                   109

-------