Water Security Initiative: Interim Guidance on Planning for Contamination Warning System Deployment .. Contamination Warning System ------- Office of Water EPA817-R-07-002 May 2007 ------- Disclaimer The Water Security Division, of the Office of Ground Water and Drinking Water, has reviewed and approved this document for publication. Neither the United States Government nor any of its employees, contractors, or their employees make any warranty, expressed or implied, or assume any legal liability or responsibility for any third party's use of or the results of such use of any information, apparatus, product, or process discussed in this report, or represents that its use by such party would not infringe on privately owned rights. This document is not a substitute for applicable legal requirements, nor is it a regulation itself. Mention of trade names or commercial products does not constitute endorsement or recommendation for use. Questions concerning this document should be addressed to: Jessica Pulz U.S. EPA Water Security Division 26 West Martin Luther King Drive Cincinnati, OH 45268-1320 (513)569-7918 Pulz.Jessica@epa.gov or Steve Allgeier U.S. EPA Water Security Division 26 West Martin Luther King Drive Cincinnati, OH 45268-1320 (513)569-7131 Allgeier. Steve @epa. gov ------- Purpose of this Document EPA intends this guidance manual to assist drinking water utilities with planning for contamination warning system deployment based on the model developed under EPA's Water Security initiative (formerly known as WaterSentinel). In particular, this manual may aid respondents to an upcoming EPA Request for Applications (RFA). Under this RFA, the Agency would make financial awards for drinking water utilities to demonstrate and evaluate contamination warning system pilots. EPA anticipates issuing this RFA in June 2007. Further, EPA plans to issue additional interim guidance on contamination warning system operation and consequence management planning in late 2007. All these interim guidance manuals will then be revised as needed based on findings of the demonstration pilots and public comment prior to being issued in final form. Along with this guidance, the Agency intends to develop outreach program to promote adoption of effective and sustainable drinking water contamination warning systems. The following is a summary of these additional documents. Request for Applications (Summer 2007) will solicit proposals for financial awards to assist drinking water utilities with demonstrating and evaluating contamination warning system pilots based on the Water Security initiative model. Interim Concept of Operations Guidance (Fall 2007) will describe the process and procedures involved in routine operation of a contamination warning system, including process and information flows, roles and responsibilities, and the initial investigation and validation of alarms. This document could be used by drinking water utilities to inform and refine component-specific designs and support deployment and operation of their contamination warning system. Interim Consequence Management Plan Guidance (Fall 2007) will assist drinking water utilities in the development and implementation of a utility-specific consequence management plan for an existing or emerging contamination warning system. This guidance will also address integration of the consequence management plan with existing plans, training and exercise scenarios, and outreach to other local, state, and federal agencies. Request for Comments EPA is soliciting suggestions and recommendations to make this interim guidance manual more complete and user-friendly. Commenters are encouraged to be as specific as possible and to provide references where appropriate. Submit suggestions by e-mail to: watersecurity@epa.gov and indicate that the message relates to the "Interim Guidance on Planning for Contamination Warning System Deployment." ------- Planning for WS-CWS Deployment Abbreviations and Acronyms APHL Association of Public Health Laboratories AWWA American Water Works Association CDC Centers for Disease Control and Prevention CFR Code of Federal Regulations CID Criminal Investigation Division CPTED Crime Prevention through Environmental Design CWA Clean Water Act DHS Department of Homeland Security EARS Early Aberration Reporting System EDS Event Detection System EMS Emergency Medical Service EPA Environmental Protection Agency ESSENCE Electronic Surveillance System for the Early Notification of Community Based Epidemics ETV Environmental Technology Verification FBI Federal Bureau of Investigation FEMA Federal Emergency Management Agency GCWW Greater Cincinnati Water Works CIS Geographic Information System HAZWOPR Hazardous Waste Operations HIPAA Heath Insurance Portability and Accountability Act ICS Incident Command System IT Information Technology LRN Laboratory Response Network MRL Minimum Reporting Level MOU Memorandum of Understanding NACWA National Association of Clean Water Agencies NELAC National Environmental Laboratory Accreditation Conference NELAP National Environmental Laboratory Accreditation Program NEMI National Environmental Methods Index NIMS National Incident Management System NLTN National Laboratory Training Network NRDM NRP NSF ORP OSHA OTC O&M PCS PIR PLC PPE QA QC RAM-W RFA ROC RODS RPTB RRU SAM SCADA SDWA SOP TEVA TEVA SPOT TOC TTEP T-JM USDA VOC VSAT1 WC IT WS WS-CWS National Retail Data Monitor National Response Plan National Sanitation Foundation Oxidation Reduction Potential Occupation Safety and Health Administration Over-the-counter (drug sales) Operation and Maintenance Polychlorinated Biphenyl Passive Infrared Programmable Logic Controller Personal Protective Equipment Quality Assurance Quality Control Risk Assessment Methodology for Water Utilities Request for Applications Receiver Operating Characteristic Real-time Outbreak and Disease Surveillance Response Protocol Tool-box Risk Reduction Units Standardized Analytical Methods Supervisory Control and Data Acquisition Safe Drinking Water Act Standard Operating Procedure Threat Ensemble Vulnerability Assessment TEVA Sensor Placement Optimization Tool Total Organic Carbon Technology Testing and Evaluation Program United States Department of Agriculture Volatile Organic Compound Vulnerability Self Assessment Tool Water Contaminant Information Tool Water Security WS Contamination Warning System May 2007 ------- Planning for WS-CWS Deployment Acknowledgements EPA's Office of Ground Water and Drinking Water would like to recognize the following individuals and organizations for their assistance and contributions in development of this document: Technical Support Center Eric Bissonette Office of General Counsel Peter Ford Water Security Division Steve Allgeier Jeffrey Pencil David Harvey Elizabeth Hedrick Mike Henrie Nancy Muzzy Brian Pickard Jessica Pulz Dan Schmelling David Travers National Homeland Security Research Center Kathy Clayton Contractor Support Yildiz Chambers, CSC John Chandler, CSC Kevin Cornell, CSC Mike Denison, CH2M Hill Bill Desing, CH2M Hill Jean Dupree, CSC Todd Elliott, CH2M Hill Katie Gavit, CSC Darcy Gibbons, CSC David Gnugnoli, CSC Yakir Hassit, CH2M Hill Gary Jacobson, CH2M Hill Reese Johnson, CH2M Hill Colm Kenny, CH2M Hill Kim Morgan, CSC Misty Pope, CSC Curtis Robbins, CH2M Hill Jerry Scott, CSC Doron Shalvi, CSC Scott Weinfeld, CSC May 2007 ------- Planning for WS-CWS Deployment Executive Summary This manual provides an overview of design and implementation considerations to assist drinking water utilities in planning for contamination warning system deployment. As a planning tool, this document describes a general framework and process for implementation and identifies available tools and resources to support implementation of a contamination warning system. What is a contamination warning system? A contamination warning system provides drinking water utilities with a systematic and comprehensive approach for monitoring and surveillance of the distribution system. Through implementation of the monitoring and surveillance strategies and a comprehensive consequence management plan, utilities can improve their ability to detect intentional or unintentional distribution system contamination. In addition, their increased ability to monitor and understand distribution system water quality may help to optimize operations and improve the overall quality of the product delivered to customers. Monitoring and surveillance components of the contamination warning system include the following: Online water quality monitoring comprises stations located throughout the distribution system that measure parameter such as chlorine, total organic carbon, conductivity, and pH among others. Software analyzes the monitoring data to establish a water quality base state. Possible contamination is indicated when a significant, unexplained deviation from the base state occurs. Sampling and analysis is the collection of distribution system samples that are analyzed for various contaminant classes as well as specific contaminants. Sampling is both routine to establish a baseline and triggered to respond to an indication of possible contamination from another component. Analyses are conducted for chemicals, radionuclides, pathogens, and toxins using a laboratory network. Enhanced security monitoring includes the equipment and procedures that detect and respond to security breaches at distribution system facilities. Security equipment may include cameras, motion activated lighting, door contact alarms, ladder and window motion detectors, area motion detectors, and access hatch contact alarms. Consumer complaint surveillance enhances the collection and automates the analysis of calls by consumers for water quality problems indicative of possible contamination. Consumers may detect contaminants with characteristics that impart an odor, taste, or visual change to the drinking water. Public health surveillance involves the analysis of health-related data to identify disease events that may stem from drinking water contamination. Public health data may include over-the- counter (OTC) drug sales, hospital admission reports, infectious disease surveillance, emergency medical service (EMS) reports, 911 calls, and poison control center calls. In addition to these monitoring and surveillance components, consequence management is a critical aspect of the overall architecture for a contamination warning system. Consequence management refers to the procedures and protocols for assessing credibility of a contamination incident and implementing response actions. Why deploy a contamination warning system? Monitoring the distribution system is the primary focus of contamination warning systems. Through the assessment of vulnerabilities to drinking water systems, water security experts have identified the distribution system as one of the most vulnerable components in a drinking water utility, with respect to contamination. Furthermore, intentional contamination, or even the threat of contamination, can have significant impacts. May 2007 ------- Planning for WS-CWS Deployment Drinking water utilities occasionally receive threats or indications of possible contamination. These contamination threat warnings can be a direct threat or an unusual observation or discovery that indicates the potential for contamination and initiates actions to investigate and potentially respond. However, these threat warnings are not standardized and are difficult to corroborate in the absence of an integrated monitoring and surveillance system and close coordination with response partners. Deployment of contamination warning systems for drinking water distribution systems provides a mechanism by which drinking water utilities can detect and respond to contamination threats and incidents. A contamination warning system is a proactive approach to managing threat warnings that uses advanced monitoring technologies/strategies and enhanced surveillance activities to collect, integrate, analyze, and communicate information to provide a timely warning of potential water contamination incidents and initiate response actions to minimize public health and economic impacts. In addition, contamination warning system implementation provides the opportunity for dual-use applications beyond security that could help to promote sustainability of the system by optimizing utility operations. Drinking water distribution systems may be accidentally contaminated through cross- connections with non-potable water, permeation of contaminated water through leaking pipes in areas of the distribution system subject to low pressures, or chemical reactions or microbial growth within the distribution system pipes. Such unintentional events that result in degradation to distributed water quality may occur with some regularity. Potential dual-use benefits of a contamination warning system could include the following: Detection of cross-connections and other distribution system water quality problems Improved relationship with public health organizations, including mutual sharing of information and alerts Enhanced knowledge of distribution system water quality leading to improved operations (e.g., more consistent disinfection residual levels, improved corrosion control, early warning of nitrification episodes, reduced disinfection byproduct levels, etc.) Identification of problem valves (closed, partially closed, inoperable) Improved coordination with local, state, and federal response organizations Reduced occurrence of tampering and vandalism Improved information technology systems and interoperability Improved consumer complaint tracking and response Improved laboratory capability and an established laboratory network Consequence management plans applicable to any water quality emergency What approach or framework should be applied for deployment of a contamination warning system? A contamination warning system is, by design, a systematic approach to monitoring and surveillance for the timely detection of drinking water contamination. As such, deployment of a contamination warning system relies on the application of system engineering principles to support coordination of technical and management activities. Through system engineering, disciplines and specialty groups are integrated in a team effort forming a structured development process that proceeds from design to implementation to operation. From the beginning of the project, system engineering principles are critical to successful planning and implementation. The primary application of system engineering for a contamination warning system is to ensure that the system - monitoring and surveillance components and consequence management - functions as an integrated whole. System engineering principles should be applied to every aspect of contamination warning system implementation, including staffing. While routine operation and maintenance of the contamination warning system should generally fall within the routine job functions of utility staff, design and implementation may involve significant time and effort from dedicated managers within the utility. Depending on utility organization and operational approach, these May 2007 iv ------- Planning for WS-CWS Deployment activities may be managed by one individual, or more likely, a core, multi-disciplinary project management team. As discussed in Section 3, deployment of a contamination warning system at a water utility should follow the typical programmatic approach in which proposed enhancements are planned, designed, implemented, tested, maintained and refined. Table ES-1 provides a summary of the design and implementation framework applied throughout the document. Table ES-1. Overview of Design and Implementation Framework Stage of Approach Planning and pre- design Design Implementation Preliminary testing Operation and maintenance Evaluation and refinement Description Developing a core implementation team, defining design objectives to guide implementation, and a preliminary assessment of existing capabilities relative to design objectives. Development of a preliminary concept of operations and development of a detailed work plan and schedule to guide implementation. Implementation of enhancements, installation of equipment, and training according to the plan. Operation of the contamination warning system for the purpose of collecting data necessary to understand system performance and finalization of the concept of operations to optimize system. Operation of the contamination warning system for the purpose of monitoring for contamination incidents and other water quality issues. Analysis of data and information generated during full operation to refine and optimize the system. Who should be involved in contamination warning system deployment? The drinking water utility is the operational hub of the system as the primary operator of the majority of monitoring and surveillance components of the contamination warning system, with the exception of public health surveillance. However, other partners may be involved in initial investigation of alarms (trigger validation) and/or consequence management activities. Figure ES-1 provides an overview of potential partners in contamination warning system implementation. As illustrated in this figure, the number and scope of partners that can become involved in responding to a contamination event can be significant. In planning for implementation of a contamination warning system, drinking water utilities should identify and engage local partners early in the process, particularly those partners such as local health departments and public health and environmental laboratories that may have a significant role in routine operations. Specific responsibilities of partners and when they are engaged may vary by utility and jurisdiction. Section 2 of the document provides additional details regarding the roles and responsibilities of external partners. May 2007 ------- Planning for WS-CWS Deployment Federal Bureau of Investigation State Emergency Management and Homeland Security Agencies State Law Enforcement Centers For Disease Control and Prevention EPA Regional Offices Local Health Department Local Fire, EMS, and HazMat Local Emergency Planning Committees Local Wastewater Utility Host Facilities Local Law Enforcement Local Civil Government Public Health and Environmental Laboratories State Drinking and Waste Water Primacy Agencies Neighboring Utilities EPA Criminal Investigation Division EPA National Response Center State Emergency Res ponders State Government Media Figure ES-1. Potential Contamination Warning System Partners What are key design considerations for contamination warning system deployment? Using this document as a guide, the utility in collaboration with local partners where appropriate, should define what it wants the system to do (defining the design basis), develop a preliminary model of how the system would function (developing a preliminary concept of operations) and compare the model to the utility's current capabilities in order to identify the gaps and inform the plan to achieve the desired goals and objectives. Design decisions to support planning for implementation of a contamination warning system are summarized in Table ES-2. It is important to apply system engineering principles to all aspects of design and implementation, particularly as they relate to utility IT systems. Table ES-2. Summary of Design Decisions to Support Planning for Contamination Warning System Implementation Component Document Section Design Decisions Online Water Quality Monitoring Water quality parameters to be monitored Use of a single monitoring station design or multiple designs in a tiered system Specific sensors and instruments integrated into a water quality monitoring station Number of water quality monitoring stations to install Methodology for determining the locations at which water quality monitoring stations should be installed Communication architecture to transmit data from monitoring locations to an operations center IT architecture used to manage and store water quality and related data Event detection software deployed to detect anomalies Sampling and Analysis List of target contaminants, including contaminant class; responsible laboratory; analytical method; laboratory certification or accreditation for the method Sampling plan that addresses the training that staff should receive; sampling equipment that should be procured; sampling locations, and rationale; and sampling frequency, and rationale Procedures for triggered sampling and analysis Site characterization procedures and responsibilities May 2007 VI ------- Planning for WS-CWS Deployment Component Document Section Design Decisions Enhanced Security Monitoring Preliminary facility list Site assessment summaries Facility risk ranking including a summary of physical security effectiveness, probability of attack, and consequence of contamination criteria Final facility list with a description of recommended improvements for each facility Communication architecture to transmit data from monitoring locations to an operations center Consumer Complaint Surveillance The utility-specific model of the consumer complaint surveillance component design An assessment of the existing consumer complaint management system An approach for enhancing the consumer complaint management system into a consumer complaint surveillance system IT architecture used to manage and store water quality complaint and related data Event detection software deployed to detect anomalies Public Health Surveillance Identification of local public health partners Identification and assessment of existing surveillance capability relative to contamination warning system objectives Improvements or additions to existing surveillance capabilities Development of a framework for communication and notification Consequence Management Objectives for consequence management plan Utility self-assessment of existing plans and response capabilities Identification and assessment of response partner capabilities Level of integration with other response plans Framework for development of consequence management plan that allows for seamless transition from routine operations and initial trigger validation to consequence management actions May 2007 VII ------- Planning for WS-CWS Deployment Water Security Initiative: Interim Guidance on Planning for Contamination Warning System Deployment ABBREVIATIONS AND ACRONYMS I ACKNOWLEDGEMENTS II EXECUTIVE SUMMARY Ill SECTION 1.0: INTRODUCTION 1 1.1 CONTAMINATION WARNING SYSTEMS - AN OVERVIEW 2 1.2 DOCUMENT OVERVIEW 7 SECTION 2.0: PROJECT PLANNING AND MANAGEMENT 8 2.1 APPLICATION OF SYSTEM ENGINEERING PRINCIPLES 8 2.1.1 Development and Management of Work Plan and Schedule 8 2.1.2 Integrated Concept of Operations 9 2.1.3 IT System Engineering 10 2.2 UTILITY STAFFING 11 2.2.1 Project Management Team 11 2.2.2 Utility Staff 12 2.3 LOCAL PARTNERS 13 2.3.1 Identifying and Engaging Partners 14 2.3.2 Considerations for Formal Agreements with Local Partners 16 2.4 COSTS 17 SECTION 3.0: DESIGN AND IMPLEMENTATION FRAMEWORK 18 3.1 PLANNING AND PRE-DESIGN 18 3.1.1 Building the Team 18 3.1.2 Defining the Utility-Specific Design Basis and Design Objectives 18 3.1.3 Preliminary Assessment and Gap Analysis 20 3.2 DESIGN 20 3.2.1 Conceptualize System 20 3.2.2 Work Plan for Implementation 21 3.3 IMPLEMENTATION 21 3.4 PRELIMINARY TESTING 21 3.4.1 Baseline Operation 22 3.4.2 Finalization of Concept of Operations and Consequence Management Plan 22 3.5 OPERATION AND MAINTENANCE 23 3.5.1 Operation 23 3.5.2 Maintenance 23 3.6 EVALUATION AND REFINEMENT 23 3.6.1 Evaluation 24 3.6.2 Refinement 24 SECTION 4.0: ONLINE WATER QUALITY MONITORING 25 4.1 MONITORING NETWORK DESIGN 26 4.1.1 Pre-Design 27 4.1.2 Design and Implementation Approach 29 4.1.3 Available Tools and Resources 30 4.2 MONITORING STATION DESIGN AND INSTALLATION 31 4.2.1 Pre-Design 31 4.2.2 Design and Implementation Approach 33 4.2.3 Available Tools and Resources 38 4.3 COMMUNICATIONS ARCHITECTURE 38 4.3.1 Pre-Design 39 May 2007 viii ------- Planning for WS-CWS Deployment 4.3.2 Design and Implementation Approach 39 4.3.3 Available Tools and Resources 40 4.4 DATA MANAGEMENT AND IT ARCHITECTURE 41 4.4.1 Pre-Design 41 4.4.2 Design and Implementation Approach 42 4.4.3 Available Tools and Resources 42 4.5 WATER QUALITY EVENT DETECTION 43 4.5.1 Planning 43 4.5.2 Implementation Approach 46 4.5.3 Available Tools and Resources 47 4.6 STAFFING AND COST CONSIDERATIONS 48 4.6.1 Staffing 48 4.6.2 Cost Considerations 49 SECTION 5.0: SAMPLING AND ANALYSIS 51 5.1 LABORATORY CAPABILITY AND CAPACITY 52 5.1.1 Pre-design 52 5.1.2 Design and Implementation Approach 55 5.1.3 Available Tools and Resources 56 5.2 SAMPLING AND ANALYSIS 56 5.2.7 Pre-design 57 5.2.2 Design and Implementation Approach 58 5.2.3 Available Tools and Resources 61 5.3 SITE CHARACTERIZATION AND FIELD SCREENING 61 5.3.7 Pre-design 61 5.3.2 Design and Implementation Approach 63 5.3.3 Available Tools and Resources 65 5.4 STAFFING AND COST CONSIDERATIONS 65 5.4.1 Staffing 65 5.4.2 Cost Considerations 66 SECTION 6.0: ENHANCED SECURITY MONITORING 67 6.1 PRE-DESIGN 69 6.2 DESIGN AND IMPLEMENTATION APPROACH 72 6.3 AVAILABLE TOOLS AND RESOURCES 75 6.4 STAFFING AND COST CONSIDERATIONS 76 6.4.1 Staffing 76 6.4.2 Cost Considerations 77 SECTION 7.0: CONSUMER COMPLAINT SURVEILLANCE 78 7.1 PRE-DESIGN 80 7.2 DESIGN AND IMPLEMENTATION APPROACH 81 7.3 AVAILABLE TOOLS AND RESOURCES 82 7.4 STAFFING AND COST CONSIDERATIONS 83 7.4.1 Staffing 83 7.4.2 Cost Considerations 84 SECTION 8.0: PUBLIC HEALTH SURVEILLANCE 85 8.1 PRE-DESIGN 86 8.2 DESIGN AND IMPLEMENTATION APPROACH 89 8.3 AVAILABLE TOOLS AND RESOURCES 90 8.4 STAFFING AND COST CONSIDERATIONS 91 8.4.1 Staffing 91 8.4.2 Cost Considerations 92 SECTION 9.0: CONSEQUENCE MANAGEMENT 94 9.1 PRE-DESIGN 94 9.2 DESIGN AND IMPLEMENTATION APPROACH 97 9.3 AVAILABLE TOOLS AND RESOURCES 98 May 2007 ix ------- Planning for WS-CWS Deployment 9.4 STAFFING AND COST CONSIDERATIONS 100 9.4.1 Staffing 100 9.4.2 Cost Considerations 100 SECTION 10.0: REFERENCES 102 APPENDIX A: GLOSSARY 104 APPENDIX B: INFORMATION SECURITY CONSIDERATIONS 106 B.I ASSESSMENT OF MATERIAL SENSITIVITY, ACCESS, AND LAWS 106 B.2 SENSITIVE MATERIALS TRACKING AND STORAGE 107 B.3 SENSITIVE MATERIALS HANDLING PROTOCOL AND PROCEDURES 107 B.4 STAFF CERTIFICATION AND BACKGROUND CHECKS 108 B.5 COORDINATION AND COOPERATION WITH PARTNERS, CONTRACTORS AND SUBCONTRACTORS 109 List of Tables TABLE ES-l. OVERVIEW OF DESIGN AND IMPLEMENTATION FRAMEWORK v TABLE ES-2. SUMMARY OF DESIGN DECISIONS TO SUPPORT PLANNING FOR CONTAMINATION WARNING SYSTEM IMPLEMENTATION vi TABLE 1-1. CONTAMINANT DETECTION CLASSES AND POTENTIAL MEANS OF DETECTION 5 TABLE 1-2. DESIGN BASIS SUMMARY BY CONTAMINATION WARNING SYSTEM COMPONENT 6 TABLE 2-1. SUMMARY OF POTENTIAL CONTAMINATION WARNING SYSTEM PARTNERS 14 TABLE 3-1. OVERVIEW OF DESIGN AND IMPLEMENTATION FRAMEWORK 18 TABLE 3-2. DESIGN BASIS CONSIDERATIONS 19 TABLE 4-1. DESIGN BASIS CONSIDERATIONS FOR ONLINE WATER QUALITY MONITORING 25 TABLE 4-2. IMPACT OF CONTAMINANT DETECTION CLASSES ON WATER QUALITY PARAMETERS 31 TABLE 4-3. STANDARD MEASURES FOR EVALUATING EDS TOOL PERFORMANCE 44 TABLE 4-4. SAMPLE MEASURES FOR EVALUATING EDS SOFTWARE 45 TABLE 4-5. ONLINE WATER QUALITY MONITORING STAFFING CONSIDERATIONS 48 TABLE 5-1. DESIGN BASIS CONSIDERATIONS FOR SAMPLING AND ANALYSIS 51 TABLE 5-2. CONSIDERATIONS FOR ANALYTICAL APPROACH TO ESTABLISHING SAMPLING AND ANALYSIS CAPABILITIES BY CONTAMINANT CLASS 52 TABLE 5-3. EXAMPLES OF BASELINE DATA SOURCES 59 TABLE 5-4. CONSIDERATIONS FOR CONTAMINANT COVERAGE FOR FIELD SCREENING 63 TABLE 5-5. SAMPLING AND ANALYSIS STAFFING CONSIDERATIONS 65 TABLE 6-1. DESIGN BASIS CONSIDERATIONS FOR ENHANCED SECURITY MONITORING AT SELECTED SITES 67 TABLE 6-2. EXAMPLE IMPROVEMENTS BY WATER UTILITY FACILITY TYPE 68 TABLE 6-3. ENHANCED SECURITY MONITORING STAFFING CONSIDERATIONS 76 TABLE 7-1. DESIGN BASIS CONSIDERATIONS FOR CONSUMER COMPLAINT SURVEILLANCE 78 TABLE 7-2. CONSUMER COMPLAINT SURVEILLANCE STAFFING CONSIDERATIONS 83 TABLE 8-1. DESIGN BASIS CONSIDERATIONS FOR PUBLIC HEALTH SURVEILLANCE 86 TABLE 8-2. PUBLIC HEALTH SURVEILLANCE STAFFING CONSIDERATIONS 92 TABLE 9-1. OVERVIEW OF RESPONSE PARTNER ROLES AND RESPONSIBILITIES FOR CONSEQUENCE MANAGEMENT ..96 List of Figures FIGURE ES-l. POTENTIAL CONTAMINATION WARNING SYSTEM PARTNERS vi FIGURE 1-1. OVERVIEW OF EPA's WATER SECURITY INITIATIVE 1 FIGURE 1-2. ARCHITECTURE OF THE WATER SECURITY CONTAMINATION WARNING SYSTEM 3 FIGURE 2-1. POTENTIAL CONTAMINATION WARNING SYSTEM PARTNERS 14 FIGURE 2-2. RECOMMENDED STRATEGY FOR ENGAGING CONTAMINATION WARNING SYSTEM PARTNERS 16 FIGURE 4-1. EXAMPLE MONITORING STATION TRADEOFF CURVE 28 FIGURE 4-2. EXAMPLE WATER QUALITY MONITORING STATION DESIGN USED IN THE INITIAL PILOT 35 FIGURE 4-3. EXAMPLE ROC CURVE 45 FIGURE 7-1. THE RECOMMENDED FILTER, FUNNEL, AND Focus APPROACH TO CUSTOMER FEEDBACK DATA OPTIMIZATION FOR UTILITY-MANAGED CONSUMER CALLS 80 FIGURE 8-1 PUBLIC HEALTH DATA SOURCES AND DESIGN OBJECTIVES 87 May 2007 ------- Planning for WS-CWS Deployment Section 1.0: Introduction This document presents a basic framework to assist drinking water utilities with planning for contamination warning system deployment based on the model developed under U.S. Environmental Protection Agency's (EPA) Water Security initiative (formerly known as WaterSentinel). Initiated in response to Homeland Security Presidential Directive 9, the overall goal of the Water Security initiative is to design and deploy contamination warning systems for drinking water utilities through a phased approach that includes conceptual design, implementation at an initial pilot utility, expansion to additional pilot utilities, and ultimately development of guidance and tools to support implementation at drinking water utilities across the nation. Figure 1-1 summarizes this process. Phase Approach Scope Design Specificity Funding DESIGN System Architecture DEMONSTRATE Initial Pilot Additional Pilots EXPAND Voluntary National Adoption ^^~~[S Applied by^T/ Apply to single Evaluate multiple Evaluate Conceptual , N Pilot utilitV ^^ U"eS ^^ C°nvert'° design ^^ ^ ["] ^ H guidance for AX . r/ AX r/ V 7 Refine !__£ \J Refine 1_J> \/ anc| ^ and enhance enhance Not applicable Low ll High- Applies to pilot utility only & ml ia s iL^ A High- Appliesto each pilot EPA Funds iW Medium - Applies to range of utilities Utility Funds Figure 1-1. Overview of EPA's Water Security Initiative Monitoring the distribution system is the primary focus of contamination warning systems. Through the assessment of vulnerabilities to drinking water systems, water security experts have identified the distribution system as one of the most vulnerable components in a drinking water utility, with respect to contamination. Furthermore, intentional contamination, or even the threat of contamination can have significant impacts. Drinking water utilities occasionally receive threats or indications of possible contamination. These contamination threat warnings can be a direct threat or an unusual observation or discovery that indicates the potential for contamination and initiates actions to investigate and potentially respond. However, these threat warnings are not standardized and are difficult to corroborate in the absence of an integrated monitoring and surveillance system and close coordination with response partners including, but not limited to public health, emergency responders, and law enforcement. Deployment of contamination warning systems for drinking water distribution systems provides a mechanism by which drinking water utilities can detect and respond to contamination threats and incidents. A contamination warning system is a proactive approach to managing threat warnings that uses advanced monitoring technologies/strategies and enhanced surveillance activities to collect, integrate, analyze, and communicate information to provide a timely warning of potential water contamination incidents and initiate response actions to minimize public health and economic impacts. May 2007 ------- Planning for WS-CWS Deployment In addition, deployment of a contamination warning system provides the opportunity for dual-use applications beyond security that could help to promote sustainability of the system by optimizing utility operations. Drinking water distribution systems may be accidentally contaminated through cross- connections with non-potable water, permeation of contaminated water through leaking pipes in areas of the distribution system subject to low pressures, or chemical reactions or microbial growth within the distribution system pipes. Such unintentional events that result in degradation to distributed water quality may occur with some regularity. In 2005, EPA documented the conceptual design for contamination warning systems in WaterSentinel System Architecture (USEPA, 2005a) and began implementation of the first WS contamination warning system pilot in partnership with the City of Cincinnati at the Greater Cincinnati Water Works (GCWW). Section 1.1 provides an overview of contamination warning systems and a summary of the design basis and Section 1.2 provides an overview of how to use this document to support design and implementation of contamination warning systems based on EPA's approach and lessons learned from the initial pilot. 1.1 Contamination Warning Systems - An Overview A contamination warning system is not merely a collection of monitors and equipment placed throughout a water system to alert of intrusion or contamination. Fundamentally, it is an exercise in information acquisition and management. Different information streams are captured, managed, analyzed, and interpreted to recognize potential contamination incidents in time to respond effectively. These data sources, when used concurrently, should support and augment each other such that the chances of detecting a contamination incident are better than using any one information source on its own. While the contamination warning system should be designed by the utility, some data sources may be outside of the utility; thus, cooperation with partners is an integral part to the success of a contamination warning system. A complete contamination warning system consists of the following monitoring and surveillance components: Online water quality monitoring comprises stations located throughout the distribution system that measure chlorine, total organic carbon, conductivity, and other parameters. Software analyzes the monitoring data to establish a water quality base state. Possible contamination is indicated when a significant, unexplained deviation from the base state occurs. Sampling and analysis is the collection of distribution system samples that are analyzed for various contaminant classes as well as specific contaminants. Sampling is both routine to establish a baseline and triggered to respond to an indication of possible contamination from another component. Analyses are conducted for chemicals, radionuclides, pathogens, and toxins using a laboratory network. Enhanced security monitoring includes the equipment and procedures that detect and respond to security breaches at distribution system facilities. Security equipment may include cameras, motion activated lighting, door contact alarms, ladder and window motion detectors, area motion detectors, and access hatch contact alarms. Consumer complaint surveillance enhances the collection and automates the analysis of calls by consumers for water quality problems indicative of possible contamination. Consumers may detect contaminants with characteristics that impart an odor, taste, or visual change to the drinking water. Public health surveillance involves the analysis of health-related data to identify disease events that may stem from drinking water contamination. Public health data may include over-the- counter (OTC) drug sales, hospital admission reports, infectious disease surveillance, emergency medical service (EMS) reports, 911 calls, and poison control center calls. As illustrated in Figure 1-2, consequence management is another key aspect of the contamination warning system architecture. Consequence management governs response actions when contamination is May 2007 ------- Planning for WS-CWS Deployment determined to be possible and includes activities that involve drinking water utilities as well as external partners. Figure 1-2. Architecture of the Water Security Contamination Warning System The basic process or conceptual model for contamination warning system operation is described as follows, moving from left to right in the diagram. Monitoring and Surveillance. As previously discussed, integration of information from a variety of data sources internal and external to the drinking water utility is a critical aspect of contamination warning system monitoring and surveillance activities. While the specific types of information streams may vary, the basic components of online water quality monitoring, sampling and analysis, enhanced security monitoring, consumer complaint surveillance, and public health surveillance are necessary to meet the design objectives as discussed later in this section. Monitoring and surveillance of these components and information streams occurs on a routine basis, in near-real time until an anomaly or deviation from the baseline or base state is detected. Event detection and Possible Determination. Event detection is the process or mechanism by which an anomaly or deviation from the baseline or base state is detected. This detection is referred to as a trigger. How event detection is implemented and the tools that are utilized may vary significantly from component to component and can include sophisticated algorithms, simple business logic, etc. This process should be automated to the extent possible. Another aspect of the contamination warning system that is tightly coupled with event detection is the initial validation of triggers to assess possibility of drinking water contamination. As discussed in greater detail throughout this document, many of the components provide non-specific indicators of contamination and thus, when a trigger occurs, validation is necessary prior to determining if contamination is "possible." If trigger validation indicates that contamination is "possible," then the credibility determination step is initiated; otherwise, the contamination warning system component returns to routine monitoring and surveillance. Credible Determination. Credibility determination is a transition from routine operation to consequence management. Credibility determination procedures are performed using information from all contamination warning system components as well as external resources when available and relevant. Through the credibility determination process, some preliminary response actions may be initiated to limit or minimize impacts of suspected contamination. If contamination is determined to be credible, additional confirmatory and response actions are initiated. If May 2007 3 ------- Planning for WS-CWS Deployment contamination can be ruled out based on additional information gathered through the credibility determination process, the system returns to routine monitoring and surveillance activities. Confirmed Determination. In this stage of consequence management, additional information is gathered and assessed to confirm drinking water contamination. Response actions initiated during credible determination are expanded and additional response activities may be implemented. Remediation and Recovery. Once contamination has been confirmed, and the immediate crisis has been addressed through response, remediation and recovery actions defined in the consequence management plan are performed to restore the system to normal operations. The architecture presented in Figure 1-2 was derived initially through the conceptual design of the contamination warning system and was refined based on lessons learned from design and implementation of the initial Water Security initiative pilot. The consequences associated with a particular contamination scenario are largely a function of the contaminant type and concentration, the location of contaminant introduction, and the relative timing of exposure, onset of symptoms, detection, and response. These results lead to a design basis for a contamination warning system that considered four primary attributes: contaminant coverage, spatial coverage, timing of detection, and reliability. Therefore, defining a design basis by the following performance objectives should guide a utility through the design and implementation of an effective contamination warning system. The system, as a whole, should be able to meet the following design objectives: Detection of a broad spectrum of contaminant classes. There are a large number of contaminants that could cause serious harm if introduced into the drinking water distribution system. As part of the contamination warning system design basis, contaminants were prioritized and then binned into 12 detection classes (see additional information below). Use of the detection classes to inform design provides more robust detection capability than analyzing for only a select number of contaminants and also avoids the challenge associated with designing a system around a list containing hundreds of potential contaminants. Achieve spatial coverage of the entire distribution system. Spatial coverage can be considered hydraulically and geographically. For components such as online water quality monitoring, spatial coverage is a function of the number of online water quality monitors and their placement throughout the distribution system. For other components such as consumer complaints or public health surveillance, spatial coverage varies geographically based on population density, population demographics (industrial vs. residential), and/or types of surveillance systems and tools used within a jurisdiction. Detect contamination in sufficient time for effective response. There are three periods associated with the evaluation of the timeline of a contamination incident, including (1) the time during which consequences (exposures, illnesses, fatalities, pipe contamination, etc.) are experienced in the population, (2) the time of initial detection, and (3) the time of response actions. A key aspect of a contamination warning system is to provide initial detection in a timeframe that allows for the implementation of response actions that result in a significant reduction in consequences. Reliably indicate a contamination incident with a minimum number of false-positives. Reliability can be considered from two perspectives. The first is operation, that is, factors such as contamination warning system component capabilities and necessary maintenance. The second is performance, defined as the ability of the system to provide information that leads decision makers to successfully infer that contamination has or has not occurred. Provide a sustainable architecture to monitor distribution system water quality. The integration of multiple monitoring and surveillance strategies already in use at the utility and public health department should improve acceptance of the system, and thus long-term sustainability. A contamination warning system should be designed as a dual-use application to May 2007 ------- Planning for WS-CWS Deployment benefit the utility in day-to-day operations while also providing the capability to detect intentional or accidental contamination incidents. The first design objective described above introduced the concept of contaminant classes. Table 1-1 presents a summary of contaminant detection classes developed during the conceptual design phase of EPA's Water Security initiative (USEPA, 2005a, USEPA, 2005b). This table shows the potential means of detection for each contaminant class by three of the components: water quality monitoring, consumer complaint surveillance, and public health surveillance (considered as two independent data streams). Enhanced security monitoring could potentially detect any contamination event, while sampling and analysis can detect contaminants within the group of target analytes for the methods employed. Thus, collectively, the five components provide comprehensive contaminant coverage, and provide a means of confirming a possible contamination incident through an independent data stream. Table 1-1. Contaminant Detection Classes and Potential Means of Detection Class 1 2 3 4 5 6 7 8 9 10 11 12 Description Petroleum products Pesticides (with odor or taste) Inorganic compounds Metals Pesticides (odorless) Chemical warfare agents Radionuclides Bacterial toxins Plant toxins Pathogens causing diseases with unique symptoms Pathogens causing diseases with common symptoms Persistent chlorinated organic compounds Water Quality X X X X X X X X X X X X Consumer Complaints X X X X X 911 calls /EMS X X X X X X Syndromic Surveillance1 X X X X Collecting and analyzing nontraditional data to detect a change or trend in the health of a population using categories of disease rather than formal diagnosis. In designing a contamination warning system, each of the design objectives should be considered for the system as a whole as well as for each component. Table 1-2 summarizes the design objectives as they relate to each of the contamination warning system components. These design objectives are presented again in Sections 4 through 8 along with a summary of design and implementation considerations for each of the monitoring and surveillance components. May 2007 ------- Planning for WS-CWS Deployment Table 1-2. Design Basis Summary by Contamination Warning System Component WS-CWS Component Online Water Quality Monitoring Sampling and Analysis Enhanced Security Monitoring Consumer Complaint Surveillance Public Health Surveillance Capability Can indicate the presence of a contaminant that significantly affects one or more monitored parameters. Can positively identify the presence of any contaminant in the suite of target analytes and above a well-defined minimum reporting level. Can detect an intrusion that may have provided the opportunity for introduction of any contaminant. Can indicate the presence of a contaminant that significantly affects one or more aesthetic qualities of water. Can detect the presence of a symptom or illness in a population which may be the result of the presence of a disease causing agent. May be able to identify the contaminant through clinical diagnosis/ testing. Contaminant Coverage High detection potential for classes 1, 2, 3, 5, 8, 9, 10, 11 and 12. Moderate detection potential for classes 4, 6, and 7. High detection potential for classes 1, 2, 3, 4, 7, and 12; Moderate detection potential for classes 5, 6, 8, 9, 10, 11. Covers all contaminant classes. High detection potential for classes 1 and 2. Moderate detection potential for classes 3, 4, and 5. Covers contaminant classes 2 through 11; detection potential varies with type of surveillance. Spatial Coverage Function of location, number, and density of monitoring stations. Function of location, number, and density of sampling stations, as well as sample type (composite vs. grab). Limited to those elements of infrastructure for which physical security can be monitored. Entire service area for contaminants with detectable taste, color, or odor characteristics. Comprehensive coverage of a particular city or county, which may include all, or a large portion of, the utility service area. Timeliness Function of hydraulic travel time from the point of contaminant introduction to the sensor, and the concentration of the contaminant. Function of sampling & analysis frequency and the total time to process the sample and analyze the results. Function of the type of security monitoring system and the time to evaluate a security breach. Function of the time from exposures to consumer reporting, complaint categorization, assessment and investigation. Function of the time from the initial exposures, the onset of symptoms, and the point at which public health officials recognize the incident as a potential water- borne illness. Reliability Rate of false positive / negative results in this application is largely unknown at this time. May be addressed through event detection systems and consequence management. Function of the reliability of sampling and analysis methods (high for established techniques). Baseline needed for reliable interpretation of results. Can be a reliable means of identifying an intrusion, especially when these breaches may involve contamination, such as in storage tanks and reservoirs. A potentially reliable indicator for contaminants with detectable characteristics if a robust complaint reporting and tracking system is in place. May be a reliable means of identifying the incidence of illness in a population, but timing of communication between drinking water and public health officials should be optimized such that appropriate response, actions could be implemented in time to reduce consequences. Sustainability Provides utility with a better understanding of water quality variability throughout distribution system and provides an opportunity to optimize distribution system operation. Provides utility with an opportunity to exercise sampling and laboratory protocols and may; provide information about previously unknown contaminants that occur in the system. Provides utility with increased physical infrastructure protection and awareness. Reduces the occurrence of nuisance tampering. Provides utility an opportunity to manage consumer information more effectively and can serve as a tool for enhanced consumer confidence. Provides an opportunity for utility and local health May 2007 ------- Planning for WS-CWS Deployment 1.2 Document Overview This document describes planning considerations for contamination warning system deployment based on the approach deployed at, and lessons learned from, the initial Water Security initiative pilot at GCWW. The primary focus of the document is on planning and pre-design. Throughout the document available resources and tools are highlighted to facilitate contamination warning system design and implementation. The lists of resources are not intended to be exhaustive, but rather a starting point to facilitate the planning process. Sections included in the document are as follows: Section 2: Project Planning and Management. This section introduces the concept of system engineering for contamination warning systems and discusses critical aspects of project planning for design and implementation. Although the drinking water utility is the primary organization involved with design and implementation, other key partners have a significant role as well and should be included in the process. Subsections focus on system engineering, utility staffing, coordination with local partners, and costing considerations. Section 3: Design and Implementation Framework. This section describes the framework for design and implementation of a contamination warning system including planning or pre-design, design, implementation, preliminary testing (start-up and baselining of components), operation and maintenance, and evaluation and refinement. Section 4: Online Water Quality Monitoring. This section provides the design basis for online water quality monitoring, identifies critical design decisions and describes design and implementation aspects relative to monitoring network design (placement of monitoring stations), monitoring station design, communications and information technology architecture, and water quality event detection. Section 5: Sampling and Analysis. This section describes the design basis for sampling and analysis, identifies critical design decisions and describes design and implementation aspects relative to laboratory capability and capacity, sampling and analysis (baseline, maintenance, triggered), and. field screening and site characterization. Section 6: Enhanced Security Monitoring. This section describes the design basis for enhanced security monitoring and a systematic methodology for selecting sites for security improvements and designing those enhancements. Section 7: Consumer Complaint Surveillance. This section describes an approach to managing customer calls and customer information. The document focuses on assessing existing call management systems and protocols based on contamination warning system design objectives and describes considerations for optimized call management, tracking, and analysis. Section 8: Public Health Surveillance. This section describes the design basis for public health surveillance and design considerations to assist with planning for implementation. The public health surveillance component of a contamination warning system relies heavily on relationships and partnerships with local health departments. In addition to describing the design basis for the public health surveillance component, this section provides background information for drinking water utilities on the types of public health surveillance that may be implemented by local or state health departments. Considerations for enhancing or expanding on these existing systems or approaches to include more timely data streams such as 911 calls or emergency medical service (EMS) events for fast-acting contaminants are also addressed. Section 9: Consequence Management. This section describes planning considerations for development of a consequence management plan for the drinking water utility, including identification and coordination with local response partners and organizations. Types of plans and information that can be leveraged to support development of this plan - both internal and external to the utility - are discussed. In addition, existing training programs and resources are identified. In addition to the sections described above, this document also includes a list of acronyms, references, a glossary (Appendix A), and information security considerations (Appendix B). May 2007 7 ------- Planning for WS-CWS Deployment Section 2.0: Project Planning and Management This section discusses critical aspects of project planning and management for deployment of a contamination warning system. Although the drinking water utility is the primary organization involved with deployment, other key partners also have significant roles and should be included in the process. As discussed throughout this section, deployment of a contamination warning system is a significant undertaking that impacts most departments or divisions within the utility at some stage. A commitment at all levels of the organization, from senior managers and supervisors to staff supporting routine operations within the utility or in the field, is essential. 2.1 Application of System Engineering Principles A contamination warning system is, by design, a systematic approach to monitoring and surveillance for the timely detection of drinking water contamination. As such, deployment of a contamination warning system relies on system engineering principles to support coordination of technical and management activities. System engineering is an interdisciplinary approach to design and implementation of systems. Through system engineering, disciplines and specialty groups are integrated in a team effort forming a structured development process that proceeds from design to implementation to operation. From the beginning of the project, system engineering principles are critical to successful planning and implementation. The primary application of system engineering for a contamination warning system is to ensure that the system - monitoring and surveillance components and consequence management as discussed in detail in Section 4 to Section 9 - functions as an integrated whole. System engineering involves the integration of monitoring and surveillance components and consequence management. Throughout planning and implementation, many activities can and should occur in parallel. However, it is necessary to reconcile activities and key stages of deployment to optimize the function of the system. For example, once a preliminary concept of operations (as discussed in Section 2.1.2) has been defined for the system, component-specific designs and consequence management planning can occur in parallel. However, it is important to reconcile the consequence management plan with the various monitoring and surveillance components as they are designed and implemented to ensure seamless transition from routine operations to consequence management. As part of system engineering, coordination strategies should be applied throughout the deployment process to ensure that there is a consistent vision and understanding of goals and objectives. To facilitate coordination, meetings involving representatives from each of the components should be held early in the planning process. It may be helpful to establish a core team with representatives for each of the component teams or supporting divisions involved in design, implementation, or operation of the contamination warning system. Regular meetings serve as a forum for component teams to share their progress with the project management team, and as an opportunity to identify synergies among the enhancements. Through discussions, critical cross-cutting issues may be identified and resolved consistently across the project, making for more efficient problem solving and refinement. In addition, these meetings could highlight dual-use applications of the system, reinforcing the value and improving sustainability. As utilities work through system deployment, it may be necessary to prioritize certain activities across components of the system based on criticality, resources, schedule, or other issues. Addressing these issues through these routine coordination meetings may help to facilitate the prioritization process. 2.1.1 Development and Management of Work Plan and Schedule The development of detailed work plans for monitoring and surveillance components as well as consequence management is critical. These work plans should outline the enhancements necessary to progress from the existing state to the desired state and should be useful in coordinating timely implementation activities. The work plan builds on the current state of the utility by identifying specific May 2007 8 ------- Planning for WS-CWS Deployment equipment, computer hardware or software, training and process modifications that are necessary to achieve specified goals. Each step in the work plan should contain detailed action items and a defined schedule to meet the goals specified by the design decisions. Coordination across components to maximize efficiency is critical in work plan development to ensure that all necessary activities are planned for, while eliminating redundancies. Although detailed schedules and work plans should be developed to support component activities, it is important to track high-level milestones across all components to ensure that activities are managed efficiently and effectively and that there is communication and coordination on overarching or cross- cutting issues. To facilitate this high-level planning, development of an integrated concept of operations can serve as a project management tool as well as a definition of system operations. 2.1.2 Integrated Concept of Operations The concept of operations is the description of the routine operation and initial trigger validation for each component of the contamination warning system. While the terminology and approach for this documentation may vary, it is important that the operational concepts be clearly documented and that utility staff and local partners understand their roles and responsibilities in operating the system. The concept of operations should be developed at the component level to present the day-to-day functioning of components and management of information to support trigger validation and initial response actions (or, in the case of consequence management, integrating information from multiple components and initiating a response). However, the development of the component concept of operations should be coordinated to ensure that information from multiple components can be integrated to better inform response decisions. It is also important to integrate day-to-day operation of the contamination warning system into routine job duties. Guidance on development of a concept of operations will be the focus of future EPA efforts. While the concept of operations for a contamination warning system is intended to guide day-to-day operations and trigger validation, it also has an important design function at the system level. A preliminary concept of operation for each component can be an effective means of determining utility- derived requirements for the design and implementation of that component. An integrated concept of operations encompasses the component-specific concept of operations by explaining how the components inter-relate, how their data streams are combined and how the system as a whole meets the design basis. It is recommended that a preliminary concept of operations be developed during the planning or pre- design stage. To develop a preliminary concept of operations, utilities should define design objectives as discussed in Section 3 and leverage the component-specific concepts presented in Section 4 - Section 9. At the conclusion of planning and pre-design, utilities should be able to develop a high-level concept of operations that includes critical information point such as the users and key decision makers involved in routine operation of a given component. As part of the system engineering approach to implementation, it may be possible to develop and define teams and roles and responsibilities for utility staff across all components, thereby presenting a cohesive, system-wide picture of what key utility personnel should do during the day-to-day operation of the contamination warning system. An important feature of this integrated view is its ability to illustrate how contamination warning system activities relate to normal job functions and provide benefits to the utility and staff beyond contamination warning. In addition, a preliminary timeline for initial trigger validation should be part of the preliminary concept of operations, and is an important consideration in early stages of consequence management plan development. With the initial draft of the concept of operations, these timelines should be projections based on desired goals for trigger validation and decision making. It will likely be necessary to refine these projections based on the design that is ultimately implemented. These timelines may also serve as a metric for evaluation of the system and individual components as part of evaluation and refinement. Development of a May 2007 ------- Planning for WS-CWS Deployment preliminary concept of operations will also serve to highlight the significance of information technology (IT) systems, data flows, and IT-based user requirements in contamination warning system deployment. 2.1.3 IT System Engineering A contamination warning system is not merely a collection of monitors and equipment placed throughout a water system to alert of intrusion or contamination. Fundamentally, it is an exercise in information acquisition and management. Different information streams should be captured, managed, analyzed, and interpreted in time to recognize potential contamination incidents and respond effectively. Information from several different databases and information systems at a water utility should be integrated to enable event detection, trigger validation, credibility determination, and management of response actions in a timely manner. In addition, data and information from local partners including, but not limited to, fire departments, public health, and law enforcement should be integrated. Thus, the success of the contamination warning system implemented at a drinking water utility will depend heavily on effective data and information management. In planning for contamination warning system implementation and development of a preliminary concept of operations, development of an inventory of the IT systems used in the operation of the components and day-to-day operations may serve as a valuable tool. It is also useful to consider information flows and how information is collected and managed throughout existing systems. An information flow diagram describes how data and information flows between system elements and users during routine operations of a contamination warning system component, including the information systems, databases, and user interfaces used during activities related to routine monitoring and surveillance, event detection, trigger validation, and credibility determination for a component. Development of a preliminary information flow diagram could serve as a useful tool for facilitating design and implementation discussions and decisions. Although each component data stream is unique, a common set of system elements (data sources, event detection, alarm or trigger notification, and data storage) can be defined. An assessment of the utility's existing capabilities relative to the design objectives of the contamination warning system and the attributes of the specific component of the system is essential for successful implementation. From an IT system engineering perspective, primary objectives for design and implementation include the following: Ability to leverage and integrate with existing utility IT systems and policies Mechanisms for timely integration and analysis of contamination warning system data and the ability to make data available to support timely decision-making Approaches and mechanisms that can be readily adapted to meet changing needs and priorities The basic objectives described above can be used to define the component-specific requirements for data management. Other considerations include the following: Electronic data management for all components. All information collected as part of the contamination warning system should be managed in a database. For each component, information should be tracked as the system transitions from routine operation to event detection, credibility determination, consequence management, and finally return to normal operations. In order to avoid duplication of effort, information should be accessible across multiple databases and systems. Data storage. The data collected should be stored in a reliable data store that has sufficient capacity and performance to support routine operations, contamination situations, and system evaluation. Historical data should be maintained at appropriate locations to enable engineers and scientists to study past normal and event situations to evaluate and optimize system performance. Automated and integrated data analysis. As data are captured in electronic format in a consistent environment, algorithms operate on the data to integrate information and facilitate data analysis. Automated anomaly detection algorithms indicate when the data may be indicative of a contamination incident, signaling the need for human involvement in the assessment process. May 2007 10 ------- Planning for WS-CWS Deployment The type and level of sophistication of algorithms will likely vary by component, from sophisticated algorithms for the analysis of online water quality and public health surveillance data to more simplistic algorithms for consumer complaints. The use of tools for spatial and temporal analysis of the indicators, such as Geographic Information Systems (GIS), should also be considered. User interface for operation and analysis. Although some contamination warning system operations are automated, human operators and analysts will assess situations and make many of the decisions necessary to manage a potential contamination incident. Thus, there is a need to develop or utilize existing user interfaces to support data analysis and decision making. If a utility's existing GIS is sufficiently robust and integrated, it can serve as an effective means for display of information from any or all components of the contamination warning system. Data exchange capability for transmission of information within the utility and between the utility and contamination warning system partners. The data collected at the utility may need to be shared with other local government agencies such as public health departments. Availability and redundancy of systems. Information systems should be designed to have system availability that is consistent with the significant importance of the contamination warning system function. Backup and recovery plans and procedures should be defined and implemented. Security, authorizations, and controls. Access to contamination warning system data and applications should be restricted to authorized personnel as determined by the utility. In addition, data encryption should be considered for information exchanged between the utility and other partners when it passes over public networks. Change management and maintenance. Throughout the life-cycle of IT software and hardware it will be necessary to implement upgrades and perform routine maintenance. From a planning perspective this should be considered both in terms of costs and the approach for implementing upgrades and performing maintenance. 2.2 Utility Staffing Approaches for staffing a contamination warning system will vary by utility, but should work within existing organizational structures and routine job functions to the extent possible. As discussed in Section 2.1, application of system engineering principles plays a critical role in successful implementation of a contamination warning system. This applies to staffing as well - planning, design, implementation, operation, and ultimately evaluation will rely on effective communication and coordination across divisions or departments within the utility. To the extent possible, generic titles and roles are provided to organize the discussion in this section. Based on the description of activities, utilities should identify individuals and departments within their own organization to fulfill these roles as appropriate, independent of job title. 2.2.1 Project Management Team While routine operation and maintenance of the contamination warning system should generally fall within the routine job functions of utility staff, design and implementation will involve significant time and effort from dedicated managers within the utility. Depending on utility organization and operational paradigm, the activities described below may be managed by one individual, or more likely, a core project management team. In addition, support from the utility director and/or board of directors, as well as all senior supervisors and managers is critical to contamination warning system implementation. Project management activities to be addressed by the management team generally include the following: Development of design goals and objectives Communication of the goals and objectives of the contamination warning system to all levels of management, staff, and external partners Prioritization of work activities and allocation of resources. Networking and establishment of agreements with external partners May 2007 11 ------- Planning for WS-CWS Deployment Coordination of procurement, installation, and inspection of equipment, hardware, and software, including licenses and maintenance agreements Coordination of teams supporting design and implementation of monitoring and surveillance components and consequence management plan Verification of reporting protocols and coordination with primacy agency Development of an overarching framework describing the project phases, goals, objectives, schedule, and milestones Coordination of IT and data management aspects of design, implementation, and operation across all aspects of the contamination warning system to ensure consistency with existing protocols and procedures and interoperability of systems Integration of contamination warning system concept of operations with existing plans and protocols that govern routine operations and activities within the utility Integration of contamination warning system consequence management plan with existing response protocols, plans, and procedures Development of evaluation plan and identification of metrics for evaluation Coordinate identification and documentation of lessons learned 2.2.2 Utility Staff While the primary focus of a contamination warning system is monitoring water quality in the distribution system, the design and implementation of a contamination warning system involves the collaboration of the utility as a whole; however, each department has its own capabilities to contribute. The information presented below is organized by general divisions or departments; actual departments and roles may vary by utility. Component-specific activities are described in "Staffing and Costing Considerations" subsections within Section 4 - Section 9. Security/Risk Evaluate physical vulnerabilities of facilities Design security enhancements to facilities Provide oversight during installation of security enhancements Coordinate planning for and response to security breaches with law enforcement Manage and coordinate response to security incidents utility-wide Coordinate training and functional drills related to consequence management Determine if security breaches provided an opportunity to contaminate water Water Quality Evaluate vulnerabilities of distribution system Select water quality parameters and sensor hardware Design/create monitoring network (placement of monitors around system) Design and oversee the conduct of tracer studies if needed Coordinate installation of monitoring stations Select target analytes and lab methods for baseline and triggered sampling Identify sampling locations and frequency for baseline sampling program Develop and implement sampling protocols Identify requirements for integration of data into event detection system (EDS) for water quality monitoring and consumer complaint surveillance component Coordinate laboratory capabilities with local laboratory network Maintain proficiency for any in-house lab and field analyses Establish water quality baselines for comparison with abnormal water quality results Develop site characterization procedures, including selection of field test equipment Coordinate with public health on surveillance activities Coordinate training on operation of all contamination monitoring system components May 2007 12 ------- Planning for WS-CWS Deployment Investigate water quality and distribution issues in reference to observed triggers Evaluate remediation options Operations and/or Distribution Coordinate with Supervisory Control and Data Acquisition (SCADA) or other communication systems Support implementation of distribution system tracer studies, if conducted Investigate distribution system operations and maintenance in reference to observed triggers Support field sampling activities Plan and implement isolation and containment options Support evaluation and implementation of remediation options Information Technology Identify data management, hardware, and software needs for utility Design and implement changes Coordinate the flow of all of the information/data streams Engineering and Planning Design and construct water quality monitors Design and construct security improvements to facilities Provide input for monitoring network design Perform hydraulic and water quality modeling Design remediation activities in event of confirmed contamination Public/Customer Affairs Identify improvements to call center to recognize and track water quality related calls Implement hardware, software, and training changes Interface with outside organizations for public health surveillance Identify appropriate customer call information streams and how to integrate into Event Detection System Provide public outreach Establish baseline levels for water-quality related customer complaints Public notification of contamination and appropriate actions Administration Financial tracking Project tracking Procurement Legal review of agreements, documents, etc. Management of contracts, grants, agreements, etc. 2.3 Local Partners Designing, implementing, and ultimately operating a contamination warning system is a complex task that relies on the coordination and cooperation of many partners. In addition, the impacts of a drinking water contamination event are not isolated within a utility, so local partners should be engaged in the complex task of responding to a contamination event, intentional or accidental, that can involve criminal activity, public health impacts, regulatory compliance, and hazardous materials response. Because it is necessary for the utility to rely on the assistance of local partners to operate the system, those partners should be involved as appropriate in the design and implementation of the system. As part of this process, the utility should consider what formal agreements can be reached with other agencies, which can vary depending on whether the utility is publicly or privately owned. Furthermore, if publicly owned, May 2007 13 ------- Planning for WS-CWS Deployment nuances of being an independent public agency, a single municipal department, or part of a public works department should be considered. 2.3.1 Identifying and Engaging Partners The utility is the operational hub of the system as the primary operator of the majority of monitoring and surveillance components of the contamination warning system, with the exception of public health surveillance. However, other partners may be involved in trigger validation and/or consequence management activities. Figure 2-1 provides an overview of potential partners in contamination warning system deployment. Federal Bureau of Investigation State Emergency Management and Homeland Security Agencies State Law Enforcement Stat Wat Centers For Disease EPA Regional Offices Control and Prevention Local Health Local Wastewater Local Law Department Utility Enforcement Local Fire, EMS, / Water \ Local Civil and HazMat f Utilitv Government .... ... Public Health and Local Emergency Host ,. . . . . . _ ... _ ..... Environmental Planning Committees Facilities , . . . Laboratories e Drinking and Waste Neighboring Utilities er Primacy Agencies EPA Criminal Investigation Division EPA National Response Center State Emergency Res ponders State Government Media Figure 2-1. Potential Contamination Warning System Partners During the early stages of the investigation of and response to a "possible" contamination incident, a utility will likely rely on local partners for assistance, and as the credibility that a contamination incident has occurred increases, the number of partners will increase. As illustrated in Figure 2-1, the number and scope of partners that can become involved in responding to a contamination event can be significant. In planning for contamination warning system deployment, drinking water utilities should identify and engage local partners early in the process, particularly those partners such as local health departments and public health and environmental laboratories that will have a significant role in routine operations. Specific responsibilities of partners and when they are engaged will vary by utility and jurisdiction. However, Table 2-1 provides a summary of possible contamination warning system partners and their possible role in design, implementation, operation, and/or response. Table 2-1. Summary of Potential Contamination Warning System Partners Partner Organization Host facilities Local health department Roles and Responsibilities Provide facilities for placement of water quality and/or enhanced security monitoring stations, including 24/7 access to utility staff for maintenance, trigger validation, and response activities. Monitor health of the population through public health surveillance as part of routine operation of the system and may have some degree of analytical capability to support sampling and analysis. Provide support during consequence management including consultation and public notification. Serve as conduit to state and Federal health departments and agencies. May 2007 14 ------- Planning for WS-CWS Deployment Partner Organization Local law enforcement Local civil government Local emergency planning committees and emergency management agencies Local fire, EMS, and Hazmat Environmental and public health laboratories Local wastewater utility Neighboring utilities Media State government State emergency responders State drinking water and wastewater primary agencies State emergency management and homeland security agencies State law enforcement EPA Regional offices and/or laboratories Federal Bureau of Investigation (FBI) Centers for Disease Control and Prevention (CDC) Roles and Responsibilities May assist in routine operation of enhanced security monitoring; provide support during consequence management through credibility determination and response. May also serve as conduit to state and national law enforcement and intelligence agencies. Should be engaged early in the planning for implementation. In the event a utility's service and/or wholesale areas span multiple jurisdictional entities, it may be necessary to engage in formal agreements among different local governments and their respective agencies to ensure cooperation and support and allocation of funding. Also, should an event occur, the elected officials of different jurisdictions should be appropriately informed of the state of the situation so that they can effectively communicate with their constituencies. Primarily support consequence management activities as a conduit to other response agencies at the state and Federal level. Can support provision of alternate water supplies, coordination, disaster declaration, and transition to National Response Plan implementation. Local fire and EMS organizations may have a role in routine operations for public health surveillance as a provider of 91 1 and/or EMS data. These organizations, as well as local Hazmat play a critical role in consequence management including site characterization activities to support credibility determination. Provide support to routine sampling and analysis activities to establish baseline and maintain analytical proficiency. In addition, provide analytical support during consequence management to assist in credibility determination as well as response and remediation efforts. State public health laboratories provide access to CDC's Laboratory Response Network. May provide analytical support for routine sampling and analysis. Should be consulted in the development and implementation of consequence management plans due to the potential impact of contamination on wastewater operations. May provide support in the event of a contamination incident through mutual aide, assisting with provision of alternate water supplies, remediation, and recovery activities. Local media organizations may serve as a valuable resource in communicating messages to the public in the event a contamination incident occurs. May provide support to implementation activities in terms of gaining cooperation from State organizations. May have a role in establishing formal agreements with State partners or coordinating funding resources. For consequence management, should be informed and engaged once contamination has been confirmed to assist in coordination of resources and communication. Provide support to consequence management phases if a contamination incident is confirmed. Should be engaged in consequence management planning to ensure efficient transition in the event a contamination incident escalates. Primacy agencies can be public health agencies as well as separate state or local environmental agencies, like state or regional water quality boards. If contamination does occur, there may be regulatory ramifications related to use of contaminated water, public notification, environmental concerns for discharged water, quality of alternative supplies, and more. Additionally, the primacy agency, along with EPA, should be consulted on any potential remediation and recovery plan. Provide support to consequence management phases if a contamination incident is confirmed. Should be engaged in consequence management planning to ensure efficient transition in the event a contamination incident escalates. Provide support to consequence management phases if a contamination incident is confirmed. Should be engaged in consequence management planning to ensure efficient transition in the event a contamination incident escalates. May assist in coordination of Federal resources and may also assist by providing analytical surge capacity during phases of consequence management. May assist in site characterization and/or consequence management plan development. Establishing a relationship with local FBI agents early in the implementation process is critical to establish and understand roles and responsibilities in the event contamination occurs. Provide oversight to the Laboratory Response Network, a network of public health laboratories with the ability to analyze for select agents based on established analytical protocols. Ensure member laboratories have appropriate training, equipment, reagents, and resources. Provide technical consultation during credibility determination and other phases of consequence management. May 2007 15 ------- Planning for WS-CWS Deployment Partner Organization EPA Criminal Investigation Division (CID) EPA National Response Center Roles and Responsibilities Provide support to consequence management phases if a contamination incident is confirmed. Should be engaged in consequence management planning to ensure efficient transition in the event a contamination incident escalates. Provide support to consequence management phases if a contamination incident is confirmed. Should be engaged in consequence management planning to ensure efficient transition in the event a contamination incident escalates. A recommended strategy for engaging partners is to first consider those partners who will have a role in routine operations of the system or will be involved as "first responders" based on the consequence management plan. Engaging the numerous partners involved in establishing a contamination warning system is a daunting challenge on its own, without the myriad other tasks the utility implementation team will be occupied with. It is not uncommon for the service area of a utility to span city limits and county borders, and into the jurisdictions of numerous police, fire, and public health agencies, not to mention the umbrella jurisdictions of hierarchical agencies like county and state emergency management, public health, and homeland security agencies, to name just a few. Therefore, it is essential to take full advantage of the existing groups and organizations in which these partners may already participate. Figure 2-2 illustrates a recommended approach for engaging partners. Engagement to form CWS Implementation Team Direct engagement with key county, state, and federal partners Regional Homeland Security Programs & Existing Local Partner Groups Engagement through existing partner groups Direct engagement with extended partners Figure 2-2. Recommended Strategy for Engaging Contamination Warning System Partners Depending on the role or type of support contamination warning system partners play in implementation, it may be necessary or desirable to establish formal agreements. Considerations for establishing these agreements are discussed in Section 2.3.2. 2.3.2 Considerations for Formal Agreements with Local Partners Inter-agency agreements, memorandums of understanding, memorandums of agreement, mutual aid agreements, and other agreements are becoming common practice in most jurisdictions. The documents contain language that is mutually agreed upon by all stakeholder agencies and generally define collaborative efforts that involve action items, equipment resources, or regional governance. When engaging local, county, state and federal partners in implementation activities, the utility should address the subject of inter-agency agreements early in the process. Addressing formal agreements early in the implementation process is extremely important, as they commit partner agencies to specific roles and actions. Without them, implementation can be stalled by inter-agency disagreements or misunderstandings, or an agency may be left responsible for costs they believed would be covered by another. May 2007 16 ------- Planning for WS-CWS Deployment The utility should first identify its own protocols for establishing formal agreements with external agencies, organizations, and partners. This includes identifying who holds the authority to enter the utility into these types agreements (who signs the document), any procedural details, like minimum or maximum review periods, paperwork routing procedures, restrictions on the types of agencies or groups the utility may enter into agreements with (public and private), or limits of commitment (monetary or other). The utility should also develop a clear understanding of the same types of information from the agencies it intends to engage. Subjects of the agreements extend beyond simply who pays for equipment; commitments should be made to provide man-power both for the implementation and operation of the contamination warning system; allocation of resources; etc. If funding is from an external source all applicable standards and regulations for establishing formal agreements should be followed. 2.4 Costs Cost of contamination warning system implementation can vary widely from utility to utility, based on a variety of factors, and perhaps most significantly based on existing capabilities across components. Factors to consider in estimating costs for specific components are embedded throughout Section 4 - Section 9. From a project planning perspective, considerations should be given to long-term issues including operation and maintenance and sustainability prior to finalizing a design. Contamination warning system deployment should be considered a significant program or initiative that will involve long-term involvement and large investments by the utility. Substantial equipment and construction may be necessary, as well as additional maintenance needs. These maintenance needs should be tracked on a system-wide basis and the labor and equipment expenditures should be included in budgets for the life of the system. All costs should be monitored and compared to budgetary constraints. Additionally, this project may involve sensitive information which should be tracked, and personnel who use or handle the information will have to be screened. This places an additional responsibility on the administrative personnel to adequately document and supervise information security issues. Considerations for information security are discussed in Appendix B. The potential cost of contamination warning system deployment should include both the tangible (equipment, installation, etc.) as well as the intangible (staff perception, morale, motivation, etc.) elements. The tangible costs should be estimated using a life-cycle approach to capture the management and coordination effort, the capital costs of the equipment, initial training, startup, testing, and calibration, and the long term operations and maintenance costs (training, calibration, spare equipment and components, maintenance contracts, chemical testing supplies, monthly communications fees, etc.). The intangible costs should be identified and recognized and a management plan developed to address any issues. Examples of intangible costs could include a staff person's perception that the operation of the system adds more work to his/her day with no additional compensation or the fear that the system will be advertised to the public as enhancing safety and security while not actually delivering on the promise. In addition to component-specific cost factors discussed throughout the remaining sections of this document, the following factors should be considered as part of project management and system engineering costs: Project Management and Coordination. Development of agreements, schedule, work plan, communication products; administration and financial tracking; routine coordination and strategy meetings. Development of Integrated Concept of Operations. Coordination of concept of operations development; analysis of component-specific concept of operations for consistency; development of integrated concept of operations documentation and procedures. IT System Engineering. Assessment of existing systems; procurement and installation of hardware and software; implementation, operation, and maintenance of systems. Evaluation and Refinement. Development and implementation of evaluation plan; identification and documentation of lessons learned; identification of refinements to optimize system performance. May 2007 17 ------- Planning for WS-CWS Deployment Section 3.0: Design and Implementation Framework Deployment of a contamination warning system should follow the typical programmatic approach in which proposed enhancements are planned, designed, implemented, tested, maintained and refined. This section provides a comprehensive framework for design and implementation. Alternate approaches for design and implementation can be considered, however the concepts described below should be addressed. Table 3-1 summarizes a recommended approach for contamination warning system design and implementation based on lessons learned from the initial pilot. While there may be some deviations in terms of how each stage is applied for a given component, all components should address planning and pre-design, design, implementation, preliminary testing, operation and maintenance, and evaluation and refinement. Table 3-1. Overview of Design and Implementation Framework Stage of Approach Planning and pre- design Design Implementation Preliminary testing Operation and maintenance Evaluation and refinement Description Developing a core implementation team, defining design objectives to guide implementation, and a preliminary assessment of existing capabilities relative to design objectives. Development of a preliminary concept of operations and development of a detailed work plan and schedule to guide implementation. Implementation of enhancements, installation of equipment, and training according to the plan. Operation of the contamination warning system for the purpose of collecting data necessary to understand system performance and finalization of the concept of operations to optimize system. Operation of the contamination warning system for the purpose of monitoring for contamination incidents and other water quality issues. Analysis of data and information generated during full operation to refine and optimize the system. Additional detail on the application of this framework to contamination warning system monitoring and surveillance components and consequence management is presented in Section 4 - Section 9. 3.1 Planning and Pre-design The initial steps in developing a contamination warning system are included in the planning and pre- design stage. The utility should develop a team to support design and implementation and define what it wants the system or component to do. 3.1.1 Building the Team As emphasized in earlier sections of the document, deployment of a contamination warning system involves an integrated team within the utility that also extends to external partners for certain aspects of monitoring, surveillance, and response. It is important that the implementation team apply the system engineering principles discussed in Section 2.1 throughout all phases of contamination warning system implementation. Utility departments and divisions should work together as an integrated team to leverage existing infrastructure, systems, protocols, and procedures to ensure effective operation of the system. 3.1.2 Defining the Utility-Specific Design Basis and Design Objectives As introduced in Section 1.1, the contamination warning system should be able to meet the following design objectives: Detection of a broad spectrum of contaminant classes Spatial coverage of the entire distribution system Detection of contamination in sufficient time for effective response May 2007 18 ------- Planning for WS-CWS Deployment Reliable indication of a contamination incident with a minimum number of false-positives A sustainable architecture to monitor distribution system water quality These concepts should be considered during each step of the pre-design process, for the components and the system as a whole. Failing to consider each design objective may result in the design of contamination warning system that accomplishes some of its goals, but as a collective system fails to meet its ultimate objective. Table 3-2 summarizes overarching design basis considerations for the contamination warning system. Table 3-2. Design Basis Considerations Design Objectives Description Design and Implementation Considerations Capability Ability to detect contamination of the distribution system through contamination warning system components. Most components provide an indirect measure of contamination; thus it is necessary to have a process for validation of triggers and coordination across components. Contaminant Coverage Contaminant classes that can be detected by the system; actual contaminants may vary from system to system depending on the manner in which components are implemented. May be influenced by several factors for each component such as type of public health surveillance or disinfectant residual. Design objectives should target coverage of as many contaminant classes as possible. Expanding to additional contaminant classes could be an objective to consider as part of system evaluation. Spatial Coverage Amount of distribution system covered by one or more of the contamination warning system components. May vary by component pending jurisdictions, number of sensors, sampling routes, security sites, etc. While the integrated contamination warning system should cover the entire distribution system, the degree of coverage by each component will vary. An aspect of system engineering should be to maximize coverage to the extent possible across all components in a way that optimizes protection through the entire system. Timeliness Function of ability to detect anomalies and conduct initial trigger validation to determine possible contamination. Development of procedures for routine operation of the system and initial trigger validation will inform design and operation of the system and ultimately impact timeliness. Reliability At the system level, reliability is characterized in terms of the rate of false alarms and occurrence of undetected contamination. Reliability of the system is influenced by design decisions made at the component level. However, system reliability can be improved by integration of information and coordination across components to maximize confidence in a system alarm. Sustainability Ability to provide drinking water utility with an understanding of the distribution system in terms of water quality and variability and information to optimize distribution system operation. Contamination warning system activities and procedures should be designed for incorporation into routine job functions to maintain the system and support dual use application. Security should not be the only consideration in developing design objectives. In the planning and pre-design stages of contamination warning system deployment, drinking water utilities should evaluate these design objectives relative to their specific needs and objectives and customize and adapt as appropriate. It is important to consider objectives beyond the security aspects of contamination warning systems, particularly dual use applications that could help to promote Sustainability of the system by optimizing utility operations. Potential dual-use benefits of a contamination warning system could include the following: Detection of cross-connections and other distribution system water quality problems Improved relationship with public health organizations, including mutual sharing of information and alerts Enhanced knowledge of distribution system water quality leading to improved operations (e.g., more consistent disinfection residual levels, improved corrosion control, early warning of nitrification episodes, reduced disinfection byproduct levels, etc.) May 2007 19 ------- Planning for WS-CWS Deployment Identification of problem valves (closed, partially closed, inoperable) Improved coordination with local, state, and federal response organizations Reduced occurrence of tampering and vandalism Improved information technology systems and interoperability Improved consumer complaint tracking and response Improved laboratory capability and an established laboratory network Consequence management plans applicable to any water quality emergency 3.1.3 Preliminary Assessment and Gap Analysis Understanding the starting point and existing resources that will form the foundation for the system is critical in the early stages of planning and pre-design. Before progress towards the desired end-state of the contamination warning system can be made or measured, a utility should fully understand the capabilities of their existing systems, procedures, and other resources. The recommended approach for conducting a thorough analysis of the utility's existing capabilities is to use personnel with varying degrees of experience with the systems being evaluated. Experience at the pilot utility demonstrated that including staff who do not routinely use a particular system can provide additional insight into what is a routine process for an experienced user. This can be accomplished by using a mix of utility personnel and/or outside consultants. The intent is to gain a complete understanding of the current state of the utility, and the benefit of using a diverse team is that it results in a more realistic plan to reach the desired capabilities. At the conclusion of this self-assessment, the utility's design and implementation team will have developed a solid understanding of their existing systems. Using the preliminary concept of operations as a benchmark, the utility can examine its existing operations by process or by component and measure the gap between current and desired capabilities. The result is a gap analysis that clearly outlines the progress that a utility should make to achieve the design objectives. For example, the analysis of the customer call center at the pilot utility discovered that the staff was already collecting very useful complaint data, and an excellent complaint management system was already in place. Only minor modifications that had very minimal impact on existing staff and processes were necessary to transition the existing system into a one that would feed the critical water quality complaints into the system. This gap analysis may be further refined following development of a preliminary concept of operations as discussed in Section 3.2. 3.2 Design The design stage of the programmatic approach for a contamination warning system encompasses the development of the plans and specifications for each component and a consequence management plan. It is critical that the performance objectives of the design basis are considered throughout this process, and the design stage is no exception. At the system level, design should go beyond information integration and event detection. It should also consider how all resources - staff, IT, communications, equipment, etc. can be leveraged across the entire project. Further, any resources at the disposal of the utility and its partners, including but not limited to staff, communications, equipment, and training opportunities, should be evaluated for possible use within the project. This will not only result in an efficient use of time and materials, but will also highlight the dual-use benefits possible through implementation of the contamination warning system (e.g., radio communication equipment purchased for the use of equipment maintenance becomes more valuable when it is also used as part of the consequence management plan). 3.2.1 Conceptualize System Before beginning physical design, consider a contamination warning system at a conceptual level, and the manner in which this conceptual model translates to physical systems and implementation. Firm understanding of the basis for design decisions should inform all aspects of the design process. This May 2007 20 ------- Planning for WS-CWS Deployment understanding can be developed, in part, though the development of a preliminary concept of operations for the entire contamination warning system. At this early stage of planning, many of the details of routine operation are unknown. However, a preliminary concept of operations can still be developed at this stage to identify the general capabilities that the fully implemented system should possess and how that capability relates to the existing resources and capabilities within the organization. The preliminary concept of operations will also establish potential roles and responsibilities and user requirements for IT-related systems. In doing so, this document outlines the characteristics of the finished system and provides the framework for the design and implementation of the individual components. Its development is a critical step towards successful implementation of a contamination warning system because it provides an initial benchmark against which to measure a utility's existing capabilities and a means by which to plan for enhancements. Note that additional details will be added to the concept of operations as the contamination warning system is designed and implemented, including a very detailed concept of operations for each component. Thus development of the concept of operations should be viewed as an iterative part of the design and implementation process, with the preliminary concept of operations serving as a starting point for the design process. 3.2.2 Work Plan for Implementation Prior to initiating implementation activities for any component, a detailed implementation plan, or work plan, should be developed that clearly identifies priorities, schedule, milestones, and resources. It may be necessary to revisit the assessment and gap analysis in order to prioritize some of the identified enhancements to maximize time and resources. This should be considered at the system level, in advance of implementation using a system engineering approach as discussed in Section 2.1. 3.3 Implementation Implementation of a contamination warning system begins with consensus on the approach identified in the component-specific work plan and completion of any associated design work (i.e., hardened access points as part of the physical security enhancements). This stage can involve significant coordination with outside consultants and contractors, depending on the capabilities and availabilities of in-house staff. Again, since a contamination warning system relies heavily on effective data management, the involvement of the information technology staff is critical. Each component-specific work plan developed during the design stage should be implemented concurrently, to the extent possible. The timelines should be carefully monitored to ensure delays do not create problems in other components' deployments. In addition, any training specified in the component-specific work plan should be conducted during the implementation stage. During this stage of the project, enhancements are implemented and installed, concept of operations and consequence management plans are reviewed, revised, and reconciled, and training for routine operation, maintenance, and consequence management should be conducted. 3.4 Preliminary Testing Once components begin to come on line, the contamination warning system transitions to the preliminary testing stage of implementation. During this period, enhancements are in place and the system is technically operational, however it is being operated in a "test" mode. Meanwhile, the concept of operations and consequence management plan can be finalized, taking into consideration the additional information and insights gained during the design and implementation stages, and reflecting the "as-built" system. May 2007 21 ------- Planning for WS-CWS Deployment 3.4.1 Baseline Operation The automated analytical capabilities of a contamination warning system depend on having a historical baseline or base state against which to match current operational conditions. Anomalies from the baseline create a trigger, which is then analyzed and validated, or dismissed, as a possible contamination event. Depending on the seasonal variability in the operating conditions for a specific water utility, the preliminary testing stage could last up to a year, or even longer. If the utility utilizes multiple water sources, blends sources at varying ratios, or uses an alternative source exclusively during different portions of the year, operations under these conditions should be experienced by the system to gain a complete picture of the normal variability in water quality parameters. Similarly, changes in source water quality, such as episodic taste and odor events, and operational changes, such as periodic changes in distribution system residual, should be incorporated into the base state to the extent possible. The more "normal" conditions that the system can be exposed to during the preliminary testing stage, the more detailed the baseline will be and the more successful the contamination warning system will be in meeting is design objectives, particularly in regards to its accuracy and reliability. During this period of preliminary testing, the system is operating at full capability. All field equipment are installed, communication and IT enhancements are in place, data streams are being transmitted to the utility and personnel have been trained to collect the information pertinent to the contamination warning system. However, the system is being operated for the purpose of collecting data necessary to understand system performance. This does not include responding to the alarms generated by the system, except in the capacity of testing procedures implemented in response to a trigger. Because a baseline has yet to be established, it is difficult to identify whether an alarm is indicative of a water quality problem, or merely an aspect of the base state that had not been previously observed by the system. This mode of operation can place the utility in a vulnerable position. Information indicating possible contamination events will be received by the utility, likely more frequently than before, without knowing the reliability of this information. It is critical, therefore, that the utility remains vigilant in its use of existing, established procedures of possible water quality problems or security concerns during this stage. An example at the pilot utility involved its consumer complaint surveillance component. Prior to implementation of their system, the pilot utility had a threshold value for the number of water quality complaints that were manually logged in a certain database within a set time period. If that number was exceeded, utility personnel began certain protocols to investigate a potential contamination event. During baseline operations, the pilot utility began identifying that a larger number of their consumer calls related to water quality in some way. However, until a baseline value for this new indicator of possible contamination events was developed, response and remediation steps were not initiated for each call or even when the old threshold value was reached. Instead, the utility continued to monitor the number of complaints that were of the same severity as those previously entered into their particular database (although now that process was more automated). When the historically validated threshold was reached, the utility initiated its established response protocol. 3.4.2 Finalization of Concept of Operations and Consequence Management Plan As discussed earlier, the concept of operations is the description of the routine operation and initial trigger validation for each component of the contamination warning system. While the terminology and approach for this documentation may vary, it is important that the operational concepts be clearly documented and that utility staff and local partners understand their roles and responsibilities in operating the system. At this stage, the system-wide concept of operations should be revised to reflect information and insights gained through design and implementation. The concept of operations should now include details of the capabilities of the components, the data streams that will be mined and the IT system infrastructure to collect and analyze the results. Roles and responsibilities for specific job descriptions or critical May 2007 22 ------- Planning for WS-CWS Deployment personnel should be defined and job-specific checklists developed to facilitate the performance of new tasks and new processes implemented as a part of system enhancements. It is critical that the development of the component concept of operations are coordinated to ensure that resources are applied in a consistent and compatible manner across components, and integrated to ensure that data from multiple components can be integrated to better inform response decisions. Recommendations regarding the development of the consequence management plan, and the steps to finalize this document are covered in separate guidance published by the EPA. However, completion of the Consequence Management Plan should be informed by final Concept of Operations to ensure the former is capable of integrating information from the multiple components and affecting a response. 3.5 Operation and Maintenance This stage represents the remaining life of the system. During this period the utility should be prepared to maintain the components to continue to meet the design basis for the life of the system. This stage should not begin until a baseline for the system is established and the consequence management plan is in place. 3.5.1 Operation With the establishment of a robust baseline and the completion of the final consequence management plan, the contamination warning system can be placed into full operation. At this stage the system is operating at full capacity and actively monitoring the water distribution system for contamination incidents and other water quality problems. Each component is feeding its data via established communication protocols into the event detection system(s) at the utility for analysis. When component- specific threshold values are reached, triggers are initiated to indicate a departure from the baseline. If that trigger can be validated by the processes and procedures detailed in the concept of operations, a water contamination event is deemed possible and the Consequence Management Plan is implemented to affect a response. If the event is further confirmed, the Consequence Management Plan guides the remainder of the response and remediation actions. If the components have been well designed and the system as a whole been automated, the day-to-day operations do not look significantly different for the majority of utility personnel involved. 3.5.2 Maintenance Maintenance of the components of the contamination warning system is the most influential determinant of the long-term success of the contamination warning system in meeting its design basis. Maintenance, in the context of a contamination warning system, refers to the activities to maintain the intended capabilities of the system. This includes the physical maintenance of equipment, upgrades of software, and the continual training of personnel. For example, "orphaned" water quality monitors are of little use in a contamination incident and it is vital that personnel responsible for their upkeep and calibration understand their importance. Specifics regarding the maintenance of particular equipment and elements of each component are included in Section 4 - Section 9. However, from a system-wide perspective, if the component designs were completed with an awareness of the need for sustainability, then they would ideally specify equipment, procedures or other enhancements that have dual-use applications. The use of the security enhancements for routine operations of a water utility will ensure that the contamination warning capabilities are fully functional in the remote chance an event were to occur. 3.6 Evaluation and Refinement Evaluation and refinement should be considered in planning for contamination warning system design and implementation. Time and resources should be built into the schedule for carrying out evaluation activities and implementing refinements to optimize system performance. May 2007 23 ------- Planning for WS-CWS Deployment 3.6.1 Evaluation The primary function of contamination warning system evaluation is to gauge the effectiveness, reliability, usability, and sustainability of the system; adjust and streamline the system and approach; and adapt to and incorporate advances in technologies, methods, and protocols that occur over time. Operation. Refers, in general, to all aspects and degrees of functionality, usability and utility. Ideally, the chosen equipment and technologies will not require an increase in skill level or manpower to operate and maintain, and will be robust enough to work under real-world conditions. More likely, however, a balance should be struck between the ideal and what is feasible, affordable, and available. Performance. Measures of performance are equally important to those of operation. Performance of the tools, components, and system refers, in general, to their ability to consistently provide accurate data in a timely manner consistent with intentions of the system design. A system that is easy to operate will not be useful unless it performs as intended to meet the overall system goals. To evaluate all system aspects fully, the performance should be measured in several different ways including the range of contaminants and contaminant classes that may be detected; the accuracy of the data produced; the ability to discern whether a data anomaly is indicative of contamination or caused by something benign and the ability to reliably detect an actual contamination event. Timeliness is an important measure of performance. A system that can detect contamination consistently but requires long periods for detection is of limited use. All aspects of the timeliness of a component's data collection, analysis and event response should be considered. Sustainability. Sustainability will attempt to measure the likelihood that the contamination warning system will become viewed as a sufficiently viable, valuable, cost-effective system resulting in relatively wide spread adoption by the drinking water utility industry. Utilities will view the system as sustainable if it provides benefits that are worth the system life cycle costs. For the benefits to be seen as sufficient, for all except a few utilities it is recognized that the contamination warning system should provide benefits other than warning of intentional contamination. Intentional contamination, although it would have high adverse consequences is viewed as a low probability event by most utilities. Therefore, part of the technical evaluation will focus on identifying and evaluating the level and degree of these dual-use benefits. Life cycle costs include costs to design, install, maintain, and operate the system. These costs will include funding for engineering and utility staff labor, equipment, consumables and spare parts. Evaluating the level and degree of these dual-use benefits will be the second key part of evaluating sustainability. In general, methods for contamination warning system evaluation can be divided into two categories: field evaluation and data analysis. Field evaluation includes activities such as drills and exercises, direct observations and performance tests, interviews, and documentation of lessons learned. Data analysis may involve simulations and/or analysis and integration of data. These methods, along with the objectives and metrics considered during evaluation will be expanded on in future guidance based on lessons learned from the initial Water Security initiative pilot. 3.6.2 Refinement As with any system, refinements to the contamination warning system are likely to be identified through routine operation and maintenance or evaluation. In order to optimize system performance and ensure that the design objectives continue to be met, it will be necessary to make some refinements to the system. Refinements could include modifications to protocols and procedures or may be more extensive such as replacement or modification of equipment. It is also important to consider that technologies may evolve over time, goals and objectives may change, or other events could occur that necessitate refinement to the system. May 2007 24 ------- Planning for WS-CWS Deployment Section 4.0: Online Water Quality Monitoring The online water quality monitoring component consists of multiple water quality monitoring stations installed at key locations throughout the distribution system with the goal of establishing the base state for water quality and using sophisticated event detection systems to monitor for water quality anomalies that could be indicative of contamination. Online water quality monitoring is included as a component of the contamination warning system due to its demonstrated potential to rapidly detect contamination through changes in several commonly measured water quality parameters (Hall, 2007a, EPA 2006). These changes may result from the aqueous chemistry of the contaminant (e.g., dissolution of an organic compound may result in an increase in the TOC concentration) or from reactions with the disinfectant residual (e.g., oxidation of a reactive contaminant consumes the free chlorine residual). While there are limited empirical data regarding the impact of many contaminants of concern on conventional water quality parameters, there has been a substantial amount of research over the past few years demonstrating that many contaminants of concern can produce measurable changes in conventional water quality parameters (Hall, 2007b). Furthermore, many of these contaminants have been shown to impact water quality at concentrations well below reported lethal dose concentrations (Hall, 2007c). Table 4-1 describes the manner in which each of the design objectives presented in Section 1.1 is defined with respect to the online water quality monitoring component, and considerations regarding how these objectives impact design and implementation. Table 4-1. Design Basis Considerations for Online Water Quality Monitoring Design Objective Description Design and Implementation Considerations Capability Can indicate the presence of a contaminant that significantly affects one or more monitored parameters. Detection of water quality changes is an indirect measure of contamination; thus operation of this component should include a process to investigate the possible cause of the anomaly. Contaminant Coverage High detection potential for classes 1, 2, 3, 5, 8, 9, 10, 11 and 12. Moderate detection potential for classes 4, 6, and 7. The specific parameters monitored will determine the contaminant coverage. Disinfectant residual type also has an impact on contaminant coverage (Szabo, 2006). Spatial Coverage Function of location, number, and density of monitoring stations. Several tools are available to design a water quality monitoring network in a manner that optimizes one or more design objectives, such as minimizing consequences over a large number of contamination scenarios. Timeliness Function of hydraulic travel time from the point of contaminant introduction to the sensor, and the concentration of the contaminant. Time to detection can be considered as a primary or secondary objective in the monitoring network design. In either case, a well calibrated distribution system model is necessary to perform this analysis. Reliability Rate of false positive / negative results in this application is largely unknown at this time. May be addressed through event detection systems and consequence management. The design elements with the greatest impact on reliability are the event detection system and the water quality monitoring stations. If reliable sensors are used and properly maintained, the capabilities of the event detection system will dominate reliability as it is defined here. Sustainability Provides utility with a better understanding of water quality variability throughout distribution system and provides an opportunity to optimize distribution system operation. The selection of parameters and monitoring locations will have a direct influence on dual-use applications that will improve sustainability, and thus should be considered in the design of the water quality monitoring system. May 2007 25 ------- Planning for WS-CWS Deployment The primary objective of this section is to describe considerations during planning for the implementation of a water quality monitoring network in a drinking water distribution system as a component of a contamination warning system. These considerations were derived from EPA's experience designing the system at the initial pilot for this program. In planning for implementation of a water quality monitoring network several key design decisions should be made, including the following: Water quality parameters to be monitored Use of a single monitoring station design or multiple designs in a tiered system Specific sensors and instruments integrated into a water quality monitoring station Number of water quality monitoring stations to install Methodology for determining the locations at which water quality monitoring stations will be installed Comprehensive concept of operations to guide routine operations and trigger validation Communication architecture to transmit data from monitoring locations to an operations center IT architecture used to manage and store water quality and related data Event detection software deployed to detect anomalies Staffing available for monitoring station equipment operation and maintenance A key objective of Section 4 is to provide information that will enable the reader to work through these design decisions in a systematic and integrated fashion. While the overall design of an online water quality monitoring system should be developed in an integrated fashion such that all elements are compatible and serve a specific function in the overall system, Section 4 considers five major design elements to facilitate the presentation of material. These elements are: Monitoring Network (Section 4.1): The spatial plan for deployment of water quality monitoring stations throughout a drinking water distribution system. The monitoring network design specifies the number and precise location of each water quality monitoring station. Monitoring Stations (Section 4.2): The specific instruments, probes, or other equipment used to monitor a water quality parameter, as configured into a monitoring station that contains all ancillary equipment (e.g., plumbing, electric, communications, etc.). Communication Systems (Section 4.3): All equipment, software, and services needed to transfer data from each water quality monitoring station to a central location (typically a SCADA control center). Data Management Systems (Section 4.4): All hardware, software, and protocols necessary to manage and store water quality and related data for event detection. The utility SCADA system will typically serve as the foundation of the data management system for the water quality monitoring network. Event Detection Systems (Section 4.5): Software or algorithms designed to analyze real-time water quality data in order to detect anomalous conditions that might be indicative of contamination. Each of these design elements is discussed in a dedicated subsection of Section 4, and is presented in the phases of pre-design, design and implementation, and available tools and resources. Section 4 concludes with a discussion of staffing and cost considerations. 4.1 Monitoring Network Design Monitoring network design is a systematic process for determining the location and number of monitoring stations deployed in a contamination warning system. The design will directly impact two important aspects of system performance: the time of detection and the spatial coverage of the system. Section 4.1 presents information useful to the design and implementation of a water quality monitoring network design, with an emphasis on activities related to planning and pre-design of the monitoring network. When applicable, references to additional resources are included and summarized in Section 4.1.3. May 2007 26 ------- Planning for WS-CWS Deployment 4.1.1 Pre-Design Prior to designing the water quality monitoring network a number of key decisions should be made that may involve some level of investigation and analysis. These decisions include the following: The objectives of the monitoring network design The level of validation needed for the distribution system model The practical upper limit on the number of monitoring stations to be installed Use of a tiered approach to monitoring network design Types of facilities that will be considered candidate locations for monitoring station installation Methodology used to design monitoring network The remainder of this section describes considerations relating to each of these design decisions and, where appropriate, references tools or resources that may be useful in this process. The pre-design phase culminates in selection of a methodology and overall framework for monitoring network design that is based on these key decisions. Network design objectives. A water quality monitoring network can be designed around a number of different objectives, some of which may be complementary while others are in some degree of conflict. Furthermore, the various network design tools available may be able to optimize towards some objectives but not others. Thus, an important step in the pre-design phase is to decide on the primary objective that the network design will attempt to optimize. Examples of design objectives include: o Minimizing the consequences to the population o Minimizing the extent of contamination o Minimizing the time to detection o Maximizing spatial coverage of the distribution system o Maximizing the number of contamination events detected o Maximizing protection of key facilities or populations Given available tools, a monitoring network design can be truly optimized to only one of these objectives; however, it is instructive to evaluate the performance of the monitoring network design with respect to the other objectives, and to consider the trade-offs involved in the selection of a primary objective. Distribution system model validation. Many approaches to monitoring network design utilize distribution system models, and thus the accuracy of the design is dependent on the accuracy of the model. For this reason, distribution system model validation is an important part of monitoring network pre-design. There are numerous approaches to model validation that vary widely with respect to complexity, cost, and resulting degree of model confidence. Approaches to validation of hydraulic and/or water quality portions of a distribution model include: o Desktop analysis to verify that hydraulic behavior matches actual system operations o Pressure studies to validate the hydraulic model o Chlorine decay studies to evaluate the water quality model o Tracer studies to validate the water quality model An up-to-date, accurate network model is useful not only for sensor network design, but also for emergency response planning, and potentially for identifying sampling locations and populations at risk following a contamination incident. Potential resources that may be useful in characterizing distribution system model performance are listed in Section 4.1.3. Maximum number of monitoring stations. In order to proceed with the design of a monitoring network, it will be necessary to establish an upper bound on the total number of monitoring stations that could be installed. This is a function of the total budget for the monitoring network May 2007 27 ------- Planning for WS-CWS Deployment and the unit cost for each monitoring station. It may also include an analysis of incremental benefit of each additional monitoring station added to the system, such as the example shown in Figure 4-1. Illustrative Example o 10 20 30 40 Number of Monitoring Stations 50 Figure 4-1. Example Monitoring Station Tradeoff Curve Tiered approach to monitoring stations. A potential means of increasing the total number of monitoring locations within a given budget is to use a tiered approach in which two or more water quality monitoring station designs, with different costs and capabilities, are deployed. Under this tiered approach, the design of the water quality monitoring network will involve a tradeoff between the total number of monitoring stations installed in the network and the unit cost (and presumably capability) of the monitoring stations, as described in Section 4.2. In the context of the design basis, this is a trade-off between spatial coverage and contaminant coverage. While a tiered design may allow more monitoring stations to be deployed, the stations that monitor for fewer parameters will have reduced contaminant coverage. Therefore, some contamination events that would have been detected by the more complex stations will go undetected even if they pass locations with simpler monitoring stations. Thus, the applicability of a tiered design should be considered at a system level to determine the optimal trade-off between spatial and contaminant coverage, and some of the monitoring network design tools can provide a means of evaluating such a tradeoff. Additional discussion of tiered monitoring stations is included in Section 4.2. Candidate facilities for installing monitoring stations. Prior to designing a water quality monitoring network, it is necessary to identify categories of feasible installation locations, such as: utility facilities, fire stations, police stations, post offices, government buildings, etc. During this initial phase of investigation, a set of general requirements can be provided to facility managers to determine if they would be willing and able to host a water quality monitoring station. These general requirements may include: security, 24/7/365 access to facility, and space for a monitoring station. At this stage it is unlikely that specific facilities will be considered, but rather categories of facilities (e.g., all fire and police stations within a given jurisdiction). Specific locations will not be investigated until a monitoring network design has been completed as described in Section 4.1.2. The purpose of identifying categories of feasible locations at this point is to determine how the monitoring network design will be constrained, and for this reason the categories of feasible locations should be as comprehensive as possible. May 2007 28 ------- Planning for WS-CWS Deployment Methodology for designing the network. The final stage in the pre-design of the monitoring network is selection of methodology for designing the monitoring network. As mentioned previously, most require use of a distribution system model including PipelineNet, Threat Ensemble Vulnerability Assessment - Sensor Placement Optimization Tool (TEVA - SPOT), and various tools built into hydraulic modeling software applications. Section 4.1.3 provides a partial listing of available tools. It is necessary for the utility to have a calibrated distribution system model in order to use these software applications. In selecting a monitoring network design tool, it is important to ensure that tool is compatible with the platform for the utility's distribution system model, and that it will support the overall approach to design. In particular, it is important to ensure that the tool can optimize the design to the desired objective(s). Furthermore, if a tiered design is used, it would be beneficial if the tool could optimize a design that incorporates monitoring stations with differing arrays of sensors. Other factors to consider in selection of a monitoring network design tool include: o Transparency and rigor of the optimization methodology o Applicability of the design and optimization methodology to detection of intentional contamination incidents o Time required to produce a design o Features that facilitate comparison of multiple designs o Visualization tools and compatibility with GIS o Usability If a distribution system model is not available, it will be difficult to optimize a sensor network design to a specific objective, and it may be impossible to systematically evaluate trade-offs for various design options. Expert-based designs can be developed without a model, but have been shown to perform poorly compared to optimization methods that utilize distribution system models (Ostfeld, 2006). The constraints on possible monitoring locations are another important design consideration. In general, designs that place monitoring stations only at utility-owned sites will not be able to perform as well as designs that allow for a large number of potential sites for locating monitoring stations. 4.1.2 Design and Implementation Approach The activities described in Section 4.1.1 describe a process for pre-design of a water quality monitoring network, culminating in documentation of an overall framework and selection of a methodology for designing the monitoring network. The approach for design and implementation of the monitoring network based on the pre-design may include the technical considerations and specifications described below. Design: Develop a comprehensive list of physical addresses for potential installation locations. This list should be based solely on consideration of the general categories identified during pre-design. GIS can be an effective tool for compiling and visualizing this information. Identify nodes in the distribution system model that correspond to each physical address. Develop a suite of design constraints in terms of number of stations, potential installation locations, and type of water quality monitoring station if tiered water quality monitoring network designs will be considered. Use a sensor placement tool, such as TEVA or PipelineNet, to develop a monitoring network design for each set of constraints. Compare the various monitoring network designs (e.g., through a tradeoff analysis, cost benefit analysis, regret analysis, etc.). Select a monitoring network design that specifies number and location of water quality monitoring stations. If a tiered design is used, also specify the type of water quality monitoring station at each location. May 2007 29 ------- Planning for WS-CWS Deployment Field verify each installation location in the design to ensure: o Access to electrical power to run the equipment Certain equipment may require higher voltage or current than may be found through common wall sockets o Access to water to run samples from the distribution system directly to the sensor station o Drainage for the discharge stream from the monitoring station. Verify the discharge option complies with applicable regulations. o Size of area that the station is being installed There should be enough room to contain the sensor station and there should be enough space for utility personnel to access the station for maintenance o Security of the location Limit access to those without a need to maintain the equipment o Accessibility, if location is not a utility owned facility Utility personnel should have 24/7/365 access to the equipment at all times in order to maintain the station or respond to an alarm event o Safety Health and safety should be addressed according to each site's safety procedures. In all cases, however, the minimum safety considerations should meet Occupation Safety and Health Administration (OSHA) requirements. As field verification finds some locations to be unsuitable, iterate through modifications to the monitoring network design and field verification until acceptable physical locations for all water quality monitoring stations have been identified. Obtain any required reviews or approvals on the design. Implementation: Develop agreements with facility owners who will host water quality monitoring stations. o Facility access o Contacts at facility and utility o Water and sewer credits Evaluation and Refinement: Through simulations, evaluate ability of the "as-built" monitoring network design to detect an ensemble of contamination incidents or other water quality anomalies. Upon revision to the distribution system model (e.g., due to system growth, changing demand patterns, recent calibration activities, etc.), evaluate potential modifications to the monitoring network design. Periodic evaluation of potential benefits of the addition of more water quality monitoring stations or the relocation of existing stations. 4.1.3 A valiable Tools and Resources The following tools and resources are available to support the design, implementation, and evaluation of a water quality monitoring network as a component of a contamination warning system: Ostfeld, et. al. "Battle of the Water Sensor Networks", in proceedings of the ASCE/EWRI Water Distribution System Analysis Symposium, August 27-30, 2006. Cincinnati, OH. Berry, J., Fleischer, L., Hart, W.E., Phillips, C.A., and Watson, J.P. 2005, "Sensor Placement in Municipal Water Networks," J. Water Resources Planning and Management, 131 (3): 237-243 (2005). Boccelli, D. L., Shang, F., Uber, J. G. and Wang, J. "Tracer Studies and Water Quality Monitoring for Evaluating Network Model Confidence." 4th International Conference on Watershed Management and Urban Water Supply, Shenzhen, China. 2004. May 2007 30 ------- Planning for WS-CWS Deployment Murray, R., Janke, R., Uber, J., Published in the Proceedings of the ASCE/EWRI Congress, Salt Lake City, UT. The Threat Ensemble Vulnerability Assessment (TEVA) Program for Drinking Water Distribution System Security. 2004. Watson, Jean-Paul, et al, "A Multiple-Objective Analysis of Sensor Placement Optimization in Water Networks", Proceedings of the ASCE/EWRI Congress, June 2004 PipelineNet: http://eh2o.saic.com/iwqss/. Hart, W. E., J. Berry, R. Murray, C. A., Phillips, L. A., Riesen, J. P., Watson, 2007. "SPOT: A Sensor Placement Optimization Toolkit for Drinking Water Contaminant Warning System Design," Proceedings of the World Environmental and Water Resources Congress, Tampa, Florida, 2007. Murray, R., W. E. Hart, and J. Berry. "Sensor Network Design for Contamination Warning Systems: Tools and Applications", Proceedings of the AWWA Water Security Congress, 2006.; TEVA: http://www.epa.gov/nhsrc/water/dw/teva.html; Info Water Sensor Location Manager (SLM): http://www.mwhsoft.com/page/pjroduct/infowaterSLM/infowaterslm feature.htm (based on the PipelineNet tool) USEPA. 2005. Water Distribution System Analysis: Field Studies, Modeling and Management: A Reference Guide for Utilities. EPA/600/R-06/028. 4.2 Monitoring Station Design and Installation Monitoring station design requires careful consideration of the selected water quality parameters to be monitored, as well as of the resources and conditions existing at the intended installation sites. Each individual location will impose unique constraints on the installation, design and operation of a monitoring station, and potentially constrain the parameters or instruments which may be used. Section 4.1 presented general guidelines regarding site constraints for installation and operation of monitoring stations, while Section 4.2 deals with the design of the actual water quality monitoring stations. Section 4.2.1 describes activities related to planning and pre-design of the monitoring stations, while Section 4.2.2 describes the remainder of the design and implementation process in summary fashion. When applicable, references to additional resources are included and summarized in Section 4.2.3. 4.2.1 Pre-Design The most significant decision made prior to designing water quality monitoring stations is selection of the parameters to be monitored. These parameters will define the capabilities of the water quality monitoring system for detection of contamination events as well as operation for dual-use applications. It will also have a significant impact on the cost and design of each water quality monitoring system. This section describes factors to consider in the selection of water quality parameters. One of the primary considerations in selecting water quality parameters for use in a contamination warning system is the potential for parameters to change in response to the presence of a contaminant at concentrations which pose a threat to public health, infrastructure, or acceptability of the water to consumers. A substantial body of research has been conducted which demonstrates that the vast majority of contaminants of concern from a security perspective do alter water quality in a detectable manner. Section 4.2.3 includes references to some of this research, and Table 4-2 summarizes the impact of the different contaminant detection classes, as presented in Table 1-1, on the water quality parameters. Table 4-2. Impact of Contaminant Detection Classes on Water Quality Parameters Class 1 2 3 4 5 Description Petroleum products Pesticides (reactive) Inorganic compounds Metals Pesticides (non-reactive) Example Contaminant Diesel Aldicarb Arsenite salts Mercuric salts Fluoroacetate TOC X X X CI21 X X COND X X May 2007 31 ------- Planning for WS-CWS Deployment Class 6 7 8 9 10,11 12 Description Chemical warfare agents Radionuclides Bacterial toxins2 Plant toxins Pathogens2 Persistent chlorinated organic compounds Example Contaminant VX Cesium-137 Botulinum toxin Ricin Vibrio cholerae PCBs TOC X X X X X CI21 X X X COND X Acronyms: TOC - total organic carbon, Cb - chlorine residual, COND - conductivity 1) Indicated contaminant classes have been shown to consume free chlorine residual. Results are not applicable to chloramine residual. 2) These contaminants are chlorine sensitive, thus it would be necessary to neutralize the chlorine residual in order to maintain potency. Many neutralizing agents would also increase TOC. In general, the results of reported research illustrate that free chlorine is the most sensitive indicator of contamination, showing significant changes from baseline values at concentrations often one to two orders of magnitude below lethal concentrations. Specifically, many contaminants were detected at concentrations around 1 mg/L, while the corresponding lethal concentration might range from 10 to 100 mg/L. In the case of pathogens and bacterial toxins, the active contaminant would generally not produce a detectable change in free chlorine residual; however, co-contaminants would often be present that would reduce free chlorine residual (a condition necessary to maintain viability of these contaminants). These results are applicable to systems using a free chlorine residual, but not to chloramines which were found to be stable in the presence of all contaminants tested, and thus chloramine (or combined chlorine) residual does not appear to provide a reliable means of contaminant detection. These studies also indicate that total organic carbon (TOC) is a particularly useful parameter for detecting the presence of many organic compounds, with a sensitivity ranging from -0.5 mg/L to more than 1 mg/L, depending on baseline levels and variability. Even at the upper end of this range, most organic contaminants should trigger a change in TOC concentration at concentrations well below the lethal concentration. Conductivity was observed to respond slightly to some inorganic contaminants, including some metals, but the response was not as strong as that observed for free chlorine residual and TOC. Nonetheless, conductivity has demonstrated the potential for detection of some contaminants that do not alter chlorine or TOC. Generally, higher concentrations of contaminants are needed to trigger a response from conductivity sensors. Beyond free chlorine residual, TOC, and conductivity, other water quality parameters may provide supporting information about potential contamination. Oxidation reduction potential (ORP) will generally behave similarly to chlorine residual, and can be used to corroborate an observed change in the chlorine residual. ORP may also serve a more prominent role in systems that use a chloramine disinfectant residual as certain oxidation reactions can occur without reacting with chloramines. pH is important to aqueous chemistry and may be useful in understanding observed changes in other parameters, such as free chlorine residual. Studies have generally shown that turbidity is an erratic and unreliable primary indictor of contamination; however, as with pH and ORP, it may be useful in understanding changes in other measured parameters. Other parameters, beyond the basic parameters described above, can be considered in the design of the water quality monitoring system. While it is of primary importance that selected parameters relate to the primary objective of the system - detection of contamination, other factors to consider in selection of water quality parameters include dual-use applications and sustainability. The collected water quality monitoring data may provide useful information and benefits to ongoing operation of the distribution system and water treatment processes. Existing distribution system water quality data should be reviewed in terms of occurrence, average concentrations, and variability to identify May 2007 32 ------- Planning for WS-CWS Deployment other parameters that may provide additional information useful to other objectives relating to water quality or system operations. The sustainability of monitoring for different parameters should be considered. The utility's existing online water quality monitoring program should be reviewed in terms of equipment, performance, and maintenance requirements. Research and industry literature, evaluations, performance studies, etc. should be reviewed to gather information on the performance and maintenance requirements of various technologies. Conduct a preliminary assessment regarding the sustainability of monitoring station designs which incorporate various sensors and technologies. Only those technologies which can be easily and affordably maintained in an acceptable operating condition will be useful in a monitoring system over a long time period. Based on identified monitoring parameters, a preliminary selection of monitoring instruments and analyzers can be made. The selection may be based on the utility's direct experience with various analyzers, consultation with other utilities regarding their experience with equipment, and a review of manufacture technical literature and third party technology evaluations (Environmental Technology Verification [ETV], Technology Testing and Evaluation Program [TTEP], National Sanitation Foundation [NSF], etc.). The parameters selected to be included in the system need not be limited to traditional water quality parameters, or those parameters used in previously published contamination warning system pilot studies. However, the selected parameters should relate to the primary objective of contamination monitoring as discussed under "system design considerations." Depending upon the identified needs, costs and constraints, it may be beneficial to use a tiered system in which two or more monitoring station designs, each with different costs and capabilities, are deployed. A tiered design may allow for more monitoring locations, but at the cost of reduced contaminant coverage at locations with the simplified design. A clear understanding of the contaminant detection potential of each system, and the resulting detection compromises throughout the distribution system is necessary to most effectively deploy a tiered system. 4.2.2 Design and Implementation Approach The activities described in Section 4.2.1 describe a process for pre-design of water quality monitoring stations, culminating in selection of parameters to be monitored as well as the potential to use tiered designs with different levels of detection capability. The approach for design, installation, and operation of water quality monitoring stations based on the pre-design may include the technical considerations and specifications described below. Design: The first phase of monitoring station design should include a visit to each candidate field installation location. An evaluation of each site should consider: Size constraints - Confirm sufficient access and space at the site for installation/fabrication and regular maintenance of analyzers Environmental conditions - Temperature, humidity, vibration, and air quality should be considered as they pertain to the monitoring system as well as the maintenance personnel Site requirements - Ensure availability of sample water, drain, electric power, and the ability to route conduit for data communications. Depending upon site conditions, it may be necessary to condition or regulate the supply water, pump the sample drain, upgrade or modify the site electrical equipment, or locate communications equipment remote from the monitoring equipment. It is also important to verify that the flow of water through the monitoring station is adequate to produce detection times on the order of two hours or less. In some cases, this may require water to be bypassed to drain in order to reduce residence time. May 2007 33 ------- Planning for WS-CWS Deployment Security and accessibility - Monitoring station level of security from tampering and disruption at each location, and degree of accessibility for utility personnel at any time. At non-utility owned sites, it may be necessary to enclose the monitoring system in a locked enclosure. Ensure that water utility personnel will have access to the monitoring station at any time in event of a contamination occurrence. Based on the site surveys and the analyzers identified during pre-design, the list of selected sensors and ancillary equipment should be finalized for each site and each tier of monitoring system. Other considerations to be addressed include the following: Remote sampling equipment details and capabilities Control and data systems including Programmable Logic Controllers (PLCs) and local data loggers Communication methods, such as radio, telephone, internet and associated equipment for the selected method Power assurance, such as an emergency generator or other uninterrupted power supply, that meets reliability and duration criteria required for the system Physical design of the water quality monitoring station (or multiple stations if tiered approach is used) can proceed in the following areas: Frame or panel design - Depending upon whether the monitoring stations will be permanently or temporarily placed in the assigned locations, different types of supporting structures may be required. Specific requirements particular to the surveyed installation sites should also be considered. Placement of analyzers and other components should be considered to result in a system of the required overall size. Electrical system design - The electrical power supply as well as the control, data and communication components should be adequately routed, enclosed and configured. Plumbing system design - Provide sample water to the instruments at a pressure which is suitable for the instruments. Use of a pressure regulator may be required to reduce the distribution system pressure to the required range. Plumbing components which will not corrode, plug or easily break are recommended. The ability to monitor and regulate flow to individual instrument or groups of instruments is useful, however rotometers or other regulating devices should be carefully selected to ensure proper operation. Use of a coarse strainer at the system inlet will prevent small particles from entering the monitoring instruments or components and impacting operation. Sampling system design - An automated system should be included to collect a water sample when potential contamination is detected. Design factors to consider include the quantity of sample to be collected to meet the requirements for laboratory analysis; materials of construction of the sample container and components to preclude contamination; method of activation - automatic or operator initiated; personnel protection from potentially contaminated sample water. Obtain any required reviews or approvals of the design. To illustrate the concepts discussed in this section, a schematic of the water quality monitoring station design used in the initial Water Security initiative pilot is shown in Figure 4-2. May 2007 34 ------- Planning for WS-CWS Deployment PLC/ Electrical Panel Radio Panel 1.J.J roc analyzer a ;a Figure 4-2. Example Water Quality Monitoring Station Design used in the Initial Pilot This design is based, in part, on a requirement for a mobile, easily relocated station. This example design includes a side-mounted TOC, PLC, and radio panel to facilitate installation at sites where space is limited or where it is necessary to move the system through small doorways or hatches to maneuver it into place. Some of the additional hydraulic and mechanical features incorporated in this design include: Supply and Drain Hoses: o For the ease of installation and relocation of the station, flexible hoses were used. o A high quality hose that complies with American Water Works Association (AWWA) standards was used to prevent crimping. Supply and Drain Pipes: o Brass supply piping and fittings are used for durability, ease of installation, and most importantly because they do not leach organic materials that could confound analysis. Sturdy CPVC piping and fittings may also be used, however careful attention to structural stability is required, as well as a flush-out period to purge the system of trace plastic and adhesive chemicals prior to placing the system on line. o Flexible tubing routes sample flow from the supply manifold and flow regulating rotometers to the individual instruments. o CPVC drain piping is used, and generously sized to ensure free flowing gravity drains. o A manual sample supply shut-off valve provides easy isolation of the system in the event of a substantial leak. o A 40-mesh Y-strainer prevents entrained particles from the distribution system from entering sensitive analyzer components. o An actuated solenoid valve routes flow to an appropriately sized container to collect a sample for laboratory analysis when possible water contamination is sensed (not shown). Flow Regulation: o A downstream pressure regulating valve provides sample water at the proper pressure to the monitoring instruments. o Flow regulating rotometers are provided to control and confirm sample flow to each instrument or group of instruments May 2007 35 ------- Planning for WS-CWS Deployment o A manual bypass valve is provided at the downstream end of the sample supply manifold. The outlet of this valve is routed directly to the system drain. The valve provides the ability to easily flush the supply pipe. For monitoring systems which are located a significant distance from the distribution system main, the bypass may be left partially open during normal system operation to reduce the travel time from the main to the station, thus reducing the time to detection of a water quality anomaly. Electrical Panel: o All necessary electrical feeds and monitoring/control signals are routed through an electrical panel to simplify the design and allow for ease of installation. Radio Panel o For a system which communicates via radio, it may be beneficial to locate the radio equipment in a dedicated panel, rather than including them in the electrical panel. Doing so facilitates establishment of a radio link, which may require that the antenna be located a distance away from the monitoring station. o The data communications and analysis aspects of a contamination warning system are further discussed in Section 4.3 The design depicted in Figure 4-2 serves as a reference only and is not to be considered a preferred design. The ultimate design of monitoring stations should be determined by the installing utility and tailored to the unique requirements of each application. Fabrication and Testing: A contract should be established with a qualified fabricator who is familiar not only with electrical and instrumented systems, but also with standards and requirements for piped and tubed hydraulic systems. A sufficiently detailed fabrication specification should be developed and included as part of the contract. Instruments and other major components may be purchased by the utility or by the fabricator. Some cost savings may be realized if purchased by the utility, as well as better assurance and control over delivery schedules. As with any instrumentation and control system, a factory acceptance test should be conducted to ensure that the specified components have been used, that the wiring is to specification and correctly installed, and that all items function as necessary. Inspection and testing of hydraulic components is also necessary to ensure proper operation and leak-free assembly. Identification and correction of errors or problems at the factory can be accomplished with less impact to cost and schedule than those found during site installation. Manufacturer specifications for some sophisticated analyzers recommend that power not be applied until they are installed in the field and connected to the water source. For these types of instruments, be sure to include an appropriate warning in the fabrication specification, and not simply rely on warnings included in the analyzer manual or packing documents. Installation: Depending upon the nature of the designed systems and the installation location, it may be necessary to contract with a separate entity for installation at the monitoring sites. For example, systems which are fully factory fabricated, but which are to be installed in facilities with difficult access may require the services of a qualified mechanical contractor who can rig the units into place. Easy access locations may be installed or fabricated in place by the instrument contractor. Other locations may require the skills of both types of service providers. Installation of the monitoring stations will involve the coordination of a number of entities involved in activities ranging from delivery of the fabricated units to the installation site to final inspection and approval of the installed equipment. A separate installation specification may be beneficial so that the installer is aware of all site conditions and expectations. The specification should clearly define the exact location for installation of the unit as well as details for connections to power, water, communications, and drain. A comprehensive installation and May 2007 36 ------- Planning for WS-CWS Deployment inspection schedule can be developed to ensure that contractors and inspectors are available at key points of the project. Finally, it is important to clearly identify all required inspections for the installed systems so they can be scheduled into the installation timeline. Start-up and Baseline: Adequate time and resources should be allocated so that the monitoring station can be commissioned and brought to a level of operations which is supported by a high level of confidence from the operating and maintenance personnel. Typical commissioning activities include: Initial configuration and calibration of instruments. Acquire calibration and operating solutions and reagents from the instrument vendor or other sources. Note that some reagents have limited shelf life from the date of manufacture, so delayed delivery may be desired. Configuration and testing of communications from the monitoring station to the central control facility Signal testing and verification at each monitoring station and at the central control facility Validation of water quality monitoring station performance (e.g., by routine comparison of sensor measurements with grab samples analyzed with an accepted, independent method). Operation and Maintenance: Operation and maintenance activities include the following: Documentation including, as-built specifications, Operations and Maintenance (O&M) manual, etc.: Purchased instruments and controls equipment are typically delivered by the manufacturer with operating and maintenance instructions. The monitoring station fabricator should also provide catalog and O&M documents for all components provided for the system. One complete set of O&M documents should be made available at the site of each monitoring station, or one set available at a central location. System design drawings should be marked-up and modified to reflect the as-build condition of each monitoring station. This is especially true for electrical design drawings. Accurate drawings should be kept with the system O&M documents and readily available to maintenance personnel and operations technicians. In order to maintain the high level of availability that is necessary for contaminant monitoring systems, a detailed maintenance and calibration plan should be developed and followed. Most instrument technicians are able to perform routine maintenance checks such as confirming sample flow rates and refilling reagent supplies. Some complex analyzers require annual or semi- annual service which may be best left to factory service personnel. A factory service contract, including the cost of replacement consumables may be beneficial. A stock of replacement parts or maintenance items should be kept in supply to enable quick repair in the event of unexpected component failure. For components that have a limited shelf life, a limited quantity of such parts should be maintained, and they should be cycled into field use prior to expiration. For monitoring stations that are installed in facilities not owned by the water utility, it is important to maintain contact with the facility host and adhere to the hosting terms of agreement. O&M schedules delineating field actions for various maintenance and calibration intervals along with record keeping requirements (i.e., log book documentation) Evaluation and Refinement: Maintenance records can provide valuable information regarding the performance and usefulness of selected monitoring instrumentation. Equipment that does not provide useful data in a cost effective manner should be evaluated for replacement with another technology or manufacturer. Periodic comparison of sensor measurements with laboratory measurements can help to ensure that received data values are an accurate representation of the actual water quality. Erroneous readings can result in false positive warnings or missed suspicious events. May 2007 37 ------- Planning for WS-CWS Deployment The state of available sensor technologies and changing contaminant monitoring trends and priorities should be followed. As monitoring equipment and recommended operating strategies evolve, utilities should evaluate whether changes should be made to their monitoring stations. 4.2.3 Available Tools and Resources The following tools and resources are available to support the design, installation, operation, and evaluation of online water quality monitoring stations for the online water quality monitoring component of a contamination warning system: American Society of Civil Engineers. Interim Voluntary Guidelines for Designing an Online Contaminant Monitoring System. US. EPA Cooperative Research and Development Agreement, X-83128301-0, December 9 2004. Grayman, Walter M., et al., Design of Early Warning and Predictive Source-Water Monitoring Systems. AWWA Research Foundation and American Waterworks Association. 2001. Hall, J. et al., "Online Water Quality Parameters as Indicators of Distribution System Contamination," mJAWWA, 99:1:66. January, 2007. Hall, J. et al., "Contaminant Minimum-Dose Threshold Concentrations for Water Quality Sensors." USEPA Report, December, 2007. Available only through WaterlSAC. USEPA, "Water Quality Sensor Response to Potential Chemical Threats in a Pilot-Scale Water Distribution System." USEPA Report, January, 2006. Available only through WaterlSAC. Szabo, J. et al., "Water Quality Sensor Response to Contamination in a Single Pass Water Distribution System Simulator." USEPA Report, January, 2007. Available only through WaterlSAC Szabo, J.G., Hall, J.S. and Meiners, G.C. "Water quality sensor responses to injected contaminants in a chloraminated pipe loop ". American Water Works Association (AWWA) Water Security Congress, Technical Session TUE6: Technology Forum B, Washington, DC, September 10-12, 2006 Hargesheimer, Erika, et al, eds. Online Monitoring for Drinking Water Utilities. AWWA Research Foundation and American Water Works Association. 2002. International Standard ISO 15839. Water quality On-line sensors/analysing equipment for water Specifications and performance tests. 2003 ETV and TTEP evaluation reports of online monitoring equipment. Available at epa.gov/etv. 4.3 Communications Architecture The objective of this task is to provide a long-term, cost effective communications system for relaying monitoring station data to the SCADA system software and ultimately to the Event Detection System (EDS) in a timely manner. There are numerous technologies available today to relay SCADA information from remote sites. These include radio (e.g., spreadspectrum), copper (e.g., cable, telephone), cellular, and fiber to name a few. Each technology has advantages and disadvantages. It is usually desirable for a utility to choose a communications system that is easy to support and is cost effective. Below is a list of activities and functions that may be required to provide a communication system for implementation and operation the online water quality monitoring component of a contamination warning system. May 2007 38 ------- Planning for WS-CWS Deployment 4.3.1 Pre-Design Prior to design, numerous activities are necessary to produce a successful communications project. Staffing the design and implementation team. It is vital that all the stakeholders are represented on the design team to provide consensus on all decisions. Otherwise, the project may be riddled with delays, lack of trust, and poor quality work. o Suggested Team Members include: Public Works or Utility: SCADA Manager IT Manager IT Network Engineer Operations and Maintenance Manager Engineering End users such as Water Quality Staff and System Operators Consultants including representatives from other project groups in reviews. Determine the schedule for the project. Allow for long delays in dealing with communications studies, right of way issues, communication providers, and contracting of sub-contractors. It will be vital to have a project manager with aggressive communication skills to implement this task. Define the requirements of the communication system needed to support the proposed network of online water quality monitoring stations. Consider communication requirements for the enhanced security systems deployed as part of the contamination warning system. Consider the rate and quantity of data to be transmitted. Evaluate existing communication system architecture used to transmit data and commands between remote facilities and the utility operations center. Assess ability of existing system to accommodate the proposed water quality monitoring network. Take into consideration the need for future growth and changing technologies. If existing communications systems are unable to meet the requirements evaluate alternatives. Identify constraints on communication alternatives, e.g., hilly terrain may make radio communications cost prohibitive. Consider an alternative communication system for just the contamination warning system or replacing the entire communications system. 4.3.2 Design and Implementation Approach The activities described in Section 4.3.1 describe a process for pre-design of a communications architecture for a water quality monitoring network, culminating in documentation of initial requirements and a preliminary assessment of the utility's existing communication architecture. The approach for design, installation, and operation of a communications system may include the technical considerations and specifications described below: Design: Select communication technology (e.g., radio, digital, phone lines, fiber, etc.). Multiple technologies may be desirable or necessary in some applications. Determine the communications protocol (e.g., Ethernet). Evaluate for integrity, loss of service, reliability, and security. Develop overall communication architecture. Include all stakeholders identified in the pre-design phase in the development of the communications architecture. Obtain necessary reviews and approvals on the proposed architecture. Schedule frequent review opportunities that include all stakeholders, and provide a high level of consensus along the way. May 2007 39 ------- Planning for WS-CWS Deployment Implementation: Select service provider (if applicable), and establish any necessary contractual relationships. Select an installer, and establish any necessary contractual relationships. Installer's experience with all the technologies needed to complete the work should be heavily weighted in their selection. Identify roles and responsibilities for procurement, installation, and testing of various components of the communications architecture. Work with service provider to get components installed and configured. Test communication pathways between water quality monitoring sites and operations center. Develop location-specific installation specifications. Procure system components that will not be provided by the service provider. Develop installation and inspection schedule. Coordinate this schedule with installation schedule for water quality monitoring stations. Install remote communications systems at water quality monitoring stations. Program PLCs at each monitoring location to record data from sensors at specified polling interval. Verify that the installed communication system at each water quality monitoring site is transmitting data back to the operations center. Start-up and Baseline: Verify data processing and storage at each water quality monitoring location. Verify data integrity during transmission from water quality monitoring locations to operations center. Stress test for communications requirements, including total bandwidth available. Operation and Maintenance: Verify that firmware and software is current with respect to vendor provided updates, patches, etc. Document all communications related settings. This includes all devices that create, push, or receive data within the architecture. This is important for calibration and equipment replacement during routine maintenance. Collect all operation and maintenance documentation. Annual inspection and maintenance of communications hardware (or follow the utility's existing O&M plan for communication systems). Maintain terms of agreement with communications service provider. Evaluation and Refinement: Annual assessment of communication system performance relative to performance specifications documented in the final design. This should include communications integrity and security. 4.3.3 Available Tools and Resources The following tools and resources are available to support the design and implementation of a communication architecture as an element of the online water quality monitoring component of a contamination warning system: National Institute of Standards and Technology, Guide to Supervisory Control and Data Acquisition and Industrial Control (SP800-82). September 2006 Instrumentation, Systems, and Automation Society, Manufacturing and Control System Security, November 2005. National Institute of Standards and Technology, Information Security (SP800-53). February 2005 May 2007 40 ------- Planning for WS-CWS Deployment Telecommunications Industry Association, Data Service Options for Spread Spectrum Systems (TIA/EIA/IS-2000-A), March 2001 4.4 Data Management and IT Architecture The objective of this task to provide support for the processing of data needed to implement the online water quality monitoring component of a contamination warning system. This includes protocols for receiving data from the monitoring stations, incorporating the data into a SCADA system, delivering it to event detection tools, providing warnings of events, providing security of data flow, ensuring backup of systems and data, and passing information to other related systems or entities. This task is a key function to the success of moving information from sensors to SCADA systems to EDS tools. The task lead should be incorporated into all other project groups to insure a complete working system. Section 4.4.1 provides a list of activities and functions to develop a data management and IT architecture for the online water quality monitoring component of a contamination warning system. 4.4.1 Pre-Design Prior to design, numerous activities are necessary to produce a successful data management and IT architecture: Staffing the design and implementation team. It is vital that all the stakeholders are represented on the design team to provide consensus throughout the project. Otherwise, the project will be riddled with delays, lack of trust, and poor quality of work. o Suggested Team Members are: Public Works or Utility: SCADA Manager IT Manager IT Network Engineer Operations and Maintenance Manager Engineering End users such as Water Quality Staff and System Operators Consultants including representatives from other project groups in reviews. Determine the schedule for this task. Consider incorporating this schedule into all project tasks. Nearly all the contamination warning system project tasks rely on some form of data management. Members of this group should be flexible to project changes and good communicators. Change management skills are critical. Define the requirements of the data management system needed to support the proposed network of online water quality monitoring stations. Two primary requirements are to manage data collected from the online water quality monitoring stations and support EDS tools. Other requirements might include, data storage, backups, remote connectivity, protocol standards, software management tools, and data transfer speeds. Evaluate potential options for managing data from online water quality monitoring stations. Consider options for collection by the SCADA system and transfer of data to the EDS tools. Evaluate potential deployment options for EDS tools considering factors such as: o Proximity to source data o Reliability o Security o System compatibility o Computing and monitoring resources Assess the ability of the existing SCADA system to serve these data management functions, including hosting the EDS tool(s). For most installations, this will be the ideal architecture. Consider implications to the health and welfare of the SCADA system. May 2007 41 ------- Planning for WS-CWS Deployment 4.4.2 Design and Implementation Approach The activities described in Section 4.4.1 describe a process for pre-design of a data management architecture for a water quality monitoring network, culminating in documentation of initial requirements and a preliminary assessment of the utility's existing SCADA and IT architecture. The approach for design installation, and operation of a communications system may include the technical considerations and specifications described below. Design: Develop an overall IT architecture. The architecture should include: o Network diagrams including hardware and software resources o Flow diagrams o Personnel interaction Obtain necessary reviews and approvals on the proposed architecture from all stakeholders. Stakeholders may include project members from other teams. Specify all hardware and software to be purchased that meets utility or facility standards. Implementation: Procure required hardware and software Coordinate installation and startup plan with project task leaders and stakeholders Install and test hardware/software according to the approved architecture Program SCADA system with tags for all water quality monitoring locations. Build screens for monitoring and EDS Tool warnings. Install and configure data migration utilities if needed Design and configure data historian and backup system Start-up and Baseline: Verify data transfer among the system components Verify data integrity at SCADA system relative to data collected at the PLC Verify data storage and archive Verify operations of the EDS system within the overall IT architecture Operation and Maintenance: Verify that software is current with respect to vendor provided updates, patches, etc. Document all configurations and settings Collect all O&M documentation Annual inspection and maintenance of IT hardware (or follow the utility's existing O&M plan for IT systems) Evaluation and Refinement: Annual assessment of IT system performance relative to performance specifications documented in the final design. 4.4.3 Available Tools and Resources The following tools and resources are available to support the design and implementation of a data management architecture as an element of the online water quality monitoring component of a contamination warning system: National Institute of Standards and Technology, Guide to Supervisory Control and Data Acquisition and Industrial Control (SP800-82). September 2006 Instrumentation, Systems, and Automation Society (ISA), Manufacturing and Control System Security, November 2005. May 2007 42 ------- Planning for WS-CWS Deployment Inter National Committee for Information Technology Standards, Information Technology - Security Techniques (17799-2005) National Institute of Standards and Technology, Information Security (SP800-53). February 2005 4.5 Water Quality Event Detection Water quality monitoring stations only provide information about general water quality conditions at a specific location and time, and in isolation are not well suited to detect contamination. This is compounded by the fact that water quality at a given location can be highly variable, with several factors affecting water quality at a given time including pump and tank operations, system demand, and source water quality. Sophisticated algorithms incorporated into EDS tools can efficiently mine the large amount of water quality data produced by these monitoring stations and detect anomalies that may be indicative of contamination or other water quality problems that relate to dual-use application of the water quality monitoring system. Thus, the EDS tool implemented as part of the water quality monitoring component of a contamination warning system is critical to the performance of the system and has a significant influence on the overall reliability of the system. Likewise, EDS tool performance impacts sustainability because a system plagued by false alarms will be quickly ignored and ultimately forgotten unless the performance can be improved through tuning of the EDS tool or inclusion of more data. The EDS tool may also impact contaminant coverage and timing of detection. Given that EDS performance impacts four aspects of the design basis (reliability, sustainability, timeliness, and contaminant coverage), selection of the tool is of paramount importance to the water quality monitoring component of a contamination warning system. Section 4.5 presents information useful to the selection and implementation of an EDS tool for water quality monitoring, with an emphasis on activities related to planning. When applicable, references to additional resources are included and summarized in Section 4.5.3. 4.5.1 Planning Unlike the previous four design elements of the water quality monitoring system, EDS tools will commonly be off-the-shelf products, and thus do not require the same level of design and pre-design as the other elements. However, given that the application of EDS to water quality is relatively new and that there are few published, third-party evaluations of these tools, careful consideration should be given to the basis for selection of an EDS tool. Thus much of the effort associated with the planning stage of water quality event detection is collection of information that will lay the ground work for design and implementation of a selection study. Suggested information collection efforts include: A review of utility water quality and operational data A preliminary assessment of metrics against which EDS tool performance can be evaluated A market survey to identify potential EDS tools A summary of EDS tool specifications for candidate tools An assessment of potential deployment environments for the EDS tools The remainder of this section describes considerations relating to each of these planning activities and, where appropriate, references tools or resources that may be useful in this process. The planning phase produces a list of candidate EDS tools and a summary of specifications, performance study results, and other available information for each tool. Part of planning for an EDS selection study is conducting a thorough review of all utility data that may be useful for water quality event detection, such as water quality and operational data. Once water quality parameters have been selected, per the pre-design activities described in Section 4.2.1, a review of existing distribution system water quality data may provide insight regarding typical values and expected variability. While the degree of variability in water quality parameters, especially free chlorine residual, May 2007 43 ------- Planning for WS-CWS Deployment is highly dependant on location, a general sense of distribution system variability may be useful in the design of the selection study and may even impact monitoring network design (see Section 4.1.1). Operational data is important because system operations can have a dramatic and predictable impact on water quality. Incorporation of operational data into water quality event detection can improve the performance of the tools as it allows for incorporation of cause and effect relationships that result in water quality changes. However, in order to use this data, the utility should have a firm understanding of the available data and knowledge regarding how operational changes impact water quality throughout the system. As with water quality variability, these relationships are location-specific. At the planning phase it is sufficient to characterize the operational data that is available so that it can be used in either the selection study or during implementation. Also important in the planning process is a precise characterization of the requirements for an EDS tool. Each utility's needs, priorities, and available resources are unique, so a ranking of the important aspects of an EDS tool is key in tool selection. A sample ranking of objectives could be: 1. An EDS tool should be compatible with current IT infrastructure and be easily integrated with existing SCADA system or other applications 2. The tool should detect a high percentage of potential contamination events (e.g., 99.99%) 3. The tool should have a minimal number of false alarms (e.g., a maximum of one per week) 4. The tool should have minimal maintenance requirements and should be supported by the tool developer 5. The tool should have dual-use functionality and be able to detect water quality anomalies that could arise from variety of causes that may be of concern to the utility (e.g., cross-connections) 6. Minimal personnel training should be necessary to implement the tool 7. The tool should be able to handle noisy and imperfect data 8. The tool should have minimal cost, both initial and ongoing, including any necessary software and hardware The tables below describe some standard performance measures that can be used to evaluate and compare EDS tools. Table 4-3 describes some performance measures that quantify the detection capability of an EDS tool, and in this table an event is defined as a continuous period of time during which water quality is anomalous at the monitoring locations. Table 4-3. Standard Measures for Evaluating EDS tool Performance Performance Measure Specificity Sensitivity False Alarm Rate Average False Alarm Length Median Detection Time Description The percent of time for which the EDS tool correctly does not alarm The percent of events that the EDS correctly identifies The frequency of false alarms The average length of a false alarm The median time it takes the EDS tool to detect an event relative to the time the event reaches the monitoring station Example The tool correctly does not alarm 99.5% of the time The tool detects 98% of events On average, the tool produces one alarm per week The duration of a false alarm ranges from 2 minutes to 3 hours, with an average of 15 minutes The delay between the time the event is at the monitoring station and the EDS tool alarms ranges from 2 minutes to 84 minutes, with an average of 6 minutes The performance of the tool, as characterized by metrics such as those in Table 4-3, should guide the selection of a tool, other factors relating to the operation, usability, and maintenance of the tool are important considerations as well. Table 4-4 describes other factors that may be considerations in the selection of an EDS tool. May 2007 44 ------- Planning for WS-CWS Deployment Table 4-4. Sample Measures for Evaluating EDS Software EDS Software Performance Measures Initial Cost Recurring Cost Calibration or Training Compatibility Customization Usability User Interface Efficiency Reliability Developer Support Description Fee for purchase and installation of EDS tool Costs incurred for labor materials necessary to support the operation and maintenance of EDS tool Procedures for training EDS tools, ability of staff to learn and implement training procedures, and requirements for re-training Compatibility with existing hardware, software, schemas, and other aspects of the planned deployment environment Ability to customize EDS tool to a particular utility environment, including: monitoring locations, water quality parameters, operational data, alarm threshold, and information displays. Ability to analyze WQ data from multiple locations in near real-time. Level of skill required to train, operate, and maintain the EDS tool Capability of the user interface to navigate the major features of the tool and to display information in a usable format. Speed and memory requirements for software Percentage of time EDS operates as designed Availability of adequate, long-term support for the operation of the EDS tool Generally, two preparatory processes are carried out by an EDS software developer before an EDS tool is deployed at a water utility in real-time operation. First, the tool is trained on a dataset from each location that will be monitored in real-time. This training dataset should be free of anomalies or events so that it can be used by the EDS tool to "learn" typical water quality patterns. For example, a 0.2 mg/L change in chlorine at one location might be normal under certain conditions, whereas it would be extremely rare (and considered anomalous) at another location. Once trained, the tool would respond differently to such a change at the two locations (it would likely alarm for the second location but not the first). Once training is complete, the EDS tool developer will typically work with the utility to configure the tool, or adjust variables within the tool to maximize event detection performance (sensitivity) while maintaining a false alarm rate that is acceptable for the utility. A receiver operating characteristic (ROC) curve, which plots 1-specifity vs. sensitivity, can be helpful in quantifying the tradeoff between detection capability and false alarm rates for a given tool. Ideally, the curve should hug the top left corner of the graph, as this represents a low number of false alarms with a high probability of detection. Figure 4-3 provides an example ROC curve. False Alarm Rate (FAR) Figure 4-3. Example ROC Curve In order to develop a candidate pool of EDS tools for consideration in the selection study, a market survey or literature review is proposed. This information gathering exercise should consider the performance metrics and EDS tool attributes discussed above, such as what information is readily available. Potential sources of information include technical and scientific literature, vendor information and websites, and information from utilities currently using or considering using EDS tools. Note that there are a limited May 2007 45 ------- Planning for WS-CWS Deployment number of tools marketed specifically for event detection in water distribution systems. However, there are many general tools for event detection or anomaly detection that could potentially be adapted to the water quality domain. Adaptation of general event detection tools to the water quality domain may require additional tool customization and/or integration efforts. The Overview of Event Detection Systems (USEPA, 2005c) describes some available tools, and also includes examples of how tools have been used at water utilities. Additional information sources are included in Section 4.5.3. The results of this market survey can be used to develop a list of candidate EDS tools for consideration in the selection study. More detailed information and product specifications should be collected for each of the candidate tools as this information will be critical to the design and conduct of the selection study as discussed in Section 4.5.2. 4.5.2 Implementation Approach The activities described in Section 4.5.1 describe a process for performing a preliminary assessment of EDS tools, culminating in a list of candidate tools and a summary of performance specifications. The approach for the selection, implementation, and operation of an EDS tool may include the technical considerations and specifications described below. Prior to selecting an EDS tool, a preliminary concept of operations as described in Section 3.2.1 should be developed. This exercise will help to define requirements for how data can be provided to the tool for analysis and may place some constraints on tool selection. Development of the concept of operations should also be tightly coupled with communications and data management activities. Planning-Selection Study: Define requirements for EDS tool performance, compatibility, usability, features, support, etc. Precisely define the metrics and attributes that will be considered in the EDS tool selection study, and document how each will be assessed in either a quantitative or qualitative manner. Perform initial analysis of candidate EDS tools against requirements to reduce the pool of candidate technologies to those that meet the most critical qualitative attributes. From the remaining list of candidate tools, develop quantitative basis for selection of an EDS tool for deployment. Options for a selection study include: o Utility-specific EDS tool evaluation using data collected from each water quality monitoring location after the monitoring stations have been found to be producing valid data. (Guidance on the design of an EDS evaluation study is currently under development by USEPA based on experience at the initial Water Security initiative pilot.) o Third party technology evaluations, either through established programs such as ETV, TTEP, NSF, or through research programs. o EDS tool performance documented in peer-reviewed literature and vendor-supplied information. o Experience and data from other utilities using EDS tools. Select EDS tool(s) for deployment at the utility based on the results of the selection study. Implementation: Verify that the utility's IT system architecture can accommodate the selected tool. Modify the architecture if necessary. Collect water quality data from all installed water quality monitoring stations to support initial training of the EDS tools. Use only data collected after it has been verified that the water quality monitoring stations are producing reliable data. Identify operational data that is expected to enhance EDS tool performance and the associated operational logic that relates water quality at a specific monitoring location to distribution system operations. Collect relevant distribution system operational data to support training of the EDS tools. May 2007 46 ------- Planning for WS-CWS Deployment Train and tune the EDS tool(s) to each water quality monitoring location using the data described above. If a utility-specific EDS tool evaluation was performed, data from the evaluation may be useful in training the tool(s) for deployment. Install and test EDS tool(s) according to the architecture developed under "data management." Modify the concept of operations as appropriate to reflect the as-built design. Start-up and Baseline: Bring the EDS online and establish connectivity with SCADA or other automated water quality data source. Train utility staff on EDS tool operation and maintenance, as well as the concept of operations for the water quality monitoring component of the contamination warning system. Monitor EDS tool performance over the first several weeks of operation, and make adjustments as necessary. Operation and Maintenance: Monitor and respond to EDS alarms in accordance with the concept of operations. Log any anomalies detected by the EDS tool(s). Update the EDS tool event library, if applicable (some tools keep a library of water quality types observed at a given monitoring station so that a repeat occurrence of the water quality type can be classified as normal). Retrain the tool if new sensors are added, water quality changes significantly, or performance is found to be unsatisfactory. Verify that EDS tools and supporting software are current with respect to vendor-provided updates, patches, etc. Evaluation and Refinement: Continuous documentation of performance during operation, e.g., false alarms, detection of true water quality anomalies, and dual-use benefits. Through simulated events, periodically evaluate the overall performance of the installed EDS tool(s). If necessary, update EDS tool configuration to optimize performance. Track and evaluate the development of new EDS tools. 4.5.3 Available Tools and Resources The following tools and resources are available to support the selection, evaluation, and implementation of EDS tools as part of the online water quality monitoring component of a contamination warning system: U. S. Environmental Protection Agency, 2005. "Overview of Event Detection Systems for WaterSentinel." http://www.epa.gov/safewater/watersecuritv/pubs/watersentinel event detection.pdf U. S. Environmental Protection Agency, 2006. "Framework for the Evaluation of Event Detection Software for Drinking Water Contamination Warning Systems." Kroll, D., King, K. (2006). Real World Operational Testing and Deployment of an On-line Water Security Monitoring Station. In Proceedings ofWDSA 2006 Symposium. Cincinnati. Klise, K., McKenna, S. (2006). Multivariate Application for Detecting Anomalous Water Quality. In Proceedings ofWDSA 2006 Symposium. Cincinnati. Jarrett, R., Robinson, G., O'Halloran, R. (2006). On-line Monitoring of Water Distribution Systems: Data Processing and Anomaly Detection. In Proceedings ofWDSA 2006 Symposium. Cincinnati. Umberg, K., Uber, J., Murray, R. (2006). Performance Evaluation of Real-time Event Detection Algorithms. In Proceedings ofWDSA 2006 Symposium. Cincinnati. May 2007 47 ------- Planning for WS-CWS Deployment Hart, D., S. A. McKenna, K. Klise, V. Cruz, and M. Wilson, 2007. "CANARY: A Water Quality Event Detection Algorithm," Proceedings of the ASCE World Environmental and Water Resources (EWRI) Congress, Tampa, Florida, 2007 4.6 Staffing and Cost Considerations Planning for the implementation of the water quality monitoring component of a contamination warning system requires involvement of a wide array of utility personnel and potentially contractor staff. Costs will be highly dependent on the utility's capabilities and intended enhancements. Therefore, the remainder of Section 4.6 illustrates the staffing considerations and cost factors that are recommended for consideration during project planning and pre-design. This section provides a summary of previously discussed information that should be considered when developing preliminary staffing plans and cost estimates. Cost considerations represent some unique aspects of implementation based on lessons learned from the initial pilot. 4.6.1 Staffing As mentioned above, staffing considerations are critical to the successful implementation of a contamination warning system. Table 4-5 offers a quick overview of which staff may be necessary to design, implement, and operate an online water quality monitoring program as part of a contamination warning system and during which phases of implementation these personnel may be needed. Table 4-5. Online Water Quality Monitoring Staffing Considerations Division or Department Water Quality Information Technology Engineering and Planning Operations and/or Distribution Host Facilities Implementation Stage PD X X X X D X X X X I X X X X X PT X X X X O&M X X X X E&R X X X Comments System end user. Defines system requirements. Lead in selection of parameters and instruments. Lead in monitoring network design. Lead in operations and maintenance. Designs, implements, and manages all IT systems used to support online water quality monitoring. Review and approval of designs. Responsible for installation oversight and inspections. Support monitoring network design. Provides domain knowledge relating to system operations. Responsible for monitoring water quality alarms 24/7/365. Support installation and maintenance of monitoring stations. Provide access to facilities during installation and maintenance activities. Contact utility in the event of a problem that could impact the monitoring station. PD = Pre-design; D = Design; I = Implementation; PT = Preliminary Testing; O&M = Operations and Maintenance; E&R = Evaluation and Refinement Building the team to implement this component of a contamination warning system will involve all divisions within the utility. Therefore, it is important to have senior leadership involved and invested during each stage to facilitate the resolution of cross-division issues. Several key personnel, like the IT manager and the Water Quality division manager, would ideally be members of this and all other component teams to facilitate the application of the system engineering principles outlined in Section 2.1. Such involvement can be a significant commitment of time and resources for these individuals, but the utility can reap substantial benefit in the long-term success of the system. Other team members' participation will be less demanding, but still critical, and will vary depending on the utility-specific gap between the initial conditions and the final planned capabilities of the contamination warning system. Furthermore, it may be useful to obtain the help of consultants and contractors to aid the utility in the implementation of an online water quality monitoring system, especially given that the level of effort required to implement this component would go beyond the human resources at many utilities. Consultants and contractors can be critical partners during each stage of the process by providing May 2007 48 ------- Planning for WS-CWS Deployment component-specific technical knowledge and installation experience, and by delivering training on the new equipment, procedures and processes. 4.6.2 Cost Considerations This section presents a summary of the design and implementation considerations discussed above that may influence costs. This list also may include other factors that were encountered during implementation of the initial Water Security initiative pilot and could be overlooked during cost estimation in the absence of this experience. Although this list of cost considerations may not be exhaustive, these factors, at a minimum, should be considered when planning. Pre-Design: Development of design objectives for each element of the online water quality monitoring component Assessment of existing communications, SCADA and IT systems Design: Development of preliminary concept of operations for the water quality monitoring component Monitoring Network: assessment and calibration of hydraulic model used in the development of the monitoring network design, followed by field verification of potential monitoring station locations and development of installation specifications for selected locations Monitoring Station: layout and design of multi-sensor instrument rack (or racks, if a tiered network approach is implemented) Communications and IT Architecture: design of communications and data management architecture; testing of communication pathways and conduct radio survey if necessary Water Quality Event Detection System: design of the event detection system architecture; design and implementation of event detection system tool evaluation study Implementation: Monitoring Network: coordination of installation at non utility-owned monitoring station locations Monitoring Station: equipment procurement and rack fabrication; transportation and installation of monitoring stations Communications and IT Architecture: establishment of contract with communication service provider(s); procurement, installation and configuration of communications and data management hardware and software Water Quality Event Detection System: procurement, installation, and testing of event detection system hardware and software; training and configuration of event detection system tools Preliminary Testing: Monitoring Station: initial calibration and support; equipment shakedown and training on operation, maintenance and alarm response Communications and IT Architecture: test of installed data management architecture and complete communication system Water Quality Event Detection System: preliminary evaluation of event detection system performance, and fine-tuning the configuration as necessary Operations and Maintenance: Monitoring Network: water and sewer credits to non-utility hosts of monitoring stations Monitoring Station: development of written documentation; scheduled and unscheduled sensor maintenance, repair and upgrades May 2007 49 ------- Planning for WS-CWS Deployment Communications and IT Architecture: monthly service fees for communications service provider(s); development of written documentation; scheduled and unscheduled communication equipment maintenance, repair and upgrades Water Quality Event Detection System: scheduled and unscheduled EDS equipment maintenance, repair and upgrades Evaluation and Refinement: Monitoring Network: ongoing hydraulic model update and recalibration; potential relocation or addition of monitoring stations Monitoring Station: equipment upgrades due to improvements in sensing technology Communications and IT Architecture: equipment upgrades due to improvements in communication technology Water Quality Event Detection System: drills and exercises, including simulations; data analysis; equipment upgrades due to improvements in event detection technology Based on experience at the pilot utility, the most significant issues will likely be the capability of the existing communication system and the upfront capital cost associated with the fabrication of the monitoring stations. Unfortunately, it may not be possible to receive an accurate estimation of the costs associated with each monitoring station until a prototype is created and unit costs can be calculated. The use of a tiered monitoring station design may reduce the eventual equipment costs, but additional prototypes would be necessary, maintenance and operation may be more complicated, and there will be a reduced contaminant warning capabilities at locations with a simplified design. The end result should be a balance between capability of the entire system and cost. Finally, the cost to operate and maintain even a modest water quality monitoring network can be significant, and should be considered early in the planning stage to ensure that the system built can be sustained. May 2007 50 ------- Planning for WS-CWS Deployment Section 5.0: Sampling and Analysis Although a critical aspect of contamination warning systems, sampling and analysis is not considered an early detection strategy. Rather, sampling and analysis serves the following three functions: Baseline monitoring. Data from samples collected and analyzed during the initial stages of implementation are used to create a "baseline" profile of contaminant occurrence in the distribution system, as well as characterize possible matrix effects on method performance. Maintenance monitoring. Ongoing sampling analysis activities maintain laboratory proficiency for techniques that may otherwise only be used in response to a triggered event and are used to continually update baseline data seasonally. Triggered sampling and analysis. Sample collection and analysis in response to contamination indicators by other contamination warning system components is part of the credibility determination and consequence management process. Triggered analyses may be specific, based on information available from other components, or may involve a broad screen to potentially detect unknown contaminants. Table 5-1 summarizes design and implementation considerations for the sampling and analy component of a contamination warning system. Table 5-1. Design Basis Considerations for Sampling and Analysis sis Design Objective Description Design and Implementation Considerations Capability Can positively identify the presence of any contaminant in the suite of target analytes and above a well-defined minimum reporting level. Assess existing laboratory capability and capacity and identify enhancements or establish laboratory networks. Contaminant Coverage High detection potential for classes 1, 2, 3, 4, 7, and 12; Moderate detection potential for classes 5, 6, 8, 9, 10, 11. Methods and analytical approaches should be identified for as many of the contaminant classes as possible. Methods should be validated for use in drinking water. Spatial Coverage Function of location, number, and density of sampling stations, as well as sample type (composite vs. grab). Consider sampling at locations identified as priority sites through sensor network design (Section 4.1) as these could be the source of triggered sampling events. Also consider aspects of the distribution system such as source water, water age, and pipe material that could result in variability or method interferences. Timeliness Function of sampling & analysis frequency and the total time to process the sample and analyze the results. Baseline monitoring should occur at a frequency sufficient to support data quality objectives, inform the design of maintenance monitoring and provide an understanding of method performance and variability in the distribution system prior to full operation of the contamination warning system. Maintenance monitoring should occur at a frequency sufficient to maintain laboratory capabilities, update baseline data to account for seasonal variability, and support dual-use applications. Reliability Function of the reliability of sampling and analysis methods (high for established techniques). Baseline needed for reliable interpretation of results. Methods utilized for baseline, maintenance, and triggered sampling and analysis should be validated for use in drinking water. In some cases where analytical methods have not been fully validated for certain classes of contaminants, procedures to demonstrate initial and ongoing proficiency should be implemented to support interpretation of results. Sustainability Provides utility with an opportunity to exercise sampling and laboratory protocols and may; provide information about previously unknown contaminants that occur in the system. Assess whether or not to enhance in-house laboratory expertise or rely on outside laboratories for support. This decision will influence level of effort and costs. As indicated previously, dual use applications should also be considered in terms of sustainability of the program. May 2007 51 ------- Planning for WS-CWS Deployment The remainder of this section is organized according to the following design elements described: Laboratory capability and capacity (Section 5.1). This includes consideration of the contaminants to monitor, analytical laboratories that would be able to support both routine and non-routine analyses, and analytical methods and data quality objectives. Sampling and analysis activities (Section 5.2). This includes consideration of sampling locations, sampling frequency, and ongoing sampling and analysis procedures. Site characterization and field screening (Section 5.3). This includes consideration of the roles that will be played by the utility and others investigating an incident, and consideration of the field testing capabilities that will be needed to conduct preliminary assessments of contaminants. Discussion of these design elements is presented in the phases of pre-design, design and implementation, and available tools and resources. Section 5 concludes with a discussion of staffing and cost considerations. 5.1 Laboratory Capability and Capacity Establishing and maintaining adequate laboratory capability to address a range of potential contaminants is a fundamental aspect of contamination warning system design and implementation. Although some of capabilities may be met with an existing or expanded utility laboratory, analyses for at least some contaminant classes should be performed by other laboratories and coordinated by the utility. In addition, sufficient capacity for these analyses should be considered to ensure that a large number of samples does not overwhelm the laboratory performing the analyses. 5.1.1 Pre-design During the pre-design phase for laboratory capability and capacity, the utility's design objectives should consider the following: Identifying potential contaminants of concern Identifying qualified laboratories that can support (and will commit to supporting) sample analysis needs that cannot be met through existing or expanded utility laboratory capabilities Identifying the analytical methods that will be used for each of the targeted analytes Evaluating the capabilities and credentials of each laboratory with respect to the identified methods to identify the most effective assignment of laboratory roles Identifying Potential Contaminants of Concern The sampling and analysis component should address the contaminant classes presented in Table 5-2. As noted in this table, consideration should be given for an analytical approach for each contaminant class. Although a goal of implementation is to maximize the number of analytes monitored in each class, there are inherent limitations in monitoring capabilities for some contaminant classes due to the need for specially equipped laboratories and the proximity of, or access to, these laboratories by the utility. Table 5-2. Considerations for Analytical Approach to Establishing Sampling and Analysis Capabilities by Contaminant Class Class 1 2 Description Petroleum products Pesticides (with odor or taste) Example Contaminants Diesel Aldicarb, fenamiphos, cyanide salts Considerations for Analytical Approach Screening for volatile and semivolatile organic compounds1 Various methods may be applicable, depending on targeted contaminants, including: liquid chromatography, gas chromatography, and spectrophotometric methods May 2007 52 ------- Planning for WS-CWS Deployment Class 3 4 5 6 7 8 9 10,11 10,11 12 Description Inorganic compounds Metals Pesticides (odorless) Chemical warfare agents Radionuclides Bacterial toxins Plant toxins Pathogens (select agents) Pathogens (non-select agents) Persistent chlorinated organic compounds Example Contaminants Arsenite salts, strychnine Mercuric chloride Sodium fluoroacetate VX Cesium-137 Botulinum Ricin Bacillus anthracis Vibrio cholerae PCBs Considerations for Analytical Approach Various methods may be applicable, depending on targeted contaminants, including inductively coupled plasma/mass spectrometry (ICP/MS) and chromatography with different detectors, Screening for heavy metals using ICP/MS Ion chromatography with conductivity detection (method is currently being validated) Analysis by a surety laboratory with access to restricted standards Screening for alpha, beta, and gamma emitters Analysis of the toxins and pathogens currently addressed by the state public health laboratory participating in the Laboratory Response Network (LRN) Analysis for a minimum of two non-select agents by procedures recommended in EPA SAM2 Gas chromatography/mass spectrometry methods Data quality objectives for the chemical analyses should include detection at the parts per million level for specific chemicals (e.g., aldicarb, rather than total organic carbon) 2 EPA Standardized Analytical Methods for Environmental Restoration following Homeland Security Events (SAM) A candidate list of target contaminants can be developed from the information in Table 5-2 and the additional details on specific contaminants and methods from the references listed in Section 5.1.3. Of these references, EPA's Water Contaminant Information Tool (WCIT) and EPA's Standardized Analytical Methods for Environmental Restoration following Homeland Security Events (SAM) are particularly useful for identifying contaminants of concern in drinking water. Identifying Laboratories to Support the Contamination Warning System It is unlikely that a water utility laboratory will have the capability to analyze for all of the target contaminants on the list that is developed. To meet the design objectives of the sampling and analysis component, the utility should establish relationships with multiple laboratories, potentially including commercial laboratories, municipal laboratories, and state laboratories. Ideally, these relationships will be in the form of contracts or purchase orders. This is the likely option for relationships with commercial laboratories, but also may be an appropriate vehicle for accessing local or state laboratory capabilities. Alternatively, an interagency agreement or memorandum of understanding may be needed. Note that developing and executing these contracts or vehicles is addressed during design and implementation (Section 5.1.2). During the pre-design phase, the goal is simply to identify the laboratories that will be part of the analysis network, work with the laboratories to establish roles, and agree upon the vehicle that will be used to access their capabilities. To ensure timely results, utilities should identify labs within close proximity to the utility, when available. The utility also should ensure that laboratories are qualified to perform analyses when these analyses are monitored under laboratory oversight programs. For regulated drinking water contaminants, laboratories should be certified through the EPA Drinking Water Laboratory Certification Program or the National Environmental Laboratory Accreditation Program (NELAP). For analyses of contaminants addressed by other programs, such as select agents under the Centers for Disease Control (CDC) Laboratory Response Network (LRN), the laboratory should be approved under these programs. Laboratories should provide documentation of certifications and accreditations. Some commercial laboratories may hold NELAP accreditation for non-regulated contaminants of concern. Determining the analytical support that each laboratory would provide can be approached using the following steps: May 2007 53 ------- Planning for WS-CWS Deployment 1. Determine which contaminants can be analyzed using the utility's current in-house laboratory capabilities and capacity 2. For the remaining list, determine which contaminants can be analyzed by the in-house laboratory by expanding capabilities (acquiring instrumentation, or certifications) or capacity (additional staff or training) 3. For the remaining list, determine which contaminants can be contracted to one or more commercial laboratories 4. For the remaining list, determine which contaminants can only be addressed through support from local or state public health or environmental laboratories Due to security restrictions, handling requirements, facility containment requirements, or instrumentation, only a limited community of laboratories will have the capability to perform analyses for toxins, select agents, radiochemicals, and chemical warfare agents. It may be difficult to identify a support laboratory in close proximity of the utility. This gap in analytical capabilities should be considered as part of the overall contamination warning system. It may be desirable to enhance credibility determination capability through other components. Identify Analytical Methods After establishing the target list of contaminants and laboratories, analytical methods should be selected for contaminants for which multiple method options are available. The utility should consider using the following steps to identify the most appropriate method to use: 1. Determine if there is an approved EPA method for measurement of the targeted analyte(s) in drinking water 2. If there is no approved EPA method, consult SAM as a resource to identify methods (Section 5.1.3) 3. If SAM does not recommend a method, consider other methods for measurement of the analyte of interest that have been validated for use in drinking water (potentially validated in other matrices if there are no drinking water methods available). In some cases, a validated analytical method may not be available or appropriate for use in routine monitoring for a given contaminant or class. For these instances utilities should identify opportunities to participate in method validation studies organized by EPA or other organizations to address this gap. Regardless of the validation status of a method, the most critical factor in ensuring that data are of known and documented quality is ongoing assessment of method performance in the laboratory performing these analyses for the contamination warning system. This assessment depends on the contaminant and technique, and may include initial and ongoing spiked samples and blanks to assess bias, precision, sensitivity, and contamination. The utility also should consider selecting methods that can be used to simultaneously measure several of the targeted analytes to improve analytical efficiency. Analytical techniques using chromatography and/or mass spectrometry are examples. For both routine and triggered method selection, both screening capability and confirmation capability of candidate methods should be taken into account. For instances where the likely contaminant has not been identified, specificity and reliability of data may not be as critical as broad detection and rapid results. For instances where a specific contaminant is suspected, specificity and reliability is paramount. Establishing Laboratory Roles When multiple laboratories are capable of performing a method, the following factors should be considered in determining which laboratory should be assigned responsibility for the analysis: Is the laboratory certified/accredited/approved for the method in drinking water? Does the laboratory's quality assurance (QA) program for the method include the following: May 2007 54 ------- Planning for WS-CWS Deployment o Sample receipt, storage and tracking protocols to chain of custody o Initial and ongoing proficiency testing and quality control (QC) analyses for all analytes and methods of interest o Data review procedures o Data storage and transfer o Documentation of personnel qualifications and training o Ability and willingness to analyze samples potentially containing unknown contaminants If no certification/accreditation/approval is available for the method, is the method currently performed routinely by the laboratory? If the method is not routinely used by the laboratory, is the method a modification of a method the laboratory routinely uses? 5.1.2 Design and Implementation Approach Design and implementation of the approach determined through the contaminant and laboratory selection process should include the technical considerations and specifications described below. Design: Develop contractual agreements and/or memoranda of understanding (MOUs) with laboratories external to the water utility o Number and frequency of samples, by method o Initial and ongoing QC requirements o Data reporting requirements (including data elements to report, data reporting forms or transfer protocol, and results turnaround time) Address logistics with off-site laboratories o Sample transport approach (i.e., lab personnel, FedEx, contracted courier) o Establish a point-of-contact at the utility and at all support laboratories Implementation: Procure instrumentation, reagents, and supplies for any expanded in-house capabilities Sign contracts and/or MOUs with external laboratories Develop SOPs for new activities Conduct initial demonstration of capabilities for new methods used at utility laboratory and/or external laboratories Implement QA plan (addressed in Section 5.2.2) Start-up and Baseline Sampling and Analysis: Identify method performance problems or matrix interference/inhibition issues. For example, high levels of background organisms may impact pathogen method performance at a given location. Resolve startup issues at laboratories that will likely arise (QC issues, documentation completeness issues, sample transfer and receipt procedures and integrity issues, sample and data flow) Operation and Maintenance: Verify ongoing acceptable performance by laboratories using proficiency testing (PT) samples analyzed under established certification/accreditation/approval programs Track availability of new, validated methods for contaminants of interest Evaluation and Refinement: Assess ongoing acceptable method and laboratory performance on drinking water samples from the utility based on QC samples and PT samples May 2007 55 ------- Planning for WS-CWS Deployment Consider implementation of expanded analytical capability based on availability of new analytical methods 5.1.3 A vailable Tools and Resources The following tools and resources are available to support design and implementation of laboratory capability and capacity as part of a contamination warning system: American Association for Laboratory Accreditation. Contains a list of NELAC approved PT sample providers, http://www.a2la.org/dirsearchnew/ptproviders.cfm Association of Public Health Laboratories (APHL). Information provided to improve the capacity and capability of public health laboratories in their response to biological, chemical, and radiological threats, as well as other public health emergencies. http ://www .aphl .org/programs/emergency_preparedness CDC-Select Agent Program. Centers for Disease Control and Prevention regulates the possession, use, and transfer of select agents, available at: http://www.bt.cdc.gov EPA Analytical Methods for Drinking Water. Information on sources of methods as well as links to the various organizations which distribute them. Environmental Protection Agency, November 2006. http://www.epa.gov/OGWDW/methods/methods.html or http://www.epa.gov/waterscience/methods EPA Drinking Water Certification Program, http://www.epa.gov/safewater/labcert/index.html EPA Laboratory Compendium. Published laboratory analytical methods that are used by industries and municipalities to analyze the chemical and biological components of wastewater, drinking water, sediment, and other environmental samples that are required by regulations under the authority of the Clean Water Act (CWA) and the Safe Drinking Water Act (SDWA). Almost all of these methods are published as regulations at Title 40 of the Code of Federal Regulations (CFR). EPA Standardized Analytical Methods [SAM] for Environmental Restoration following Homeland Security Events REVISION 3.0, EPA/600/R-07/015 http: //www .epa. gov/nhsrc/ Laboratory Response Network (LRN). use this tool to identify laboratories close enough in proximity that could serve as contract laboratories to analyze samples potentially containing contaminants of interest, available at: http://www.bt.cdc.gov/lrn NELAC Institute. Contains a listing of NELAP accredited labs, http://www.nelac-institute.org/ National Environmental Methods Index (NEMI). Use NEMI to compare and contrast the performance and relative cost of analytical, text, and sampling methods for environmental monitoring. US Geological Survey, Environmental Protection Agency http: //www .nemi. gov/ Water Contaminant Information Tool (WCIT). Secure (password protected), on-line database that provides current, reliable information on chemical, biological, and radiological contaminants of concern for water security. Environmental Protection Agency, December 2006 http://www.epa.gov/wcit 5.2 Sampling and Analysis After target contaminants and support laboratories have been identified, the utility should develop and implement an approach for baseline, maintenance, and triggered sampling and analysis. This ultimately should be documented in a comprehensive sampling and analysis plan, as indicated in Section 5.2.2. However, this process can begin with consideration of the factors that should be documented in this plan, as discussed in Section 5.2.1. May 2007 56 ------- Planning for WS-CWS Deployment 5.2.1 Pre-design During the pre-design phase for sampling and analysis, the utility's design objectives should consider, but not be limited to: Sampling locations Sampling frequency Sampling procedures Sampling Location Considerations In addition to adapting the approach used for locating sensor stations (Section 4.1) to identify sampling points for baseline and maintenance monitoring, the following factors should be considered in selecting potential sampling locations: Proximity of location to utility (a sampling location situated further from the utility may have greater vulnerability, such as potentially decreased chlorine residual) Accessibility of location (ease of access for samplers to collect large samples and to gain access to sample collection location) Percentage of output from each plant in system (consider increased sampling of key locations that receive finished water from plants with a higher relative output) Age of the water being sampled (extremely aged water may potentially have a decreased chlorine residual, and/or increased potential for the presence of biofilms compared to water that is 1 day old) Age and composition of piping that finished water has passed through prior to being sampled Open source of finished water (an open reservoir of finished water could prove to be more vulnerable than an underground holding tank) Locations where backflow into the system could pose a threat Other key locations, such as fire stations, elevated tanks, and pump stations Other key locations as identified through assessments for online water quality (Section 4.1) and/or enhanced security monitoring (Section 6.1) Because triggered samples may come from anywhere in the distribution system, a primary goal of location selection for baseline monitoring should be to collect water from locations that is representative of as large a region of the distribution system as possible. The first set of analyses of a triggered sample will likely include the same methods and procedures that are used for baseline monitoring The utility also should plan to sample from key locations throughout the distribution system so that data can be compared to finished water data collected at the treatment plants. The utility may consider using the each source water treatment plant as a control by which to compare all data from baseline and triggered sampling events. The purpose of baseline sampling from multiple and diverse locations is to determine if the water is homogeneous with respect to contaminants detected, levels detected, frequency of detections and method performance. Data should only be pooled from multiple locations if it is scientifically justifiable to do so. This determination will involve statistical analysis of data. Sampling Frequency Considerations To support development of representative and complete baseline data, sampling frequency should consider the following: Size of the utility's distribution system and service community Flow rates through various parts of distribution system High pressure points within the system Seasonal affects (i.e., increased intake of surface water into plant due to rainy season or snow/ice melts) Changes in water source (if different sources are used during different times of the year) May 2007 57 ------- Planning for WS-CWS Deployment Frequency of customer complaints (i.e., increased customer complaints during stagnant, hot seasons) The utility may choose to collect samples more regularly while establishing a baseline for the potential contaminants, and then adjust the sampling schedule to sample sufficiently to maintain sampling and analysis response capabilities, continually update baseline data and to address seasonal or other issues that may influence change in baseline levels of contaminants that already are present in the system. Considerations for New or Modified Sampling and Analysis Procedures Some aspects of sampling and analysis for the contamination warning system will be different from those used routinely for compliance monitoring. The utility should consider the following factors when planning for these new procedures: Procedures, training, and equipment to concentrate large bulk samples for select pathogen and toxin analysis for ease of transport to the state public health/LRN laboratory (volumes up to 100 L may need to be collected at each sampling location). Additional safety equipment that may be needed to protect samplers from exposure to potential contaminants (e.g., goggles, gloves, face shields, laboratory coats). Procedures for ongoing field methods QC to aid in confirmation of site characterization determinations; this should include corrective actions for QC failures Procedures and training on proper chain of custody and evidentiary sample handling training to ensure that sample documentation is addressed properly from sample collection, shipping to method support lab, receipt at method support lab and throughout handling at the method support lab during analysis Training on collection, packaging, and transport/shipping procedures for drinking water samples that may contain disease causing agents or materials considered to be hazardous by commercial shippers Training of utility laboratory staff on new analytical capabilities. Procedures for ongoing analytical QC for new methods to assess data quality; this should include corrective actions for QC failures Procedures for data review for each new method (both those added to in-house laboratory capabilities as well as those performed by external laboratories) A response protocol in the event of a positive result 5.2.2 Design and Implementation Approach Design and implementation of the sampling and analysis activities determined through the pre-design process should include the technical considerations and specifications described below. Design: Develop a preliminary concept of operations that describes the process flow for routine sampling and analysis and establishes roles and responsibilities Establish sampling locations and a baseline monitoring sampling schedule based on pre-design considerations. Sampling design should strive to collect data to address both spatial differences ("snapshot" differences from samples collected over a short period of time) and longer term trends. Snapshot differences may be differences between sampling locations and treatment plants and between sampling locations when collected over a short interval (such as 1 month). Longer term trends (sampling location-specific trends and regional aggregate trends) is useful only when sampling continues over a period of time in which a trend may be anticipated. For water utilities, that period of time may be as long as 1 year to capture seasonal and operational changes. Analyze historical data for any contaminants of concern based on contamination warning system objectives to inform sampling and analysis plan Develop standard operating procedures for sampling that address the following: May 2007 58 ------- Planning for WS-CWS Deployment o Sample collection procedures, containers, and preservatives o Safety equipment use o Sample packaging and transport Develop or modify existing chain-of-custody, data reporting forms, or other utility forms needed to meet new sampling and analysis needs unique to the contamination warning system Identify field sampling equipment that should be procured Identify laboratory instruments that should be procured Develop list of field sampling supplies that should be procured and stocked for ongoing sampling Develop list of laboratory supplies that should be procured and stocked for ongoing analyses Develop a QA Project Plan that addresses the following: o Sampling QA/QC (e.g., type and frequency of field QC to include in sampling activities) o Analytical QA/QC o Data review procedures, and corrective actions in the event of QC failures. Implementation: Procure field sampling equipment Procure laboratory instruments and establish service agreements Procure field sampling supplies that should be procured and stocked for ongoing sampling Develop list of laboratory supplies that should be procured and stocked for ongoing analyses Conduct training on sampling procedures o Specialized procedures for sample collection o Safety considerations o Disease causing and hazardous materials packaging and shipping procedures Establish "notification levels" for each contaminant; further action will be needed if a contaminant is detected above this level. Until a baseline level of contaminants is established, these notification levels should be based on relevant data from studies conducted by the utility, as well as health advisory and other levels available through the sources listed in Section 5.2.3. After baseline sampling analysis is conducted, these notification levels should be adjusted. Consider the sources of data that can help establish a baseline of contaminant levels. Table 5-3 provides examples of sources to consider. Revise concept of operations based on design and implementation activities Table 5-3. Examples of Baseline Data Sources Baseline Data Historical data for specific contaminants and water quality from the following: 1. the point of entry to distribution system 2. within the distribution system 2 Method performance data at the treatment plant and throughout the distribution system for specific contaminants (spike recoveries, interferences, matrix effects). 3 Targeted contaminants detected at treatment plants and from individual sampling locations within the distribution system 4 Non-targeted contaminants that are detected at the treatment plants and within the distribution system 5 Tentatively identified compounds (TICs) that are detected at the treatment plants and within the distribution system. 6 Levels of contaminants detected (quantified and semi-quantified for targeted, non-targeted and TICs) at the treatment plant and individual locations 5 Frequency of detections of targeted and non-targeted contaminants, as well as tentatively identified compounds in treatment plant water, individual locations and multiple locations overtime 7 Contaminant specific control charts Trend charts Start-up and Baseline Sampling and Analysis: In general, a baseline monitoring program should proceed through phases of activity, culminating in the development of a maintenance monitoring program. Those phases are described below. May 2007 59 ------- Planning for WS-CWS Deployment o Phase 1: SOPs and necessary resource document development for critical activities related to baseline monitoring should be developed. Initial demonstrations of capability (IDC) and minimum reporting limits (MRLs) for each method and contaminant should be established. Data reporting requirements and protocols should be established. o Phase 2: Following the development of SOPs and completion and review of IDC data, finished water at the treatment plants is analyzed with respect to contaminant occurrence (contaminants detected, levels detected and frequency of detections) and method performance. Finished water from the treatment plant serves as a benchmark for comparison of contaminant occurrence and method performance of water from the distribution system. All future baseline and triggered sampling events may include the source treatment plant water as a control. o Phase 3: Regular surveillance monitoring of strategic/priority locations should be initiated and conducted at regular intervals to establish baseline for these locations and to determine if there are seasonal or regional trends. o Phase 4: A survey study should be performed to determine contaminant occurrence and method performance in the distribution system. Sample collection locations should be selected with the goal of achieving spatial coverage of the distribution system and to capture a wide range of conditions in water age, pressure zones and pipe material. This phase of study is designed to survey the distribution system for contaminant occurrence and method performance. The results from this survey study may result in the design of Phase 5 (focused studies). o Phase 5: Based on the results of Phases 2-4, short-term, focused studies may be conducted to look more closely at possible differences in contaminant occurrence or method performance within the distribution system. If significant differences are found, these findings may influence the selection locations for maintenance monitoring. o Phase 6: The final phase should be the analysis of results from Phases 1 - 5 to establish the management, interpretation and use of baseline data and to establish a maintenance monitoring program. Emphasis should be placed on access and interpretation of baseline data during a triggered sampling and analysis event. Document progress of sampling and maintain communication with laboratories to ensure contractual obligations are being met Consideration for baseline data should include storing the data in a manner that allows easy retrieval and use for interpretation of data from triggered sampling events. Consider adjusting sampling schedule after evaluating results from the first year of sampling and analysis Consider a shift from the baseline monitoring phase to the maintenance monitoring phase after a year of baseline sampling and analysis is complete Finalize concept of operations document to reflect results of baseline sampling and analysis and transition to maintenance monitoring Operation and Maintenance: Development of a maintenance monitoring program should consider results and data evaluation from the baseline monitoring period as well as cost considerations and dual benefit. For contaminants that fall under drinking water regulation, less frequent maintenance monitoring may be warranted, whereas, contaminants that may not otherwise be monitored under any program may be given priority for more frequent maintenance monitoring. Maintenance monitoring should strive to maintain capabilities and serve to collect data that may be used to update baseline data (Table 5-3). Maintain baseline data (i.e., tabular data, control charts, method performance by location) Perform periodic maintenance of sampling equipment, as necessary to maintain response capabilities Restock sampling equipment as necessary (bottles, cubitainers) or laboratory supplies (reagents or consumables) May 2007 60 ------- Planning for WS-CWS Deployment Evaluation and Refinement: Analyze baseline data to assist in determining when contaminant levels in triggered samples exceed baseline levels Evaluate ease of use of accessibility of baseline data during a triggered event Review target contaminant list and consider if new methods for additional contaminants are available and could be implemented Provide additional training to sampling teams and/or laboratory staff to address deficiencies observed during baseline/maintenance sample analyses (i.e., contaminated blanks, high false positives/negatives) 5.2.3 Available Tools and Resources The following tools and resources are available to support design and implementation of sampling and analysis as part of a contamination warning system: EPA Health Advisory Levels: http://www.epa.gov/waterscience/criteria/drinking EPA Region 3 Risk Based Concentrations: http://www.epa.gov/reg3hwmd/risk/human/index.html EPA Region 6 Human Health Medium-Specific Screening Levels (HHMSSLs): http: //www .epa. gov/earth 1 r6/6pd/rcra_c/pd-n/screen .htm EPA Region 9 Preliminary Remediation Goals (PRGs): http://www.epa.gov/region9/waste/sfund/prg/faq.htm EPA Response Protocol Toolbox (RPTB), Module 3 and 4, EPA-817-D-03-003, http://cfpub.epa.gov/safewater/watersecurity/publications.cfm EPA Water Training Opportunities, Workshops/Training, November 2006 http://www.epa.gov/water/training.html National Laboratory Training Network (NLTN). Dedicated to improving laboratory practice of public health significance through quality continuing education. http: //www .phppo .cdc. gov/nltn/ Training resources for packing and shipping etiologic agent samples and hazardous materials. Examples include: o http://www.ercweb.com/classes/ o http: //saf-t-pak .com/ The Water Quality Data Elements User Guide, http://acwi.gov/methods/ 5.3 Site Characterization and Field Screening Sections 5.1 and 5.2 address activities associated with laboratory-based analyses performed on samples transported from the field. These activities should be addressed both on a routine basis and during a response. Additional activitiessite characterization and field screeningshould be addressed in the field specifically for response. Site characterization is the process of collecting information at a site to support the evaluation of a drinking water contamination threat. This process may include site evaluation, sample collection, and field screening. Field screening involves rapid sample testing in the field to evaluate any potential safety, chemical, biological or radiochemical hazards present at the site and to provide the laboratory with preliminary information that may help focus their analytical activities. 5.3.1 Pre-design During the pre-design phase for site characterization and field screening, the utility's design objectives should consider, but not be limited to: May 2007 61 ------- Planning for WS-CWS Deployment The role that the utility will play in site characterization activities, versus roles by other organizations, such as law enforcement and hazardous materials response units Current site characterization expertise among these organizations, and coordination across these organizations, versus the expertise and coordination needed to fulfill each organization's role to fully and effectively evaluate a potential contaminant threat site Current field screening equipment and expertise, versus the equipment and expertise needed to rapidly test for contaminants in the field Site Characterization Details on the site characterization process for a potential water system contamination incident are provided in the EPA Response Protocol Toolbox (RPTB). Site characterization is intended to provide important information to guide activities not only of water utility managers and staff, but also external first responders (such as local law enforcement and HazMat teams) and other government agencies that may be involved (such as the FBI and the EPA's CID. Information gathered during site characterization is combined with other information to perform a threat evaluation, the results of which may feed back into additional site characterization activities. During pre-design, the utility should consider the following steps for addressing site characterization: Define the potential scope of site characterization activities, based on the RPTB, the utility's own emergency response planning materials, or other resources (see Section 5.3.3). Identify the organizations that should be involved in site characterization. Although this will depend on the scope of the incident, the objective of pre-design is to identify a comprehensive list of these organizations. Work with these organizations to map roles for each site characterization activity to the responsible organization. The same activity also may involve different organizations, depending on the contaminant (such as a toxic industrial chemical versus a chemical warfare agent). Work with these organizations to assess the level of training and expertise of each to fulfill their role and identify shortfalls that should be addressed. Through the consequence management process (see Section 9), identify exercises and other opportunities to test and maintain a high level of coordination among the organizations that may be involved in site characterization. Field Screening Although field screening is one of many potential site characterization activities, it merits some specific pre-design consideration because it relies on appropriate equipment and training to be effective. Two types of rapid testing should be considered: Testing of materials other than the water (safety screening) to determine whether the environment around the potential contamination site can be safely accessed (or accessed with appropriate personal protective equipment [PPE]) Testing of the water (water screening) or other media (e.g., contents of discarded containers or suspicious residues) to determine whether samples can be safely handled or transported to laboratory(ies) for analysis and to provide the laboratory with preliminary information on potential contamination to help focus subsequent laboratory analyses. Field screening capabilities should address the target parameters presented in Table 5-4. Although one goal of pre-design for field screening is to maximize the number or type of contaminants that can be tested in the field, there are practical considerations that may limit this, including equipment cost and initial and ongoing training time and cost. May 2007 62 ------- Planning for WS-CWS Deployment Table 5-4. Considerations for Contaminant Coverage for Field Screening Screening Type Safety Safety Water Water Water Water Water Water Target Parameter Radioactivity (alpha, beta, and gamma) VOC(PID), LEL, CO, H2S,O2 Cyanide Chlorine residual pH/ conductivity/ ORP Turbidity Chemical Warfare Agents (VX, sarin, etc.)1 Toxicity1 Considerations for Field Testing Equipment Geiger counters and scintillators, equipment that can distinguish gamma/beta from alpha/beta emissions Multi-meters used for confined space entry (photoionization detector with other meters) In-field cyanide detector (e.g., a colorimeter or spectrophotometer) In-field cyanide detector (e.g., a colorimeter or spectrophotometer) Electrode detector Turbidimeter; most measure light scattering M272 Water Testing Kit or similar commercial versions Commercial toxicity test kit Considerations for Training The level of sophistication and expense vary widely. Vendor training typically is required. Some HazMat units may be able to provide training Instrument manual and training videos can be sufficient; training from an experienced user is ideal Instrument manual and periodic QC samples are usually sufficient Instrument manual and QC samples are usually sufficient Instrument manual. None beyond normal utility procedures for these measurements Instrument manual and QC samples are usually sufficient Instrument manual and QC samples are usually sufficient Vendor training sometimes required Comments May be expanded to water testing with a special probe Detects chemicals in air Tests water for cyanide ion, but not combined forms Absence of residual chlorine may indicate a problem Abnormal pH or conductivity may indicate a problem High turbidity may indicate a problem May also detect some pesticides and common chemicals Should establish a baseline Note: M272 and toxicity testing kits are time-consuming to use. These kits are generally only used if initial screening is inconclusive, or if the situation indicates that these screening tests may be relevant. A candidate list of target contaminants can be developed from the information in Table 5-4 and the additional details on specific field screening equipment and capabilities available from the RPTB. 5.3.2 Design and Implementation Approach This section addresses considerations for design and implementation of site characterization and field screening capabilities. Develop a customized site characterization plan based on the circumstances of the threat warning that integrates with the concept of operations and consequence management plan (Section 2.1 and Section 9, respectively). This customized plan may be adapted from a generic site characterization plan or as part of a response to a specific contamination threat. It is impossible to predict every possible scenario, so it is best to specify example scenarios that each warrants a different level of response. The site characterization team uses the customized plan as the basis for reporting their observations/data at the investigation site. Develop a health and safety plan to address any concerns that may arise in the field o Appropriate PPE o Emergency call list of agencies and individuals that should be notified in an emergency (e.g., hospitals, HazMat, MEDTox, fire and police) May 2007 63 ------- Planning for WS-CWS Deployment o 40-hour OSHA training Select Site Characterization Team and Team Leader o Experience o Training o Availability o Anticipated level of response. Note that outside agency intervention may be necessary (e.g., HazMat). Develop basis for personal protective equipment and field screening equipment o Preliminary information from monitoring station, personnel, or customer complaint o Direct experience and availability of various equipment types (utility vs. HazMat team) o Suspected contaminant Select primary components of the sampling kit o Bottle types and preservatives o Number of containers needed o Labels, Chain-of-Custody o Shipping materials o Name and address of receiving laboratory Assess resources available for expanded efforts Implementation: Ensure equipment for use in the field investigation is available and ready o Instrumentation is pre-calibrated o Communication devices and power supplies are tested and operational o Equipment for sampling (e.g., coolers, bottles, preservatives) o Appropriate documentation (field logbooks, chain of custody, health and safety plans, standard operating procedures, etc.) Prepare lists of field trained individuals within the organization o Prepare call lists for expanded situations (police, fire, HAZMat, etc.) o Schedule training at required intervals Health & Safety Use of equipment, instrumentation, sampling procedures Shipping hazardous materials Procure necessary instrumentation o Schedule routine maintenance and calibration o Secure back-up parts, and consumables Prepare site investigation kits o Personnel protective equipment o Screening instrumentation o Sampling kits o Documentation Develop SOPs as necessary Develop working relationships with local fire, police and HAZMat personnel o Meet with other organizations (Police, Fire, HazMat, etc.) to agree upon each parties roles and responsibilities in an expanded situation Perform background Site Hazard Assessment at representative collection sites Start-up and Baseline: Site inspections to observe normal surroundings Establish initial and ongoing QC requirements Establish background levels of contaminants May 2007 64 ------- Planning for WS-CWS Deployment Operation and Maintenance: Maintenance and calibration of screening equipment as per manufacturer's requirements or specifications Periodic testing of instrumentation with independent measurements Evaluation and Refinement: Track and evaluate the development of new field equipment technologies Provide additional training to address deficiencies observed during site characterization 5.3.3 Available Tools and Resources The following tools and resources are available to support design and implementation of site characterization and field screening as part of a contamination warning system: EPA Response Protocol Toolbox (RPTB), Module 3, EPA-817-D-03-003, http://cfpub.epa.gov/safewater/watersecurity/publications.cfm Resources for Strategic Site Investigation and Monitoring, United States Office of Solid Waste and EPA 542-F-01-030b, Environmental Protection Emergency Response September 2001 Agency (5102G) http: //www. epa. gov/tio/ Improving Sampling, Analysis, and Data Management for Site Investigation and Cleanup United States Office of Solid Waste and EPA-542-F-01-030a Environmental Protection Emergency Response April 2001Agency (5102G) http://www.epa.gov/tio/ EPA Water and Wastewater Security Product Guide, September 2005, http://cfpub.epa.gov/safewater/watersecuritv/guide/index.cfm 5.4 Staffing and Cost Considerations Planning for the implementation of the sampling and analysis component of a contamination warning system will likely involve a more limited range of utility staff than other components, but consideration should be given to both routine (baseline and maintenance) and triggered sampling events, the latter of which may involve a wider array of utility staff. Cost factors will be driven by the analyses the utility laboratory will perform, the role of commercial laboratories, and the cost basis for support by state or local public health or other laboratories. 5.4.1 Staffing Table 5-5 offers a quick overview of which staff may be necessary to design and implement an expanded, multi-laboratory sampling and analysis program as part of a contamination warning system and during which phases of implementation these personnel may be needed. Table 5-5. Sampling and Analysis Staffing Considerations Division or Department Water Quality Information Technology Security/Risk Administration Implementation Stage PD X D X X X X I X X X X PT X O&M X X ER X Comments Involvement in all aspects of sampling and analysis activities Support to data transfer and management aspects of laboratory results Site characterization support Contracting vehicles or MOUs needed to access external laboratory assets May 2007 65 ------- Planning for WS-CWS Deployment 5.4.2 Cost Considerations This section presents a summary of the design and implementation considerations discussed above that may influence costs. This list also may include other factors that were encountered during implementation of the initial Water Security initiative pilot and could be overlooked during cost estimation in the absence of this experience. Although this list of cost considerations may not be exhaustive, these factors, at a minimum, should be considered when planning. Cost considerations for activities performed by the utility: Additional staff to support sampling or analysis activities Field screening equipment Laboratory instruments Service and preventative maintenance contracts for new instrumentation Sample collection containers Additional laboratory reagents, standards, and disposables Safety equipment (e.g., face shields, respirators, flammable cabinets) Additional costs for hazardous and/or biological waste disposable (e.g., procure an autoclave, contract waste disposal) Training: o Disease causing agent and hazardous materials shipping o Site safety (e.g., OSHA HAZWOPR) o New laboratory instruments o Field screening equipment Proficiency testing to maintain certification/accreditation or demonstrate proficiency for methods that are not covered under the certification program Laboratory information management system changes Initial and ongoing costs of special permits (e.g., CDC, USDA) that may be required Cost considerations for use of external laboratories: Costs that may be incurred based on a minimum vs. maximum number of samples analyzed (e.g., cost per sample for 10 samples vs. 100 samples) Additional costs that may be charged if samples are received after hours, exceed daily capacity, require additional analyses Additional costs for sample processing (e.g., concentration for pathogen samples) at the laboratory, rather than in the field Transport costs (e.g., courier, shipping costs) based on proximity to the support laboratory In the process of determining costs the utility should determine whether expanding in-house analytical capability is cost effective and sustainable. This will not be an option for some contaminants (such as analysis of select agents, which is restricted to CDC LRN laboratories). However, expanding in-house capabilities to analyze others may be more cost effective if this capability can not only address contamination warning system needs, but also enable the utility to shift some current compliance monitoring sample analyses from a commercial laboratory to the utility's in-house laboratory. May 2007 66 ------- Planning for WS-CWS Deployment Section 6.0: Enhanced Security Monitoring Enhanced security monitoring includes the systems, equipment, and procedures that detect and respond to security breaches at distribution system facilities such as pump stations, reservoirs and storage vessels that are vulnerable to contamination. The monitoring strategy includes detection by physical security systems such as alarms and cameras, witness accounts, notifications by perpetrators, media, and law enforcement, as well as associated response methods. A security breach is an unauthorized intrusion into a secured facility that may be discovered through direct observation, an alarm trigger, or signs of intrusion (cut locks, open doors, cut fences). Security alarms are a common threat warning for a utility but are often unintentionally caused by routine operation and maintenance activities. Actual security breaches usually are the result of criminal activity such as trespassing, vandalism, and theft, rather than attempts to contaminate the water. Under the contamination warning system model, enhanced security monitoring should be designed to help discriminate between security breach alarms and notifications that may be related to a contamination incident and those resulting from other activities. Table 6-1 summarizes design basis considerations for enhanced security monitoring. Table 6-1. Design Basis Considerations for Enhanced Security Monitoring at Selected Sites Design Objective Capability Contaminant Coverage Spatial Coverage Timeliness Reliability Sustainability Description Can detect an intrusion that may have provided the opportunity for introduction of any contaminant. Covers all contaminant classes. Limited to those elements of infrastructure for which physical security can be monitored. Function of the type of security monitoring system and the time to evaluate a security breach. Can be a reliable means of identifying an intrusion, especially when these breaches may involve contamination, such as in storage tanks and reservoirs. Provides utility with increased physical infrastructure protection and awareness. Reduces the occurrence of nuisance tampering. Design and Implementation Considerations System should be capable of detecting and assessing breaches that could provide access to the water supply at distribution system facilities where contaminant addition would impact a significant number of customers. System should detect and assess breaches at all possible entry points, at facilities selected for security enhancements, which could provide access to the water supply, regardless of contaminant quantity, type or method of injection. Improvements should be focused on distribution system facilities such as pump stations, wells, reservoirs, and storage tanks where a large volume of water could be contaminated and impact a significant number of users. Service connections and hydrants should generally not be considered due to the high number of nodes and the low benefit/cost ratio of hardening those appurtenances. System should be designed such that alarms produced allow responders to quickly make an assessment and generate the proper response. For many facilities, use of video to assess alarms may be critical. System should be designed such that video images and proper response procedures are used to minimize false alarms. System should utilize equipment that is robust, does not have substantial maintenance requirements, and does not produce frequent false alarms. The overall objective of this section is to describe considerations and a process for design and implementation of the enhanced security monitoring component of a contamination warning system to guide planning activities. In planning for implementation of enhanced security monitoring, several key design decisions should be made including the following: Facilities at which to install security enhancements Type of security enhancements for consideration (e.g., alarms, motion sensors, video) Prioritization framework for ranking sites and security enhancements May 2007 67 ------- Planning for WS-CWS Deployment Communications architecture for transmission of data and alarms Approach for implementation Approach for operation and maintenance A key objective of this section is to provide information to enable the reader to consider these decisions in a systematic process. The primary design element for enhanced security monitoring is physical security enhancements, along with integrated communications architecture and data management. Physical security enhancements should focus more heavily on detection and assessment of a potential contamination event at a facility and less on preventing the event from occurring due to challenges with and feasibility of implementing improvements to prevent an adversary from gaining access to a facility. The physical security enhancements should be designed in conjunction with the design of the communications architecture and data management to ensure the physical security system design objectives are met. Communications architecture for enhanced security monitoring should be closely coordinated with the same communications systems for online water quality monitoring since they often may share common systems. Unique considerations for enhanced security monitoring, particularly with regard to the transmission and storage of video data should be identified and considered in the development of a comprehensive communications and data management architecture for these components. The initial assessment process for enhanced security monitoring is one of the most critical aspects of this component of the contamination warning system. The process is similar to the approach taken by most water utilities in conducting their vulnerability assessments in response to the Public Health Security and Bioterrorism Response Act of 2002 as described in Instructions to Assist Community Water Systems in Complying with the Public Health Security and Bioterrorism Preparedness and Response Act of 2002 (USEPA, 2003). In contrast to the vulnerability assessment process which evaluated all possible threats to the entire water utility, a distribution system physical risk assessment focuses only on intentional distribution system water contamination. Findings from the previously completed vulnerability assessments may be utilized to identify the critical facilities within the distribution system. For some contamination warning system components, the dual benefit of detecting accidental or naturally occurring contamination events is possible. In the case of enhanced security monitoring, the dual benefit is the ability to detect all types of intrusions including those involving intentional contamination. Table 6-2 provides example preliminary recommendations for security improvements for typical water utility facilities. The actual recommendations should be utility-specific. The process for selecting the improvements is described in subsequent sections of this document. Table 6-2. Example Improvements by Water Utility Facility Type Facility Type Typical Recommended Improvements Construct a structure over vents to prevent addition of contaminants Contact switches and alarms for hatches and access points Finished Water Reservoirs and Ground Level , Harden hatches and covers for which alarms can not be feasibly added ^ - Cover or install barriers for overflow pipes that are vulnerable to contamination Develop written procedures for isolating tanks and reservoirs in the event of suspected contamination and provide training for these procedures May 2007 68 ------- Planning for WS-CWS Deployment Facility Type Pump Stations Elevated Storage Tanks Typical Recommended Improvements Interior motion detectors to detect intruders entering through windows and vents in areas that provide access to water pumps and pipes Contact switches for doors to detect intruders entering areas that provide access to water pumps and pipes Camera(s) that are activated by motion detectors or contact switches Card access reader Video and communication interfaces Lighting improvements for camera systems where needed Develop written procedures for isolating and turning off pumps in the event of suspected contamination and provide training for these procedures Interior motion detectors Audible alarms at facility Develop written procedures for isolating tanks and reservoirs in the event contamination and provide training for these procedures of suspected 6.1 Pre-design Pre-design and planning for enhanced security monitoring involves the following: Determine design basis threat Develop preliminary facility list Site assessments Perform risk ranking to assess risk before improvements To evaluate how well existing and future proposed security systems and procedures protect facilities from contamination, it is important to define the specific potential threats to those facilities. Before determining the effectiveness of protection systems, agreement should be reached defining types and capabilities of the adversaries who may attempt to contaminate the system. This is important because the effectiveness of a protection system can vary greatly depending upon the adversary. For example, a standard steel door with hinge protection should delay or defeat a vandal attempting to enter a facility, but a sophisticated, trained adversary such as a terrorist armed with the proper tools and equipment could defeat the door easily and quickly. The same variance in effectiveness often applies to detection and alarm systems. Of equal importance is defining the types and quantities of contaminants that may be used. In order to evaluate the contamination risk of each facility, the design basis threat should be defined. This includes identifying the capabilities of adversaries as well as the quantity and type of contaminants that could be used to contaminate the water supply. The recommended design basis threat is a highly sophisticated adversary or group of adversaries with the resources and ability to access all contaminants represented by the classes listed in Table 1-1. This type of adversary has the ability to gain expertise in the areas of distribution system design, operation, hydraulics, drinking water treatment, chemistry, microbiology, etc. The next step during pre-design and planning is to preliminarily identify which utility facilities may be at the highest risk for contamination based on factors including: consequence of contamination, site location, access, and visibility. Selection of these preliminary sites can be facilitated through review of previously conducted vulnerability assessments, which may have characterized relevant attributes of each facility even if the focus of the assessment was not on contamination. If a distribution system model is available, it could be used to estimate the consequences of contamination at various distribution system facilities. The process for estimating consequences using a distribution system model is described below. The preliminary list of facilities may be evaluated further through site assessments for potential enhanced security monitoring upgrades. May 2007 69 ------- Planning for WS-CWS Deployment After the preliminary facilities list has been developed, the next step is to conduct detailed site assessments of each facility on the list with a focus on existing physical security systems, communication capabilities (i.e., transmission of alarms, video, etc.), proximity to the public, terrain, adjacent land uses, site access, site lighting, alarm and detection systems, and physical barriers such as fencing and hardened structures. Using the observations made during the site assessments, an evaluation should be performed regarding the possible modes of entry for contaminants (i.e., dumping contaminants directly into reservoir, injecting into pipe tap, etc.) and volumes of contaminants that could practically be delivered to each of the facilities and put into the water without arousing suspicion. For example, the site configuration of some facilities may not allow a heavy truck to get close enough to the facility to deliver 1,000's of gallons of the contaminant, while other facilities may not be accessible at all by a vehicle in which an adversary may be limited to quantities that could be transported by hand. The contamination risk of each facility can be evaluated after facility assessments have been conducted and the design basis threat has been defined. The risk assessment provides the basis for prioritizing which facilities should be considered for security improvements and which security improvements would be most cost-effective in terms of ability to reduce risk of contamination. The contamination risk at a facility is a function of three primary parameters: Effectiveness of the facility's existing physical security system Probability that the facility may be targeted by an adversary Consequences of a contamination event at the facility There are several methods available to calculate risk. The method most commonly used during the 2002 vulnerability assessment process was the Risk Assessment Methodology for Water Utilities (RAM-WSM) developed by Sandia National Laboratory. A method very similar to RAM-WSM can be used to compare the risk of intentional contamination at distribution system facilities. Other risk assessment methods are available including VSAT (Vulnerability Self-Assessment Tool) developed by the National Association of Clean Water Agencies (NACWA). The effectiveness of a facility's existing physical security system can be assessed during the detailed site assessment. The considerations used to estimate the effectiveness of a security system include detection, delay, and response. Traditionally, physical security systems (i.e., door alarms, motion detectors, etc.), have been designed to detect a security breach early enough to provide adequate delay and allow sufficient time for law enforcement to respond and prevent the adversary from completing their intended act. However, for the case of potential contamination, the consequences can in some cases be quickly eliminated or significantly reduced through an operational response triggered by a security alarm or notification (i.e., isolating a finished water storage tank, shutting down pumps, etc.). Estimating the relative probability that an adversary may attack one of a utility's facilities compared to another may be somewhat difficult to assess. However, it is likely that some facilities may be more attractive targets compared to others. Some aspects of the facility that could be used to estimate the probability of attack include: RecognizabilityHow easy would it be to recognize a facility as a water utility facility that provides access to drinking water? Visibility to Surrounding PublicIf a facility is visible to the public living and working near the facility, an attack on that facility may be deterred due to the increased probability of the public witnessing the attack. However, a sophisticated adversary may not be deterred significantly from attacking a visible facility. Access to and Ability to Deliver ContaminantThis estimates how difficult it would be to get a contaminant to a facility and add it to the system. This is a function of the possible modes of entry for contaminants (i.e., dumping contaminants directly into reservoir, injecting into pipe tap), and volumes of contaminants that could practically be delivered to each of the facilities and put into the water without arousing suspicion. Also, some utilities may not provide vehicle access which would limit the amount of contaminant that could be delivered. May 2007 70 ------- Planning for WS-CWS Deployment Effectiveness of AttackThis estimates how effective a potential attack would be on a facility based upon the ability to add a sufficient amount of a contaminant to reach a lethal concentration in the distribution system. The consequences of a contamination event occurring at a facility ideally should be estimated using the utility's distribution system model and GIS to simulate how a contaminant would spread and how many people would be affected by the incident. The following parameters should be considered in a modeling analysis to estimate the resulting consequences of contamination at any distribution system facility: Volume of contaminant added (depends on contaminant and site characteristics) Concentration of contaminant added (depends on contaminant availability) Toxicity of contaminant (depends on contaminant type) Duration of contaminant addition (depends on site characteristics) Duration of model simulation (recommend minimum of 24 hours) Type of storage tank mixing modeled (i.e., completely mixed, plug flow, etc.) The results of the site assessment, described previously, can be used to define and constrain the contamination scenarios that are modeled (e.g., the volume of contaminant that can be delivered to a location, the duration of contaminant addition, etc.). Furthermore, additional understanding may be gained by varying parameters such as the time of day that the contaminant is introduced at the facility because as demand patterns vary over the course of a day, the system hydraulics may change, which may impact the total number of exposures significantly. The risk of contamination for each facility is calculated utilizing the selected risk assessment equation and the estimated values for physical security effectiveness, probability of an attack, and consequences of an attack. Once calculated, the facilities should be sorted in order of highest to lowest risk. Section 6.2 provides additional detail for designing security systems for the selected facilities. In addition, most of the physical security improvements have communication and data management requirements that should be considered during pre-design. A robust, reliable, and secure architecture should be developed. The same system developed in Section 4.3 for online water quality monitoring may in some cases be used for enhanced security monitoring. Utilities may decide, however, to have a dedicated architecture for enhanced security monitoring. Pre-design considerations unique to enhanced security monitoring include the following: Determination of requirements of the communication system needed to support the proposed network of physical security improvements. Consider communication requirements for the online water quality monitoring stations deployed as part of the contamination warning system, and evaluate the feasibility of using a single architecture to support both. Assessment of existing communication system architecture used to transmit data and commands between remote facilities and the utility central control location. Assess ability of existing system to accommodate the proposed enhanced security monitoring systems. Evaluate use of the existing data recording systems (e.g., SCAD A) for managing data from cameras, contact alarms and other remote devices. If existing communications systems and data recording systems are unable to meet the requirements, the utility desires to separate security data from process control SCADA systems, or the utility desires to transmit security data to locations (e.g., security guard station) outside the SCADA network, evaluate alternatives. Identify constraints on communication alternatives such as hilly terrain that may make radio communications cost prohibitive. Alternative communication methods include SCADA, Tl lines, digital cellular services, and private radio network. A potentially significant data management challenge for enhanced security monitoring could be the management of video data from remote sites. The ability to transmit video for intrusion alarm assessment can be the most demanding communications network requirement of the physical security improvements. The video transmission option that provides the best resolution with the quickest response is full streaming video over fiber optic lines. The installation of fiber optic lines to remote facilities is often necessary, making this option very costly. The other video May 2007 71 ------- Planning for WS-CWS Deployment options are various technologies to compress and package video clips for transmission. Transmission options such as Tl line and digital cellular services were described above. 6.2 Design and Implementation Approach Once ranked, the facilities are evaluated further to identify methods for reducing risk of contamination to the facilities. To reduce risk, physical security system effectiveness should be increased or the consequences or probability of attack reduced. To determine how security system effectiveness could be increased, conceptual design and associated cost estimates of security improvements should be completed. Table 6-2 showed a list of typically recommended improvements. These recommendations may, however, vary from utility to utility and may be dependent upon several factors that are unique to each utility. Methods to reduce risk by increasing physical security system effectiveness include increasing detection and assessment capabilities, improving delay, and improving responses to an alarm. Means to improve the utility's security systems should include capital improvements, such as surveillance and monitoring equipment and alarms, facility structural improvements, and procedure modifications. In addition, response can be improved by developing written procedures for isolating each facility in the event that contamination is suspected. It is recommended that these written procedures be developed and the procedures incorporated into standard operating procedures and consequence management training and drills. In addition to increasing the effectiveness of the physical protection systems, contamination risk can be reduced by decreasing the consequences of a contamination event. If cameras are installed, the video images can confirm that an alarm was caused by an intruder and the operator can take action to mitigate a potential contamination event, including disabling pumps, closing valves, or changing the hydraulic grade line to prevent the spread of a potential contaminant. It is recommended that detailed procedures like this be developed for each facility. In addition, a timely "do not drink" or "do not use" order given to the public would in many cases reduces the consequences of a contamination event. Procedures and guidance for issuing "do not drink" and "do not use" orders should be developed as part of the consequence management plan. Decreasing the probability of an attack, especially for a sophisticated adversary, may be difficult to achieve but some options are available. Removing or covering signs that identify the facilities as a utility facility could be considered. However, since the adversary may be sufficiently sophisticated to identify all facilities, removing or covering signs is not recommended. For most facilities, it likely will not be possible to make them more visible to the surrounding public to deter attacks. However there may be facilities where clearing brush or small trees or other Crime Prevention through Environmental Design (CPTED) strategies, as discussed in Interim Voluntary Security Guideancefor Water Utilities (AWWA, 2004), may be beneficial. Barriers could be added at some facilities to prevent vehicles that could be used to carry large quantities of contaminants from entering the site. However, these facilities would still be subject to attack with smaller quantities of the most potent contaminants. The effectiveness of barriers should be reviewed further during design, although the cost of an effective barrier at even one facility may limit the budget available to enhance security at other locations. To help prioritize improvements, the cost of improving security systems at each site can be evaluated in the context of the associated risk reduction that would result from making the recommended improvements. Based on the preliminary design, planning level capital costs should be calculated for each facility. Methods for calculating planning level capital costs are described in detail in Section 6.3.2. For each facility, the risk score following improvements is estimated and the difference between this score and the original risk score is calculated and the difference is expressed as risk reduction units (RRU). The costs to benefits of for each facility are expressed in terms of the capital costs of May 2007 72 ------- Planning for WS-CWS Deployment improvements per RRU ($/RRU). After developing the preliminary design for security improvements at each facility, the risk of contamination for each facility should be recalculated. The values of RRU are utilized in the facility prioritization process. If any modifications to the preliminary design concepts are made as a result of the cost analysis or facility prioritization process, the RRU should be recalculated accordingly. Once calculated, the list of facilities and associated cost benefit ratios should be sorted in order of increasing $/RRU. From this list, the facilities that may receive enhanced security monitoring improvements can be selected based on the ranking and available budget for enhanced security monitoring. Design: Establish goals for detection, delay and response that should help either prevent the adversary from successfully contaminating the water or mitigate the consequences of a contamination event by successfully detecting the adversary and minimizing the response time. Security systems should be selected that will allow these goals to be met. Coordinate system design with the concept of operations as described in Section 2.1 to help establish the procedure aspects of the security systems and define the requirements for equipment and data systems. Select, specify, and locate equipment type and required equipment features based on factors such as building configuration and design criteria. For example, for camera systems, the available line of site, required resolution and speed of data transmission must be considered. For motion detectors, the range of detection, sensitivity and potential facility obstructions must be considered. Once the design criteria and functional goals are defined, methods can be evaluated to achieve them. The following are features and equipment that may be included in an enhanced security monitoring system. Hardening - The focus of physical security improvements is on detection and assessment rather than delay. Consequently, significant hardening of building structures such as doors and windows is generally not recommended, but expenditures on facility alarms and cameras are recommended. Hardening of exterior reservoir access hatches and vents, however, is recommended where installing cameras or hatch/vent contact alarms is difficult or cost prohibitive. Concrete or metal enclosures around hatches or vents can physically hide and/or make it very difficult to add contaminates reservoirs through vents. o Vents - design to make contaminant addition difficult while still allowing ventilation o Hatches- add contact alarms if routing electrical conduit feasible or bury/hide Facility access control - Access control systems allow a means for employees to automatically disable alarms upon entering a facility and provide records of employee egress. o Numeric keypads o Electronic locks - replace lock cylinders on doors and controls access on a per-key basis using programmable keys o Coded credentials - proximity cards o Biometric devices - fingerprint and iris scanners Contact switches - Contact switches are recommended for all exterior doors or for interior doors providing access to areas such as pump rooms where an intruder could contaminate water. Exterior reservoir hatches and valve vault hatches that provide access to water piping should also have contact switches. Contact alarms may be discrete for specific entry location identification or daisy-chained to provide a general facility intrusion alarm. Cameras - Contact alarms, motion detectors, or changes in camera image fields may be used to activate cameras to assess the alarm. Pan-tilt-zoom cameras can cover more area than stationary cameras but should be able to quickly turn and focus to area of concern. Camera field of view and distance from object impact the ability to identify intruders. Adequate lighting is necessary for assessment. Other factors to consider are resolution and lenses. o Stationary o Pan-tilt-zoom May 2007 73 ------- Planning for WS-CWS Deployment o "Smart" cameras that recognize suspicious behavior and/or motion in a limited part of the view Lighting - Adequate lighting is extremely important when using cameras for identification purposes. Lamps should quickly illuminate to full brightness. Motion detectors - Motion detectors are used to detect intruders gaining access through windows, and storage tank ladders and standpipes. Interferences from shadows and other causes should be considered. o Microwave o Passive infrared (PIR) o Microwave - PIR - Dual technology minimizes false alarms on ladders. Glass break sensors - Glass break sensors may be used in lieu of or in combination with motion detectors. o Acoustic o Shock o Acoustic - Shock - Dual technology greatly reduces false alarms from background noises. Audible alarms - Audible alarms are mainly for deterrence. Impacts on neighbors may be a concern. Heavy duty conduit and integrity monitoring - Heavy duty conduit and integrity monitoring protect power and communications wiring. Obtain any required code reviews or approvals on the design especially relating to egress. Video system types - Full streaming video provides real time capabilities but is often not a viable option due to the cost of installing the communications network. Two video packaging technologies include compressed video clips and flash memory. Compressed video clips systems are sent in packages over SCADA systems and typically includes digital video recorders installed in the monitored facilities to store images for potential criminal investigation. Flash memory systems include a video server to convert analog video signals to digital images and a processor to provide intrusion event video packaging and transmission. Flash memory may capture intrusion events but lacks the large video storage capabilities of compressed video clips. Video transmission - Good video resolution and transmission speed are the key requirements for rapid assessment. o Tl lines - Determine capability and interface requirements. Firewalls may be an issue. o Private radio networks - Perform path measurements to locate repeater sites. o Digital cellular service - Digital cellular service is a relatively recent technology development and is not available in all areas of the United States. It requires coordination with the local wireless carrier to determine the best way to link data from remote facilities to the command center. Multiple technologies may be desirable or necessary in some applications. For example, existing Tl lines may be available for some facilities and would provide excellent video transmission and response. Since Tl lines are costly to install and may require relatively high monthly communication fees, alternative video transmission options should be considered for areas without Tl lines. Implementation: Delivery Options: Traditional, Design-build (develop 30 to 50% design documents and contractor completes design); hardware and equipment could be procured by the utility or supplied by the contractor as part of the construction contract. Pre-selection of eligible contractors Bid and award of contract Construction - inspection, contractor oversight, contract modifications Document security issues Select communications service provider (if applicable), and establish any necessary contractual relationships. May 2007 74 ------- Planning for WS-CWS Deployment Select an installer, and establish any necessary contractual relationships. Installer's experience with all the technologies needed to complete the work should be heavily weighted in their selection. Identify roles and responsibilities for procurement, installation, and testing of various components of the communications architecture. Work with service provider to get components installed and configured. Test communication pathways between enhanced security monitoring sites and the operations center. Develop location-specific installation specifications. Procure system components that may not be provided by the service provider. Install remote communications systems at selected sites. Refine concept of operations based on as-built design Training of security personnel regarding the new monitoring systems, how they should interface with them, and how they should respond to an alarm Preliminary Testing: Initial configuration and calibration of enhanced security monitoring equipment, including verification of the following: o Contact switches and glass break sensors activation and transmission o Cameras work in conjunction with contact alarms o Lighting works in conjunction with contact alarms and is adequate for video resolution o Video transmission provides the speed and resolution specified for a full assessment by bench testing Troubleshooting Contractor functional and performance testing Refine concept of operations, if necessary, based on any modifications Operation and Maintenance: In-house versus contracted O&M services Documentation including, as-built specifications, O&M manual, etc. Periodic maintenance and calibration, including consumables. Annual inspection and maintenance of contact switches, motion sensors, glass break sensors, cameras and other physical security equipment. Unexpected maintenance events Amortized replacement costs Evaluation and Refinement: Evaluate frequency and cause of false alarms Conduct drills and exercises to identify refinements to concept of operations and to assess and improve equipment performance and alarm response 6.3 Available Tools and Resources The following tools and resources are available to support the design, installation, operation, and evaluation of physical security improvement for the enhanced security monitoring component of a contamination warning system: AWWA. Interim Voluntary Security Guidance for Water Utilities, 2004. AWWA. Guidelines for the Physical Security of Water Utilities, 2006. Garcia, Mary Lynn. The Design and Evaluation of Physical Protection Systems, 2001. The Integrated Physical Security Handbook. Philpot and Einstein, Homeland Defense Journal. 2006. http://www.phvsicalsecuritvhandbook.org May 2007 75 ------- Planning for WS-CWS Deployment USEPA Water and Wastewater Security Product Guide. http://crpub.epa.gov/safewater/watersecurity/guide/ Sandia Corporation. Risk Assessment Methodology for Water Utilities (RAM-W), 2002. National Association of Clean Water Agencies (NACWA). Vulnerability Self-Assessment Toolfor Water & Wastewater Utilities (Version 3.2 Update), 2005. http://nacwa.org/pugs/index.cfm USEPA. 2003. Instructions to Assist Community Water Systems in Complying with the Public Health Security andBioterrorism Preparedness Response Act of 2002 (EPA 810-B-02-001). 6.4 Staffing and Cost Considerations Planning for the implementation of the enhanced security monitoring component of a contamination warning system requires involvement of a wide array of utility personnel and potentially contractor staff. Costs may be highly dependent on the utility's capabilities and intended enhancements. Therefore, the remainder of Section 6.3 illustrates the staffing considerations and cost factors that are recommended for consideration during project planning and pre-design. 6.4.1 Staffing As mentioned above, staffing considerations are critical to the successful implementation of a contamination warning system. Table 6-3 offers a quick overview of which staff may be necessary to design, implement, and operate an online water quality monitoring program as part of a contamination warning system and during which phases of implementation these personnel may be needed. Table 6-3. Enhanced Security Monitoring Staffing Considerations Division or Department Water Quality Information Technology Engineering and Planning Operations and/or Distribution Security/Administrative IT Implementation Stage PD X X X X X X D X X X X X X I X X X X X X PT X X X X X O&M X X X X E&R X X X X X X Comments Helps define monitoring system requirements Designs, implements, and manages all IT systems used to support communications and data management needs of enhanced security monitoring Review and approval of designs. Responsible for installation oversight and inspections. Support security system design. Provide input during design. Supports selection of distribution system security equipment and improvements. Lead in operations and maintenance of equipment. Support installation of security systems. Lead in selection of distribution system security equipment and improvements. May be responsible for monitoring and responding to alarms. Designs, implements, and manages all IT systems used to support enhanced security monitoring. PD = Pre-design; D = Design; I = Implementation; Maintenance; E&R = Evaluation and Refinement PT = Preliminary Testing; O&M = Operations and Building the team to implement this component of a contamination warning system should involve all divisions within the utility. Therefore, it is important to have senior leadership involved and invested during each stage to facilitate the resolution of cross-division issues. Several key personnel, like the IT manager and the Water Quality division manager, would ideally be members of this and all other component teams to facilitate the application of the system engineering principles outlined in Section 2.1. Such involvement can be a significant commitment of time and resources for these individuals, but the utility can reap substantial benefit in the long-term success of the system. Other team members' participation may be less demanding, but still critical, and may vary depending on the utility-specific gap between the initial conditions and the final planned capabilities of the contamination warning system. Furthermore, it may be useful to obtain the help of consultants and contractors to aid the utility in the May 2007 76 ------- Planning for WS-CWS Deployment implementation of an enhanced security monitoring system, especially given that the level of effort required to implement this component would go beyond the human resources at many utilities. Consultants and contractors can be critical partners during each stage of the process by providing component-specific technical knowledge and installation experience, and by delivering training on the new equipment, procedures and processes. 6.4.2 Cost Considerations This section presents a summary of the design and implementation considerations discussed above that may influence costs. This list also may include other factors that were encountered during implementation of the initial Water Security initiative pilot and could be overlooked during cost estimation in the absence of this experience. Although this list of cost considerations may not be exhaustive, these factors, at a minimum, should be considered when planning. Development of design objectives for both elements of the enhanced security monitoring component Assessment of existing security, communications, SCADA and IT systems Development of preliminary concept of operations for the enhanced security monitoring component, including decisions regarding design basis threat, facilities to be protected, monitoring equipment to be deployed and the approach for communicating and responding to alarms. Field verification of preliminary locations and development of installation specifications for selected locations Layout and design of security equipment and communications system components (for each site). Even if the enhancements are handled as a design-build, significant design work may be necessary to produce the preliminary drawings. Design of communications and data management architecture; testing of communication pathways and conducting radio survey if necessary Installation of security and communications equipment at each site Establishment of contract with communication service provider(s); procurement, installation and configuration of communications and data management hardware and software; depending on existing infrastructure and topography, laying fiber optic cable or erecting antennas may add significant cost. Initial equipment shakedown and training on operation, maintenance and alarm response Communications and IT Architecture: test of installed data management architecture and complete communication system Development of written documentation; scheduled and unscheduled security monitoring equipment maintenance, repair and upgrades Service fees for communications service provider(s); development of written documentation related to communications architecture; scheduled and unscheduled communication equipment maintenance, repair and upgrades Equipment upgrades due to improvements in sensing and or communication technology Drills and exercises Based on experience at the pilot utility, the most significant issues may likely be the capability of the existing communication system and the capital costs associated with the installation of the security monitoring equipment. Finally, the cost to maintain and upgrade security monitoring equipment can be significant, and should be considered early in the planning stage to ensure that the system implemented can be sustained. May 2007 77 ------- Planning for WS-CWS Deployment Section 7.0: Consumer Complaint Surveillance Located throughout a utility's distribution network, consumers can provide near real-time input regarding changes in water characteristics discernable through the senses. Consumers may detect contaminants with characteristics that impart an odor, taste, or visual change to the drinking water. Complaints from residential, commercial and industrial consumers are routinely reported to water utilities on a very timely basis. As such, consumer complaints may provide one of the earliest warnings of a possible contamination incident for contaminants in classes 1 through 5, if an effective system is in place to detect anomalous trends in complaints and quickly respond to them. Generally, utilities document reports of unusual water characteristics and use them to identify and address water quality problems. The procedures and systems used to handle these reports are commonly referred to as the utility's consumer complaint management system. As part of a contamination warning system, the complaint management system should extend beyond just managing complaints - the system should monitor the complaint handling process and identify when conditions could be indicative of a water quality problem. This can be achieved by identifying information within the consumer complaint management system that can serve as an indicator of a possible contamination event when monitored and compared against a threshold value. If the collective information indicates an anomalous pattern in water quality calls, the contamination warning system triggers an alarm, followed by further investigation. By expanding on existing systems that manage consumer complaints, consumer complaint surveillance provides dual-use benefits to the utility by enhancing its customer service as it captures early signs of a potential water quality issue. Table 7-1 summarizes the design basis for consumer complaint surveillance and provides considerations as to how these objectives impact design and implementation. Table 7-1. Design Basis Considerations for Consumer Complaint Surveillance Design Objective Capability Contaminant Coverage Spatial Coverage Timeliness Reliability Sustainability Description Can indicate the presence of a contaminant that significantly affects one or more aesthetic qualities of water. High detection potential for classes 1 and 2. Moderate detection potential for classes 3, 4, and 5. Entire service area for contaminants with detectable taste, color, or odor characteristics. Function of the time from exposures to consumer reporting, complaint categorization, assessment and investigation. A potentially reliable indicator for contaminants with detectable characteristics if a robust complaint reporting and tracking system is in place. Provides utility an opportunity to manage consumer information more effectively and can serve as a tool for enhanced consumer confidence. Design and Implementation Considerations Detection of aesthetic changes is an indirect measure of contamination. Contaminant coverage may also be a function of consumer education. In addition, population diversity may impact the ability to detect an aesthetic change in water quality. While consumer complaint surveillance can encompass the entire service area, it is important to consider how calls are handled outside the utility if multiple jurisdictions or municipalities are served. Time to detection is a function of the system and procedures used to detect and investigate an anomaly in the number, type, or distribution of water quality complaints. Reliability is a function of educated consumers and their ability to notice aesthetic changes and ultimately report these to the utility in a timely manner. It is also a function of the system used to detect anomalies. Enhancements to the consumer complaint management system should be integrated with routine job functions and should improve customer service/satisfaction and potentially day-to-day operations in a call center. The objective of this section is to assist in the planning for implementation of consumer complaint surveillance as part of a contamination warning system. Although many utilities currently implement some consumer complaint monitoring and management activities, typically these activities are not integrated in a manner to support contamination warning system objectives, such as the timely May 2007 78 ------- Planning for WS-CWS Deployment recognition of possible contamination. The consumer complaint surveillance component should provide utilities with a mechanism to enhance their current call management system and incorporate consumer call information with other contamination warning system monitoring and surveillance data. Design decisions to consider include determining how to: Educate consumers. Utility consumers who are aware of indicators of potential contamination, including suspicious activity and unusual water characteristics, and who are educated on how to report such indicators, are an invaluable tool in detection of a potential water quality incident. Capture all complaints. A "funnel" for collecting all water quality complaints into the consumer complaint surveillance system should exist. For example, a unified call center with a widely publicized telephone number in place to capture the largest percentage of potential complaints. In addition, procedures should be in place to capture complaints that are directed to other points inside the utility that are initially received by other agencies. In cases where customer calls are managed outside of the utility, priority should be given to tunneling water quality related calls to the utility in a timely and efficient manner. Electronically manage data. All water quality complaints should be entered into an electronic database as they are received and categorized by type. A complaint record is carried through the process with information being added to it as it is received or investigations are conducted, and duplicate data entry is minimized or eliminated. Each complaint should be tracked from receipt to closure and retained in a historical database. Automate and integrate data analysis. Once all of the information (water quality complaint data and location) is collected for a call, it can be evaluated based on utility known parameters. This can be accomplished using manual procedures already in place for review of complaint data or through an automated event detection system. An event detection system is an automated software and/or hardware system that analyzes data in near "real-time" for information that may be indicative of a contamination incident based on a utility pre-determined threshold. For example, the frequency or locations of water quality complaints could provide indication of possible contamination Establish procedures and protocols. Written standard operating procedures (SOPs) for every step in the water quality complaint handling process is an essential attribute of consumer complaint surveillance. These SOPs should facilitate effective and timely communications, including clear guidance regarding the decision process to determine appropriate response actions, such as field investigations and sampling and analysis. All personnel potentially involved in the consumer complaint management system should be trained in these procedures. Address additional training of personnel. Trained and dedicated personnel may be crucial to implementing a successful consumer complaint surveillance process. These people are the front line contacts who interact with the consumers and process their input. The experience and professionalism of the utility personnel are critical to timely and accurate recording of data, and their judgmentin conjunction with automated and integrated data analysismay be vital in assessing when a possible contamination incident has occurred. Achieve continuous system improvement. The consumer complaint management system should be evaluated routinely (at least every five years) for two reasons: 1) to gauge how well it is meeting the intended goals of consumer complaint surveillance as well as other benefits derived from enhancements to the consumer complaints management system and 2) is to identify technological innovations or procedural modifications that warrant changes to the system. Throughout the enhancement process, the utility should also be mindful that the efficacy of the consumer complaint surveillance component is contingent upon the occurrence of data collection, analysis and notification steps in a timeframe that allows for effective response actions to mitigate a water contamination incident. May 2007 79 ------- Planning for WS-CWS Deployment 7.1 Pre-Design Pre-design for the consumer complaint surveillance component of a contamination warning system involves completing an assessment and gap analysis of a utility's call center procedures and call management system. The purpose of the assessment and gap analysis is to determine to what extent the current consumer complaint management system procedures and data systems can be modified and recommend enhancements to consumer complaint management systems to meet the attributes of an effective consumer complaint surveillance component Consumer Complaint Surveillance Model The design of a consumer complaint surveillance component of a contamination warning system is based on a model of "Funnel, Filter and Focus" as described below: Funnel. All customer calls should be directed to one-point of contact within the utility Filter. Utility employees who routinely handle calls should be able to respond to billing, metering reading and general water quality concerns. Calls concerning more specific water quality concerns should be forwarded to other appropriate personnel Focus. Personnel with training and experience in water quality should gather in-depth information for the consumer and make a determination of the need for field sampling. Figure 7-1 illustrates this model. In cases where consumer calls are managed outside of the utility, a similar approach should be applied to manage water quality related calls. LJJ CC. LJJ Primary Source: Water Utility Call Center Secondary Source: ^ I ^ Secondary Other Agencies ^^^ I ^f Source: ^ * r Other Utility Departments Non Water Quality Related Calls Non Water Quality Related Calls (As determined by Call Center Staff) Water Quality Calls Related to System Operations (Main Breaks, ^~~ ^ Operations Staff / o o Water Quality Specialist analyzes remaining complaints for indications of possible contamination Figure 7-1. The Recommended Filter, Funnel, and Focus Approach to Customer Feedback Data Optimization for Utility-Managed Consumer Calls May 2007 80 ------- Planning for WS-CWS Deployment Assessment and Gap Analysis of Existing Consumer Complaint Management Activities Based on experience at the pilot utility, an assessment and gap analysis is an effective tool to assess the current consumer complaint management activities and identify enhancements that should achieve the objectives of a contamination warning system. Such an assessment involves a thorough investigation of the utility call center and its call handling procedures. The goal is for the consumer complaint surveillance team to develop an understanding of the entire call management system. During the assessment, the consumer complaint surveillance team gathers an accurate picture of the utility's call management system. Superimposing this picture onto the model of a consumer complaint surveillance component outlined in Figure 7-1, identifies the gaps where enhancements could bring the existing complaint management system into alignment with the model. Key considerations in the analysis include: Identification of consumer complaint surveillance data streams. The utility should identify existing data streams related to water quality calls and the systems currently used to process these calls (e.g., call and work management systems, if available). As the utility assesses the data upon which a consumer complaint surveillance system is to be built, it is important to consider the sources from which that data is to be derived. It may be necessary to consider how data is to be aggregated and integrated from such disparate sources as third-party call management system hosts, other organizations within the city/jurisdiction, or other sources. Establishing a trigger and process for event detection. Once the existing data steams are identified, the next step is to establish a trigger level(s) based on background consumer complaint call volumes at a particular time. It should be recognized that consumer complaint call volume may vary based on the day of the week or time of day. Depending on how complaints are captured the data analysis can be completed by breaking the data into complaint categories by day of the week and/or by month. This process should be a method for counting the frequency and spatial distribution of water quality related consumer complaints over time and notifying responsible stakeholders once the characteristics of these data surpass the established level. Ultimately the goal is to answer the question, when does the complaint frequency rise to a level that indicates a significant change in water quality has occurred. Automation can occur through simple frequency counts based on time and location(s) of water quality complaints or could be a more sophisticated algorithm, such as a programmed EDS which operates behind the scenes of the existing consumer complaints management system hardware and software data systems. Alarm notification and user interfaces. Once a consumer complaint surveillance trigger level is surpassed, the user should be notified of the event. The notification process should attempt to utilize existing hardware and software systems available to the extent possible, such as e-mail and/or text messaging. Multiple notification procedures may be necessary to ensure coverage for both business and non-business hours. Other user interfaces to consider are graphical information systems for display of alarm data. Linkage of data systems. It is critical to recognize during the assessment process that an effective call management system for consumer complaint surveillance should be able to link across associated data bases such as: call tracking, customer information and the utility's asset management systems. This ability should allow the utility to quickly identify the location of the complaint and facilitate geographical analysis of the complaint data. It would also provide the utility a significant dual-use benefit by facilitating tracking of all maintenance related issues from call receipt until work-order close-out. 7.2 Design and Implementation Approach The activities described in Section 7.1 describe a process for pre-design of a consumer complaint surveillance system, culminating in documentation of an overall framework and selection of a methodology for designing the system. The design and implementation of the consumer complaint May 2007 81 ------- Planning for WS-CWS Deployment surveillance component should be based on the outcomes of pre-design and may include the considerations described below. Design: Perform additional reviews of historical water quality consumer call data analysis (if available) Clarify requirements of software/hardware needs (including event detection system) Develop a detailed method for automating data collection and analysis Refine notification protocols (e.g., text message, e-mail, pager) Develop end-to-end test protocols to verify system functionality Develop a preliminary concept of operations to describe routine operation of the component and additional user-specific standard operating procedures as needed Create a work plan and schedule to meet the design and time requirements of the project Implementation: Install and test necessary hardware/software or other equipment for data gathering and automation Install and configure EDS tools Revise concept of operations as appropriate to reflect the routine operation of the component as- built Conduct staff training on consumer complaint surveillance tools, systems, and procedures Preliminary Testing: Integrate the consumer complaint surveillance concept of operations into routine operations Collect and evaluate additional baseline data used in event detection Refine EDS tool configuration Operation & Maintenance: Monitor and respond to EDS alarms in accordance with the concept of operations. Log any anomalies detected by the EDS tool(s). Reconfigure the EDS tool if performance is found to be unsatisfactory. Verify that EDS tools and supporting software are current with respect to vendor-provided updates, patches, etc. Conduct periodic staff training on concept of operations Evaluation & Refinement: Continuous documentation of performance during operation, e.g., false alarms, detection of true water quality anomalies, and dual-use benefits Through simulated events, periodically evaluate the overall performance of the operational consumer complaint surveillance component. If necessary, update procedures or EDS tools to optimize performance. Track and evaluate the development of new EDS tools 7.3 Available Tools and Resources The following tools and resources are available to support design and implementation of consumer complaint surveillance as part of a contamination warning system: Whelton, A., Dietrich, A.M., Gallagher, D.L, Roberson, A. "Using Customer Feedback for Improved Water Quality and Infrastructure Monitoring," submitted to Journal of the American Waterworks Association. (Dec. 2006). Dietrich, A.M. "Aesthetic issues for drinking water," Journal Water and Health, Volume 4, Supplemental 1 (2006). May 2007 82 ------- Planning for WS-CWS Deployment Whelton, A., Dietrich, A.M., Burlingame, G.A., Cooney, M.F., "Detecting Contaminated Drinking Water: Harnessing Consumer Complaints," submitted to J. Amer. Water Works Assoc. (Dec. 2006). Lauer, William C. "Water Quality Complaint Investigator's Field Guide." American Water Works Association (2004). 7.4 Staffing and Cost Considerations Planning for the implementation of the consumer complaint surveillance component of a contamination warning system requires involvement of a wide array of utility personnel and potentially contractor staff. Costs may vary based on the utility's existing capabilities and the extent of enhancements. Therefore, the remainder of Section 7.4 illustrates the staffing considerations and cost factors that are recommended for consideration during project planning. 7.4.1 Staffing Owing to the potential complexity of creating or modifying call management systems to accommodate the objectives of a contamination warning system, a variety of staff, both internal and external to the utility should be considered. Depending on in-house expertise, it may be necessary to consider engaging third- party contractors to modify existing systems. Table 7-2 provides an overview of the staff and resources that may be engaged to design, implement, and operate a consumer complaint surveillance component as part of a contamination warning system. Table 7-2. Consumer Complaint Surveillance Staffing Considerations Division / Department Water Quality IT Engineering Supply Distribution Administration (internal Call Center Outsourced Call Center City Call Center Phase PD X X X X X D X X X X X I X X X X X PT X X X X X X X O&M X X X X X X X ER X X X X X X X Comments Provide water quality information during investigations of water quality complaints Designs, implements, and manages all IT systems used to support call management system N/A Provide investigation support to water quality calls Provide investigation support to water quality calls Recognize water quality related complaints and pass on to Water Quality Personnel Provide call center support Provide call center support PD = Pre-design; D = Design; I = Implementation; Maintenance; E&R = Evaluation and Refinement PT = Preliminary Testing; O&M = Operations and Building the team to implement this component of a contamination warning system may involve man divisions within the utility. Therefore, it is important to have senior leadership involved and invested during each stage to facilitate the resolution of cross-division issues. Several key personnel, such as the IT manager and the water quality division manager or equivalent, would ideally be members of this and all other component teams to facilitate the application of the system engineering principles outlined in Section 2.1. Such involvement can be a significant commitment of time and resources for these individuals, but the utility can reap substantial benefit in the long-term success of the system. Other team members' participation may be less demanding, but still critical, and may vary depending on the utility- specific gap between the initial conditions and the final planned capabilities of the contamination warning system. If the call center is outsourced, the utility should incorporate the requirements of the consumer complaint surveillance component of the contamination warning system into the contract with the call center contractor to the extent possible. May 2007 83 ------- Planning for WS-CWS Deployment 7.4.2 Cost Considerations This section presents a summary of the design and implementation considerations discussed above that may influence costs. This list also may include other factors that were encountered during implementation of the initial Water Security initiative pilot and could be overlooked during cost estimation in the absence of this experience. Although this list of cost considerations may not be exhaustive, these factors, at a minimum, should be considered when planning. Baseline assessment costs may include inspection and formal assessment report development by a project team to document the utility's existing call management system hardware, software and business management practices and to assess current capabilities to meet the contamination warning system objectives. Concept of operations costs include document development to specify the utility process and information systems to be used during routine operation and initial trigger validation of the consumer complaint surveillance component. Modification of existing call management software or the assessment and procurement of new call management software to meet the contamination warning system objectives for this component. Assessment of existing consumer complaint data streams to determine an approximation of event trigger level and applicability of statistical algorithms. The cost may also include coding utility- specific automated analysis tools along with the consumer complaints management system data extraction and transformation for data processing. Design and testing of consumer complaint surveillance alarm notification. This process may require the procurement of new hardware and/or software to ensure the timely display of alarms. Additional cost may also be incurred to incorporate and test GIS display of possible consumer complaint surveillance water contamination alarms, if this capability is desired. Deployment and testing of automated analysis tools along with the consumer complaints management system data extraction and transformation for data processing. Training on enhancements and concept of operations. Testing of existing call management software or new call management software to verify operation is consistent with the design. Software and hardware upgrades for call management system, event detection tools and alarm notification system Periodic evaluation and recalibration of consumer complaint trigger values, based on a review of historic data. Refinement efforts could also focus on a review of the effectiveness of the concept of operations. Conducting a consumer complaint surveillance evaluation drill. Consider one drill per year to include all personnel identified within the concept of operations for this component. May 2007 84 ------- Planning for WS-CWS Deployment Section 8.0: Public Health Surveillance Public health surveillance systems gather and analyze health-related data to identify anomalies that might indicate unusual incidence of disease. The role of public health surveillance in a contamination warning system is to gather and analyze data for investigation that will augment traditional epidemiological surveillance (which often relies on an astute clinician to notice and report anomalies). When anomalies are detected by public health analysis, coordinating with the utility will assist in determining whether the anomaly is related to water. The involvement of public health experts in a contamination warning system also adds a unique area of expertise that can not only support this component, but also consequence management. Some public health data, such as OTC drug sales and emergency room visits, may be suited to detecting biological contaminants found in contaminant classes 10 and 11 (Table 1-1). Other data, such as EMS records, 911 call data, and Poison Control Center data may be better at detecting fast-acting chemicals, like those found in contaminant classes 1-9. Public health surveillance is performed by many entities, including local health departments, 911 call centers, Poison Control Centers, and nationwide surveillance systems, such as the National Retail Data Monitor, which gathers and analyzes OTC data. Depending on the utility's area of service, there could be numerous local health departments (the locality could include city, county, or regional departments). Each of these entities is responsible for different areas, and some parts of the utility service area may not be covered by all of the public health partners. For example, a county health department may cover the entire utility service area, but 911 call data may be limited to one designated public service answering point (such as one city within the service area). Because of this, coordination with as many entities as feasible is needed to cover as much of the service area population as possible. Public health surveillance systems that would be appropriate for a contamination warning system can be grouped into two categories: Traditional surveillance systems include hospital disease reporting and laboratory reports. These are generally collected by the health department(s) and analysis is performed using morbidity rates (e.g., the number of measles cases per county per year). However, collection of these data can be relatively slow, with a lag time of days or weeks. Therefore, these data may be more useful for follow up investigation and false-alarm evaluation, but other more timely approaches to data collection and analysis should be considered. Syndromic surveillance aims to use any data available in as near real-time as possible to detect possible outbreaks based on statistical analysis of "syndromes," or categories of disease. This surveillance approach may be more useful for quick detection of the contaminants of concern. Syndromic surveillance can also involve performing "fused analysis," whereby information from many sources is analyzed together to detect possible anomalies. Some examples of syndromic surveillance programs currently being used by health departments include BioSense, the National Retail Data Monitor (NRDM), RODS and ESSENCE (see Section 8.3 for additional detail). The overall objective of this section is to describe options and a process to develop a design and implementation strategy for the public health surveillance component of a contamination warning system, based on existing capabilities in the area where the utility is located. This strategy will help local public health identify routine health changes more quickly and aid in cooperation with the utility to determine whether a possible water contamination is the cause of the health anomaly. Design basis considerations to achieve successful public health surveillance are summarized in Table 8-1. May 2007 85 ------- Planning for WS-CWS Deployment Table 8-1. Design Basis Considerations for Public Health Surveillance Design Objective Capability Contaminant Coverage Spatial Coverage Timeliness Reliability Sustainability Description Can detect the presence of a symptom or illness in a population which may be the result of the presence of a disease causing agent. May be able to identify the contaminant through clinical diagnosis/ testing. Covers contaminant classes 2 through 1 1 ; detection potential varies with type of surveillance. Comprehensive coverage of a particular city or county, which may include all, or a large portion of, the utility service area. Function of the time from the initial exposures, the onset of symptoms, and the point at which public health officials recognize the incident as a potential water- borne illness. May be a reliable means of identifying the incidence of illness in a population, but timing of communication between drinking water and public health officials should be optimized such that appropriate response, actions could be implemented in time to reduce consequences. Provides an opportunity for collaboration between utility and local health department(s). Design and Implementation Considerations Consider appropriate algorithms and the concept of fused analysis to look at many results concurrently (Burkam, et. al). Include public health data streams for both fast-acting chemicals and biological contaminants. Should include participation of numerous agencies and partners; any one type of data may not cover the entire service area. Where possible, automation of data collection and analysis as well as alerts will increase timeliness. New data streams such as 91 1 or EMS data should be considered. Develop a well defined communications protocol and Concept of Operations, including role descriptions and a firm commitment to investigation. Should be able to be performed without compromising other roles of public health agencies. 8.1 Pre-design Design of the public health surveillance component depends primarily on existing public health surveillance systems within the utility's service area and will vary greatly between utilities. Integration of public health surveillance in the contamination warning system will involve relationship building as much as analytical implementations. Appropriate roles, limits of information sharing, and how investigations will proceed should be addressed, with the end goal of communicating and responding to possible events more effectively. The pre-design process for implementation of public health surveillance in a contamination warning system should consider the following: Identification of local public health partners Identification and assessment of existing capability Development of a framework for communication and notification Identification of Local Public Health Partners The first step in the pre-design process is to identify current relationships between the utility and public health partners. If there is already a high degree of cooperation and coordination between the utility and local health partners, an expansion of these relationships should be considered to include a contamination warning system. If no relationship exists, points of contact should be identified for those who can effectively share information and participate in joint investigations of possible contamination events. A point of contact in the local health department(s) is particularly important. In addition to local public health departments, it is also important to engage the local or regional poison control center, local fire departments, dispatch centers, and in some cases State health departments as well. Because the contamination warning system will engage agencies that may, under other circumstances, have little need to interact, an effective way of communicating information should be established early in the planning process. May 2007 86 ------- Planning for WS-CWS Deployment Identification and Assessment of Existing Surveillance Capability Data streams that could be available for public health surveillance should be identified with the help of local health departments, fire departments, Poison Control Centers, and other partners. These data streams may include EMS, 911, laboratory tests, hospital data, Poison Control Center calls, and OTC data. Surveillance capability may vary from jurisdiction to jurisdiction in terms of degree of automation, type of surveillance, and area covered. It is important to assess the capability of the existing surveillance systems relative to the contamination warning system objectives of contaminant coverage, spatial coverage, timeliness (degree of automation), and reliability. The relationships between possible public health data and the design objectives of spatial coverage, timeliness, and contaminant coverage are shown in Figure 8-1. It is important to note that, while automation of data analysis is a desired goal of a contamination warning system, this may not always be the most cost-effective means of improving surveillance capabilities. Response times and communication protocols also may be improved significantly by evaluating and optimizing manual processes and procedures. The costs, organizational feasibility, and enhanced surveillance capabilities yielded by these improvements should be weighed against the costs and benefits of automated data analysis options. Consideration should also be given to the ability to detect both fast acting contaminants and contaminants with a long latency period. SPATIAL TIMELINES OF DATA SOURCE COVERAGE DATA LOCATION COLLECTION f5Tj5fBil City and/or County Near Real-Time TO^-lmr3 Department EMS 911 City and/or County ^^ NeaMTme JWl ""S'0"81 Poison Control Near Real-Time ^53s>i Center ^Poison Wdfc Civ/County/ w*w Regional t^v*-, City/County/ ^ Regional LPH Days or Weeks Hosp tats INITIAL CONTAMINANT DETECTED Fast-Acting Chemicals (Classes 1-9) Fast-Acting Chemicals (Classes 1-9) Fast-Acting Chemicals (Classes 1-9) Pathogens (Classes 10-11) Pathogens (Classes 10-11) Figure 8-1 Public Health Data Sources and Design Objectives At a minimum, data should be applicable to the detection of biological and chemical contaminants though either formal diagnosis or results (i.e., a laboratory test) or syndromic surveillance (i.e., OTC data profile). Especially important to consider is whether the data collection process can be automated, which can decrease time and effort of collection and analysis and thereby increase sustainability. Not only will this be helpful as part of the contamination warning system, but also will benefit the local health department in performing faster epidemiological analyses. In the absence of data automation, other approaches for achieving the design objectives should be considered. These approaches could include the following: May 2007 87 ------- Planning for WS-CWS Deployment Increased manual surveillance activities through increased staff to increase frequency of data monitoring Increased breadth and scope of manual surveillance activities Working cooperatively with local health departments to fund and add staff positions to their operations dedicated to surveillance activities focused on waterborne contamination Because the data streams should be complementary to each other and to data available at the utility, possible anomalies may be quickly verified or discounted. For example, a high volume of 911 calls could be compared with EMS run volume and run location to verify whether the high volume of 911 calls indicates a real event, or is just due to chance. Data should also be representative of the entire utility service area; if a utility serves multiple counties, then data from each county should be utilized. Public health data under consideration also should be compared with utility data to see where they might supplement each other. An example would be mapping 911 call data against consumer complaints calls to provide an investigative starting point. Development of a Framework for Communication and Notification Utility and public health partners should evaluate any current notification protocols. A call list, phone tree, or other established contact order may already exist. If a protocol exists, the organizations should determine whether the current protocol is appropriate for a contamination warning system, and then modified and/or updated into a preliminary concept of operations, as discussed in Section 2.0. If there is no communication protocol available, it should be created. The communication protocol should involve personnel who understand the concepts of a contamination warning system, can interpret data presented to them in relation to possible contamination events, and will know how to proceed with the investigation. Where possible, automation of alerts, such as e-mail alerts generated by an analysis program and sent to predefined recipients, should be considered. Roles also may need further refinement to clearly define who will be doing what at each stage. This will ensure not only that every investigative duty is performed, but also that efforts are not duplicated. Identifying who does what also will help effectively use the expertise of public health officials. As part of the assignment of roles and responsibilities, a strong commitment to follow-through on these roles and responsibilities should be emphasized. When developing a communication protocol, issues related to the Health Insurance Portability and Accountability Act (HIPAA) should be considered. This law recognizes that advances in electronic technology could erode the privacy of health information and mandates privacy protections for individually identifiable health information. Health officials have access to data that, due to HIPAA limitations or other constraints, are not usually available to utilities. Information typically protected under HIPAA includes any information that can be used to identify an individual, and how this is interpreted may vary by jurisdiction. For example, public health officials may determine that age, sex, and address, data cannot all be included, as this information could identify a patient even though the "Patient ID" field is not provided. However, communication between the utility and local public health agency(ies) is essential, and data sharing should be conducted to effectively investigate anomalies. For example, based on discussions among the utility and local health departments participating in EPA's initial contamination warning system pilot, the following data elements were used for analysis: Patient location (zip code or address for EMS and 911 data, respectively) Event date and time Chief complaint (used to categorize into syndromes). May 2007 88 ------- Planning for WS-CWS Deployment Care should be taken with access to data presented in a User Interface to ensure they are compliant with HIPAA. More information on HIPAA and compliance with can be found at the Department of Health and Human Services website (http://www.hhs.gov/ocr/hipaa/) At the conclusion of the pre-design stage, the utility should have an understanding of the public health surveillance tools within the utility's service area and approach for engaging and coordinating with public health partners to leverage or adapt these tools to meet the contamination warning system design objectives. This approach should stress cooperation among public health organizations and the utility to improve existing data surveillance, event detection, and communication practices without compromising current public health or utility services. 8.2 Design and Implementation Approach The activities discussed in Section 8.1 describe design factors that should be considered when designing and implementing a public health surveillance component within a contamination warning system. Design: Develop a preliminary concept of operations to describe the process flow, data streams, and roles and responsibilities Establish how the data should be efficiently gathered o Assess methods for automating data and/or adding data streams, including data that may not have been electronically captured previously (e.g., 911 and/or EMS data) o If automation is not feasible and/or cost effective, consider alternative methods of providing public health support to the contamination warning system (e.g., increased frequency or scope of manual surveillance activities or staff support to local health departments to focus on waterborne contamination) o Consider participation in BioSense, RODS, ESSENCE, or another syndromic surveillance program to improve data collection capabilities, and determine if the area is covered by the NRDM In collaboration with local health departments, research and identify appropriate analysis tools o Consider methods that can analyze data by time (i.e., counts per day) and by location (i.e., counts in a spatial cluster) in as near real-time as possible o Research available literature to determine which algorithm(s) are appropriate for the data. Possible analysis tools include regression models, multivariate models (e.g., CUSUM), and spatial analysis o Consider a fused analysis approach Establish a means for displaying data in a HIPAA-compliant way in one user-friendly location, such as a User Interface on a centrally-accessible website. Implementation: Optimize data collection and analysis procedures and protocols through implementation of refined procedures or installation of software and equipment to support automation o IT mapping of data on servers o Computer code for automating and optimizing datasets Install analysis tools decided upon in design stage o Software to run statistical algorithms o Computer code for automating analysis o Display of results on a user-friendly interface Hire additional staff that may have been identified in the design stage Conduct training on analysis tools and interpretation of results, particularly for new analysis tools Hold meetings between utility and public health agency(ies) to educate on roles and responsibilities Participate in table-top and other communication exercises Implement data agreements, where necessary (e.g., a contract with the Poison Control Center) May 2007 89 ------- Planning for WS-CWS Deployment Preliminary Testing: Based on analysis of historical data, determine when an anomaly requires investigation relative to contamination warning system objectives. This "alert level" may be revised based on results from data generated during the preliminary testing stage. Determine acceptable false-alarm levels (How often is it acceptable to investigate a result that is not a true anomaly?) Adjust roles and communication protocols based on results from table-top exercises and other exercises Revise concept of operations as appropriate based on design and implementation activities and results from preliminary testing activities Operation and Maintenance: Use communication protocol for investigation o Contact appropriate people during anomaly investigation o Share information agreed upon during pre-design stages o Update contact lists as necessary Continue data gathering and analysis Perform data maintenance activities as necessary. o Software upgrades o Archive old records Evaluation and Refinement: Evaluate ease of interpretation and usefulness of data, including sensitivity (false-alarm) measurements and data quality issues. Determine how well communication protocol is working, and consider methods of improvement. o Consider further automation of alerts, such as email or text message o Consider improvements to any manual data-gathering methods o Consider information shared and if it is useful Adjust alarm thresholds as necessary, based on sensitivity measurements. Adjust data gathering frequency as necessary. This could also include automation of data not previously automated. 8.3 Available Tools and Resources The following resources are available to assist in design and implementation of the public health surveillance component: Early Aberration Reporting System (EARS). EARS is a software program developed by the CDC for the purpose of syndromic surveillance. Versions of the program can be run using either SAS or EARS. Public health data can be run through EARS to detect possible outbreaks or bioterrorism events. Analysis capabilities include categorization of symptoms into syndromes using text search string functions, aberration detection using CUSUM methods (Cl-Mild, C2-Medium, and C3-Ultra), and graphic representation of the analysis using graphs and maps, http://www.bt.cdc.gov/surveillance/ears/ Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE). Developed by the Department of Defense and Johns Hopkins University Applied Physics Laboratory, ESSENCE aims to collect and analyze a variety of data sources for the early recognition of abnormal community disease patterns that could result from natural causes or terrorist activities. ESSENCE uses data from military and civilian databases of patient visits, OTC sales, chief complaint data from Emergency Rooms, 911 calls, Poison Control Center Calls, laboratory records, as well as weather and community events to perform a fused analysis using spatial and temporal algorithms. http://www.geis.fhp.osd.mil/GEIS/SurveillanceActivities/ESSENCE/ESSENCE.asp May 2007 90 ------- Planning for WS-CWS Deployment Real-time Outbreak and Disease Surveillance (RODS). RODS was originally developed by the University of Pittsburgh in collaboration with Carnegie Mellon to provide a computer-based public health surveillance system for the early detection of disease outbreaks. RODS looks at emergency room and OTC data into one user interface; it collects hospital data in near real-time and analyzes it using Recursive Least-Square and What's Strange About Recent Events algorithms. It has been implemented at health departments in numerous states, http://rods.health.pitt.edu/ National Retail Data Monitor (NRDM). The NRDM was also developed by the University of Pittsburgh in collaboration with Carnegie Mellon to gather and monitor information on OTC drug sales for possible outbreak detection. This data is collected for use in the RODS analysis tool. The data is gathered from over 20,000 retail stores throughout the country, and is available to public health departments free of charge. http://rods.health.pitt.edu/NRDM.htm BioSense. BioSense is the national program designed to improve the nation's capabilities for real-time bio-surveillance and situational awareness headed by the CDC. It operates using an Internet-accessible system that allows users to visualize information about public health trends from early detection data sources, using advanced algorithms to analyze the data and provide a nationwide, real-time picture. BioSense is also discussed in Section 1.0 of this document, www.cdc.gov/biosense/ FirstWatch. FirstWatch is a software program designed to integrate and analyze data from various health and public safety sources (i.e., 911 call, police dispatches, etc.) FirstWatch displays many data streams on one user interface and performs analysis using predetermined thresholds. Automation for analysis is already built into the program. Its use has been growing, and is now used in some capacity by over 50 cities in the U.S. and Canada, www.firstwatch.net SaTScan. _SatScan is a statistical program developed for the purpose of cluster analysis using both spatial and temporal methods. It includes models for case-control (Bernoulli model), Poisson based population, space-time permutation, ordinal, exponential, and normal distribution analysis. Information about the significant clusters can be output to files for plotting using a separate GIS system. SatScan has been used extensively in recent years for syndromic surveillance purposes, www.satscan.org HIPAA. The Health Insurance Portability and Accountability Act was passed in 1996 to protect the privacy of patient medical data by limiting who can have access to individually identifiable medical data. http: //www .hhs. gov/ocr/hipaa/ 8.4 Staffing and Cost Considerations In planning the public health surveillance component of a contamination warning system, it is essential to incorporate staff from a wide range of partners, some of which may have no previous experience with public health data collection and investigation. Cost factors will vary substantially, depending on how many partners participate, and their capabilities. 8.4.1 Staffing During planning, input from numerous entities should be gathered, including local health department epidemiologists, fire departments/EMS crew, 911 dispatch operators, Poison Control Centers, as well as the utility. Each of these can provide unique input on advantages and disadvantages of proposed public health surveillance systems. These agencies should be engaged early in the planning process, such that their effort on a day to day basis (i.e., during operation and maintenance) is minimized. At later stages of operation and maintenance, there should at a minimum be adequate staff to interpret any alert generated by the public health event detection component and successfully communicate details of the alert to appropriate utility staff for investigation. Where appropriate, additional staff dedicated to manual surveillance activities could be considered in place of automated data streams. Table 8-2 provides a summary of staffing considerations for the public health surveillance component. May 2007 91 ------- Planning for WS-CWS Deployment Table 8-2. Public Health Surveillance Staffing Considerations Division or Department Implementation Phase PD D I PT O&M E&R Comments Utility Project Manager Water Quality X X X X X X Provide investigation support to public health alerts Provide investigation support to public health alerts Public Health Partners Epidemiologist Fire /EMS 911 Poison Control Center X X X X X X X X X X X X X X X X X X X May include one or more epidemiologist(s); will be the main person performing surveillance of data streams PD and D activities include identification of possible data streams; O&M activities will mainly be maintenance PD &D activities include identification of possible data streams; O&M activities will mainly be maintenance Roles may be based on service agreement with the poison control center; may include data sharing and toxicological expertise Other Support Partners IT Support X X X X X X May be managed at the city or regional level PD = Pre-design; D = Design; I = Implementation; PT = Preliminary Testing; O&M = Operations and Maintenance; E&R = Evaluation and Refinement 8.4.2 Cost Considerations This section presents a summary of the design and implementation considerations discussed above that may influence costs. This list also may include other factors that were encountered during implementation of the initial Water Security initiative pilot and could be overlooked during cost estimation in the absence of this experience. Although this list of cost considerations may not be exhaustive, these factors, at a minimum, should be considered when planning. Costs associated with the incorporation of public health surveillance in a contamination warning system will be based heavily on the current status of public health data systems and existing relationships between local health departments and the utility. Estimating costs will only be possible after the basic decisions presented above are considered. Substantial costs could be incurred with the purchase of new software, development of code to automate systems integration, and initial training. However, investment in these resources now could prevent even higher labor expenses during operation and maintenance. In making decisions on how to collect data, automating data streams may be more expensive during installation due to equipment and software procurement; however, this initial cost may be offset in the long run by reduced effort needed by staff to continue data collection. Key cost considerations should include the following: Determining contacts and developing a Concept of Operations. Effort will be needed to generate these contacts and documents Computer hardware or software for data collection and analysis. It may be necessary to procure computer hardware or software and/or develop custom software for the purposes of data gathering, analysis, and display. Where possible, available and experienced resources should be leveraged to reduce these costs. Determining and testing alert levels. It will take some effort to determine and test appropriate alert levels to maximize effective investigation while minimizing false positives. Training on utilizing new analysis tools. These costs will depend on how many new tools are implemented, and the overall number of people who need training Table-top and communication exercises. Resources will be necessary to plan, support, and participate in table-top and other response exercises. May 2007 92 ------- Planning for WS-CWS Deployment In general, the more agencies that are involved with public health surveillance, the more expensive it will be in terms of training costs, coordination, and level of effort. However, involving more agencies also means increasing potential resources in terms of software, equipment and expertise. Leveraging existing systems could offset the cost of increased coordination by reducing the need for developing new systems. May 2007 93 ------- Planning for WS-CWS Deployment Section 9.0: Consequence Management As discussed and illustrated in Section 1, consequence management plays a critical role in a contamination warning system. When triggers from one or more of the monitoring and surveillance components discussed in Section 4 - Section 8 have been validated, consequence management governs the response, remediation, and recovery actions. A consequence management plan that successfully guides these actions is a cornerstone of an effective contamination warning system, and it is essential to have the plan in place and tested prior to operation of any contamination warning system components. While development of a consequence management plan can occur in parallel with design and implementation of monitoring and surveillance components of the contamination warning system, it is important to reconcile the consequence management plan with routine operations as the system evolves through the application of system engineering principles as discussed in Section 2.1. In planning for contamination warning system deployment, it is important to recognize that while the integrated, routine, and active approach for monitoring and surveillance of public health and water quality in the distribution system may be a new concept, drinking water utilities should have an existing emergency response plan that can serve as a starting point for consequence management. In response to the terrorist attacks of 2001, Congress passed the Public Health Security and Bioterrorism Preparedness and Response Act of 2002 (the Bioterrorism Act) which required drinking water utilities to prepare or revise, where necessary, an emergency response plan that incorporates the results of vulnerability assessments. Consequence management plans developed in support of contamination warning systems should build on the utility's existing emergency response plan, focusing specifically on the contamination threat to the distribution system and should integrate response and decision-making with the routine operation of the system as defined in the concept of operations. EPA previously provided guidance on response to drinking water contamination in a suite of six modules that composed the Response Protocol Toolbox and companion Response Guidelines (USEPA, 2004a-h). Many of the concepts presented in the Response Protocol Toolbox are applicable to development of a consequence management plan for contamination warning systems. This section provides high-level considerations to assist utilities and local partners in planning for development of a consequence management plan to support contamination warning system implementation. 9.1 Pre-design Designing a consequence management plan is a critical task as part of contamination warning system deployment. Pre-design activities should be carefully conducted, such that subsequent steps can be effectively executed, with the result being a complete utility-specific consequence management plan that presents a comprehensive framework for response to validated contamination warning system triggers. Roles and responsibilities should be defined and assessments conducted, both at the utility and with local partners. These preliminary assessments can help identify existing plans, both within the utility and partner organizations, and training opportunities that can be integrated as part of the effort to develop the consequence management plan for the contamination warning system. Pre-design activities for development of a consequence management plan include the following: Identification of objectives for the consequence management plan Self-assessment Identification of partners Development of a preliminary work plan and path forward May 2007 94 ------- Planning for WS-CWS Deployment Consequence Management Plan Objectives Prior to development of a consequence management plan to support contamination warning system implementation, drinking water utilities should define the objectives of the plan. Most utilities have detailed emergency response plans and/or action plans that may address contamination of the distribution system. For the purposes of contamination warning system deployment, objectives of the consequence management plan may include the following: Clearly defined roles and responsibilities for utility and response partners in all stages of consequence management Comprehensive decision-making framework to support timely response to a trigger from one or more contamination warning system components Guidance on use of specific response procedures Seamless transition from routine operations to consequence management activities Integration with local and regional response plans Determining objectives of the consequence management plan early in the process should enable the utility to effectively use information from their emergency response plans to create a comprehensive consequence management plan applicable to a contamination warning system. Utility Self-Assessment A self assessment of the utility's existing emergency response plans and overall preparedness is the first step in developing a consequence management plan to support contamination warning system implementation. The purpose of the self assessment is to identify what procedures are already in place regarding planning, preparedness, and response and assess these procedures relative to the objectives of the contamination warning system consequence management plan. There are two primary aspects of this assessment: existing plans and response resources and capabilities. The utility should first review their current response procedures and related documents for different events to determine what elements of a consequence management plan they may already have. Examples of the types of plans that should be considered include the following: Plans for responding to a water contamination or water quality event, cross connections, chemical spills near source water, intentional contamination of the water system Plans for responding to increased or overwhelming consumer complaint calls, or calls reporting illness from the water Plans for responding to facility alarms, reports of suspicious persons near utility facilities, or threats made to the system, both directly to the utility and though third parties (police, media, etc.) Operational plans that address issues such as depressurization or power outage Severe weather response plans Civil disorder response plans Mutual aid agreements with other utilities Issuing of water-use restrictions Risk communication and public notification plans As plans are reviewed, the situation addressed, utility divisions are involved, and outside agencies involved should be captured to identify gaps that should be addressed through consequence management planning activities. In addition to an assessment of existing operational plans, the utility should conduct an assessment of response resources and capabilities. This involves identifying assets (e.g., staff, equipment) as well as training needs that are required to implement the existing plans and operations. Throughout the development of the consequence management plan, the utility should maintain a list of items or resources May 2007 95 ------- Planning for WS-CWS Deployment that should be acquired, enhanced, or improved. During design and implementation, the list can be revisited and shortfalls in training, equipment, and other resources can be resolved. An aspect of the utility self-assessment related to response resources and capabilities includes assessing Incident Command System (ICS) training and National Incident Management System (NIMS) compliance. ICS is a flexible command and control system designed to manage any magnitude of emergency. ICS is a key component of NIMS, and NIMS is a key component of the National Response Plan (NRP) which directs, among other things, how command over an incident is escalated from the local to state to federal level, and back down again. More information on ICS and NIMS can be found at the Federal Emergency Management Agency's (FEMA) website (http://www.fema.gov/emergency/nims/index.shtm). There are many reasons to have staff trained in ICS and NIMS. ICS has been used effectively since the 1970s and is a proven system. The response partners engaged will most likely be well versed in ICS and NIMS, and should expect your staff to be as well. Additionally, as directed by Homeland Security Presidential Directive 5, full NIMS compliance is a requirement for receiving federal preparedness funds. The utility should also consult their state emergency management and/or homeland security agency, as many states have NIMS requirements that are more stringent than the federal requirements. Identification of Response Partners Identification of partners to support contamination warning system design and implementation is discussed in Section 2.0. For the purposes of development of the consequence management plan, specific contacts should be identified within each of those organizations to discuss roles and responsibilities specifically related to response, remediation, and recovery actions. Table 9-1 provides an overview of the roles and responsibilities response partners may play in implementing the consequence management plan, and thus, when they should be engaged in the planning process. Table 9-1. Overview of Response Partner Roles and Responsibilities for Consequence Management Response Partner Drinking water utility Local health department Local law enforcement Local civil government Local emergency planning committees and emergency management agencies Local fire, EMS, and Hazmat Environmental and public health laboratories Local wastewater utility Neighboring utilities (water and/or wastewater) Media State government State emergency responders State drinking water and wastewater primary agencies State emergency management and homeland security agencies State law enforcement EPA Regional offices and/or laboratories Federal Bureau of Investigation Centers for Disease Control and Prevention Operational Response Y Y Y Y Public Health Response Y Y Y Y Y Site Characterization Y Y Y Y Y Y Criminal Investigation Y Y Y Y Y Y Y Y Expanded Sampling Y Y Y Y Y Y Y Y Y Y Y Laboratory Analysis Y Y Y Y Y Y Y Risk Communication Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Remediation and Recovery Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y May 2007 96 ------- Planning for WS-CWS Deployment Response Partner EPA Criminal Investigation Division EPA National Response Center ns n) C (ft 0 C += O ns Q. <- ns to ns O P Q) o: Y Y As response partners are identified, it may also be necessary to work with them to assess their existing plans and capabilities to ensure there should be a seamless transition as an event escalates through the stages of response, remediation, and recovery. Different response partners may have different areas of expertise, and likewise might have different levels of response preparedness. These discrepancies should be identified in the pre-design phase, and so that resolving them can be accomplished through design and implementation activities. Work Plan to Guide Consequence Management Plan Development Following the utility self-assessment and identification of response partners, the next step in the pre- design process is to develop a work plan to guide development of the consequence management plan. The work plan should identify a framework or process for plan development, including engagement of local partners, communications and roll out, and closing of any gaps related to training or capability. In addition, major goals and milestones may be established in the work plan, such that the objectives defined at the start of pre-design become practical, concrete targets. The work plan should aim at directing steps to meet these targets and staying on schedule. Completion of a well-defined work plan should aid in the success of developing a consequence management plan as the utility moves forward through design and implementation activities. 9.2 Design and Implementation Approach Design and implementation activities associated with consequence management plan development include the following: Consequence management plan development. Develop a framework and approach for credibility determination, confirmation, and remediation and recovery. The development process may include the following: o Defining an ICS structure for the utility o Engagement of local partners and alignment with existing plans at the local, regional, and state levels o Reconciliation with the concept of operations as defined for the monitoring and surveillance components o Integration of site characterization plan Communication strategy o Develop a strategy and framework for communications within the utility, between the utility and external partners, and customers. o Develop a risk communication strategy and message maps for communicating information during a suspected or confirmed contamination incident. Implementation: Procure equipment needed to augment utility or local partner capabilities. Equipment may include sampling equipment, personal protective equipment, field screening equipment, communications equipment, etc. See Section 5.3 for a discussion of site characterization equipment, coordination, and planning. May 2007 97 ------- Planning for WS-CWS Deployment Training o Develop training materials that address all roles and responsibilities o Conduct training on the plan. The scope of this training should include all levels of staff within the utility that have a role in consequence management as well as local partners. o Ensure appropriate staff are trained and certified in ICS and NIMS. Revise the consequence management plan as necessary based on feedback from training and reconciliation with routine operations and initial trigger validation for each of the components. Preliminary Testing: Conduct additional training on the consequence management plan as necessary. Design and implement drills and exercises to test the consequence management plan that involve drinking water utility staff as well as local response partners. Refine and finalize the consequence management plan, including reconciliation with the concept of operations to ensure a smooth transition from routine operations, initial trigger validation, and consequence management across all components of the system. Operations and Maintenance: Deploy plan as necessary in response to validated triggers from one or more of the contamination warning system components. Conduct ongoing training to ensure that new staff are familiar with the consequence management plan. Update the plan as necessary based on enhancements or modifications to monitoring and surveillance aspects of the system or local response partner capabilities. Evaluation and Refinement: Conduct routine drills and exercises to evaluate the operation and performance of various aspects of the consequence management plan Refine the plan as appropriate based on lessons learned through drills and exercises 9.3 Available Tools and Resources The following tools and resources are available to support development of a consequence management plan for a contamination warning system: Response Protocol Toolbox Developed by the EPA, this series of six documents covers topics such as communications and notifications, threat evaluation, site characterization, sample analysis, and response actions to help the water sector prepare for and respond to contamination threats and incidents. o Overview (EPA-817-D-03-007) http://www.epa.gov/safewater/watersecurity/pubs/guide response overview.pdf o Water Utility Planning Guide - Module 1 (EPA-817-D-03-001) http://www.epa.gov/safewater/watersecuritv/pubs/guide response modulel.pdf o Contamination Threat Management Guide - Module 2 (EPA-817-D-03-002) http://www.epa.gov/safewater/watersecurity/pubs/guide response module2.pdf o Site Characterization and Sampling Guide - Module 3 (EPA-817-D-03-003) http://www.epa.gov/safewater/watersecurity/pubs/guide_response_module3.pdf o Analytical Guide - Module 4 (EPA-817-D-03-004) http://www.epa.gov/safewater/watersecuritv/pubs/guide response module4.pdf o Public Health Response Guide - Module 5 (EPA-817-D-03-005) http://www.epa.gov/safewater/watersecurity/pubs/guide response module5.pdf o Remediation and Recovery Guide - Module 6 (EPA-817-D-03-006) http://www.epa.gov/safewater/watersecuritv/pubs/guide response module6.pdf May 2007 98 ------- Planning for WS-CWS Deployment Response Protocol Toolbox: Response Guidelines. An action oriented document to assist drinking water utilities, laboratories, emergency responders, state drinking water programs, technical assistance providers, and public health and law enforcement officials during the management of an ongoing contamination threat or incident. The Response Guidelines are not intended to replace to Response Protocol Toolbox and they do not contain the detailed information contained within the six complete modules. The Response Guidelines are to be viewed as the application of the same principles contained in the Response Protocol Toolbox during an actual incident. The Response Guidelines have been developed to provide an easy to use document for field and crisis conditions. Finally, users are encouraged to adapt the Response Guidelines as necessary to meet their own needs and objectives. http://www.epa.gov/safewater/watersecurity/pubs/rptb response guidelines.pdf Incident Command System (ICS) Training and National Incident Management System (NIMS) Compliance ICS is a command and control system designed to grow and contract to manage any magnitude of emergency. ICS is a key component of NIMS, and NIMS is a key component of the National Response Plan (NRP) which directs how command over an incident is escalated from the local to state to federal level. Additionally, as directed by Homeland Security Presidential Directive (HSPD) 5, full NIMS compliance is a requirement for receiving federal preparedness funds More information on ICS and NIMS can be found at FEMA's website: http://www.fema.gov/emergencv/nims/index.shtm Federal Emergency Management Agency (FEMA) FEMA is the federal agency responsible for responding to and aiding the recovery from natural or man-made disasters. The have developed the ICS and NIMS training program, as well as the National Response Plan (NRP) to establish an all-hazards response to emergencies for communities throughout the United States. www.fema.gov Department of Homeland Security (DHS) DHS offers many resources relating to water security, included training and research under the areas of awareness, prevention, protection, response and recovery. The website is also contains information relating to bioterrorism laws, regulations and policies, as well as background information about the Homeland Security Presidential Directives, www.dhs.gov Centers for Disease Control and Prevention (CDC) The CDC offers response plans for agents, diseases, and other events through its Emergency Preparedness and Response branch. Specific training opportunities offered include those for Bioterrorism, Chemical, and Radiation Emergencies, www.bt.cdc.gov Federal Bureau of Investigation (FBI) The FBI is a major partner in investigating terrorist activities, and may be one of the responders to a contamination event. The FBI can also support local and state enforcement agencies, www.fbi.gov Water Information Sharing and Analysis Center (WaterlSAC) Online database containing information, expert analysis, and government alerts. Provides tools for water security and links to other agencies, such as homeland security, law enforcement, and public health. www.waterisac.org The Association of State Drinking Water Administrators (ASDWA) A professional organization that supports states in their efforts to assure quality drinking water, and encourages coordination between state drinking water agencies. Provides tools and technical materials for area wide optimization programs, data management and security, www.asdwa.org May 2007 99 ------- Planning for WS-CWS Deployment American Water Works Association (AWWA) AWWA is a professional organization dedicated to improving water quality and supply. They provide numerous resources and training tools for use by utilities, including Water 101: Security Planning and Partnership for Safe Water online courses, www.awwa.org Utilities Helping Utilities: An Action Plan For Mutual Aid and Assistance Networks For Water and Wastewater Utilities A document developed by AWWA to help utilities develop Water and Wastewater Agency Response Networks (WARNs), a mutual aid and assistance program to be used after a utility has sustain damage from man-made or natural disasters. http://www.awwa.org/Advocacy/govtaff/ Effective Risk and Crisis Communication during Water Security Emergencies. This report summarizes results from three water security risk communication message mapping workshops conducted by U.S. EPA's National Homeland Security Research Center during 2005/2006. It provides information about effective message development and delivery that could be useful to water sector organizations as they develop their respective risk communication plans. Message mapping is a process by which users can predict 95 percent of questions likely to be asked by the media and others following an incident, prepare clear and concise answers to the questions along with supporting information ahead of time, and practice effective message delivery before a crisis occurs. nttp://www.epa.gov/nhsrc/pubs/reportCrisisCom040207.pdf 9.4 Staffing and Cost Considerations Staffing and cost considerations for development and implementation of a consequence management plan as part of a contamination warning system may vary significantly from utility to utility based on existing capabilities and the approach used to develop the consequence management plan. Section 9.4.1 and 9.4.2 provide general considerations for staffing and costs, respectively. 9.4.1 Staffing Response partners involved in development and implementation of the consequence management plan are discussed in Section 9.1. The response partners needed to support consequence management activities may vary from utility to utility; however, once identified, they should be engaged from the pre-design through the evaluation and refinement process. For the drinking water utility implementing a contamination warning system, staffing considerations may vary based on the approach taken to develop the plan. Throughout development and implementation of the plan, it may be necessary to engage representatives from each utility department or division. Ultimately, the departments or divisions engaged may also vary based on response actions needed. While senior managers may have a more significant role in development of the plan, all levels of staff may need to be engaged in some aspect of training, participation in drills and exercises, and plan implementation. 9.4.2 Cost Considerations Costs associated with development and implementation of a consequence management plan for contamination warning systems may vary based on existing plans and capabilities within the utility and the jurisdictions included in the utility's service area. Most of the costs associated with development of the consequence management plan are related to staff time. In planning for contamination warning system implementation, utilities should also consider costs associated with training, equipment (e.g., communications), and development and implementation of drills and exercises to evaluate the plan. Like the development of the consequence management plan, most of the costs associated with training and development of drills may be labor costs, and may vary based on how many people are involved. May 2007 100 ------- Planning for WS-CWS Deployment When identifying cost considerations for developing the consequence management plan, it should be considered that some of the equipment and training necessary to address gaps here my already be planned for implementation under one of the other contamination warning system components. As such, these costs should be discussed amongst the project management team. Identifying instances such as this may not only succeed in reducing overall costs, but strengthens the concept that a contamination warning system (and in particular a consequence management plan) requires the components to function together. May 2007 101 ------- Planning for WS-CWS Deployment Section 10.0: References AWWA. 2004. Interim Voluntary Security Guidance for Water Utilities. Hall, J. Zaffiro, A. D., Marx, R. B., Kefauver, P C., Krishnan, E. R, Haught, R C., Herrmann, J. G. 2007a. Online Water Quality Parameters as Indicators of Distribution System Contamination. Journal of the American Waterworks Association, 99 (l):66-77. Hall, J., Szabo, J.G. 2007b. Evaluation of Water Quality Monitoring Technologies to Respond to Changes in Drinking Water Quality. Presented at GWRC On Line Monitoring Workshop; KIWA Facility, Nieuwegein, NETHERLANDS, March 21-22. Hall, J.S., Haught, R., Rahman, M., Richardson-Coy, R. and Piao, H. 2007c. The bench-scale minimum dose threshold experiment. EPA/600/R-07/002. Water Information Sharing and Analysis Center (WaterlSAC) (www.waterisac.org). Szabo, J.G., Hall, J.S. and Meiners, G.C. 2006. Water quality sensor responses to injected contaminants in a chloraminatedpipe loop. American Water Works Association (AWWA) Water Security Congress, Technical Session TUE6: Technology Forum B, Washington, DC, September 10-12, 2006 Szabo, J. et al. 2007. Water Quality Sensor Response to Contamination in a Single Pass Water Distribution System Simulator. EPA-600-R-07-001. Available only through WaterlSAC USEPA. 2003. Instructions to Assist Community Water Systems in Complying with the Public Health Security and Bioterrorism Preparedness and Response Act of 2002. EPA-810-B-02-001. USEPA. 2004a. Response Protocol Toolbox Overview. (EPA-817-D-03-007). USEPA. 2004b. Response Protocol Toolbox: Water Utility Planning Guide -Module I. (EPA-817-D-03- 001). USEPA. 2004c. Response Protocol Toolbox: Contamination Threat Management Guide -Module 2. (EPA-817-D-03-002). USEPA. 2004d. Response Protocol Toolbox: Site Characterization and Sampling Guide -Module 3. (EPA-817-D-03-003). USEPA. 2004e. Response Protocol Toolbox: Analytical Guide - Module 4. (EPA-817-D-03-004). USEPA. 2004f Response Protocol Toolbox: Public Health Response Guide -Module 5. (EPA-817-D- 03-005). USEPA. 2004g. Response Protocol Toolbox: Remediation and Recovery Guide -Module 6. (EPA-817- D-03-006). USEPA. 2004h. Response Protocol Toolbox: Planning for and Responding to Drinking Water Contamination Threats and Incidents Response Guidelines. USEPA. 2005a. WaterSentinel System Architecture. EPA-817-D-05-003. USEPA. 2005b. WaterSentinel Contaminant Selection. Sensitive Information, Limited Distribution, For Official Use Only. May 2007 102 ------- Planning for WS-CWS Deployment USEPA. 2005c. Overview of Event Detection Systems. EPA-817-D-05-001. May 2007 103 ------- Planning for WS-CWS Deployment Appendix A: Glossary Anomaly. Deviations from an established baseline or base state. Specifically, a water quality anomaly is a deviation from an established water quality base state at a specific location. Base State. Normal conditions that result from typical system operation. The base state includes predictable fluctuations in measured parameters that result from known changes to the system. For example a water quality base state includes the effects of draining and filling tanks, pump operation, and pipe flushing, all of which may alter water quality in a somewhat predictable fashion. Baseline Data. Baseline data is all available chemical, radiochemical, pathogen and toxin analytical data relative to a baseline sample that may be used to determine "possible" or "credible" contamination. Baseline data may be contaminant and location-specific control charts and tabulated contaminant data. Concept of Operations (Con Ops). A process for routine operation of a drinking water contamination warning system, which establishes specific roles and responsibilities, process and information flows, and procedural activities. The Con Ops includes the process for validation of a contamination warning system trigger and determining whether or not contamination is "possible." Consequence Management Plan. Provides a decision-making framework that governs when, how, what, and who will be involved in making decisions in response to a "possible" contamination incident in order to minimize the response timeline and implement operational or public health response actions appropriately. "Credible." In the context of the credibility determination process, water contamination is characterized as 'credible' if information collected during the investigation of "possible" contamination corroborates information from the validated contamination warning system trigger. Credibility Determination. Contamination warning system triggers will be investigated to determine whether or not they are indicative of "possible" contamination. Credibility determination is the subsequent investigation to determine whether or not additional information, including data from other monitoring and surveillance components, corroborates the information from the validated trigger. If the additional information corroborates the trigger, contamination is considered 'credible.' Event Detection System (EDS). A system designed specifically to detect anomalies from the various monitoring and surveillance components of a contamination warning system. An EDS may take a variety of forms, ranging from complex set of computer algorithms to a simple set of heuristics that are manually implemented. In essence, an EDS is a data mining tool that supports the efficient analysis of large amounts of monitoring and surveillance data to pick out possible anomalies while at the same time minimizing false alarms. EDS Alarm. A notification from the EDS tool that an anomaly has been detected. Contamination warning system alarms may be visible and/or audible, and may initiate automatic notifications such as pager or e-mail alerts. Most EDS alarms require some degree of validation before they are considered indicative of "possible" contamination. Field Screening. Performing a series of tests to evaluate any potential chemical, biological or radiochemical dangers present at the site. Job Function. A description of the duties and responsibilities of a specific job within an organization. May 2007 104 ------- Planning for WS-CWS Deployment Monitoring and Surveillance. Element of a contamination warning system that provide a standardized set of information streams used in the detection of potential contamination incidents. "Possible." In the context of the contamination warning system concept of operations, water contamination is characterized as "possible" if the cause of a trigger cannot be identified and/or determined to be benign. Risk Reduction Units (RRUs). The difference between the calculated risk before security improvements versus after security improvements. A measure of enhanced security. RRU = (Risk Before improvements) - (Risk After Improvements) Security Breach. An unauthorized intrusion into a secured facility that may be discovered through direct observation, an alarm trigger, or signs of intrusion (cut locks, open doors, cut fences). Site characterization. The process of collecting information from an investigation site to support the evaluation of a drinking water contamination threat Standard Operating Procedure (SOP). A step-by-step list of actions that guide the user in the implementation of a specific task. Syndromic surveillance. Collecting and analyzing nontraditional data to detect a change or trend in the health of a population using categories of disease rather than formal diagnosis. Target Contaminant. A contaminant that has been identified by the EPA for monitoring under the Water Security Sentinel Initiative. Target contaminants are monitored using drinking water confirmatory methods. Reported results are qualitative and quantitative Threat Warning. An unusual occurrence, observation or discovery that indicates a potential contamination incident and initiates actions to address this concern. Trigger. Information from a monitoring and surveillance component that an anomaly has been detected. Trigger Validation. The process of investigating potential causes of a contamination warning system trigger to either rule out contamination or determine that contamination is "possible." May 2007 105 ------- Planning for WS-CWS Deployment Appendix B: Information Security Considerations Because the Water Security initiative is first and foremost a homeland security program with a counter- terrorism focus, information security is extremely important. Certain utility materials and materials developed in support of the program can potentially be exploited by adversaries to defeat the system. Therefore, it is necessary for partner utilities to develop formal procedures and protocols for the identification, handling, tracking, and overall security of any sensitive documents involved with the development, implementation, and operation of their contamination warning system. Training on and abiding by these procedures and protocols should be considered for any and all staff who may access sensitive materials as part of their job; this includes utility, partner, contractor and subcontractor personnel. Some key elements and steps in developing and implementing an information security strategy include: Assessment of Material Sensitivity, Access, and Law Sensitive Materials Tracking and Storage Sensitive Materials Handling Protocols and Procedures Staff Certification and Background Checks Coordination and Cooperation with Partners, Contractors and Subcontractors Additionally, it is recommended that utility program management designate a team responsible for the development, operation, and maintenance of an information security strategy, and reconciliation with partner agencies sensitive information programs. This team should be comprised of members with backgrounds in general and/or information security, information technology, emergency response, and law and law enforcement. B.1 Assessment of Material Sensitivity, Access, and Laws The first step to implementing an information security program is to identify potentially sensitive materials. The information security team should consider existing utility and partner materials, as well as materials that will be developed. Examples include a utility's sensitive facilities list, security procedures and emergency response procedures, hydraulic models, facility maps and blueprints, locations and types of contamination warning system enhancements and certain contact lists and notification procedures. As materials are reviewed, the team can begin developing a tiering system for different levels of sensitivity. It is recommended that the specific terminology used in this tiering system draw on accepted nomenclature (e.g., terms such as "For Official Use Only", "Sensitive", "Confidential", and "Proprietary"). However, the terms chosen and the level of sensitivity they reflect should be clearly defined in the information strategy document, and whether they truly afford any legal protection. Also as part of the assessment, local, state, and federal laws and regulations should be reviewed to determine what legal protections may be afforded to sensitive materials. Because local, state, and federal governments are involved in contamination warning system implementation, the federal Freedom of Information Act (FOIA), as well as similar laws and regulations passed in other jurisdictions may be applied to certain types of information, especially if the utility itself is part of a government entity. The FOIA legislation, implemented in 1967, governs the disclosure of documents and information controlled by the U.S. government (this includes documents submitted to the government by an outside government agency or private entity). In general, most information held by the government should be made available to the public or other entities if requested, provided said information is not covered by one of nine exemptions (such as certain types of confidential business information, or CBI, and National Security Information, which has been designated exempt to protect the security of the Nation). Since the legislation went into effect, and particularly in the aftermath of 9/11, certain information, particularly relating to critical infrastructure, like water utilities, has been determined to be covered by one or more of the exemptions. Additionally, many other jurisdictions have exempted sensitive security information May 2007 106 ------- Planning for WS-CWS Deployment related to critical infrastructure from there own regulations that are similar to FOIA. Local, state, and federal laws related to disclosure of government held information should be reviewed to determine what disclosure requirements and exemptions might apply to the information being assessed. It is recommended that both the utility's legal counsel and that of the respective governing entity be engaged in this process. It is important to remember that sensitivity assessment of materials will remain an ongoing process throughout the life of the contamination warning system; some materials may have their sensitivity determination reconsidered while new materials will have to be placed in the ranking system. It is also important to remember that new documents produced from sensitive materials should also be assessed for their sensitivity, and possibly included in the sensitive materials program. B.2 Sensitive Materials Tracking and Storage Maintaining an effective tracking and storage system for sensitive materials is an important aspect of drinking water security. Additionally, it will be necessary to exchange information with partner agencies as part of the development and implementation of a contamination warning system. Their sensitive data should be protected as thoroughly as the utility's own documents, both so that the partners maintain confidence in their partnership with the utility, and so that the utility does not become the conduit for exploitable information falling into the wrong hands. As materials are being assessed, a tracking and storage system should be developed. Materials can be assigned tracking numbers which identify the date of receipt or creation, creating or owning agency, form of the material (electronic, paper, etc.) kind of material (map, document, blueprint, etc.), whether the material is a duplicate or is a number in a series of duplicates, and other identifying pieces of information. As part of the tracking system, a system for logging materials in and out should be implemented. This could be as simple as a paper or electronic log sheet maintained by a responsible person or group within the utility which lists the date, tracking number, and signature, initial, or other mark of the person obtaining the material. More complex systems can involve confirmatory phone calls or e-mails for materials that are mailed or sent electronically, or by using the receipt and tracking services of the U.S. Postal Service and other carriers. Storage refers to where the materials are stored when they are checked-in. The utility should decide whether a central storage location will be used, or whether different divisions will have storage areas for the materials they primarily use for their implementation efforts. In deciding on storage procedures and location, a utility should consider such factors as whether the location is physically secure from break-in, whether a severe rain storm or other natural event might damage it, fire protection, and whether electronic materials are protected from power failure. B.3 Sensitive Materials Handling Protocol and Procedures Proper handling of materials is important for the same reasons that storage and tracking are important. It is a wasted effort to expend resources protecting documents while in storage, only to have them stolen from a car, lost on the bus, left in a printer tray, or emailed to "reply to all". Sensitive materials handling in this context refers to how those materials should be protected when checked-out of central storage. Before or while materials are being assessed for sensitivity, handling procedures and protocols should be developed. Handling protocols are fairly common anywhere sensitive information is used, from government agencies to corporations intent on keeping proprietary or other sensitive information a secret, and they can range from common sense handling procedures, like locking them in a drawer when not in use, to intensive procedures used for classified information. The sensitive material handling procedures should be specific to different forms of media and transmitting media, like e-mailing electronic May 2007 107 ------- Planning for WS-CWS Deployment attachments. Some resources that can be found online include EPA's Office of Science and Technology's Confidential Business Information plan and the Toxic Substances Control Act Confidential Business Information Protection Manual. However, aside from minimum protections required by EPA, it will be up to the utility and its partners to determine how conservative their sensitive materials plan is. Some common precautions include: Shredding extra or damaged paper documents Locking documents up at night or when not in use for extended periods of time, even when in secure facilities Employing cover sheets for covering sensitive material when leaving one's workspace for a short period of time, or when a person not authorized to view the material enters the office. For electronic files, this would mean enabling the computer screen saver or locking the computer. Utilizing encryption software for electronic transmittal of materials. Only using company or agency email addresses (for example, not gmail, AOL, or hotmail accounts). Downloading attachments immediately and deleting the transmitting email. Prohibiting saving of sensitive materials on an unprotected Local Area Network (LAN). Double wrapping materials when transporting them outside secure facilities. Double wrapping consists of wrapping documents in an inner envelope or packaging that contains the "please return to" information, as well as clear markings that the packaging contains sensitive or proprietary information. An outer envelope or packaging should contain the same "please return to" information but not the sensitive or proprietary information label, so as not to arouse the curiosity of whoever finds it. Requiring the same level of protection for electronic media, like thumb drives, as are required for paper documents. Calling the recipient before faxing a document, and requiring a confirmatory phone call back. Not leaving sensitive materials unattended in cars, even if the car is alarmed or locked. Utilizing hotel room safes These types of precautions should also be observed whenever the sensitivity of a document or material is in question, including for materials produced from sensitive materials, but have yet to be assessed for their sensitivity. Responsible handling of sensitive materials extends to protecting the data they contain as well. Precautions should be taken to ensure that discussions of sensitive materials, or documents produced from sensitive materials (and hence might contain sensitive materials themselves), are afforded equal precautions. Care should be taken during conference calls to ensure all participants are cleared to discuss sensitive information, and that others cannot join the call without the host's knowledge. Similarly, using speaker-phone when discussing sensitive materials should be avoided. Sensitive materials should not be discussed in public places. To help this effort, persons cleared for access to sensitive materials should be provided updated lists regularly of who else is cleared to view sensitive materials. B.4 Staff Certification and Background Checks Part of ensuring the security of sensitive materials is ensuring the integrity of those who will be handling the materials. Background checks in general, and specific training on the handling of sensitive materials, will help achieve this goal. If not already done, background checks should be performed on all staff involved in the design, implementation, and operation of the major components of the contamination warning system: this includes utility, partner, contractor and subcontractor staff. While it may not be necessary to perform a background check on a member of a maintenance crew whose day-to-day responsibilities will be largely unaffected by the program, it will be necessary to perform them for staff who, say, have access to the locations of monitoring stations. The utility should assess its existing rules regarding which personnel May 2007 108 ------- Planning for WS-CWS Deployment should have background checks, how extensive those checks are, and how they should adjust them based on new roles and responsibilities resulting from implementing the contamination warning system. It is strongly recommended that any staff who will be handling sensitive materials receive background checks. In addition to background checks, a certification or training process should be implemented for staff who will be handling sensitive materials. At a minimum, staff should certify that they have read and understand relevant materials related to the handling of sensitive materials, and lists should be kept showing who has received such training and when they are due for recertification; signed nondisclosure agreements can be utilized as part of this certification process. B.5 Coordination and Cooperation with Partners, Contractors and Subcontractors As mentioned, it may be necessary to provide certain sensitive materials to (or receive from) response partners and contractor personnel. While it is up to the utility to decide what level of precaution is appropriate for their internal sensitive materials, they should be cognizant that their partners may have more or less restrictive measures. Prior to the exchange of any sensitive materials, the utility should provide their sensitive materials handling guidance to the partner agency and request the partner agency's equivalent guidance. This way, any conflicts between the two can be settled before materials are exchanged. Additionally, formal non-disclosure agreements should be provided by contractors and subcontractors to the utility, and the utility should check to see if any similar agreements partners have with their contractors and subcontractors are consistent with the utility's procedures. To ease this process, it is recommended that the utility seek guidance in developing their own sensitive materials program from such partners as local counter-terrorism and intelligence groups, such as the FBI local field office, or local Joint Terrorism Task Force (JTTF). Other local partners, such as public health agencies, who have experience maintaining data with special handling requirements, can also be consulted. May 2007 109 ------- |