&EPA United States Environmental Protection Agency Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Pilot Monitoring and Surveillance Water Quality Monitoring Enhanced Security Monitoring Customer Complaint Surveillance ublic Health Surveillance Possible Contamination Consequence Management Sampling and Analysis Response Office of Water (MC-140) EPA-817-R-14-001G April 2014 ------- ------- Disclaimer The Water Security Division of the Office of Ground Water and Drinking Water has reviewed and approved this document for publication. This document does not impose legally binding requirements on any party. The findings in this report are intended solely to recommend or suggest and do not imply any requirements. Neither the U.S. Government nor any of its employees, contractors or their employees make any warranty, expressed or implied, or assumes any legal liability or responsibility for any third party's use of or the results of such use of any information, apparatus, product, or process discussed in this report or represents that its use by such party would not infringe on privately owned rights. Mention of trade names or commercial products does not constitute endorsement or recommendation for use. Questions concerning this document should be addressed to: Elizabeth Hedrick U.S. EPA Water Security Division 26 West Martin Luther King Drive Mail Code 140 Cincinnati, OH 45268 (513)569-7296 Hedrick.Elizabeth@epa.gov or Steve Allgeier U.S. EPA Water Security Division 26 West Martin Luther King Drive Mail Code 140 Cincinnati, OH 45268 (513)569-7131 Allgeier.Steve@epa.gov ------- Acknowledgements The Water Security Division of the Office of Ground Water and Drinking Water would like to recognize the following individuals and organizations for their assistance, contributions, and review during the development of this document. • Jeff Swertfeger, Greater Cincinnati Water Works • Mike Tyree, Greater Cincinnati Water Works • Matthew Magnuson, U.S. Environmental Protection Agency • Jennifer Calles, City of Phoenix Water Services Laboratory, Environmental Services Division ------- Executive Summary The goal of the Water Security Initiative (WSI) is to design and demonstrate an effective multi- component warning system for timely detection and response to drinking water contamination threats and incidents. A contamination warning system (CWS) integrates information from multiple monitoring and surveillance components to alert the water utility to possible contamination, and uses a consequence management plan to guide response actions. The first CWS pilot under WSI was deployed in Cincinnati, Ohio, in partnership with the Greater Cincinnati Water Works (GCWW). System design objectives for an effective CWS are: spatial coverage, contaminant coverage, alert occurrence, timeliness of detection and response, operational reliability and sustainability. Metrics for the sampling and analysis (S&A) component are defined relative to the system metrics common to all four monitoring and surveillance components of the CWS, but the component definition provides an additional level of detail relevant to the S&A component. Evaluation techniques used to quantitatively or qualitatively evaluate each of the metrics include analysis of empirical data from routine operations, drills and exercises, modeling and simulations, forums and an analysis of lifecycle costs. This report describes the evaluation of data collected from the S&A component from the period of March 2008 - June 2010. The major outputs from the evaluation of the Cincinnati pilot include: 1. Cincinnati Pilot System Status, which describes the post-implementation status of the Cincinnati pilot following the installation of all monitoring and surveillance components. 2. Component Evaluations, which include analysis of performance metrics for each component of the Cincinnati pilot. 3. System Evaluation, which integrates the results of the component evaluations, the simulation study, and the benefit-cost analysis. The reports that present the results from the evaluation of the system and each of its six components are available in an Adobe portfolio, Water Security Initiative: Comprehensive Evaluation of the Cincinnati Contamination Warning System Pilot (USEPA 2014a). Sampling and Analysis Component Design Although not an early detection component, S&A plays a critical role in the CWS due to the potential to confirm or rule out contaminants in drinking water samples collected throughout the pilot utility's distribution system during investigation of validated CWS component alerts as part of the credibility determination process. Unlike other CWS monitoring and surveillance components, the S&A component affords the potential to identify specific contaminants and, in many instances, determine the concentration of these contaminants in drinking water. For the Cincinnati pilot, baseline methods were selected for their ability to detect and confirm contaminants which EPA has identified as being of particular concern in drinking water. During a potential contamination incident, drinking water samples are collected and analyzed with the goal of identifying and confirming, or ruling out the presence of specific contaminants (i.e., incident response sampling and analysis). Field response personnel perform site characterization activities including site safety screening and rapid field testing of water samples. These activities can provide information rapidly to assist decisions regarding site safety and potential contaminants, and can inform or focus subsequent sample collection and analysis. Upon completion of sample collection and field analyses, samples are packaged and transported to the utility laboratory and/or partner laboratories; ------- analytical results are used to supplement investigation of validated alerts from other components of the system. The design elements of the S&A component include 1) field and laboratory capabilities, 2) routine sampling and analysis and 3) incident response procedures. Field and laboratory capabilities (instrumentation, methods and laboratories) are those that would be used to perform screening and confirmatory analyses for a wide range of contaminants in a Possible contamination incident. Routine sampling and analysis is performed to establish baseline data for contaminant occurrence in the distribution system and to evaluate method performance for the field and laboratory methods implemented for the S&A component. Incident response sampling and analysis occurs when an alert from one of the monitoring and surveillance components is determined to be Possible water contamination. Procedures and protocols are developed in the incident response procedures design element The following subsections describe the five design objectives that provided a basis for evaluation of the S&A component, including spatial coverage, contaminant coverage, timeliness of response, operational reliability and sustainability. Each subsection includes a description of the design objective, and a summary of the data that was used to determine how well the component met the design objective. S&A data was collected from GCWW and their laboratory and emergency response partners. The design objectives of contaminant coverage and timeliness of response were evaluated using both empirical data and results from a computer model simulating the Cincinnati pilot. The simulation study allowed pilot performance to be evaluated in more than 2,000 different contamination scenarios. A scenario is defined as a simulation of a contamination incident using a specified contaminant at a pre-determined location, time, and injection rate. Contaminant coverage, operational reliability and sustainability for S&A were evaluated using empirical data from the Cincinnati pilot. For more information on this topic, see Section 2.0. Methodology Several methods were used to evaluate S&A performance. Data was tracked over time to illustrate the change in performance as the component evolved during the evaluation period. Statistical methods were also used to summarize large volumes of data collected over either the entire or various segments of the evaluation period. Data was also evaluated and summarized for each reporting period over the evaluation period. In this evaluation, the term reporting period is used to refer to one month of data that spans from the 16th of the indicated month to the 15th of the following month. Thus, the January 2008 reporting period refers to the data collected between January 16th 2008 and February 15th 2008. Additionally, six drills and two full-scale exercises designed around mock contamination incidents were used to practice and evaluate the full range of procedures, from initial detection through response. Because there were no contamination incidents during the evaluation period, there is no empirical data to fully evaluate the detection capabilities of the component. To fill this gap, a computer model of the Cincinnati CWS was developed and challenged with a large ensemble of simulated contamination incidents in a simulation study. An ensemble of 2,015 contamination scenarios representing a broad range of contaminants and injection locations throughout the distribution system was used to evaluate the effectiveness of the CWS in minimizing public health and utility infrastructure consequences. The simulations were also used for a benefit-cost analysis, which compares the monetized value of costs and benefits and calculates the net present value of the CWS. Costs include implementation costs and routine operation and maintenance labor and expenses, which were assumed over a 20 year lifecycle of the CWS. Benefits included reduction in consequences (illness, fatalities and infrastructure damage) and dual-use benefits from routine operations. IV ------- Design Objective: Spatial Coverage Spatial coverage includes the spatially diverse and hydraulically significant area of the GCWW distribution system covered through routine and incident response sampling and analysis. During a Possible contamination incident, samples can be collected from any location within the GCWW service area. To establish baseline contaminant occurrence and evaluate method performance throughout the distribution system, samples were routinely collected from 23 pre-identified sampling locations over a one-year period. The pre-identified locations were selected to be representative of different pressure and mixing zones, different source waters and extremes in water age. Many of the routine sampling locations were selected because they were locations equipped with water quality monitors or enhanced security surveillance equipment, and thus, could be the site of a water quality monitoring (WQM) or enhanced security monitoring component alert. Other locations were of strategic interest (tanks, reservoirs, etc). Additionally, a survey study of 54 different locations was performed over a two-month period. Following completion of baseline monitoring, the utility transitioned to maintenance monitoring and is continuing to collect samples from 31 strategic locations throughout the distribution system to maintain proficiency in field and laboratory methods and to update contaminant baseline data. For more information on this topic, see Section 4.0. Design Objective: Contaminant Coverage Contaminant coverage is the ability to detect a wide range of contaminants of concern to water security and to be able to detect these contaminants under a wide range of contamination scenarios. GCWW established baseline occurrence data for 32 different targeted priority contaminants using in-house field and laboratory capabilities or partner laboratories. Twelve contaminants for which GCWW had detection capabilities were evaluated in the simulation study in addition to five contaminants for which methods and laboratory partners were identified, but no baseline data was collected. The simulation study allowed evaluation of more than 2,000 different contamination scenarios to determine contamination scenario coverage. Contamination scenario coverage was calculated as the percent of simulated contamination scenarios that were detected either through analysis during site characterization (water quality parameter or rapid field testing) or laboratory analysis. Fiigher detection rates were observed for scenarios involving toxic chemicals with rapid symptom onset (>88% for site characterization analysis and >98% for laboratory analysis). Lower detection rates were observed for scenarios involving biological agents with delayed symptom onset (ranging from 12% to 72% for site characterization analyses and 23% to 72% for laboratory analyses). This is explained by the fact that sampling and analysis was never initiated for some scenarios, as the threat level never advanced to a Possible contamination determination, so by default no detection occurred. Other scenarios which involved biological agents with delayed symptom onset were first detected by the public health surveillance component, and sampling did not occur soon enough to capture a water sample containing the contaminant. This finding underscores the importance of a multi-component CWS which does not rely solely on public health surveillance for detection of drinking water contamination incidents, but involves multiple monitoring and surveillance components. For example, in many scenarios involving biological agents with delayed symptom onset, WQM detected contaminated water while it was still in the distribution system, allowing for the automated sampling devices at each WQM location to capture a sample that did contain detectable concentrations of the biological agent. In these scenarios, the contaminant was also detected during site characterization and/or laboratory analysis. For more information on this topic, see Section 5.0. ------- Design Objective: Timeliness of Response For the S&A component, timeliness of response is defined as a portion of the incident timeline that begins with the recognition of a Possible contamination incident and ends with a determination regarding whether or not the contamination is detected or confirmed by field or laboratory analyses. Based on data gathered during drills and exercises, it is estimated that the timeline for escalation of an incident, from recognition of a Possible incident to Credible determination would be between 9 hours and 1.5 days depending on the contaminant. Empirical timeline data was used to parameterize the simulation model, and therefore timeline data output closely matched the inputs. Simulation study results demonstrated a consistent timeline availability of results from site characterization following a Possible contamination determination (-2.5 to 3 hours) as the process is consistent regardless of the contaminant, though some time delays would occur if local HazMat response was activated. More variability in the timeline from a Possible contamination determination to availability of laboratory results was observed, and was expected due to differences in transport time to the GCWW laboratory vs. partner laboratories, and the time differences involved in analytical methods for toxic chemicals vs. biological agents. The time from the Possible contamination determination to laboratory results for most of the toxic chemicals ranged from ~8 to 15 hours, whereas for the biological agents, the time ranged from ~1 to 2 days. For more information on this topic, see Section 6.0. Design Objective: Operational Reliability Operational reliability quantifies the percent of time that the S&A component is available and producing complete and accurate data. Analysis of the operational reliability of the S&A component considers metrics including component availability, data completeness, method accuracy, and method precision. Empirical data collected during the evaluation period demonstrated the overall dependability of component operations. During the course of 26 months of maintenance monitoring, only one short period of downtime (13 hours) was experienced by the GCWW laboratory as a result of a severe weather incident. Furthermore, high data completeness percentages were recorded for each of the S&A sub- components (> 88% for one field and four laboratory sub-components). Finally, method accuracy and method precision data were within established method limits/tolerances during baseline monitoring for each of the methods and laboratories supporting the S&A component. For more information on this topic, see Section 7.0. Design Objective: Sustainability Sustainability is defined in terms of the cost-benefit trade-off. Empirical data as well as feedback documented during component forums were used to evaluate costs, benefits and ability of the utility to comply with procedures and sampling plans. Costs were estimated over the lifecycle of the system to provide an estimate of the total cost of ownership. Table ES-1 demonstrates the value of the major cost elements used to calculate the total lifecycle cost of the S&A component. It is important to note that the Cincinnati CWS was a pilot research project, and as such incurred higher costs than would be expected for atypical large utility installation. VI ------- Table ES-1. Cost Elements used in the Calculation of Lifecycle Cost Parameter Implementation Costs Annual O&M Costs Renewal and Replacement Costs1 Salvage Value1 Value $2,543,918 $42,795 $260,482 ($11,269) Calculated using major pieces of equipment. To calculate the total lifecycle cost of the S&A component, all costs and monetized benefits were adjusted to 2007 dollars using the change in the Consumer Price Index between 2007 and the year that the cost or benefit was realized. Subsequently, the implementation costs, renewal and replacement costs and annual operation and maintenance costs were combined to determine the total lifecycle cost: S&A Total Lifecyde Cost: $3,436,060 A similar S&A component implementation at another utility should be less expensive when compared to the Cincinnati pilot as it could benefit from lessons learned and would not incur research-related costs. The benefits of the S&A component at the Cincinnati pilot include: • Increased field and laboratory preparedness for responding to all hazard events • Improved working relationship with emergency response partners (HazMat) and partner laboratories (Ohio Department of Health and local contract laboratories) • Increased in-house field and laboratory analytical capabilities, including volatile gas and radiation meters, ultrafiltration concentration equipment, and a GC-MS for semi-volatile analyses • Better characterization of the distribution system with respect to contaminants of concern to water security • Improved procedures for incident response S&A The utility has absorbed the O&M cost for the component and has designated personnel to support ongoing sampling and analysis efforts associated with maintenance monitoring. This has allowed the utility to comply with the sample collection and analysis schedule designated for maintenance monitoring, which demonstrates acceptance and suggests sustainability of the S&A component. For more information on this topic, see Section 8.0. VII ------- Table of Contents LIST OF FIGURES x LIST OF TABLES x SECTION 1.0: INTRODUCTION 1 1.1 CONTAMINATION WARNING SYSTEM DESIGN OBJECTIVES 1 1.2 ROLE OF SAMPLING AND ANALYSIS IN THE CINCINNATI CWS 2 1.3 OBJECTIVES 2 1.4 DOCUMENT ORGANIZATION 3 SECTION 2.0: OVERVIEW OF THE SAMPLING & ANALYSIS COMPONENT 4 2.1 FIELD AND LABORATORY CAPABILITIES 6 2.2 ROUTINE SAMPLING AND ANALYSIS 8 2.2.1 Baseline Monitoring 8 2.2.2 Maintenance Monitoring 9 2.3 INCIDENT RESPONSE PROCEDURES 10 2.4 SUMMARY OF SIGNIFICANT S&A COMPONENT MODIFICATIONS 11 2.5 TIMELINE OF S&A DEVELOPMENT PHASES AND EVALUATION-RELATED ACTIVITIES 13 SECTION 3.0: METHODOLOGY 14 3.1 ANALYSIS OF EMPIRICAL DATA FROM ROUTINE OPERATIONS 14 3.2 DRILLS AND EXERCISES 14 3.2.1 S&A Drill 1 (May 7, 2008) 14 3.2.2 S&A Drill: July 15, 2008 15 3.2.3 Full Scale Exercise 2: October 1, 2008 15 3.2.4 S&A Drill: March 31,2009 16 3.2.5 S&A Drill: April 23, 2009 16 3.2.6 CCS /S&A Drill: September 16, 2009 16 3.2.7 Full Scale Exercise 3: October 1, 2009 16 3.2.8 BT'Agent S&A Drill: May 10, 2010 17 3.3 SIMULATION STUDY 17 3.4 FORUMS 22 3.5 ANALYSIS OF LIFECYCLE COSTS 22 SECTION 4.0: DESIGN OBJECTIVE - SPATIAL COVERAGE 24 4.1 SPATIAL COVERAGE 24 4.2 SUMMARY 25 SECTION 5.0: DESIGN OBJECTIVE - CONTAMINANT COVERAGE 27 5.1 CONTAMINANT DETECTION POTENTIAL 27 5.2 CONTAMINANT DETECTION LIMIT 28 5.3 CONTAMINATION SCENARIO COVERAGE 28 5.4 SUMMARY 32 SECTION 6.0: DESIGN OBJECTIVE - TIMELINESS OF RESPONSE 33 6.1 TIMELINE OF INCIDENT RESPONSE SAMPLING AND ANALYSIS 33 6.2 SUMMARY 42 SECTION 7.0: DESIGN OBJECTIVE - OPERATIONAL RELIABILITY 43 7.1 AVAILABILITY 43 7.2 DATA COMPLETENESS 44 7.3 METHOD ACCURACY 44 7.4 METHOD PRECISION 49 VIM ------- 7.5 SUMMARY 50 SECTION 8.0: DESIGN OBJECTIVE - SUSTAINABILITY 52 8.1 COSTS 52 8.2 BENEFITS 54 8.3 COMPLIANCE 55 8.4 SUMMARY 56 SECTION 9.0: SUMMARY AND CONCLUSIONS 57 SECTION 10.0: REFERENCES 59 SECTION 11.0: ABBREVIATIONS 60 SECTION 12.0: GLOSSARY 61 IX ------- List of Figures FIGURE 2-1. TIMELINE OF S&A COMPONENT ACTIVITIES 13 FIGURE 4-1. SURVEY SAMPLE SITES 25 List of Tables TABLE 2-1. SAMPLING AND ANALYSIS DESIGN ELEMENTS 4 TABLE 2-2. SAMPLING AND ANALYSIS ROLES AND RESPONSIBILITIES 5 TABLE 2-3. SAMPLING AND ANALYSIS - FIELD METHODS 7 TABLE 2-4. SAMPLING AND ANALYSIS - LABORATORY INSTRUMENTATION 7 TABLE 2-5. BASELINE MONITORING PHASES 8 TABLE 2-6. SIGNIFICANT S&A COMPONENT MODIFICATIONS 11 TABLE 3-1. S&A DRILL VARIATIONS 17 TABLE 3-2. FIELD TESTING CAPABILITIES 19 TABLE 3-3. LABORATORY TESTING CAPABILITIES 21 TABLE 4-1. STRATEGIC AND PRIORITY SAMPLE LOCATIONS FOR BASELINE MONITORING 24 TABLE 5-1. TARGET WATER QUALITY PARAMETERS FOR BASELINE MONITORING AT GCWW 27 TABLE 5-2. CONTAMINANT CLASSES FOR BASELINE MONITORING AT GCWW 28 TABLE 5-3. SITE CHARACTERIZATION DETECTION STATISTICS 29 TABLE 5-4. LABORATORY ANALYSIS DETECTION STATISTICS 31 TABLE 6-1. S&A TIMELINESS DURING DRILLS AND EXERCISES 35 TABLE 6-2. S&A TIMELINE ESTIMATES FOR INCIDENT RESPONSE 36 TABLE 6-3. S&A TIMELINE ANALYSIS (SIMULATION STUDY RESULTS) 36 TABLE 6-4. SCENARIOS WITH WATER QUALITY PARAMETER RESULTS AVAILABLE PRIOR TO CREDIBLE AND CONFIRMED CONTAMINATION DETERMINATION (SIMULATION STUDY RESULTS) 37 TABLE 6-5. SCENARIOS WITH RAPID FIELD TEST RESULTS AVAILABLE PRIOR TO CREDIBLE AND CONFIRMED CONTAMINATION DETERMINATION (SIMULATION STUDY RESULTS) 38 TABLE 6-6. SCENARIOS WITH LABORATORY RESULTS AVAILABLE PRIOR TO CREDIBLE AND CONFIRMED CONTAMINATION DETERMINATION (SIMULATION STUDY RESULTS) 39 TABLE 6-7. SCENARIOS WITH WATER QUALITY PARAMETER RESULTS PRIOR TO PUBLIC HEALTH RESPONSE (SIMULATION STUDY RESULTS) 40 TABLE 6-8. SCENARIOS WITH RAPID FIELD TEST RESULTS PRIOR TO PUBLIC HEALTH RESPONSE (SIMULATION STUDY RESULTS) 40 TABLE 6-9. SCENARIOS WITH LABORATORY RESULTS PRIOR TO PUBLIC HEALTH RESPONSE (SIMULATION STUDY RESULTS) 41 TABLE 7-1. S&A SUB-COMPONENT AVAILABILITY 44 TABLE 7-2. S&A SUB-COMPONENT DATA COMPLETENESS 44 TABLE 7-3. FIELD INSTRUMENT ACCURACY ESTIMATES AND QC CHECK FREQUENCY 45 ------- TABLE 7-4. 2007 PROFICIENCY TESTING SAMPLE RESULTS 46 TABLE 7-5. 2009 PROFICIENCY TESTING SAMPLE RESULTS 47 TABLE 7-6. SUMMARY OF GCWW FILTER CONCENTRATION RECOVERY TRIALS 48 TABLE 7-7. METHOD-SPECIFIC RECOVERY QC CRITERIA FOR EACH CONTAMINANT CLASS 49 TABLE 7-8. FIELD INSTRUMENT PRECISION ESTIMATES 50 TABLE 7-9. METHOD-SPECIFIC PRECISION QC CRITERIA FOR EACH CONTAMINANT CLASS 50 TABLE 8-1. COST ELEMENTS USED IN THE CALCULATION OF LIFECYCLE COST 52 TABLE 8-2. IMPLEMENTATION COSTS 53 TABLE 8-3. ANNUALO&M COSTS 54 XI ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 1.0: Introduction The purpose of this document is to present the results of evaluation of the sampling and analysis (S&A) component of the Water Security Initiative (WSI) contamination warning system (CWS) pilot project at the Greater Cincinnati Water Works (GCWW). The evaluation covers the period March 2008 to June 2010 when the S&A component was fully operational. This evaluation was implemented by examining the performance of the S&A component relative to the design objectives established for the CWS. 1.1 Contamination Warning System Design Objectives The goal of a CWS is to detect possible water contamination in a timely manner so that consequences can be mitigated through operational responses. Early detection is accomplished by using an integrated system of monitoring and surveillance components. S&A is not an early detection component; however, it is critical to consequence management. To determine the efficacy of the Cincinnati CWS, performance was evaluated against the following CWS design objectives as applied to the S&A component: • Spatial Coverage. The objective for spatial coverage is to ensure that S&A response capabilities extend throughout the distribution system and to the entire population served by the drinking water utility. The degree of coverage depends on the location and density of potential sampling points in the distribution system, and the hydraulic connectivity of each monitoring location to downstream regions and populations. Spatial coverage includes the spatially diverse and hydraulically significant area covered through routine sampling and analysis to establish contaminant occurrence and method performance throughout the distribution system. Metrics evaluated under this design objective include: number of samples collected at various locations during baseline monitoring and the rationale for location selection. Potential sampling locations during incident response may be anywhere in the distribution system, and are not limited to the locations used for routine sampling and analysis. • Contaminant Coverage. Prior to the Cincinnati CWS project, an interagency research and analysis effort identified more than 200 contaminants that could cause serious harm if introduced into a drinking water distribution system. These contaminants were prioritized based on their toxic/infectious dose, stability in water, and availability. Contaminant selection for baseline monitoring was designed to achieve broad coverage of the contaminant classes of concern that these prioritized contaminants represented with a sub-set of chemicals, radiochemicals, pathogens and biotoxins selected based upon availability of analytical methods for the drinking water matrix. Metrics used to assess contaminant coverage of the S&A component include: contaminant detection potential, contaminant detection limit, and contamination scenario coverage. • Timeliness of Response. A key objective of a CWS is to provide initial detection and validation of a contamination incident in a timeframe that allows for the implementation of response actions that result in a significant reduction in consequences. For the S&A component, timeliness of response is defined as a portion of the incident timeline that begins with the recognition of a Possible contamination incident and ends with a determination regarding whether or not the contamination is detected or confirmed by field or laboratory analyses. This metric is only applied to incident response S&A and not routine sampling. Metrics associated with timeliness of response include: time for response partner notification, time for Site Characterization Team mobilization and deployment, time for site approach and field safety screening, time for sample collection, time for rapid field testing, time for sample preparation and transport, time for sample disposition, time for laboratory mobilization, time for laboratory sample analysis and time for data review and reporting. The metric 'time for response partner notification' is characterized in ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System a separate document, Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot (USEPA, 2014b), while the remaining timeline metrics are discussed in this document. • Operational Reliability. Analysis of the operational reliability of the S&A component considers metrics that quantify the overall availability and functionality during routine operation of the S&A component. Metrics used to assess the operational reliability of the S&A component include: method/instrument/laboratory availability, data completeness, method accuracy and method precision. • Sustainability. Sustainability of the S&A component is dependent upon the overall acceptability to the utility, which is a function of the perceived cost-benefit trade-off. Metrics used to assess Sustainability of the S&A component include: lifecycle costs, benefits (primary and dual-use) and compliance with component operational requirements. The design objectives provide a basis for evaluation of each component as well as the entire integrated system. Because the deployment of drinking water CWSs is a new concept, design standards or benchmarks are unavailable. Thus, it is necessary to evaluate CWS components against the design objectives on a relative scale. This includes evaluation of the deployed component relative to the baseline state of the utility prior to deployment, as well as evaluation of the components relative to each other. 1.2 Role of Sampling and Analysis in the Cincinnati CWS Under the WSI, a multi-component design was developed to meet the above CWS design objectives. Specifically, the WSI CWS architecture utilizes four monitoring and surveillance components common to the drinking water industry and public health sector: water quality monitoring (WQM), enhanced security monitoring (ESM), customer complaint surveillance (CCS) and public health surveillance (PHS). Information from these four components is integrated under the Cincinnati Pilot Consequence Management Plan to establish the credibility of possible contamination incidents and to inform response actions intended to mitigate consequences. Although not an early detection component, S&A plays a critical role in the CWS due to the potential to confirm or rule out contaminants in drinking water samples collected throughout the pilot utility's distribution system during investigation of validated CWS component alerts as part of the credibility determination process. Unlike the CWS monitoring and surveillance components, the S&A component affords the potential to identify specific contaminants and, in many instances, determine the concentration of these contaminants in drinking water. During a potential contamination incident, drinking water samples are collected and analyzed with the goal of confirming or ruling out the presence of specific contaminants or contaminant classes (i.e., incident response sampling and analysis). Even though results from sample analyses may not be available until several hours or longer after sample collection, S&A is critical for corroborating validated alerts from the monitoring and surveillance components and in possible attribution of illness or other adverse consequences to drinking water contamination. In addition to sample collection and laboratory analysis, the S&A component includes the site characterization activities of site safety screening and rapid field testing of water samples. These S&A activities provide site safety information for ensuring worker protection and can inform or focus subsequent sample collection and analysis. 1.3 Objectives The overall objective of this report is to evaluate how well the S&A component functioned as part of the CWS deployed in Cincinnati (i.e., how effectively the component achieved the design objectives). It will ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System also characterize factors that impact the sustainability of S&A in a CWS. This evaluation will cover the component as a whole, as well as the individual S&A design elements as described in Section 2.0. Data collection during baseline monitoring, routine operation, drills and exercises and computer simulations yielded sufficient data to evaluate performance of the S&A component for each of the stated design objectives. The data sources used in the evaluation are presented in Section 3.0. In summary, this document will discuss the approach for analysis of this information and present the results that characterize the overall operation, performance, and sustainability of the S&A component as part of the Cincinnati CWS. 1.4 Document Organization This document contains the following sections: • Section 2: Overview of the S&A Component. This section introduces the S&A component of the Cincinnati CWS and describes each of the major design elements that make up the component. A summary of significant modifications to the component, made as a result of experience gained during the pilot which had a demonstrable impact on performance, is presented at the end of this section. • Section 3: Methodology. This section describes the data sources and techniques used to evaluate the S&A component. • Sections 4 through 8: Evaluation of S&A Performance against the Design Objectives. Each of these sections addresses one of the design objectives listed in Section 1.1. Each section begins with the definition of the subject design objective in the context of the S&A component and introduces the metrics that will be used to evaluate the component against that design objective. Each supporting evaluation metric is discussed in a dedicated subsection, including an overview of the analysis methodology employed for that metric followed by presentation and discussion of the results. Each section concludes with a summary of the evaluation of the subject design objective. • Section 9: Summary and Conclusions. This section provides a high-level assessment of how well the S&A component of the Cincinnati CWS met the design objectives. • Section 10: References. This section lists all sources and documents cited throughout this report. • Section 11: Abbreviations. This section lists all acronyms approved for use in the S&A component evaluation. • Section 12: Glossary. This section defines terms used throughout the S&A component evaluation. ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 2.0: Overview of the Sampling & Analysis Component The S&A component involves the collection, testing, and interpretation of results from GCWW water samples collected as part of routine and incident response sampling and analysis. The sampling and analysis activities were performed by qualified field and laboratory personnel at GCWW, the Cincinnati Fire Department (CFD) and Greater Cincinnati Hazardous Materials (HazMat) units, the Ohio Department of Health (ODH) State laboratory and contract laboratories. The S&A component, as deployed, was a result of modifications and expansions to existing GCWW capabilities and protocols as identified by a multifaceted evaluation and assessment process. A summary description of the pre- and post-implementation status of the S&A component can be found in Water Security Initiative: Cincinnati Pilot Post-Implementation System Status (USEPA, 2008). Two types of sampling are performed as part of the S&A CWS component: routine (baseline and maintenance monitoring phases) and incident response sampling and analysis. Routine sampling during the baseline monitoring phase was designed to determine contaminant occurrence and method performance under normal circumstances using a suite of methods that could be used during incident response sampling and analysis (baseline methods). During the maintenance monitoring phase, routine sampling confirms there are no changes in baseline contaminant occurrence or method performance during normal (i.e., non-incident) sampling and analysis. Routine monitoring also allows for the practice and refinement of sampling protocols. This process began during the initial phase of the Cincinnati pilot, and continues as part of the ongoing CWS project. In contrast, incident response samples are collected in response to validated alerts from the monitoring and surveillance components (WQM, ESM, CCS and PHS) during consequence management as part of the credibility determination process. For methods with previously established baseline data, incident response S&A results (contaminant occurrence and method performance) are compared and results exceeding baseline data are reported for possible utility response action. A description of the three S&A design elements is shown in Table 2-1, though the design elements are described more fully in Sections 2.1 through 2.3. Table 2-1. Sampling and Analysis Design Elements Design Element 1. Field and laboratory capabilities 2. Routine sampling and analysis 3. Incident response procedures Description Build field and laboratory capability and capacity that would be necessary to perform screening and confirmatory analyses for a wide range of contaminants in a possible contamination incident. Select sampling locations, frequencies, quality assurance, and data quality objectives for routine sampling and analysis to establish baseline data for contaminant occurrence in the distribution system and to evaluate method performance. Establish roles, responsibilities, and procedures that will be used by the utility and others investigating a potential contamination incident. It should be noted that the titles of these design elements are different than those used in the Water Security Initiative: Cincinnati Pilot Post-Implementation System Status (USEPA, 2008). The design elements were modified to more accurately present the approach used to design the S&A component. Many users within different job functions are involved in building and operating the S&A component with the above design elements. Table 2-2 describes the various job functions of those directly involved in the operation of the S&A component of the Cincinnati CWS, including GCWW personnel, contract laboratories and partner agencies. Although not involved directly with S&A activities, the GCWW Water ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Utility Emergency Response Manager (WUERM) initiates all S&A response actions following the determination of Possible contamination. Table 2-2. Sampling and Analysis Roles and Responsibilities Personnel/Organization Laboratory Program Manager Water Quality & Treatment Chemist Site Characterization Team Leader Site Characterization Team Distribution Valve Operator Cincinnati Fire Department or Greater Cincinnati HazMat Units California Control Operator Water Quality and Treatment Shift Chemist Ohio Department of Health (ODH) Role in Sampling and Analysis Component • Coordinates sample flow and laboratory analysis of routine samples • Informs the appropriate staff of any non-routine sampling needs • Performs data review and updates baseline control charts • Compares sample results to baseline control charts • Provides advice regarding the chemical and microbiological analyses of samples • Coordinates the delivery of samples to external labs • Receives lab data and reports it to the WUERM • Maintains baseline and maintenance monitoring field method data • Performs laboratory analysis as needed • Reports field analysis results to the WUERM and Laboratory Program Manager to help guide sample flow and laboratory analysis • In conjunction with the WUERM develops and implements situation-specific site characterization and sampling plan for sample collection and field testing • Manages and leads the site characterization and sampling teams according to the Site Characterization Plan • Implements the site characterization Standard Operating Procedures from the GCWW manual titled Standard Operating Procedures for Site Characterization and Sampling • Reports site characterization results to WUERM and consults regarding specific response needs • Functions as on-site coordinator with HazMat and other emergency responders • Performs site characterization and sampling activities as directed by the Site Characterization Team Leader • Performs site characterization and sampling activities as directed by the Site Characterization Team Leader • Performs site characterization and sampling (as required) • Monitors Supervisory Control and Data Acquisition alerts, and reviews operational data to support the investigation of alerts • Assumes CWS responsibilities of Water Quality and Treatment Chemist during off-hours; support sample analysis • Performs screening and confirmatory analyses for radiochemical analyses per regulatory schedule and for bioterrorism threat (BT) agents during incident response sampling and analysis ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Personnel/Organization GCWW Laboratory Contract Laboratory Role in Sampling and Analysis Component • Performs confirmatory analyses for semi-volatiles • Performs semi-volatiles screening of routine samples • Performs confirmatory analyses for volatile constituents of gasoline fuel in routine samples • Performs volatiles screening of routine samples • Performs screening for metals and • Performs contingency or surge analyses of samples for volatiles, semi- volatiles, metals, carbamates and total cyanide 2.1 Field and Laboratory Capabilities Field and laboratory testing capabilities were built to provide analytical capabilities for a baseline suite of contaminants and methods. Laboratories were selected to support baseline monitoring, maintenance monitoring, and incident response sampling and analysis. Contaminants and methods were selected during design of the S&A component to provide wide contaminant coverage using readily available screening and confirmatory methods. The rationale for selection of contaminants and methods is presented as guidance for utilities in the document Water Security Initiative: Guidance for Building Laboratory Capabilities to Respond to Drinking Water Contamination (USEPA, 2013). In order to build effective field and laboratory testing capabilities for response to a wide range of contamination scenarios, enhancements of existing GCWW capabilities were implemented and new capabilities were acquired. When possible, in-house enhancements were provided to GCWW in the form of equipment and/or training opportunities. Through a combination of enhancements to GCWW's field and laboratory capabilities, partnering with the CFD and Greater Cincinnati HazMat units and the ODH laboratory, and contracting with commercial laboratories, a laboratory network was established with broad detection capabilities for chemical, radiochemical and biological contaminants. A complete description of equipment enhancements, laboratory capabilities, and agreements can be found in Water Security Initiative: Cincinnati Pilot Post-Implementation System Status (USEPA, 2008). Tables 2-3 and 2-4 present the field and laboratory testing capabilities used to support baseline and maintenance monitoring and incident response sampling and analysis. ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Table 2-3. Sampling and Analysis - Field Methods Safety Screening Contaminant Class Radioactivity (alpha, beta, and gamma) General hazards Volatile organic compounds (VOC) and combustible gases Methodology Hand-held device HazCat (explosives, oxidants, etc.) Hand-held device Comments May be expanded to water testing with a special probe and procedure Should be performed by trained HazMat responder Detects chemicals in air Rapid Field Testing Contaminant Class Cyanide Chlorine residual pH/conductivity/ORP Turbidity Chemical Warfare Agents (VX, sarin, etc.) General toxicity Arsenic Methodology Portable colorimeter Portable colorimeter Portable electrochemical detector Portable turbidimeter Test Kit Test Kit Test Kit Comments Tests water for cyanide ion, but not combined forms Absence of residual may indicate a problem Abnormal pH or conductivity may indicate a problem High turbidity may indicate a problem May also detect some pesticides and common chemicals Only used as an optional screening procedure during incident response due to poor interpretive value at GCWW Rapid, easy to use Table 2-4. Sampling and Anal) Contaminant Class VOCs indicative of gasoline (i.e., BTEX) Semi-volatile organic compounds (SVOCs) Metals Carbamate Pesticides Total Cyanide* Total Organic Carbon* Radiochemicals BT Agents: Select agents and toxins /sis - Laboratory Instrumentation Instrumentation Gas Chromatography with Mass Spectrometry Detection (GC-MS) using purge and trap Gas Chromatography with Mass Spectrometry Detection using liquid-solid extraction Inductively Coupled Plasma - Mass Spectrometry (ICP-MS) Direct injection + high-performance liquid Chromatography (HPLC) with Post Column Derivatization and Fluorescence Detection Colorimetry with Reflux Distillation Extraction Persulfate-ultraviolet Spectrophotometry Alpha Beta Scintillation Sealer or Gas Flow Low- Background Proportional Detector High Purity Germanium Gamma Spectrometry Real-time polymerase chain reaction (PCR) and immunoassay Not used during baseline monitoring, but is a current capability. ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System 2.2 Routine Sampling and Analysis Routine S&A encompasses activities within both baseline and maintenance monitoring. GCWW's distribution system is complex with two treatment plants: Richard Miller Treatment Plant (Miller) and the Charles M. Bolton Treatment Plant (Bolton), which feed into a common distribution system. A baseline monitoring program was initiated to address specific questions about water in the GCWW system regarding differences in contaminant occurrence and method performance between GCWW's two water treatment plants which have different source waters, differences between sampling locations in the distribution system, temporal trends and water age, the effects of distribution system materials and other factors. Once initial contaminant baselines were established, ongoing maintenance monitoring was performed to update and maintain baseline data, and to maintain incident response sampling and analysis capabilities. Baseline and maintenance monitoring activities were accomplished using field and laboratory capabilities as described in Section 2.1. 2.2.1 Baseline Monitoring Baseline monitoring is a special purpose contaminant monitoring program that is intended to establish baseline occurrence of contaminants in the distribution system using methods that would be used during incident response sampling and analysis. The frequency of sample collection and the limited number of samples collected made it unlikely that baseline monitoring would detect a transient, localized contamination incident. The objectives of baseline monitoring at the Cincinnati pilot were to 1) establish and ensure ongoing laboratory preparedness for incident response, 2) establish baseline contaminant occurrence (levels and frequency of detections) and method performance (interferences, precision, accuracy as percent recovery) in the distribution system and (3) provide information for developing a long-term maintenance monitoring program. To accomplish these objectives, a phased approach to sample collection and analysis was developed. An overview of these phases can be found in Table 2-5. Table 2-5. Baseline Monitoring Phases Phase Sampling Phases Evaluation and Data Analysis Phase 1 Phase 2 Phase 3 Phase 4 Phase 5 Phase 6 Title Initial demonstration of capability Comparison of finished water from treatment plants Monthly monitoring of strategic sampling locations Survey study of the distribution system Focused distribution system studies Data analysis and recommendation for maintenance monitoring Description Development of standard operating procedures, establishment of precision, accuracy, and reporting limits. Analysis of spiked water samples from two treatment plants over a one month period to establish initial quality control (QC) limits for water not subjected to distribution system conditions. Regular sampling and analysis of strategic locations for one year to monitor contaminant occurrence and method performance overtime. Sampling and analysis from 54 locations spatially distributed and not previously sampled to determine if contaminant occurrence and method performance is different from treatment plants. Based on previous phases of baseline monitoring, determine if additional studies are needed and perform them. Perform exploratory data analysis, compile summary statistics, and perform statistical analysis to detect differences and trends. ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System The focus of baseline monitoring at GCWW was to determine background concentrations of priority contaminants in the drinking water during routine operation and performance of field and laboratory instrumentation and methods. Additional discussion of targeted water quality parameters and priority contaminants is presented in Section 5.1. It should be noted that while baseline monitoring followed the general approach described in Table 2-5, some contaminants were monitored at a different frequency because of practical limitations (e.g., analysis costs, personnel limitations, etc.). A more detailed description of the phased approach is provided below. Phase 1: Phase 1 results included the development of standard operating procedures and necessary resource documents for critical activities related to baseline monitoring. An initial demonstration of capability (IDC) was performed to establish analyst proficiency, method performance (precision, accuracy, and recovery) and minimum reporting limits for each method, where applicable. Data reporting requirements and protocols were also established. Phase 2: Phase 2 monitoring was conducted to determine if the finished waters from the two treatment plants and source waters are different with respect to contaminant occurrence or method performance. During Phase 2 sampling and analysis, no SVOC or VOC priority contaminants and no BT agents were detected in either of the two treatment plant waters. Significant differences in matrix spike recoveries between treatment plants were observed for two of the carbamates and one metal analyte. Phase 3: Regular surveillance monitoring of eighteen strategic locations and five priority locations was conducted at regular intervals for one year to establish baseline data for these locations and to determine if there were seasonal or regional trends. Long-term monitoring of strategic and priority sites revealed some seasonal trends for detectable non-priority contaminants and water quality parameters; none of the priority contaminants were detected above the minimum reporting limit or with sufficient frequency to perform trend analysis. Phase 4: Survey sampling of sites in the GCWW distribution system was performed during Phase 4 to determine target analyte occurrence and to evaluate method performance in water collected from various locations in the GCWW distribution system. Phase 4 survey samples for chemical analyses and field screening were collected from 54 different sites throughout the distribution system between May 2007 and June 2007. Only a few samples were collected from each of two sites for radiochemical analyses during Phase 4. Only two survey sites were monitored for BT agents during June 2007. None of the priority contaminants were detected above the minimum reporting limit. Three priority contaminant spikes were outside the recovery control limits established in Phase 2 but were within method QC limits. Phase 5: No focused studies were performed as a result of lessons learned from baseline monitoring. Phase 6: The final phase of baseline monitoring was the analysis of results from Phases 1 - 4 to establish the management, interpretation and use of baseline data at the Cincinnati pilot and to establish a maintenance monitoring program. Baseline monitoring results were used to construct a database of priority and non-priority contaminant occurrence at numerous sites throughout the distribution system. Control charts indicating site-specific contaminant levels and matrix-dependent analyte spike recoveries were developed and will be maintained through long-term maintenance monitoring. This information will be valuable for interpreting analytical results during incident response sampling and analysis. 2.2.2 Maintenance Monitoring When the one-year baseline monitoring period concluded, GCWW implemented a maintenance monitoring schedule that would allow them to maintain proficiency in methods and update contaminant baseline data. The location and schedule for maintenance monitoring sampling was based on baseline ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System results, historical data, regulatory requirements, and sustainability concerns. Maintenance monitoring began in April 2008; because maintenance monitoring is based in part upon sustainability requirements, it is planned to continue for the foreseeable future. 2.3 Incident Response Procedures Incident response sampling and analysis encompasses activities associated with field safety screening, sample collection, rapid field testing, sub-sampling, sample transport, chain-of-custody, laboratory coordination, sample analysis, data review, and results reporting. Initiation of these activities is contingent on a Possible contamination determination originating from one or more CWS validated component alerts. Incident response involves GCWW personnel and emergency response partners (e.g., CFD HazMat) for site characterization as well as partner laboratories described in Section 2.1 for sample analysis. Several activities conducted during incident response sampling and analysis drills and exercises are discussed in more detail in the Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot (USEPA, 2014b). Incident response sampling and analysis differs from routine monitoring in that the goal is to investigate the nature of contamination and to confirm or rule out specific contaminants and contaminant classes during Possible contamination incidents (i.e. those arising from validated CWS component alerts). Since contamination incidents are rare, regular practice of procedures via drills and exercise serves to familiarize personnel with protocols, identify potential procedural refinements and provide an opportunity to collect performance metrics to evaluate timeliness of response. As part of the S&A component implementation, standard operating procedures for field safety screening, sampling and rapid field testing were developed, with input from local HazMat response teams. These standard operating procedures are available to all utility and response partners in GCWW's manual, Standard Operating Procedures for Site Characterization and Sampling. Procedures cover activities including: • Pre-sampling guidelines from drinking water sources • Communications and results reporting • Sample container labeling, packaging, and chain-of-custody • Decontamination of personnel and equipment • Sampling from accessible water taps • Sampling from fire hydrants • Sampling from water towers • Sampling from underground tanks or reservoirs • Sampling from WQM stations • Sub-sampling from grab samples • Use of field safety equipment • Use of rapid field test kits Following a Possible contamination determination, site characterization is performed to assess the safety of the site where samples will be collected. There are two conditions under which the Site Characterization Team operates: "low hazard", where sampling and rapid field testing can be conducted 10 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System by GCWW, or "high hazard," where sampling and/or rapid field testing are better conducted by a HazMat response team. Based on the hazard level assessment made by the WUERM, sampling by the appropriate parties can be performed as outlined in GCWW's manual, Standard Operating Procedures for Site Characterization and Sampling. A complimentary two-volume set titled, Field Guide for Water Emergencies, was also developed. These guides contained point-of-use standard operating procedures for sampling, water quality parameter and rapid field testing and were printed in large font and laminated to be durable and field-friendly. They were designed to be used by GCWW field personnel and HazMat partners during incident response sampling and analysis. A Laboratory Response Plan (Laboratory Response Plan for Water Security Incidents at the Greater Cincinnati Water Works} was also developed to ensure effective and efficient coordination of sampling, sample analyses and reporting of results during incident response. The plan includes procedures for notifying laboratory partners, description of the baseline suite of methods, chain-of-custody, shipping samples to outside laboratories, analysis of samples, data review, and results reporting to the WUERM. It also contains information on identifying laboratories for non-baseline suite methods. Coordination of laboratory analyses is primarily the responsibility of the GCWW Laboratory Program Manager. The Laboratory Program Manager is responsible for notifying the appropriate external (contract) and GCWW laboratories, along with alerting response partners that sample analyses will be required once informed by the WUERM. In situations where radiochemical or BT agent analyses are requested, the Laboratory Program Manager notifies the Cincinnati Health Department (CHD), and CHD then relays the information to the appropriate ODH Laboratory. The Laboratory Program Manager ensures that proper chain-of-custody forms are utilized to keep track of samples as they are received from the field and sent to the appropriate laboratories for analysis, and also verifies that samples were received by the laboratories. The Laboratory Program Manager also ensures that any required sample submission forms, analytical request forms or other sample information accompanies the samples. Data review and reporting procedures for incident response are significantly different than those used for routine and compliance monitoring. The Laboratory Program Manager is the primary contact for receiving analytical results from in-house or external laboratories or through partner agencies (e.g., CHD). The Laboratory Program Manager reviews all analytical data to ensure that appropriate QC supports the sample results prior to reporting to the WUERM. In the event a contaminant is detected, the Laboratory Program Manager evaluates the results in comparison with the baseline database prior to reporting to the WUERM. 2.4 Summary of Significant S&A Component Modifications Component modifications were implemented to refine the S&A component (to field and laboratory testing capabilities and to routine and incident response sampling and analysis procedures) during the evaluation period. These modifications are summarized in Table 2-6 and will serve as a reference when discussing the results of the evaluation presented in Sections 4.0 through 8.0. Additional description of some of the modifications is provided below. Table 2-6. Significant S&A Component Modifications ID Component Modification Date Field and Laboratory Capability Modifications 1.1 Modification A contract was established by GCWW with Test America, Savannah for volatiles, semi-volatiles, metals, carbamates and total cyanide analyses as required to support maintenance monitoring or incident response S&A. August 2008 11 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System ID 1.2 Component Modification Cause Modification Cause Contract with Mobile Analytical Services, Inc (MASI) for carbamate pesticide analysis discontinued due to completion of baseline monitoring. The toxicity test kit was eliminated by GCWW during site characterization and will only be used as an optional screening procedure during incident response. Also, the test will only be performed in the laboratory of the Richard Miller Treatment Plant, instead of in the field. During baseline monitoring, it was found that the assay response when conducted in the field by various field personnel was high and variable which made it an unreliable diagnostic indicator of potential contaminants. Date November 2008 Routine Sampling and Analysis Modifications 2.1 2.2 Modification Cause Modification Cause Ultrafiltration of bulk samples for routine or incident response samples was eliminated by GCWW. Instead, bulk samples (10-100 L samples) will be transported to ODH for Ultrafiltration and analysis. The GCWW laboratory does not have the appropriate biosafety level to process samples potentially containing BT agents, and would not be able to safely concentrate bulk samples for BT agent screening in an emergency. A shift from baseline monitoring to maintenance monitoring was implemented as part of routine sampling activities. Maintenance monitoring required less frequent sampling and analysis than baseline monitoring. GCWW and EPA agreed that baseline monitoring would only last for one year, after which maintenance monitoring would commence. February 2008 February 2008 - April 2008 Incident Response Procedures Modifications 3.1 3.2 3.3 3.4 Modification Cause Modification Cause Modification Cause Modification A change in the headspace VOCs standard operating procedure for field analysis using the test instrument was implemented. As originally described in the standard operating procedure, sub-samples should be taken from large volume grab samples to perform headspace VOC measurement and the subsamples are shaken to encourage volatilization. In the modified standard operating procedure, a sub- sample is not prepared, nor is the large volume sample shaken. Instead the probe is used to measure VOCs immediately upon opening of the large volume sample container. GCWW adopted this alternative headspace VOCs procedure because they believed it to be equally informative. HazMat roles and responsibilities were modified such that HazMat units will continue to provide support during high hazard incident response but GCWW will maintain all water testing equipment and supplies. GCWW may call on HazMat when a site and/or a water sample are deemed hazardous. GCWW will conduct field screening tests during low hazard incident response. During drills and exercises conducted at GCWW, participants and observers determined that high hazard situations would require the support of trained HazMat units. Multiple revisions were implemented to field safety screening, rapid field testing, and sampling standard operating procedures. Site characterization and sampling drills and exercises demonstrated that the standard operating procedures required modifications to reflect what was actually done in the field and that standard operating procedures could be better written for use by HazMat teams. A Laboratory Response Plan was developed for GCWW to provide a procedures manual to facilitate smooth laboratory operations during incident response sampling events (e.g., sample receipt and disposition, COC, results reporting). May 2008 January 2009 November 2008 -June 2009 March - May 2009 12 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System ID 3.5 3.6 Component Modification Cause Modification Cause Modification Cause Need for standardized procedures for the Laboratory Program Manager and chemists at GCWW. User-friendly (condensed) and field-friendly (laminated) standard operating procedures (Field Guide for Water Emergencies) were developed for site characterization and field screening and rapid field testing (Volume 1) and sampling activities (Volume II). Need for portable, point of use standard operating procedures that are durable, and user friendly. Sample volume collection was reduced for BT agent analysis from 100 L to 20 L to reflect practical feasibility of sample collection at multiple GCWW sites. It is not feasible for GCWW to collect 1 00 L per sample location when it is anticipated that there would be multiple locations to collect from during a Possible contamination incident. Date March -June 2009 March 2010 2.5 Timeline of S&A Development Phases and Evaluation-related Activities Figure 2-1 presents a summary timeline for deployment of the S&A component, including milestone dates indicating when significant component modifications and drill and exercise evaluation activities took place. The timeline also shows the completion date for design and implementation, along with the subsequent optimization and real-time monitoring phases of deployment. Aug-08 External Analytical Services Contract Established Design & Implementation Complete Jan-08 Jan-08 May-09 Laboratory Response Plan Developed I Jan-09 / Jun-09 HazMat Unit Roles and / User- and Field-Friendly Responsibilities Modified / SOPs Developed FSE2 Oct-08 Wind Storm Sep-08 Jun-10 CCS/S&A Drill Sep-09 FSE3 Oct-09 Drill 3 Mar-09 Drill 4 Apr-09 BT Agent Drill May-10 Optimization Jan-08 - Mar-08 Real-time Monitoring Apr-OS -Jun-10 End of Data Collection Jun-10 Figure 2-1. Timeline of S&A Component Activities 13 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 3.0: Methodology The following section describes the evaluation techniques that were applied to the S&A component. The analysis of the S&A component was conducted using four evaluation techniques: empirical data from routine operations, results from drills and exercises, a simulation study, findings from forums such as lessons learned workshops and results of an analysis of lifecycle costs. 3.1 Analysis of Empirical Data from Routine Operations This evaluation includes data on the performance, operation, and sustainability of the S&A component from March 16, 2008 to June 15, 2010. In this evaluation, the term 'reporting period' is used to refer to a month of data which spans from the 16th of one month to the 15th of the next month. Thus, the March 2008 reporting period refers to the data collected between March 16th, 2008 and April 15th, 2008. Baseline monitoring data is another source of empirical data used within this component evaluation to characterize the component, including contaminant detection potential, method accuracy, and method precision for the target analytes and analytical methods identified for baseline monitoring. 3.2 Drills and Exercises Drills and exercises were conducted to characterize key aspects of component performance for particular activities that cannot be characterized via routine sampling activities. Findings from drills and exercises were used to evaluate the incident response sampling and analysis process, as conducted by participant personnel, and to determine whether procedures were followed correctly and in a timely manner. Drills and exercises also provided an opportunity to identify component modifications required to be more consistent with observed sampling and analysis procedures or to create more sustainable protocols. All of the drills and exercises that were designed to test and evaluate the Cincinnati pilot were compliant with Homeland Security Exercise and Evaluation Program guidelines (DHS, 2013), and performance data was captured to allow documentation of improvements in the response timeline. The results from the drills and exercises were used to evaluate the timeliness of response design objective. Brief descriptions of eight drills and exercises conducted for the purpose of component evaluation are provided below: • S&A Drill 1 (May 7, 2008) • S&A Drill 2 (July 15,2008) • CWS Full Scale Exercise 2 (October 1,2008) • S&A Drill 3 (March 31,2009) • S&A Drill 4 (April 23, 2009) • CCS/S&A Drill (September 16, 2009) • CWS Full Scale Exercise 3 (October 1, 2009) • S&A BT Agent Drill (May 10, 2010) 3.2.1 S&A Drill 1 (May 7, 2008) Description: GCWW personnel initiated a three-day drill (May 7th through 9th) designed to evaluate incident response sampling and analysis, along with related consequence management activities. The drill consisted of three phases, each with unique activities and objectives: 14 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System • Phase 1: Site Characterization Team Deployment, Sampling, and Field Screening o This phase was designed to exercise and evaluate GCWW's response to a simulated alert originating from WQM, including site characterization practice of various activation and deployment procedures. • Phase 2: GCWW Sample Management and Disposition o Phase 2 was designed to exercise and evaluate GCWW protocols for sample receipt and distribution for in-house and external analyses. • Phase 3: Sample Analysis, Data Management and Interpretation o Phase 3 of the drill was designed to exercise and evaluate GCWW and contract laboratory procedures for sample analyses, data reporting, and data interpretation. Relevant Participants: S&A relevant participants are listed in Table 3-1. 3.2.2 S&A Drill: July 15, 2008 Description: The objective of this drill was to evaluate the site characterization procedures outlined in the consequence management plan following a WQM alert. The drill was conducted in two parts: 1) a morning session conducted by the GCWW Site Characterization Team working alone under simulated low-hazard conditions, and 2) an afternoon session conducted jointly by the GCWW Site Characterization Team and CFD HazMat under simulated possibly hazardous conditions. During the morning session, the GCWW Site Characterization Team performed all of the site characterization activities as outlined in the consequence management plan, except sub-sampling which was conducted in the afternoon session. This included: 1) initial response/deployment, 2) site approach, 3) field safety screening, 4) sample collection, 5) rapid field testing and 6) sample preparation and transport of the BT agent sample to the GCWW Richard Miller Treatment Plant laboratory. In addition, the morning session tested procedures for laboratory analysis and reporting protocols, including packaging, labeling and transporting a BT agent sample. Relevant Participants: S&A relevant participants are listed in Table 3-1. 3.2.3 Full Scale Exercise 2: October 1, 2008 Description: A comprehensive full scale exercise was conducted on October 1, 2008 to test the overall detection and response components of the Cincinnati CWS. The S&A component site characterization and sample collection activities and procedures were coordinated with local emergency response partners (HazMat and Cincinnati Police Department [CPD]). The exercise was designed to simulate events requiring sampling and analysis support activities, including site characterization in response to a simulated "hazard" situation. The exercise afforded the opportunity to evaluate labor hours and several timelines) associated with site characterization activities that included safety screening, rapid field testing, sample collection and sub-sampling for laboratory analyses. Role of S&A: The Site Characterization Team was deployed to two separate sites to conduct rapid field testing. The Site Characterization Team coordinated closely with the CPD to secure the site and determined that contacting HazMat was necessary to safely conduct a perimeter search and collect field samples. Relevant Participants: Site Characterization Team Leader (GCWW), Site Characterization Team (GCWW), Laboratory Program Manager (GCWW), GCWW WUERM, HazMat Unit (CFD) 15 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System 3.2.4 S&A Drill: March 31, 2009 Description: GCWW personnel conducted a Sample Analysis drill. The drill focused on four areas that included procedures for 1) notification of appropriate contacts for requesting laboratory-based sample analyses, 2) sample receipt, documentation, and disposition to appropriate laboratories for the baseline contaminant suite of analyses, 3) analytical data review and 4) analytical results reporting following a simulated Possible contamination incident. The purpose of this drill was to practice and evaluate protocols and standard operating procedures rather than evaluate the performance of those personnel engaged in the drill. Relevant Participants: S&A relevant participants are listed in Table 3-1. 3.2.5 S&A Drill: April 23, 2009 Description: GCWW personnel conducted a site characterization drill to evaluate the implementation of revised site characterization procedures outlined in the consequence management plan and the standard operating procedures for site characterization and sampling. This objective was evaluated by observing the response of GCWW Site Characterization Team to a simulated WQM alert. The drill was designed to observe two Site Characterization Teams conducting identical field procedures simultaneously, but independently of each other. Both teams were coordinated by a single Site Characterization Team Leader. Relevant Participants: S&A relevant participants are listed in Table 3-1. 3.2.6 CCS / S&A Drill: September 16, 2009 Description: GCWW personnel conducted a CCS/site characterization drill to evaluate the alert recognition and investigative procedures associated with the CCS component and to evaluate implementation of the site characterization procedures as they relate to field deployment and investigation following a CCS alert. This was the first time the GCWW Site Characterization Team practiced their protocols and field procedures for responding to a CCS alert. Relevant Participants: S&A relevant participants are listed in Table 3-1. 3.2.7 Full Scale Exercise 3: October 1, 2009 Description: A comprehensive full scale exercise was conducted on October 1, 2009 to provide GCWW Incident Command System (ICS) second-in-command personnel and local response partner agencies the opportunity to exercise their protocols related to detection of and response to a drinking water contamination incident. Procedures such as response partner integration, response time and time for credibility determination were evaluated. Role of S&A: The WUERM, Site Characterization Team Leader, and Site Characterization Team conducted site characterization activities. Site characterization activities for this full scale exercise differed from the previous full scale exercise based on the need to identify sampling points in public areas of the distribution system due to the CCS alerts. The Site Characterization Team and Team Leader deployed to the field, conducted a site investigation, performed rapid field tests and conducted sub- sampling procedures. Samples were then transported back to the GCWW laboratory. Relevant Participants: Site Characterization Team Leader (GCWW), Site Characterization Team (GCWW), Laboratory Program Manager (GCWW), GCWW WUERM and HazMat Unit (CFD) 16 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System 3.2.8 BT Agent S&A Drill: May 10, 2010 Description: On May 10, 2010, GCWW and ODH personnel conducted a BT Agent S&A drill in order to practice site characterization and partner laboratory capabilities, including internal notification procedures to prepare to receive and analyze samples using the Laboratory Response Network (LRN) BT Agent Screening Protocol. The Site Characterization Team collected and packaged two samples for BT agent analysis and EPA transported the samples to the ODH Laboratory. The ODH Laboratory analyzed the samples and reported both polymerase chain reaction (PCR) and culture results to CHD and GCWW. Relevant Participants: S&A relevant participants are listed in Table 3-1. Observed timelines from these drills and exercises will be discussed in Section 6.0. Labor hours expended during drills and exercises will be discussed in Section 8.0. Table 3-1. S&A Drill Variations Variations Time of Drill (N = Normal business hours, A = After hours) Drill Participants WUERM, GCWW Site Characterization Team Leader, GCWW Site Characterization Team, GCWW Laboratory Program Manager, GCWW Chemistry Laboratory Program Manager, GCWW Laboratory Analysts, GCWW Program Manager (Test America, Savannah) Center for Public Health Preparedness and ODH Laboratory and CPD Liaison Director, CHD Microbiology Laboratory Supervisor, ODH Laboratory Bioterrorism Coordinator, ODH Laboratory Drill 1 5/7/08 N Drill 2 7/15/08 N Drill 3 3/31/09 N Drill 4 4/23/09 N CCS/S&A 9/16/09 N BT Drill1 5/10/10 N Number of Participants 0 1 5 1 1 0 1 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 3 0 0 0 0 0 1 6 0 0 0 0 0 0 0 1 1 4 0 0 0 0 0 0 0 0 1 2 1 0 0 0 1 1 1 During this drill, one participant at GCWW served as the Site Characterization Team Leader as well as the Laboratory Program Manager. 3.3 Simulation Study Evaluation of certain design objectives relies on the occurrence of contamination incidents with known and varied characteristics. Because contamination incidents are extremely rare, there is insufficient empirical data to fully evaluate the detection and response capabilities of the Cincinnati CWS. To fill this gap, a computer model was developed using Cincinnati pilot data. This allowed the model CWS to be 17 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System challenged with a large ensemble of simulated contamination scenarios. For the S&A component, simulation study data was used to evaluate the following design objectives: • Contaminant Coverage: Analyses conducted for this design objective demonstrate the detection statistics for site characterization (including water quality parameter and rapid field testing) and laboratory analysis. • Timeliness of Response: Analyses conducted to evaluate this design objective quantify the number of scenarios in which field or laboratory results contributed to the determination that contamination was Credible or Confirmed, and the number of scenarios in which field or laboratory results were available prior to the public health response. Statistical analyses characterize the various timeline metrics, such as the time that elapsed between determination that contamination was Possible and the time that field or laboratory results were available. A broad range of contaminant types, producing a range of symptoms, was utilized in the simulation study to characterize the detection capabilities of the monitoring and surveillance components of a CWS. For the purpose of the simulation study, a representative set of 17 contaminants was selected from the comprehensive contaminant list that formed the basis for CWS design. These contaminants are grouped into the broad categories listed below (the number in parentheses indicates the number of contaminants from that category that were simulated during the study). A description of the manner in which the critical concentration, which is the concentration that would produce adverse health effects (or aesthetic problems in the case of the nuisance chemicals), was derived is also provided for each contaminant category. • Nuisance Chemicals (2): these chemical contaminants have a relatively low toxicity and thus generally do not pose an immediate threat to public health. However, contamination with these chemicals can make the drinking water supply unusable. The critical concentration for nuisance chemicals was selected at levels that would make the water unacceptable to customers, e.g., concentrations that result in objectionable aesthetic characteristics. • Toxic Chemicals (8): these chemicals are highly toxic and pose an acute risk to public health at relatively low concentrations. The critical concentration for toxic chemicals was based on the mass of contaminant that a 70 kg adult would need to consume in one liter of water to have a 10% probability of dying (LD10). • Biological Agents (7): these contaminants of biological origin include pathogens and toxins that pose a risk to public health at relatively low concentrations. The critical concentration for biological agents was based on the mass of contaminant that a 70 kg adult would need to consume in one liter of water, or inhale during a showering event, to have a 10% probability of dying (LD10). Development of a detailed CWS model required extensive data collection and documentation of assumptions regarding component and system operations. To the extent possible, model decision logic and parameter values were developed from data generated through operation of the Cincinnati CWS, although input from subject matter experts and available research was utilized as well. The simulation study used several interrelated models, four of which are relevant to the evaluation of S&A: EPANET, Health Impacts and Human Behavior (HI/HB), Site Characterization, and Laboratory Analysis. Each model is further broken down into modules that simulate a particular process or attribute of the model. The function of each of these models, and their relevance to the evaluation of S&A, is discussed below. 18 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System EPANET EPANET is a common hydraulic and water quality modeling application widely used in the water industry to simulate contaminant transport through a drinking water distribution system. In the simulation study, it was used to produce contaminant concentration profiles at every node in the GCWW distribution system model, based on the characteristics of each contamination scenario in the ensemble. The concentration profiles were used to determine the number of miles of pipe contaminated during each scenario, which is one measure of the consequences of that contamination scenario. Health Impacts and Human Behavior Model The HI/HB model used the concentration profiles generated by EPANET to simulate exposure of customers in the GCWW service area to contaminated drinking water. Depending on the type of contaminant, exposures occurred during one showering event in the morning (for the inhalation exposure route), or during five consumption events spread throughout the day (for the ingestion exposure route). The HI/HB model used the dose received during exposure events to predict infections, onset of symptoms, health-seeking behaviors of symptomatic customers and fatalities. The primary output from the HI/HB model was a case table of affected customers, which captured the time at which each transitioned to mild, moderate and severe symptom categories. Additionally, the HI/HB model outputted the times at which exposed individuals would pursue various health-seeking behaviors, such as visiting their doctor or calling the poison control center. The case table was used to determine the public health consequences of each scenario, specifically the total number of illnesses and fatalities. Furthermore, EPANET and the HI/HB model were run twice for each scenario; once without the CWS in operation and once with the CWS in operation. The paired results from these runs were used to calculate the reduction in consequences due to CWS operations for each simulated contamination scenario. Site Characterization and Sampling Model In the Cincinnati CWS model, the Site Characterization and Sampling model was developed based on procedures GCWW uses to investigate sites in the distribution system associated with Possible contamination incidents. This module encompasses Site Characterization Team mobilization, travel time, deployment, site approach, field safety screening, sample retrieval, rapid field testing, grab and sub- sampling for laboratory analyses, sample packaging in the field and transport to GCWW for disposition of samples to partner laboratories. Table 3-2 depicts GCWW's field safety screening, rapid field testing, and water quality parameter testing capabilities for the 17 contaminants evaluated in the simulation study. Table 3-2. Field Testing Capabilities Type1 Nuisance Chemical 1 Nuisance Chemical 2 Toxic Chemical 1 Toxic Chemical 2 Toxic Chemical 3 Toxic Chemical 4 Toxic Chemical 5 Toxic Chemical 6 Toxic Chemical 7 Toxic Chemical 8 Field Safety Screening •/ •/ Rapid Field Testing •/ •/ s s •/ Water Quality Parameter Testing •/ S •/ S •/ S •/ •/ S 19 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Type1 Biological Agent 1 Biological Agent 2 Biological Agent 3 Biological Agent 4 Biological Agent 5 Biological Agent 6 Biological Agent 7 Field Safety Screening Rapid Field Testing Water Quality Parameter Testing •/ S •/ •/ S •/ S Note that the 17 contaminants modeled in the simulation study were assigned generic IDs for security purposes. The Site Characterization and Sampling model is initiated in response to Possible contamination as determined through the investigation of an alert from one or more of the monitoring and surveillance components. Once the threat level reaches Possible, the WUERM has the responsibility for determining sampling locations. The Site Characterization Team will begin mobilization and deploy to alert locations identified by the WUERM. Samples retrieved from the sites are returned to the central GCWW laboratory and/or partner laboratories for analysis. Laboratory Analysis Model The Laboratory Analysis model was based on procedures and methods GCWW and partner laboratories use to process and analyze samples and report results from Possible contamination incidents. The Laboratory Analysis model encompasses notification of laboratories to prepare for sample receipt and analysis, sample disposition to method laboratories, sample analysis, data review and results reporting to the WUERM. In the Laboratory Analysis model, certain samples are transported directly from the field location to the laboratory performing analyses, so no time delay is included in these instances for sample receipt and disposition at the GCWW central laboratory. All other samples are transported to the GCWW central laboratory for sample disposition and analysis. Laboratory mobilization begins at the time the WUERM notifies the Laboratory Program Manager to mobilize, which includes assembling staff and calibrating instrumentation. In the model, laboratory analyses begin when the laboratory has mobilized and samples are received. The model allows analyses to be performed concurrently once the two criteria above are met. Under special conditions, "triggered" laboratories may be activated to analyze for contaminants outside of the baseline suite. While these laboratories were not utilized during baseline monitoring, they were identified using a variety of resources, including EPA's Laboratory Compendium, referral and direct telephone contact. Through direct telephone contact with the performing laboratory, analytical methods, sample transport times, laboratory mobilization times, sample analysis times, minimum concentration to detect and results reporting protocols and times were established for use in the simulation study model. Analyses that these laboratories would perform for the additional five contaminants are called "triggered" analyses. While all 17 contaminants evaluated in the simulation study theoretically could be confirmed by the S&A component, they can only be detected by S&A if the sample is sent to the laboratory that analyzes for that specific contaminant, and if the contaminant concentration is above the minimum concentration to detect. Table 3-3 indicates which of these contaminants would be detected via analyses performed as a part of GCWW's baseline suite of analysis (conducted either in-house or at a partner laboratory), and which would be detected through "triggered" analyses. This table also indicates the ratio of the critical concentration to the minimum reporting limit for each contaminant. The ratio was calculated to 20 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System determine whether the minimum reporting limit was sufficient to detect water contaminated at concentrations equal to or greater than the critical concentration. Ratios greater than 1.0 demonstrate the contaminants that can be detected at concentrations below the critical concentration. As can be seen from the ratios in Table 3-3, all laboratory methods employed in the Cincinnati CWS could detect contaminants at concentrations significantly smaller than the critical concentration. Table 3-3. Laboratory Testing Capabilities Type1 Nuisance Chemical 1 Nuisance Chemical 2 Toxic Chemical 1 Toxic Chemical 2 Toxic Chemical 3 Toxic Chemical 4 Toxic Chemical 5 Toxic Chemical 6 Toxic Chemical 7 Toxic Chemical 8 Biological Agent 1 Biological Agent 2 Biological Agent 3 Biological Agent 4 Biological Agent 5 Biological Agent 6 Biological Agent 7 Critical Concentration/ Minimum Reporting Limit 2.00 x 104 2.00 x 104 1,470 3.39 x 104 3.69 x 10s 5.80 x 104 6,680 4.08 x 104 57.0 6.60 x 107 2.25 x 104 4.93 x 105 2430 90.7 20.0 5.79 x 104 3.30 x 105 Baseline Suite or Triggered Analysis Baseline Suite Baseline Suite Baseline Suite Baseline Suite Baseline Suite Baseline Suite Baseline Suite Baseline Suite Triggered Baseline Suite Triggered Baseline Suite Triggered Triggered Triggered Baseline Suite Baseline Suite Note that the 17 contaminants modeled in the simulation study were assigned generic IDs for security purposes. The following assumptions used in the design of the S&A model are important to consider when evaluating the simulation study results presented in this report: • In order for a sample to be sent for "triggered" analyses, there must be some indication or evidence of the contaminant from other components to prompt the analyses. This evidence may come from site characterization results, water quality changes produced by the contaminant and/or the symptoms reported by exposed individuals. • Contaminant injection locations at utility facilities with enhanced security monitoring (ESM) capabilities were the only sites that produced a positive field safety screening result for Nuisance Chemical 1 or the Toxic Chemical 8. The implicit assumption is that neat material is present at the injection location, which triggered a field safety screening hit. • Special conditions exist for field analyses when the first alert that occurs in a scenario is either a WQM or ESM alert. In these scenarios, personnel that are deployed for alert investigations perform some site characterization activities as part of the investigation of the component alert. Specifically, during the investigation of the first WQM alert, the following site characterization activities were performed by the Water Quality and Treatment Technician who was deployed to inspect the water quality monitoring station: Site Characterization Team mobilization, deployment, site approach and field safety screening. During enhanced security monitoring alert investigations, site approach and field safety screening was performed by enhanced security 21 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System monitoring alert investigation personnel, who then communicated the screening results to the Site Characterization Team upon the team's arrival at the site. • The concentration in a sample was the concentration at the node to which the Site Characterization Team is deployed at the time the sample was collected. This concentration was obtained from the EPANET output. • For CCS and PHS alerts, it is assumed that the Site Characterization Team will be deployed to the residences of one of the individuals whose case is associated with the alert. If the contaminant used in the scenario is a chemical, the sample concentration is assumed to be equal to the concentration at the node at the time the person was exposed (i.e., contaminated water remains in household plumbing for at least a few hours). • Samples collected at a node associated with a PHS alert for scenarios involving some of the biological agents had a concentration of zero. This assumption is based on the relatively long delay between exposure to some biological agents and onset of symptoms followed by the health seeking behavior that produced the alert. • Samples containing a contaminant concentration above the minimum concentration to detect always produced a positive result for rapid field tests and laboratory analyses. It was assumed that there were no false negative results. • There were no QA issues that would negate the results of laboratory analysis. However, it was assumed that a QA review of the data was performed, resulting in a one-hour delay between generation of results and reporting them to the WUERM. 3.4 Forums Feedback and suggestions on all aspects of the S&A component were captured during monthly staff interviews during the evaluation period (March 2008 - June 2010) as well as during a lessons learned workshop (September 2009). Information gathered through these forums provided insight regarding which pieces of the component were acceptable to the end-users and others that required modification or enhancement. • Monthly Interviews with Utility Staff: GCWW staff members were contacted monthly in order to track maintenance monitoring efforts. These interviews were generally conducted by telephone and the information was used to update and track S&A performance data. • Lessons Learned Workshop: The purpose of the lessons learned workshop was to allow GCWW and partner personnel (i.e., ODH, CHD, CFD) to provide feedback regarding the performance, operation and sustainability of the S&A component during the evaluation period. The group expressed specific feedback regarding the strengths and weaknesses of the S&A component in the context of establishing an effective system for routine and incident response sampling and analysis. 3.5 Analysis of Lifecycle Costs A systematic process was used to evaluate the lifecycle cost of the S&A component, which represents the overall cost of the S&A component over the lifecycle of the Cincinnati CWS, which is assumed to be 20 years for the purpose of this analysis. The analysis includes implementation costs, component modification costs, annual operations and maintenance (O&M) costs renewal and replacement costs, and the salvage value of major pieces of equipment. Implementation costs include labor and other expenditures (equipment, supplies, and purchased services) for installing the S&A component. Implementation costs were summarized in Water Security Initiative: Cincinnati Pilot Post-Implementation System Status (USEPA, 2008), which was used as a primary data 22 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System source for this analysis. In that report, overarching project management costs incurred during the implementation process were captured as a separate line item. However, in this analysis, the project management costs were equally distributed among the six components of the CWS, and are presented as a separate line item for each component. Component modification costs include all labor and expenditures incurred after the completion of major implementation activities in December 2007 that were not attributable to O&M costs. These modification costs were tracked on a monthly basis, summed at the end of the evaluation period and added to the overall implementation costs. It should be noted that implementation costs for the Cincinnati CWS may be higher than those for other utilities given that this project was the first comprehensive, large-scale CWS of its kind and had no experience base to draw from. Costs that would not likely apply to future implementers (but which were incurred for the Cincinnati CWS) include overhead for EPA and its contractors, cost associated with deploying alternative designs and additional data collection and reporting requirements. Other utilities planning for a similar large-scale CWS installation would have the benefit of lessons learned and an experience base developed through implementation of the Cincinnati CWS. Annual O&M costs include labor and other expenditures (supplies and purchased services) necessary to operate and maintain the component and investigate alerts. O&M costs were obtained from procurement records, maintenance logs, investigation checklists, and training logs. Procurement records provided the cost of supplies, repairs, and replacement parts, while maintenance logs tracked the staff time spent maintaining the S&A component. To account for the maintenance of documents, the cost incurred to update documented procedures following drills and exercises conducted during the evaluation phase of the pilot was used to estimate the annualized cost. Investigation checklists and training logs tracked the staff hours spent on investigating alerts and training, respectively. The O&M costs were annualized by calculating the sum of labor and other expenditures (supplies and purchased services). Labor hours for both implementation and O&M were tracked over the entire evaluation period. Labor hours were converted to dollars using estimated local labor rates for the different institutions involved in the implementation or O&M of the S&A component. The renewal and replacement costs are based on the cost of replacing these major pieces of equipment at the end of their useful life. The useful life of S&A equipment was estimated using field experience, manufacturer-provided data, and input from subject matter experts. Equipment was assumed to be replaced at the end of its useful life over the 20-year lifecycle of the Cincinnati CWS. The salvage value is based on the estimated value of each major piece of equipment at the end of the lifecycle of the Cincinnati CWS. The salvage value was estimated for all equipment with an initial value greater than ~$1,000. Straight line depreciation was used to estimate the salvage value for all major pieces of S&A equipment based on the lifespan of each item. All of the cost parameters described above (implementation costs, enhancement costs, O&M costs, renewal and replacement costs, and salvage value) were used to calculate the total lifecycle cost for the S&A component, as discussed in Section 8.1. 23 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 4.0: Design Objective - Spatial Coverage Analysis of spatial coverage of the S&A component describes the geographic scope of sampling locations across the GCWW service area. A description of the rationale for location selection for each of the phases of baseline monitoring and those that would be targeted during incident response is presented. 4.1 Spatial Coverage Definition: Spatial coverage includes the spatially diverse and hydraulically significant area covered through routine and incident response sampling and analysis, as determined by sampling locations selected during Phase 1 and tested during Phases 2, 3, and 4 of baseline monitoring (described in Section 2.2.1). The objective of these sampling locations was to establish contaminant occurrence and method performance throughout the distribution system. Analysis Methodology: Empirical data of the number and types of samples collected at various locations. Results: In theory, samples can be collected from any location within the GCWW service area. However, for purposes of routine and maintenance monitoring, strategic locations were identified such that samples could be collected in an efficient manner. These strategic locations were based on the placement of online water quality monitors and enhanced security monitoring equipment. These locations provided 18 strategic and 5 priority locations to conduct regular sample collection as part of Phase 3 of baseline monitoring; the strategic and priority sampling locations may also be used as incident response sampling sites in the event of a suspected contamination incident. Table 4-1 illustrates the strategic and priority sampling locations and their associated treatment plants. Table 4-1. Strategic and Priority Sample Locations for Baseline Monitoring Sampling Location Strategic Locations Pump Station Elevated Tank Ground Tank Reservoir Fire Station Tank Treatment Plant Priority Locations Reservoir Elevated Tank Booster Station Associated Treatment Plant Miller 4 4 1 2 3 - 1 Miller 2 1 1 Bolton - - - - - - 1 Bolton - - - Interface 1 - - - - 1 - Interface - 1 - Due to logistical issues at some of the strategic locations (i.e., the large volume of water required for BT agent testing), eleven alternate locations within the distribution system were also selected for routine monitoring. However, all 18 of the strategic locations were sampled during the first year of baseline monitoring to practice procedures associated with these sites. Phase 4 of baseline monitoring was a survey study of 54 different sampling sites over a two month period. Sample collection locations were selected primarily from GCWW's total coliform rule monitoring route. The goal of the survey study was 24 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System to sample spatially diverse locations in a short period of time as well as to capture a range of conditions in water age, pressure zones and pipe material. Figure 4-1, below, is a map of greater Cincinnati showing the survey locations (red dots denote survey sampling locations). The colored shading in different areas of this map demonstrates the various pressure zones and service areas in GCWW's distribution system. Figure 4-1. Survey Sample Sites Following baseline monitoring, a transition to maintenance monitoring occurred which involved continued sampling at a total of 31 strategic locations. During incident response sampling and analysis, sampling location will be determined using information gathered from other components of the CWS and can include any locations covered by baseline monitoring as well as customer taps, fire hydrants and tanks. 4.2 Summary The spatial coverage design objective for the S&A component was achieved through routine sample collection from strategic, priority, and survey sampling locations throughout GCWW's distribution system. Baseline monitoring was performed to establish contaminant occurrence (levels and frequency), method performance, seasonal trends and to establish laboratory preparedness for incident response. Sampling locations were selected to leverage WQM and ESM locations, as well as existing GCWW sampling plans (e.g., total coliform rule monitoring route, tank routes). It was important to characterize baseline contaminant occurrence and method performance at locations that may be sampled during incident response sampling and analysis. Following completion of baseline monitoring, the utility 25 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System transitioned to maintenance monitoring and is continuing to collect samples from 31 strategic locations throughout the distribution system to maintain proficiency in field and laboratory methods and to update contaminant baseline data. 26 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 5.0: Design Objective - Contaminant Coverage As described in Section 2.1, the S&A component was designed to enhance laboratory capability and capacity to perform screening and confirmatory analyses for a wide range of contaminants under routine and non-routine conditions. When possible, in-house capabilities at GCWW were used to build broad contaminant detection capabilities. If GCWW did not have the capability in-house, field and laboratory equipment or additional personnel training were acquired. For targeted priority contaminants outside of GCWW's range of capabilities, a support laboratory network was established. Background levels of priority contaminants were documented during baseline monitoring. During maintenance monitoring, a reduced sampling and analysis schedule was established which allowed the utility and partner laboratories to maintain proficiency in methods and to update contaminant baseline data. Three metrics were used to assess contaminant coverage of the S&A component: contaminant detection potential, contaminant detection limit and contaminant scenario coverage. 5.1 Contaminant Detection Potential Definition: Contaminant detection potential is the ability to detect specific contaminants or contaminant classes through field or laboratory analysis of water samples, as determined by the S&A design element field and laboratory testing capabilities (Section 2.1). Analysis Methodology: Description of field and laboratory methods deployed during baseline and maintenance monitoring for the S&A component. Results: Prior to implementation of the Cincinnati pilot, EPA compiled a list of high priority contaminants of interest to water security. From the high priority contaminant list, EPA identified a subset of chemical, radiochemical, and microbiological contaminants to monitor during the Cincinnati pilot at GCWW. A suite of field and laboratory methods to detect these contaminants was devised. Table 5-1 presents the target water quality parameters identified for analysis during baseline and maintenance monitoring at GCWW. The instruments/test kits used to test for these parameters (see Table 2-3) provide screening capability, and are not considered confirmatory. Table 5-1. Target Water Quality Parameters for Baseline Monitoring at GCWW Target Parameters Free cyanide Free chlorine pH, conductivity, ORP, turbidity Chemical Warfare Agents Radioactivity (alpha, beta, gamma) VOCs and combustible gases Toxicity* Arsenic * Field test performed during baseline monitoring, but discontinued during maintenance monitoring Table 5-2 demonstrates the contaminant class analytical capabilities identified for baseline monitoring at GCWW. Laboratory capabilities for each of these contaminant classes were successfully implemented, either at GCWW or at external partner laboratories. These capabilities would be used during response to Possible contamination incidents. The laboratory methods used for analytes in these contaminant classes are considered confirmatory methods, with the exception of the BT Agent method. Subsequent culture analysis would be needed to confirm PCR results. 27 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Table 5-2. Contaminant Classes for Baseline Monitoring at GCWW Contaminant Classes VOCs Carbamates SVOCs Radiochemicals Total cyanide BT Agents Metals In some instances, it may be necessary to seek analytical confirmation for contaminants outside of the analytical capabilities described above. This process would only be performed if there was information available from S&A or other CWS components indicating a suspected contaminant. While it may be possible to utilize GCWW capabilities, support from partner laboratories would be required in some cases; the process of identifying a capable support laboratory would take extra time. Depending on the information available during the incident, existing methods could be used to target the suspected contaminant. For example, the methods used to detect metals or SVOCs can expand beyond the target analytes in the contaminant classes described above to detect hundreds of additional contaminants. 5.2 Contaminant Detection Limit Definition: The S&A component metric for contaminant minimum reporting limits is defined as the lowest concentration of a specific baseline contaminant that can be empirically measured and reliably reported using a confirmatory method as determined by the performing laboratory. Method detection limits and minimum reporting limits for targeted chemical contaminants were empirically determined by each laboratory conducting analyses. An estimated detection limit based on information provided by the manufacturer as well as published literature and field experience is used for contaminants detected by screening methods. Analysis Methodology: Minimum reporting limits for each baseline contaminant and analytical method, both screening and confirmatory, were determined during Phase 1 of baseline monitoring based on empirical laboratory results, published reports, and manufacturer information. Results: Estimates of rapid field test screening detection limits were based on published literature, manufacture information, and GCWW experience with field equipment. Empirical data from samples collected during routine sampling was also compiled by analyte and method for each of the contaminants targeted during baseline monitoring. 5.3 Contamination Scenario Coverage Definition: Contamination scenario coverage is defined as the ratio of contamination incidents that are actually detected to those that are theoretically detectable based on field and laboratory detection capabilities of the component. Analysis Methodology: Simulation study results were used to characterize contamination scenario coverage of the S&A component, including both site characterization and laboratory analyses. Site characterization detection is characterized by results from both water quality parameter testing and rapid field testing. If either water quality parameter tests or rapid field tests indicated a deviation from the established baseline, the site characterization component of S&A "detects" a scenario. Detection by laboratory analysis of the 17 contaminants modeled in the simulation study may occur if 1) samples are collected and analyzed as part of the baseline suite of analyses that GCWW performs in response to Possible or Credible contamination incidents (total of 12 target contaminants), or 2) if there is sufficient information in the scenario to "trigger" analyses for any of the five contaminants that were not part of the 28 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System baseline suite but that were modeled for the simulation study. Refer to Section 3.3 for a description of the assumptions used in the simulation study that are relevant to S&A results. Results: Table 5-3 presents the site characterization detection statistics for each of the 17 contaminants under evaluation in the simulation study. A total of 1,666 contamination scenarios out of a possible 2,015 (83%) were detected by either water quality parameter tests and/or rapid field tests in the simulation model. All scenarios that could have been detected by rapid field tests (for Nuisance Chemical 1, Toxic Chemical 1, Toxic Chemical 2, Toxic Chemical 7 and Toxic Chemical 8) were detected. The column labeled "Non-Detect Description" indicates the reason for scenarios that were not detected by site characterization for each contaminant, accompanied by the total number of scenarios that were not detected (as presented in the column on the far right). Many more contamination scenarios were detected which involved rapid symptom onset as compared to the biological agents with delayed symptom onset. The GCWW Site Characterization Team is equipped with field tests that can indicate the presence of Nuisance Chemical 1 and some of the toxic chemicals, whereas that availability of verified technologies for field testing for biological agents in water samples is limited. Of the 349 contamination scenarios that were not detected by site characterization activities, 224 were scenarios involving biological agents with delayed symptom onset, in which the first detection of the contamination incident was made by the PHS component. For these scenarios, sampling did not occur soon enough to capture a water sample containing the contaminant. In 45 of the contamination scenarios that were not detected, a Possible water contamination determination was not reached; in these instances, a Site Characterization Team would not have been deployed to collect samples. Finally, the remaining 80 contamination scenarios were not detected by WATER QUALITY parameter testing due to the simulated change in C12 or total organic carbon not deviating enough from the normal range, or due to an inability to detect the contaminant via rapid field testing. Table 5-3. Site Characterization Detection Statistics Contaminant Nuisance Chemical 1 Nuisance Chemical 2 Toxic Chemical 1 Toxic Chemical 2 Toxic Chemical 3 Toxic Chemical 4 Toxic Chemical 5 Toxic Chemical 6 Toxic Chemical 7 Toxic Chemical 8 Biological Agent 1 Total Scenarios 119 119 119 119 119 119 119 119 119 119 119 Scenarios Not Detected 0 47 0 0 14 0 0 1 0 0 0 Percent Detected 100%1 61% 100%1 100%1 88% 100% 100% 99% 100%1 100%1 100% Non-Detect Description N/A Threat level did not reach Possible Contaminant concentration below detection limit N/A N/A Contaminant concentration below detection limit N/A N/A Contaminant concentration below detection limit N/A N/A N/A Number of Scenarios N/A 39 8 N/A N/A 14 N/A N/A 1 N/A N/A N/A 29 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Contaminant Biological Agent 2 Biological Agent 3 Biological Agent 4 Biological Agent 5 Biological Agent 6 Biological Agent 7 Total Scenarios 119 119 119 119 113 117 Scenarios Not Detected 0 0 58 33 93 103 Percent Detected 100% 100% 51%2 72%2 18%2 12%2 Non-Detect Description N/A N/A PHS was first to detect and scenario involved a biological agent Contaminant concentration below detection limit PHS was first to detect and scenario involved a biological agent Contaminant concentration below detection limit PHS was first to detect and scenario involved a biological agent Threat level did not reach Possible Contaminant concentration below detection limit PHS was first to detect and scenario involved a biological agent Threat level did not reach Possible Contaminant concentration below detection limit Number of Scenarios N/A N/A 40 18 31 2 71 3 19 82 3 18 100% contamination scenario detection by rapid field testing capability. 2 The detection statistics for these biological agents indicate the frequency at which a WATER QUALITY parameter test showed a deviation in either free chlorine or total organic carbon in these contamination scenarios. Table 5-4 presents the laboratory analysis detection statistics for each of the 17 contaminants under evaluation in the simulation study. A total of 1,729 contamination scenarios out of a possible 2,015 (86%) were detected by laboratory analysis in the simulation study. The column labeled "Non-Detect Description" indicates the reason for scenarios that were not detected by laboratory analysis for each contaminant, accompanied by the total number of scenarios that were not detected (as presented in the column on the far right). Many more contamination scenarios were detected which involved toxic chemicals and biological agents with rapid symptom onset (1 through 3) as compared to the biological agents with delayed symptom onset (4 through 7). Similar to the site characterization detection results, for some of the scenarios involving biological agents (223 scenarios), sampling was not initiated until PHS component alerts had occurred, which did not occur in time to capture a sample containing the contaminant. In 45 of the contamination scenarios that were not detected, a Possible water contamination determination was not reached; in these instances, a Site Characterization Team would not have been deployed to collect samples. Three of the contamination scenarios not detected were due to the fact that the scenario involved a contaminant that was not part of the baseline suite of analyses and the scenario did not contain sufficient information to prompt "triggered" analysis. Therefore, the contaminant was not detected in these three scenarios because the contaminants were not detectable by the baseline suite of methods. 30 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Finally, in 13 of the contamination scenarios, the contaminant was not detected as it was present in a concentration below the detection limit of the method (in some cases there was no contaminant in the sample). Scenarios wherein the samples that were collected did not contain any of the contaminant were characterized. For all these scenarios, it was determined that the WQM component was the first to produce an alert. However, water quality parameters had not changed above a specified threshold; therefore, the investigation was aborted. This logic was built into the model to be representative of GCWW's investigation practices for WQM alerts; if there is only one WQM alert, and if that alert does not "appreciably" change water quality, GCWW investigators will wait for more information to come in from other WQM stations before proceeding with the investigation. If no additional information becomes available, the WQM investigation is terminated before reaching Possible for that scenario. Therefore, in the contamination scenarios where this logic applied, samples were not collected until subsequent alerts were produced by other components. For the 13 scenarios where the contaminant was not detected, the PHS component was the next to produce alerts. Because samples were collected from PHS alert locations days after the first and only WQM alert in most cases, they did not contain the contaminant and thus yielded a negative sample result. Table 5-4. Laboratory Analysis Detection Statistics Contaminant Nuisance Chemical 1 Nuisance Chemical 2 Toxic Chemical 1 Toxic Chemical 2 Toxic Chemical 3 Toxic Chemical 4 Toxic Chemical 5 Toxic Chemical 6 Toxic Chemical 7 Toxic Chemical 8 Biological Agent 1 Biological Agent 2 Biological Agents Biological Agent 4 Biological Agent 5 Total Scenarios 119 119 119 119 119 119 119 119 119 119 119 119 119 119 119 Scenarios Not Detected 0 39 0 0 0 2 0 1 0 0 0 0 0 42 33 Percent Detected 100% 67% 100% 100% 100% 98% 100% 99% 100% 100% 100% 100% 100% 65% 72% Non-Detect Description N/A Threat level did not reach Possible N/A N/A N/A Contaminant not detectable by the baseline suite of methods and "triggered" analysis was not initiated N/A Contaminant concentration below detection limit N/A N/A N/A N/A N/A PHS was first to detect and scenario involved a contaminant with delayed symptom onset Contaminant concentration below detection limit PHS was first to detect and scenario involved a contaminant with delayed symptom onset Triggered laboratory analysis not initiated; incorrect laboratory Contaminant concentration below detection limit Number of Scenarios N/A 39 N/A N/A N/A 2 N/A 1 N/A N/A N/A N/A N/A 39 3 31 1 1 31 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Contaminant Biological Agent 6 Biological Agent 7 Total Scenarios 113 117 Scenarios Not Detected 79 90 Percent Detected 30% 23% Non-Detect Description PHS was first to detect and scenario involved a contaminant with delayed symptom onset Threat level did not reach Possible Contaminant concentration below detection limit PHS was first to detect and scenario involved a contaminant with delayed symptom onset Threat level did not reach Possible Contaminant concentration below detection limit Number of Scenarios 71 3 5 82 3 5 5.4 Summary The S&A component of the Cincinnati pilot met the contaminant coverage design objective through successful implementation of field and laboratory methods for all target water quality parameters and priority contaminants identified during design of the component. Though it is possible that a contamination incident may involve a contaminant outside of the suite of baseline methods utilized by GCWW, the utility is prepared to implement the process of identifying a capable support laboratory if information is available from field screening or from other CWS components about the suspected contaminant. Furthermore, approximate detection limits were identified using manufacturer information for each of the field methods, and reporting limits were identified based on laboratory data for each of the confirmatory methods included in GCWW's baseline suite of analyses. Simulation study results demonstrated a contaminant scenario detection rate of 83% for site characterization results (water quality parameter tests and rapid field tests) and 86% for laboratory analysis results. The predominant reason that non-detects occurred was explained by scenarios in which contamination by a biological agent was first detected by the PHS component. By the time that the Site Characterization Team was deployed to collect samples, the contaminated water had already passed through the system, leading to non-detects in field and laboratory analyses. This finding underscores the importance of a multi-component CWS which does not rely solely on PHS for detection of drinking water contamination incidents, but involves multiple monitoring and surveillance components. For example, in many scenarios involving biological agents with delayed symptom onset, WQM detected contaminated water while it was still in the distribution system, allowing for the automated sampling devices at each WQM location to capture a sample that did contain detectable concentrations of the biological agent. In these scenarios, the contaminant was also detected during site characterization and/or laboratory analysis. 32 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 6.0: Design Objective - Timeliness of Response Analysis of the timeliness of response by the S&A component considers metrics that quantify the time for field safety screening, sample collection, rapid field test procedures, sample packaging and transport, laboratory analysis, data interpretation and results reporting. The timeline for incident response sampling and analysis is used as the basis for analysis under this design objective. Results from drills and exercises are used to evaluate this design objective. 6.1 Timeline of Incident Response Sampling and Analysis Definition: This timeliness of contaminant detection is defined as a portion of the incident timeline that begins with the recognition of a Possible contamination incident and ends with a determination regarding whether or not the contamination is detected or confirmed by field or laboratory analyses. This metric is meant to measure S&A activities during incident response to a contamination incident as opposed to routine sampling. Timeliness is divided into discreet activities in order to capture and differentiate specific processes that affect the overall time required to detect or confirm the presence of a baseline contaminant using the baseline suite of analytical methods. These activities include: 1) Time to deploy to contamination site - a portion of the incident timeline that begins with notification from the WUERM that the Site Characterization Team should deploy and proceeds through: Site Characterization Team briefing and assignment of responsibilities; identifying equipment and supplies needed for site characterization and field sampling; calibration of field instruments and preparation of field reagents and standards; loading supplies and personnel into response vehicle; departing the utility; and arrival at the perimeter of the sampling location. 2) Time for site approach and field safety screening - a portion of the incident timeline that begins after the Site Characterization Team has arrived on scene and includes preparation of necessary equipment and donning of appropriate personal protective equipment to begin site approach, continual visual observation and reporting of site conditions and continual monitoring and reporting of radiation and atmospheric gas levels to ensure site safety. 3) Time for sample collection - a portion of the incident timeline that begins with the initiation of sampling and ends with retrieval of a sample(s) for rapid field testing and/or laboratory analysis. 4) Time for sample analysis a) Time for rapid field testing - a portion of the incident timeline that begins with initiation of rapid field testing and ends with reporting of results to the WUERM by the Site Characterization Team Leader, b) Time for laboratory sample analysis - a portion of the incident timeline that begins with receipt and disposition of the sample(s) for laboratory-based analysis and ends with data reporting to the GCWW Laboratory Program Manager. 5) Time for laboratory data review and results reporting - a portion of the incident timeline that begins with the receipt of laboratory analytical results by the GCWW Laboratory Program Manager, involves comparison of data to baseline contaminant occurrence and method performance and ends with reporting of laboratory results to the WUERM. Analysis Methodology: Data is derived from incident response sampling and analysis timelines as measured during drills and exercises and data sources include field results forms, chain-of-custody forms and after action reports. Results are reported (hrs:min) and are updated as appropriate. Furthermore, an overall estimated timeline for sampling and analysis was estimated based on data gathered during drills 33 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System and exercises. In some cases, partner laboratories would conduct sample analysis of analytes that had not previously been targeted during baseline monitoring; these laboratories were contacted to gather time estimates for sample analysis and results reporting. Simulation study results were used to characterize the average time between a Possible water contamination determination and availability of site characterization results/laboratory results to evaluate the timeliness of S&A results for each contaminant. Furthermore, simulation study results were evaluated to determine the number of contamination scenarios in which S&A results (site characterization and laboratory results) had an impact on elevating the threat level to Credible or Confirmed. The time of S&A results was compared to the time of Credible and Confirmed contamination. Simulation study results were also used to evaluate the number of contamination scenarios in which S&A results played a role in activating the public health response by comparing the time of S&A results to the time that the public health response was initiated. Contamination scenarios included in the analyses described above were those wherein field or laboratory results were available (which varied for each contaminant). Results: Observed timelines were derived from S&A drills and exercises. These times reflect the practical application of sample collection, laboratory analysis, and data interpretation as part of incident response sampling and analysis performed in response to various practice contamination scenarios representative of the baseline suite of contaminants. In the event that methods beyond the baseline suite were needed during incident response, more time may be necessary to identify an appropriate method and locate a capable facility. Depending on the method needed, this could add significant time onto the overall S&A response timeline. A summary of S&A times observed during drills and exercises at the Cincinnati pilot can be found in Table 6-1. A range of time, from approximately 3.5 to 6 hours, was necessary to complete Phase 1 activities. In some instances, time to complete Phase 1 and Phase 2 was measured together. In most instances, data interpretation and reporting (Phase 3) was simulated because there was no actual contamination present. Therefore, assumptions based on laboratory and utility experience are necessary to contrive estimates for this phase. 34 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Table 6-1. S&A Timeliness during Drills and Exercises Drill Type Date Phase 1: Site Characterization, Sample Collection and Transport Phase 2: Sample Analysis Phase 3: Data Interpretation and Results Reporting S&A Drill May 2008 375 min 23 hours S&A Drill July 2008 315 min Not applicable (simulated) Not applicable (simulated) Full Scale Exercise 2 October 2008, Location 1 225 min Not applicable (simulated) October 2008, Location 2 6 hours Not applicable (simulated) S&A Drill April 2009 Team A 189 min Team B 206 min NA NA S&A Drill September 2009 159 min NA NA Full Scale Exercise 3 October 2009 224 min 17 min (rapid field test) Not applicable (simulated) S&A Drill May 2010 268 min 103.5 hours Not applicable (simulated) 35 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Timeline estimates for each of the S&A activities that would occur during incident response to a contamination incident from the time of deployment of the Site Characterization Team through data review and results reporting are presented in Table 6-2, and were based on observed timelines during drills and exercises. A range of time estimates is listed for some of the S&A activities given that the timeline varies between the samples that would be transported to and analyzed in-house at GCWW's laboratory as compared to those that would be transported to and analyzed by partner laboratories. Furthermore, the time for sample analysis varies for the different laboratory methods that would be conducted. These estimates were used in the simulation study to represent the timeline for incident response. Table 6-2. S&A Timeline Estimates for Incident Response S&A Activity Site Characterization, Sample Retrieval, and Sample Disposition Sample Transport Laboratory Mobilization Sample Analysis Data Review & Results Reporting Total Time Estimate (minutes) 180 30 - 240 30-120 240-1,680 60 540-2,040 Simulation study results are presented in Table 6-3 and demonstrate the average time that elapsed from the Possible contamination determination to site characterization results and the time that elapsed before laboratory results for each contaminant and overall. Based on the data, there is not much variation between contaminants in the time that elapsed from a Possible determination to site characterization results, which demonstrates that the site characterization process is consistent regardless of the contaminant. The main factor that would delay availability of site characterization results would be contamination scenarios that would require involvement of a HazMat unit to assist with sampling and analysis. Time delays would occur while waiting for a HazMat unit to arrive on the scene, and HazMat personnel would require more time to complete site characterization activities due to the difficulty of handling samples and conducting field tests while wearing full-body personal protective equipment. The average time from Possible to the availability of laboratory results varied as a function of analyte, ranging from ~8 hours to almost 2 days. Variation in time to laboratory results exists due to differences in transport time to the GCWW laboratory vs. partner laboratories and the time required to complete analytical methods, which is typically longer for some biological agents as compared to chemical contaminants. Table 6-3. S&A Timeline Analysis (simulation study results) Contaminant Nuisance Chemical 1 Nuisance Chemical 2 Toxic Chemical 1 Toxic Chemical 2 Toxic Chemical 3 Toxic Chemical 4 Toxic Chemical 5 Average Time Possible to Site Characterization Results (minutes) 194 148 176 171 150 153 150 Total Scenarios 119 72 119 119 105 119 119 Average Time Possible to Laboratory Results (minutes) 463 750 615 580 501 604 531 Total Scenarios 119 80 119 119 119 119 119 36 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Contaminant Toxic Chemical 6 Toxic Chemical 7 Toxic Chemical 8 Biological Agent 1 Biological Agent 2 Biological Agent 3 Biological Agent 4 Biological Agent 5 Biological Agent 6 Biological Agent 7 All Contaminants Average Time Possible to Site Characterization Results (minutes) 151 172 171 151 151 182 182 199 168 167 166 Total Scenarios 118 119 119 119 119 119 61 86 20 14 1,666 Average Time Possible to Laboratory Results (minutes) 553 917 1,870 558 832 928 2,471 2,404 1,715 1,903 1,072 Total Scenarios 119 119 119 119 119 119 119 119 110 114 1,970 Tables 6-4 to 6-7 demonstrate the impact of water quality parameter results, rapid field test results, and laboratory results on the threat level for all relevant contamination scenarios. Each table indicates the number of scenarios (for each contaminant and overall) in which water quality parameter results, rapid field test results or laboratory results were available prior to the time when Credible or Confirmed contamination was reached. Table 6-4 shows that on average, water quality parameter results played a role in elevating the threat level to Credible in slightly less than half of scenarios. Results were available in more scenarios involving biological agents compared to the toxic chemicals, due to the lengthier timeline involved when investigating contamination by a biological agent. On average, slightly more than half of all scenarios have water quality parameter results prior to a Confirmed determination, and therefore played a role in elevating the threat level. Results were not available for Toxic Chemical 8 as it did not result in a change in the water quality parameters under evaluation in the simulation model. Table 6-4. Scenarios with Water Quality Parameter Results Available Prior to Credible and Confirmed Contamination Determination (simulation study results) Contaminant Nuisance Chemical 1 Nuisance Chemical 2 Toxic Chemical 1 Toxic Chemical 2 Toxic Chemical 3 Toxic Chemical 4 Toxic Chemical 5 Toxic Chemical 6 Toxic Chemical 7 Toxic Chemical 8 Biological Agent 1 Biological Agent 2 Biological Agent 3 Biological Agent 4 Total Scenarios 119 72 119 115 105 119 119 118 106 0 119 119 119 61 WQ Parameter Results Prior to Credible 77 48 3 4 3 1 92 77 89 0 2 95 72 30 Percent (Prior to Credible) 65% 67% 3% 3% 3% 1% 77% 65% 84% - 2% 80% 61% 49% WQ Parameter Results Prior to Confirmed 90 48 5 70 52 3 93 83 90 0 4 95 81 36 Percent (Prior to Confirmed) 76% 67% 4% 61% 50% 3% 78% 70% 85% - 3% 80% 68% 59% 37 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Contaminant Biological Agent 5 Biological Agent 6 Biological Agent 7 All Contaminants Total Scenarios 86 20 14 1,530 WQ Parameter Results Prior to Credible 59 2 0 654 Percent (Prior to Credible) 69% 10% 0% 43% WQ Parameter Results Prior to Confirmed 61 2 0 813 Percent (Prior to Confirmed) 71% 10% 0% 53% Table 6-5 displays the number of scenarios where rapid field test results played a role in elevating the threat level to a Credible or Confirmed determination with five contaminants relevant to this analysis. While there is not a wide availability of rapid field tests that have been verified for detection of specific contaminants in water, the tests that are available can be very useful to investigators in terms of focusing the follow-on laboratory analytics if field results suggest that a particular contaminant is present in the samples, which can expedite the overall investigation. Results of rapid field tests suggesting the presence of certain contaminants can also quickly elevate the credibility determination. Some of the tests are considered reliable enough by GCWW to elevate the investigation to a Confirmed determination which allows for rapid enactment of operational changes and public notification. Table 6-5. Scenarios with Rapid Field Test Results Available Prior to Credible and Confirmed Contamination Determination (simulation study results) Contaminant Nuisance Chemical 1 Nuisance Chemical 2 Toxic Chemical 1 Toxic Chemical 2 Toxic Chemical 3 Toxic Chemical 4 Toxic Chemical 5 Toxic Chemical 6 Toxic Chemical 7 Toxic Chemical 8 Biological Agent 1 Biological Agent 2 Biological Agent 3 Biological Agent 4 Biological Agent 5 Biological Agent 6 Biological Agent 7 All Contaminants Total Scenarios 119 - 119 119 - - - - 119 119 - - - - - - - 595 RFT Results Prior to Credible 73 - 1 0 - - - - 11 94 - - - - - - - 179 Percent (Prior to Credible) 61% - 1% 0% - - - - 9% 79% - - - - - - - 30% RFT Results Prior to Confirmed 81 - 4 50 - - - - 87 94 - - - - - - - 316 Percent (Prior to Confirmed) 68% - 3% 42% - - - - 73% 79% - - - - - - - 53% Laboratory results were available prior to a Credible determination for relatively few scenarios as shown in Table 6-6. It is likely the case that most scenarios were elevated to a Credible threat level based on information from the monitoring and surveillance components, the public health sector or site characterization. Laboratory results played a more noticeable role in elevating the threat level to Confirmed for various contaminants. 38 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Table 6-6. Scenarios with Laboratory Results Available Prior to Credible and Confirmed Contamination Determination (simulation study results) Contaminant Nuisance Chemical 1 Nuisance Chemical 2 Toxic Chemical 1 Toxic Chemical 2 Toxic Chemical 3 Toxic Chemical 4 Toxic Chemical 5 Toxic Chemical 6 Toxic Chemical 7 Toxic Chemical 8 Biological Agent 1 Biological Agent 2 Biological Agent 3 Biological Agent 4 Biological Agent 5 Biological Agent 6 Biological Agent 7 All Contaminants Total Scenarios 119 80 119 119 119 119 119 119 119 119 119 119 119 81 88 34 27 1738 Laboratory Results Prior to Credible 3 47 0 0 0 0 0 5 0 0 0 3 0 0 16 10 3 87 Percent (Prior to Credible) 3% 59% 0% 0% 0% 0% 0% 4% 0% 0% 0% 3% 0% 0% 18% 29% 11% 5% Laboratory Results Prior to Confirmed 56 55 0 0 12 0 46 29 0 0 0 89 39 1 61 10 3 401 Percent (Prior to Confirmed) 47% 69% 0% 0% 10% 0% 39% 24% 0% 0% 0% 75% 33% 1% 69% 29% 11% 23% Results from S&A also play an important role in public health response. While public health agencies will initiate some response actions before the identity of the contaminant is known, full public health response, including issuance of prophylaxis and consistent treatment of the injured will typically be implemented only after the identity of the contaminant is known. Results from either field testing or laboratory analysis thus may provide information critical to public health response. Tables 6-7 to 6-9 present the number of scenarios (for each contaminant and overall) in which either water quality parameter results, rapid field test results, or laboratory results were available prior to, and therefore played a role in activating the public health response. In these tables, no results are presented for Nuisance Chemicals 1 and 2 as these contaminants, while having the potential to render the water supply unusable, have relatively low toxicity and were not assumed to produce public health consequences in the simulation model. Table 6-7 shows that slightly less than half of all scenarios overall have water quality parameter results prior to public health response. However, there is a large difference between the toxic chemicals and Biological Agent 1 and the remaining contaminants. While water quality parameter results were available prior to the public health response in only 31% of scenarios or fewer for the toxic chemicals and Biological Agent 1, results for Biological Agents 2 through 7 were available prior to public health response in 98% to 100% of scenarios. Based on the simulation study results, it is more likely that water quality parameter results would play a role in activating a public health response in scenarios involving biological agents. Results were not available for the Toxic Chemical 8 as it did not result in a change in the water quality parameters under evaluation in the simulation model. 39 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Table 6-7. Scenarios with Water Quality Parameter Results Prior to Public Health Response (simulation study results) Contaminant Nuisance Chemical 1 Nuisance Chemical 2 Toxic Chemical 1 Toxic Chemical 2 Toxic Chemical 3 Toxic Chemical 4 Toxic Chemical 5 Toxic Chemical 6 Toxic Chemical 7 Toxic Chemical 8 Biological Agent 1 Biological Agent 2 Biological Agent 3 Biological Agent 4 Biological Agent 5 Biological Agent 6 Biological Agent 7 All Contaminants WQ Parameter Results before PH Response - - 31 4 10 13 4 6 6 - 6 98 98 41 67 4 2 390 Total Scenarios - - 99 98 92 99 98 99 96 - 100 98 100 42 67 4 2 1094 Percent - - 31% 4% 11% 13% 4% 6% 6% - 6% 100% 98% 98% 100% 100% 100% 36% Table 6-8 displays the number of scenarios where rapid field test results were available prior to public health response with four contaminants relevant to this analysis (i.e., the contaminants which could be detected by GCWW's rapid field test kits). While rapid field test results played a role in the public health response in 23% of the contamination scenarios for Toxic Chemical 1, they were not as instrumental to the response for scenarios involving contaminants Toxic Chemical 2, Toxic Chemical 7 and the Toxic Chemical 8 with 5%, 8%, and 0%, respectively, of results available prior to public health response. It is likely that information available from the other CWS components and from the public health community had more of an impact in activating the public health response than rapid field testing results. Table 6-8. Scenarios with Rapid Field Test Results Prior to Public Health Response (simulation study results) Contaminant Nuisance Chemical 1 Nuisance Chemical 2 Toxic Chemical 1 Toxic Chemical 2 Toxic Chemical 3 Toxic Chemical 4 Toxic Chemical 5 Toxic Chemical 6 Toxic Chemical 7 Toxic Chemical 8 Biological Agent 1 Biological Agent 2 RFT Results before PH Response - _ 23 5 - - _ - 8 0 - - Total Scenarios - _ 99 99 - - _ - 98 99 - - Percent - _ 23% 5% - - _ - 8% 0% - - 40 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Contaminant Biological Agents Biological Agent 4 Biological Agent 5 Biological Agent 6 Biological Agent 7 All Contaminants RFT Results before PH Response - _ - - - 36 Total Scenarios - _ - - - 395 Percent - _ - - - 9% Table 6-9 demonstrates that laboratory results did not play a large role in activating the public health response in most contamination scenarios for the toxic chemicals and three of the biological agents. Public health response was not modeled for the two nuisance chemicals so they are not included in Table 6-9. For two contaminants (Biological Agent 3 and Biological Agent 5), laboratory results clearly contributed to activating the public health response, as the results were available prior to the response in a high percentage of scenarios (100% and 83%, respectively). Due to differences in symptom onset time of the toxic chemicals and biological agents, it is thought that the simulation study results are similar to what may be expected in a real contamination incident. Laboratory results may not play a significant role in chemical incidents where data from the CWS monitoring and surveillance components and public health sector could result in initiating a public health response prior to the time that laboratory results are available. Even though laboratory results may not play a significant role in initiating a public health response, laboratory analysis of drinking water samples is critical in attributing illness to water consumption, regardless of when the results become available. Conversely, with longer symptom onset times, it may be that analytical results for biological agents are available prior to the time when a full response is initiated and less information may be available from monitoring and surveillance components. Table 6-9. Scenarios with Laboratory Results Prior to Public Health Response (simulation study results) Contaminant Toxic Chemical 1 Toxic Chemical 2 Toxic Chemical 3 Toxic Chemical 4 Toxic Chemical 5 Toxic Chemical 6 Toxic Chemical 7 Toxic Chemical 8 Biological Agent 1 Biological Agent 2 Biological Agent 3 Biological Agent 4 Biological Agents Biological Agent 6 Biological Agent 7 All Contaminants Laboratory Results before PH Response 0 5 12 3 3 0 2 0 0 19 100 0 83 24 6 257 Total Scenarios 99 99 97 99 98 100 98 99 100 98 100 100 100 87 93 1,467 Percent 0% 5% 12% 3% 3% 0% 2% 0% 0% 19% 100% 0% 83% 28% 6% 18% 41 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System 6.2 Summary Based on drill and exercise data, the S&A component demonstrates effective timeliness for response to possible contaminant incidents. In general, the time for response to a chemical contamination incident from recognition of a Possible incident to a Credible determination would be approximately 9 to 14 hours, depending on the contaminant. For a biological contamination incident, the estimated response timeline would be between 9 hours and 1.5 days, depending on the contaminant. As described above, the response timeline would increase if information was available from field sampling or from another CWS component about the suspected contaminant, which suggested that identification of an external support laboratory would be necessary. The benefits of evaluating the S&A component during multiple drills and exercises were demonstrated as utility personnel exhibited improved timeliness for certain key activities required for incident response sampling and analysis. When comparing the first four drills, which were based on WQM alerts, the time to pack up site characterization equipment and deploy to the contamination site decreased almost 50% from the first two drills to the April 2009 drill. During the September and October 2009 drills this metric increased in time, but this was likely due to a difference in drill scenarios. Full Scale Exercise 3 in October 2009 was based on a CCS alert and the process for the WUERM determining a sampling location may have increased the amount of time that the Site Characterization Team was required to wait between initial notification that equipment should be prepared and receiving orders to depart for the sampling location. Another metric, sample collection, can be dramatically decreased if only a grab sample is required. During most drills, the time for sample collection was relatively consistent including sub- sampling. Simulation study results demonstrated a consistent timeline availability of results from site characterization following a Possible contamination determination as the process is consistent regardless of the contaminant, though some time delays would occur if HazMat response was activated. More variability in the timeline from a Possible contamination determination to availability of laboratory results was observed (ranging from ~8 hours to nearly 2 days), and was expected due to differences in transport time to the GCWW laboratory vs. partner laboratories, and the time differences involved in analytical methods for chemical contaminants vs. biological agents. Simulation results analyses demonstrated variability among contaminants in terms of the role that water quality parameter results, rapid field tests and laboratory results played in activating the public health response, or in elevating the threat level. In general, results that are available sooner are more likely to have an impact on decisions to activate the public health response or to elevate the threat level, which ultimately relates to the decision to enact operational changes and issue public notification. Regardless of the timeliness of confirmatory laboratory results, however, the intrinsic value of laboratory confirmation is in the ability to attribute illness to contamination of drinking water in the distribution system. 42 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 7.0: Design Objective - Operational Reliability Analysis of the operational reliability considers metrics that quantify the overall availability of the five S&A sub-components (GCWW - field screening, GCWW - laboratory, ODH - radiochemistry laboratory, ODH - BT agent screening, and Test America, Savannah [TAS]) during maintenance monitoring, as well as the data completeness exhibited by the sub-components. Four metrics will be used to assess the operational reliability of the S&A component: availability, data completeness, method accuracy and method precision. 7.1 Availability Definition: The availability of sampling and analysis activities (GCWW and support laboratory methods) is defined as the percentage of time that sampling capabilities and all baseline analytical methods are operational and functioning within the limits of pre-established standards for acceptable operation during ongoing maintenance monitoring. These standards require full deployment potential and function of all S&A sub-components as they are currently defined and implemented according to GCWW's Site Characterization Plan. Data sources include laboratory inquiries (method and analyst availability), maintenance logs (equipment and reagents, QC data), and after action reports (drills and exercises). For performance evaluation purposes, 100% availability was assumed for any required emergency response support including GCWW site characterization and sampling teams as well as local (CFD/HazMat, CPD, CHD), state (Ohio EPA, ODH) and federal (Centers for Disease Control and Prevention [CDC], Federal Bureau of Investigation, EPA) partners. Analysis Methodology: Component availability was recorded through tracking of each sub-component (GCWW - field screening, GCWW - laboratory, ODH - radiochemistry laboratory, ODH - BT agent screening and TAS) on a monthly basis during the evaluation period. It was recorded as the percentage of time that each sub-component was available per month/total hours per month. The cause of any sub- component downtime was characterized and noted. For example, a data collection failure for the S&A component could be the result of field testing equipment malfunction or operator error, sampling errors (no sample collected or improperly collected), method or laboratory availability or analytical failure (method or laboratory performance). Any period of time longer than one hour that any sub-component was not available was considered downtime. Results: Only one instance of downtime was reported for the GCWW laboratory sub-component during the evaluation period due to a system event associated with the Hurricane Ike windstorm on September 14, 2008, which affected the entire GCWW service area with considerable damage and power outages throughout the Cincinnati area. The GCWW laboratory sub-component experienced downtime because Richard Miller Treatment Plant laboratory lost power for approximately 13 hours during the storm. During the power outage, maintenance monitoring was not impacted as no samples were in the process of being collected or analyzed. Utility personnel responded to the event by testing water samples throughout the service area using portable pH and chlorine meters, and collecting/processing an increased number of total coliform compliance monitoring samples. These sampling capabilities had been developed prior to implementation of the Cincinnati pilot, and the field sampling instrumentation used during the windstorm response had already been purchased by GCWW. No other S&A sub-components experienced any downtime during the evaluation period. S&A sub-component availability results are presented below in Table 7-1. 43 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Table 7-1. S&A Sub-component Availability S&A Sub-component GCWW - field sampling GCWW - laboratory ODH - radiochemistry lab ODH - BT agent screening TAS Availability (March 2008 -June 2010) 100% 99.9% 100% 100% 100% 7.2 Data Completeness Definition: Measurement of data completeness is based on the number of samples analyzed (per reporting period) from each of the five S&A sub-components (GCWW - field screening, GCWW - laboratory, ODH - radiochemistry laboratory, ODH - BT agent screening and TAS) including the samples prescribed in the maintenance monitoring schedule and those requested during drills/exercises or possible contamination incidents. Data is considered complete if received by the GCWW Laboratory Director or WUERM in usable condition. Analysis Methodology: Data completeness was tracked based on the amount of field- and laboratory- based data requested from each S&A sub-component (GCWW - field screening, GCWW - laboratory, ODH - radiochemistry laboratory, ODH - BT agent screening, and TAS) on a monthly basis during the evaluation period. It was recorded as a percentage for each sub-component (number of samples analyzed yielding usable sample results / number of samples requested) x 100%. Results: The number of samples requested for field and laboratory testing, both at GCWW and contract laboratories, varied by month, as per the maintenance monitoring schedule, and sample analysis required for drills and exercises. Field samples were collected and analyzed by GCWW every month of the evaluation period excluding one month (March 2009), resulting in 96% data completeness for field sampling at GCWW. With the exception of three months (March through May 2009) when no laboratory samples were collected or analyzed, all requested laboratory samples were completed by GCWW each month during maintenance monitoring, resulting in 88% data completeness for GCWW laboratory analysis. All maintenance monitoring samples collected for compliance monitoring requirements were analyzed as per the required schedule, resulting in 100% data completeness for the ODH laboratory. All sample analyses were completed by partner support laboratories for drills and exercises, resulting in 100% data completeness. These results are summarized in Table 7-2. Table 7-2. S&A Sub-component Data Completeness S&A Sub-component GCWW - field sampling GCWW - laboratory ODH - radiochemistry lab ODH - BT agent screening TAS - carbamate pesticides, total cyanide Data Completeness (March 2008 -June 2010) 96% 88% 100% 100% 100% 7.3 Method Accuracy Definition: Accuracy is a measure of the overall agreement of a measurement to a known value. For the S&A component, accuracy is defined as the extent [(measured concentration / nominal concentration) x 44 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System 100] to which the measured concentration of a targeted contaminant in reagent water agrees with the true or nominal concentration. This metric applies only to baseline sampling and analysis. Analysis Methodology: Component metrics for field and laboratory method accuracy for all contaminant classes identified for the Cincinnati pilot (Table 5-2) were documented during baseline monitoring. Accuracy is determined as percent recovery of QC samples (proficiency test, IDC, initial precision and recovery [IPR], Ongoing Precision and Recovery [OPR] and matrix spike samples) or performance evaluation samples. Performance evaluation sample results are used when available however, in the absence of commercially available performance evaluation samples, accuracy is determined by analysis of QC samples. Results: The method accuracy results presented below are separated into four sections: 1) field instrument accuracy estimates and QC check frequency, 2) initial demonstration of capability conducted by GCWW for SVOC and free cyanide analysis, 3) initial and ongoing precision and recovery conducted by GCWW for the LRN filter concentration procedure for BT agents and 4) method-specific recovery QC criteria. Field instrument accuracy was checked before each use, according to the manufacturers' instructions. Table 7-3 presents the field instrument accuracy estimates and GCWW's QC check frequency for all target parameters measured during baseline monitoring. The accuracy estimates are based on manufacturer information. Table 7-3. Field Instrument Accuracy Estimates and QC Check Frequency Instrument Portable Colorimeter Portable Electrochemical Detector Portable Turbidimeter Hand-held device Hand-held Device Test Kit Test Kit Text Kit HazCat Analyte Chlorine Cyanide ORP PH Conductivity Turbidity VOCs and combustible gases Radioactivity Chemical warfare agents General toxicity Arsenic Explosives, oxidants Accuracy Estimate +/-200 ug/L +/-50 ug/L +/- 1 mV +/-0.1 pH unit +/- 10 microsiemen +/- 0.01 NTU Semi-quantitative +/- 1 count per minute Detect/Non-detect Not available Semi-quantitative Semi-quantitative QC Check Frequency Daily Daily Daily Daily Daily Daily Daily Daily NA Daily Daily Daily Chemical Contaminants - Initial Demonstration of Capability For the priority contaminants and analytical methods identified for baseline monitoring in the Cincinnati pilot, an IDC was performed for each laboratory method that the laboratory had not yet been certified to conduct on drinking water samples. For the methods that the laboratory had already been certified by EPA for drinking water, no IDC was performed. For the targeted chemical analytes, this only included the SVOC method and automated colorimeter analysis for free cyanide, both performed at GCWW. The IDC established analyst proficiency, method performance (precision, accuracy and recovery) and minimum reporting limits for each method. IDCs were not performed for the remainder of the targeted chemical and radiochemical analytes/analytical methods, as the laboratories (GCWW, TAS, MASI, and ODH) held drinking water certification for the analyses performed during the study. 45 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System To demonstrate proficiency with the SVOC method and the automated colorimeter analysis for free cyanide, two proficiency test studies were conducted by the GCWW laboratory. Wibby Environmental, a National Environmental Laboratory Accreditation Conference (NELAC) accredited proficiency test vendor, prepared PT samples in July of 2007 and March of 2009, and GCWW analyzed them in the same months. Only the 2009 proficiency test study included samples for metals because GCWW did not have ICP-MS capabilities in 2007. NELAC acceptance limits were used for the contaminants, which are 99% confidence limits calculated using NELAC criteria. Three target analytes did not have established NELAC acceptance criteria, so acceptance criteria listed by NELAC for other similar compounds were used. The proficiency test recovery results from Wibby are summarized in Tables 7-4 and 7-5 below, for 2007 and 2009 respectively. Table 7-4. 2007 Proficiency Testing Sample Results Instrumentation Automated colorimeter Gas Chromatography with Mass Spectrometry Detection using liquid-solid extraction, multiple analytes PT Sample Concentration 0.424 mg/L 5.05 ug/L 10.8ug/L 8.9 ug/L 11. 7 ug/L Laboratory Result 0.074 mg/L 4.90 ug/L 10. 9 ug/L 14.3 ug/L 14.4 ug/L Percent Recovery 17.45% 97.03% 100.93% 160.67% 123.07% Acceptance Criteria 75-125% Correct Identification 55-145% 55-145% 55-145% Results for some of the analytes examined in the 2007 proficiency test tests did not show acceptable recovery. The cause of these unacceptable recoveries was investigated, resulting in some method improvements and also improvements in the specifications for and the handling of the proficiency test samples. As a result, the 2009 proficiency testing showed marked improvements in recovery. Table 7-5 demonstrates that the proficiency testing results were within acceptance criteria for the 2009 proficiency test samples. 46 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Table 7-5. 2009 Proficiency Testing Sample Results Instrumentation Automated colorimeter Gas Chromatography with Mass Spectrometry Detection using liquid-solid extraction, multiple analytes Inductively Coupled Plasma Spectrometry, multiple analytes PT Sample Concentration 0.253 mg/L 1.69 ug/L 10.6 ug/L 9.80 ug/L 9.62 ug/L 24.5 ug/L 5.22 ug/L Laboratory Result 0.216 mg/L 1.84 ug/L 8.44 ug/L 5.54 ug/L 8.78 ug/L 24.2 ug/L 3.30 ug/L Percent Recovery 85.38% 108.88% 79.62% 89.59% 57.59% 98.78% 63.22% Acceptance Criteria 75-125% Correct Identification 55-145% 55-145% 55-145% 55.1 -145% 60.0-145% BT Agents - Initial and Ongoing Precision and Recovery For the BT Agent Screening Protocol identified for baseline monitoring, GCWW performed initial and ongoing proficiency tests to demonstrate and monitor proficiency using the LRN filter concentration procedure. A protocol was established to determine recovery of a vegetative bacterial surrogate, which involves spiking a phosphate buffered saline (PBS) reference matrix for the initial and ongoing precision and recovery tests (IPR and OPR tests, respectively), or drinking water samples (matrix spikes) with viable enterococci (Enterococcus faecalis) to achieve a known concentration of target analyte (approximately 100 colony forming units [CPU]). The spiked sample was concentrated and target recovery determined by enumeration of enterococci in the concentrated sample (retentate) according to EPA Method 1600. GCWW analysts completed initial and ongoing demonstration of capability by conducting a total of 25 reference matrix (PBS) recovery determinations (IPR and OPR sample analyses). In addition, GCWW analysts analyzed a total of six (three duplicate sets) matrix spike samples. A summary of these results is provided in Table 7-6. 47 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Table 7-6. Summary of GCWW Filter Concentration Recovery Trials Sample Set 1 (IPR) 2(IPR) 3 (IPR) 4 (IPR) 5 (IPR) 6 (IPR) 7 (IPR) 8 (MS) 9 (MS) 10 (MS) 11 (OPR) 12(OPR) 13 (OPR) Date 10/13/2006 10/26/2006 01/04/2007 01/09/20073 01/23/2007 01/25/2007 02/01/2007 02/06/2007 02/27/2007 06/26/2007 11/14/2007 12/05/2007 02/19/2008 Enterococci Spike Recovery (CPU/filter) 2 0 20 61 232 62 58 56 76 54 62 54 55 67 76 65 72 83 39 56 59 50 59 47 64 Positive Control (CPU/filter) 32 27 (+ NaPP) 28 (- NaPP) 33 35 35 36 26 28 31 29 26 26 26 Negative Control (CPU/filter) 0 0 0 0 0 0 0 0 0 0 0 0 0 Enterococci % Recovery (Spike/Pos Cont) 1-2% 21% 6% 25%2 69% 64% 62% 84% 60% 69% 60% 61% 74% 84% 72% 80% 92% 43% 62% 66% 56% 66% 52% 71% Sample filtered without sodium polyphosphate (NaPP) amendment 2 Sample (> 50%) lost/spilled during membrane filtration 3 Sample concentration performed at slower rate (~ 0.4 L/min) Initial recovery trials (sample sets 1 and 2, Table 7-6) resulted in low reference matrix (PBS) recoveries of enterococci (2% to 20%). However, sample sets 3 through 13 indicated consistent enterococci recoveries greater than 50% (52% to 92%) for both reference (PBS) and matrix samples processed. Proficiency with the filter concentration procedure improved significantly during the course of these recovery trials (~2% to > 80%). This may simply be a reflection of improved technique as a function of practice and familiarity with the procedures. Consistent demonstration of enterococci recovery in excess of 50% would provide a high level of confidence in the efficiency of the filter concentration procedure. Target (enterococci) recoveries of 50% (lower limit) are considered acceptable, based on discussions with CDC method developers, ODH Laboratory and EPA. Duplicate matrix spike recovery determinations suggest that recovery of enterococci from GCWW drinking water (43% to 92%) is not that different than recovery from PBS samples (52% to 84%) based on a limited number of six available observations. 48 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Matrix Spike Samples Recovery QC criteria for the methods used in baseline monitoring for each contaminant class is shown in Table 7-7. The radiochemical laboratory (ODH) holds Ohio EPA drinking water certification for all of the analyses performed by that laboratory during this study. The laboratory performs method QC and regularly meets all QC acceptance criteria. Therefore, no matrix spike samples were specified or analyzed for radiochemical parameters during any sampling phase. The purpose of Phase 2 of baseline monitoring was to determine if the finished water from the two treatment plants and source waters are different with respect to contaminant occurrence or method performance; therefore, differences in mean recovery of spiked analytes between samples collected at the Bolton and Miller treatment plants were statistically analyzed. No differences in analyte concentration were observed between the treatment plants for SVOC or VOC matrix spike recoveries. A significant difference in analyte concentration of two of the carbamates and one of the metals was observed for matrix spike samples. These data sets were different at the 99% confidence level, indicating the impact of the drinking water matrix (i.e., water composition) on the analytical results. The objective of Phase 4 of baseline monitoring was to evaluate whether water within the piping of the distribution system affected analytical recovery. This was investigated by comparing matrix spike recoveries from Phase 4 samples with Phase 2 samples. Each plant was only compared to the Phase 4 sampling locations supplied by that plant. The matrix spike sample recoveries observed during Phase 4 differed from those observed during Phase 2 for one SVOC, one carbamate and one metal analyte, indicating some potential change in the drinking water matrix. It is unclear whether the cause of the differing matrix spike recoveries between Phases 2 and 4 was due to piping materials or some other factor (such as changing water composition). Table 7-7. Method-Specific Recovery QC Criteria for each Contaminant Class Contaminant Class Metals VOCs SVOCs Carbamates Radiochemicals BT Agents Number of Analytes 2 5 4 4 3 6 QC Specifications 70% to 130% 70% to 130% 70% to 130% and Detect/Non-detect 80% to 120% +/- 2 Std Deviations NA 7.4 Method Precision Definition: Precision is defined as the measure of agreement among repeated measurements of the same property under identical or substantially similar conditions; expressed generally in terms of the standard deviation. This metric applies only to baseline sampling and analysis. Analysis Methodology: Method precision was evaluated during baseline monitoring for all chemical priority contaminants (metals, volatiles, semi-volatiles and carbamate pesticides) identified for the Cincinnati pilot. Precision is determined as the percent relative standard deviation of replicate measurements [(standard deviation of replicate measurements / mean of measurements) * 100] at a mid- calibration range for laboratory-based methods. 49 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Results: Precision estimates for each field instrument and target parameter used during baseline monitoring are reported in Table 7-8 below. For some of the field instruments, precision was estimated to be 20% RPD based on manufacturer's information. Table 7-8. Field Instrument Precision Estimates Instrument Portable Colorimeter Portable Electrochemical Detector Portable Turbidimeter Hand-held device Hand-held Device Test Kit Test Kit Text Kit HazCat Analyte Chlorine Cyanide ORP PH Conductivity Turbidity VOCs and combustible gases Radioactivity Chemical warfare agents General toxicity Arsenic Explosives, oxidants Precision Estimate 20% RPD 20% RPD 20% RPD 20% RPD 20% RPD 20% RPD 20% RPD 20% RPD Detect/Non-detect Detect/Non-detect 20% RPD 20% RPD Precision QC criteria for the methods used in baseline monitoring for each contaminant class is shown in Table 7-9. Table 7-9. Method-Specific Precision QC Criteria for each Contaminant Class Contaminant Class Metals VOCs SVOCs Carbamates Radiochemicals BT Agents Number of Analytes 2 5 4 4 3 6 Precision (maximum RSD) 20% RSD 20% RSD 30% RSD and Detect/Non-detect 20% RSD +/- 2 Std Deviations NA 7.5 Summary The S&A methods and laboratories effectively met the design objective of operational reliability, as data collected during the evaluation period demonstrated overall availability, reliability and acceptable method performance. During the course of 26 months of maintenance monitoring, only one S&A sub-component (GCWW - laboratory) experienced a short period of downtime (13 hours) due to a highly unusual windstorm and subsequent power outage in the Cincinnati area. Though this downtime occurred, the GCWW laboratory sub-component demonstrated a high percentage of availability overall: 99.9%. The remaining four sub-components (GCWW - field screening, ODH - radiochemistry laboratory, ODH - BT agent screening and TAS) were continually available throughout the duration of the evaluation period (100%). The high data completeness percentages recorded for each of the S&A sub-components for the overall evaluation period (Table 7-2) demonstrated GCWW's successful transition from baseline monitoring to maintenance monitoring. The majority of samples that were prescribed by the maintenance monitoring 50 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System schedule, or for drills and exercises, were collected and analyzed. Finally, method accuracy and method precision data were within established method limits/tolerances during baseline monitoring for each of the methods and laboratories supporting the S&A component. 51 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 8.0: Design Objective - Sustainability Sustainability is a key objective in the design of a CWS and each of its components, which for the purpose of this evaluation is defined in terms of the cost-benefit trade-off. Costs are estimated over the lifecycle of the system to provide an estimate of the total cost of ownership, including the capital cost to implement the system and the cost to operate and maintain the system. The benefits derived from the system are defined in terms of primary and dual-use benefits. The primary benefit of a CWS is the potential reduction in consequences in the event of a contamination incident; however, such a benefit may be rarely, if ever, realized. Thus, dual-use benefits that provide value to routine utility operations are an important driver for Sustainability of the system. Ultimately, the Sustainability of the system is also reflected by the ability of utility and partner agencies to uphold and apply the protocols and procedures necessary to operate and maintain the CWS. The three metrics that will be evaluated to assess how well the Cincinnati CWS met the design objective of Sustainability are: costs, benefits, and compliance. The following subsections define each metric, describe how it was evaluated, and present the results. 8.1 Costs Definition: Costs are evaluated over the 20-year lifecycle of the Cincinnati CWS, and comprise costs incurred to design, deploy, operate and maintain the S&A component since its inception. Analysis Methodology: Parameters used to quantify the implementation cost of the S&A component were extracted from the Water Security Initiative: Cincinnati Pilot Post-Implementation System Status (USEPA, 2008). The cost of modifications to the S&A component made after the completion of implementation activities were tracked as they were incurred. O&M costs were tracked on a monthly basis over the duration of the evaluation period. Renewal and replacement costs, along with the salvage value at the end of the Cincinnati CWS lifecycle were estimated using vendor supplied data, field experience and expert judgment. Note that all costs reported in this section are rounded to the nearest dollar. Section 3.5 provides additional details regarding the methodology used to estimate each of these cost elements. Results: The methodology described in Section 3.5, was applied to determine the value of the major cost elements used to calculate the total lifecycle cost of the S&A component, which are presented in Table 8- 1. It is important to note that the Cincinnati CWS was a research effort, and as such incurred higher costs than would be expected for a typical large utility installation. A similar S&A component implementation at another utility should be less expensive as it could benefit from lessons learned and would not incur research-related costs. Table 8-1. Cost Elements used in the Calculation of Lifecycle Cost Parameter Implementation Costs Annual O&M Costs Renewal and Replacement Costs1 Salvage Value1 Value $2,543,918 $42,795 $260,482 ($11,269) Calculated using major pieces of equipment as presented in Table 8-4. Table 8-2 below presents the implementation cost for each S&A design element, with labor costs presented separately from the cost of equipment, supplies and purchased services. 52 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Table 8-2. Implementation Costs Design Element Project Management Field and Laboratory Testing Capabilities Routine Sampling and Analysis Incident Response Sampling and Analysis TOTAL: Labor $102,749 $366,817 $1,078,384 $412,790 $1,960,740 Equipment, Supplies, Purchased Services - $135,876 $197,263 $319,539 $652,679 Component Modifications (deletions in parentheses) - - - ($69,500) ($69,500) Total Implementation Costs $102,749 $502,694 $1,275,647 $662,829 $2,543,918 1 Project management costs incurred during implementation were distributed evenly among the CWS components. The first design element, project management, includes overhead activities necessary to design and implement the component. The field and laboratory testing capabilities design element includes the analytical equipment required for field screening and sampling kits were identified and provided to GCWW. This also includes the process of establishing communication between GCWW and HazMat. The third design element, routine sampling and analysis, includes design and execution of baseline monitoring to achieve defined objectives. Based on the results of baseline monitoring, the follow-on maintenance monitoring program was developed. The final design element, incident response sampling and analysis, includes the cost of defining analytical requirements and addressing gaps by providing GCWW the equipment needed to perform SVOC analysis, as well as ultrafiltration concentration. A laboratory network capable of analyzing drinking water samples was established. Overall, the routine sampling and analysis design element had the highest implementation costs (50%). A significant amount of labor was involved in designing the baseline monitoring program, collecting and analyzing samples, and conducting statistical analysis using analytical results. The total implementation cost for field and laboratory testing capabilities and incident response sampling and analysis were lower at 19% and 26%, respectively. Implementation costs for project management were significantly lower at 4%. The component modification costs represent the labor, equipment, supplies, and purchased services associated with enhancements to the S&A component after completion of major implementation activities in December 2007. The cost associated with the SmartCycler PCR instrument was eliminated as the equipment was not utilized and was transferred to an EPA laboratory. Similarly, the cost associated with the toxicity test kits was eliminated as the utility decided to discontinue use of the toxicity testing capability due to unacceptable variability in the assay response when conducted in the field by various field personnel. The annual labor hours and costs of operating and maintaining the S&A component, broken out by design element, are shown in Table 8-3. 53 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Table 8-3. Annual O&M Costs Design Element1 Field Test/Chemistry Supplies Procedures TOTAL: Total Labor (hours/year) - 615 615 Total Labor Cost ($/year) - $23,795 $23,795 Supplies and Purchased Services ($/year) $19,000 - $19,000 Total O&M Cost ($/year) $19,000 $23,795 $42,795 1 Overarching project management costs were only incurred during implementation of the S&A component and are not applicable for annual O&M costs. Annual O&M costs for the field test and chemistry supplies include those costs related directly to ongoing maintenance of field and laboratory equipment (i.e., Gas Chromatograph-Mass Spectrometer (GC-MS) warranty, service and replacement costs [site characterization instruments], purchase of reagents and standards). Most of the O&M labor hours reported under procedures are spent on maintenance monitoring, in-house training and drills and exercises. Two of the major cost elements presented in Table 8-1, the renewal and replacement costs and salvage value, were based on costs associated with major pieces of equipment installed for the S&A component. One of the biggest expenditures was a GC-MS for semi-volatile analyses. The utility also procured field equipment, including volatile gas and radiation meters. To calculate the total lifecycle cost of the S&A component, all costs and monetized benefits were adjusted to 2007 dollars using the change in the Consumer Price Index between 2007 and the year that the cost or benefit was realized. Subsequently, the implementation costs, renewal and replacement costs, and annual O&M costs were combined, and the salvage value was subtracted to determine the total lifecycle cost: S&A Total Lifecycle Cost: $3,436,060 Note that in this calculation, the implementation costs and salvage value were treated as one-time balance adjustments, the O&M costs recurred annually, and the renewal and replacement costs for major equipment items were incurred at regular intervals based on the useful life of each item. 8.2 Benefits Definition: The benefits of CWS deployment can be considered in two broad categories: primary and dual-use. Primary benefits relate to the application of the CWS to detect contamination incidents, and can be quantified in terms of a reduction in consequences. Primary benefits are evaluated at the system-level and are thus discussed in the report titled Water Security Initiative: Evaluation of the Cincinnati Contamination Warning System Pilot (USEPA, 2014c). Dual-use benefits are derived through application of the CWS to any purpose other than detection of intentional and unintentional drinking water contamination incidents. Dual-use benefits realized by the S&A component are presented in this section. Analysis Methodology: Information collected from forums, such as data review meetings, lessons learned workshops and interviews were used to identify dual-use applications of the S&A component of the CWS. 54 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Results: Operation of the S&A component of the CWS has resulted in benefits beyond providing field and laboratory analytical response to contamination incidents. These key dual-use benefits and examples identified by the utility include: 1. Ongoing practice of field and laboratory response protocols: • Practice of standard protocols for site characterization, sample collection, sample analysis and data interpretation in response to CWS alerts as part of drills and exercises afforded GCWW an opportunity to fine-tune skills that may be beneficial to areas beyond response to possible contamination incidents. In addition, these drills and exercises improved partnerships with agencies involved in any hazard event, including CFD, HazMat and partner laboratories. 2. Availability of field equipment and laboratory instrumentation for other projects: • Acquisition of equipment for purposes of detecting priority contaminants can also be utilized for other sampling procedures. For example, the purchase of GC-MS instrumentation for SVOC analysis has enabled GCWW to perform in-house compliance monitoring, and allows them to offer this analytical capability to other utilities. In addition, utility personnel reported that the 800 MHz hand-held radios, procured under the site characterization project area to enhance field communications, proved to be extremely useful for communications between personnel deployed in the field and the Incident Commander during response to the September 14, 2008 windstorm. 3. Improved water quality from expanded analytical capability: • Expanded analytical ability during baseline monitoring allowed analysis of many samples for metals which GCWW does not normally target. During this testing, it was discovered that some areas of the distribution system contained significantly higher levels of iron than others; these sites corresponded to sites where "water age" was greater than average. When GCWW has knowledge of these areas, GCWW flushes hydrants in these areas often. After flushing, iron concentrations in these areas were lowered to the same levels as other service areas. GCWW had previous knowledge of some high iron areas, but this program resulted in identification of more areas that would benefit from flushing. Improved water quality in these areas was a direct result of baseline monitoring as part of the CWS. 8.3 Compliance Definition: Compliance captures the acceptability of the S&A component by measuring the willingness of persons and organizations to monitor, maintain, and actively participate in the CWS. The use of each S&A activity (sampling and laboratory analysis) during drills and exercises, maintenance monitoring and during incident response is tracked to represent the acceptability of the CWS. Analysis Methodology: This metric is measured by documenting the percentage of maintenance monitoring samples collected per month by GCWW personnel and analyzed per month by GCWW and partner support laboratories, as specified in the maintenance monitoring plan. Another measure of compliance is the attendance of utility staff in drills and/or exercises (# staff in attendance / # staff expected to attend). Results: Overall, the S&A component demonstrated excellent compliance throughout the evaluation period. 100% attendance was documented at all scheduled trainings, drills and exercises. In addition, a high percentage of maintenance monitoring was completed; with the exception of one month for GCWW 55 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System field sampling (March 2009) and three months for GCWW laboratory analysis (March through May 2009), 100% of all maintenance monitoring samples were collected and analyzed (Table 7-2). These compliance measures bolster sustainability by providing a strong indicator that the utility was easily able to comply with component procedures during the pilot, which bodes well for their continuing interest in doing so. For instance, it is apparent that the utility has been able to incorporate maintenance monitoring sample collection and analysis into routine sampling routes and laboratory analyses. This indicates that the component procedures do not represent an excessive intrusion into routine activities, but rather represent value-added. 8.4 Summary While the total implementation cost of the S&A component was $2,543,918, the ongoing cost for O&M is $42,795. This much lower annual cost required to maintain the component, which is a small fraction of GCWW's overall O&M budget for its various operations, supports the long-term viability of the component. While the O&M cost relates directly to maintenance monitoring for priority contaminants, many of the samples collected for maintenance monitoring also support compliance monitoring for regulated contaminants. During the evaluation period, the utility achieved a high compliance rate for collecting and analyzing most samples required per the maintenance monitoring schedule, which demonstrates overall ability of the utility to implement the component, as currently designed. The utility has also derived many dual-use benefits from implementation of the S&A component including increased preparedness for responding to all hazard incidents, improved familiarity towards working with emergency response partners and partner laboratories and increased in-house field and laboratory capabilities. 56 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 9.0: Summary and Conclusions This document provides a comprehensive evaluation of how effectively the S&A component of the Cincinnati pilot achieved the five applicable CWS design objectives used to characterize performance: spatial coverage, contaminant coverage, timeliness of contaminant detection, operational reliability, and sustainability. To conduct the evaluation, data sources including empirical data, drill and exercise data, forums (including monthly staff interviews and a lessons-learned workshop), and cost data were utilized. Overall, the personnel supporting the S&A component demonstrated exceptional performance with respect to each of the design objectives. GCWW personnel, response partners, and contract laboratories invested considerable time during design and implementation of the component to achieve the design goals and to ensure acceptable performance of routine and incident response procedures. Furthermore, the utility and response partners demonstrated dedication to the pilot study through regular attendance at activities designed to evaluate the component including multiple component drills, full scale exercises, and the lessons learned workshop. For spatial coverage, the utility effectively collected and analyzed samples at designated strategic, priority, and survey sampling locations throughout the distribution system during baseline monitoring. Currently, ample baseline data for each location evaluated during baseline monitoring is available and is stored in relevant locations in GCWW's laboratory including a pre-existing Water Quality and Treatment database, spreadsheet databases (for field test data) or on instrumentation computers (e.g., GC-MS library) for tentatively identified compound data and method performance data). This data can be accessed for analysis by the utility, and will be utilized during incident response sampling and analysis. During an incident, historical data would be needed to compare with incident response data. Following completion of baseline monitoring, the utility transitioned to maintenance monitoring and is continuing to collect samples from 31 strategic locations throughout the distribution system to maintain proficiency in field and laboratory methods and to update contaminant baseline data. For contaminant coverage, GCWW achieved successful implementation of field and laboratory methods for all target water quality parameters and priority contaminants identified during design of the component. Furthermore, through enhancement of field and laboratory capabilities, the utility is now able to target a wide variety of possible water contaminants, and is familiar with the process of identifying a capable support laboratory if necessary during a Possible contaminant incident. Simulation study results demonstrated a contaminant scenario detection rate of 83% for site characterization results (water quality parameter tests and rapid field tests) and 86% for laboratory analysis results. For timeliness of contaminant detection, the utility exhibited improved response procedures during subsequent component drills and exercises, and reduced the time required for certain key activities required for sampling and analysis incident response. Based on data gathered during drills and exercises, the timeline for incident response, from recognition of a Possible incident to a Credible determination would be between 9 hours and 1.5 days, depending on the contaminant. Simulation study results demonstrated an average availability of site characterization results (varied, but were within the timeframe to take significant actions.) within ~3 hours of the Possible determination across all relevant contamination scenarios, and an average availability of laboratory results within -18 hours across all relevant contamination scenarios. Variability in the data from the simulation study was observed among contaminants in terms of the role that water quality parameter results, rapid field tests and laboratory results played in activating the public health response, or in elevating the threat level. In general, results that are available sooner are more likely to have an impact on decisions to activate the public health 57 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System response or to elevate the threat level, which ultimately relates to the decision to enact operational changes and issue public notification. The S&A component effectively met the design objective of operational reliability, as data collected during the evaluation period demonstrated the overall stability of component operations. During the course of 26 months of maintenance monitoring, only one short period of downtime (13 hours) was experienced by the GCWW laboratory sub-component. The remaining four sub-components (GCWW - field screening, ODH - radiochemistry laboratory, ODH - BT agent screening, and TAS) were continually available throughout the duration of the evaluation period (100%). Furthermore, high data completeness percentages were recorded for each of the five S&A sub-components (> 88%). Metrics data used to characterize the sustainability of the S&A component exemplifies the long-term viability of the component. During the evaluation period, the utility achieved a high compliance rate for collecting and analyzing most samples required per the maintenance monitoring schedule, which demonstrates overall acceptability of the component, as currently designed. Furthermore, the utility has absorbed the cost for operation and maintenance of the component, and has designated personnel to support ongoing sampling and analysis efforts associated with maintenance monitoring. The dual-use benefits that have been afforded from implementation of the S&A component also support the long-term stability of the component, including increased preparedness for responding to all hazard events, improved familiarity towards working with emergency response partners and partner laboratories and increased in-house field and laboratory capabilities. One of the primary limitations of this analysis is the absence of data from an actual contamination incident. While it is clearly not desirable that a contaminant incident occur, data from such an incident would be useful to accurately characterize S&A component performance with respect to many of the design objectives and their associated metrics. Though drills and exercises were extremely beneficial towards improving GCWW and response partner familiarity with incident response procedures, and improving overall timeliness of response, it is important to remember that these drills and exercises only provide estimates of the times involved. For instance, these drills and exercises only occurred during normal working hours; it is expected off-hour incidents would have different timelines. In summary, the S&A component was effective in meeting each of the five design objectives established for the pilot CWS in Cincinnati, and is adequately prepared to help implement the Cincinnati Pilot Consequence Management Plan when any of the other early detection CWS components suggest Possible contamination. As noted earlier, the component drills and full scale exercises were identified as one of the most valuable aspects of the CWS. These events allow the utility to practice and refine response procedures, become familiar with field test kits and equipment, understand roles and responsibilities when working with emergency response partners, and to practice packing, shipping and proper documentation for samples being shipped to external support laboratories. While this evaluation is specific to the S&A component deployed in Cincinnati, it should aid other utilities in design and implementation of an S&A component as part of a CWS or to simply improve their existing S&A program for responding to contamination incidents. 58 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 10.0: References U.S. Department of Homeland Security. 2013. Homeland Security Exercise and Evaluation Program. https://hseep.dhs.gov/support/HSEEP_Revision_Aprl3_Final.pdf U.S. Environmental Protection Agency. 2008. Water Security Initiative: Cincinnati Pilot Post- Implementation System Status. EPA 817-R-08-004. U.S. Environmental Protection Agency. 2013. Water Security Initiative: Guidance for Building Laboratory Capabilities to Respond to Drinking Water Contamination. EPA 817-R-13-001. U.S. Environmental Protection Agency. 2014a. Water Security Initiative: Comprehensive Evaluation of the Cincinnati Contamination Warning System Pilot EPA 817-R-14-001. U.S. Environmental Protection Agency. 2014b. Water Security Initiative: Evaluation of the Consequence Management Component of the Cincinnati Contamination Warning System Pilot. EPA817-R-14-001F. U.S. Environmental Protection Agency. 2014c. Water Security Initiative: System Performance Evaluation of the Cincinnati Contamination Warning System Pilot. EPA 817-R-14-001A. 59 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 11.0: Abbreviations The list below includes acronyms approved for use in the S&A component evaluation. Acronyms are defined at first use in the document BT Bioterrorism-Threat CCS Customer Complaint Surveillance CDC Centers for Disease Control and Prevention CFD Cincinnati Fire Department CPU Colony-forming Unit CHD Cincinnati Health Department CPD Cincinnati Police Department CWS Contamination Warning System DHS Department of Homeland Security EPA Environmental Protection Agency ESM Enhanced Security Monitoring GC-MS Gas Chromatograph-Mass Spectrometer GCWW Greater Cincinnati Water Works HazMat Hazardous Material Response HPLC High Performance Liquid Chromatography ICP-MS Inductively Coupled Plasma-Mass Spectrometer IDC Initial Demonstration of Capacity IPR Initial Precision and Recovery LRN Laboratory Response Network MASI Mobile Analytical Services, Inc MS Matrix Spike NELAC National Environmental Laboratory Accreditation Conference ODH Ohio Department of Health OPR Ongoing Precision and Recovery ORP Oxygen Reduction Potential PBS Phosphate Buffered Saline PCB Polychlorinated biphenyls PCR Polymerase Chain Reaction PHS Public Health Surveillance PT Proficiency Test QC Quality Control RFT Rapid Field Testing S&A Sampling and Analysis SVOC Semi-volatile Organic Compound TAS Test America, Savannah VOC Volatile Organic Compound WQ Water Quality WQM Water Quality Monitoring WUERM Water Utility Emergency Response Manager 60 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Section 12.0: Glossary Accuracy. Accuracy is a measure of the overall agreement of a measurement to a known value. Alert. Information from a monitoring and surveillance component indicating an anomaly in the system, which warrants further investigation to determine if the alert is valid. Alert Investigation. A systematic process, documented in a standard operating procedure, for determining whether or not an alert is valid, and identifying the cause of the alert. If an alert cause cannot be identified, contamination is Possible. Baseline. Normal conditions that result from typical system operation. The baseline includes predictable fluctuations in measured parameters that result from known changes to the system. For example, a water quality baseline includes the effects of draining and filling tanks, pump operation, and seasonal changes in water demand, all of which may alter water quality in a somewhat predictable fashion. Baseline analysis. In the simulation study, analytical methods performed by laboratories (GCWW and its partners) capable of supporting analyses included in GCWW's baseline suite. These laboratory partnerships were established during the evaluation period. Benefit. An outcome associated with the implementation and operation of a contamination warning system that promotes the welfare of the utility and the community it serves. Benefits are classified as either primary or dual-use. Benefit-cost analysis. An evaluation of the benefits and costs of a project or program, such as a contamination warning system, to assess whether the investment is justifiable considering both financial and qualitative factors. Biotoxins. Toxic chemicals derived from biological materials that pose an acute risk to public health at relatively low concentrations. Bolton. The Greater Cincinnati Waterworks' Charles M. Bolton Treatment Plant. Bulk volume (of contaminant). The total volume of a contaminant solution that is injected into the distribution system during a contamination scenario. Confirmed. In the context of the threat level determination process, contamination is Confirmed when the analysis of all available information from the contamination warning system has provided definitive, or nearly definitive, evidence of the presence of a specific contaminant or class of contaminant in the distribution system. While positive results from laboratory analysis of a sample collected from the distribution system can be a basis for confirming contamination, a preponderance of evidence, without the benefit of laboratory results, can lead to this same determination. Confirmatory methodology. A methodology that confirms, with high confidence, the presence of a contaminant or suggests conclusively that it is absent. Consequence management. Actions taken to plan for and respond to possible contamination incidents. This includes the threat level determination process, which uses information from all monitoring and surveillance components as well as sampling and analysis to determine if contamination is credible or 61 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System confirmed. Response actions, including operational changes, public notification and public health response, are implemented to minimize public health and economic impacts and ultimately return the utility to normal operations. Consequence management plan. Documentation that provides a decision-making framework to guide investigative and response activities implemented in response to a Possible contamination incident. Contamination incident. The introduction of a contaminant in the distribution system with the potential to cause harm to the utility or the community served by the utility. A contamination incident may be intentional or accidental. Contamination scenario. Within the context of the simulation study, parameters that define a specific contamination incident, including: injection location, injection rate, injection duration, time the injection is initiated and the contaminant that is injected. Contamination warning system. An integrated system of monitoring and surveillance components designed to detect contamination in a drinking water distribution system. The system relies on integration of information from these monitoring and surveillance activities along with timely investigative and response actions during consequence management to minimize the consequences of a contamination incident. Costs, implementation. Installed cost of equipment, IT components, and subsystems necessary to deploy an operational system. Implementation costs include labor and other expenditures (equipment, supplies, and purchased services). Cost, life cycle. The total cost of a system, component, or equipment over its useful or practical life. Life cycle cost includes the cost of implementation, operation & maintenance and renewal & replacement. Costs, operation & maintenance. Expenses incurred to sustain operation of a system at an acceptable level of performance. Operational and maintenance costs are reported on an annual basis, and include labor and other expenditures (supplies and purchased services). Costs, renewal & replacement. Costs associated with refurbishing or replacing major pieces of equipment (e.g., water quality sensors, laboratory instruments, IT hardware, etc.) that reach the end of their useful life before the end of the contamination warning system lifecycle. Coverage, contaminant. Specific contaminants that can potentially be detected by each monitoring and surveillance component, including sampling & analysis, of a contamination warning system. Coverage, spatial. The areas within the distribution system that are monitored by, or protected by, each monitoring and surveillance component of a contamination warning system. Credible. In the context of the threat level determination process, a water contamination threat is characterized as Credible if information collected during the investigation of Possible contamination corroborates information from the validated contamination warning system alert. Data completeness. The amount of data that can be used to support system or component operations, expressed as a percentage of all data generated by the system or component. Data may be lost due to QC failures, data transmission errors and faulty equipment among other causes. 62 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Distribution system model. A mathematical representation of a drinking water distribution system, including pipes, junctions, valves, pumps, tanks, reservoirs, etc. The model characterizes flow and pressure of water through the system. Distribution system models may include a water quality model that can predict the fate and transport of a material throughout the distribution system. Dual-use benefit. A positive application of a piece of equipment, procedure, or capability that was deployed as part of the contamination warning system, in the normal operations of the utility. Ensemble. The comprehensive set of contamination scenarios evaluated during the simulation study. Evaluation period. The period from January 16, 2008 to June 15, 2010 when data was actively collected for the evaluation of the Cincinnati contamination warning system pilot. The evaluation period for S&A was from March 2008 to June 2010. Field results. Field results include information collected from Site Characterization activities including the site hazard assessment, field safety screening, water quality testing and rapid field tests. This does not include the results of the laboratory analysis conducted on samples collected at the end of the site characterization process. HazMat. A specially trained unit of professionals with the responsibility of containing incidents related to hazardous materials. This organization plays a critical role in consequence management including site characterization activities to support credibility determination. Hydraulic connectivity. Points or areas within a distribution system that are on a common flow path. Incident Commander. In the Incident Command System, the individual responsible for all aspects of an emergency response; including quickly developing incident objectives, managing incident operations and allocating resources. Incident timeline. The cumulative time from the beginning of a contamination incident until response actions are effectively implemented. Elements of the incident timeline include: time for detection, time for alert validation; time for threat level determination and time to implement response actions. Job function. A description of the duties and responsibilities of a specific job within an organization. Maintenance monitoring. A phase of sampling and analysis which occurs after completion of baseline monitoring. During maintenance monitoring, routine sampling confirms there are no changes in baseline contaminant occurrence or method performance during normal (i.e., non-incident) sampling and analysis. Metric. A standard or statistic for measuring or quantifying an attribute of the contamination warning system or its components. Miller. Greater Cincinnati Water Works' Richard Miller Treatment Plant Model. A mathematical representation of a physical system. Model parameters. Fixed values in a model that define important aspects of the physical system. Module. A sub-component of a model that typically represents a specific function of the real-world system being modeled. 63 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Monitoring & surveillance component. Element of a contamination warning system used to detect unusual water quality conditions, potentially including contamination incidents. The four monitoring & surveillance components of a contamination warning system include: 1) online water quality monitoring, 2) enhanced security monitoring, 3) customer complaint surveillance and 4) public health surveillance. Nuisance chemicals. Chemical contaminants with a relatively low toxicity, which thus generally do not pose an immediate threat to public health. However, contamination with these chemicals can make the drinking water supply unusable. Optimization phase. Period in the contamination warning system deployment timeline between the completion of system installation and real-time monitoring. During this phase the system is operational, but not expected to produce actionable alerts. Instead, this phase provides an opportunity to learn the system and optimize performance (e.g., fix or replace malfunctioning equipment, eliminate software bugs, test procedures and reduce occurrence of invalid alerts). Pathogens. Microorganisms that cause infections and subsequent illness and mortality in the exposed population. Possible. In the context of the threat level determination process, a water contamination threat is characterized as Possible if the cause of a validated contamination warning system alert is unknown. Precision. Precision is defined as the measure of agreement among repeated measurements of the same property under identical, or substantially similar conditions; expressed generally in terms of the standard deviation. Primary benefits. Benefits that are derived from the reduction in consequences associated with a contamination incident due to deployment of a contamination warning system. Priority contaminant. A contaminant that has been identified by the EPA for monitoring under the Water Security Initiative. Priority contaminants may be initially detected through one of the monitoring and surveillance components and confirmed through laboratory analysis of samples collected during the investigation of a Possible contamination incident. Public health incident. An occurrence of disease, illness, or injury within a population that is a deviation from the disease baseline in the population. Public health response. Actions taken by public health agencies and their partners to mitigate the adverse effects of a public health incident, regardless of the cause of the incident. Potential response actions include: administering prophylaxis, mobilizing additional healthcare resources, providing treatment guidelines to healthcare providers, and/or providing information to the public. Radiochemicals. Chemicals that emit alpha, beta, and/or gamma particles at a rate that could pose a threat to public health. Real-time monitoring phase. Period in the contamination warning system deployment timeline following the optimization phase. During this phase, the system is fully operational and is producing actionable alerts. Utility staff and partners now respond to alerts in real-time and in full accordance with standard operating procedures documented in the operational strategy. Optimization of the system still occurs as part of a continuous improvement process, however the system is no longer considered to be developmental. 64 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System Routine operation. The day-to-day monitoring and surveillance activities of the contamination warning system that are guided by the operational strategy. To the extent possible, routine operation of the contamination warning system is integrated into the routine operations of the drinking water utility. Safety screening. Portable field screening methodologies (e.g., volatile gas detectors or radioactivity meters) used by the Site Characterization Team during site approach to scan the area in the vicinity of the sampling location for potential hazards such as toxic gases or radioactivity. Salvage value. Estimated value of assets at the end of the useful life of the system. Screening methodology. An analytical methodology that may identify a contaminant, but does not provide a high level of confidence that a specific contaminant is present. Select agents. Biological agents and toxins (as declared by the U.S. Department of Health and Human Services) that have the potential to pose a severe threat to public health and safety and as such, their possession, use, or transfer is regulated. Simulation study. A study designed to systematically characterize the detection capabilities of the Cincinnati drinking water contamination warning system. In this study, a computer model of the contamination warning system is challenged with an ensemble of 2,023 simulated contamination scenarios. The output from these simulations provides estimates of the consequences resulting from each contamination scenario, including fatalities, illnesses, and extent of distribution system contamination. Consequences are estimated under two cases, with and without the contamination warning system in operation. The difference provides an estimate of the reduction in consequences. Site characterization. The process of collecting information from an investigation site to support the investigation of a contamination incident during consequence management. Threat level. The results of the threat level determination process, indicating whether contamination is Possible, Credible or Confirmed. Threat level determination process. A systematic process in which all available and relevant information available from a contamination warning system is evaluated to determine whether the threat level is Possible, Credible or Confirmed. This is an iterative process in which the threat level is revised as additional information becomes available. The conclusions from the threat evaluation process are considered during consequence management when making response decisions. Time for Confirmed determination. A portion of the incident timeline that begins with the determination that contamination is Credible and ends with contamination either being Confirmed or ruled out. This includes the time required to perform lab analyses, collect additional information, and analyze the collective information to determine if the preponderance of evidence confirms the incident. Time for contaminant detection. A portion of the incident timeline that begins with the start of contamination injection and ends with the generation and recognition of an alert. The time for contaminant detection may be subdivided for specific components to capture important elements of this portion of the incident timeline (e.g., sample processing time, data transmission time, event detection time, etc.). Time for Credible determination. A portion of the incident timeline that begins with the recognition of a Possible contamination incident and ends with a determination regarding whether contamination is Credible. This includes the time required to perform multi-component investigation and data integration, 65 ------- Water Security Initiative: Evaluation of the Sampling and Analysis Component of the Cincinnati Contamination Warning System implement field investigations (such as site characterization and sampling), and collect additional information to support the investigation. Toxic chemicals. Highly toxic chemicals that pose an acute risk to public health at relatively low concentrations. Triggered analysis. In the simulation study, certain analytical methods were performed by laboratories who were capable of supporting analyses outside of GCWW's baseline suite. While these laboratory partnerships had not been established during the evaluation period, telephone contact was made to ascertain information regarding sample analysis turnaround time and shipping logistics to accurately parameterize the process in the Cincinnati contamination warning system model. Water Utility Emergency Response Manager. A role within the Cincinnati contamination warning system filled by a mid-level manager from the drinking water utility. Responsibilities of this position include: receiving notification of validated alerts, verifying that a valid alert indicates Possible contamination, coordinating the threat level determination process, integrating information across the different monitoring and surveillance components, and activating the consequence management plan. In the early stages of responding to Possible contamination, the Water Utility Emergency Response Manager may serve as Incident Commander. 66 ------- |