www. e pa. go v/resea rc h
technical BRIEF
INNOVATIVE RESEARCH FOR A SUSTAINABLE FUTURE
Deployment of Real-Time Analytics and Modeling at the
City of Flint, Michigan Water System
INTRODUCTION
The City of Flint, Michigan owns and operates the public water system that provides drinking water to
its nearly 100,000 residents. In 2014 the City of Flint, Michigan ("City") ceased drawing treated water
under its agreement with the Detroit Water and Sewage Department and began treating water from its
Flint River source. The change in source water and the lack of corrosion control treatment following
that change allowed lead to leach into the drinking water. The U.S. Environmental Protection Agency
(EPA) found that the drinking water provided by the City of Flint to its residents posed an imminent
and substantial endangerment to residents' health.
EPA, the City, and the utility recognized that a detailed analysis of the drinking water distribution
system was needed. Drinking water distribution systems are complex systems, typically encompassing
hundreds or thousands of miles of pipes linked together through a complex array of valves, pumps, and
storage facilities, to efficiently treat and convey water from a raw water source to the customer. The
ability of a water utility to successfully manage such a feat under the best of circumstances is difficult
but when an emergency occurs, that difficulty is multiplied. Thus, when an emergency occurs, proper
understanding of the system and how it works are key to limiting damage and furthering recovery. An
accurate water distribution system network model can provide this understanding.
EPA formed a Safe Drinking Water Task Force to support the City on October 16, 2015. On January 21,
2016 the EPA issued a Safe Drinking Water Act Emergency Administrative Order to the City and the
State of Michigan to put in place necessary actions to protect public health. The Safe Drinking Water
Task Force included providing support from EPA scientists and technical experts. The Task Force
identified the need for an updated and accurate water distribution system network hydraulic model for
the City. EPA began working with the City to update and calibrate their network model in March 2016.
To help improve and optimize operations in the system, EPA worked with representatives from the
utility, the state of Michigan, the City, CitiLogics, and ARCADIS.
Maintaining an updated and accurate water distribution system network model can be difficult for any
water system. It was recognized to be a particularly difficult problem, given future uncertainties
regarding their long-term water source and a myriad of operational issues and needs that were being
addressed by the City and the utility. As a result, EPA suggested the need for a data integration
U.S. Environmental Protection Agency	EPA/600/S-18/353
Office of Research and Development	October 2018

-------
framework that could allow the utility to easily assess model accuracy and more efficiently update
their network model, e.g., by incorporating new or modified infrastructure elements (e.g., pumps and
reservoirs) or revising operational rules. To meet this need, EPA deployed the EPANET-RTX (RTX) real-
time data integration framework to update, calibrate and confirm the accuracy of the City's network
hydraulic model and to provide a lasting data integration framework [1, 2, 3, 4].
This technical brief summarizes the deployment of the prototype RTX real-time analytics (RTX:LINK)
and data integration framework technologies to the City. It presents a high-level summary of the
network hydraulic model calibration and accuracy assessment that were conducted and describes the
benefits to the City once the technologies are fully deployed and adopted by the City. Finally, this
technical brief demonstrates that RTX and RTX:LINK technologies can enable more efficient model
calibration and accuracy assessment and make it easier for water utilities to update their network
model as their operations change or are updated.
DEPLOYMENT OF REAL-TIME ANALYTICS - RTX:LINK
The first step for EPA in assisting the City with updating, calibrating and assessing the accuracy of their
network model to reliably predict water system behavior was to access to the city's SCADA database.
The City initially provided their SCADA data for network model calibration as comma-separated values
(CSV) files, collected from the data stored in the SCADA historian database, and provided to EPA on
compact disks or email attachments. After some discussion and interactions with City staff, the project
team worked with utility staff to perform quality assurance on the data and implement some
configuration changes to the way SCADA was being collected, resulting in shorter and more consistent
data retrieval (e.g., one-minute data intervals for tank and reservoir levels and pump statuses, and 15-
minute intervals for control valve positions, pressure measures, and flow measures). These changes
were important for constructing diurnal demand curves and properly regulating the water supplies into
the network model. As these CSV files were obtained, they were uploaded into a cloud database
equipped with dashboard views for easy viewing, analyzing and management.
2
U.S. Environmental Protection Agency
Office of Research and Development

-------
\ /'
SCADA
I
I
T
Historian
RTX:LINK
(Raspberry Pi device)
Hardware Data Dkxto


WOT 1
OX-MV
: y
[ WTW1
©
Dc:




lea:
Dashboards
LIGHT EMITTER
LIGHT RECEIVER
One-way Data Diode
Fig. 1. RTX:LINK deployment to the City of Flint
Phone
Laptop, tablet
However, it was quickly recognized that the City needed a data integration framework that could allow
the utility to easily assess model accuracy and update their network model more efficiently, e.g., by
incorporating new or modified infrastructure elements (e.g., pumps and reservoirs) or revising
operational rules. To meet this need, the manual CSV file transfer process was replaced by the
installation of RTX:LINK. RTX:UNK is an RTX application that provides a simple, secure, read-only access
to key operational data streams (e.g., those stored within a SCADA historian), through web-based
dashboards for trending, analysis, and alerting [5], RTX:UNK was deployed to the City along with a
one-way data diode between the RTX:LINK SCADA historian replication device (raspberry pi) and the
city's public internet connection, to prevent malicious attacks on the SCADA network from the public
internet (Fig. 1). The deployment of RTXiLINK to the City provides managers and operators;
•	Anywhere and anytime access to their SCADA data analytics via a mobile dashboard,
•	Ability for easy increased attention to potential operational and emergency problems related to
hydraulic data, equipment, and facility operations. This allows the utility to quickly catch
potential problems observable through SCADA data (e.g., potential problems related to pump
operation, flows, pressures, tank operation and disinfectant management), and
•	Improved water quality management including the ability to monitor disinfectant loss and
disinfection byproduct formation in the distribution system. Such data are a function of
residence time, which is affected by day-to-day system operational policies and decisions,
particularly those associated with water stored in tanks. By providing real-time, accurate
information about operational effects on tank residence time and mixing, real-time analytics
can improve awareness of tank water quality, which can lead to operational changes that
improve disinfectant management.
3
U.S. Environmental Protection Agency
Office of Research and Development

-------
RTX:LINK also enabled the City to more easily deploy real-time modeling, e.g., real-time water age
mapping. RTX:LINK also provided the City with the capability to more easily adopt model-based
predictive, real-time analytics (using real-time SCADA data and an updated network model) which is
critical in allowing the early detection and awareness of operational problems in the distribution
system.
EPANET-RTX BASED NETWORK MODEL
The next step was to update the City's network model with EPANET-RTX to provide the functionalities
for data access, data transformation and data synthesis (including real-time analytics). The RTX data
integration framework that was deployed to the City uses CitiLogics' Polaris™ Work Bench data
integration environment (Fig. 2)1. Fig. 2 is a conceptual illustration of how the RTX library can be
harnessed to build a data integrated model. The RTX-based data integrated, real-time model (real-
time model) deployed to the City is depicted as the light blue shaded box within Fig. 2. In Fig. 2, the
inputs to this real-time model are shown in the boxes to the left of the blue shaded box while the
outputs, specifically "Prediction" and "Accuracy Assessment" and "Network Diagnostics and Calibration
Needs," are shown below. The setup of the real-time model began by importing the City's existing
network model, developed in 2009. The original model was obtained as an EPANET file that was
exported from the City's commercial software application2 [6].
The setup of the real-time model consists of three principle tasks as shown in Fig. 2:
•	"Data Access and Integration" is the process for integrating a network model with real data.
The initial, manual SCADA data integration process was later replaced by a live SCADA data feed
with the deployment of RTX:LINK.
•	"Asset Mapping" is the process in which all the key infrastructure assets (e.g., pumps, valves,
tank, and reservoirs) for the City were associated with their appropriate SCADA data streams.
•	The "Data Pipeline Configuration" process is the final step. Raw SCADA data streams are
filtered and processed by removing errors, bad data, and dropped signals. Data pipeline
configuration refers to the process of converting raw SCADA data streams into more accurate
and useful network model measures and boundary conditions. The design of RTX incorporates a
data time-series processing library for efficiently removing data outliers, filtering noisy data,
and bridging data gaps, as well as transforming and combining data by basic mathematical
operators.
1	Polaris Work Bench is an RTX-based commercial software program.
2	An EPANET file is a computer text file with a ".inp" extension in its file name which describes a water distribution system.
An .inp file specifies all the infrastructure of the water system, including their characteristics, operating instructions, as well
as other hydraulic and water quality configuration parameters.
U.S. Environmental Protection Agency
Hla Office of Research and Development

-------
Start
Model Export
(EPANET .inp)
SCADA
Data
Dump
(data .csv)
Mapping
Spreadsheet
(data.xlsx)
CitiLogics Polaris/
Tm
Work Bench
Network
Diagnostics/
Calibration
Needs
Prediction
Accuracy
Assessment
Data Access &
Integration
Asset Mapping
Data Pipeline
Configuration
Polaris
WorkBench
EPANET-RTX Data Integration
Real-Time Modeling
Fig 2. Commercial application using EPANET-RTX
EPANET-RTX: EASIER CALIBRATION AND ACCURACY ASSESSMENT
Once the configuration of the City's real-time model was completed, the resulting network model was
calibrated and then tested for its ability to predict operational behavior. The real-time model made
testing, calibration, and accuracy assessment much easier. For instance, an RTX-based real-time model
allows the user to simulate any historical time period supported by the underlying data, while
producing accuracy metrics for every SCADA measure and comparing model predictions with
observations. The resulting error statistics are more meaningful because they reflect the true
operational record from the SCADA historian database. Through this systematic process of replacing
operations' assumptions with data3, any significant output errors can be interpreted as
misrepresentations of the infrastructure or its condition, and not of the operational decisions. The
capability to select and evaluate a particular simulation period, e.g., a week in August 2017, with a
detailed understanding of the associated operations taking place (an understanding supported by
3 Typical network models (e.g., EPANET) assume a pump turns on, for example, at 6 AM every day and off at 3 PM; an RTX-
based real-time model can automatically incorporate the actual pump statuses that occurred during the time period of
interest.
U.S. Environmental Protection Agency
Office of Research and Development

-------
data), provides the basis for effective and efficient model calibration and a continuous assessment of
model accuracy.
CITY OF FLINT - NETWORK MODEL CALIBRATION
The calibration of the City's real-time model was a collaborative effort between EPA, the City,
CitiLogics, and Arcadis. The calibration consisted of a 16-week period from July 29, 2016 until
November 18, 2016. For the calibration period, field pressure and flow data and the City's SCADA
historical record for the water system were accessed and used to calibrate the network model using
CitiLogics' Polaris Work Bench.
Multiple site visits were made by project team representatives to ensure an accurate representation of
the pumping and control facilities on the treatment plant grounds, at the primary booster pump
station, and the two ground level reservoirs and associated pumping facilities. These data were then
used to support the modifications made to key facility infrastructure representations and operating
characteristics (e.g., pump head-discharge and valve loss curves) in the network model.
Multiple field studies were also implemented to collect additional hydraulic data. Pressure and flow
data were collected by the Michigan Department of Environmental Quality. EPA and CitiLogics
personnel deployed pressure monitors and collected pressure data. Network model node elevation
data were reviewed and updated. Fire hydrant flow tests were conducted by ARCADIS to evaluate the
significant differences that were observed between actual and expected head losses4 (ranging from 3
to 8 ft./mile) in transmission mains feeding the city's two ground level reservoirs. Due to these field
studies, the network model's C factors5 were reduced significantly for all pipe size ranges, including
several transmission mains.
Network model base demands were updated to reflect billing records as late as 2015. Water usage
billing data from approximately 50,000 individual customer accounts (representing approximately
900,000 customer water meter readings) from February 2013 to September 2016 were put into a
billing database. The data was then processed to estimate accumulated average water usage rates that
were used to update the model base demands in the EPANET model file. Along with other changes,
the calibration process resulted in a significantly improved network model.
4	Head loss is the resistance to water flow due to friction.
5	A value used in a hydraulic network model to describe the smoothness of the interior of a pipe. The higher the C factor, the
smoother the pipe, the greater the flow capacity, and the smaller the friction or energy losses from water flowing in the pipe.
Ml U.S. Environmental Protection Agency
Hi! Office of Research and Development

-------
Comparing the SCADA historical record with the calibrated model's output over the 16-week
calibration period displayed a high level of model accuracy. Table 1 below shows the accuracy results
for the City's network model over the 16-week calibration period in 2016.
Table 1. Calibrated Model Accuracy Results Over 16-Week Calibration Period in 2016*

Hydraulic Head (FT)
Tank Level (FT)
Pipe Flow (GPM)
Tank Flow (GPM)
Period
Avg
MAE
Avg R
Avg
MAE
Avg R
Avg
MAE
Avg R
Avg
MAE
Avg R
2016
3.20
0.87
0.76
0.99
273
0.94
217
0.97
* Results expressed as the average (Avg) mean absolute errors (MAE) over all measurements within a category. R
corresponds to Pearson correlation coefficients. FT, feet; GPM, gallons per minute.
CITY OF FLINT - NETWORK MODEL ACCURACY ASSESSMENT
The accuracy assessment for the updated and calibrated City's network model was performed for a
similar but slightly shorter period in 2017 (July 29th to Oct. 27th)6. The selection of this time period was
to test the network model's ability to simulate hydraulic events for a different operational period. The
period selected in 2017 included a significant operational difference from 2016. Specifically, one of the
City's two major ground reservoirs was out of service for the entire time. Thus, the 2017 accuracy
assessment period tested the ability of the network model to recreate the behavior observed in 2017
without the hydraulic interaction that existed between the two ground level storage facilities in 2016,
during the time period for the network model calibration. Table 2 below shows the accuracy results for
the City's network model over the 13-week accuracy assessment period in 2017.
6 During week 14 of the accuracy assessment period, a SCADA reservoir level signal at the operating ground level reservoir
experienced a string of invalid values (sudden and sustained decreases and increases of 5-15 feet without reason) that
continued for more than 24 hours. Due to the nature of these errors it was impossible to recover the true signal using available
filtering operations, and because of the large capacity of the operating reservoir, the errors propagated to the calculated
diurnal demand which, in turn, had a significant impact on model simulation accuracy that then continued for the duration
of the accuracy assessment period. Rather than have such errors potentially bias the interpretation of the accuracy
assessment results, it was decided to shorten the accuracy assessment period from 16 to 13 weeks.
¦V U.S. Environmental Protection Agency
HI Office of Research and Development

-------
Table 2. Accuracy Assessment Results Over 13-Week Accuracy Assessment Period in 2017

Hydraulic Head (FT)
Tank Level (FT)
Pipe Flow (GPM)
Tank Flow (GPM)
Period
Avg
MAE
Avg R
Avg
MAE
Avg R
Avg
MAE
Avg R
Avg
MAE
Avg R
2017
3.08
0.79
0.57
0.97
304
0.93
348
0.92
Avg, average; FT, feet; GPM, gallons per minute; MAE, mean absolute errors over all measurements within a category; R,
Pearson correlation coefficients.
BENEFITS TO THE CITY OF FLINT
Real-Time Data Analytics - The deployment of RTX:LINK hardware and software to the City provides
utility managers and operators with an easy, web-based access to all their water distribution system
operational data through a highly secure connection. Fig. 3 shows an example view of the RTX:LINK
dashboards illustrating system pressures and water quality. These Grafana-based dashboards are
easily user- configured to provide the analytics of most interest to the City or water utility.7
The real-time analytics dashboards provide utility managers and operators their key utility operational
data streams via a smart phone APP - fostering improved communication about operations between
operators, engineers, and managers.
Real-Time Model - The City now has an updated, calibrated, and validated (through the accuracy
assessment) network model. More importantly, the City now has an updated, modern SCADA data-
based real-time model of their water system operations that can be more easily maintained and used
in day-to-day management of their water system operations. The RTX-based hardware/software
system integrates the City's SCADA data with their hydraulic and water quality network model for real-
time prediction of system wide flows, pressures, and water quality. The real-time model includes a
continuous accuracy evaluation of the real-time predictions, using an unprecedented volume of
streaming, real-time (SCADA) operational data. These accuracy metrics can now be tracked
automatically, greatly decreasing the effort and costs required to maintain the City's network model
while, at the same time, enhancing confidence in its use.
7 Grafana - https://grafana.com/ is an open source platform for analytics and monitoring that is well-suited for time-series
data.
U.S. Environmental Protection Agency
Office of Research and Development
8

-------
SS Dashboard List <•



< Q >
O Last 6 hours C
Facilities


Summaries


Cedar Street Reservoir

System Pressures


¦Cr
CS2 / Plant
O
Wate» Quality


0-
West SKIo Reservoir
a




SS System Pressures -



& < Q, > O Last 2 days Refresh every 1 m 0
125 |M


,S2 Pressure






-




^rj 58 Water Quality -
CSR Cnionne

10 A*
<1 C < Q. > Ou»l20«y» . C
CS2
t 3 r - -


? --

7/2916.00 7/26 COCA 7/2
•» »' Ln» CNonrw — incoming Oilorm#
6M00 7/261600
WSR Chlorine
7/270000 7/27 Ot00
7/251600 7/260000
— 36*lmCNonw — Pl«m Supphr Chtonr*
7/260100 7/261600 7/2700:00 7/270*00
CS2 Turbidity
t


Z 0J

7/141ft00 7/26 0300 lOt
— WSRCNOrm*
MOO 7/361600
7/270000 7/77 0800
7/251600 7/360000
— CS2TWfiWy
7/26 0000 7/26 16 03 7/270000 7/27 0800
Fig. 3. City of Flint real-time analytics dashboards
Operations and Management - During the model calibration process, the real-time model was
instrumental in identifying excessive head losses in city's transmission and distribution mains and
ineffective reservoir pumping operations that were resulting in excessive water age in storage.
The real-time model can be used to assist with asset management and sensor placement (e.g., to
support improved operations or compliance monitoring). Finally, the real-time model can now be used
by the City to support future model upgrades and engineering studies.
The real-time model can also be used to monitor and help manage water quality. Water age is a
parameter that can be monitored for an indication of water quality. Water age is a major factor in in
the loss of water quality within drinking water distribution systems. Water quality deteriorates from
the time treated water leaves the treatment plant until it is used by the customer. As the water travels
through the distribution system, it undergoes various chemical, physical and aesthetic transformations
due to its interactions with pipe walls (including biofilms, pipe scale, and contaminants) thus impacting
water quality. A longer water age can allow disinfectants more time to react with naturally occurring
materials in the distribution system to form disinfection by-products (DBPs). Water age is generally
driven by customer demand and water system design and operation.
9
U.S. Environmental Protection Agency
Office of Research and Development

-------
Fig. 4 shows a snap-shot of a prototype, real-time water age map for the City. This snap-shot map of
real-time water age provides an illustration of a continually updated view of the last 24 hours of water
age for the City, as a function of actual operations. A continuously updated, real-time model can
provide utility managers with the information and the tool needed to set specific water quality goals
(e.g., lowering water age) and help them achieve them.
Day 22, 12:00 AM

Fig, 4, Real-time water age map for City of Flint.
Note: Water age is not directly correlated with disinfectant chlorine residuals or other water quality parameters
and does not reflect the City's efforts to perform booster chlorination at its storage tanks.
FOR MORE INFORMATION
The intended audience for RTX:LINK is water utilities and their consultants and software developers.
The intended audience for EPANET-RTX is software developers. EPANET-RTX and RTX:L!NK are open-
source software projects. If collaborators are interested, there are various ways to get involved (e.g.,
connecting to the code repository, looking over coding conventions and using the issues tracker to
make a feature request and communicate with the developers). To learn more about RTX:LIMK and
real-time modeling using EPANET-RTX or ongoing enhancements, visit the repositories of EPANET-RTX
and RTX;LINI< at https://github.com/OpenWaterAnalytics.
10
U.S. Environmental Protection Agency
Office of Research and Development

-------
REFERENCES
[1]	Uber, J., Hatchett, S., Hooper, S., Boccelli, D., Woo, H., and Janke, R. (2014). Water Utility Case
Study of Real-Time Network Hydraulic and Water Quality Modeling Using EPANET-RTX Libraries.
Cincinnati, Ohio: U.S. Environmental Protection Agency. EPA 600-R-14-350.
[2]	U.S. Environmental Protection Agency. (2014). "Enhancements to the EPANET-RTX Open Source
Libraries 2014," (Technical Brief). EPA/600/S-14/310.
[3]	U.S. Environmental Protection Agency. (2015). "Enhancements to the EPANET-RTX (Real-Time
Analytics) Software Libraries (Fiscal Year 2015)," (Technical Brief). EPA/600/S-14/441
https://cfpub.epa.gov/si/si_public_record_report.cfm?dirEntryld=309673 accessed on July 27, 2018.
[4]	U.S. Environmental Protection Agency. (2012). http://wateranalytics.org/epanet-rtx/index.html and
https://github.com/OpenWaterAnalytics/epanet-rtx, accessed on July 26, 2018.
[5]	U.S. Environment Protection Agency. (2016). "RTX:LINK-An EPANET-RTX Software Tool For Water
Utilities To Implement Real-Time Analytics," (Technical Brief). EPA/600/S-16/287
https://cfpub.epa.gov/si/si_public_record_report.cfm?dirEntryld=328356, accessed on July 26, 2018.
[6]	Rossman, L.A. (2000). EPANET 2 Users Manual. Cincinnati, OH: U.S. Environmental Protection
Agency, Office of Research and Development, National Risk Management Research Laboratory Report,
EPA/600/R-00/057.
CONTACT INFORMATION
For more information, visit the EPA Web site at https://www.epa.gov/homeland-security-research
Technical Contacts: Robert Janke (ianke.robert@epa.gov)
Regan Murray (murray.regan@epa.gov)
General Feedback/Questions: Amelia McCall (mccall.amelia@epa.gov)
DISCLAIMER
The U.S. Environmental Protection Agency funded and collaborated in the work described here. It has
been subjected to the Agency's review and has been approved for publication. Note that approval does
not signify that the contents necessarily reflect the views of the Agency. Mention of trade names,
products, or services does not convey EPA approval, endorsement, or recommendation.
11
U.S. Environmental Protection Agency
Office of Research and Development

-------
U.S. EPA's Homeland Security Research Program (HSRP) develops products based on scientific
research and technology evaluations. Our products and expertise are widely used in preventing,
preparing for, and recovering from public health and environmental emergencies that arise from
terrorist attacks or natural disasters. Our research and products address biological, radiological, or
chemical contaminants that could affect indoor areas, outdoor areas, or water infrastructure.
HSRP provides these products, technical assistance.
12
U.S. Environmental Protection Agency
Office of Research and Development

-------