EPA 910-R-16-007
December 2016
Guidance for Quality Assurance Project
Plans for Water Quality Modeling Projects
U.S. Environmental Protection Agency
EPA Region 10
Office of Environmental Review and Assessment
Seattle, Washington
-------
Contact
Ben Cope, Office of Environ mental Review and Assessment
EPA Region 10
cope.ben@epa.gov
Ackno wled gements
EPA thanks the following individuals for reviewing a draft of this document and providing input:
Dino Marshalonis, EPA
Donald Brown, EPA
Paul Pickett, Washington Department of Ecology
Cristiana Figueroa-Kaminsky, Washington Department of Ecology
Greg Pelletier, Washington Department of Ecology
Teizeen Mohamedali, Washington Department of Ecology
Bill Kammin, Washington Department of Ecology
Tom Gries, Washington Department of Ecology
Curtis DeGasperi, King County
Ryan Michie, Oregon Department of Environmental Quality
2
-------
Contents
I. Introduction 6
A. Scope of Guidance 6
B. Document Template 6
II. Problem Definition and Management Objectives 6
III. Conceptual Model: Key Processes and Variables 7
IV. Technical Approach 7
A. Overview 7
B. Model Selection 7
C. Software Development Quality Assessment 8
V. Model Development 8
A. Model Boundaries 8
B. Spatial and Temporal Resolution 8
C. Source Characteristics 9
D. Data Availability and Quality 9
E. Time Frame of Simulation 10
F. Data Gaps 10
G. Important Assumptions 10
H. Model Calibration 10
I. Model Parameters 11
VI. Model Evaluation and Acceptance 11
A. Model Uncertainty and Sensitivity 11
B. Model Acceptance 12
VII. Documentation in Model Reports 13
VIII. Peer Review 13
IX. Management Scenarios 14
X. Project Organization 14
A. Project Team/Roles 14
B. Expertise and Special Training Requirements 15
C. Reports to Management 15
D. Project Schedule 15
XI. Data Management 15
3
-------
XII. Recordkeeping and Archiving 15
XIII. QAPP Review and Approval 15
XIV. Implementation and Adaptive Management 16
XV. References 16
4
-------
FOREWORD
EPA Region 10 has created this guidance to assist in the effective and efficient planning of water
quality modeling projects through the development of a Quality Assurance Project Plan (QAPP).
This document was preceded by two general guidance documents for environmental modeling.
EPA's national quality assurance program office, the Office of Environmental Information (OEI),
issued a guidance document on model project planning in 2002: Guidance for Quality Assurance
Project Plans for Modeling (EPA, 2002). In 2009, EPA's Council on Regulatory Environmental
Modeling (CREM) issued a guidance document on all aspects of regulatory modeling: Guidance
on the Development, Evaluation, and Application of Environmental Models (EPA, 2009). The
scope of these documents included all types of environmental modeling, which span a diverse
range of media, scales, and regulatory settings. This guidance is focused specifically on QAPPs
for surface water quality models. While this guidance is generally consistent with the previous
guidance documents, it is intended to simplify QAPP development by focusing on the unique
challenges and practical constraints facing water quality modelers. The topic sequence is
intended to provide a template for both the QAPP and the model development report for a
project. The goal is to foster a streamlined planning process while encouraging strategic project
planning, problem solving, and transparency in the earliest stages of water quality modeling
projects.
5
-------
I. Introduction
A Quality Assurance Project Plan (QAPP) sets the stage for a successful water quality modeling
project. While modeling projects can vary widely in scope and complexity, this document
provides a template of topics and issues that should apply to most surface water quality modeling
projects.
The information in a QAPP is not only relevant to the initial planning of the project but remains
relevant in the final documentation of the model. In fact, much of the information in a QAPP
should be incorporated into the final documentation. In this sense, the QAPP is the first draft of
the final model development report, and the sequence of topics covered in this guidance can be
used as a template not only for the QAPP but also for the initial chapters of the final model
development report.
For complex projects, the QAPP can be used as a "living document" that evolves, through
revisions or addenda, as model development proceeds and the system is better understood.
A. Scope of Guidance
This guidance document is focused specifically on quality assurance planning for surface water
quality models used within EPA Region 10. EPA has published two national guidance
documents that apply broadly to all types of environmental models: Guidance for Quality
Assurance Project Plans for Modeling (EPA, 2002) and Guidance on the Development,
Evaluation, and Application of Environmental Models (EPA, 2009). Model developers are
encouraged to review these documents for additional detailed information not provided in this
streamlined guidance.
B. Document Template
Beginning with Section II below, this guidance steps through the topics that should be covered in
a QAPP. It is recommended that the project team use this topic sequence as the table of contents
of both the QAPP and final model report, with any additional topics or subtopics as needed for
the particular project.
II. Problem Definition and Management Objectives
The QAPP should begin with a concise overview of the water quality problem to be assessed
using the model, including background information that provides a historical, scientific, and
regulatory perspective for the project.
6
-------
The management context for the project should also be described. Water quality models are
commonly used as decision support tools for regulatory actions and restoration projects. Some
examples include Total Maximum Daily Load (TMDLs), National Pollutant Discharge
Elimination System (NPDES) permits, Superfund cleanup actions, and National Estuary Program
(NEP) assessments. It is often helpful to formulate the specific assessment questions to be
answered using the model.
III. Conceptual Model: Key Processes and Variables
With the problem and management context outlined, this section of a QAPP provides a
description of the system characteristics that are hypothesized to be influencing the water quality
endpoint(s) of concern. Conceptual drawings ("cartoons"), process flow diagrams, and other
figures will assist the reader in understanding the system to be assessed, including boundary
conditions, processes, sources, and linkages between pollution sources and water quality
endpoints of concern.
IV. Technical Approach
A. Overview
Since this QAPP is written for a modeling project, the description of the technical approach may
be as simple as stating that a predictive model will be built and describing the types of
predictions required. However, more complex projects may warrant additional explanation. For
example, there may be parallel projects to the model development, such as data collection,
analyses outside of the model, linkages between multiple models, and other efforts that are
important to model development. The full scope of the technical work for the project should be
described.
B. Model Selection
A wide variety of models are used in water quality assessment. Models can be divided into two
broad categories: empirical (or statistical) models and mechanistic (or process) models.
Empirical models are based on a statistical relationships in environmental and experimental data.
Mechanistic models are based on a structure of equations that explicitly represent physical,
chemical, and/or biological processes. Because of the widespread use of mechanistic models in
water quality programs, this guidance is focused on the development of this type of model. At
the same time, many of the project planning steps described in this document are equally relevant
to the development of empirical models.
For mechanistic models, the model framework is the software tool that will be used in the project
(e.g., QUAL2Kw, HSPF, EFDC/WASP). Model development entails construction of a site-
specific representation of the waterbody using a particular model framework. There is usually
7
-------
more than one available model framework that can simulate the processes and variables of
interest. This section of the QAPP should identify the candidate model frameworks and the
rationale for selecting the chosen framework.
Some of the factors that may influence the selection of a model framework include:
• fit between model capabilities and project needs
• proprietary vs non-proprietary software
• model pedigree (e.g., widespread use) and support community
• capability to simulate all processes of interest
• familiarity of model developers with the tool
• complexity of the problem/tool/products
• data needs relative to available data
EPA discourages the use of proprietary models for regulatory projects, because the model
framework codes cannot be shared outside the agency (EPA, 2009). Fortunately, the vast
majority of water quality problems can be assessed using model frameworks with open source
codes.
C. Software Development Quality Assessment
In some projects, the selected model framework does not simulate a particular process or has
some other limitation that requires a code modification. The QAPP should describe anticipated
code changes and how the new coding will be quality-tested and documented in the project file.
V. Model Development
The following sections provide a general guide for planning the assembly of the model.
A. Model Boundaries
The QAPP should include a map of the study area and a description of the spatial domain of the
model. This description should include all boundary conditions to be represented in the model,
such as upstream/downstream boundaries, tributaries, groundwater inflows, point source
discharges, dams, water withdrawals/returns, bed sediment influences, air deposition, and
weather.
B. Spatial and Temporal Resolution
This section describes the spatial and temporal resolution of model predictions. For gridded
models, the spatial resolution will be defined by the sizing of the computational cells within the
grid. Coarse grids can be a concern for water quality assessment, because the model will average
8
-------
water quality impacts across large cell distances and potentially underestimate local impacts.
The QAPP should describe the rationale for the choices made in setting up the grid. This may
include the interplay between practical considerations (e.g., computer run times, data limitations)
and programmatic requirements (specific impacts and areas of concern). For example, cell
layout may be influenced by the need to provide predictions at specific locations, such as near
point source discharges, at tributary confluences, or at varied depths in stratified reservoirs.
C. Source Characteristics
This section describes the current knowledge about all known pollution sources in the study area
and the plan for incorporating them into the model. Sources often range from well-defined
sources (e.g., permitted point sources, dams) to diffuse sources (e.g., agricultural lands,
groundwater) that may be estimated as part of the modeling project. The QAPP should describe
the data and/or calculation method for estimating source contributions to receiving waters.
D. Data Availability and Quality
This section describes the available data for the project, the organizations that collected the data,
and quality assurance of the data. For water quality modeling, general categories of data
commonly include:
• Environmental Characteristics Data - e.g., bathymetry, slope, stream network
linkages, waterbody dimensions
• Hydrological Data - e.g., flow, velocity, water elevation, withdrawals
• Meteorological Data - e.g., air temperature, dew point, precipitation, wind
• Water Quality Data - e.g., temperature, dissolved oxygen, nutrients, algae
• Source Data - e.g., point source data, dam operations, agricultural drain data
• Watershed Condition Data - e.g., land use, riparian shade, air deposition
Summary tables on the available data are recommended to include parameter, sample type (e.g.,
grab, continuous), frequency (e.g., hourly, monthly), monitoring station, data collection
organization, period of record, and availability of quality assurance information for the data.
Key monitoring stations should be depicted on a map, such as USGS gauge stations, water
quality sampling locations, and weather stations.
Water quality models are commonly developed with existing data collected by various agencies
and organizations. Many agencies routinely conduct monitoring programs within established
quality management systems, such as USGS, EPA, and state environmental agencies. Detailed
data quality evaluations are not generally warranted for data collected and analyzed by these
agencies in the QAPP phase. Nevertheless, model developers are experienced users and
interpreters of environmental data, and data handling and review during the model development
process provides a layer of quality review. In the course of incorporating data into the model,
9
-------
errors and/or quality concerns about the data will occasionally be unearthed. Any concerns or
uncertainties about data quality should be documented in the QAPP (if known early in the
project) and/or model documentation reports.
If a data source has a less established quality management system, the project team should
consider a thorough review of the quality assurance information for the dataset during QAPP
development to identify potential issues.
E. Time Frame of Simulation
Once the data is summarized, the QAPP should identify the time frame that will be simulated by
the model. This is typically the period of time with the greatest data coverage across all data
types (e.g., flow, water quality, weather, and point source data).
F. Data Gaps
As the planning focuses on the time frame of the simulation, the gaps in the data can be
identified and discussed. These data limitations should be summarized along with the general
approaches the model developer will take to address them in the model. In many cases, the
importance of a particular data gap is not known at the QAPP stage of the project, but
information can be gleaned during model development. For example, if a particular boundary
condition is uncertain due to a lack of data, the influence of that model input can be evaluated
using sensitivity analysis. This will help identify the most important data gaps. This, in turn,
assists in future monitoring plans.
G. Important Assumptions
Water quality models can be viewed as a combination of theory, observations, and assumptions
that represent a "best available understanding" of a waterbody. The model software incorporates
the mathematical equations from the peer-reviewed literature in water quality science, and the
calculations are driven by the available data for the study area. The model is rounded out by
numerous assumptions and troubleshooting in the setup, calibration, and evaluation process. To
the extent practicable, the QAPP should list the likely assumptions to be employed in the model.
H. Model Calibration
After identifying the available data for the project, the focus can turn to the available data from
locations within the model domain that will be used in the model calibration process. Model
calibration is an iterative investigation of model representativeness across multiple lines of
evidence, so specific sub-tasks are difficult to predict in the planning phase. Nevertheless, the
QAPP can identify the general sequence of tasks to be undertaken, such as the common approach
of calibrating hydrodynamics prior to calibrating water quality. The QAPP should also describe
10
-------
elements of the model that are likely to be important in the calibration process, such as key
monitoring locations, data gaps, and model parameters.
I. Model Parameters
The QAPP should describe the rates and constants in the model that are likely to be adjusted to
improve model fit. The default or starting values for the rates and constants should also be
noted. Starting values may be based on default values in the model software and user manuals,
values used in other models, scientific literature, or guidance such as EPA's "Rates, Constants,
and Kinetics Formulations in Surface Water Quality Modeling" (EPA, 1985). Once the
calibration process is completed, the selected values for all of the rates and constants in the
model should be included in a table in the model report (see Documentation in Model Reports
below).
VI. Model Evaluation and Acceptance
At the end of the calibration process, the model developer and project team has gained
knowledge about the waterbody as a biogeochemical system, as well as insights into the key
uncertainties in the model and supporting data. The QAPP can help a project team anticipate the
tasks to be undertaken in this model evaluation phase.
A. Model Uncertainty and Sensitivity
Development of water quality models is an inherently uncertain enterprise. Common sources of
uncertainty include the following:
System complexity
Simplifying assumptions of the model
¦ Uncertainty in process kinetics and rate constants of the model
¦ Data Limitations
¦ Gaps
¦ Limitations in boundary condition and calibration data
Measurement error
Specific sources of uncertainty identified during the project planning phase should be described
in the QAPP, along with plans for assessing uncertainty during model development. Uncertainty
and sensitivity analyses are complementary activities to model calibration, performed as part of
an investigation into different sources of error in the predictions. Important findings in this
iterative investigation should be documented in the model development report. Key data
limitations identified through this process should be summarized for consideration in the
planning of any future monitoring.
11
-------
B. Model Acceptance
A project reaches an important milestone when a model is accepted for use in the development of
TMDLs, NPDES permits, or other water program decision. The most common model
development goals are (1) to minimize the difference between simulated and observed water
quality and (2) to capture the spatial and temporal patterns in the observed water quality
conditions. Progress toward achieving these goals is commonly captured in error statistics and
graphical plots. However, model quality goes beyond these core evaluations. Several parallel
tasks to achieve overall model quality are pursued alongside efforts to reduce model error,
including:
1) Incorporation of all available observations of the system (e.g., geometry, flow, boundary
inputs/withdrawals, meteorology) for the time period simulated.
2) Reasonable estimation methods and assumptions to fill gaps in the observations.
3) Calibration of model parameters and unmeasured boundary conditions within reasonable
bounds to improve agreement between simulated and observed water quality.
Identification of key parameters/processes through model calibration and sensitivity
analysis.
5) Clear communication of key assumptions during model development with the project team.
6) Clear written documentation of all important elements in the model, including model setup,
boundary conditions, assumptions, and known areas of uncertainty.
Peer review.
Progress on all of these fronts will factor into the decision to accept a model for use in a
decision-making process.
Taken as a whole, identification of these model acceptance elements is consistent with the Data
Quality Objective (DQO) process for environmental sampling, analysis, and use. However,
unlike DQOs for monitoring and analysis, most water quality modeling QAPPs do not include
quantitative acceptance criteria for model error. This is based on the following considerations:
1) Overall model quality cannot be fully captured in numeric error statistics.
2) Model error can vary widely depending on the system characteristics and simulated
parameters, and the irreducible error cannot be predicted at the outset of the project.
3) It may not be possible to reduce error below numeric acceptance criteria without additional
data collection, and this can significantly impact the project schedule, budget, and
management goals. A decision to delay model acceptance for additional data collection is a
12
-------
major management decision that should not be pre-judged by criteria in the project
planning document.
4) Model acceptance is a policy decision of regulatory agency management and should
involve consideration of numerous factors and goals in model quality (described above).
The QAPP should ensure good project planning through informed decision making without
setting pass-fail goals or constraining management review and decisions.
VII. Documentation in Model Reports
As noted in the introduction, the information in a QAPP is not only relevant to the initial
planning of the project but remains relevant in the final documentation of the model. In fact,
much of the information in a QAPP should be incorporated into the final documentation. In this
sense, the QAPP is the first draft of the final model development report. The sequence of topics
covered in this guidance can be used as a table of contents template not only for the QAPP but
also for the opening chapters of the final model development report.
Model documentation may be accomplished in one document at the end of the project or in
stages during different phases of the project. For complex projects, the project team may elect to
write separate reports on hydrodynamic calibration, water quality calibration, and model
application scenarios. The amount of detail provided in the model documentation should be
sufficient as to allow for an independent recreation of the results by someone who is experienced
and technically proficient with the model(s).
The use of models in the regulatory arena requires more detailed documentation than is found in
scientific journals. In addition, a model that underlies a decision (such as a TMDL or NPDES
permit) is subjected to the same public comment process as the regulatory action, even in cases
when the model has been previously peer-reviewed. The expected sequence of the
documentation process should be considered in the project scoping and schedule.
VIII. Peer Review
Peer review can provide major benefits to a modeling project, and opportunities for peer review
should be considered in project planning. Because of limited budgets in water quality agencies,
many projects will conduct internal peer review, in which the project team provides the model
and documentation to a modeler within the same organization but outside the project team. In
more complex and influential projects, the project team may invite outside modelers to peer
review the work and/or participate in an advisory group that reviews the model. In the highest
profile projects, the agency may devote time and funds to contract with external experts to
review the model.
13
-------
When requesting peer review from external experts, it is important to define the boundaries of
the review. This is typically accomplished in "charge questions" to the reviewing panel that
focus their review and analysis. These questions should be focused on the scientific soundness
of the model and opportunities for improving model accuracy rather than policy matters such as
the reviewers' opinion on the suitability of the model for the proposed action.
More detailed guidance on peer review can be found in EPA's Peer Review Handbook (EPA,
2006).
IX. Management Scenarios
A modeling project often begins with a general understanding that a predictive tool is needed to
diagnose a water quality problem and chart a path toward restoration. As the QAPP nears
completion and the modeling approach is better defined, the capabilities and limitations of the
model should be considered in anticipation of the model application phase of the project. The
QAPP should outline the key assessment questions for the project and how model scenarios can
be crafted to answer them.
Commonly examined scenarios include:
- Existing condition (results from the calibrated model)
- Critical condition (e.g., changes to existing condition when flows or weather are more severe)
- Future condition (e.g., with higher source loadings due to increased population and development)
- Natural condition (removal of all known anthropogenic influences)
- Source assessment (isolating impacts of one source or group of sources)
- Future restored condition (e.g., the condition attained by reducing sources)
By anticipating the scenario phase during QAPP development, the project team can identify
model capabilities that will be needed to investigate specific questions. This may influence the
manner by which a particular source or system characteristic will be represented in the model
setup.
X. Project Organization
A. Project Team/Roles
This section should identify all project team members including modeling leads, QA manager,
and policy/program staff that may or may not have expertise in modeling. The roles and
responsibilities of each team member in the context of the project should also be described. In
describing roles, the plan should identify communication roles to ensure that the full project team
tracks the status and progress of the modeling project.
14
-------
B. Expertise and Special Training Requirements
This section identifies the expertise and training required for the project. If the current expertise
or training of the project team is lacking in a particular area, the necessary training to be
undertaken should be identified and addressed in the project plan.
C. Reports to Management
This section identifies the role of senior management and key junctures of the project when the
team will communicate progress and/or issues to agency management. In addition to
communicating to management at key junctures, the project team should inform management of
major deviations from the QAPP in a timely manner, such as delays in the model development
schedule, changes in technical approach, and unforeseen data or model framework limitations.
D. Project Schedule
With the selection of the technical approach in the QAPP, the project team can develop a
schedule for major model development and application tasks. The process of developing a
schedule may identify parallel or dovetailing sub-tasks that bring efficiencies and other benefits
to the project.
XI. Data Management
Data management for a modeling project can range from maintenance of basic spreadsheets to
the development of large relational databases. The QAPP should describe the general approach
for data gathering, organizing, and sharing. It is particularly important to organize the model and
underlying data prior to the public comment phase of the project so that it can be readily shared
upon request.
XII. Recordkeeping and Archiving
A model and supporting information may be useful to a water quality agency for years after
project completion. The QAPP can anticipate future uses of the information and set up a specific
archiving plan to retain access to the model project file over the long term.
XIII. QAPP Review and Approval
This section contains the review path for the QAPP, culminating in signature from the QA
manager for the agency. All project team members that would benefit from the information
should be part of the informal reviews of a draft QAPP. A smaller group of project leads, such
as the lead model developer and overall project manager, should provide formal signoff on the
document.
15
-------
A project team may elect to share a draft QAPP with an advisory group and/or stakeholders prior
to approval. Early engagement with outside parties can improve buy-in from stakeholders, bring
in local knowledge, and/or lead to improvements in the plan.
XIV. Implementation and Adaptive Management
The complexity of model development virtually guarantees that some element of the project will
not proceed as planned. The hope is that a well-devised plan will identify and address the most
significant challenges. If an obstacle does arise that requires a change in approach, the project
team can return to the QAPP as a topic template for evaluating the ripple effects on other aspects
of the project. As noted in the introduction, a QAPP can be a "living document" that evolves,
through revisions or addenda, as model development proceeds and new aspects of the system are
understood. Significant changes in technical approach should be reviewed and approved by the
project team and QA manager.
XV. References
EPA. 1985. Rates, Constants, and Kinetics Formulations in Surface Water Quality Modeling.
Second Edition. Environmental Research Laboratory. Publication EPA/300/3-85/040.
June 1985.
EPA. 2002. Guidance for Quality Assurance Project Plans for Modeling. Office of
Environmental Information. Publication EPA/240/R-02/007. December 2002.
EPA. 2006. Peer Review Handbook. Third edition. Science Policy Council. Publication EPA-
100-B-06-002.
EPA. 2009. Guidance on the Development, Evaluation, and Application of Environmental
Models. Council on Regulatory Environmental Modeling (CREM). Publication
EPA/100/K-09/003. March 2009.
16
------- |