July 27, 1999
EPA-SAB-EPEC-99-014
Honorable Carol M. Browner
Administrator
U.S. Environmental Protection Agency
401 M Street, SW
Washington, DC 20460
Subject: Second Review of the Index of Watershed Indicators
Dear Ms. Browner:
On October 14-15, 1998, the Ecological Processes and Effects Committee of the
Science Advisory Board (SAB) conducted a second review of the Office of Water's
Index of Watershed Indicators (IWI), a web site that uses easily understandable maps
to depict the condition of all watersheds in the United States. By using a common set
of indicators for every watershed, the web site and its companion printed materials
allow members of the public not only to gain information about particular watersheds,
but also to compare their local watersheds to others across the nation.
The primary strengths of the IWI lie in its use of watershed units to organize and
present data from a variety of sources, the high quality presentation on an Agency web
site of both the composite information and the underlying data, and its ability to
highlight data gaps through map presentations and thereby to stimulate improved data
collection and reporting. The Committee continues to feel that the IWI concept is a
good one and worth building upon. By providing public access to important, but
disparate, types of environmental data in a systematic and organized manner, the IWI
should facilitate local discussion and decision-making regarding watershed issues.
The IWI used indicators that were readily available, enabling it to provide a great
deal of useful information in a short amount of time. However, those readily available
indicators are not necessarily optimal for achieving IWI's goal of "measuring progress
toward EPA's goal that all watersheds will be healthy and productive places." Most of
the existing IWI indicators measure chemical stressors, but not their biological or
ecological effects. Just as one would not assume that her child had lead poisoning
simply because lead paint was discovered on the walls of the house, nor assume that
her child was completely healthy just because she did not exhibit lead poisoning,
conclusions about the health (or illness) of organisms and ecosystems should not be
drawn solely on the basis of ambient chemical measures in water and sediments.
Indicators are needed that more directly measure ecological health.
-------
In most instances (and in IWI's defense), there are no nationwide measurements
of ecological health that can be easily taken "off the shelf" and added as IWI indicators.
Although the Committee has made specific suggestions in this report for the
development of indicators from existing data, in many cases the basic monitoring
needed to assess ecological health doesn't exist, or exists only in some states. Thus,
although the conclusions of the report are directed at our specific charge questions,
this review raises a larger issue of ecological monitoring that the Agency may want to
address across media and target organisms: "what data are needed to assess
watershed health?"
Specific to IWI, the Committee recommends that the Agency develop a strategic
plan to articulate IWI's goals and objectives, identify target audiences and clientele,
define terms, and identify data gaps. The Agency should also develop a conceptual
model to guide the selection of additional data layers and refinements to the integrating
algorithm.
In lieu of direct measures of ecological health, the IWI has adopted surrogate
indicators from readily available data, but has not yet evaluated indicator performance
to demonstrate that changes in the indicator correspond to meaningful changes in
environmental quality. This evaluation is crucial, and should be undertaken for the IWI
as it currently exists as well as for the expanded IWI planned for the future. The
Agency should also undertake research to refine the composite index that combines
the individual indicators, which currently falls short of the goal of characterizing
watershed condition and vulnerability. The Agency should determine the sensitivity of
the composite index to variation of the composite indicators, conduct analyses to
assign differential weights to the individual indicators based on their relative importance
as predictors of watershed integrity, and ensure that the composite index does not
reward (or punish) organizations just because they collected and reported a lot of data.
We commend the Agency for taking a very positive step in making watershed
information available in such a user-friendly format on the IWI web site (accessed
through Surf Your Watershed). The Committee feels, however that the time has come
to put the IWI on a more sound scientific footing. In some cases, this will require
longer term effort to develop additional data layers in cooperation with the Office of
Research and Development, other federal and state data gathering organizations, and
public and private researchers.Because the IWI has the potential to serve as a nexus,
both within and outside of the Agency, for a wide variety of data on watershed condition
and vulnerability, the Committee encourages the Office of Water and other Agency
program
-------
offices to push ahead with proposed improvements. We hope these comments are
helpful to the Agency in working toward that end, and we look forward to your
response.
Sincerely,
Dr. Joan M. Daisey, Chair
Science Advisory Board
Dr.
Ecological Processes and
Effects Committee
Dr. Carol Johnston, Acting Chair
for the IWI Review
Ecological Processes and
Effects Committee
-------
NOTICE
This report has been written as part of the activities of the Science Advisory
Board, a public advisory group providing extramural scientific information and advice to
the Administrator and other officials of the Environmental Protection Agency. The
Board is structured to provide balanced, expert assessment of scientific matters related
to problems facing the Agency. This report has not been reviewed for approval by the
Agency and, hence, the contents of this report do not necessarily represent the views
and policies of the Environmental Protection Agency, nor of other agencies in the
Executive Branch of the Federal government, nor does mention of trade names or
commercial products constitute a recommendation for use.
Distribution and Availability: This Science Advisory Board report is provided to the
EPA Administrator, senior Agency management, appropriate program staff, interested
members of the public, and is posted on the SAB website (www.epa.gov/sab).
Information on its availability is also provided in the SAB's monthly newsletter
(Happenings at the Science Advisory Board). Additional copies and further information
are available from the SAB Staff.
-------
U.S. ENVIRONMENTAL PROTECTION AGENCY
SCIENCE ADVISORY BOARD
ECOLOGICAL PROCESSES AND EFFECTS COMMITTEE
Review of the Index of Watershed Indicators
October 13-14, 1998
ACTING CHAIR
Dr. Carol A. Johnston, Natural Resources Research Institute, University of Minnesota,
Duluth, MN
MEMBERS
Dr. Miguel F. Acevedo, University of North Texas, Denton, TX
Dr. William J. Adams, Kennecott Utah Copper Corp., Magna, UT
Dr. Lisa Alvarez-Cohen, University of California-Berkeley, Berkeley, CA
Dr. Kenneth W. Cummins, Tarpon Bay Environmental Laboratory, South Florida Water
Management District, Sanibel, FL
Dr. Leslie A. Real, Department of Biology, Emory University, Atlanta, GA
Dr. Frieda B. Taub, School of Fisheries, University of Washington, Seattle, WA
Dr. Terry F. Young, Environmental Defense Fund, Oakland, CA (EPEC Chair)
CONSULTANTS
Dr. William H. Smith, School of Forestry and Environmental Studies, Yale University,
New Haven, CT
FEDERAL EXPERTS
Dr. Thomas R. Loveland, EROS Data Center, U.S. Geological Survey, Sioux Falls, SD
Dr. Lou Steyaert, EROS Data Center, U.S. Geological Survey, Sioux Falls, SD (on
location at NASA Goddard Space Flight Ctr, Greenbelt, MD)
SCIENCE ADVISORY BOARD STAFF
Ms. Stephanie Sanzone, Designated Federal Officer, EPA Science Advisory Board
(1400), 401 M Street, S.W., Washington, DC 20460
Ms. Mary L. Winston, Management Assistant, EPA Science Advisory Board (1400), 401
M Street, S.W., Washington, DC 20460
-------
TABLE OF CONTENTS
1. EXECUTIVE SUMMARY 1
2. INTRODUCTION 3
3. GENERAL COMMENTS 4
4. STRATEGIC PLAN FOR THE IWI 5
4.1 Essential Components 5
4.1.1 Goals and Objectives 5
4.1.2 Clarification of Index Classification: "Condition" and "Vulnerability"
6
4.2 Data Gaps 8
5. WATERSHED CHARACTERIZATION 10
5.1 Developing a Conceptual Model 10
5.2 Evaluation of Index Performance 11
5.3 Sources of Uncertainty 11
5.4 Redundancies Among Indicators 13
5.5 Recommended Additional Indicators 13
5.5.1 Indicators of Effects 14
5.5.2 Terrestrial Indicators 15
5.5.3 Riparian Habitat Indicators 17
5.5.4 Lake Indicators 19
5.6 Research Needs 19
6. INTEGRATION ALGORITHM 21
7. PRESENTATION AND DOCUMENTATION ISSUES 24
7.1 Map Data Intervals 24
7.2 IWI Presentation 24
7.3 IWI Documentation 25
8. CONCLUSIONS AND RECOMMENDATIONS 26
9. REFERENCES CITED R-1
APPENDIXA. CHARGE TO THE COMMITTEE A-1
-------
1. EXECUTIVE SUMMARY
On October 13-15, 1998, the Ecological Processes and Effects Committee of the
Science Advisory Board met to review and comment on the Index of Watershed
Indicators (IWI) developed by the Office of Water. The stated purpose of the IWI is to
provide available data on aquatic resources in a Geographic Information System (CIS)
format to interested parties, including state and tribal governments and members of the
public, for assessing the condition and vulnerability of watersheds. Phase I of the IWI,
released in 1997, consisted of information on 15 indicators (or data layers) presented
individually and in aggregate. In a previous review (EPA-SAB-EPEC-ADV-97-003), the
Committee supported in concept Agency plans to include 6 additional indicators (i.e.,
biological integrity, habitat, groundwater, coastal condition indicator, air deposition, and
downstream effects) and further recommended that land use change and other
indicators of terrestrial condition be considered. The Committee also recommended
that the algorithm used to calculate composite scores for watershed condition and
vulnerability be examined prior to the Agency's release of a revised version of the IWI.
The primary focus of this second EPEC review, as reflected in the Charge to the
Committee, is to follow up on the previous Committee recommendations by providing
comments on IWI strategic directions, further evaluation of individual indicators, and
the integrating algorithm. The Committee applauds early Agency efforts on the IWI,
but recommends strengthening the scientific basis of IWI.
The IWI currently highlights watersheds with low water quality, as defined by the
state water quality standards. This approach does not capture other reliable indicators
of poor water quality, such as biological indices, because only a handful of states
include biological criteria in their water quality standards. More importantly, the IWI
does not characterize watershed condition (i.e., "health") and vulnerability as claimed.
The Committee supports the Office of Water's efforts to incorporate new indicators and
adjust the integrating algorithm to achieve the stated goals of the IWI. To this end, the
Committee recommends the following:
a) The Agency should develop a strategic plan to articulate IWI's goals and
objectives, identify its target audiences and clientele, define terms, and
identify data gaps in order to assure that the IWI can be updated
efficiently with additional data layers and provide an improved
assessment of condition. A Quality Assurance/Quality Control (QA/QC)
plan should be included in the strategic plan.
b) The Agency should develop a conceptual model for the IWI that can be
used to guide the selection of additional data layers and refinements to
the integrating algorithm.
-------
c) The Agency should add more indicators of biological and ecosystem
effects to the IWI.
d) The Committee reiterates its previous recommendation to include
terrestrial indicators, but recommends waiting until the Multi-Resolution
Land Characteristics (MRLC) data set becomes available to develop
indices related to land use. The Agency should develop better riparian
habitat indicators to replace its proposed riparian indicators (Indicator 16 -
Forest Riparian Habitat and Indicator 18 - Agricultural/Urban Habitat).
e) The Agency should evaluate each existing and proposed indicator to
demonstrate that changes in the indicator correspond to meaningful
changes in environmental quality. The Agency should better document
data sufficiency and other sources of indicator uncertainty and describe
sensitivity of the composite index to variation of the component indicators.
f) The Agency should revisit the current integrated index, which falls short
of the goal of characterizing watershed condition and vulnerability. As
part of this exercise, the Agency should undertake the appropriate
analyses to assign differential weights to the individual indicators based
on their relative importance as predictors of watershed integrity.
g) The integration algorithm must be flexible with respect to the amount of
data used to compile composite indicators, so as not to reward (or
punish) organizations just because they collected and reported a lot of
data.
h) Watershed quality and vulnerability are individual measures that should
be reported separately.
-------
2. INTRODUCTION
On October 13-15, 1998, the Ecological Processes and Effects Committee of the
Science Advisory Board met to review and comment on the Index of Watershed
Indicators (IWI) developed by the Office of Water. The stated purpose of the IWI is to
provide available data on aquatic resources in a Geographic Information System (CIS)
format to interested parties, including state and tribal governments and members of the
public, for assessing the condition and vulnerability of watersheds. Phase I of the IWI,
released in 1997, consisted of information on 15 indicators (or data layers) presented
individually and in aggregate. In a previous review (EPA-SAB-EPEC-ADV-97-003), the
Committee supported in concept Agency plans to include 6 additional indicators (i.e.,
biological integrity, habitat, groundwater, coastal condition indicator, air deposition, and
downstream effects) and further recommended that land use change and other
indicators of terrestrial condition be considered. The Committee also recommended
that the algorithm used to calculate composite scores for watershed condition and
vulnerability be examined prior to the Agency's release of a revised version of the IWI.
The primary focus of this second EPEC review, as reflected in the Charge to the
Committee, is to follow up on the previous Committee recommendations by providing
further evaluation of individual indicators and the integrating algorithm. The Committee
also provides recommendations for future directions for the IWI. The Charge to the
Committee (attached) from the Office of Water contains 13 questions in three general
areas: a) the strategic plan for the IWI; b) the algorithm used to calculate watershed
scores; and c) the proposed indicators of terrestrial condition. The Committee's
response to these questions is contained in the sections that follow.
-------
3. GENERAL COMMENTS
The primary strengths of the IWI lie in its use of watershed units to organize and
present data from a variety of sources, the high quality presentation on an Agency web
site of both the composite information and the underlying data, and its ability to
highlight data gaps through map presentations and thereby to stimulate improved data
collection and reporting. The Committee continues to feel that the IWI concept is a
good one and worth building upon. By providing public access to important, but
disparate, types of environmental data in a systematic and organized manner, the IWI
should facilitate local discussion and decision-making regarding watershed issues.
The Committee commends the Agency for its continuing efforts to develop and
refine the IWI. The Agency has responded to several concerns raised previously by
EPEC (EPA-SAB-EPEC-ADV-97-003) by including new terrestrial indicators, improving
data documentation, and increasing coordination with other offices and agencies. At
present, however, efforts to improve the IWI are hampered by the absence of a
strategic plan and future vision for the IWI, and a clear conceptual basis to guide the
selection and weighting of indicators for the composite index. Our responses to many
of the charge questions relate back to these two critical issues.
-------
4. STRATEGIC PLAN FOR THE IWI
4.1 Essential Components
Charge Question 1. Does the strategic plan for IWI include the critical
development activities, and are the time frames envisioned
appropriate/adequate ?
Although the IWI initially was launched with a relatively narrow focus (i.e.,
presentation of water quality-related measures organized by watersheds), its popularity
with the public and its potential future value as a nexus of watershed management
information strongly suggest that this focus should be expanded. The potential
evolution of the IWI from primarily a vehicle for presenting information reported under
Clean Water Act (CWA) Section 305(b) to a source of more comprehensive information
on watershed condition will be more effective if guided by a strategic plan. The
strategic plan should articulate the IWI's goals and objectives, identify its target
audiences and clientele, and maintain continuity during the long-term evolution of the
program. Although a draft of the strategic plan was not available for Committee review,
we understand that such a plan is currently under development. The Committee urges
the Agency to consider the issues outlined below as it drafts the strategic plan for the
IWI.
4.1.1 Goals and Objectives
One essential component of the strategic plan should be an articulation of the
goals and objectives of the program. During the course of the Agency's presentation to
the Committee, two distinct goals were presented regarding the type of information that
the IWI seeks to present:
a) Identify the watersheds where water quality goals, as represented by
numerical water quality standards adopted pursuant to the Clean Water
Act, are not being met; and
b) Characterize the overall health of each watershed and the country's
watersheds as a group.
These two goals are mirrored in the introduction to the printed version of the IWI
provided to the Committee (U.S. EPA, 1997a). The Committee agrees that both goals
are important, but is particularly supportive of the latter goal because it will be far more
valuable in the long term to inform watershed management and educate the public.
As currently implemented, IWI accomplishes the goal of identifying non-attainment of
305(b) water quality goals, but does not accomplish the goal of characterizing
watershed health. The IWI strategic plan should provide the vision for making a
-------
smooth transition between the two goals. This transition would involve identifying
watershed characteristics related to ecological health for which data layers would
ultimately be developed, such as water quality, habitat quality and landscape
characteristics, and hydrology. It would also require developing an improved algorithm
that mathematically groups the measures so that each group carries an appropriate
weight (see Section 6).
The strategic plan also should address the audience and clientele for IWI both
inside and outside the federal government. The popularity of the IWI web site is a
measure of its success in reaching a variety of clients and users and may lead IWI
developers to attempt to be all things to all people. However, some of those clients are
more important to the success of IWI than others. The strategic plan should identify
and prioritize clients based on IWI objectives so that the needs and desires of high
priority clients can be addressed before those of other users. Clients within EPA (e.g.,
Assistant Administrators or staff from other EPA offices), as well as external clients,
should be included in the identification/prioritization process. Such a prioritization will
help the Office of Water to sort out the myriad requests and suggestions that it may
receive from users, so as to focus its energy and resources on those changes that will
further IWI and broader Agency objectives.
Identification of the audience will also guide the selection and presentation of
information in the IWI. For example, if one of the main audiences is local watershed
management groups, then presenting the broader picture of overall watershed health
will be essential. In addition, the ability to incorporate locally developed information
(that meets quality control standards, but is not available from a national database)
also will be important. If one of the main audiences is EPA internal management or
Congress, for whom a progress report on national water quality improvement is
important, then the IWI should be capable of showing changes for water quality
indicators , both individually and in aggregate, over time. The ability to showcase
gradual improvements or declines is important to the general public, as well, and is not
currently incorporated into the IWI; this aspect of the IWI should be discussed in the
strategic plan.
4.1.2 Clarification of Index Classification: "Condition" and "Vulnerability"
IWI indicators are currently grouped into "condition" and "vulnerability"
categories. However, the description of these terms in the IWI printed materials (U.S.
EPA, 1997a) is confusing. "Condition" is defined as "existing water quality," but none
of the original 15 IWI indicators shows existing water quality. For example, there are no
maps showing average chemical concentrations in stream water or sediments, only
exceedences above national criteria. Rather, most of the IWI indices (1, 3a, 3b, 5, 9,
10) measure compliance (e.g., violations of NPDES permits, non-attainment of 305(b)
designated uses), which is not the same as condition or vulnerability. A watershed
may be perfectly compliant with established thresholds, yet still have water quality
-------
problems (i.e., poor condition) or contain ecosystems susceptible to pollution (i.e.,
high vulnerability). Compliance indices are a measure of regulatory success and
reporting, and reflect the Agency's role as water quality regulator, rather than indicate
ecological condition or vulnerability perse.
"Vulnerability," as defined in the IWI printed materials (U.S. EPA 1997a), is
designed to show "where discharges and other stressors impact the watershed and
could, depending on the natural and manmade factors present in the watershed,
cause future problems to occur." This definition is not clear. What constitutes
vulnerability? Is an ecosystem more vulnerable when a given stressor increases
(increased exposure) or when a sensitive assemblage of organisms is present, or both?
How does vulnerability equate to risk (which relates exposure to effects)? In general,
we believe vulnerable systems are those that require significantly less unit change in
the stressor to cause an effect. Based on the Committee's interpretation, only indices 8
and 15 indicate ecosystem vulnerability.
Several of the IWI indicators are measures of exposure to pollution or other
stressors, more commonly called pressure indicators (U.S. EPA 1998a). Indices 3c, 4,
6, and 14 are indices of current measured exposure; indices 11, 12a, 12b, and 12c are
indices of potential (modeled) exposure; and indices 7a, 7b, and 13 are indices of past
exposure trends (measured).
Three of the new IWI indices that appear in Enviromapper (16 - Forest Riparian
Habitat, 18 - Agricultural/Urban Riparian Habitat, and 20 - Soil Permeability Index)
provide background information about the state of the resource, but are not classified
as indicators of condition nor vulnerability. These indices provide useful background
information. Also, there are appropriate links via the MapLibrary to the Center for
Environmental Information and Statistics (CEIS) Water Atlas
(http://www.epa.gov/ceisweb1/ceishome/atlas/nationalatlas/wateratlas.html) and other
web sites (e.g., NRCS, USGS) with good background information. The Committee
applauds these links to related sites.
The strategic plan and IWI documentation in general should better define the
terms "condition" and "vulnerability," and clarify the relationship of pressure and
compliance indices to condition and/or vulnerability.
-------
4.2 Data Gaps
Charge Question 2. Does the plan include a mechanism for IWI users and
developers to communicate data needs and gaps to those responsible for data
collection?
An important secondary purpose of the IWI, acknowledged repeatedly during
the presentation to the Committee, is to highlight data gaps. The IWI partially
accomplishes this goal by including "insufficient data" designations on IWI maps, and
according to Agency staff, early IWI maps have already stimulated several states to
come forth with new, improved data. However, some types of data are missing from
the IWI; there are no maps showing biological effects of chemical contaminants; and,
more importantly, there are no maps showing information about other essential aspects
of watershed condition, such as native biota, habitat quality, and hydrology. The
strategic planning process should identify the highest priorities among those missing
pieces.
The strategic plan also should describe mechanisms for ongoing feedback to
EPA administrators, state producers of 305(b) reports, and other data gatherers within
and outside of EPA about the data needs. In developing IWI, the staff already has
gained substantial collective knowledge about specific data sets that would be
desirable, but do not currently exist or would be prohibitively expensive to develop into
watershed indicators. That knowledge is valuable, and should be channeled back to
those who can support appropriate data gathering efforts.
Data providers other than states are an important data source that often has
been disregarded by developers of the IWI and should be cultivated. The strategic
plan should address ways that the Agency can reach out to potential data suppliers
who are not yet aware of the IWI.
When non-EPAdata sources are used, it is an important professional courtesy
to inform database owners (particularly listed contact persons) about how their data are
being used. These contacts may provide reciprocal benefits: the producers of the data
may be able to help IWI to interpret and use the data and, conversely, the IWI may be
able to help the data producers to improve the utility of their data in the future. The
strategic plan should identify mechanisms to inform database owners (particularly
listed contact persons) about how their data are being used.
A related issue is quality assurance procedures for existing data. At present, the
IWI seems dependent on users or state agencies to communicate problems or
mistakes found in the data by, for example, phoning to report problems. However,
many dissatisfied users may simply leave the web site. The IWI Strategic Plan should
include a quality assurance/quality control (QA/QC) plan that addresses the sources of
uncertainty described in Section 5.3 of this review, as well as such issues as locational
8
-------
accuracy of data, potential for errors in data transfer from original sources (e.g., data
entry from hardcopy 305(b) reports, downloading digital data), and procedures for
resolving data conflicts where watersheds cross state boundaries.
-------
5. WATERSHED CHARACTERIZATION
Charge Question 3. Are the IWI indicators (current and proposed), taken as a
whole, adequate to characterize watershed condition and vulnerability? Are there
redundancies among indicators?
The Committee concludes that the current set of IWI indicators, even when
augmented by some of the proposed indicators, is not adequate to characterize
watershed condition and vulnerability. However, the following sections provide
suggestions on how the Agency can broaden the utility of the IWI and move in the
direction promised in the IWI documentation of providing information on watershed
condition and vulnerability.
5.1 Developing a Conceptual Model
Selecting the most appropriate indicators of watershed condition (and
vulnerability) requires an assessment of the types of ecological characteristics that
should be represented, an understanding of the relationship between an indicator and
the ecological characteristic, and knowledge of the manner in which the indicator will
respond to a change in an environmental stressor(s) (see, e.g., U.S. EPA, 1998a). This
understanding can be used to construct a conceptual model that depicts the
interrelationships among the ecological characteristics and relates changes in stressors
to changes in the indicators to changes in environmental condition/vulnerability. Such
a conceptual model would provide a unifying framework for selecting and interpreting
the many types of environmental information (e.g., pressure, state, and compliance
indicators) that are available about watersheds, while highlighting areas where
indicators or data should be developed.
The Committee recommends that the Agency develop a conceptual model for
the IWI that can be used to guide the selection of additional data layers and
refinements to the integrating algorithm. This need not be an overly time-consuming
exercise, but at a minimum should identify a list of ecological characteristics to be
included, and their indicators. A statement of the physical basis for each indicator, i.e.,
what it measures, and its relationship to watershed condition/vulnerability also should
be included. General ecosystem models (see, e.g., Odum, 1994; Mclntire and Colby,
1979) or conceptual models developed for watershed risk assessments (e.g., U.S.
EPA, 1996a-d) may provide a good starting point. A conceptual model that relates
changes in indicators to changes in environmental conditions/vulnerability also would
be an important public education tool. Once the conceptual models are in place,
further refinements could be attempted by the use of mathematical ecosystem,
watershed, and landscape models, specifically those capable of being linked with
spatial frameworks (e.g., see the variety of approaches reported in Goodchild et al.,
1993; NCGIA, 1996).
10
-------
5.2 Evaluation of Index Performance
Once the hypothesized relationships between multiple indicators and watershed
condition are described, it is necessary to evaluate the resulting index by
demonstrating that changes in the index correspond to meaningful changes in
watershed quality. We strongly recommend, therefore, that the Agency provide
information on how well the indicators used in the IWI monitor the health of the
watersheds. This evaluation of the IWI is particularly important because the IWI results
may be used by the states as the basis of corrective actions. Ideally, evaluation should
be undertaken for the IWI as it currently exists as well as for the expanded IWI planned
for the future.
Index evaluation could be done in any of the following three ways. First, IWI
indicators could be evaluated for watersheds for which there are biological indicators of
watershed health. Comparison of the two sets of indices (IWI vs. biological indicators)
would be very helpful in assessing the performance of the current approach, as well as
addressing the broader issue of how to assess watershed health. Watershed data
collected as part of the Office of Research and Development's Mid-Atlantic Integrated
Assessment (MAIA) project might be useful in this regard. Second, a semi-quantitative
sensitivity analysis could be performed to determine the ability of the current indicators
and the integrating algorithm to detect small to moderate changes in the environment.
In other words, can the current index be used to detect improvement in watershed
conditions, and if so, what are its detection limits (i.e., how much improvement would
have to occur before it could be detected by the index )? Third, a retrospective
comparison of IWI indicator changes to watershed improvement as measured
biologically and chemically at one or more sites could be used to answer the question
of whether or not watershed quality improvements are reflected in the IWI approach.
5.3 Sources of Uncertainty
Distinct from the question of which indicators have a sound ecological basis is
the issue of indicator uncertainty and the related question of data sufficiency
thresholds (e.g., what sample size or temporal and spatial coverage of data points is
necessary before a data layer is included in the Index.) A misleading or incorrect
sense of watershed condition or vulnerability may result where indicator methods differ
from state to state (e.g., Indicator 1 - Designated Uses); where indicators are adopted
with limited data (e.g., Indicator 2 - Fish and Wildlife Consumption Advisories); where a
few, scattered measurements are used to represent the entire watershed (e.g.,
Indicator 4 - Contaminated Sediments); where a statewide or basinwide measurement
is assigned to all watersheds within that larger geographic unit (e.g., Indicators 7a and
7b - Wetland Loss); or where the indicator is measured in a portion of the watershed
that is not representative of the conditions in general (e.g., Indicator 5 - Ambient Water
Quality, metals). Although the IWI document (U.S. EPA, 1997a) provides some
qualitative information about indicator uncertainty and data sufficiency thresholds, more
11
-------
quantitative information would improve indicator interpretation. For example, the
statement for Indicator 7b that "all available data were used" implies low uncertainty,
but the fact that the data set used (Dahl, 1990) contains only 48 values for the
conterminous United States (one value per state) indicates a much higher level of
uncertainty.
This discrepancy suggests the need for some indication of the limits and
uncertainties associated with reported indicator values. The IWI document should
characterize the uncertainty associated with assignments of watershed conditions and
vulnerability in the very beginning of the document. Attempts should be made to
identify the major sources of uncertainty in IWI calculations. These sources of variation
include: a) variation in sample size across different indicators and within indicators
across states; b) measurement and reporting errors across data sets; and c) the loss of
information associated with moving from fine-scaled spatial data to coarser-scaled
averaged data (e.g., in spatial data that are highly aggregated, the average value
almost never occurs).
The Committee thus recommends that IWI provide some indication of the error
associated with indicator values. This recommendation was made in the previous
EPEC review of the IWI, but has yet to be addressed adequately. As a first
approximation, error tends to decrease with increasing sample sizes used to arrive at
indicator values. Consequently, we recommend that sample size distributions
accompany each map and that a measure of variability in indicator values be given
along with the integrated watershed score.
In addition, data sufficiency thresholds vary widely among the current indicators.
For example, Indicator 5 (Cu, Cr, Ni, and Zn) has a higher threshold than others,
requiring 20 observations representing a minimum of five sites, and therefore most of
the map is in the "insufficient data" category. This is in contrast to other maps (e.g.,
Indicators 7a, 7b, and 12a), where no data sufficiency thresholds are applied. The
rationale for selecting sufficiency thresholds should be made explicit and the
relationship of this selection to the uncertainty of the indicator should be clarified.
Additional uncertainty is introduced by combining heterogeneous variables into a single
indicator, as was done for Indicators (Ammonia, Dissolved Oxygen, Phosphorus, and
pH). These four conventional pollutants have vastly different anthropogenic sources,
and combining them obscures their individual contribution to water quality.
12
-------
5.4 Redundancies Among Indicators
There are several reasons why it may be important to identify redundancies
among indicators included in the IWI (i.e., where multiple data layers are in fact
providing information about the same stressor or effect). The first is the concern, given
that the current algorithm gives equal weight to all indicators except Indicator 1, that
some stressors or effects may be "double counted" in the integrated watershed score if
more than one data layer relates to that stressor or effect. This concern would be
moderated if the algorithm were modified so that indicators providing information about
the same ecological characteristic are clustered and weighted as a group. A second
issue associated with redundancy is the desirability of providing the maximum amount
of information with the minimum required data.
Redundancies among indicators could be evaluated using the conceptual
model recommended in Section 5.1 to relate the indicators to ecological condition in
order to define a parsimonious set of indicators. Another approach would be to
conduct statistical analysis of the indicators and combinations of them. For example,
as a first step, pairwise scatter diagrams and simple regressions of indicators would
reveal whether some of these indicators may be redundant. Multivariate methods
could be used after potential relationships are uncovered by these simple bivariate
methods. Multivariate analysis is a method to reduce the number of variables in a data
set, and also could be used to design weighting scores for redundant variables. An
additional benefit of conducting these exercises would be that descriptive
characteristics of the indicators can be compared for consistency.
5.5 Recommended Additional Indicators
Charge Question 4. Are the priorities for development of additional indicators
appropriate?
In the Committee's view, watershed characterization requires information not
only on the presence of environmental stressors (e.g., concentrations of chemical
contaminants and measures of landuse change), but also on environmental effects or
responses. In short, characterizing watershed condition (or state of health) requires
assessment of information about several attributes relating to structure, function, and
composition of the ecological resource. In order to determine whether the priorities for
development of additional indicators are appropriate, therefore, the Committee
recommends that the IWI identify the set of major ecological attributes that should be
included in a watershed assessment, determine which of these are not reflected in the
current IWI data layers, and identify which missing attributes can be added most
efficiently in the near term. The following sections provide suggestions for additional
high-priority indicators and offer recommendations for improving the relevance of
existing IWI information. In particular, indicators of biological effects should be added,
13
-------
because the numbers, types, and condition of organisms living in a watershed provide
a strong indication of watershed conditions.
When selecting new indicators, the Office of Water should take greater
advantage of the wealth of published information available on ecological indicators,
much of which has been produced by EPA (e.g., McKenzie et al., 1992; Jones et al.,
1997; U.S. EPA, 1998a, 1998b). The IWI has used numerous EMAP products, but
could use the results from the Office of Research and Development's Ecological
Indicators Research to a greater extent.
5.5.1 Indicators of Effects
The Committee is concerned that the majority of the IWI indicators are based
upon chemical measurements either in the water or the sediments without providing
complementary information on ecological effects. In addition, the current IWI
assessment is "blind" to many problems not associated with the four conventional
pollutants and heavy metals. For example, although "potential for agriculture runoff"
and "fish advisories" have the potential for identifying insecticide problems, it would be
more useful to have actual chemical concentrations and measures of ecological effects
(e.g., biotic integrity).
While the present set of indicators provides a useful screening level measure of
watershed condition, these measures are predominately exposure measures and not
biological effects measures. This reflects the regulatory bias towards these data by the
federal government and the states in implementing the Clean Water Act and regulating
point sources. The condition of a given watershed should not be measured solely as a
function of water quality (i.e., chemistry), but also by its biological integrity/quality. The
Agency has released guidance documents on biological assessment and biocriteria
(e.g., U.S. EPA 1996e; 1998c) and has begun to place more emphasis upon collection
of biological data at the state level with a view towards providing these data in the
305(b) state reports. We support this action and recommend that biological indices be
incorporated in the IWI. Biological data supplied in 305(b) reports or from other
sources should constitute a separate data layer in the IWI to complement chemical
water quality assessment information. When assessing biotic integrity, preference
should be given to indices that include measured biological conditions, rather than an
index that is an indirect surrogate for biotic integrity (e.g., lack of heavy metals, no
exceedences of pH or dissolved oxygen reference levels).
14
-------
5.5.2 Terrestrial Indicators
Charge Question 10. Are the data sets that underlie the proposed terrestrial
indicators used appropriately? How can the proposed indicators be improved for
purposes of the IWI?
Charge Question 9. Are there other available data sets or indicators of terrestrial
condition that the Agency should evaluate for use in the IWI?
The Committee reiterates its previous recommendation that a valid watershed-
scale assessment must identify and integrate both terrestrial and aquatic
environmental indicators (EPA-SAB-EPEC-ADV-97-003) and encourages the Agency to
continue to work in that direction. Efficient development of terrestrial indicators will be
facilitated by clear articulation of management goals and objectives, recognition that
terrestrial indicators may need to vary on a regional basis, and a strategy to organize
watersheds into larger landscape units. Management goals and objectives must be
clearly presented, since there is no statutory requirement for comprehensive monitoring
of terrestrial systems analogous to CWA 305(b) reporting. Terrestrial management
goals can be based on benefit flows (products or services) from ecosystems to society,
or on inherent ecosystem condition (i.e., system integrity and sustainability). The
alternative (benefit flows versus system integrity) chosen to guide terrestrial indicator
selection should be consistent with the definition of "condition" developed for the
overarching strategic plan.
Terrestrial indicators may differ from the aquatic indicators that are used to
characterize watershed condition in at least two respects. First, relevant indicators will
vary among watersheds because different terrestrial ecosystem types (e.g., forest,
rangeland, and agroecosystems) cannot be reflected by the same descriptors.
Second, in order to organize and identify specific terrestrial ecosystem types,
watersheds must be integrated into larger units (e.g., ecoregions) than the USGS
hydrologic units that form the basis of IWI data layers. Addressing these issues up
front in the strategic plan will allow for smoother additions of data layers in the future.
The Committee understands that a number of the previously proposed terrestrial
indicators have been set aside due to problems encountered with the available data.
The Committee agrees with the Agency's decision to table the Partners in Flight
(Terrestrial Indicator B), Diversity of Natural Land Cover Types (Terrestrial Indicator A3),
Agriculture (Terrestrial Indicator A2), and Human Use (Terrestrial Indicator A1)
indicators. The latter three may be reconsidered when the MRLC data set becomes
available (see below).
15
-------
The "Conterminous U.S. Land Cover Characteristics Data Set" database
(Loveland et al., 1991), which served as the basis for Terrestrial Indicators A, C, and D,
is not optimal for this application because of its generalization of land cover types,
coarse spatial resolution (1 km pixels), and the use of mixed classifications
(forest/agriculture). A better alternative will be the Multi-Resolution Land
Characteristics (MLRC) data, which are available now for states east of the Mississippi
River, and which will be available by January 2000 for the entire conterminous U.S.
The Committee recommends, therefore, that the Agency utilize the MRLC data as they
are released, rather than continuing with use of the Conterminous U.S. Land Cover
Characteristics Data Set to compute terrestrial indices requiring land cover data. If the
Agency decides that it is more important to release the information quickly than to wait
for national coverage, then terrestrial indices could be released for multi-state
geographic regions as MRLC data become available.
The Agency derived several of its proposed terrestrial indicator maps from the
Conterminous U.S. Land Cover Characteristics Data Set by interpreting the proportion
of urban, agricultural, and forested land from their original legend descriptions (Jones
et al., 1997). Pixels containing mixed forest/agricultural land were counted as having
100% cover in both categories, leading to overestimation of total land cover. This
assumption also generated serious errors in Indicator 18 - "Agricultural/Urban Riparian
Habitat", resulting in gross over-representation of riparian agricultural land cover in the
states of Maine, Massachusetts, New Hampshire, Vermont, and northern Minnesota.
These errors are evident when Indicator 18 is compared with NRCS's "Percent of Non-
Federal Area in Cultivated Cropland"
(http://www.nhq.nrcs.usda.gov/land/lgif/m2081l.gif). Pending its revision, we strongly
recommend that Indicator 18 be removed from the IWI web site.
The data sets used to compute the "Roads Crossings Streams" map (Terrestrial
Indicator E) are the 1:1,000,000 scale USGS Digital Line Graphs (DLGs) for rivers and
roads. Line density on these 1:1,000,000 DLGs is greatly influenced by cartographic
convention (e.g., rivers and roads are not shown where they are too dense to map).
Thus, the 1:100,000 scale USGS Transportation DLGs and joint EPA-USGS National
Hydrography Data set, already available in digital form, would be more appropriate.
Also, EPA may wish to consider the Federal Highway Administration's "Highway
Statistics Information Retrieval System" to weight roads by usage, because a bridge
crossed by only a few cars a day will have much less impact on aquatic life than one
that is crossed by thousands of cars a day.
In selecting or developing terrestrial indicators, the Office of Water will have to
consider how much of the terrestrial data are available in the immediate (1-2 year) time
frame and what might be useful, but would only be available on a longer time frame.
The Agency also should consider additional sources of information on watersheds,
including Appendix B of the recently released National Research Council
pre-publication report, New Strategies for America's Watersheds, which can be viewed
16
-------
online in the "Reading Room" of the National Academy Press (www.nap.edu), and a
recent USDA publication entitled America's Private Land, A Geography of Hope (NRCS,
1996). The latter contains watershed information on nationwide maps, including the
following:
a) Dominant Land Uses;
b) Patterns of Agricultural Diversity;
c) Net Gains and Losses in Irrigated Cropland Acreage;
d) Sediment Delivered to Rivers and Streams from Sheet & Rill Erosion
(shows pattern similar to IWI Indicator 12c);
e) Confined Livestock Concentration;
f) Potential Nitrogen and Phosphate Loss from Farm Fields (Nitrogen loss
data shows patter similar to IWI Indicator 12b);
g) Pesticide Runoff Potential for Field Crop Production (used in IWI Indicator
12a);
h) Pesticide Leaching Potential for Field Crop Production (this is the
potential for groundwater contamination from pesticides, and the spatial
distribution is MUCH different than that for runoff); and
i) Freshwater Consumption as a Percentage of Local Average Precipitation.
There are also many relevant spatial data sets for water that can be downloaded from
the USGS Water Resources web page (http://water.usgs.gov/lookup/getlist), including
agricultural chemical uses and "Risk of Nitrate Contamination in Groundwaters of the
United States."
The IWI has already used the output from simulation models to predict pollutant
loadings (Indices 12a, 12b, 12c), and should also consider their use to predict
ecological response to environmental stressors. For example, the EPA Pesticides and
Industrial chemical Risk ANalysis and Hazard Assessment (PIRANHA) model
(developed by the EPA Environmental Research Laboratory in Athens, GA) could be
used to predict aquatic organism risks from toxic chemicals.
5.5.3 Riparian Habitat Indicators
The fundamental importance of riparian vegetation to the structure and function
of stream ecosystems has been well documented (e.g., Cummins et al., 1989, Gregory
et al., 1994). Two of the Terrestrial Indicator maps distributed to EPEC in July 1998
that remain on the IWI web site (http://www.epa.gov/surf2/iwi) as of January 1999 are
riparian indicators: Indicator 16 - "Forest Riparian Habitat" and Indicator 18 -
"Agricultural/Urban Riparian Habitat" (Terrestrial Indicators C and D, respectively, in the
materials received by the Committee). The renaming of these maps is an improvement,
as it more correctly reflects their contents than the former names ("Riparian Habitat
Integrity" and "Riparian Habitat Vulnerability", respectively). However, "Forest Riparian
Habitat" is an inappropriate measure of riparian condition in the prairie and desert
17
-------
states because riparian areas were never forested there, and "Agriculture/Urban
Riparian Habitat" overestimates agricultural influences in the northern U.S. states (see
Section 5.5.2). The Committee strongly urges, therefore, that the Agency work to
develop better riparian habitat indicators. Because riparian habitats have been shown,
in essentially all carefully studied watersheds, to be major regulators of in-stream biotic
resources, very significant effort should be expended to incorporate their characteristics
into the IWI.
Stream density (i.e., length of stream per unit area of watershed) is a simple yet
powerful indicator that could be added. Studies have shown that there is an inverse
relationship between stream order (or width) and riparian influence on in-stream
ecology. In other words, the smaller the stream (i.e., the lower the order), the greater
the influence of the riparian zone. In any given drainage basin, stream orders 1 to 3
will be maximally influenced by riparian cover, and will also dominate stream density.
At the watershed scale, the result is a direct relationship between stream density and
potential riparian influence; the higher the stream density, the greater the proportion of
the watershed potentially under direct riparian control of biological integrity. At the
cumulative watershed level, high drainage density, and its coupled proportionally large
riparian control, would infer greater resilience in the face of disturbance to headwater
tributary riparian areas. However, a fixed percent removal of riparian cover would
involve a much greater area of channels in high drainage density watersheds than in
those having low drainage density.
An additional riparian indicator, the percent of intact riparian cover, would be a
reliable predictor of overall in-stream biotic condition. The intact nature of riparian cover
can be inferred from the condition of the general vegetation cover. For example,
complete watershed cover by old or second growth forest would almost always include
good riparian cover. A profitable way to develop the indicator would be to intersect a
stream map with a map of potential terrestrial plant cover (e.g., potential vegetation
types listed by Omernik, 1987). The combination of these two factors provides a basis
for assessing the potential influence of riparian habitat on the aquatic resources of the
watershed. The ecoregion vegetation cover type for the watersheds provides an
indication of expected riparian cover. Thus, high drainage density watersheds in
minimally disturbed terrestrial or grassland ecoregions would be indicative of high
quality watersheds. Further, estimates of removal of vegetation cover, even at 1 square
km resolution, in high drainage density areas should reliably predict watershed
impairment at the level of aquatic biotic resources. This information could be obtained
from NRCS's "Percent Change in Forested Land Cover, 1982-1992"
(http://www.nhq.nrcs.usda.gov/land/sgif/m2310s.gif) and EPAs North American
Landscape Characterization (NALC) program (U.S. EPA, 1993).
Additional resolution could be achieved by designating climax and pioneer
riparian vegetation cover types characteristic of each ecoregion. For example, for most
watersheds in forested and grassland ecoregions, some species of alder would be
18
-------
expected to be dominant in both deciduous and con if erf crested areas as well as
grassland areas in second growth (re-growth) condition. Although riparian condition
(i.e., percent intact riparian habitat) might be inferred from run-off data from the USGS
National Water Information System (http://waterdata.usgs.gov/nwis-w/US/), other
riparian inventory data bases would need to be sought. In the long-term, a national
riparian inventory, similar to the wetlands inventories that have been introduced, will be
required. A significant research question for the future will be the most efficient (cost-
effective) way to accomplish this, for example through the use of a volunteer/intern
work force in collaboration with USDA, especially USFS. A critical point will be
designation of percent intact riparian habitat in agricultural lands, in as much as
watershed condition could be noted good if riparian coverage is high.
5.5.4 Lake Indicators
In addition to the general call for the inclusion of indicators of biological effects,
the Committee recommends that IWI be expanded to include indicators for lakes and
reservoirs. Lakes are important ecosystems which are highly valued for human use,
and for which there are abundant chemical and biological data suitable for indicator
development (e.g., Kanciruk et al., 1986; Omerniket al., 1988; U.S. EPA1998c).
Because lakes are not uniformly distributed nationwide, lake indicators might be
developed for only a subset of watersheds, comparable to the approach already used
for estuaries (Indicator 15).
5.6 Research Needs
Charge Question 5. What additional steps, including additional research, could
be undertaken to improve watershed assessments?
The primary focus of the IWI developers has been on utilizing existing indicators
and watershed assessment information. This focus is appropriate for the initial IWI
goal of providing water resource information on a watershed basis for citizens and
resource managers. In this report, the Committee suggests expanding the IWI effort to
truly report on watershed condition by evaluating ecological attributes in addition to
water quality. In order to accomplish this purpose, additional existing databases and
classes of indicators (e.g., biological indicators) should be considered. There are a
number of areas, however, where additional research will be needed to develop new
indicators and additional environmental data will need to be collected. Important
research areas underlying watershed assessment include: development and validation
of terrestrial and landscape indicators; measures of ecological processes, such as
organic matter turnover rates in streams in the form of riparian-derived vascular plant
litter; techniques for aggregating individual indicators into representative indices that
take into account biotic and abiotic variables, and missing values in data sets;
weighting methods that incorporate community differences expected in different
habitats or biomes; and understanding the transport, fate, and effects of atmospheric
19
-------
contaminants on watersheds.
Further effort should be devoted to continuing improvement of the IWI
methodology, with particular attention to issues discussed in this report, including the
use of conceptual and simulation models to support the integration algorithm,
sensitivity analysis to identify "key" indicators and to assess known or hypothetical
changes in water quality or biological integrity, uncertainty estimation, and evaluation
methods. The IWI strategic plan provides an opportunity to describe and prioritize key
additional information that needs to be developed, as well as to indicate ways in which
the information may be generated (e.g., ORD research activities, STAR grant program,
programs underway at other federal agencies). Many improvements to the IWI,
however, including most of those suggested in this report are possible with existing
information.
20
-------
6. INTEGRATION ALGORITHM
Charge Question 6. Is the algorithm used to integrate the individual indicators
into a watershed score appropriate, given the objectives of the IWI?
Charge Question 7. How should the proposed additional indicators be
incorporated into the integrated index?
The algorithm used in Phase I of the IWI calculates a watershed score by
summing the point contributions for the seven "condition" indicators (to produce a
watershed condition score) and for the eight "vulnerability" indicators (to produce a
vulnerability score). Within the condition score, the greatest weight (6x, with a
maximum contribution of 18) is accorded to Indicator No. 1 (state 305(b) assessments
of the extent to which surveyed surface waters meet Water Quality Standards and
designated uses). The remaining six condition indicators can contribute a total of 12
points.
While noting that the current approach is inherently subjective and not based on
science, the Committee emphasizes that the selection of the most appropriate
algorithm will depend on the goals of the IWI as stated in the strategic plan. In
general, a good algorithm should have the following qualities:
a) Respond to the overall objectives of the project;
b) Be amenable to the inclusion of new indicators; and
c) Be transparent.
As noted earlier, in order for the algorithm to respond to the objectives of the
project, those objectives must be well defined. For example, if the main objective is to
highlight the watersheds with low water quality as defined exclusively by CWA Section
305(b), then the present algorithm may be sufficient with minor modifications. If,
however, the objective is to characterize watersheds with respect to overall water
quality, then the present algorithm is insufficient. That is, it would not capture
indicators of poor water quality that are not explicitly captured by 305(b) water
chemistry data, such as biological indicators. Finally, if the objective of the IWI, as
stated in the documentation provided to the Committee, is to characterize watersheds
with respect to overall conditions and vulnerability, then the current algorithm falls far
short of this goal.
The algorithm must be flexible with respect to the amount of data used to
compile the composite indicators. That is, since the data sets from individual
watersheds will be widely disparate across the country, the algorithm must have built-in
functionality to handle differing amounts of data without skewing the resulting
composite indicator. The present algorithm assigns a score that is equivalent to better
21
-------
water quality when there are insufficient data, skewing the composite to favor
watersheds with few data sets. The composite score of a watershed should not
degrade solely because a larger data set is available for its calculation; in other words,
the algorithm should be designed to reward (not punish) organizations just because
they have collected and reported a lot of data.
It is crucial that any algorithm used to compile data and produce an overall
composite indicator be easily understandable to those who may utilize the composite
data. A truly transparent algorithm would allow the user to independently judge the
usefulness of the resulting parameter and make informed decisions about the
appropriate application of such results.
The Committee has a number of specific recommendations for improvement of
the algorithm. First, the Committee believes that watershed condition and watershed
vulnerability (or susceptibility) are individual measures that should be reported
separately. This would avoid the awkwardness associated with some of the presently
applied descriptors, e.g. "more serious water quality problems, low vulnerability."
Second, as mentioned earlier, indicators should be grouped into composites that
represent broad measures of watershed condition that could then be weighted (if
desired) to achieve an overall quantitative descriptor of watershed quality and a
separate descriptor of watershed vulnerability. This exercise would be greatly aided by
the development of a conceptual model, as discussed in Section 5.1. A consistent
intellectual framework for the selection and integration of multiple indicators would
allow orderly inclusion of additional indicators as they are developed.
Appropriate composites for watershed condition, most of which would have both
aquatic and terrestrial components, would include: water quality, habitat quality and
extent, hydrology, energy and nutrient flows, and native biota. Composites for
watershed vulnerability would include sensitive biological communities, landscape
change, and critical habitats (including, e.g., percent intact riparian habitat). Within
these composite categories, it would be important to include indicators of potential
stressors and measures of biotic integrity as they apply to keystone, rare and
endangered, and commercially important species. Development of vulnerability
composites will be complicated somewhat by the fact that the factors contributing to
vulnerability will often be stressor specific (e.g., watershed and surface water
characteristics that describe sensitivity to acid deposition are given by Marcus et al.,
1983).
Individual indicators should be used to calculate composites in a manner that
would be amenable to inclusion or exclusion of available indicators. Efforts should be
made to ensure that results would not change simply due to choice of indicator sets
but would actually be tied to additional information. For example, instead of summing
individual indicator scores to arrive at a composite, individual scores could be
normalized then averaged, thereby allowing any number of indicators to be easily
22
-------
incorporated without skewing the data. Insufficient data sets would therefore not be
assigned any value and would not be included in the average.
Arbitrarily chosen calculation methods that are used solely to spread the
composite scores more evenly across the watersheds should be avoided. Specifically,
in the current algorithm, if insufficient 305(b) data are available then other data are
summed and tripled. A more logical, quantitative, and defensible method for
incorporating additional data sets should be applied (see above).
In any case, the Committee feels that the assignment of weights in the
integrating algorithm requires validation. As noted above, we recommend that IWI
undertake a sensitivity analysis of the Index calculations to determine which indicators
are the most robust indicators of response in watershed quality. In this way, the
indicators could be weighted according to their relative sensitivities. The Committee
recognizes that this would require additional effort and may take some time. However,
a sensitivity analysis will lead to a more natural weighting and remove any biases
associated with an arbitrary over-weighting of selected indicators.
Finally, it is essential that whatever method is chosen to assign composite
scores to watersheds be made explicit. Both the web site and printed documents
associated with the IWI should include not only a description of the algorithm, but
worked examples illustrating actual calculations.
23
-------
7. PRESENTATION AND DOCUMENTATION ISSUES
7.1 Map Data Intervals
Charge Question 8. Should different map intervals/data breakpoints (see hard
copy of map legend for example) be used for any of the individual indicators or
the roll-up index?
The data breakpoints or intervals are not consistent across the various IWI
indicator maps. For example, terciles are used for several indicators whereas a
different number of intervals is used in other indicators. The selection of breakpoints is
not always clear and needs to be better documented since decisions about breakpoints
for narrative categories can greatly influence the message portrayed by a map. (For
discussion of this and other presentation issues, see Monmonier, 1996). One way of
addressing the question of breakpoints is to provide a histogram of the data distribution
together with the map, as done in several figures (e.g., Figures 2.3, 2.4, 2.5) of "An
Ecological Assessment of the United States Mid-Atlantic Region" ( Jones et al., 1997).
It may also be important to provide some examples of how the maps change if different
breakpoints are selected.
7.2 IWI Presentation
Charge Question 11. Are the IWI results presented in an understandable
manner? Are the methods used to disseminate the results (i.e., published map
reports and the web site) appropriate?
As highlighted early in this report, the Committee feels that the presentation of
IWI results on the web site and in published hardcopy maps has been very effective.
The large number of visitors to the IWI web site is a testimony to the interest that has
been generated by the effort. While issues still remain about the clarity of the
integration algorithm and issues associated with individual data layers, we continue to
feel that the Agency has taken a very positive step in making watershed information
available in such a user-friendly format.
24
-------
7.3 IWI Documentation
Charge Question 12. Are the algorithms used in the IWI (e.g., to generate data
layers from source data, and to integrate multiple indicators) clearly
documented?
Charge Question 13. Are there adequate metadata available to users to describe
the content, quality, condition and other characteristics of the IWI indices and
their source data? Do the metadata meet applicable Federal Geographic Data
Committee (FGDC) content standards?
As discussed in Section 6, documentation of the algorithms used in the IWI
requires further improvement so that IWI users can more easily trace the assumptions
and decision rules used to transform source data to data layers to composite indicators
to calculation of IWI scores. With regard to metadata, documentation about the IWI
data layers is provided by links to ORD's Environmental Information Management
System (EIMS). This documentation was hard to find on the Web page at the time of
the EPEC review meeting, but the links have since been made more prominent. The
use of EIMS is a good start, but there should be additional efforts to provide metadata
that meet applicable Federal Geographic Data Committee (FGDC) content standards.
When an IWI data set is derived from another data set within EIMS, there should be a
link to that EIMS page. For example, the description of the IWI indicator "Ambient
Water Quality Data - Four Conventional Pollutants" should have a link to the EIMS
description of EPAs STOrage and RETrieval System (STORET) data set from which it
was derived. When an IWI data set is derived from a data set from a federal source
outside of EPA, there should be links to Web pages of those data providers (e.g., the
NRCS Natural Resources Inventory (NRI), the Fish and Wildlife Service Wetlands
Status and Trends data, the U.S. Census Bureau). Not only will this supply
information about data lineage, it will also simplify the task of writing metadata for IWI
products, because the metadata may already have been written for the source data set.
25
-------
8. CONCLUSIONS AND RECOMMENDATIONS
The presentation of IWI results on the web site and in published hardcopy maps
has been very effective. While issues still remain about the clarity of the integration
algorithm and the quality of individual data layers, we continue to feel that the Agency
has taken a very positive step in making watershed information available in such a
user-friendly format. The scientific basis of IWI, however, should be improved.
The IWI currently highlights watersheds with low water quality, as defined by the
state water quality standards. This approach does not capture other reliable indicators
of poor water quality, such as biological indices, because only a handful of states
include biological criteria in their water quality standards. More importantly, the IWI
does not characterize watershed condition (i.e., "health") and vulnerability as claimed.
The Committee supports the Office of Water's efforts to incorporate new indicators and
adjust the integrating algorithm to achieve the stated goals of the IWI. To this end, the
Committee recommends the following:
a) The Agency should develop a strategic plan to articulate IWI's goals and
objectives, identify its target audiences and clientele, define terms, and
identify data gaps in order to assure that the IWI can be updated
efficiently with additional data layers and provide an improved
assessment of condition. A Quality Assurance/Quality Control (QA/QC)
plan should be included in the strategic plan.
b) The Agency should develop a conceptual model for the IWI that can be
used to guide the selection of additional data layers and refinements to
the integrating algorithm.
c) The Agency should add more indicators of biological and ecosystem
effects to the IWI.
d) The Committee reiterates its previous recommendation to include
terrestrial indicators, but recommends waiting until the Multi-Resolution
Land Characteristics (MRLC) dataset becomes available to develop
indices related to land use. The Agency should develop better riparian
habitat indicators to replace its proposed riparian indicators (Indicator 16 -
Forest Riparian Habitat and Indicator 18 -Agricultural/Urban Habitat).
e) The Agency should evaluate each existing and proposed indicator to
demonstrate that changes in the indicator correspond to meaningful
changes in environmental quality. The Agency should better document
data sufficiency and other sources of indicator uncertainty and describe
sensitivity of the composite index to variation of the component indicators.
26
-------
f) The Agency should revisit the current integrated index, which falls short
of the goal of characterizing watershed condition and vulnerability. As
part of this exercise, the Agency should undertake the appropriate
analyses to assign differential weights to the individual indiators based on
their relative importance as predictors of watershed integrity.
g) The integration algorithm must be flexible with respect to the amount of
data used to compile composite indicators, so as not to reward (or
punish) organizations just because they collected and reported a lot of
data.
h) Watershed quality and vulnerability are individual measures that should
be reported separately.
27
-------
9. REFERENCES CITED
Cummins, K.W., M.A. Wilzbach, D.M. Gates, J.B. Perry, and W.B. Taliaferro. 1989.
Shredders and riparian vegetation. Bioscience 39: 24-30.
Dahl, I.E. 1990. Wetlands losses in the United States, 1780's to 1980's. U.S.
Department of the Interior, Fish and Wildlife Service, Washington, D.C. 21 pp.
Goodchild, M.F., B.O. Parks, and L.T. Steyaert. (Eds). 1993. Environmental Modeling
and CIS. Oxford University Press.
Goodchild, M.F., L.T. Steyaert, B.O. Parks, M.P. Crane, C.A. Johnston, D.R. Maidment,
and S. Glendinning. 1996. CIS and Environmental Modeling: Progress and Research
Issues. CIS World, Fort Collins, CO.
Gregory, S.V., F.J. Swanson, W.A. McKee, and K.W. Cummins. 1991. An ecosystem
perspective of riparian zones. BioScience 41: 540-551.
Jones, K.B., K.H. Riitters, J.D. Wickham, R.D. Tankers ley Jr., R.V. O'Neill, D.J.
Chaloud, E.R. Smith, and AC. Neale. 1997. An Ecological Assessment of the United
States Mid-Atlantic Region. Office of Research and Development, U.S. Environmental
Protection Agency, Washington, DC. EPA/600/R-97/130
Kanciruk, P., J.M. Eilers, R.A McCord, D.H. Landers, D.F. Brakke, and R.A Lindhurst.
1986. Characteristics of lakes in the eastern United States, vol. III. Data compendium
of site characteristics and chemical variables. EPA/600/4-86/007c. U.S. Environmental
Protection Agency, Washington, DC 439 pp.
Loveland, T.R., J.W. Merchant, D.O. Ohlen, and J.F. Brown. 1991. Development of a
land-cover characteristics database for the conterminous U.S. Photogrammetric
Engineering and Remote Sensing, v. 57, no. 11, p. 1453-1463.
Marcus, M.D., B.R. Parkhurst, and F.E. Payne. 1983. An Assessment of the
Relationship among Acidifying Depositions, Surface Water Acidification, and Fish
Populations in North America. EA-3127, Volume 1, Final Report. Electric Power
Research Institute, Palo Alto, CA
McKenzie, D.H., D.E. Hyatt, and V.J. McDonald (Eds). 1992. Ecological Indicators.
Volumes I and II. Elsevier Applied Science, London.
Mclntire, C.D. and J.A. Colby. 1978. A hierarchical model of lotic ecosystems.
Ecological Monographs 48:167-190.
R-1
-------
Monmonier, M. 1996. How to Lie With Maps (2nd Edition). University of Chicago
Press, Chicago, IL.
NCGIA. 1996. Third International Conference/Workshop on Integrating CIS and
Environmental Modeling. Santa Fe, NM, Santa Barbara, CA. National Center for
Geographic Information and Analysis.
Odum, H.T. 1994. Ecological and General Systems: An Introduction to Systems
Ecology.
Omernik, J.M. 1987. Ecoregions of the conterminous United States. Annals of the
Association of American Geographers ZZ(1): 118-125. Map suppl., map scale
1:7,500,000.
Omernik, J.M., D.P. Larsen, C.M. Rohm, and S.E. Clarke. 1988. Summer total
phosphorus in lakes: a map of Minnesota, Wisconsin, and Michigan, USA.
Environmental Management 12(6): 815-825.
U.S. Department of Agriculture. 1996. America's Private Land: A Geography of Hope.
Natural Resources Conservation Service, Washington, DC. Program Aid 1548.
U.S. Environmental Protection Agency. 1993. North American Landscape
Characterization (NALC). EPA/600/R-93/135
U.S. Environmental Protection Agency. 1996a. Clinch Valley Watershed: Ecological
Risk Assessment - Planning and Problem Formulation. Risk Assessment Forum.
EPA/630/R-96/005a
U.S. Environmental Protection Agency. 1996b. Big Darby Creek Watershed:
Ecological Risk Assessment -Planning and Problem Formulation. Risk Assessment
Forum. EPA/630/R-96/006a
U.S. Environmental Protection Agency. 1996c. Middle Snake River Watershed:
Ecological Risk Assessment - Planning and Problem Formulation. Risk Assessment
Forum. EPA/630/R-96/008a
U.S. Environmental Protection Agency. 1996d. Waquoit Bay Watershed: Ecological
Risk Assessment - Planning and Problem Formulation. Risk Assessment Forum.
EPA/630/R-96/009a
U.S. Environmental Protection Agency. 1996e. Biological Criteria: Technical Guidance
for Streams and Small Rivers. Office of Water. EPA 822-B-96-001
U.S. Environmental Protection Agency. 1997a. The Index of Watershed Indicators.
R-2
-------
Office of Water. E PA/841/R/97/010
U.S. Environmental Protection Agency. 1998a. Evaluation Guidelines for Ecological
Indicators (Draft Document). Office of Research and Development, October, 1998.
U.S. Environmental Protection Agency. 1998b. Communicating the Condition of
Terrestrial Ecosystems: A Focused Investigation of Terrestrial Ecosystem Health
Indicators. Office of Policy. (Prepared by ICF Kaiser under Contract No. 68-W5-0012).
U.S. Environmental Protection Agency. 1998c. Lake and Reservoir Bioassessment
and Biocriteria: Technical Guidance Document. Office of Water (EPA-841-B-98-007).
R-3
-------
APPENDIX A. CHARGE TO THE COMMITTEE
The Office of Water will be developing additional IWI data layers and library
maps as time goes on. The goal is to release information on a quarterly basis, and to
build the IWI with more current data. As part of this ongoing effort to improve the
scientific quality of the integrated watershed information provided by the IWI, the
Office of Water requests that EPEC provide peer review and advice on the strategic
plan for IWI, the algorithm used to calculate watershed scores, and the proposed
indicators of terrestrial condition. Specifically, the Agency requests the SAB to
consider the following questions:
Strategic Plan for IWI:
1. Does the strategic plan for IWI include the critical development activities,
and are the time frames envisioned appropriate/adequate?
2. Does the plan include a mechanism for IWI users and developers to
communicate data needs and gaps to those responsible for data
collection?
Watershed Characterization:
3. Are the IWI indicators (current and proposed), taken as a whole,
adequate to characterize watershed condition and vulnerability? Are
there redundancies among indicators?
4. Are the priorities for development of additional indicators appropriate?
5. What additional steps, including additional research, could be undertaken
to improve watershed assessments?
Roll-Up Algorithm:
6. Is the algorithm, used to integrate the individual indicators into a
watershed score, appropriate, given the objectives of the IWI?
7. How should the proposed additional indicators be incorporated into the
integrated index?
8. Should different map intervals/data breakpoints (see hard copy of map
legend for example) be used for any of the individual indicators or the
roll-up index?
A-1
-------
Terrestrial Indicators:
9. Are there other available data sets or indicators of terrestrial condition that
the Agency should evaluate for use in the IWI?
10. Are the data sets that underlie the proposed terrestrial indicators used
appropriately? How can the proposed indicators be improved for
purposes of the IWI?
Presentation and Documentation:
11. Are the IWI results presented in an understandable manner? Are the
methods used to disseminate the results (i.e., published map reports and
the web site) appropriate?
12. Are the algorithms used in the IWI (e.g., to generate data layers from
source data, and to integrate multiple indicators) clearly documented?
13. Are there adequate metadata available to users to describe the content,
quality, condition and other characteristics of the IWI indices and their
source data? Do the metadata meet applicable Federal Geographic Data
Committee (FGDC) content standards?
A-2
-------
United States Science Advisory EPA-SAB-EPEC-99-014
Environmental Board (1400) July 1999
Protection Agency Washington, DC vwvwepa.gov/sat)
&EPA AN SAB REPORT: REVIEW
OF THE INDEX OF
WATERSHED INDICATORS
PREPARED BY THE ECOLOGICAL
PROCESSES AND EFFECTS
COMMITTEE OF THE SCIENCE
ADVISORY BOARD
------- |