September 30, 1997
EPA-SAB-EPEC-ADV-97-003
Honorable Carol M. Browner
Administrator
U.S. Environmental Protection Agency
401 M Street, SW
Washington, DC 20460
Subject: Advisory on the Development of Phase II of the Index of
Watershed Indicators
Dear Ms. Browner:
On May 15, 1997, the Ecological Processes and Effects Committee of the
Science Advisory Board met to review and provide comments on the National
Watershed Assessment Project (NWAP), recently renamed the Index of Watershed
Indicators (IWI), being developed by the Office of Water. The purpose of IWI is to
provide available data in a Geographic Information System (CIS) format to interested
parties, including state and tribal governments and members of the public, for
assessing the condition and vulnerability of watersheds. The Committee's comments
take the form of an SAB Advisory, because they are intended to guide improvements to
IWI in the future (Phase II) rather than to revise the existing version, which has already
been released to the states. This approach was agreed to by the SAB because of the
schedule for release of Phase I (i.e., late June) that had already been established by
the Office of Water.
This letter contains the Committee's overall comments on IWI, as well as
responses to the five specific questions contained in the Charge. Although review
materials provided to the Committee consisted of the January 1997 version of IWI, at
the review meeting Agency staff circulated a revised (draft final) version of IWI that had
been provided to states and EPA Regions on May 1, 1997. The following comments,
therefore, are based primarily on the January version, but include some of the
Committee's observations about changes that occurred between the January and May
drafts.
Overall, the Committee feels that the basic concept of IWI, i.e., the provision to
the public, decision-makers, and scientists of integrated watershed information maps
for the nation, is a major step forward, but IWI is not yet applied by EPA in a clear and
constant way. Nonetheless, the approach shows much promise. Despite its
-------
enthusiastic support for state-of-the-science integrated environmental assessments,
such as those conducted under the Agency's Environmental Monitoring and
Assessment Program (EMAP), the Committee also recognizes an important role for less
data-intensive watershed assessments such as IWI. The IWI approach will be valuable
for a variety of reasons. It provides baseline maps at the national and watershed levels
that show the current status of a variety of indicators. Integrating the information at the
watershed, state, and national level is a very valuable product in itself, and we applaud
the Agency for making the information easily available to the public via the World Wide
Web. In addition, the preparation of watershed maps may help the states better
understand the condition of their watersheds and perhaps seek to improve the
environmental conditions of them. At a minimum, the maps, by highlighting gaps in the
data, may act as an incentive for state and local environmental managers to provide
additional existing data to EPA or to seek finer-scale indicator data for their own use in
watershed management. We also feel that displaying watershed information in a map
format may encourage people to ask questions about the data and its integration.
Finally, the maps will provide a basis for Agency discussions with states about the
status of and environmental improvements in their watersheds. In summary, we very
much applaud the approach and the drive to disseminate the information on the Web
and think that IWI will be a useful tool, but since IWI is very much a works-in-progress,
the approach needs to be further evaluated after more experience is gained with its use
before definite conclusions can be drawn about its utility in practice.
We are less enthusiastic about the content of some of the specific data layers,
but recognize that the Agency was working with existing data sources that, despite their
inadequacies, were often the only available alternatives. Many of the comments that
follow relate to ways in which the scientific clarity and rigor of IWI can be improved,
while retaining its current utility as a policy tool. Although the Committee did not
evaluate the process employed by IWI to weight and combine the various watershed
indicators to calculate total scores for watershed condition or vulnerability, we
recommend that this algorithm receive peer review prior to the release of IWI Phase II.
The Charge to the Committee contained five questions in regard to the
development of Phase II of IWI, which we deal with in turn below.
Charge Question 1: Are the purposes of NWAP/IWI clearly stated?
Yes, the purposes of NWAP/IWI described in the Charge to the Committee are
clear and reasonable. The Committee noted, however, that this language differed is
some respects from the description of the purposes of NWAP/IWI in the review
documents. As discussed further under Charge Question 4, the documentation for IWI
Phase I should be reorganized and integrated into a single document, rather than a
confusing series of attachments. The purposes of IWI should be clearly described in
that integrated document.
-------
Charge Question 2: Would the new data layers proposed for Phase II
provide the most useful enhancements to NWAP assessments for
both the watershed condition and for vulnerability? Should
additional new data layers be considered?
The current set of data layers included in Phase I of IWI are focused largely on
water quality measures, with only minor consideration of terrestrial and landscape
characteristics of watersheds that influence watershed condition. Thus, in addition to
the six data layers proposed for inclusion in Phase II (biological integrity, habitat,
groundwater, coastal condition indicator, air deposition, and downstream effects), the
Committee suggests the addition of land use change and other indicators of terrestrial
condition that influence watershed water quality. Additional terrestrial data layers that
should be considered include: a) an index of potential phosphorus runoff from
terrestrial sources, similar to Map 12b for nitrogen; b) terrestrial species at risk (a
logical companion to Map 8: Aquatic/Wetland Species at Risk); c) indicators of
terrestrial habitat quality, fragmentation, and soils; and d) population density. The
population change data in Map 13 cannot be evaluated in the absence of population
density data because a given percentage increase in the population in a watershed has
a much greater impact in watersheds where population density is very high (e.g.,
coastal California) than in watersheds where population density is very low (e.g., the
Adirondack Mountains of New York State).
Charge Question 3: Are the description and display of data limitations and
uncertainties in Phase I an appropriate model to use in Phase II?
How can these descriptions be improved?
It is highly likely that, as databases are refined and new ideas are incorporated
into the indicators, the maps will change over time. As new maps are developed, they
will replace the original maps. In order to avoid confusion and assist in communication
to users of IWI, therefore, we recommend that each of the indicator maps be annotated
to specify the version and release date (e.g., Version 01-97). As new indices are
developed or suggested changes are brought forth and incorporated, it will be
important that the system allow for trends to be back-calculated using the new
approaches. A key part of IWI is the development of trend data. Therefore, the ability
to plot the trends with new or revised indices will be important.
In addition, the Committee recommends that each map at the national level
should contain an explicit explanation of the degree of uncertainty that the Agency feels
is associated with the information that it displays. An uncertainty icon, such as a
barometer scaled from 0 to 10, should also be displayed on each map to give a quick
visual summary of the level of uncertainty associated with the map. The inclusion of
uncertainty information will assist decision-makers in the use of the IWI maps and
-------
should stimulate efforts to improve the databases and/or components of the indicator to
decrease that uncertainty.
Other critical information that should be included for each map includes the
source of the underlying data, sampling methods, and the geographic scale of
aggregation of the source data. Where source data are published, those publications
should be fully referenced in the map documentation. For example, the wetland loss
data used as one of the data layers for Map 7 are from a published document that
should be referenced (see Attachment A to this letter regarding this data source, which
was incorrectly attributed to the National Wetlands Inventory).
Information about the sampling methods and geographic aggregation of source
data is included for some maps, but not all. For example, the Natural Resources
Conservation Service (NRCS) NRI (does this stand for Natural Resources Inventory?)
data set is described well for Map 12a, but not for Map 7. If space is a limitation,
perhaps those data sources that were used for several of the maps (e.g., STORET,
NRI) could be put in the introductory documentation and then referenced on the
individual maps that used them. The narrative descriptions of data sources would be
enhanced by including inset maps showing the spatial aggregation of source data (e.g.,
for Map 7, a map of state boundaries for the historic wetland loss rates used and a map
of the 6-digit accounting unit boundaries for the NRI-derived wetland loss rates used )
Maps showing the separate data layers that were integrated into composite data
layers (e.g., Maps 3a, b, and c) were very helpful. The use of such supplemental maps
is encouraged for Phase II, and could be increased in future revisions of the Phase I
maps. For example, maps for each of the four heavy metals that went into the
composite "Ambient Water Quality Data - Four Toxic Pollutants" (Map 5) should be
shown, particularly in view of the changes that occurred in this composite map when a
difference suite of heavy metals was used for the April 18, 1997 vs. the January 9,
1997 version (see comments in Attachment A regarding Map 5). Similarly,
supplemental maps should be used to show the individual data layers that were used to
generate Maps 6, 7, and 11.
Charge Question 4: Can the Phase I approach to integration of the
separate data layers be improved in Phase II to provide a more useful
illustration of how to understand the overall condition and
vulnerability of watersheds?
With regard to the overall IWI documentation, the Committee recommends that
the maps, data caveats, and description of the scoring/index algorithm be integrated
into a single document. Members noted that this had been partially achieved in the
May 1 version of the documentation insofar as the maps and data caveats are now
displayed on facing pages. Nonetheless, the description of the algorithm used to
-------
weight and combine the various watershed indicators should be part of the integrated
documentation, rather than a separate attachment.
The decision rules used to combine individual data layers into a composite index
should be stated in the supporting documentation because changes in these decision
rules can lead to changes in the composite maps even when the source data remain
the same. Indeed, the maps provided to the Committee for review had already been
significantly revised by the time of the review meeting. In some cases, the changes
resulted in significantly different characterizations of watersheds due solely to changes
in the manner of data presentation or aggregation, rather than to changes in the
underlying data themselves. The Committee's comments on some of the changes that
occurred between different versions of specific maps are contained in Attachment A.
Overall, the Committee concluded that the best maps are those derived from a
single data layer that was collected using consistent methods nationwide (i.e., Maps
12, 13, and 14). Maps that combine multiple and/or disparate data layers are much
more subjective and difficult to interpret, making them less meaningful. Maps that
combine finely sampled data with coarsely sampled data into a composite index should
be used with extreme caution. For example, the wetland losses data set used in Map 7
contains only 50 values (one for each of the 50 states), whereas the NRI data set used
for the recent wetland loss rates consists of thousands of data samples distributed
throughout the country. The composite index is only as precise as its least precise
component (in this case, the 50 values from the Wetland Losses report), a fact that is
not evident from the map.
The rationale behind including or excluding separate data layers in composite
index maps needs to be stated explicitly. For example, the two versions (dated January
9, 1997 and April 21, 1997) of Maps 5 and 6 used different combinations of chemicals,
with no explanation for why the original suite of chemicals was used in the January
version, nor why it was deemed necessary to change it for the April version (also see
comments on Maps 5 and 6 in Attachment A to this letter).
Data thresholds used also should be stated explicitly. Sometimes this
information is contained in the map legends, but sometimes it is not. For example, the
data thresholds used to define low, moderate, and high levels of wetland loss
presumably changed between January and April because the two version of Map 7
were drastically different despite the use of identical source data (see Attachment A); it
was unclear what ranges of percent wetland loss were used for these low, medium, and
high categories.
Charge Question 5: What additional steps, including additional research,
could be undertaken to improve watershed assessments?
-------
The watershed assessment maps and associated data base will realize their
greatest benefit if they are linked to the Agency's Environmental Monitoring and
Assessment Program (EMAP), especially EMAP's regional assessments such as the
Mid-Atlantic Integrated Assessment (MAIA) program. There needs to be close
coordination between these efforts, because both data bases can enrich one another.
There may also be the opportunity in this linking to make the IWI assessments more
landscape-based, and less tied simply to water quality. A second important link would
be to The Nature Conservancy's ecoregional planning effort now underway for all of the
United States. Many of these TNC planning efforts are based on watersheds or
subwatersheds and have already synthesized data of value to IWI. Conversely, IWI
could provide data of value to the TNC efforts. One TNC watershed-based plan that is
virtually complete is for the Columbia Plateau, which would be worth looking into
carefully. In general, many federal and state agencies are engaged in ecoregional
planning programs of different kinds, often watershed-based for data management
purposes, although terrestrial in focus. In general, IWI needs to place itself in this
milieu of regional planning and regional troubleshooting.
The Committee is encouraged by the initiation of the IWI project and would
welcome the opportunity for continuing dialog with the Agency about IWI, such as
reviewing the maps to be produced in Phase II and reviewing the integration and
assessment algorithms as they are developed and refined.
We hope these comments will be useful to the Office of Water in the
development of Phase II of IWI, and we look forward to a response from Robert
Perciasepe, Assistant Administrator for Water.
Sincerely,
/signed/
Dr. Genevieve M. Matanoski, Chair
Executive Committee
/signed/
Dr. Mark A. Harwell, Chair
Ecological Processes and Effects
Committee
Attachment
-------
Attachment A: Comments on Specific IWI Maps
Upon comparison of the June 1997 "Index of Watershed Indicators" maps
(received at the May 15 EPEC Advisory Review of the National Watershed Assessment
Project) with the January 1997 draft maps received as part of the review packet prior to
the meeting, the Committee noted that differences between the two sets of maps were
in some instances substantial. The following are selected comparisons and
evaluations of the new and old maps:
Map 1. There were some changes in this map, presumably because the 1/10/97
version was for 1992-1994, whereas the 4/18/97 version was for 1997-1996. The
data sufficiency threshold description ("...assessments were considered
sufficient to characterize the condition of a watershed if more than 20% of the
total perennial stream miles...met all designated uses") conflicts with one of the
data categories shown in the legend (<20% met).
Map 3. There were big changes in this map. Although inclusion of the Supplemental
Maps (3a, b, c) helps, the decision rules used to combine the 3 maps and define
data sufficiency thresholds for the composite map were not documented.
Map 5. There were big changes in this map. Why were different metals used in the
1/9/97 version (Cd, Cu, Pb, and Hg) versus the 4/21/97 version (Cu, Cr, Ni, Zn)?
Map 6. There were big changes in this map. Why were different pollutants used in the
1/9/97 version (NH4, N03, P, and suspended sediments) versus the 4/21/97
version (NH4, DO, P, and pH)?
Map 7. There were extreme changes in this map, which are difficult to explain given
that the data sources and data categories (low, moderate, and high levels of
wetland loss) are identical for both the 1/7/97 and 4/21/97 versions. This
change illustrates that the decision rules used to combine and categorize the
source data can be manipulated to skew the depiction one way or the other. A
much higher proportion of the country has a "high level of wetland loss" on the
4/21/97 maps than in the earlier version. Supplemental maps should be included
to show the recent (NRI) and historical (USFWS) data.
ERROR OF FACT: the USFWS data used are no! from the National Wetlands
Inventory (which is not yet completed); they are from the National Wetlands
Status & Trends Reports, which should be correctly cited in the documentation:
Dahl, T.E. 1990. Wetland Losses in the United States, 1780's to 1980's.
U.S.D.I., Fish and Wildlife Service, Washington, D.C., 21 p.
A-l
-------
Map 9. There were big changes in this map. As with Map 7, these changes are difficult
to explain given that the data sources and data categories are identical for both
the 1/8/97 and 4/21/97 versions. Why does the pollution picture improve so
much between the 1/8/97 and 4/21/97 versions?
Map 11. There were big changes in this map, presumably because of the use of an
additional data source (1978 USGS land use data) and different data categories.
Supplemental maps should be included to show both the Census and 1978
USGS land use data. The "Description of the Data Layer" published with the
4/21/97 map is much more informative than the "Scoring for Urban Runoff
Potential" descriptions; the latter is basically a restatement of the map legend.
A-2
-------
U.S. ENVIRONMENTAL PROTECTION AGENCY
SCIENCE ADVISORY BOARD
ECOLOGICAL PROCESSES AND EFFECTS COMMITTEE
May 15, 1997
CHAIR
Dr. Mark A. Harwell, Rosenstiel School of Marine and Atmospheric Science,
University of Miami, Miami, Florida
MEMBERS
Dr. William Adams, Kennecott Utah Copper Corp, Magna, Utah
Dr. Kenneth W. Cummins, South Florida Water Management District, West Palm
Beach, Florida
Dr. Virginia Dale, Environmental Sciences Division, Oak Ridge National Laboratory,
Oak Ridge, Tennessee
Dr. Carol Johnston, Natural Resources Research Institute, Duluth, Minnesota
Dr. Frederick K. Pfaender, Carolina Federation for Environmental Programs,
University of North Carolina, Chapel Hill, North Carolina
Dr. William H. Smith, School of Forestry and Environmental Studies, Yale
University, New Haven, Connecticut
CONSULTANTS
Dr. Kenneth L. Dickson, University of North Texas, Denton, Texas
Dr. Peter Kareiva, Department of Zoology, University of Washington, Seattle, Washington
SCIENCE ADVISORY BOARD STAFF
Ms. Stephanie Sanzone, Designated Federal Official, US EPA, Science Advisory Board
(1400), 401 M Street, SW, Washington, DC 20460
Ms. Wanda R. Fields, Staff Secretary, US EPA, Science Advisory Board (1400F), 401 M
Street, SW, Washington, DC 20460
-------
NOTICE
This report has been written as part of the activities of the Science Advisory Board,
a public advisory group providing extramural scientific information and advice to the
Administrator and other officials of the Environmental Protection Agency. The Board is
structured to provide balanced, expert assessment of scientific matters related to problems
facing the Agency. This report has not been reviewed for approval by the Agency and,
hence, the contents of this report do not necessarily represent the views and policies of
the Environmental Protection Agency, nor of other agencies in the Executive Branch of the
Federal government, nor does mention of trade names or commercial products constitute
a recommendation for use.
-------
DISTRIBUTION LIST
Administrator
Deputy Administrator
Assistant Administrators
Deputy Assistant Administrator for Science, ORD
Director, Office of Science Policy, ORD
Director, Office of Air Quality Planning and Standards, OAR
Director, National Center for Environmental Assessment, ORD, RTP, NC
Director, Office of Air Quality Planning and Standards, OAR, RTP, NC
EPA Regional Administrators
EPA Laboratory Directors
EPA Headquarters Library
EPA Regional Libraries
EPA Laboratory Libraries
Library of Congress
National Technical Information Service
Congressional Research Service
------- |