UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON D.C. 20460
OFFICE OF THE ADMINISTRATOR
SCIENCE ADVISORY BOARD
October 12, 2005
EPA-SAB-ADV-06-001
The Honorable Stephen L. Johnson
Administrator
U.S. Environmental Protection Agency
1200 Pennsylvania Avenue, N.W.
Washington, D.C. 20460
Subject: Advisory on EPA's Regional Vulnerability Assessment Program
Dear Administrator Johnson:
The Environmental Protection Agency's (EPA) Office of Research and Development
requested that the Science Advisory Board (SAB) provide advice on the methodological
approach used in EPA's Regional Vulnerability Assessment (ReVA) Program. The goal of
EPA's ReVA Program is to develop tools and methods to estimate future ecosystem
vulnerability and illustrate trade-offs associated with alternative environmental and economic
policies. The SAB was specifically asked to provide advice on improving the effectiveness of
the ReVA web-based Environmental Decision Toolkit for communicating ecological risk and
condition to risk managers. The enclosed SAB report addresses EPA's charge questions to the
Panel and provides recommendations for improvements in ReVA.
Although ReVA is not yet fully developed, the SAB finds that the objective of developing a
suite of tools to integrate and synthesize environmental data to provide screening level estimates
of ecosystem vulnerability on a regional scale is very important. While doing this presents many
challenges, the SAB believes the ReVA project offers real promise and warrants continued effort
and resources. Once fully developed, documented, and supported with effective user interfaces,
such tools and methods could be of great value to local and regional resource managers for
assessing current and future conditions and making risk management decisions. The SAB
strongly recommends continued support of the efforts of EPA's Office of Research and
Development to develop and refine ReVA.
While it shows considerable promise, before ReVA is ready for application to real problems,
a more systematic effort should be devoted to articulating a clear basis for the choice and
-------
validation of the underlying analytical methods on which it relies. Once this has been done, a
much improved documentation explaining and justifying the choice of those methods will also
be needed so that users can clearly understand ReVA's strengths and limitations. The SAB also
recommends that EPA provide additional documentation on processes for acquiring and
assembling data, quality assurance reviews, spatial data integration, and the statistical tools used
in ReVA.
The SAB underscores the need for EPA to provide additional resources and in-house
expertise to fully develop ReVA and to better leverage outside expertise by working closely with
other government agencies and academic institutions. The SAB looks forward to your
consideration of and response to the enclosed advisory report, and stands ready to offer further
assistance as this effort continues.
Sincerely,
/Signed/ /Signed/
Dr. M. Granger Morgan, Chair Dr. Kenneth Cummins, Chair
Science Advisory Board Regional Vulnerability
Assessment Advisory Panel
Science Advisory Board
-------
NOTICE
This report has been written as part of the activities of the EPA Science Advisory Board, a
public advisory group providing extramural scientific information and advice to the
Administrator and other officials of the Environmental Protection Agency. The Board is
structured to provide balanced, expert assessment of scientific matters related to the problems
facing the Agency. This report has not been reviewed for approval by the Agency and,
hence, the contents of this report do not necessarily represent the views and policies of the
Environmental Protection Agency, nor of other agencies in the Executive Branch of the
Federal government, nor does mention of trade names or commercial products constitute a
recommendation for use. Reports of the EPA Science Advisory Board are posted on the EPA
website at http://www.epa.gov/sab.
in
-------
U.S. Environmental Protection Agency
Science Advisory Board
Regional Vulnerability Assessment Advisory Panel
CHAIR
Dr. Kenneth Cummins, Co-Director, Institute for River Ecosystems, Humboldt State
University, Arcata, CA
MEMBERS
Dr. Cynthia Gilmour, Senior Scientist and Principal Investigator, Smithsonian Environmental
Research Center, Edgewater, MD (Member of the SAB Ecological Processes and Effects
Committee)
Dr. Charles Hawkins, Professor, Department of Aquatic, Watershed, and Earth Resources;
Director, Western Center for Monitoring and Assessment of Freshwater Ecosystems, Utah State
University, Logan, UT (Member of the SAB Ecological Processes and Effects Committee)
Dr. Orie Loucks, President, ICValue, Inc., Oxford, OH
Dr. William Mitsch, Professor, Olentangy River Wetland Research Park, The Ohio State
University, Columbus, OH (Member of the SAB Ecological Processes and Effects Committee)
Dr. Michael C. Newman, Professor of Marine Science, School of Marine Sciences, Virginia
Institute of Marine Science, College of William & Mary, Gloucester Point, VA (Member of the
SAB Ecological Processes and Effects Committee)
Dr. Ganapati Patil, Director, Center for Statistical Ecology and Environmental Statistics, The
Pennsylvania State University, University Park, PA
Dr. Charles Rabeni, Leader, Missouri Cooperative Fish and Wildlife Research Unit, U.S.
Geological Survey, Columbia, MO (Member of the SAB Ecological Processes and Effects
Committee)
Dr. Mark Ridgley, Professor and Chair, Department of Geography, University of Hawaii at
Manoa, Honolulu, HI
Dr. James Sanders, Director, Skidaway Institute of Oceanography, Savannah, GA (Member of
the SAB Ecological Processes and Effects Committee)
Dr. David Stoms, Associate Researcher, Institute for Computational Earth System Science,
University of California at Santa Barbara, Santa Barbara, CA
IV
-------
Mr. Timothy Thompson, Senior Environmental Scientist, Science, Engineering, and the
Environment, LLC, Seattle, WA (Member of the SAB Ecological Processes and Effects
Committee)
SCIENCE ADVISORY BOARD STAFF
Dr. Thomas Armitage, Designated Federal Officer, U.S. Environmental Protection Agency,
Washington, D.C.
-------
TABLE OF CONTENTS
1. EXECUTIVE SUMMARY vii
2. INTRODUCTION 1
3. CHARGE TO THE PANEL 1
4. ADVISORY PROCESS 2
5. RESPONSE TO THE CHARGE QUESTIONS 3
5.1 Question 1 3
5.1.1 Question la 3
5.1.2 Question Ib 7
5.2 Question 2 11
5.2.1 Question 2a 11
5.2.2 Question 2b 14
5.2.3 Question 2c 19
5.3 Questions 19
5.4 Question 4 22
5.4.1 Question 4a 22
5.4.2 Question 4b 23
6. REFERENCES 25
APPENDIX A: SENSITIVITY OF THE CRITICALITY MEASURE IN ReVA 28
VI
-------
1. EXECUTIVE SUMMARY
The Science Advisory Board Regional Vulnerability Assessment Advisory Panel was charged
with providing advice to EPA's Office of Research and Development on the approach used in
the Agency's Regional Vulnerability Assessment (ReVA) Program and on improving the
effectiveness of the web-based ReVA Environmental Decision Toolkit (EDT) for
communicating ecological condition and risk. Geographic information system technologies and
quantitative integration and assessment methods are being developed in ReVA and will be used
to derive future vulnerability estimates that include: syntheses of modeled ecological drivers of
change (i.e., estimated changes in pollution and pollutants, resource extraction, spread of non-
indigenous species, land use change, and climate change) and resulting changes in stressor
patterns. Integrative and visualization tools are being incorporated into ReVA. These tools will
be used to illustrate the trade-offs associated with alternative environmental and economic
policies in the context of dynamic stakeholder values.
It is the opinion of the SAB that the suite of tools in ReVA can be exceptionally useful to
local and regional resource managers for assessments of current and future regional conditions.
In ReVA, spatially explicit data are coupled into a statistical platform (S-Plus) to facilitate rapid
reanalysis and display of data. This capability can be very useful in answering the range of
questions ReVA may address. The SAB notes, however, that there are a number of limitations
associated with the methodological approaches used in ReVA and that the application of ReVA
could be substantially improved by providing additional documentation of the underlying
processes. The SAB urges that future development of the ReVA be consistent with the
principles embodied in EPA's Guidance on the Development, Evaluation, and Application of
Regulatory Environmental Models (U.S. EPA Office of Science Policy, 2003). The SAB
strongly encourages EPA to continue efforts to develop ReVA and provides specific comments
and recommendations in response to EPA's charge questions.
Question 1. Strengths and Limitations of the ReVA Approach
ReVA is intended to provide an overview of current and future regional conditions. ReVA may
also serve as apriority setting tool to target areas for more focused risk assessments of specific
problems. Please comment on the strengths and limitations of the ReVA approach as it applies
to these uses.
Overall, the SAB finds that the major strengths of ReVA are in the areas of data
integration and visualization, particularly in the development of tools in these areas for
resource managers and planners.
The SAB finds that ReVA provides a very promising methodology for compiling existing
and other disparate spatially integrated data sets in a cohesive way for a region. ReVA
also provides new methods to synthesize existing data in a spatial framework.
The SAB acknowledges that development of the ReVA has been an extraordinary and
elegant effort by a dedicated and highly skilled team. The SAB also notes that ReVA is
not yet fully developed. The SAB finds, however, that a good deal of knowledge about
Vll
-------
what presently constitutes ReVA resides solely with the developers. Outside reviewers
cannot discern what ReVA is from information that is currently available. The SAB feels
strongly that ReVA could be substantially improved by providing additional
documentation of the underlying processes and a framework and indicators to assess
ecosystem vulnerability.
The SAB recommends that, in order to improve ReVA as a tool for providing an
overview of regional conditions, EPA should develop: 1) overarching conceptual models
for ReVA documenting what ReVA is, the main objectives of ReVA, and the main
questions being asked in ReVA; and 2) clear basic documentation on what constitutes the
ReVA methodology, including the underlying processes for acquiring and assembling
data, quality assurance reviews, and spatial data integration.
The SAB finds that, as presently described, ReVA has limited overall use as a priority
setting tool to target areas for more focused risk assessment. However, ReVA can
provide useful information for this purpose. This is further discussed in Section 5.1.2
below.
The SAB recommends that EPA use caution when ReVA is applied to aggregate
individual stressors into a single map or value. While such aggregations can provide
useful information to assist in identifying areas for more focused risk assessment, the
underlying statistical methods for aggregating and/or integrating multiple stressors into a
single value are still in their infancy. Use of these methods may lead to erroneous
interpretations. This is further discussed in Section 5.1.2 below.
The SAB finds that ReVA's focus on moderate to high probability/lower incremental
impact stressors that change gradually over time precludes evaluation of important
regional differences in ecological qualities such as keystone habitat. At finer scales, such
issues emerge as extremely important.
The SAB finds that a good future application of ReVA would be to evaluate low
probability/rapid or "cusp-driven" changes with highly adverse consequences. Examples
might include: a sudden shift in agricultural practice to widespread use of genetically
engineered crops with reductions in heavy pesticide applications, pulses of organo-
phosphorus pesticides into streams in a small county, sudden atmospheric releases of
potentially acutely toxic chemicals, and changes in policy relative to timber harvesting.
Such events would seem to be more relevant at smaller scale applications where change
can be more rapid and pervasive, and would be worthy of additional ReVA research
efforts in the future.
Question 2. Effectiveness of the Web-based ReVA Environmental Decision Toolkit
Please comment on the effectiveness of the web-based ReVA Environmental Decision Toolkit
(EDT) in communicating ecological condition and vulnerability to decision-makers at regional
to local scales. Please provide input as to the level of analytical capability needed in ReVA for
intended audiences as well as approaches to presenting available information and uncertainty.
Vlll
-------
EPA provided two data sets to demonstrate the ReVA Web-based Environmental Decision
Toolkit (EDT). The Sustainable Environment for Quality of Life (SEQL) data set in ReVA
contained information obtained from counties in the Charlotte, North Carolina region. The Mid-
Atlantic Regional Assessment data set contained information from eight states in the Mid-
Atlantic region.
The SAB finds that EPA has used the example data sets to demonstrate excellent
examples of ReVA applications for very limited regions. The SAB finds that the spatial
development maps in the EDT use color effectively and that vulnerability is well
described in the EDT. However, the SAB finds that ecological condition is not as well
described because temporal dynamics have not been captured. This could be addressed
by linking the data layers to models that enable consideration of temporal information.
The SAB recognizes that the EDT is still under development. However, given the lack of
documentation for the EDT, the SAB recommends that EPA compile and publish a
separate document on compilation, organization, extrapolation, and types of data/layers
in the ReVA EDT. An example format that could be used to develop such a document is
Table ES-2 in the executive summary of the SAB publication, A Framework for
Assessing and Reporting on Ecological Condition (EPA Science Advisory Board, 2002).
It would also be helpful to include statements describing quality of, and confidence in,
the data.
The SAB recommends that more resource efforts be expended toward developing
mechanistic models to be coupled with the spatially explicit data in ReVA. This is
potentially ReVA's most powerful application. The SAB also recommends that the
models developed and used in ReVA be listed on the ReVA web site (e.g., watershed
models and ozone model). The SAB also recommends that EPA explore the potential
coupling of ReVA with dose/response models.
The SAB recommends that the strengths and utility of the integration methods in ReVA
be tested using a relatively limited set of environmental and landscape data. The SAB
finds that a back-cast demonstration of ReVA in a simpler system would be an effective
way to illustrate the utility and potential power of the methods and to answer focused
questions. The SAB recommends that EPA allocate additional resources to the ReVA
program to: 1) run back-casts to conduct field validation of the integration methods; 2)
apply the integration methods using a more limited number of land/resource variables;
and 3) explore sensitivity and uncertainty in ReVA with back-casts.
The SAB finds that the elements of the ReVA EDT have been assembled into a web-
based application that can be applied by regional and local decision makers to conduct
scenario analysis. By scenario analysis the SAB means the articulation of future contexts
which could plausibly (not necessarily probably) be defined by variations in present-day
natural and social processes that together could lead to ecological vulnerability and
management priorities different from those likely to occur under a continuation of
present-day patterns and processes (Ringland, 1998; Schwartz, 1991). The SAB
IX
-------
recommends that the developers of ReVA carefully qualify the limitations of scenario
analysis as currently conducted in ReVA, and distinguish it from forecasting. In this
regard, the SAB notes that scenario analysis does not prescribe significance and is not
probabilistic or predictive in any mechanistic fashion. Scenario analysis is simply
application of a set of conditions observed in the past to project a plausible future case.
The SAB finds that ReVA in this context is therefore best suited for use as a screening
tool.
The SAB finds that, while developing web-based applications is a laudable goal, the
computing power needed to handle and process information is likely to be too great to
practically allow such web-based applications in the near future. The SAB therefore
recommends that EPA include strong cautions against using the interface tool for actual
decision making if the application cannot be practically applied.
The eleven integration and assessment methods in ReVA have been developed from a
vast literature encompassing multiple disciplines, software, and decision tools. The SAB
finds that these methods offer great promise for further development and future use. The
SAB recommends, however, that additional documentation be provided to support the
ReVA methods that have been adopted for data integration, landscape modeling, and
integrative assessments. The SAB recommends that a methodology document and user's
manual be prepared as an integral part of the EDT to address these issues. A precise
description of each integration and assessment method should be included in the
document. Basic documentation of the ReVA methodology, as well as metadata for the
entire methodology, should also be included. The user's manual should provide
information needed to understand how much uncertainty is associated with the EDT
presentation of ecosystem vulnerability, and guidance to assist users in selecting
methods. It would be useful to include a table of assessment questions and integration
methods in the document with an indication of which methods (or suite of methods) are
most appropriate for answering the questions.
Question 3. Usefulness of the ReVA Approach to Decision-makers
Please comment on the usefulness of the ReVA approach to decision makers in allowing them to
see the overall consequences of future development, and mitigation, conservation, and
restoration activities.
The SAB finds that the usefulness of the ReVA approach to decision makers could be
improved by: (1) explicitly acknowledging the differences between forecasting and
scenario analyses, (2) continuing efforts to improve or enhance the ecological conditions
database, (3) validation and/or improvement of the ecological condition integration
methods, (4) incorporation of commercially-available decision-assisting software, and (5)
recognition within ReVA that ecological vulnerability decisions must also consider
equity, efficiency and effectiveness. Effectiveness means getting the job done (e.g.,
reducing vulnerability) regardless of cost; efficiency refers to output divided by input
(e.g., benefit-cost ratio) or achieving a given level of vulnerability reduction at the lowest
possible resource use or cost, and hence does consider the cost (e.g., use of various
-------
resources) involved; equity is some notion of fairness. If ReVA is not to be used in
priority setting it need not consider equity, efficiency, and effectiveness. However, to the
degree that ReVA (or any other entity or tool) concerns itself with priority setting, it
should consider equity, efficiency, and effectiveness. The SAB recommends that EPA
explore adding tools and data layers to ReVA in these areas to make it more useful in the
decision making process.
Question 4. Issues Associated with use of ReVA at Multiple Scales and Future Research
Priorities
Please provide input on issues encountered as the information and approaches in ReVA are used
at finer scales. Please also provide input on future ReVA research priorities and alternative
applications of ReVA methods for decision making at multiple scales.
The SAB finds that as ReVA is applied at finer scales it is likely to be used by a large
number of decision-makers with varying levels of scientific and technical expertise. In
order to further develop ReVA for use at finer scales, the SAB encourages EPA to
provide additional information documenting and explaining issues related to the choice of
methods and indicators, and to provide exemplars where available.
The SAB has identified a number of research priorities and applications to support further
development of ReVA methods for decision making at multiple scales. 1) Research is
needed to provide information about the minimum amount of data needed for advice and
guidance used in decision making. 2) In addition to providing information about the
vulnerability of geographic areas, it would be useful to develop ReVA tools to identify
geographic areas of highest "value." However, as discussed in Section 5.4.2, this would
be a complex task. 3) Integration methods, applications, and futures tools in ReVA
should be validated. 4) ReVA should contain data sets describing simpler scenarios that
span resource issues. 5) Analyses should be conducted to determine whether ReVA is
providing data describing the critical parameters for assessing vulnerability. 6) Users
should be provided information about the confidence in data used for projections. 7)
Spatial problems (scale effects) associated with the ReVA map representations should be
resolved.
In summary, the SAB strongly recommends continued support of the efforts of EPA's
Office of Research and Development to develop ReVA. The SAB finds that the ReVA
methods and web-based Environmental Decision Toolkit hold great promise as tools that can
assist local and regional resource managers in assessing current and future conditions.
However, the utility of ReVA could be better supported by providing additional
documentation of the underlying processes. The SAB encourages EPA to continue
developing ReVA, and to provide documentation on: what constitutes ReVA, the framework
and indicators for assessing ecological condition in ReVA, and the conceptual models
underlying ReVA. A methodology document and user's manual should also be developed
for the ReVA Environmental Decision Toolkit. The user's manual should document the
ReVA statistical tools in a manner that is clear and accurate with analytical and empirical
supporting evidence.
XI
-------
Advisory on EPA's Regional Vulnerability Assessment
Methods for Multi-Scale Decision Making
An Advisory by the Science Advisory Board
Regional Vulnerability Assessment Advisory Panel of the
Ecological Processes and Effects Committee
2. INTRODUCTION
This report transmits the advice of the U.S. Environmental Protection Agency (EPA) Science
Advisory Board (SAB) Regional Vulnerability Assessment Advisory Panel of the Ecological
Processes and Effects Committee. The Panel met on October 26-27, 2004 to provide advice to
EPA's Office of Research and Development on Regional Vulnerability Assessment Methods for
Multi-Scale Decision Making. EPA's Office of Research and Development is developing
approaches for comprehensive regional-scale environmental assessments that can inform
decision-makers at multiple scales about current and anticipated environmental conditions and
vulnerabilities. A suite of predictive tools and methods is being incorporated into the Regional
Vulnerability Assessment Environmental Decision Toolkit to enable decision-makers to
determine the magnitude, extent, and distribution of current and anticipated environmental
vulnerabilities within a geographic region.
In the context of EPA's Regional Vulnerability Assessment (ReVA) Program, environmental
vulnerabilities have been defined as risks of serious degradation of ecological goods and services
that are valued by society. Spatial data are used in ReVA to depict: 1) the current patterns of
condition and distribution of resources and human demographics in a region, 2) variability in the
sensitivity of resources and human populations to various stresses in a region, and 3) the
estimated spatial distribution of stressors in a region. Geographic information system
technologies and quantitative integration and assessment methods are being developed for use in
ReVA to derive future vulnerability estimates that include syntheses of modeled ecological
drivers of change (i.e., estimated changes in pollution and pollutants, resource extraction, spread
of non-indigenous species, land use change, and climate change) and resulting changes in
stressor patterns. Integrative and visualization tools are being developed and incorporated into
ReVA. These tools will be used to illustrate the trade-offs associated with alternative
environmental and economic policies in the context of dynamic stakeholder values. The
following two regional case examples were provided to the panel to illustrate the application of
ReVA methods and tools: 1) an assessment of data from the Mid-Atlantic region of the U.S., and
2) an assessment of data for decision-makers in a 15-county region around Charlotte, North
Carolina.
3. CHARGE TO THE PANEL
EPA's Office of Research and Development requested advice from the Science Advisory
Board on the approach used in ReVA, and on improving the effectiveness of the ReVA
integration toolkit (the ReVA web-based Environmental Decision Toolkit or EDT) for
communicating current and future condition and risk to clients and users. Specifically, EPA
-------
sought advice regarding the following questions:
Question 1. Strengths and Limitations of the ReVA Approach
ReVA is intended to provide an overview of current and future regional conditions. ReVA may
also serve as a priority setting tool to target areas for more focused risk assessments of specific
problems. Please comment on the strengths and limitations of the ReVA approach as it applies
to these uses.
Question 2. Effectiveness of the Web-based ReVA Environmental Decision Toolkit
Please comment on the effectiveness of the web-based ReVA Environmental Decision Toolkit
(EDT) in communicating ecological condition and vulnerability to decision-makers at regional to
local scales. Please provide input as to the level of analytical capability needed in ReVA for
intended audiences as well as approaches to presenting available information and uncertainty.
Question 3. Usefulness of the ReVA Approach to Decision-makers
Please comment on the usefulness of the ReVA approach to decision makers in allowing
them to see the overall consequences of future development, and mitigation, conservation, and
restoration activities.
Question 4. Issues Associated with use of ReVA at Multiple Scales and Future Research
Priorities
Please provide input on issues encountered as the information and approaches in ReVA are used
at finer scales. Please also provide input on future ReVA research priorities and alternative
applications of ReVA methods for decision making at multiple scales.
4. ADVISORY PROCESS
To establish the ReVA Advisory Panel, the EPA Science Advisory Board Staff Office
published a Federal Register notice requesting nominations to augment the expertise of members
on the SAB's Ecological Processes and Effects Committee (EPEC). The SAB Staff Office then
identified a subset of nominees for consideration as panelists. The final panel was selected after
requesting public comments on the nominees and further evaluating them against EPA Science
Advisory Board selection criteria. The members of the advisory panel included ecologists on the
Ecological Processes and Effects committee as well as additional members with expertise in
decision science and environmental decision making, analysis of land use change, the use of
geographic information system technology to analyze environmental stressors and effects, and
statistics.
The advisory was conducted in a two-day face-to-face public meeting. At the public meeting,
the advisory panel heard presentations from EPA's Office of Research and Development on: 1)
an overview of the ReVA Program, 2) spatial data and landscape models in ReVA, 3) integration
methods in ReVA, 4) future vulnerability estimates, and 5) the ReVA integration toolkit for
-------
communicating risk and uncertainty to users and clients. The panel also heard presentations
addressing application of ReVA tools and methods to decision making. These presentations
were delivered by: EPA's Office of Research and Development, EPA's Region 3 Office, and the
Land Use and Environmental Planning Division, University of North Carolina - Charlotte Urban
Institute. The panel then deliberated on each of the charge questions and developed the final
SAB report.
5. RESPONSE TO THE CHARGE QUESTIONS
The Panel Chair decided that the SAB panel could most effectively respond to EPA's charge
questions if the questions were considered in subparts. Responses to charge question one are
therefore provided in two subparts (la and Ib), responses to charge question two are provided in
three subparts (2a, 2b, and 2c), the response to charge question three is provided in one part, and
responses to charge question four are provided in two subparts (4a and 4b).
5.1 Question 1. ReVA is intended to provide an overview of current and future regional
conditions. ReVA may also serve as a priority setting tool to target areas for more
focused risk assessments of specific problems. Please comment on the strengths and
limitations of the ReVA approach as it applies to these uses.
5.1.1 Question la. Comment on the strengths and weaknesses of ReVA as a tool to
provide an overview of current and future regional conditions.
It is the opinion of the SAB that the suite of tools in ReVA can be exceptionally useful to
local and regional resource managers for assessments of current and future regional conditions.
In ReVA, spatially explicit data are coupled into a statistical platform (S-Plus) to facilitate rapid
reanalysis and display of data. This capability can be very useful in answering the range of
questions ReVA intends to address. The SAB chose to explore limitations (as opposed to
"weaknesses") of ReVA, and found that the lack of documentation on what constitutes ReVA,
and the lack of a framework and adequate indicators to assess ecological condition are the most
important limitations to application of ReVA.
Strengths of ReVA as a Tool to Provide an Overview of Current and Future Regional Conditions
Overall, the SAB finds that the major strengths of ReVA are in the areas of data integration
and visualization, particularly in the development of tools in these areas for resource managers
and planners. The SAB notes the following major strengths of ReVA:
ReVA provides a very promising methodology for compiling existing (e.g., Mid-Atlantic
Integrated Assessment [MAIA]) and other disparate spatially integrated data sets in a
cohesive way for a region.
ReVA provides new methods to synthesize existing data in a spatial framework.
Integration approaches for multivariate data are being developed in ReVA.
ReVA offers the power of those simple summary indicators, combined with spatial
visualization, for communicating the concept of "vulnerability" to the lay public.
Strong emphasis has been placed on integrating ReVA with "customer" needs.
-------
An interactive interface is being developed to enable the use of ReVA tools by resource
managers and planners.
Limitations of ReVA as a Tool to Provide an Overview of Regional Conditions
The SAB feels strongly that ReVA is limited by a lack of documentation of the underlying
methodology, and especially the lack of a framework and indicators to assess ecological
condition. While some of these factors are outside of the ReVA developer's control, the SAB
finds that the power of the ReVA approach is limited specifically by:
The lack of basic documentation of the ReVA framework and methodology.
The lack of availability of ecosystem-specific data.
The lack of good indicators of ecological condition.
The complete lack of calibration, verification and sensitivity demonstrations on the
ReVA summary indicator models.
Inherent weaknesses in using solely spatial data to make predictions. In this regard,
ecological condition is not presently well described in ReVA because temporal dynamics
have not been captured in ReVA. This could be addressed by linking data layers to
models that enable the consideration of temporal information.
Oversimplification of the complex relationships among stressors and resources to predict
"vulnerability."
The SAB is aware that ReVA staff and consultants continue to work on resolving issues, and
believes that recommendations in this report will enhance their efforts
Recommendations to Improve ReVA as a Tool for Providing an Overview of Regional Conditions
In order to improve ReVA as a tool for providing an overview of regional conditions, the
SAB recommends and encourages the ReVA program develop the following:
Conceptual models for ReVA. As discussed in the response to charge question 2a,
different "levels" of conceptual models should be provided. An overarching conceptual
model should describe what ReVA is and the main objectives and questions being asked
in ReVA. Other levels of conceptual models should describe the processes in ReVA,
including relationships between data sets used to assess ecological condition and
vulnerability. It may be useful to present the conceptual models in box and arrow
diagrams.
Clear basic documentation on what constitutes the ReVA methodology, including the
underlying processes for acquiring and assembling data, quality assurance reviews, and
spatial data integration.
Documentation on the development and application of the summary indicators in ReVA,
including external verification of indicator applicability, sensitivity, and sources of
uncertainties.
A process to evaluate the performance of indicators developed for assessing ecological
condition.
Increased use of response measures and ecological endpoints.
-------
Use of more sophisticated measures than species abundance to assess condition (e.g.,
Index of Biotic Integrity).
Common goals for ecological valuation and assessment within EPA.
Increased transparency in providing information on data sets used in ReVA. In this
regard, the SAB recommends that EPA provide information documenting the key data
sets needed to evaluate ecosystem vulnerability and how the lack of available data may
affect such evaluations.
Discussion of Strengths and Limitations of the ReVA Process and Toolkit
In seeking a basic understanding of what comprises ReVA and the ReVA "toolkit," the SAB
discussed questions relative to: 1) whether ReVA is a tool ready for immediate implementation;
2) whether ReVA is a methodology for assembling data and information into a format against
which local or regional decision processes can be developed for specific questions; and 3)
whether the broad definitions, data sets, futures projection methods, and statistical integration
methods used to develop a single index of "vulnerability" are appropriate for their intended use.
Much of this discussion focused on whether ReVA processes and tools were sufficiently
documented and transparent.
Transparency of ReVA
While the SAB acknowledges that development of the ReVA has been an extraordinary and
elegant effort by a dedicated and highly skilled team, it is apparent that a good deal of knowledge
about what constitutes ReVA resides solely with the developers. Outside reviewers cannot
discern what ReVA is from information that is currently available. In the parlance of EPA's
Risk Assessment Paradigm (U.S. EPA, 1984), ReVA is not transparent. The SAB notes, based
upon its working understanding of the ReVA Program, that EPA has completed, or is working
on, the following ReVA activities:
Developed clearly articulated goals and objectives as represented by the research strategy
(Smith et al., 2000);
Compiled an extensive set of spatially-explicit data on the Mid-Atlantic from several
sources as a pilot set of information from which to develop and test integration and
vulnerability methods;
Developed and applied a set of quality assurance, data and spatial normalization
procedures, and compiled the data into a single GIS-database;
For certain data, extrapolated limited information sets to broader regional scales using
commonly accepted statistical interpolation methods for geographic data;
Demonstrated the utility of coupling the spatially-explicit data sets with mechanistic
models that provide a method for forecasting changes in certain environmental
parameters;
Developed novel and potentially applicable statistical methods to integrate a divergent set
of environmental parameters into a single assessment of "vulnerability;"
Developed web-based tools to explain what ReVA is and demonstrate how the data sets,
interpolations, and integration methods can be combined to help make environmental
decisions; and
-------
Has begun to develop specific regional decision-assisting tools for a range of clients
including EPA program and regional offices, sister federal agencies, and state and local
agencies.
From this understanding, the SAB believes that ReVA is a methodology1. The strength of
ReVA lies in the standards for assembling the data sets, quality assurance reviews, and methods
for interpolating limited data with an eye to understanding and addressing specific regional
questions. ReVA's greatest opportunity lies in developing the application and integration
methods to address specific problems in specific regions. Having said that, ReVA suffers from
not having a single source document that articulates what it is, and the specific procedures
followed to compile data, provide quality review, and apply these data. The SAB explicitly
recommends that EPA develop and make available to the public and ReVA clients a concisely
written description of the ReVA methodology, and the tools that have been and may be
developed with ReVA.
The ReVA "Toolkit"
From the understanding that ReVA is a methodology, the SAB has sought to distinguish
between the methodology and what has been termed the ReVA "toolkit." The ReVA developers
used the terms "process" and "toolkit" interchangeably; this injected ambiguity into the SAB's
understanding. The SAB recommends that "toolkit" should be reserved to mean the decision-
assisting elements nested within the overall ReVA methodology and presented on the ReVA
websites. In the response to charge question two, the SAB identified strengths and limitations of
the elements that comprise the "toolkit," and has provided recommendations for further
development of those "toolkit" elements.
The SAB believes it is imperative that when EPA is developing applications for the ReVA
methodology, the developers must make clear the difference between "forecasting" and
"scenario analysis" to project future vulnerability. While this is discussed more fully in the
response to charge question three, the SAB intends "forecasting" to mean application of well
defined, calibrated and validated mechanistic models. Mechanistic models are applied using the
baseline spatial data as inputs to the model, with outputs as changes over time with quantifiable
uncertainties. An example of a forecast is the application of the "PM2.5" model to project future
ozone levels for the Clear Skies Initiative. Scenario analysis is the exploration of potential
changes in the overall landscape using the baseline spatial data coupled with the good
visualization tools presented with geographic information technology. For example, if
populations grow by 20% and the impacts associated with population growth are known, a
scenario analysis can be conducted. The planned use of ReVA in the Sustainable Environment
for Quality of Life (SEQL) program in Charlotte North Carolina is an example of a scenario
analysis.
1 The ReVA methodology may be viewed as the framework and set of procedures used to apply models and
quantification tools in the ReVA Environmental Decision Toolkit to the data in order to evaluate ecosystem
condition and vulnerability.
-------
5.1.2 Question Ib. Comment on the strengths and weaknesses of ReVA as a priority
setting tool to target areas for more focused risk assessment.
The SAB finds that, as presently described, ReVA is not well suited for use as a priority -
setting tool to target areas for more focused risk assessment. The strengths and limitations of
ReVA for this use are discussed below. The SAB notes that EPA may wish to consider
developing ReVA as a tool for measuring or characterizing vulnerability and/or helping clients to
conceptualize and measure vulnerability well for their purposes and, as discussed below, to
provide information that can be of use in priority-setting.
Strengths of ReVA as a Priority Setting Tool to Target Areas for More Focused Risk Assessment
As noted above, ReVA's strengths include: its value as a tool for presentation of complex
information and integration of multi-variate data, the unique and promising integration tools in
ReVA, and the ability ReVA provides to conduct exploratory analyses with the data layers and
weighting factors coupled in the toolkit. Stressor/resource overlays are a powerful application of
spatially explicit data and may be used, with other information, to assist in priority setting and
targeting areas for more focused risk assessments of specific problems. As discussed below,
ReVA presently has limited overall use as a priority setting tool, but it can provide useful
information for the arduous task of priority setting. Within the same set of strengths and
limitations described previously, ReVA has the following additional strengths for use in risk
assessment:
Within the ReVA layers, the impacts of individual stressors can be assessed and
evaluated using GIS-analysis tools and presentations. The power of GIS is the overlays
that can be generated and viewed for multiple stressors.
Overlays of multiple stressors can be used to help target geographic areas where it may
be appropriate to conduct focused risk assessment and/or restoration activities.
Mechanistic models can be coupled to the baseline GIS-data to project future risks and
uncertainties.
ReVA enables relatively easy risk communication with the visual display of complex
spatial information.
Limitations of ReVA as a Priority Setting Tool to Target Areas for More Focused Risk
Assessment
The same limitations noted previously are applicable to ReVA's potential use in risk
assessment. The SAB also notes that EPA should use caution when ReVA is applied to
aggregate individual stressors into a single map or value. While such aggregations can provide
useful information to assist in identifying areas for more focused risk assessment, the underlying
statistical methods for aggregating and/or integrating multiple stressors into a single value are
still in their infancy. Use of these methods may lead to erroneous interpretations. Until issues
are resolved the SAB feels that ReVA will have limited overall use as a priority-setting tool.
The SAB notes that the Stressor-Resource Matrix Analysis in ReVA is based on
summing correlation coefficients. Summing these coefficients has little meaning and is
-------
misleading. However, the correlation coefficient does provide an indication of indirect
cause-effect links. The SAB suggests that EPA may wish to consider using graph-
theoretic approaches that utilize adjacency and reachability matrices (Bodini, Giavelli
and Rossi, 1994; Chorley and Kennedy, 1971; Craig, 1981; Gould, 1986; Hage and
Harary, 1983; Harary, Norman and Cartwright, 1965; Levins, 1974; Maruyama, 1963;
Maruyama, 1968; Phillips, 1993; Puccia and Levins, 1985; Puccia and Levins, 1991;
Roberts, 1976; Roberts, 1978; Slingerland, 1981). These are easy to program and
explain, do not require quantitative (ratio or interval-level) data, and are found in almost
all introductory texts on graph theory. The outcome will indicate the number of nth
order paths leading from one variable (cause) to another (effect). Since this approach
also identifies the variables and phenomena involved, it is more useful for management
and policy purposes than correlation. However, it will be necessary to provide expert
judgment to set up the original adjacency matrices. Expert judgment may be provided
by the developers or users of ReVA. However, if the users provide expert judgment,
their sense of ownership will increase, and their understanding will likely be greater as
well. The SAB recommends that EPA link graph theory with the notion of stability and
instability, since the latter can be viewed as a dimension or manifestation of
vulnerability. In particular, the SAB recommends that EPA should look into pulse
stability and loop analysis. Loop analysis and the theory of pulse processes are two
methods that use only the information portrayed in signed digraphs to enable inferences
about system stability. Pulse processes, the less complicated of the two, is treated fully
in Roberts (1976). Loop analysis was introduced by Levins (1974) and popularized by
Puccia and Levins (1985). A chapter-length overview is provided in Puccia and Levins
(1991), and examples of applications are available in Bodini, Giavelli, and Rossi (1994)
and Slingerland (1981).
The concept of "valued resources" in ReVA is simplistic. It appears to be defined
without respect to people and/or their need for or interest in the "resource" (i.e., in
disregard of the demand for the resource and its constituent factors such as accessibility).
The value of resources appears to be assessed only with respect to the "resources in
watersheds," yet the concept of resources as something of value to people or society
needs to be addressed.
As illustrated in the following three expressions, considerable differences may exist in
the possible conceptualization of risk, vulnerability, and related factors:
a) Vulnerability = (Stressors) X (Resources) This represents the ReVA approach.
b) Risk to Watershed = (Probability of Event, Situation, etc.) X (Damage) X
(Vulnerability of the Watershed)
c) Risk = (Probability) X (Damage) X (Trust) X (Liability) X (Consent)
The second two expressions clearly suggest that: society may wish to prioritize actions as
they affect risk and not vulnerability, and that those who must prioritize actions will face
multiple, conflicting objectives. These objectives are determined by factors such as
which risks to minimize or mitigate and which aspects of risk to minimize (e.g., expected
risk, worst-case risk, and variance or semi-variance). The SAB notes that the ReVA team
-------
cannot be expected to know what the objectives will be, or how the decision-makers and
stakeholders will wish to prioritize them. This limits ReVA's overall usefulness as a
priority-setting tool.
Priority setting is difficult because "vulnerability" encompasses many different
dimensions and is related to a host of concepts that are poorly defined in any widely
accepted way (e.g., stability, resilience, resistance, elasticity, robustness, viability,
ecological condition, etc.). Vulnerability is ambiguous enough to often be left out of
policymaking. Vulnerability under differing contextual environments, under cumulative
effects and impacts, and in the light of conflicting expert opinion is only rarely addressed
in a coherent way. However, the SAB notes that if ReVA were viewed more as an expert
system than as an education/facilitation tool, some selected effects could always be
evaluated regardless of whether they were identified as important by different users.
Use of ReVA to Target Areas for More Focused Risk Assessment at Different Temporal and
Spatial Scales
The response to charge question 4a below discusses issues associated with application of
ReVA at fine scales. The SAB also notes that as ReVA evolves, EPA should consider
addressing the following issues encountered when risk assessments are conducted at different
temporal and spatial scales. More focused (local scale or shorter times) risk assessments are
more likely to have relatively abrupt, intense, and less incremental stressor scenarios than larger
regional studies. ReVA is presently structured to be applied in assessments of large scope (i.e.,
regional-level assessments). All areas needing closer scrutiny may not be identified when ReVA
is initially used to target areas for further study. This is because factors applied in ReVA to drive
the identification of vulnerabilities at coarse scales are not as well defined at fine scales (e.g.,
factors such as percent forest cover, percent agricultural land cover on slopes, and non-native
species distribution). As ReVA is used at finer scales, these issues will become important.
ReVA's focus tends toward moderate to high probability/lower incremental impact stressors
that gradually change through time. This precludes evaluation of important regional differences
in ecological qualities such as keystone habitat. For example, the flatwoods of the Carolinas and
Georgia contain small features (Carolina Bays) that are important beyond their physical size to
determining biodiversity in an area. At fine scales, such issues emerge as extremely important.
A good, future application of ReVA would be to evaluate low probability or rapid changes
with highly adverse consequences. Examples might include: a shift in agricultural practice
toward increasing use of genetically engineered crops to reduce the use of pesticides, pulses of
organo-phosphorus pesticides into streams in a small county, sudden atmospheric releases of
potentially acutely toxic chemicals, and changes in policy relative to timber harvesting. Such
events would seem to be more relevant at smaller scale applications where change can be more
rapid and pervasive, and would be worthy of additional ReVA research efforts in the future.
The SAB also notes that the ReVA approach as presented focuses on watersheds and requires
the fitting of data that do not blend into this context seamlessly (e.g., air pollutants that distribute
in airsheds or ecological entities that conform to ecoregions or other spatial units). Economic,
-------
infrastructure, and demographic information do not conform to the watershed context. This
causes a certain level of difficulty in vulnerability assessment or decision making. With more
focused studies, the watershed context may be more relevant yet other larger scale issues
involving airsheds, human demographics or economics may simply become ambient
"background."
As the scale of the application becomes finer, the ratio of partially informed to fully informed
people involved in applying ReVA will increase. As the focus of vulnerability assessments
changes from a broad to narrow focus, it will also be necessary to involve different groups of
people in the assessment. This will place a heavy burden on the participants in the assessment
and those who must coordinate the participation of others in the assessment. The SAB notes that
ReVA does not presently contain much specific guidance for application of methods and tools.
The SAB notes that an important future consideration for the developers of ReVA is the
benefit of making assessment tools available to skilled and knowledgeable professionals versus
the inherent dangers associated with making "decision tools" available to less knowledgeable
public or private groups. One can assume that fine scale applications of ReVA will be
undertaken with less input from diverse professionals and with fewer resources. Also, as scales
change in assessment, the participants and concerns also shift. This means that local
professionals must address different sets of concerns in order to effectively use ReVA to identify
areas for more focused risk assessments. Use of professional or best judgment is central in many
places throughout the ReVA approach and implementation of the associated web tool. However,
ReVA provides minimal guidance about how to approach this aspect of the methodology. The
SAB notes that this is unfortunate because, in the absence of such guidance and presence of so
many options, the cumulative application of the ReVA by diverse, smaller groups may result in a
chronic degree of discord. The SAB suggests that straightforward Bayesian techniques such as
Bayesian belief networks could be incorporated into ReVA to help fill this gap and provide some
bases for the professional judgment activities. Application of Bayesian belief networks in
systems that interact directly with human users, such as decision support systems, requires
effective user interfaces. The Bayesian Belief Network exploits probability theory to provide a
single framework for supporting multiple calculations and communications. It also allows for
unbiased inspection and interrogation for a wide range of observers. Relevant examples and
information concerning application of Bayesian belief networks are available in the literature
(Borsuk, Stow, and Reckhow, 2003; Druzdel, 1996; Hukkinen, 1993; Probability Theory and
Bayesian Belief Nets, 2005; Varis and Kuikka, 1989). While ReVA in this connection is not
expected to be a decision making tool, it may be developed as an interface tool, becoming more
useful in the decision making process. As noted previously, ReVA presently has limited overall
use as a priority-setting tool, but it may provide information and tools that can be of use in this
process. Examples may be seen in the literature discussing such methods and tools inclusive of
partially ordered sets and Hasse diagrams (Patil and Taillie, 2004a).
The SAB also notes that priority setting, if done properly, should be tailored to the following
information:
The kind of input information available. The measurement scale (categorical, ordinal,
interval, ratio) of the data and the expressions of preferences should be considered.
10
-------
The kind of output needed. The output needed might include a complete ranking (e.g.,
best to worst); an incomplete ranking (e.g., acceptable sites versus unacceptable ones,
sites needing attention versus those that do not, and the best site out of candidate sites
considered); and ratio-level weights (e.g., for resource allocation).
The level and kind of uncertainty involved with the input and output information (e.g.,
40% chance that site A is the best, and 60% that it is second-best).
5.2 Question 2. Please comment on the effectiveness of the web-based ReVA
Environmental Decision Toolkit (EDT) in communicating ecological condition and
vulnerability to decision-makers at regional to local scales. Please provide input as
to the level of analytical capability needed in ReVA for intended audiences as well as
approaches to presenting available information and uncertainty.
5.2.1 Question 2a. Comment on the effectiveness of the ReVA Environmental Decision
Toolkit (EDT) in communicating ecological condition and vulnerability to decision
makers.
The SAB reviewed three different versions of the web-based EDT (the public, client, and
research versions). These versions of the EDT are on different websites in various stages of
development. The SAB found that it was somewhat difficult to follow pathways on different
websites to evaluate the EDT. Two example data sets were provided by EPA to demonstrate the
EDT. The Sustainable Environment for Quality of Life (SEQL) data set contained information
obtained from counties in the Charlotte, North Carolina region. The Mid-Atlantic Regional
Assessment data set contained information from eight states in the Mid-Atlantic region. Both of
these data sets were used to provide excellent examples of ReVA applications for specific
regions representing different spatial scales. The SAB notes that the spatial development maps
in the EDT use color effectively. Vulnerability is well described in the EDT, but ecological
condition is not as well described because temporal dynamics have not been captured. This
could be addressed by linking the data layers to models that enable the consideration of temporal
information.
The SAB recognizes that the EDT is still under development. However, given information
that is currently available, the SAB notes the following concerns about the effectiveness of the
EDT in communicating ecological condition and vulnerability to decision-makers. Most of these
concerns focus on uncertainty and the lack of available documentation for the EDT.
As indicated previously, it is difficult to understand from currently available information
what the toolbox is, what tools are in the toolbox, and where the toolbox is located. The
SAB questions whether EPA has defined the tools as maps, indices or the techniques
used to generate maps and indices.
It is difficult to understand what decisions the EDT was developed to influence. The
model and tools in the EDT are presented without a major justification that they are
needed.
11
-------
Lack of quantification is a problem in some components of the EDT. In particular, units
on the maps are confusing.
It appears that the tools in the EDT are, at present, limited to relatively few
environmental issues.
Information provided to the SAB suggests that the EDT will be used by "the public,"
"clients," and "researchers," but it is difficult to determine specifically who will use the
EDT.
For the most part the models applied in the EDT are "behind the scenes." Conceptual
models have not been presented and it is not possible to evaluate the underlying science
supporting the EDT. This science should be carefully and transparently documented. A
strength of the ReVA EDT is its potential value as an adaptive management tool that will
enable users to experimentally compare selected policies by evaluating alternative
hypotheses about the systems being managed. However, this will require presentation of
clearly articulated conceptual models that specify key state variables and describe the
dynamic interrelationships between them. Different "levels" of conceptual models in
ReVA should be described. An overarching conceptual model should describe what
ReVA is, the main objectives and questions being asked in ReVA, and what constitutes
the ReVA methodology. Other levels of conceptual models should describe the
underlying processes in ReVA, including the relationships between data sets used to
assess ecological condition and vulnerability. It may be useful to present the conceptual
models as box and arrow diagrams. Too much text is included on the websites where the
EDT is located. There appears to be little difference between reading a report and
viewing the EDT websites.
Flow diagrams of ecosystems and underlying mechanisms are needed in the EDT, not
just cause and effect models.
Strengths and Limitations of Elements in the EDT
The following is a listing of what the SAB believes comprises the strengths and limitations of
various elements of the ReVA EDT. The SAB provides recommendations for further
development for each of those EDT elements:
Element 1. An extensive set of spatially explicit data, formulated to be displayed on a map
system that has gone through a "standardized" evaluation for data quality.
Strengths: ReVA's real power to date is in the demonstrated exercise to bring divergent
spatial data into a single, useable source. Analysis using spatially explicit data is a well-
founded, scientifically defensible method for extrapolating and interpreting broader
conditions from limited existing data. Representation of spatial data is a powerful tool
for risk communication to users and the general public.
12
-------
Limitations: As noted previously, ReVA currently provides very limited documentation
of available databases/layers; and limited transparency of construction/extrapolation of
data layers, scale, and definition of uncertainty in extrapolation of data. The connections
between the current data layers used to indicate vulnerability and actual ecological
condition are tenuous, at best.
Recommendations: The SAB recommends that EPA compile and publish a separate
document on compilation, organization, extrapolation, and types of data/layers in the
ReVA toolkit. An example format that could be used to develop this document is Table
ES-2 in the executive summary of the SAB publication, A Framework for Assessing and
Reporting on Ecological Condition. It would also be helpful to include a statement of the
quality and confidence levels of the data.
Element 2. Mechanistic models that can be applied to the base spatial data to project future
conditions or trends. These mechanistic models may have been developed within the ReVA
program, or by separate/independent researchers who use the base data for projections.
Strengths: Mechanistic models are a well-defined, scientifically defensible means of
forecasting future trends. When coupled with spatially explicit data, they are a powerful
tool for forecasting future trends and defining the uncertainties associated with
projections. Coupled with geographically based displays, these are a powerful tool for
communicating risks to decision makers and the general public.
Limitations: The spatial data in ReVA appear to have been coupled with a number of
mechanistic models but the inventory of coupled models was not apparent at the ReVA
website or in the literature provided to the SAB. Mechanistic models used in ReVA
appear to be narrowly focused on forecasting changes in relatively few parameters (e.g.,
eutrophication, air quality [ozone, sulfur, urban growth]), and are constrained by the data,
assumptions, and calibration.
Recommendations: The SAB believes that EPA should focus more resources on
developing mechanistic models to be coupled with the spatially explicit data in ReVA.
This is potentially ReVA's most powerful application. Where models have been
developed, those uses should be listed on the ReVA web site (e.g., watershed models and
ozone models). One potential application that the SAB would like to see explored is the
coupling of ReVA with dose/response models.
Element 3. A series of data integration methods.
Strengths: As noted previously, when the integration methods are combined with spatial
visualization tools in ReVA, they offer simple, understandable summary indicators for
communicating the concept of "vulnerability" to the lay public.
Limitations: The SAB believes that the statistical integration methods developed and
used in ReVA have not been demonstrated to be statistically sound. The methods should
be validated and the levels of uncertainty associated with the methods should be
13
-------
identified. The underlying statistical models are not transparent. Complete
documentation on those models should be made available. Furthermore, for the models
to have utility, they should be subjected to field verification and/or validation, with some
assessment of external sensitivity and uncertainty. The models also assume ecological
relationships that are not implicitly evident from landscape data. Finally, the models may
be too ambitious; attempting to integrate too many factors at once. The SAB notes that
validating the integration methods may be challenging because some subjectivity is
associated with the concept of ecological condition. However, the ReVA data integration
methodology requires some level of understanding about whether the assignment of an
index of vulnerability is adequate and accurate, or at least bounding the uncertainty. The
SAB also notes that there are several applications in ReVA that could lend themselves
well to a validation exercise if adequate data are available. Three such applications are
EDT future projections concerning invasive species, resources extraction, and pollutants.
For example, modeled predictions of endangered species spread could be compared with
actual observations.
Recommendations: The strengths and utility of the integration methods should be tested
using a relatively limited set of environmental and landscape data. The SAB finds that a
"back-cast" demonstration of ReVA in a simpler system to answer focused questions
would be an effective way to illustrate the utility and potential power of the methods.
The SAB recommends that EPA allocate additional resources to the ReVA program to: 1)
run back-casts and conduct field validation of the integration methods; 2) apply the
integration methods using a more limited number of land/resource variables; and 3)
explore sensitivity and uncertainty with back-casts. A recommended system that might
be used to complete this work is described in the response to charge question three.
Use of Web-based Interface Tool
The elements of the ReVA EDT have been assembled into a web-based application that can
be applied by regional and local decision makers to conduct scenario analysis. The SAB finds
that, while this is a laudable goal, the computing power needed to handle and process
information may be too great to practically allow such web-based applications in the near future.
ReVA's current demonstration product on the web is a good, functional demonstration for
marketing the tool to potential regional and local decision makers. The SAB endorses EPA's
efforts to develop front-end, user-friendly interfaces for decision makers to explore the effects of
land use changes on environmental resources. However, EPA should be careful to include strong
cautions against using the interface tool for actual decision making. This is further discussed in
the response to charge question three.
5.2.2 Question 2b. Provide input as to the level of analytical capability needed in the
ReVA EDT for intended audiences.
General Comments on Analytical Capability
EPA has indicated that ReVA is expected to be a priority setting tool to target areas for more
focused risk assessment. The strengths and limitations associated with using ReVA for that
14
-------
purpose have been discussed above. The SAB notes that ReVA has also been presented as a
framework for environmental decision making and for communicating ecological condition and
vulnerability at multiple scales. Dual products of ReVA are: 1) the integration and assessment
methods in the "tool box" described in the EPA document, "Regional Vulnerability Assessment
for the Mid-Atlantic Region: Evaluation of Integration Methods and Assessment Results"
(Smith, E, L. Tran, and R. O'Neill, 2003), and 2) the web-based Environmental Decision Toolkit
(EDT) for data analysis and visualization. The two products have distinct roles in accomplishing
ReVA's purposes but are also intimately related.
The SAB finds that the analytical capability needed in ReVA is a multi-faceted issue. It
appears that ReVA is expected to provide an EPA system for communicating ecological
condition and risk to several intended audiences, such as the science audience, the decision-
maker audience, and the public audience. It is not exactly clear, however, who the intended
audiences are and what their needs are. The science audience expects scientific credibility,
quantitative accuracy, and rigorous exposition. The decision maker audience expects simplicity,
defensibility, and visualization. The public audience expects transparency and user-friendliness.
To use the ReVA EDT wisely, all audiences should know the key assumptions and provisos
behind the analytical models and their input data.
The SAB finds that the current ReVA approach provides more layered geographic
information content than quantitative analytics. However, the eleven integration and assessment
methods in ReVA have been developed from a vast literature encompassing multiple disciplines,
software, and decision tools. These methods offer great promise for further development and
future use. The SAB notes, however, that the ReVA methods for data integration, landscape
modeling, and integrative assessments appear to have been adopted through what might be
viewed as a somewhat ad hoc process that could be improved by documentation of additional
reasoning or validation. The SAB notes that the credibility of the ReVA toolbox and toolkit
needs to be addressed.
The SAB believes that careful definitions and descriptions, statistically sound methods, and
independently reproducible calculations must be provided in ReVA. Currently, ReVA's
discussions of limitations and sensitivities generally evolve into declarations with limited
supporting evidence. The SAB therefore recommends that appropriate personnel provide
critically needed expertise on data, analytical methods, and ecological interpretation to further
develop ReVA. In this regard, ReVA needs to be able to address the issues of uncertainty and
the currently missing, but extremely important, elements of statistical and practical importance:
false alarm, false discovery rate, and scale effects. Additionally, ReVA needs a methodology
handbook that provides careful documentation of the ReVA statistical tools in a manner that is
clear and accurate with analytical and empirical supporting evidence. To accomplish all of this,
the SAB recommends that the ReVA Program add to its existing manpower. It appears to be
quite unlikely that the ReVA Program will be able to satisfactorily address these critical needs
within the limits of its current manpower resources.
The SAB believes that the ReVA Program is an important EPA initiative, and finds that
ReVA staff are developing applications for new methodologies and technology with
considerable skill and insight. For example, critical areas are identified as extreme score
15
-------
watershed neighborhoods in the GIS-layered maps, and rankings are largely procured using
index-crunching methods involving uncertainty and ambiguity. The SAB notes, however, that
the ReVA Program may also benefit by recognizing the more plausible view of critical areas
identified as neighborhoods that have extreme scores, and prioritizing them without having to
crunch indicators criteria into indexes (Patil and Taillie, 2004b).
Specific Comments on Analytical Capability
The SAB finds that the analytical concepts, definitions and descriptions of ecological
condition and vulnerability provided in the ReVA documentation are insightful, although in
places they are not complete, clear, accurate, or precise enough. The SAB provides the
following observations and suggestions to clarify and improve the presentation of relevant
integration and assessment methods.
Explanation of "correlation." Many decision-makers confuse correlation with causation.
Decision-makers need to know the problems associated with any analysis that sums
correlation coefficients (as is done in the Stressor-Resource Matrix approach to
vulnerability analysis).
Title of the "Toolbox" Document. The title is misleading and could be changed to
"ReVA for Mid-Atlantic Region: Evaluation of Integration and Assessment Methods."
This would help clarify the expectations.
Conceptualization of Ecological Condition and Vulnerability. A case has been made for
vulnerability to have both a single directional gradient and a multidirectional gradient. A
clearer conceptualization for ecological condition would also be helpful.
Simple Sum. The discussion of the Simple Sum method should be clarified to ensure that
it is accurate and that statements concerning skewness and its effects on values, averages,
and variabilities are not misinterpreted by the reader.
Methods Ranking Distance to a Reference Condition. Methods used to rank watersheds
by distance to a reference condition include the "state-space method," "principal
component analysis" (PCA), and "criticality analysis." The SAB notes that it is not clear
how such a distance measure describes criticality analysis and PCA. In the case of PCA,
after axes have been rotated through any one of a number of different algorithms, the
concept of "distance to a benchmark" appears to be so distant as to be meaningless. It is
the opinion of the SAB that a method known as Technique for Order-Preference by
Similarity to Ideal Solution (TOPSIS) would be more relevant (Hwang and Yoon, 1981).
Principal Component Analysis. The discussion of Principle Component Analysis could
be improved by providing a clearer and more detailed discussion concerning combining
principal components and the roles of eigenvalues and eigenvectors. The discussion
should be clarified to ensure that it is accurate and is not misinterpreted.
State Space Analysis. The SAB notes that this is an innovative concept but it needs more
16
-------
work. The SAB recommends that EPA move beyond consideration of the most
vulnerable corner into the consideration of the most vulnerable candidate watersheds
closest to the corner.
Criticality Analysis. The SAB finds that the concept of natural state in the ReVA
documentation is interesting. However, the documentation tends to be overly simplistic
concerning issues of ambiguity and uncertainty. It is the finding of one SAB ReVA
Advisory Panel member that the triangular and rectangular aspects of fuzzy numbers are
over-rated and that sensitivity to location is under-rated. This is addressed in more detail
in Appendix A of this report provided by ReVA Advisory Panel member Dr. Ganapati
Patil.
Cluster Analysis. The SAB notes that it is good to see the limitations of cluster analysis
described and analyzed in terms of the instabilities of the clustering methods. ReVA
might benefit from consideration of the spatially constrained clustering tools. Spatially
constrained clustering helps locate the edges of homogeneous regions, resulting in closed,
areal boundaries. Spatially constrained clustering has been applied in landscape ecology.
Applications have involved ecological variables, environmental variables, and
biophysical variables for exploring ecologically homogeneous as well as geographically
contiguous clusters such as habitat patches, biophysical settings, and soil zones
(Burrough, 1989, Fortin, 1994; Fortin andDrapeau, 1995; Legendre, 1987; Legendre and
Fortin, 1989). Software for the analysis is also available (Boundary Seer, 2001).
Change Analysis. The SAB notes that the ReVA documentation appears to confuse
change and difference analysis. It should refer to difference analysis for method-based
rating comparisons and change (map) analysis for future-present comparisons.
Self-Organizing Maps. The SAB believes that issues of watershed incomparability and
meaningfulness of the ordination in terms of environmental features should be examined.
Analytic Hierarchy Process. The ReVA documentation is not clear in this area,
particularly with regard to the hierarchical levels and their numbers. No differential
weights have been assigned to indicators to represent sensitivities to within and between
group indicators, particularly when eigenvalues and eigenvectors are available.
Ambiguities are not addressed in the discussion. For example, the description of the
diagram as hierarchical is confusing because the bottom two levels are not a hierarchy,
but a Cartesian product. Further, the method may be reasonable if the correlations within
each group of indicators are very high and the between group correlations are very low.
However, if the within-group correlations are only moderately high and the between
group correlations are low, then better top level weights can be computed from the
eigenvalues of the principal components analysis, and better second level weights from
the eigenvectors.
The SAB recommends that a methodology document and user's manual approach
(approximately 25 pages in length) be prepared as an integral part of the toolkit to address these
issues. A precise description of each integration and assessment method should be included.
17
-------
Basic documentation of the ReVA methodology, as well as metadata for the entire methodology,
should be included. It would also be useful to include a table of the assessment questions and
integration methods in the document, with an indication of which methods (or suite of methods)
are most appropriate for answering the questions.
Much more additional advice should be provided about statistical tools in ReVA and how to
use them. The SAB notes that the analytical capabilities provided on the ReVA public website
should be perceptive and insightful. As noted above, some targeted audiences need greater
analytical capabilities to handle the tools they receive. The SAB recommends that EPA provide
more information to decision makers about the analytical methods in ReVA so decision makers
can decide which tools to use. Users should be familiar with multivariate statistics in order to
understand that different algorithms (e.g., axis-rotation procedures) will yield different or
differently weighted principal components or "factors" in factor analysis, and that this may
influence the results of analyses. The SAB notes that relatively few users (as opposed to
researchers actively using statistical modeling) will have this familiarity. Few users will be
familiar with fuzzy data sets and Kohonen self-organizing maps. However, it may not be
necessary to provide extensive information about these procedures if simple conceptual
explanations of the procedures are available using metaphors and analogies.
The SAB notes that the analytical capabilities offered to ReVA's users should be
sophisticated, and therefore information about how to use ReVA tools should be provided to
users. Clearly, users will need information about the watersheds being compared. This is
because the standardization in ReVA (scaling from 0 to 1) implies that watershed evaluation
criteria (e.g., number of aquatic species) are comparable among all watersheds evaluated. In
fact, the natural biotic diversity of different habitats may vary greatly within a region and there
may be significant variation in the best possible criteria values observed under pristine
conditions. Without such knowledge, blind reliance on the indices produced can be misleading
and the indices can be inaccurate. EPA should provide ReVA users with the capability of
performing different kinds of standardization. This will enable users to analyze their own data.
In order to complete these kinds of analyses, users need to recognize what the standardization is
doing and the ranges within the watersheds. The SAB also notes that it will be helpful if, in
developing the EDT, EPA recognizes color-blindness of some users and develops outputs
accordingly.
The SAB's Ecological Processes and Effects Committee recently recommended a hierarchical
structure for reporting on ecological condition because it revealed tradeoffs between sets of
indicators in meaningful categories as indicators are aggregated upwards. These SAB
recommendations were published in the document, "Framework for Assessing and Reporting on
Ecological Condition" (U.S. EPA Science Advisory Board, 2002). The SAB notes that ReVA
may also benefit from hierarchical integration methods as well. Currently, the only fully
hierarchical method explored in ReVA is the analytical hierarchy process (AHP). ReVA
documentation should highlight the relationship of AHP to the recommendations provided in the
SAB EPEC document cited above.
18
-------
5.2.3 Question 2c. Provide input as to approaches for presenting available information
and uncertainty in the EDT.
As discussed above, the SAB finds that a more extensive "user's manual" is needed to fully
understand the adequacy of the approach used to present data in the EDT and the ability of the
EDT to present uncertainty. With regard to uncertainty, there are two issues of concern to the
SAB. The first issue of concern is that it is difficult to judge the adequacy of the presentation of
information and uncertainty without more specific details describing the implementation of
ReVA. It is difficult to know definitely whether the information and uncertainties are presented
effectively because important details remain unclear. For example, it is not clear how one would
weigh or prioritize effects and vulnerabilities using the ReVA approach. It is also not clear what
process or rules one would use. In the absence of clear guidance, many diverse decisions will be
made and will influence the presentation of the state of vulnerability. Some guidance about
selecting methods is included in the ReVA documentation. However, information providing an
in-depth understanding of the methods is not presently available. The SAB notes that ReVA
users presently appear to explore the use of methods until a feeling emerges that the best
integration approach has been found.
The second issue of concern with regard to uncertainty is that it is difficult to know how much
uncertainty is associated with the EDT presentation of system vulnerability. This is because the
ReVA definition of vulnerability does not include all essential aspects of Cairns' generally
accepted definition of ecosystem vulnerability (Cairns and Dickson, 1977). The ReVA
presentation of vulnerability appears to be indifferent to some important qualities of ecosystem
vulnerability as defined by Cairns. Cairns defined ecosystem vulnerability as "susceptibility of
an ecosystem to irreversible damage," and he identified three major issues associated with
ecosystem vulnerability: 1) elasticity or the ability to return to an original, pre-stress condition,
2) inertia or the ability to resist change in function or structure, and 3) resilience or the number of
times that the ecosystem is able to recover to its normal state. The SAB notes that two
ecosystem qualities may experience the same level of a stressor but have very different levels of
inertia. Two ecosystem qualities may change identically with stress but one may be more
capable of rebounding after the stressor is eliminated. Some ecosystem qualities may rebound
only once or twice but others could potentially rebound many times before permanent damage is
established. The SAB believes that the ReVA EDT should incorporate these differences in key
characteristics in order to present ecosystem vulnerability. The SAB finds that the current
presentation of vulnerability in ReVA does not appear to allow these qualities to be visualized.
5.3 Question 3. Please comment on the usefulness of the ReVA approach to decision
makers in allowing them to see the overall consequences of future development, and
mitigation, conservation, and restoration activities.
Within the context of improving the ecological evaluation data and integration methods, the
SAB endorses the continued development of ReVA. The SAB finds that the ReVA methodology
can be a useful component in evaluating the overall consequences of future development,
mitigation, conservation, and restoration activities. While ReVA is not a unique product within
the realm of landscape, urban, or decision-planning software tools that use geographic
information technology, its important contribution to this field is its emphasis on critical or
19
-------
vulnerable habitat evaluations.
ReVA's utility can be improved by (1) explicitly acknowledging the differences between
forecasting and scenario analyses, (2) continuing efforts to improve or enhance ecological
conditions database, (3) validation and/or improvement of the ecological condition integration
methods, (4) incorporation of commercially-available decision-assisting software, and (5)
recognition within ReVA that ecological vulnerability decisions must also consider equity,
efficiency and effectiveness - including social justice issues. The SAB recommends that EPA
explore adding tools and data layers to ReVA to make it more useful in the decision making
process.
Forecasting Versus Scenario Analysis
As indicated in the response to charge question one, there are two "futuring" functions that
can be used in the ReVA methodology: 1) mechanistic forecasting models, and 2) scenario
analysis. The SAB strongly recommends that the developers of ReVA provide a clear indication
of the differences between the functions of forecasting and scenario analysis to project future
vulnerability.
Forecasting mechanistic models are defined as mathematical algorithms designed to answer
relatively narrow questions and predict changes to environmental parameters over a defined time
frame. The coupling of a comprehensive spatial data set (such as the one provided by ReVA)
with well-defined, calibrated and validated mechanistic models provides a powerful ability to
predict changes over time in environmental conditions with quantifiable uncertainties. An
example of a forecast application mentioned previously is the application of the "PM2.5" model
with ReVA to project future ozone levels for the Clear Skies Initiative. Other examples of the
utility of geographic information system/mechanistic model coupling and forecasting include the
fate, transport, and bioaccumulation prediction functions developed by EPA for risk assessments
on the Hudson River, the Housatonic River, and the Lower Fox River. Another excellent
example of the linking of spatially-explicit information with dynamic ecological models is the
Across Trophic Level System Simulation (ATLSS) (Duke-Sylvester and Gross, 1999). The
ReVA methodology is well suited as a tool to explore regional or watershed level questions such
as how agricultural nutrients exported from midwestern states impact the vulnerability of the
Gulf of Mexico. The SAB recommends that additional resources be allocated by EPA to further
develop ReVA for use in this fashion.
The SAB believes that development of ReVA and its applications is an effort in scenario
analysis; the exploration of potential changes to the overall landscape using the baseline spatial
data coupled with good visualization tools presented with geographic information technology.
The ReVA web-based Environmental Decision Toolkit, with weighting factors, spatial
integrators, and color map representations, appears to be well suited for this use. For example,
the SEQL program in Charlotte North Carolina plans to use ReVA to create and compare
alternative development scenarios. In this context, ReVA will be used to develop decision tools
to help build consensus on density and location of new development in order to minimize
creation of new transportation demand, promote clean air, and plan for sustainable community
infrastructure while preserving potentially vulnerable habitats.
20
-------
The SAB believes that the developers of ReVA must be careful to qualify the limitations of
analysis as currently conducted in ReVA, and distinguish it from forecasting. In this regard, the
SAB notes that scenario analysis does not prescribe significance and is not probabilistic or
predictive in any mechanistic fashion. Scenario analysis is simply application of a set of
conditions observed in the past to project a plausible future case. The SAB finds that ReVA in
this context is therefore best suited for use as a screening tool. In essence, the ReVA approach is
equivalent to low resolution modeling used by landscape planners. An additional problem
associated with scenario analysis in ReVA is that as one evaluates more localized areas, small
events may have a greater influence on vulnerable habitats. The SAB therefore recommends that
ReVA explicitly include conditional statements regarding the predictive (or lack of predictive)
power in its scenario analysis components. The SAB also recommends that ReVA not be used
as the sole tool for evaluating local conditions. For example existing protocols that utilize
Indexes of Biotic Integrity (IBI) can be used in environmental bioassessment.
Ecological Conditions Data
The limitations associated with the ecological conditions data in ReVA have been discussed
above. The SAB notes that the ReVA approach to decision making could be made more useful
through increased use of response measures and ecological endpoints, and use of a process to
evaluate the performance of indicators developed to assess ecological condition. External
verification of indicator applicability, sensitivity, and sources of uncertainty is also needed.
Again, the SAB EPEC Framework Document, referred to above, can provide useful guidance.
Validation and Confirmation of the Ecological Condition Integration Methods
The SAB noted in the response to charge question two that serious questions remain
regarding the integration methods used in ReVA. The integration methods are unique and
elegant applications. However, there is a need for a careful description of the methodologies, an
evaluation of the statistical soundness of the methods, the capability of reproducibility of the
methods (demonstration of similar results among multiple users), field validation of the
integration methods, and a discussion of uncertainty. The SAB believes these actions are
achievable, endorses continued effort in this area, and recommends that EPA provide resources
(either direct budget or personnel) to complete this evaluation.
Decision making Process and Software
The SAB finds that there are serious limitations associated with the decision making tools and
process developed for use in ReVA. Although EPA has tried to incorporate a decision process
into ReVA, key decision tree concepts are not presently included in the approach. Conceptual
models and/or guidelines for setting priorities are important elements that are not presently part
of ReVA. The limitations of ReVA as a priority setting tool, discussed in the response to charge
question one, limit its usefulness in decision making. A number of commercially available
software packages support prioritization and decision assistance. These software packages could
be applied to the ReVA process. The SAB believes that EPA should incorporate commercially
available decision assistance software into ReVA instead of trying to develop de novo decision
21
-------
assistance software.
The SAB notes that priority setting for decision making must consider equity, efficiency and
effectiveness. Effectiveness means getting the job done (e.g., reducing vulnerability) regardless
of cost; efficiency refers to output divided by input (e.g., benefit-cost ratio), and hence does
consider the cost (e.g., use of various resources) involved; equity is some notion of fairness. If
ReVA is not to be used in priority setting, it need not consider equity, efficiency, and
effectiveness. However, to the degree that ReVA (or any other entity or tool) concerns itself
with priority setting, it should consider equity, efficiency, and effectiveness.
5.4 Question 4. Please provide input on issues encountered as the information and
approaches in ReVA are used at finer scales. Please also provide input on future
ReVA research priorities and alternative applications of ReVA methods for decision
making at multiple scales.
5.4.1 Question 4a. Provide input on the issues encountered as the information and
approaches in ReVA are used at finer scales.
ReVA has been demonstrated within a region (on a multi-state scale for the Mid-Atlantic
Region), and it is being developed for a "local" 15-county area surrounding Charlotte, North
Carolina. ReVA may potentially be applied at even larger and smaller scales. The SAB notes
two issues in applying ReVA at scales finer than the Mid-Atlantic. The first issue is that at finer
scales, the number of stakeholders involved in the analysis frequently increases. Whereas at
regional scales, the decision maker may be an agency manager making decisions on regional
priorities, at finer scales decisions are made that directly affect the use of lands and the quality of
life. Such decisions will concern a large segment of the population. The implication for ReVA
of the increased number of actors using the tools at finer scales is that ReVA must be developed
for users with a significantly lower level of scientific and technical expertise. The tool must
balance scientific rigor with clarity and simplicity of concepts and application. ReVA's role as
an educational tool in relation to its original multicriteria decision making role should increase at
finer scales.
The second issue deals with the choice of indicators to be used at finer scales. ReVA is a
framework and an approach, but the choice of condition and resource indicators is left to the
discretion of users. Thus users have an opportunity to select indicators myopically, overlooking
processes operating at scales above that of the area, or exports of stressors to adjoining areas.
Hierarchy theory advises that patterns at any local scale are conditioned by processes at larger
scales. For example, an indicator that only considers habitat fragmentation within assessment
units in an area could underestimate the overall impacts on migratory bird species if the corridor
function is lost. Similarly, an area may be a source of a stressor on a neighboring area even if
that stressor does not manifest prominently in the vulnerability assessment of the source region.
This cross-area issue is exemplified by agricultural nutrients exported from midwestern states.
The export of these nutrients can have an impact on the vulnerability of the Gulf of Mexico.
This also underscores the importance of selecting indicators that respond to policy options so that
the effects of scenarios can be examined. These concerns apply at any scale, but are likely to be
most prevalent at fine scales. At these scales, local problems typically dominate the discussion,
22
-------
but local problems may require regional solutions. The SAB encourages EPA to document and
explain these issues related to the choice of indicators, and to provide exemplars where available.
Further, EPA should consider tracking exports to adjoining areas and making this information
available for decision makers. These factors need not be included in the integration methods
because they do not affect vulnerability of the assessment units within an area, but they would
alert decision makers when a potential decision would create new problems for someone else.
5.4.2 Question 4b. Provide input on research priorities and alternative applications of
ReVA methods for decision making at multiple scales.
The SAB notes that the methods and applications in ReVA can provide the kind of
information sought by a wide range of organizations, including conservation groups and other
nongovernmental organizations. These organizations often work in areas that are data-poor and
ReVA can provide them with important and useful information. The SAB notes that the
following research priorities and applications can support further development of ReVA
methods for decision making at multiple scales.
Because many organizations work in regions that are data-poor, research is needed to provide
further information about the minimum amount of data needed for advice and guidance in
decision making. It is important to examine how much certainty is lost as the amount of
available information is reduced, and also whether there is a core set of metrics that will
always be needed by decision-makers.
ReVA currently provides information about the vulnerability of geographic areas. An
alternative and very useful application of ReVA would be to provide information that
could be of use in identifying geographic areas of the "highest value." The SAB notes,
however, that this will be a very complex task because the innate values associated with
ecosystems are very "context-specific." The values of resources will differ depending on
factors such as adjacent land uses and the size of the human population living in the vicinity.
The SAB notes that alternative applications of ReVA will require validation, and additional
data input files are needed to understand uncertainty. Clearly, integration methods must be
validated. Validation of ReVA methods is an important research issue.
It will be important to determine whether ReVA is providing data that describe the critical
parameters for assessing vulnerability. For example, an analysis should be conducted to
determine whether the nitrogen and phosphorus thresholds used in ReVA provide
information needed for the assessments of vulnerability. If major data sets are not useful to
users, the data sets should be dropped out of ReVA. In addition, the "core measures" in
ReVA should be identified. The SAB notes, however, that the philosophy of using a single
index should not be embedded within ReVA.
It would be very useful to provide data sets describing simpler "scenarios." This would
enable the users of ReVA to more easily understand and identify problems that span resource
issues. For example, data could be made available from high mountain lakes in California.
User groups are interested in the fisheries in these lakes. Exotic species in these lakes have
23
-------
affected native biodiversity and altered community structure, and the U.S. Forest Service is
interested in managing the lakes to maintain biodiversity. There are clearly identified
resource values associated with the lakes. There are also two primary resource stressors,
introduced fish and increased nutrient loading. Data sets from these lakes describe a simpler
scenario than the Mid-Atlantic regional information currently provided in ReVA. Well-
defined data at a fine scale such as the high mountain lakes in California can be "scaled up"
to evaluate the hydrologic cataloging unit and regional levels.
Research is needed to develop a roadmap for validation of ReVA futures tools. Validation of
ReVA methods will depend upon confidence in the futures data layers. The SAB notes that
many of the variables in ReVA are computed from others (e.g., in the case study phosphorus
is computed from sediment) and validation of these relationships is necessary. The SAB also
notes that two other important aspects of the ReVA futures tools must be validated.
Validation of substitution of space for time must be conducted to ensure that ReVA is not
extrapolating beyond the range of data. These issues have been carefully examined through
research conducted at the U.S. Forest Service HJ. Andrews Experimental Forest in Oregon
(Andrews Experimental Forest LTER, 2002). Work must also be conducted to validate
predictions made using configurations of data that have not been seen previously. ReVA will
be subject to criticism if validation of the futures data layers is not undertaken.
The SAB also recommends the following relatively minor but important improvements in the
ReVA documentation and visualization: 1) users should be provided information about
confidence in data used in the framework for projections; 2) some of the maps in ReVA have
defective labels and should be corrected; 3) EPA must be careful in explaining to users what
scenarios mean; and 4) spatial problems (scale effects) associated with ReVA map
representations should be resolved. For example, if the North Carolina streams biological
data currently in ReVA are expressed at a regional scale, the stressor results appear to be
different from stressors results associated with individual streams. It is important to examine
the relevant scales of stressors in ReVA.
In summary, the SAB strongly supports the efforts of EPA's Office of Research and
Development to develop ReVA. The suite of tools in ReVA can assist local and regional
resource managers in assessing current and future conditions. The SAB notes, however, that the
usefulness of ReVA could be greatly improved by providing additional documentation. The
SAB encourages EPA to continue developing ReVA, and to provide documentation on: what
constitutes ReVA, the framework and indicators for assessing ecological condition in ReVA, the
conceptual models underlying ReVA, clear basic documentation of the underlying methodology
for acquiring and assembling data, quality assurance reviews, and spatial data integration. A
methodology document and a user's manual should also be developed for the ReVA
Environmental Decision Toolkit documenting the ReVA statistical tools in a manner that is clear
and accurate with analytical and empirical supporting evidence.
24
-------
6. REFERENCES
Andrews Experimental Forest LTER. 2002. H. J. Andrews Experimental Forest, Long Term
Ecological Research, http://www.fsi.orst.edu/lter/pubs.cfm?topnav= 11
Bodini, A., G. Giavelli, and O. Rossi. 1994. The qualitative analysis of community food webs:
implications for wildlife management and conservation. Journal of Environmental
Management., 41: 49-65.
Borsuk, M.E., C.A. Stow, and K.H. Reckhow. 2003. Integrated approach to total maximum
daily load development for Neuse River Estuary using Bayesian probability network model
(Neu-BERN). Journal of Water Resources Planning and Management, July August 2003,
124(4): 271-282.
Boundary Seer (2001). Software for Geographic Boundary Analysis. TerraSeer, Inc., Ann
Arbor, MI. www.terraseer.com
Burrough, P.A. 1989. Fuzzy mathematical methods for soil survey and land evaluation.
Journal of Soil Science, 43:193-210.
Cairns, J.P., and K. Dickson. 1977. Recovery of streams from spills of hazardous materials. In:
J.P. Cairns, K. Dickson, and E. Herricks (eds.), Recovery and Restoration of Damaged
Ecosystems, pp. 24-42. University Press of Virginia, Charlottesville
Chorley, R. J., and B. Kennedy. 1971. Physical Geography, A Systems Approach. Prentice-
Hall, Englewood Cliffs.
Craig, R. 1981. Natural systems. In: Future Trends in Geomathematics, R. G. Craig and M.L.
Labovitz (eds.), pp. 265-274. London: Pion.
Druzdel, MJ. 1996. Qualitative verbal explanations in Bayesian belief networks. Artificial
Intelligence and Simulation of Behavior Quarterly, special issue on Bayesian networks, 94:43-
54.
Duke-Sylvester, S. M. and L. J. Gross. 1999. Integrating spatial data into an agent-based
modeling system: Ideas and lessons from the development of the ATLSS (Across Trophic Level
System Simulation). In: Integrating GIS andAgent-BasedModeling Techniques for
Understanding Social and Ecological Processes, R. Gimblett (ed.) University of Arizona
Fortin, MJ. 1994. Edge detection algorithms for two-dimensional ecological data. Ecology,
75:956-965.
Fortin, MJ. and P. Drapeau. 1995. Delineation of ecological boundaries: comparisons of
approaches and significance tests. Oikos, 72:323-332.
25
-------
Gould, P.R. 1986. Allowing, forbidding, but not requiring: a mathematic for a human world. In:
Complexity, Language, and Life: Mathematical Approaches, J. L. Casti and A. Karlqvist (eds.),
Berlin, Springer.
Hage, P. and F. Harary. 1983. Structural Models in Anthropology. Cambridge University Press.
Harary, R.Z., R.Z. Norman, and D. Cartwright. 1965. Structural models: An Introduction to the
Theory of Directed Graphs. Wiley, New York, 415 pp.
Hukkinen, J. 1993. Bayesian analysis of agricultural drainage problems in California's San
Joaquin Valley. Journal of Environmental Management, 37:183-200.
Hwang, C.L. and K. Yoon. 1981. Multiple Attribute Decision Making Methods and
Applications: A State-of-the-Art Survey. Springer-Verlag, New York
Legendre, P. 1987. Constrained clustering. In: Developments in Numerical Ecology, P.
Legendre and L. Legendre (eds.), pp. 289-307. NATO AST series, vol. G 14, Berlin: Springer.
Legendre, P. andMJ. Fortin. 1989. Spatial and ecological analysis. Vegetatio, 80:107-138.
Levins, R. 1974. The qualitative analysis of partially specified systems, New YorkAcad. Sci.
Ann. 231:123-138.
Maruyama, M. 1963. The second cybernetics: deviation-amplifying mutual causal processes.
American Scientist, 51 (2): 164-179.
Muruyama, M. 1968. Mutual causality in general systems. In: A General Systems Approach to
Positive/Negative Feedback and Mutual Causality, J.H. Milsum (ed.), pp. 80-100. Pergamon
Press, Oxford.
Patil, G. P. and C. Taillie. 2004a. Multiple indicators, partially ordered sets, and linear
extensions: multi-criterion ranking and prioritization. Environmental and Ecological Statistics,
11(2): 199-228.
Patil, G. P. and C. Taillie 2004b. Upper level set scan statistic for detecting arbitrarily shaped
hotspots. Environmental and Ecological Statistics, 11:183-197.
Phillips, J. D. 1983. Biophysical feedbacks and the risks of desertification. Annals of the
Association of American Geographers, 83:630-640.
Probability Theory and Bayesian Belief Nets. 2005.
http://www.dcs.qmw.ac.uk/~norman/BBNs/idxlist.htm
Puccia, CJ. andR. Levins. 1985. Qualitative Modeling of Complex Systems. Harvard
University Press, Cambridge.
26
-------
Puccia, C. and R. Levins. 1991. Qualitative modeling in ecology: loop analysis, signed
digraphs, and time averaging. In: Qualitative Simulation Modeling and Analysis, P. Fishwick
and P. Luker (eds.), Chapter 6, pp. 119-143. Springer-Verlag, New York.
Ringland, G. 1998. Scenario Planning: Managing for the Future. Wiley, New York.
Roberts, F.S. 1976. Discrete Mathematical Models. Prentice-Hall, Englewood Cliffs.
Roberts, F.S. 1978. Graph Theory and Its Applications to Problems of Society. Society for
Indus, and Appl. Math., Philadelphia.
Schwartz. P. 1991. The Art of the Long View. Currency-Doubleday, New York.
Slingerland, R. 1981. Qualitative stability of geologic systems, with an example from river
hydraulic geometry. Geology, 9 (Oct):491-493.
Smith, E.R., L. Tran, and R. O'Neill. 2003. Regional Vulnerability Assessment for the Mid-
Atlantic Region: Evaluation of Integration Methods and Assessments Results. EPA/600/R-
03/082, U.S. Environmental Protection Agency, Office of Research and Development, Research
Triangle Park, N.C.
Smith, E.R., R.V. O'Neill, J.D. Wickham, K.B. Jones, L. Jackson, J.V. Kilaru, and R. Reuter.
2000. The U.S. EPA 's Regional Vulnerability Assessment Program: A Research Strategy for
2001 - 2006. U.S. Environmental Protection Agency, Office of Research and Development,
Research Triangle Park, N.C.
U.S. EPA. 1984. Risk Assessment and Management: Framework for Decision Making. EPA
600/9-85-002, Washington, D.C.
U.S. EPA Science Advisory Board. 2002. A Framework for Assessing and Reporting on
Ecological Condition: An SAB Report, T. F. Young and S. Sanzone (eds.) EPA-SAB-EPEC-02-
09, U.S. Environmental Protection Agency Science Advisory Board. Washington, D.C.
U.S. EPA Office of Science Policy. 2003. Draft Guidance on the Development, Evaluation, and
Application of Regulatory Environmental Models. U.S. EPA Office of Science Policy, Office of
Research and Development, Washington, D.C. Available at:
http://www.epa. gov/osp/crem/library/CREM%20Guidance%20Draft%2012 03 .pdf
Varis, O. and S. Kuikka. 1989. Application of Bayesian influence diagrams in environmental
decision making under high uncertainty. In: Proceedings of the International Conference on
Multiple Criteria Decision Making: Applications in Industry and Service. Asian Institute of
Technology, Bankok, 6-8 December, 1989.
27
-------
Appendix A: Sensitivity of the Criticality Measure in ReVA2
The ReVA report "Regional Vulnerability Assessment for the Mid-Atlantic Region:
Evaluation of Integration Methods and Assessments Results" (Smith, E., L. Iran, and R. O'Neill,
2003) claims that the criticality measure is insensitive to the definition of natural state.
However, little evidence is actually offered to support this claim. The report simply considers
two possible natural states, the second allegedly having greater uncertainty than the first, and
observes that empirically (i) there is not much difference in the corresponding criticality values
and (ii) typically the criticality values are smaller with the second definition of natural state.
A mathematical analysis of the sensitivity issue indicates that:
Changing the uncertainty of the natural state has only a slight numerical effect on the
measure. Further, the effect is to increase the criticality value when uncertainty is
increased.
The criticality measure can be sensitive to changes of location (in indicator space) of the
natural state. The criticality value can increase or decrease depending on the nature of
the change of location.
The criticality measure would be about the same if the "fuzzy" numbers were ignored and
criticality was simply defined as the (squared) Euclidian distance from the given
watershed to the (midpoint) of the natural state.
The ReVA report does not give a precise definition of the criticality measure. For defmiteness,
the following may be supposed:
For each variable, the values associated with actual watersheds are crisp numbers rather
than "fuzzy" numbers.
For each variable, the "fuzzy" number associated with the "natural state" is either
symmetric triangular or rectangular over an interval of length L and midpoint M.
The distribution on the parameter a is uniform.
Integration is achieved by summing the (squared) fuzzy distances across all the variables.
The report is completely silent on the foregoing issues. The conclusions in this appendix do not
depend critically on these issues, except for symmetry of the fuzzy numbers. Of course,
antisymmetry of the fuzzy numbers would be an expression of uncertainty about the location of
the natural state, i.e., the midpoint of the uncertainty interval would vary with the parameters .
Fix a particular indicator variable, and let Wbe the value of that variable on the watershed in
question. Putting aside the notational pyrotechnics in the appendix of the report, the (squared)
fuzzy distance between the watershed and the natural state is
Rectangular: D2R = (W-M}2 +L2 (1)
2 The comments and suggestions in Appendix A were provided by one member of the Panel (Dr. G.P. Patil). The
Panel agreed that these comments and suggestions should be included in this appendix to the report.
28
-------
Triangular: D2 = (W -M)2 +L2. (2)
36
In either case, the fuzzy distance is the sum of two terms: (i) the squared Euclidian distance
between the watershed and the midpoint of the fuzzy number and (ii) a correction to account for
uncertainty. Three conclusions can be drawn at this point: (i) the uncertainty correction serves
to increase the fuzzy distance, (ii) the uncertainty correction is small compared with the
locational distance (W -M)2 unless the watershed is located within the interval L of uncertainty,
and (iii) those who claim that fuzzy distance is insensitive must also claim that Euclidian
distance is insensitive.
For the two natural state scenarios considered in the ReVA report, the most common change was
to replace the triangular with the rectangular membership function, keeping L and M the same.
The corresponding change in the fuzzy distance is
D2 -D2 = L2 -L2 = L2. (3)
12 36 36
If this change is made for TV indicator variables, then the integrated criticality measure will
increase by
N-L2. (4)
36
This is the effect of incorporating "fuzziness" into the definition of the criticality measure. But,
comparing the legends in Figures 9 and 10 of the ReVA report we see that the integrated
criticality measure has generally decreased. Thus, there must have been other changesin
locationof the natural state that offset this tendency to increase.
This matter can be examined for item (7) on page 18 of the ReVA report - soil loss. The actual
data are not available, so it is assumed that the first (lowest) quintile occurs at a value Q and the
second quintile at a value 2Q. Then, for scenario I, the membership function is triangular on the
interval from 0 to Q (so L = Q and M = Q12 ) while for scenario II, the membership function is
rectangular on the interval from 0 to 2Q (so L = 2Q and M = Q ). Inserting these values in
equations (1) and (2) gives
D2 =(W-QI2)2 +Q2
1 36
and
Thus,
29
-------
38
It follows that D\ > Z)2 if and only if W < Q ~ Q. Thus, Scenario II results in an increase in
36
criticality only for watersheds in the lower quintile. For watersheds in the upper quintile,
criticality can decrease and by a substantial amount as a result of changing the definition of
natural state from Scenario I to Scenario II.
One would need the actual data to quantify the numerical decrease. For example, if the
watershed values were uniformly distributed across the unit interval, one would have Q=l/5 and
the upper quintile interval would be (0.8, 1.0). Taking W=0.9=4.5Q to be the midpoint of this
interval, gives
~>o \
3.522 =-3.5Z2. (5)
Comparing with equation (4), one sees that this decrease due to a locational change in one
variable would offset the increase due to replacing the triangular with rectangular membership
function in about 62 variables. Also, observe that if fuzziness was discarded and the "natural
state" intervals were degenerate at their midpoints, the only effect would be to replace 38/36 by
3 1/36 in the preceding analysis.
A similar analysis was carried out for item 5 (Forest Inventories). This is an interesting example
since the definition of "natural state" varies with the particular watershed W. For scenario I, the
membership function is triangular on the interval from 0 to W(so L = W and M = W 12 ) while
the membership function for scenario II is degenerate at W(so L = 0 and M = W ). Here, one
finds that
n i ~,r '
36
where the 10/36 would be replaced by 9/36 if there were no fuzziness, just midpoints. Here,
scenario II always has smaller criticality than scenario II. The magnitude of the decreases varies
with the watershed; watersheds in the upper quintile produce larger decreases. For a watershed
at the midpoint of the upper quintile under uniformity (W=4.5Q),
which is only slightly smaller in magnitude than the decrease given in equation (5).
30
------- |