United States EPA Science Advisory EPA-SAB-EPEC-02-001
Environmental Board (1400A) November 2001
Protection Agency Washington DC www.epa.gov/sab
THE SCIENCE TO ACHIEVE
RESULTS (STAR) WATER
AND WATERSHEDS
GRANTS PROGRAM: AN
EPA SCIENCE ADVISORY
BOARD REVIEW
A REVIEW BY THE ECOLOGICAL
PROCESSES AND EFFECTS
COMMITTEE (EPEC) OF THE EPA
SCIENCE ADVISORY BOARD
-------
November 26, 2001
EPA-SAB-EPEC-02-001
Honorable Christine Todd Whitman
Administrator
U.S. Environmental Protection Agency
1200 Pennsylvania Avenue, NW
Washington, DC 20460
Re: Review of the Science to Achieve Results (STAR) Water and Watersheds
Extramural Grants Program
Dear Governor Whitman:
A panel of the EPA Science Advisory Board recently reviewed the Water and
Watersheds component of the Agency's extramural grants program, Science to Achieve Results
(STAR). The overall STAR program, currently funded at over $100 million per year, represents
a significant investment by the Agency in extramural research to support EPA's mission. For
this reason, STAR was the focus of recent reviews by the U.S. General Accounting Office and a
joint panel of the Agency's SAB and Board of Scientific Counselors. The Agency requested the
current SAB review of the STAR Water and Watersheds program because it is one of the longest
running components of STAR, having funded grants for approximately five years.
The SAB panel found that the Water and Watersheds program has provided relevant and
useful information. We have provided recommendations for mid-course corrections to the
program help ensure that the results will be used more effectively. The remainder of this letter
highlights our findings and recommendations. We look forward to your consideration of and
response to the enclosed report.
Background
The Science to Achieve Results (STAR) program provides a mechanism for the Agency
to engage academic researchers in work that supports the Agency's mission. Run by the Office
of Research and Development (ORD), the objectives of the STAR program are, in part, to
involve the best academic scientists in research efforts targeted at Agency priorities and to train a
cadre of environmental scientists for the future.
-------
Within the STAR program, the Water and Watersheds component is designed to
complement the Agency's work on ecosystem assessment and restoration. The specific
objectives of the Water and Watersheds component are to:
a) Develop an improved understanding of the natural and anthropogenic processes
that govern the quantity, quality, and availability of water resources in natural and
human-dominated systems;
b) Develop an understanding of the structure, function, and dynamics of the
terrestrial and aquatic systems that comprise watersheds; and
c) Promote integration across the biological, physical, and social sciences in the area
of watershed management.
Since 1996, approximately $36 million in Water and Watershed grants has been awarded
to academic researchers. These grants have required that the researchers use interdisciplinary
teams (representing biological, physical, and social sciences) to address watershed research
questions.
Results from many of the multi-year grants are not yet available to the Agency, and many
of the research teams have not had time to publish their results for the use of other scientists. In
order to review the program, therefore, the SAB panel attended a three-day meeting at which the
STAR Water and Watershed scientist teams were required to present their interim results to the
Agency. Our assessment of the research program and our recommendations for mid-course
corrections are based on information gathered at that meeting, as well as on written materials
provided by the Agency.
Conclusions
The Panel agreed that the Water and Watersheds program is an important component of
STAR and covers subject areas critical to the Agency's goals of protecting water quality and
participating in collaborative management of watershed resources. The scientific quality of the
research is high, and the program is producing a crop of younger researchers with experience in
practical applications of sophisticated academic research. An additional long-term benefit of the
program is to further legitimize within universities the pursuit of research on questions that cut
across traditional academic boundaries yet are relevant to the Agency's mission. The Panel
strongly recommends that STAR WW be retained as a major, focused program within EPA.
To date, much of the research has been focused on a subset of the Water and Watersheds
program objectives. The research predominantly has targeted water quality in human-dominated
systems. Within this subject area, many of the projects have generated models and other
"decision tools" that can be used to analyze the effects of watershed management schemes on
nature and the people who live in the watershed. These tools have broken new ground by more
rigorously linking knowledge about natural processes on the one hand and the social and
political drivers of human activities on the other hand.
-------
The Panel feels, however, that the current focus of the grant program may be forcing too
much homogeneity among projects while precluding some important areas of watershed
research. The requirement that each project incorporate social science, ecological, and physical
science research components has become a barrier to research on pressing questions that involve
a different mix of disciplines. Accordingly, the program should be refocused around
fundamental issues in watershed science, rather than on funding integrated research per se.
Interdisciplinary research will probably still be common under this scenario, although the
mixture of disciplines may shift over time as progress is made in addressing important
problems.
The charge questions presented to the SAB also asked whether the results of the Water
and Watersheds program are likely to be useful. The Panel found that most of the research
grants appear directly relevant to on-the-ground watershed management decisions. The primary
client base for most of the grants is local, and the information that is generated should be both
useful and understandable to the local groups. Little evidence was presented to the Panel,
however, to demonstrate that regional or national agencies will apply the information and tools
generated by the grants. Moreover, the collective results of the research grants have not yet been
used effectively. Now that a number of the projects have been completed, valuable insights can
be gained by analyzing the results of groups of projects.
Mid-Course Corrections
We suggest the Agency consider the following mid-course corrections for the STAR
Water and Watersheds program:
a) In conjunction with the Agency's program and regional offices, ORD should
identify known information gaps that limit effective watershed management and
target these for research. The Panel has provided examples of gaps that could be
targeted, such as developing a classification system for aquatic ecosystems that
comprise watersheds and establishing baseline (or "reference") conditions against
which watershed management success can be measured. These targeted
information gaps must be defined far less globally than the broad annual themes
currently used in the STAR WW Requests for Applications.
b) ORD should continue to promote research that is policy-relevant, but judge
relevance directly rather than using integration across academic disciplines as an
indirect measure. Projects that represent only one discipline, yet address critical
questions should be funded. The Panel has provided suggestions, including a
proposed template of questions for grant recipients, which may help accomplish
this purpose.
c) ORD also should continue to reserve some of the Water and Watersheds funding
for research that involves several academic disciplines, because it is one of the
few sources of such research funds, and because Agency funding will continue to
develop needed capacity for multi-disciplinary research within the academic
community. For example, the inclusion of social sciences in STAR research
-------
projects has produced insights that would not have arisen from an ecological
research focus alone. Because truly integrated research is complex and
organizationally cumbersome, however, these grants should be longer-term and
for larger amounts than those presently provided. Planning grants would be an
effective means of enhancing the integration of research questions in grant
proposals.
d) ORD should far more aggressively pursue its plans to produce "State of the
Science" reports that review and analyze the collective findings of STAR-funded
research. We recommend that ORD commission groups of researchers to
synthesize cross-project findings on a variety of issues. Typical questions might
include the following: What ecological endpoints were used most often and how
can these be improved to better represent changes in ecological condition? Did
multiple researchers make the same adaptations to common watershed models
and can more useful versions of the models now be published? Did the individual
projects independently arrive at the same study elements and similar sequence for
their execution, and can this experience now streamline management of
interdisciplinary research?
e) The Agency should develop a process systematically to distill and communicate
STAR research findings to its program and regional offices and to state agencies.
This recommendation was made earlier by the joint SAB/BOSC review of the
overall STAR program and remains relevant. The Panel provides several specific
suggestions that may be used by the Agency to accomplish this task. Although we
are well aware of the benefits of maintaining a free marketplace of research ideas
in the academic sector, we also concur with the earlier recommendation by the
SAB/BOSC that STAR scientists should work more directly with Agency
scientists and managers.
f) Should the Agency wish to measure more quantitatively the utility of the STAR
Water and Watersheds program, the Panel has provided a number of metrics that
might be used.
In sum, the Committee's relatively detailed review of the Water and Watersheds
component of the STAR program yielded conclusions strikingly similar to those of the
SAB/BOSC panel that reviewed the entire STAR program a year ago. While we have made
numerous specific suggestions to sharpen the focus of the program and derive more value-added
from its results, we agree with the previous SAB/BOSC panel's conclusions that, overall, "the
-------
STAR program is structured and managed so as to generate high-quality science, conducted by
well-qualified scientists, on topics that are relevant to the environmental problems identified in
the EPA Strategic Plan." We look forward to your response.
Sincerely,
/Signed/ /Signed/
Dr. William Glaze, Chair Dr. Terry F. Young, Chair
EPA Science Advisory Board Ecological Processes & Effects Committee
EPA Science Advisory Board
-------
NOTICE
This report has been written as part of the activities of the EPA Science Advisory Board,
a public advisory group providing extramural scientific information and advice to the
Administrator and other officials of the Environmental Protection Agency. The Board is
structured to provide balanced, expert assessment of scientific matters related to problems facing
the Agency. This report has not been reviewed for approval by the Agency and, hence, the
contents of this report do not necessarily represent the views and policies of the Environmental
Protection Agency, nor of other agencies in the Executive Branch of the Federal government, nor
does mention of trade names or commercial products constitute a recommendation for use.
Distribution and Availability: This EPA Science Advisory Board report is provided to the EPA
Administrator, senior Agency management, appropriate program staff, interested members of the
public, and is posted on the SAB website (www.epa.gov/sab). Information on its availability is
also provided in the SAB's monthly newsletter (Happenings at the Science Advisory Board).
Additional copies and further information are available from the SAB Staff [US EPA Science
Advisory Board (1400A), 1200 Pennsylvania Avenue, NW, Washington, DC 20460-0001; 202-
564-4533].
-------
U.S. Environmental Protection Agency
EPA Science Advisory Board
Ecological Processes and Effects Committee
STAR Water and Watersheds Review Panel
CHAIR
Dr. Terry F. Young, Environmental Defense, Oakland, CA
MEMBERS
Dr. Miguel F. Acevedo, Department of Geography, University of North Texas, Denton, TX
Dr. Steven M. Bartell, Senior Associate, Cadmus Group, Inc., Oak Ridge, TN
Dr. Gregory R. Biddinger, Environmental Issues Advisor, ExxonMobil Refining and Supply Company,
Fairfax, VA
Dr. Kenneth W. Cummins, Senior Advisory Scientist, CA Cooperative Fishery Research Unit and
Fisheries Dept, Humboldt State University, Arcata, CA
Dr. Cynthia C. Gilmour, Associate Curator, The Academy of Natural Sciences, Estuarine Research
Center, St. Leonard, MD
Dr. Lawrence L. Master, Chief Zoologist, Association for Biodiversity Information, Boston, MA
Dr. Charles A. Pittinger, Procter and Gamble Co., Cincinnati, OH1
Dr. Frieda B. Taub, Professor Emeritus, School of Aquatic and Fishery Sciences, University of
Washington, Seattle, WA
CONSULTANTS
Dr. Richard Perritt, Director, Environmental Science & Policy Program, University of Southern Maine,
Gorham, ME
Dr. Margaret Shannon, Associate Professor, Environmental Law & Policy, School of Law, State
University of New York at Buffalo, Amherst, NY
INVITED EXPERT
Dr. Timothy McDaniels, Professor, School of Community and Regional Planning, University of British
Columbia, Vancouver, BC
SCIENCE ADVISORY BOARD STAFF
Ms. Stephanie Sanzone, Designated Federal Officer, EPA Science Advisory Board (1400A), Washington,
DC
Ms. Mary Winston, Management Assistant, EPA Science Advisory Board (1400A), Washington, DC
with SoBran, Inc., Dayton, OH
-------
TABLE OF CONTENTS
1. EXECUTIVE SUMMARY 1
2. INTRODUCTION 6
2.1 Background 6
2.2 Statement of the Charge 8
2.3 SAB Review Procedures 9
3. RESPONSE TO THE CHARGE QUESTIONS 11
3.1 General Comments 11
3.2 A Practical Understanding of Watersheds 13
3.2.1 Watershed Ecosystems 14
3.2.2 Human Dimensions of Watersheds 15
3.2.3 Building A Body of Knowledge 16
3.3 Making a Difference: Evaluating STAR WW Research Findings 17
3.3.1 Evidence of Program Impact 17
3.3.2 Enhancing Policy Relevance and Stakeholder Participation 18
3.3.3 A More Systematic Approach to Evaluating Program Success 21
3.4 Integration of Ecological, Physical, and Social Sciences 23
3.4.1 Defining Integrated Research 23
3.4.2 A Need for Integrated, Basic Research 26
3.5 Major Advancements or Breakthroughs 27
3.5.1 Watershed Science 27
3.5.2 Interdisciplinary Integration 28
3.6 Perception of STAR WW Within and Outside the Research Community 30
3.7 Enhancing the Value of Current STAR WW Projects 32
3.7.1 Cross-Study Evaluation of Projects 32
3.7.2 Building Capacity for Integrated Approaches 33
4. SUMMARY RECOMMENDATIONS 36
LITERATURE CITED R-l
APPENDIX A: SUMMARY OF PROJECTS FUNDED BY THE STAR WATER AND
WATERSHEDS PROGRAM A-l
111
-------
1. EXECUTIVE SUMMARY
The Science to Achieve Results (STAR) Program is an EPA extramural grants program begun
in 1995 as a means to "include this country's universities and non-profit centers in EPA's research
program and to ensure the best possible quality of science in areas of highest risk and greatest
importance to the Agency"(EPA, 1999). Since its beginning, the STAR Program has grown to
approximately $100 million/year in research grants over 32 topic areas (see Table 1). STAR Requests
for Applications (RFA) are developed by the Office of Research and Development in consultation with
representatives of EPA program and regional offices. The Water and Watersheds (STAR WW)
portion of the STAR Program issues joint RFAs with the National Science Foundation and (since
1998) the U.S. Department of Agriculture. Since 1996, STAR Water and Watersheds has funded
over 50 watershed research grants totaling approximately $36 million (Table 2). Of these grants, 35
have been funded by EPA for a total of approximately $28 million. The overall STAR Program has
been reviewed with respect to its management structure and alignment with the Agency's research
priorities (e.g., U.S. GAO, 2000; SAB-BOSC, 2000). The present review, however, is the first
external evaluation to examine the scientific quality and likely utility of the STAR Water and Watersheds
research findings.
In a 1999 report, Evaluating Federal Research Programs, the Committee on Science,
Engineering, and Public Policy (COSEPUP) (a joint committee of the National Academy of Sciences,
National Academy of Engineering, and Institute of Medicine) recommended that expert peer reviewers
evaluating a research program look at scientific quality, relevance, and benchmarking (i.e., the stature
and influence of the research as compared to other research programs, including those in other
countries) (NRC, 1999). For the SAB panel's review of the STAR WW program, the Agency's
charge focused on relevance (Questions 1-3), and benchmarking (Question 4). Although there was not
a specific question on the scientific quality of the STAR WW research, the Panel commends the
program on the quality of the researchers and the research. Research on the processes-both human
and natural-that shape watersheds is important to EPA's mission to protect human health and the
environment, and the STAR program should retain Water and Watersheds as a focus area. The STAR
WW grants are supporting research that likely would not be done by academic researchers absent the
STAR program.
The NRC (1999) also recommended that both research and mission agencies should have as
one of their goals "the goal of developing and maintaining adequate human resources in fields critical to
their missions." The Panel applauds the STAR WW program for its focus on developing capacity in
the extramural research community for interdisciplinary thinking and problem-solving. The STAR WW
program will have long-lasting effects because of its role in facilitating and strengthening interdisciplinary
research within the academic community.
-------
The Charge Questions posed by the Agency, and the Panel's summary responses, are given
below:
Charge Question 1: Are the STAR Water and Watersheds grants, taken collectively,
likely to produce a body of research that will improve our practical understanding of: a)
natural and anthropogenic processes that govern the quantity, quality, and availability of
water resources in natural and human-dominated systems, andb) the structure, function,
and dynamics of the terrestrial and aquatic ecosystems that comprise watersheds?
Summary Response: Yes, the STAR WW grants have increased our practical knowledge of
watershed sciences. The inclusion of social sciences in the STAR WW program has produced insights
into decision-making on watershed management that would not have arisen from an ecological research
focus alone. On the other hand, the advancements have been limited to particular subject areas in the
watershed sciences. The STAR WW projects have focused primarily on linking the effects of
anthropogenic processes on water quality and biotic integrity measures. Within this focus, there is a
preponderance of work on models that link some aspect of social science with aspects of physical and
ecological components.
Many of the STAR WW projects are local case studies that currently have limited applicability
elsewhere. In order to enhance scientific understanding from the STAR WW research, the Panel
recommends that the Agency analyze and synthesize the results of groups of like projects. Future
STAR WW research could then focus on critical information gaps.
Charge Question 2: Are the research findings likely to make a difference in
environmental protection (i.e., are research results influencing Agency programs,
directions, or regulations? influencing other organizations and other researchers?)
Summary Response: Individually, most of the STAR WW grants appear directly relevant to
on-the-ground watershed management decisions. The primary client base for most of the grants is
local, and the information that is generated should be both useful and understandable to the local
groups. Little evidence was presented to the Panel, however, to indicate that the knowledge developed
in the grants is being applied by Agency staff or other local constituencies outside of the watershed
where the research was conducted. The Agency does not have a systematic process to collect
information on the application of STAR WW research results, and is just beginning to consider ways
of distilling and communicating STAR WW research findings. Improvements in this arena are timely
because many more of the multi-year grants are now nearing completion. Interdisciplinary integration
and stakeholder involvement—while important emphases of the STAR WW program—are not
sufficient to ensure that funded research will have utility to decision-makers. The Panel suggests
additional steps that could enhance the policy relevance, applicability, and ecological protection
afforded by STAR WW projects. The Panel also suggests some possible metrics for which data could
be collected to support future evaluations of program success.
-------
Charge Question 3: Is the requirement that grant proposals integrate ecological,
physical and social sciences producing a unique body of research? Would funding each of
the "circles" [in the Venn diagram] individually have the same outcome? Is the
integrated approach so important that it is giving us new insights into decision making at
the water shed scale?
Summary Response: Yes, the requirement to include ecological, physical and social sciences
in most of the STAR Water and Watersheds projects to date has produced a unique body of research.
The STAR WW program, both in its focus and its interdisciplinary nature, provides a source of
research funding that is rare within federal research programs. Funding projects within the individual
discipline groups (or "circles") would not have produced the same results. Most of the progress made
by STAR WW projects occurred at the interface between disciplines and/or at the interface between
scientists and stakeholders. In particular, the integration of socioeconomics and management issues into
watershed research is a very encouraging, unique and beneficial aspect of the STAR WW program.
The requirement to integrate social, physical, and biological sciences in every project appears,
however, to be forcing too much homogeneity among projects while precluding some important areas
of watershed research. The Panel recommends that future STAR WW Requests for Applications
retain some, but not exclusive, emphasis on interdisciplinary projects, and that they allow the mix of
disciplines to be determined primarily by the important science questions that need to be answered. A
portion of program funding could be set aside to ensure support for projects that include social science
research and one other of the discipline areas. Similarly, a portion of program funding might be
reserved for a few large, multidisciplinary projects. For these projects, the Agency should consider
providing planning grants, an increased level of funding, and longer grant periods, commensurate with
the additional complexity of the proposed research.
Charge Question 4: As a result of the Water and Watersheds program, do we see any
major advancements or breakthroughs in watershed science or interdisciplinary
integration across the relevant disciplines?
Summary Response: The Panel did not see evidence of major breakthroughs in watershed
science but did conclude that STAR WW was producing valuable opportunities to link the natural and
social sciences relevant to watershed assessment and management. Given the emphasis on
interdisciplinary research, which requires additional time and effort by researchers, it may be too early
to expect major advances in interdisciplinary integration. Advances from the currently funded projects
are likely to take the form of integrated application of existing models and more refined decision tools
for watershed management.
-------
Charge Question 5: How is the STAR WWprogram perceived within and outside the
research community?
Summary Response: In general the Panelists felt that the data provided in the pre-meeting
materials plus their experience at the STAR Progress Review did not provide a factual basis to assess
how the STAR WW program is viewed by the rest of the research community. The Panel agreed,
however, that a positive answer to this question would be a significant indication that the STAR WW
funding was achieving its goal of expanding the appreciation of integrated research on watershed
management. Measures of awareness and acceptance of STAR WW research could be developed as
part of a more comprehensive evaluation of program success.
Charge Question 6: What changes would [the Panel] recommend to the [STAR WW]
program managers?
Summary Response: Based on the materials provided by the Agency, and the STAR WW
researcher presentations in San Francisco, the Panel suggests a number of mid-course corrections to
enhance the impact of the STAR WW program. The recommended changes include cross-study
evaluation of existing STAR WW projects to distill and synthesize program results; improved
dissemination of STAR WW research findings to the larger research community and to potential users;
and redirection of future RFAs. The Panel's summary recommendations are as follows:
Recommendation 1: The Panel strongly recommends that STAR WW be retained as a major, focused
program within EPA.
Recommendation 2: In order to meet the Program's stated objectives, the Panel recommends that
STAR WW Requests for Applications focus more on fundamental issues in watershed science, rather
than on funding integrated research per se. Interdisciplinary research will probably still be common
under this scenario. Specifically, the Agency should:
a) Pursue a more balanced approach to addressing the program's objectives. The Panel
notes that STAR WW projects have focused primarily on anthropogenic processes,
water quality issues, biotic integrity measures, and human-dominated systems. The
Panel recommends placing additional emphasis on natural systems and reference
conditions, on the understanding of water quantity issues, and on ecosystem processes
and dynamics related to the maintenance of native communities and species.
b) Retain some, but not exclusive, emphasis on interdisciplinary projects, and allow the
mix of disciplines to be determined by important and relevant science questions that
need to be answered. In particular, replace the Venn diagram with a broader definition
of interdisciplinary research, and fund projects that only include one or two disciplines
when needed to address gaps in our understanding of watersheds. Continue to
emphasize the integration of social sciences with ecological research.
c) For a small number of particularly complex, truly integrated, multi-disciplinary projects,
consider providing planning grants, an increased level of funding, and longer grant
-------
periods.
Recommendation 3: The Panel believes that benefits from the existing STAR WW research grants and
practical application of research results could be significantly enhanced. Steps to do this include:
a) Cross-study evaluations to analyze and synthesize the results of groups of projects
(e.g., through convened panels of internal and external scientists);
b) Disseminate research results in useful forms (e.g., peer-reviewed literature, web-based
products, simplified glossy products) to business, government, and science sectors;
c) Improve delivery of extramural research results to Agency scientists and program
managers;
d) Continue to build capacity for trans-disciplinary work related to the Agency's mission
by, for example, enhancing inter-disciplinary and inter-project thinking and
communications; and
e) Provide support for fuller engagement of EPA STAR program managers in relevant
scientific and management communities, and for increased interaction with funded
scientists.
Recommendation 4: If the Agency desires a more methodical measure of STAR WW benefits in the
future, the Panel suggests that the Agency identify sets of measures that correspond to the specific
program objectives to be achieved, then determine means of gathering information on the measures.
The Panel provides examples of such measures in this report.
-------
2. INTRODUCTION
2.1 Background
The Science to Achieve Results (STAR) Program is an EPA extramural grants program begun
in 1995 as a means to "include this country's universities and non-profit centers in EPA's research
program and to ensure the best possible quality of science in areas of highest risk and greatest
importance to the Agency"(EPA, 1999). Since its beginning, the STAR Program has grown to
approximately $100 million/year in research grants over 32 topic areas (see Table 1). STAR Requests
for Applications (RFA) are developed by the Office of Research and Development in consultation with
representatives of EPA program and regional offices. All STAR research proposals undergo external
scientific peer review and, for those that receive a "very good" or "excellent" score on scientific merit,
an internal relevancy review. The relevancy review, conducted by representatives of EPA program and
regional offices, identifies "which proposals are most relevant, responsive, timely and complementary to
the intramural research program" (EPA, 1999).
The Water and Watersheds portion of the STAR Program issues joint RFAs with the National
Science Foundation and (since 1998) the U.S. Department of Agriculture. Since 1996, STAR WW
has funded over 50 watershed research grants totaling approximately $36 million (Table 2). Of these
grants, 35 have been funded by EPA at a total of approximately $28 million. The goals of the STAR
Water and Watersheds program are to:
a) develop an improved understanding of the natural and anthropogenic processes that
govern the quantity, quality, and availability of water resources in natural and
human-dominated systems;
b) develop an understanding of the structure, function, and dynamics of the terrestrial and
aquatic ecosystems that comprise watersheds; and
c) promote integration across the biological, physical, and social sciences in the area of
watershed management.
Beginning in 1996, the Water and Watersheds RFA shifted toward requiring greater integration
of ecological, physical, and social sciences relevant to watersheds. The program's conceptual
approach to integrated watershed research was embodied in a Venn diagram showing areas of
intersection among ecological, physical, and social science research (Figure 1). In 1996, projects were
required to demonstrate incorporation of at least two of the research categories (i.e., Areas 2, 3 and 4
of Figure 1), with most desirable proposals including all 3 categories (i.e., Area 1 of Figure 1). In
subsequent RFAs, only projects falling within Area 1 were considered for funding. In addition, the
RFAs have emphasized different focus areas each year (e.g., watershed restoration, Total Maximum
Daily Load development).
-------
Table 1. STAR RESEARCH AREAS (1995 - 2000)
AIR
Indoor Air Quality
Health Effects of Paniculate Matter & PM
Centers
Air Pollution Chemistry and Physics
Air Toxics
Mercury Fate and Transport
WATER
Drinking Water
Risk-based Decisions for Contaminated
Sediments
Water and Watersheds
Health Effects of Arsenic
HUMAN HEALTH
Exposure of Children to Pesticides
Endocrine Disrupters
Children's Environmental Health & Disease
Prevention Research Centers
Human Health Risk Assessment
Role of Interindividual Variability in Human
Susceptibility
Children's Vulnerability to Toxic Substances
in the Environment
Exposure to Waste Combustion Products
Chemical Mixtures in Environmental Health
ECOLOGY
Ecological Assessment and Indicators
Global Climate Change
Regional Scale Assessment and Analysis
Ecology & Oceanography of Harmful Algal
Blooms
Ecosystem Restoration
OTHER
Analytical and Monitoring Methods
Environmental Fate & Treatment of Toxics
& Hazardous Wastes
Environmental Statistics
High Performance Computing
Technology for Sustainable Environment
Decision-making & Valuation for Envir.
Policy
General Solicitation: Exploratory Research
Socioeconomic Projects Related to Pollution
Prevention
Program on Bioremediation
Futures: Detecting the Early Signals
The overall STAR Program has been reviewed with respect to its management structure and
alignment with the Agency's research priorities (e.g., GAO, 2000; SAB-BOSC, 2000). The present
review, however, is the first external evaluation to examine the scientific quality and likely utility of the
STAR Water and Watersheds research findings.
-------
Table 2. STAR Water and Watersheds Funding and Focus Areas for 1996-2000
Year
1996
1997
1998
1999
Integration
Requirements
2 discipline
categories
3 discipline
categories
3 discipline
categories
3 discipline
categories
Focus Areas
urban/ suburban;
public/stakeholder
involvement
watershed
restoration;
public/stakeholder
involvement
TMDL;
public/stakeholder
involvement;
education &
outreach
TOTALS
# New Grants
8 EPA
4NSF
10EPA
4NSF
9 EPA
2NSF
2USDA
8 EPA
2NSF
2USDA
35 EPA
12NSF
4USDA
Total EPA
(SlOOO's)
6,872
8,131
6,535
6,556
28,093
Total All Partners
(SlOOO's)
8,572
10,475
8,260
9,077
36,384
Source: NSF-EPA Partnership for Environmental Research web site (www.nsf. gov/home/crssprgm/'l. For
summary tables of STAR WW projects, see Appendix A.
2.2 Statement of the Charge
The EPA's Office of Research and Development, which administers the STAR Program,
requested that the Science Advisory Board evaluate several aspects of the STAR Water and
Watersheds (STAR WW) program. After discussions between the Agency and members of the
SAB's STAR WW Review Panel, the following charge questions were adopted to focus the review:
Question 1: Are the STAR Water and Watershed grants, taken collectively, likely to produce a
body of research that will improve our practical understanding of: a) natural and anthropogenic
processes that govern the quantity, quality, and availability of water resources in natural and
human-dominated systems, and b) the structure, function, and dynamics of the terrestrial and
aquatic ecosystems that comprise watersheds?
-------
Question 2: Are the research findings likely to make a difference in environmental protection
(i.e., are research results influencing Agency programs, directions, or regulations? influencing
other organizations and other
researchers?)
Questions: Is the requirement that grant
proposals integrate ecological, physical
and social sciences producing a unique
body of research? Would funding each of
the "circles" individually have the same
outcome? Is the integrated approach so
important that it is giving us new insights
into decision making at the watershed
scale?
Question 4: As a result of the Water and
Watersheds program, do we see any
major advancements or breakthroughs in
watershed science or interdisciplinary
integration across the relevant disciplines?
Ihysical Science
earch
Figure!. STAR Water and
Watersheds Diagram Used to Show
Interdisciplinary Integration.
Question 5: How is the program perceived within and outside the research community?
Question 6: What changes would you recommend to the program managers?
2.3 SAB Review Procedures
The STAR WW Review Panel (the Panel) was composed of 9 members of the SAB's
Ecological Processes and Effects Committee, augmented by 3 panelists (an invited expert from Canada
and 2 SAB consultants) with expertise in geography, sociology, public participation, economics, and
decision-making. The panel held a public teleconference meeting on April 3, 2001 and a public face-
to-face meeting in San Francisco on April 20, 2001. Prior to the meetings, the panel reviewed a
package of written materials prepared by the Agency, which included abstracts of all STAR WW
projects, a sample STAR WW Request for Applications, summary information on STAR WW
products and likely clients, and previous evaluations of the overall STAR Program. During the April 3
meeting, the panel was briefed on the STAR WW program by Agency officials, discussed the charge
questions, and requested additional information from the Agency. Based upon the written materials
provided and the Agency briefing on April 3, pre-meeting comments were submitted by individual
panelists prior to the April 20 meeting and these comments were shared among the panel members and
with the Agency and interested members of the public. In addition, the panelists attended an Agency-
sponsored meeting of STAR WW researchers on April 18-19, 2001 in order to hear first hand about
-------
the research funded by the STAR WW program. A public teleconference call of the Panel was held on
June 1, 2001 to continue discussion of Panel responses to the charge questions. Although opportunity
was provided for public comment, no comments were received for any meeting of the panel.
10
-------
3. RESPONSE TO THE CHARGE QUESTIONS
3.1 General Comments
In a 1999 report, Evaluating Federal Research Programs., the Committee on Science,
Engineering, and Public Policy (a joint committee of the National Academy of Sciences, National
Academy of Engineering, and Institute of Medicine) recommended that expert peer reviewers
evaluating a research program look at scientific quality, relevance, and benchmarking (i.e., the stature
and influence of the research as compared to other research programs, including those in other
countries) (NRC, 1999). For the SAB panel's review of the STAR WW program, the Agency's
charge focused on relevance (Questions 1-3), and benchmarking (Question 4). Although there was not
a specific question on the scientific quality of the STAR WW research, the Panel commends the
program on the quality of the researchers and the research. Research on the processes-both human
and natural-that shape watersheds is important to EPA's mission to protect human health and the
environment, and the STAR program should retain Water and Watersheds as a focus area. The STAR
WW grants are supporting research that likely would not be done by academic researchers absent the
STAR program. The Panel strongly recommends that STAR WW be retained as a major, focused
program within EPA
The NRC (1999) also recommended that both research and mission agencies should have as
one of their goals "the goal of developing and maintaining adequate human resources in fields critical to
their missions." The Panel applauds the STAR WW program for its focus on developing capacity in
the extramural research community for interdisciplinary thinking and problem-solving. The STAR WW
program will have long-lasting effects because of its role in facilitating and strengthening interdisciplinary
research within the academic community.
The Panel notes also that the value of integrated, multi-disciplinary research is not unique to
watershed assessment and management. Thus, the Agency should consider ways to bring the same
sort of integrated thinking, including social and natural sciences and stakeholder involvement, to other
decision-making arenas. Other arenas that would be enhanced by such research include, for example,
new product development and selection of control strategies for environmental releases to manage air
quality. In this example, future STAR funding might include airshed/watershed interactions and how
their management could be integrated on a local and regional scale.
And finally, the charge to the Panel included questions about the STAR WW program's
potential to improve practical understanding of watershed processes, to provide research findings that
will make a difference in environmental protection, to integrate ecological, physical, and social sciences,
and to produce breakthroughs in watershed science or interdisciplinary integration. The Panel
concluded, however, that program "success" did not require that all of the program objectives be met
equally. Realistic expectations for a scientific research program might include progress on some, but
not all, of the objectives implicit in the charge questions.
11
-------
The Panel's responses to the charge questions posed by the Agency are based on review of
STAR WW project abstracts and some final project reports, and researcher presentations. The Panel
did not conduct a detailed review of all program outputs, and indeed the majority of the funded projects
are still ongoing.
12
-------
3.2 A Practical Understanding of Watersheds
Charge Question 1: Are the STAR Water and Watershed grants, taken
collectively, likely to produce a body of research that will improve our practical
understanding of: a) natural and anthropogenic processes that govern the
quantity, quality, and availability of water resources in natural and
human-dominated systems, andb) the structure, function, and dynamics of the
terrestrial and aquatic ecosystems that comprise watersheds?
Summary Response: Yes, the STAR WWgrants, taken collectively, have
increased our practical knowledge of water shed sciences. The inclusion of social
sciences in the STAR WW program has produced insights into decision-making
on water shed management that would not have arisen from an ecological
research focus alone. On the other hand, the advancements have been limited to
particular subject areas in the watershed sciences. The STAR WW projects have
focused primarily on linking the effects of anthropogenic processes on water
quality and biotic integrity measures. Within this focus, there is a preponderance
of work on models that link some aspect of social science with aspects of
physical and ecological components.
Many of the STAR WW projects are local case studies that currently have limited
applicability elsewhere. In order to enhance scientific under standing from the
STAR WW research, the Panel recommends that the Agency analyze and
synthesize the results of groups of like projects. Future STAR WW research
could then focus on critical information gaps.
According to the materials provided to the Panel, the goals of the Water and Watersheds
program are to: a) develop an improved understanding of the natural and anthropogenic processes that
govern the quantity, quality, and availability of water resources in natural and human-dominated
systems; b) develop an understanding of the structure, function, and dynamics of the terrestrial and
aquatic ecosystems that comprise watersheds; and c) promote integration across the biological,
physical, and social sciences in the area of watershed management. Charge Question 1 asks whether
the body of research being funded by STAR WW is likely to meet the first two program goals.
(Comments on the third goal-integration across disciplines-are contained in Section 3.4.)
The Panel interprets the phrase "practical understanding" in Charge Question 1 as meaning that
the research findings would be relevant to decisions facing managers of watershed landscapes and
regional water supplies. Although we discuss the relevance of STAR WW research in Section 3.3, our
general conclusion was that the results of most of the STAR WW projects appear to be relevant to
management decisions in the watersheds where the work is being conducted. Moreover, since
stakeholders are involved in almost all STAR WW projects, the products of the research likely will be
incorporated into the public decision process. The STAR WW portfolio, therefore, meets the
"relevance" test in the targeted watersheds.
13
-------
3.2.1 Watershed Ecosystems
It seems likely that the STAR WW research ultimately will improve our general understanding
of watershed ecosystems, but the advancements to date have been limited to particular subject areas.
The current STAR WW program goals encompass a great deal of scientific territory: natural AND
anthropogenic processes; water quantity AND quality; natural AND human-dominated systems. Given
the broad scope of the STAR WW program, it is not surprising that the portfolio of funded projects
does not adequately treat all of these research areas. That said, the Panel has the following
observations about the balance of research funded to date under the STAR WW program:
a) Natural vs. human-dominated ecosystems
There is a strong emphasis on human-dominated ecosystems in the STAR WW projects funded
to date. This focus may be appropriate in light of the emphasis of other funding programs on more
natural (less heavily modified) ecological systems. However, in order to manage or restore watershed
functions, managers need data on reference conditions against which to compare the human-dominated
watersheds. This argues for more research on natural (i.e., less human-dominated) systems. At the
individual project level, researchers should give greater attention to the issue of reference condition
when designing watershed studies and interpreting research findings. In this regard, a classification of
aquatic ecosystems to match that in use for terrestrial ecosystems (Federal Geographic Data
Committee™FGDC—standard) would be an important step in allowing comparisons between
watersheds. That is, the transferability of information between systems requires a common basis for
defining the systems.
b) Water quality vs. water quantity
The STAR WW program seems to be supporting more work on water quality than on water
quantity issues. More emphasis on quantity would be helpful, since the amount of water and its
apportionment among user groups (e.g., agriculture, municipalities, and environment) is arguably the
paramount issue in many areas of the country.
c) Ecosystem structure vs. function and dynamics
The STAR WW projects supported to date are strong on structural description but functional
(or process) measurements are not as widely embraced and little has been done to investigate
dynamics. However, it should be noted that the intended distinction between dynamics and function or
process is not made clear. In addition to water quality and biotic integrity measures, projects should
address landscape structure, maintenance of an array of native communities and habitats, hydrology and
geomorphology, ecological processes (such as carbon and nutrient cycling), and disturbance regimes.
d) Freshwater vs. estuarine systems
The current STAR WW portfolio contains relatively few projects that focus on estuarine
systems and the interactions between upland watersheds and estuarine or coastal watersheds or, more
14
-------
importantly, on the interactions between upland watersheds and the coastal ecosystems downstream.
The U.S. population increasingly is concentrated in coastal areas, with associated stress on coastal
ecosystems. Further, large coastal ecosystems have an important influence on global cycles and on the
oceans. Coastal watersheds are historically under-represented in funding by federal agencies because
of perceived "jurisdictions" in research funding. However, management of many large drainage basins
in the U.S. is being driven by effects in coastal systems downstream. Examples include management of
the enormous Mississippi River basin in relation to the "dead zone" in the Gulf of Mexico, management
of Md-Atlantic watershed of the Chesapeake Bay to respond to Bay degradation, and the links
between water management in southern Florida and coral reef loss in the Florida Keys.
3.2.2 Human Dimensions of Watersheds
While the human dimensions influencing water quality and watershed conditions were evident in
most of the STAR WW research projects, there was little evidence of real integration of findings about
human systems and non-human components of watershed ecosystems. In sum, human dimensions
research efforts paralleled the research on biological, physical and chemical elements. But human
dimensions were not treated as critical parts of a synthetic analysis with other components to achieve a
more integrated and system-wide approach to research for understanding changes in watersheds. The
Panel had the following observations about the STAR WW research projects' contributions to our
understanding of the influence of anthropogenic processes on watersheds and stakeholder involvement
in watershed assessment and management:
a) Anthropogenic Processes in Watersheds
Much of the STAR WW research made concrete contributions towards describing how human
behavioral factors determine land use patterns, residential and farm management practices, and general
urban and suburban landscape sprawl and economic growth. These human activities in turn affect
environmental conditions in STAR WW project areas. Research findings were most effective at
explaining causal relationships among particular land use types and watershed impacts when
researchers focused on smaller study areas where human settlement and activity patterns corresponded
with hydrological boundaries of watersheds. This research examined such problems as linking growth
of recreational land use in a lake watershed with controlling problems of lake sedimentation and
phosphorus loading; studying farm agrochemical and crop tillage practices to determine best rural
watershed strategies; and using GIS programs to relate coastal town expansion with ensuing problems
of E. coli water pollution. Smaller watershed and human area research also was effective in developing
community participation schemes and models for decision-support systems based on the local context.
There is a need for the STAR WW program to collect and distill the results and disseminate the lessons
from smaller watersheds and particular forms of human settlements in order to increase understanding
and reach a wider range of social, economic and watershed contexts.
Few STAR WW studies focused on larger river basin systems and fewer examined a wide
range of human factors and their influence on watersheds. Yet, those that did provided valuable results.
The framework applied in the Baltimore Ecosystem Study, for example, integrated information on long
periods of historic urban and human settlement processes with relatively large-scale modeling of
15
-------
ecological river basin changes. Such integration efforts will provide important lessons for long range
management of waters and watersheds.
b) Stakeholder Involvement
Several STAR WW research projects addressed stakeholder involvement by investigating why
some watershed projects are more successful than others in organizing social capital for making
decisions and implementing solutions to social and environmental problems. Many of the projects
incorporated human values, beliefs and perceptions by promoting feedback among researchers and the
public, and sharing information and gathering watershed data along side community-based organizations
and watershed citizen groups. A subset of the STAR WW projects took a more complex approach to
involvement using applied social science research, where researchers surveyed community members of
watersheds and analyzed attributes for citizen role models and leadership through community-based
organizations, examining the formal arrangement for citizen participation within a process of local
watershed management and planning. Here, research focused on understanding the characteristics of
effective planning and citizen inputs involving questions of trust, legrtimization, interpreting scientific data,
differences in manager and public perspectives about water and watershed protection, and the overall
qualities needed to build more effective community problem-solving organizations. Research about the
workings of stakeholder involvement should be encouraged and can point to desirable collaborative
citizen-manager-scientist- cooperation and management systems.
3.2.3 Building A Body of Knowledge
The Agency should continue to encourage STAR WW researchers to publish their results in the
peer reviewed literature so that the STAR WW program will make lasting and significant contributions
to our understanding of watershed ecosystems. In addition, the improvement in understanding of
watershed ecosystems from the STAR WW work could be greatly enhanced by targeted efforts to
analyze the results of groups of STAR WW projects (e.g., to synthesize modifications made to existing
watershed models). The Agency should increase its efforts to distill and synthesize transferable lessons
from STAR WW projects to create a body of knowledge from the many individual projects. Likely
outcomes of such cross-project evaluations are discussed in Section 3.7. Efforts to "mine" the project
results would greatly increase the benefits from the STAR WW program at relatively low additional
cost. Program synthesis efforts would benefit from increased support from EPA for STAR program
managers engagement in WW-related scientific communities.
16
-------
3.3 Making a Difference: Evaluating STAR WW Research Findings
Charge Question 2: Are the research findings likely to make a difference in
environmental protection (i.e., are research results influencing Agency
programs, directions, or regulations? influencing other organizations and
other researchers?)
Summary Response: Individually, most of the STAR WW grants appear
directly relevant to on-the-ground watershed management decisions. The
primary client base for most of the grants is local, and the information that
is generated should be both useful and understandable to the local groups.
Little evidence was presented to the Panel, however, to indicate that the
knowledge developed in the grants is being applied by Agency staff or other
local constituencies outside of the watershed where the research was
conducted. The Agency does not have a systematic process to collect
information on the application of STAR WW research results, and is just
beginning to consider ways of distilling and communicating STAR WW
research findings. Improvements in this arena are timely because many
more of the multi-year grants are now nearing completion. Interdisciplinary
integration and stakeholder involvement—while important emphases of the
STAR WW program—are not sufficient to ensure that funded research will
have utility to decision-makers. The Panel suggests additional steps that
could enhance the policy relevance, applicability, and ecological protection
afforded by STAR WW projects. The Panel also suggests some possible
metrics for which data could be collected to support future evaluations of
program success.
Although it is too early to gauge the ultimate value of this program, it is easy to speculate that
the STAR WW research results will promote more effective and efficient environmental protection.
For example, a number of models have been adapted by STAR WW researchers to integrate
ecological and social (including economic) attributes. The body of research on these decision tools
should produce transferable lessons. Even more important, a number of prominent research groups
now have learned how to focus research on questions of interest to watershed stakeholders, and citizen
groups evidently have learned a great deal about the relevance and functioning of various ecological
attributes. STAR WW research projects seem to be having important impacts on watershed
management at local and regional levels. These are important and valuable outcomes.
3.3.1 Evidence of Program Impact
The Panel was provided summary information from STAR WW researchers on likely clients for
their research (EPA, 2001-"Examples of STAR Water/Watersheds Grantees and Their Clients"). In
some areas, anecdotal evidence of research impact was provided: e.g., fecal contamination detection
methods developed and evaluated in a 1995 STAR WW grant apparently have been incorporated in
17
-------
an EPA standard, and Kahl et al. (EPA, 2001, pp. 52-53) mentioned that their study results will be
used by the Agency, in response to a Congressional mandate, to help determine the effectiveness of the
Clean Air Act. The examples of STAR WW clients provided prior to the review indicated that several
state agencies, local governments, and stakeholder groups were at least interested in the information
being developed by these selected projects. Many of the examples of potential clients, however, were
not specific enough to allow the Panel to determine whether novel methodologies were developed or
whether the results of the grant work truly are being used.
One approach to evaluating the impact of STAR WW research projects is to assess the
likelihood that results of the type being generated by these grants are needed and applicable. Potential
indicators of applicability could be the number of peer-reviewed publications that deal with an
application of the research and the quality of web material generated by the grants. This type of
analysis was done by the Panel for a sample of STAR WW grants and the partial data indicate that the
fraction of publications dealing with the application of the results was very low for 1995, and shifting to
application-oriented publications after 1995. The existence of web material, which is one mechanism to
promote application of research findings, was very low in the grants sampled by the Panel. (In an
informal check of 15 STAR WW grants from 1995-1998, the Panel found that, other than brief
progress reports to EPA, only three had results available on web sites.) STAR WW should emphasize
the use of more web-based material to facilitate dissemination of results and models, but EPA should
include adequate support for these efforts in WW grants. In Section 3.3.3, the Panel suggests an
expanded set of metrics that might be used to evaluate the impact of the STAR WW program.
Based on the STAR WW researchers' presentations, materials provided to the Panel, and the
partial analysis described above, the Panel concluded that some of the broader benefits seen from the
STAR WW program include:
a) Developing a cadre of skilled, interdisciplinary scientists and managers capable of
tackling watershed issues and problems;
b) Building a wider and potentially more adaptable "toolbox" of resources for use in
watershed assessment and management;
c) Collaboration among multiple levels of government, including municipalities (which can
be both regulator and regulated), and the private sector (e.g., watershed associations);
and
d) Facilitating and strengthening interdisciplinary research within universities.
3.3.2 Enhancing Policy Relevance and Stakeholder Participation
Since 1996, the STAR WW program has emphasized interdisciplinary research. Being
interdisciplinary, however, is not sufficient on its own to ensure that research is policy relevant. What is
needed is that the research be oriented to address questions that bear directly on specific policy
decisions. It was clear from the STAR WW research presentations that the research questions, focus,
18
-------
and relevance could be sharpened in many cases by input from stakeholders. It is also possible (though
not proven) that more immediate and direct impacts could be made at lower cost to address particular
issues. Elsewhere in this report, the Panel recommends that the STAR WW program begin to target
known information gaps. As part of this shift, for example, researchers might query local planners (e.g.,
county commissioners, sanitation engineers, urban planners) regarding the pressing issues, decisions,
and challenges they face that could benefit from better understanding of the ecological, social, and
economic consequences of various management options under consideration. It also would be helpful
to increase emphasis in grant application reviews and in project plans and reports on transferability and
application of a project's results to other watersheds.
To demonstrate a potential method for focusing STAR WW research on explicit questions
relevant to watershed management, the Panel suggests a series of questions that could be answered by
grant recipients (see Figure 2). The basis for these questions is policy analysis structured in terms of
multiple objectives. Alternatively, if analysts are interested in policy analysis in terms of economists'
approaches to benefit/cost analysis, then those approaches can be built onto the series of questions
posed to STAR WW researchers. In asking the researchers to respond to these questions, we are in
effect asking them to think through and create a summary of the policy objectives, alternatives, and their
consequences for the policy decision(s) relevant to their project. Benefits of this type of analysis
include a) creating an explicit decision framework for the policy questions addressed in the project, in
order to focus attention of the researchers and interested parties on the key issues, and b) fostering
communication among interested parties about the policy decision and the contribution of the research
project. There are a variety of guide books for policy analysis that may be consulted for discussion of
the various approaches. More broadly, we suggest that investigators consider the writings of Morgan
and Henrion (1994) on the "Ten Commandments for Good Quantitative Policy Analysis."
19
-------
Research Questions
1. Summarize the basic research questions your project is attempting to answer.
2. Outline the fundamental, integrative questions of science that are addressed by these research
questions.
3. Explain how the research questions relate to Agency science priorities as given in the Office of
Research and Development's Strategic Plan.
Decision Framework
4. What are the specific environmental management decisions regarding water and watersheds that your
research questions are intended to help inform?
5. In what watershed(s) is your work being conducted? Will the research be generalizable to other
watersheds or other policy decisions? If yes, please explain how.
6. What are the fundamental ecological system attributes that affect, or will be affected by, the decisions?
7. What are the fundamental societal values that matter to stakeholders, agencies and other interested
parties in making these policy decisions? For guidance on how to clarify public values as a basis for
defining objectives for ecological risk management decisions, see, for example, the appendices to the
recent EPA guidelines for ecological risk assessment. Some examples of objectives that could be relevant
for the most of the policy decisions associated with water and watersheds research projects could
include:
-Promoting ecological health or ecological integrity within the watershed;
-Avoiding adverse effects on private property rights of landowners;
-Minimizing direct costs to governments, organizations and individuals
-Promoting beneficial uses of water
-And others, depending on the context, and what matters to interested parties.
8. What are some of the broad alternatives that could be considered for this environmental management or
policy decision?
9. What are the major uncertainties that arise in considering the impacts of these alternatives, in terms of
the objectives?
10. In a few sentences, what are the key value tradeoffs that arise in comparing and selecting among the
broad alternatives?
Figure 2. Example Questions to Sharpen Policy Relevance of STAR WW Projects
20
-------
3.3.3 A More Systematic Approach to Evaluating Program Success
Although expert review may be the most effective means of evaluating a research program's
quality and relevance, an additional method for evaluating the results of a research program is
bibliometric analysis. (For a discussion of the strengths and weaknesses of this approach in the context
of the Government Performance and Results Act, see NRC, 1999). If the Agency wishes to augment
expert review with a measurement approach in the future, the Panel suggests that the Agency identify
sets of metrics that are relatively easy to measure and that correspond to the specific program
objectives to be achieved (e.g., see Table 3). This might be done in a variety of ways: for example, by
asking grantees to explicitly address the question in their reports, or through questionnaires to
stakeholders and Agency staff. Serious deliberation will be required to define measures that reflect
STAR WW program objectives without themselves leading to unintended consequences.
The selection of program performance measures requires definition of "environmental
protection" in terms of process (e.g., conforming with regulations, stakeholder groups developed,
institutional developments); effective planning (e.g., preventing adverse impacts, ensuring sustainable
natural resources); or environmental outcomes (e.g., restoration or remediation of perturbed systems).
The types of metrics will differ depending on the goal, e.g., the audience to be influenced.
21
-------
Table 3. Potential Evaluation Measures for the STAR WW Program
POTENTIAL STAR WW
GOALS
• B>M
-------
3.4 Integration of Ecological, Physical, and Social Sciences
Charge Question 3: Is the requirement that grant proposals integrate
ecological, physical and social sciences producing a unique body of research?
Would funding each of the "circles" [in the Venn diagram] individually have
the same outcome? Is the integrated approach so important that it is giving us
new insights into decision making at the watershed scale?
Summary Response: Yes, the requirement to include ecological, physical and
social sciences in most of the STAR Water and Watersheds projects to date has
produced a unique body of research. The STAR WWprogram, both in its focus
and its interdisciplinary nature, provides a source of research funding that is
rare within federal research programs. Funding projects within the individual
discipline groups (or "circles") would not have produced the same results.
Most of the progress made by STAR WW projects occurred at the interface
between disciplines and/or at the interface between scientists and stakeholders.
In particular, the integration of socioeconomics and management issues into
watershed research is a very encouraging, unique and beneficial aspect of the
STAR WW program.
The requirement to integrate social, physical, and biological sciences in each
project appears, however, to be forcing too much homogeneity among projects
while precluding some important areas ofwatershed research. The Panel
recommends that future STAR WW Requests for Applications retain some, but
not exclusive, emphasis on interdisciplinary projects, and that they allow the
mix of disciplines to be determined primarily by the important science questions
that need to be answered. A portion of program funding could be set aside to
ensure support for projects that include social science research and one other of
the discipline areas. Similarly, a portion of program funding might be reserved
for a few large, multidisciplinary projects. For these projects, the Agency
should consider providing planning grants, an increased level of funding, and
longer grant periods, commensurate with the additional complexity of the
proposed research.
3.4.1 Defining Integrated Research
An integrative approach to the study of watersheds is essential to progress in understanding
how watersheds function and to developing management tools that will allow both humans and
ecosystems to prosper. Further, an integrated understanding of both the complex sources of stressors
and the multi-dimensional nature of how stresses are initiated and ultimately controlled, could only come
from a strongly interdisciplinary research program. The Panel recommends that the STAR WW
program continue, and that the interdisciplinary nature of the projects and the emphasis on projects with
stakeholder involvement both be retained.
23
-------
The Panel has some concern, however, that the STAR WW program's definition of
"integration" in its recent RFAs, i.e., the intersection of three circles of a Venn diagram (Figure 1), may
be too restrictive a definition of integrated watershed science. Interesting and important watershed
science questions may be proposed within the intersection of two, rather than three, of the research
categories. For example, the most productive areas of integration appear to be those at the interface of
the ecological and social sciences, and the STAR WW may wish to fund such work even in the
absence of physical research questions.
While the Panel feels that it is critically important to retain an integrative approach to watershed
research within EPA, the current STAR WW definition of integrated research forces too much
homogeneity among projects. As investigators struggle to fit their research programs within the
intersection of the three designated discipline groups, the types of studies being done actually may be
narrowed. While a number of the STAR WW researchers discussed how to conduct interdisciplinary
projects and how to work with stakeholders, the Panel heard little discussion of the fundamental
questions in watershed science or state-of-the-art hypotheses in any of the three designated disciplines.
Absent a conceptual framework for integrating the required disciplines, most investigators apparently
developed hypotheses within their discipline areas and combined them later. This resulted in projects
with a clumping of disciplinary groups, but often without significant work at the intersection of the
"circles." It appears that research excellence is being sacrificed somewhat to accommodate formation
of large interdisciplinary teams.
The Panel applauds the STAR WW program for its efforts to bring together historically
separate disciplines that generally act independently. However, the absence of an existing conceptual
framework other than the Venn diagram to drive this research was a limitation. If an acceptable
conceptual framework for these types of studies can be extracted from the existing project results (see
Section 3.7.1), then it can be used to drive the next generation of funding. Until that time, the Panel
recommends the following:
a) Eliminate the requirement that every future project include all elements of the Venn
diagram.
The use of the Venn diagram to drive integration was a worthwhile aspect of previous funding
efforts. It seems fair to say that this requirement drove innovation, even though the challenge of truly
operating at the intersection of the three circles was not well met by the researchers. As the STAR
WW program moves forward, however, the research program should be refocused around
fundamental issues in watershed science, rather than on funding integrated research per se.
Interdisciplinary research will probably still be common under this scenario.
Future RFAs should seek proposals that target specific technical needs identified by analyzing
the existing STAR WW portfolio, and request project proposals that target known gaps in knowledge
in ecological, physical, or social sciences. The option to address any subject area where researchers
can show that end results will be utilizable both in a particular location and across watersheds should be
open. The criteria for whether a study is eligible for funding also should include its potential to advance
the overall practice of integrated research on watersheds.
24
-------
b) For the most complex interdisciplinary projects (e.g., Area 1 of the Venn diagram),
consider the use of planning grants, an increased level of funding, and longer grant
periods.
Ambitious, innovative efforts such as the STAR Water and Watersheds program that seek to
integrate research activities bring with them some risk that good science will be done, but not in a truly
integrated fashion. Projects that focus on integration across the entire Venn diagram could benefit from
a step in which they compete for small planning grants. These grants would be used to build a solid
integrative proposal. This step would reduce the risks of failed approval of submissions for the
researchers, and reduce the Agency's overall risk that supported research will not deliver on the
promised integration.
In addition, interdisciplinary research requires more time and resources than more traditional
disciplinary research because of the need to develop research teams and relationships, develop
hypotheses that are informed by the perspectives of multiple disciplines, and conduct integrated
analyses of research results. The Panel is concerned that the current STAR WW program may not
provide either the time or funding necessary for such research. Projects of the complexity required by
the recent STAR WW RFA have high overhead costs associated with project management that are not
fully supported at the current level of funding and the quality of research appears to have suffered as a
result. The most successful efforts are the ones where interdisciplinary groups were already in existence
and STAR WW funding built on past and existing funding. This suggests that the funding level for STAR
WW projects (approximately $1 million) is not high enough to fund integrated efforts by a large number
of investigators, and/or that three years is not enough time to meet the program objectives when multi-
disciplinary groups arise de novo.
If, as recommended above, the Agency relaxes the requirements on the extent of
interdisciplinarity required, it may be possible to enhance the success of the more complex
interdisciplinary projects by offering small planning grants, providing additional funding, allowing longer
time frames for project completion, or by other means. The Panel supports the current Agency policy
of providing no-cost extensions automatically at the first request, and potentially for a second year if
there is progress with a possibility of renewal.
c) Continue to encourage integration of social and natural science research by setting
aside some portion of STAR WW funds for projects that integrate aspects of human
systems with non-human components of watershed ecosystems.
Physical and ecological scientists historically have worked together to do research on
watersheds, and so the inclusion of social sciences in the STAR WW program in particular has added a
valuable new perspective. In some cases, the social sciences requirement of the program has produced
insights into decision-making on watershed management that would not have arisen from an ecological
research focus alone. In other cases, however, the social science aspects of the projects seemed to be
conducted in parallel, rather than being truly integrated, with the ecological research. Future projects
25
-------
that include a social sciences component should both better integrate social and natural sciences and
improve the quality of the social science component.
3.4.2 A Need for Integrated, Basic Research
The Panel believes that a primary motivator behind the current requirement for the inclusion of
social science research in all STAR WW projects is the recognition of the influence of humans on
watersheds and the desire to understand the linkages among human and natural systems. This
understanding should lead to both policy relevant and integrated science insights. While it will usually
be the case that integrated research will be more applicable to decision-making, integrated research is
not synonymous with applied research. Even in the STAR WW scheme, there is a role for integrated,
basic research on, for example, ecological system processes. While basic, integrated watershed
science may not yield decision support tools or other applications in the near term, such research is
critical in the longer term for improving our understanding of how ecological systems function.
The relationship between interdisciplinary research (including social, economics, ecological, and
hydrological research) and applicability of research (including utility for stakeholders) can be used to
define different categories of research (Figure 3). Populating different sectors of this integration-
applicability space with STAR WW projects has been accomplished by the design of the RFA over the
life of the program. In 1995, the STAR WW competition produced several disciplinary-basic research
projects; whereas in 1996 interdisciplinary work was
encouraged, moving the research effort towards
integrated, basic (IB) projects and some integrated,
applied (IA) projects. After 1997, RFA emphasis
on both interdisciplinary and relevancy has driven the
proposals in both directions towards the integrated,
applied (IA) research quadrant (Figure 3) with slight
variation in focus (e.g., restoration, TMDL).
DkciplMiary,
The current STAR WW program emphasis
has resulted in interdisciplinary, applied research
topics, but fewer IB projects are currently being
conducted in STAR WW. It is important to
recognize that science of types IB and disciplinary-
applied (DA) should still attract the interest of the
STAR WW program. Two examples come to mind:
type IB projects to improve our basic understanding of the human or ecological processes being
modeled are crucial to the success of the applicability and integration; and type IB projects that attempt
to construct unified theoretical underpinnings of integrated watershed science are important, even
though their applicability may occur at a later time.
Figure 3. Interdisciplinary and Policy-
Relevant Research
26
-------
3.5 Major Advancements or Breakthroughs
Charge Question 4: As a result of the Water and Watersheds program, do we
see any major advancements or breakthroughs in watershed science or
interdisciplinary integration across the relevant disciplines?
Summary Response: The Panel did not see evidence of major breakthroughs in
watershed science, but did conclude that STAR WW was producing valuable
opportunities to link the natural and social sciences relevant to watershed
assessment and management. Given the emphasis on interdisciplinary
research, which requires additional time and effort by researchers, it may be
too early to expect major advances in interdisciplinary integration. Advances
from the currently funded projects are likely to take the form of integrated
application of existing models and more refrned decision tools for watershed
management.
3.5.1 Watershed Science
Based on the information provided by the Agency and the STAR WW researchers, it appears
that the major advancements stimulated by the STAR Water and Watersheds Program have been
largely in the form of new integration of traditional approaches to studying watersheds. Many of the
projects combined watershed ecological or hydrological models with spatial information summarized
using geographic information system (GIS) technologies. Often, the results of social assessments (e.g.,
surveys, stakeholder groups) were incorporated into this framework with the objective of developing
decision support capabilities. Such integration likely will provide valuable support to decision-making
and watershed management. However, examination of the individual project components (e.g.,
physical, ecological, or hydrological) failed to identify any significant breakthroughs or compelling
intellectual advancements in the underlying science. The measures of water quality (e.g., nutrients,
PCBs, E. co//'), ecological integrity (e.g., biodiversity, Index of Biotic Integrity), and watershed
processes, for example, appear to be fairly standard in concept and practice. The described modeling
approaches either modify existing watershed models (e.g., HSPF), or develop traditional
compartmental models (with their corresponding strengths and limitations). The spatial modeling
approaches reported in several abstracted project descriptions have been used for about 10-15 years.
Given the emphasis on interdisciplinary integration, the ability of STAR WW to produce major
advances in watershed science may not be the right measure of program success. The adaptation and
application of models to different analysis scenarios and data sets, while a more incremental approach
to science, is valuable nonetheless. For this approach to add up to a contribution to watershed science,
however, there would have to be a conscious effort to bring the different models and results together
and see what could be learned in the aggregate.
27
-------
While acknowledging that it is likely too early to assess the extent to which the STAR WW
program will produce breakthroughs in watershed science, it is timely to consider whether the program
is configured so as to lead to this result in the longer term. In general, the STAR WW projects did not
seem to have identified either gaps in watershed science or unanswered questions, which the project
then targeted for study. Rather, teams of researchers seemed to have formed for the express purpose
of seeking STAR WW funding. Thus, the capacity of the STAR WW program to produce advances in
watershed science sometimes seemed limited by the way in which the teams were formed.
One important research issue highlighted by several of the STAR WW projects was the
apparent mismatch between the scale of the hydrology models and the scale of the land use decisions,
social institutions, and other aspects of the watershed. This showed up in the reliance on stream
segment approaches, for example, in contrast to the more integrative concept of the "catchment."
Having hydrological models focused on stream segments greatly reduces the capacity of these projects
to contribute to advancing watershed science in a more interdisciplinary context, partly because of the
mismatch in scales of analysis. This issue could be addressed in both the requirements stated in future
RFAs and the evaluation criteria for proposal review.
3.5.2 Interdisciplinary Integration
As noted above, the development of interdisciplinary research teams takes time, and thus
advancement in interdisciplinary work was most evident in cases where the research team and the
project had been initiated prior to STAR funding. Sometimes the project, like the Baltimore Long
Term Ecological Research (LTER) project, had been initiated several years before receiving STAR
WW funding, but the STAR funding fit perfectly the kind of integrated, interdisciplinary work intended
within an LTER site. Not surprisingly, teams put together specifically for STAR WW grants achieved
less integration in their research. In presentations where individual researchers seemed to have
conceptualized and worked on "their piece of the problem," then little integration was evident. In
projects where the team pre-dated the STAR WW grant or specifically worked to focus on integrated
results, then there was more evidence of integrated outcomes. Of course, while the Panel members
listened to the presentations given at the STAR WW researchers' meeting in San Francisco and
reviewed the written materials provided, what we could not know was the extent to which new
relationships among interdisciplinary scientists were initiated or strengthened through STAR WW
funding. The panel did see evidence that collaborations begun due to STAR WW funding requirements
had been continued because they were interesting and useful to the researchers. On the whole, it seems
that the STAR WW program is having a positive impact in moving toward more interdisciplinary
approaches in watershed science, policy, and practice.
The Panel acknowledges the scarcity of publication outlets for interdisciplinary research. Yet,
after reviewing the articles attributed to the STAR WW grants to date, it appeared that most of the
publications were fairly traditional in scope and content. It is difficult to know if this is the result of the
constraints of publication requirements or the result of scientists writing that which is most familiar to
them. Either way there needs to be greater emphasis on developing publishable papers that are based
on interdisciplinary contributions - not simply disciplinary pieces of the larger project - and attempts to
28
-------
publish in the journals available to interdisciplinary work. This means that researchers have to figure out
how to write together, a process that will greatly enhance their ability to work across disciplines.
An opportunity is also available in the form of a significant comparative "data base" of STAR
WW projects for analysis. Whatever the suggestions for improved integration, these STAR WW
projects represent a significant investment of research funding and researcher time and commitment.
There is much to be learned by looking across the STAR WW projects in systematic ways. For
example, the San Francisco researchers' meeting offered the opportunity to both engage researchers in
dialogue across projects as well as to include the large number of watershed scientists, policy makers,
and practitioners across all groups in a dynamic discussion. It was unfortunate that neither occurred
given the time and cost of the meetings. The opportunity to take what has been learned from STAR
WW projects to date and use it to refine or critique models or proposed policy interventions should not
be lost.
29
-------
3.6 Perception of STAR WW Within and Outside the Research Community
Charge Question 5: How is the STAR WW program perceived within and
outside the research community?
Summary Response: In general the Panelists felt that the data provided in the
pre-meeting materials plus their experience at the STAR Progress Review did
not provide a factual basis to assess how the STAR WW program is viewed by
the rest of the research community. The Panel agreed, however, that a positive
answer to this question would be a significant indication that the STAR WW
funding was achieving its goal of expanding the appreciation of integrated
research on water shed management. Measures of awareness and acceptance of
STAR WW research could be developed as part of a more comprehensive
evaluation of program success.
In general the Panelists felt that the data provided in the pre-meeting materials plus their
experience at the STAR WW Progress Review did not provide a factual basis to assess how the STAR
WW program is viewed by the rest of the research community or by those outside the research
community most likely to use the research findings. Based on its own experience, however, the Panel
concluded that the STAR WW program likely is viewed as a valuable, and largely unique, source of
funding to address key questions that need to be answered to manage and conserve watershed values.
Some panelists also felt, from anecdotal data, that some members of the research community are
struggling with the value of developing funding requests on such complex projects with low probability
of success. That said, STAR WW publication lists and web citations described earlier demonstrate
that respectable within-discipline research is being done. The dearth of "integrated" interdisciplinary
publications may be a reflection of the relative scarcity of journals devoted to such papers, although
there are journals, such as Environmental Science and Technology, that actively seek such papers.
With regard to the appreciation of the STAR WW work outside of the researchers involved,
the Water and Watersheds research is more useful to local and state agencies, elected officials, and
community stakeholders than to federal program managers. The appreciation of this work and its
value, therefore, likely would be greatest among local stakeholders in the watershed and the regional
authorities responsible for its management. This may well be appropriate, however, since many of the
decisions that directly affect a watershed's ecological condition are made at the local and regional
governmental levels. As with the research community, however, there was little reported on the
measurement of awareness and satisfaction of stakeholders with the technical approaches employed by
STAR WW researchers.
If the Agency wishes to understand the level of recognition and appreciation of STAR WW
research, a systematic effort might be made to gain feedback from both the research and non-research
communities. Over time, as STAR WW results are published in the peer reviewed literature and other
researchers begin to cite them, it will be possible to assess the reaction of the general research
community to the program. With regard to non-research communities, funded researchers could be
30
-------
asked to collect this information as an assessment of their research at the local and regional level. For
example, periodic questionnaires to community participants in the project, and potential users of the
information and other researchers could provide a basis for evaluating a project's strengths and
weaknesses. By compiling this information for all (or a subset of) projects, the value of the STAR WW
program to the non-research community could be assessed.
31
-------
3.7 Enhancing the Value of Current STAR WW Projects
The Panel concluded that the Agency could and should do more to extract value from the set of
existing STAR WW projects. Suggested activities include supporting cross-project synthesis of
findings on a variety of issues and enhancing communication of research findings to the larger research
community and to potential users.
3.7.1 Cross-Study Evaluation of Projects
To its credit, the STAR program staff have already recognized the value of extracting
knowledge from the STAR WW researchers as a group. The workshop and subsequent report on
lessons-learned, for example, yielded some good generalizable information that can be used by future
researchers. The STAR program should continue to extract knowledge from the entire set of WW
projects by a more expanded cross-study evaluation. The Panel encourages periodic repetition of the
"lessons learned" workshops as a means of distilling common themes and findings from STAR WW
projects. It would be useful to have periodic workshops to critique and synthesize particular sets of
tools (e.g., DSS, modeling approaches) so as to consolidate and make widely available (through
publication and web dissemination) the collective wisdom of different approaches to problems that are
being addressed by the research teams. An alternative approach would be to ask an organization such
as the National Center for Ecological Analysis and Synthesis (a Center funded by NSF and the
University of California) to have a working group target specific cross-cutting questions, compare the
results of different research projects, and summarize and disseminate any conclusions that can be
drawn. Contracting one person to do a peer reviewed synopsis or state-of-the-science report on a
topic likely will not be adequate to the need.
Cross-project evaluations and synthesis of STAR WW findings should:
a) Identify major gaps in understanding or information (e.g., the need for a standard
classification system for the aquatic ecosystems that comprise watersheds; the need for
tools for comparative analysis of the Net Environmental Consequences of various
watershed management options) that can be targeted in future grants.
b) Identify future watershed research priorities. The STAR WW research should not only
create innovation at the margins of the integration between disciplines but should also
highlight the principal scientific weaknesses in the elements of the experimental design
and supporting models. Looking broadly across the STAR WW research projects and
focusing on where there were barriers or limitations to successful integration would lead
to identification of areas for future research funding. The researchers themselves are
best able to identify those barriers and to provide a prioritized list of recommended
basic research to improve the ability to integrate with confidence in the future.
c) Identify improvements in data collection, analysis and modeling in association with
watershed science so that integration with social and economic disciplines can be
optimized.
32
-------
In addition, cross-project evaluations could be used to:
a) Develop a database that links STAR WW projects and associated public policy
choices or decisions that the research findings will help to inform. (Input could be
derived in part from researchers' responses to questions such as those included in
Figure 2.) The database might be made accessible from a web site, and could provide
links to the specific projects for those seeking more information.
b) Deduce the common elements of a framework for integrated watershed research that
might provide the basis for defining and addressing fundamental interdisciplinary science
questions important to the Agency. A trans-disciplinary conceptual model could be
extracted by overlaying the designs of the various projects to identify the common
elements and the sequence of their execution. This extracted framework could then be
the backbone for further analysis of the work to identify gaps in knowledge.
c) Develop a best practices design manual for integrated assessments of watersheds that
brings together lessons learned from STAR WW projects. Such a guide might include
suggestions for project design and management, integrated analysis, stakeholder
involvement, and so forth.
3.7.2 Building Capacity for Integrated Approaches
An important result of the STAR WW funding of integrated studies on watersheds should be
that it builds capacity for design and execution of integrated studies and use of results to make
decisions. In its current form, the program takes a very critical step towards advancing society's ability
to manage watersheds in an integrated fashion. Additional, more aggressive efforts to communicate the
results of the programs and formalize research networks would go a long way towards assuring that
such integrated work is supported during the early stages of development.
The STAR WW program efforts to date have been useful in bringing together the principal
investigators and some of their team members to share their research and experiences on the difficulties
in working across disciplines. Yet, STAR WW managers and researchers could and should do more
to ensure that results of the research are expeditiously disseminated in useful form to communities, and
to business, government, and science sectors. Proposals should be encouraged to include a plan to
disseminate results and tools in a useful form (e.g., in a web-compatible format for posting by the EPA
or a research center with a durable web site, with documentation, and with training). While the Panel
agrees that researchers have an obligation to publish in scientific journals, this should not be the sole
means of communicating research results. The Agency also has a responsibility to further disseminate
research findings and possible applications to watershed managers and decision-makers by, for
example, organizing regional workshops, training, disseminating STAR WW materials on web sites, and
developing state-of-the-science reports derived through multi-expert synthesis and peer review.
The Panel also urges the Agency and the STAR WW researchers to give attention to the
application of results and external communication of results to decision-makers and practitioners in
33
-------
agencies, organizations, etc. Grant reports should explicitly address how the results may be applied to
watershed management problems. The potential for a grant to enhance watershed management not
only in the geographic study area of the grant but elsewhere should be a significant factor considered in
the evaluation of grant proposals, and transferability should be reassessed as results become available.
To improve dissemination of STAR WW results and build capacity for integrated approaches,
the Panel suggests the following:
a) Require Communication Plans
The STAR program should require investigators to develop communication plans that include
publication in the peer-reviewed literature, web-based products and summary documents for other
scientists, EPA program managers, and the public, and production of working decision tools when that
is a goal of the project. Project budgets should include the cost of an effective communication plan and
therefore funding for these projects may need to be increased accordingly.
b) Document DSS and Other Tools
Many of the STAR WW researchers are developing decision support systems and other tools.
Many researchers are also developing improvements to existing modeling techniques, or even in some
cases new modeling techniques. If these grants are to have impacts to other watersheds outside the
study area, it is essential that any tools and models developed be fully documented with adequate
metadata so that they could be used elsewhere by a technically competent person not involved in the
original project. The tools and models should be made freely available on the web by EPA, or at least
be linked to EPA's web site if housed on a relatively permanent site elsewhere.
c) Create Fora for Presentations of Integrated Research
Professional societies or journals that are dedicated to the individual disciplines usually are not
effective venues for individual submissions of highly integrated studies. Thus, the Agency might
consider supporting special sessions dedicated to STAR WW integrated research at meetings of
professional societies (e.g., Society of Environmental Toxicology and Chemistry, Ecological Society of
America, or Society for Risk Analysis), or holding an annual STAR WW researchers' review in
conjunction with an existing scientific meeting as a means of expanding the awareness of STAR
research in the scientific community. The Agency also should support supplemental issues devoted to
STAR WW research in journals dedicated to integrated studies.
d) Integrate Intramural and Extramural Research
EPA's mandate to strengthen the quality of internal science could be aided by finding ways to
have EPA scientists and managers become more aware of WW projects and by strengthening
interactions between Agency scientists and STAR WW researchers. One means of enhancing the
integration of the Agency's intramural and extramural research programs relating to watersheds would
be to use more effectively the annual STAR WW researcher meetings. There were few EPA
34
-------
researchers present at the STAR WW meeting attended by the Panel. The Agency should consider a
different format for the meetings that focuses researchers on lessons learned, innovations, and
breakthroughs (even minor) in thinking and research. The target audience for the meetings should be
broadened to include EPA researchers, regional program offices, and state and local regulators, as well
as interested researchers from watershed programs in other agencies (e.g., U.S. Geological Survey's
National Water-Quality Assessment Program-NAWQA). Another means of enhancing awareness of
STAR WW research within the Agency would be to bring "best practice" STAR WW researchers into
the EPA regional and program offices to present research applicable to the particular offices (e.g.,
research conducted in specific regions could be presented to that EPA regional office).
To increase engagement in scientific communities, the Agency should support greater interaction
between STAR program managers, funded STAR researchers, and others in the research communities,
including greater participation by STAR program managers in national and international scientific
meetings. Greater interaction with the scientific community will enhance the value of STAR managers to
synthesize program results and to disseminate that information to management and regulatory
communities.
e) Consider Establishing Formalized Research Networks
At this point there does not appear to be a formal mechanism to regularly bring researchers
interested in the integrated assessments together in a trans-disciplinary meeting. Creating a standing
research network and a web-based clearinghouse could stimulate such regular and formal interaction,
both physically and virtually. The STAR WW program, either on its own or in partnership with NSF or
others, could fund the development of formal networks. NSF, for example, has funded a network
called ecological circuitry. The NSF network is designed to bring together both the thought leaders in
an area and their graduate students to work on common projects. Although there is a lead organization
that coordinates the network, the staff of the organizations physically spend time at other network
member facilities. The NSF network may not be as interdisciplinary as the integrated watershed
research might demand, but it should serve as a worthwhile model to consider. Given the fact that
many of the STAR projects have a local to regional flavor, STAR WW also might organize networks
on a regional (e.g., Northwest) or on an ecotype (e.g., arid lands, forested watershed) basis.
f) Support Web-based Distribution of Information Related to STAR WW Studies
The STAR program should facilitate the communication of both the results of individual projects
as well as the results of lessons learned via the Internet. This facilitation could occur either by setting
the expectation as part of the individually funded projects or such web-based systems could be
established as part of the scope of any research networks. The web-based system can act as a
clearinghouse of information on best practices, tools, and other lessons learned.
35
-------
4. SUMMARY RECOMMENDATIONS
Charge Question 6: What changes would [the Panel] recommend to the [STAR WW]
program managers?
Based on the materials provided by the Agency, and the STAR WW researcher presentations
in San Francisco, the Panel suggests a number of mid-course correction to the STAR WW program.
These recommendations are discussed in the previous sections in response to the charge questions and
summarized below:
Recommendation 1: The Panel strongly recommends that STAR WW be retained as a major, focused
program within EPA (see Section 3.1).
Recommendation 2: In order to meet the Program's stated objectives, the Panel recommends that
STAR WW Requests for Applications focus more on fundamental issues in watershed science, rather
than on funding integrated research per se. Interdisciplinary research will probably still be common
under this scenario. Specifically, the Agency should:
a) Pursue a more balanced approach to addressing the program's objectives. The Panel
notes that STAR WW projects have focused primarily on anthropogenic processes,
water quality issues, biotic integrity measures, and human-dominated systems. The
Panel recommends placing additional emphasis on natural systems and reference
conditions, on the understanding of water quantity issues, and on ecosystem processes
and dynamics related to the maintenance of native communities and species. (See
Section 3.2)
b) Retain some, but not exclusive, emphasis on interdisciplinary projects, and allow the
mix of disciplines to be determined by important and relevant science questions that
need to be answered. In particular, replace the Venn diagram with a broader definition
of interdisciplinary research, and fund projects that only include one or two disciplines if
the projects address important gaps in our understanding of watersheds. Within these
guidelines, continue to emphasize the integration of social sciences with ecological
research. (See Section 3.4)
c) For a small number of particularly complex, truly integrated, multi-disciplinary projects,
consider the use of planning grants, an increased level of funding, and longer grant
periods. (See Section 3.4)
Recommendation 3: The Panel believes that benefits from the existing STAR WW research grants and
practical application of research results could be significantly enhanced (see Section 3.7). Steps to do
this include:
36
-------
a) Cross-study evaluations to analyze and synthesize the results of groups of projects
(e.g., through convened panels of internal and external scientists);
b) Disseminate research results in useful forms (e.g., peer-reviewed literature, web-based
products, simplified glossy products) to business, government, and science sectors;
c) Improve delivery of extramural research results to Agency scientists and program
managers;
d) Continue to build capacity for trans-disciplinary work related to the Agency's mission
by, for example, enhancing inter-disciplinary and inter-project thinking and
communications; and
e) Provide support for fuller engagement of EPA STAR program managers in relevant
scientific and management communities, and for increased interaction with funded
scientists.
Recommendation 4: If the Agency desires a more methodical measure of STAR WW benefits in the
future, the Panel suggests that the Agency identify sets of measures that correspond to the specific
program objectives to be achieved, then determine means of gathering information on the measures.
The Panel provides examples of such measures in this report. (See Section 3.3.3)
37
-------
LITERATURE CITED
Morgan, G. and M. Henrion. 1994. Uncertainty. Cambridge University Press.
National Research Council. 1999. Evaluating Federal Research Programs: Research and the
Government Performance and Results Act. Committee on Science, Engineering, and Public
Policy. National Academy Press, Washington, DC.
U.S. EPA Science Advisory Board. 2000. A Joint SAB/BOSC Report: Review of the Science to
Achieve Results (STAR) Program. EPA-SAB-EC-00-008. March 2000.
U.S. Environmental Protection Agency. 1999. Responses to Self-Study Questions for Board of
Scientific Counselors' Review of ORD's Science to Achieve Results (STAR) Program. Office
of Research and Development, December 1999.
U.S. General Accounting Office. 2000. Environmental Research: STAR Grants Focus on Agency
Priorities, but Management Enhancements Are Possible. RCED-00-170. September 11,
2000.
R-l
-------
APPENDIX A: SUMMARY OF PROJECTS FUNDED BY THE STAR
WATER AND WATERSHEDS PROGRAM
(Source: NSF-EPA Partnership for Environmental Research web site atwww.nsf.gov/home/crssprgm/ unless
otherwise noted. *Source: EPA/ORD)
1995 STAR Water and Watersheds
PROJECT TITLE
A Comparative Institutional Analysis of
Conjunctive Management Practices
Among Three Southwestern States
A Comparison of Agricultural vs
Forested Basins: Carbon and Nutrien
Cycling within the Hyproheic Ecotone
of Streams
Alternate States and Ecosystem
Metabolism in Lakes: Interactions of
Nutrients and DOC
An Ecoregion-Specific Comparison of
Stream Community Responses to
Nutrient Gradients Using Both Survey
and Experimental Approaches
Carbon Exchange Dynamics in a
Temperate Forested Watershed: A
Laboratory and Field Multidisciplinary
Study
Characterization of Metal Ion
Complexation and Aggregation of Humic
Substances
Characterization of Metal Ion
Complexation and Aggregation of Humic
Substances
Contemporary Water and Constituent
Balances for the Pan-Arctic Drainage
System: Continent to Coastal Ocean
Fluxes
Principal
Investigate r(s),
Institution
EdellaC. Schlager
University of Arizona
School of Public
Administration and
Policy
DavidS. White
Murray State
University
Hancock Biological
Station
James F. Kitchell
University of
Wisconsin,
Madison
Robert Jan Stevenson
University of
Louisville
Department of Biology
Lynn Walter
University of
Michigan
Department of
Geological Sciences
W. Robert Carper
Wichita State
University
Chemistry Department
Cynthia K. Larive
University of Kansas,
Lawrence
Chemistry Department
Bruce J. Peterson
Marine Biological
Laboratory
State
AZ
KY
WI
KY
MI
KS
KS
MA
NSF
Number
9524483
9524721
9509595
9524759
9524454
9524865
9524514
9524740
EPA/
USDA
Number
EPA
R824781
EPA
R824786
EPA
R824783
EPA
R824978
Funding
($)
198,000
300,000
763,403
376,200*
800,000*
143,001
221,401
959,987
A-l
-------
Detecting Fecal Contamination and Its
Sources in Water and Watersheds
Development and Application of
Spectroscopic Probes for Measurement
of Microbial Activity in Aquatic
Ecosystems
Development of Geomorphological
Artificial Neural Networks (GANNs) for
Modeling Watershed Runoff
Diffusion Rate Limitations in
Heterogeneous Porous Media: Model
Structure, Scale and Geologic
Characterization
Environmental Change and Adaptive
Resource Markets: Computer-Assisted
Markets for Water Allocation
Fluvial Responses to Climate Change
and Human Activities in Burgundy,
France
Formation and Propagation of
Large-Scale Sediment Waves in
Periodically Disturbed Mountain
Watersheds
Geomorphic, Hydrologic and Ecological
Connectivity in Columbia River
Watersheds: Implications for
Endangered Salmonids
Influences of Watershed Land Use on
Stream Ecosystem Structure and
Function
MarkD. Sobsey
University of North
Carolina, Chapel Hill
Department of
Environmental
Sciences and
Engineering
Carol Arnosti
University of North
Carolina, Chapel Hill
Curriculum in Marine
Sciences
Rao S. Govindaraju
Kansas State
University
David L. Freyberg
Stanford University
Department of Civil
Engineering
VernonL. Smith
University of Arizona
Michael D. Blum
Southen Illinois
University
Gary Parker University
of Minnesota
St. Anthony Falls
Hydraulic Lab
Hiram W. Li
Oregon State
University
Department of
Fisheries and
Wildlife
Patricia F. McDowell
University of Oregon
Department of
Geography
Judy L. Meyer
University of Georgia
Institute of Ecology
NC
NC
KS
CA
AZ
IL
MN
OR
GA
9524535
9524268
9524758
9524430
9409525
9506643
9524358
9524854
9524819
EPA
R824782
EPA
R825159
EPA
R824768
EPA
R824779
EPA
R824773
+
R824774
EPA
R824777
400,000
405,811*
198,000
9,960
280,000
891,052*
500,000
A-2
-------
In-Situ Assessment of the Transport
and Microbial Consumption of Oxygen
in Ground-water
Integrated Ecological Economic
Modeling and Valuation of Watersheds
Integrating Planning, Forecasting, and
Watershed Level Ecological Risk
Assessment Techniques: A Test in the
Eastern Cornbelt Plains Ecoregion of
Ohio
Modeling Temporal Rainfall via a Fractal
Geometric Approach
Norwalk Virus-Like-Particles (VLPs) for
Studying Natural Groundwater
Disinfection
Resistance of Communities to Chronic
Haloaromatic Contamination from
Biogenic and Anthropogenic Sources
Response and Compensation to a
Bivalve Invasion by an Aquatic
Ecosystem
Tadashi Yoshinari
New York State
Department of
Health
Wadsworth Center
Research
Laboratories
Robert Costanza
University of
Maryland - Center for
Environmental
Science
Chesapeake Biological
Lab
Steven I. Gordon
Ohio State University
Department of City
and Regional
Planning
CarolosE. Puente
University of
California, Davis
Land, Air & Water
Resources
Hydrologic Science
Program
Mary C. K. Estes
Baylor College of
Medicine
Division of Molecular
Biology
Stanley B. Grant
University of
California, Irvine
Department of Civil
and
Environmental
Engineering
David E. Lincoln
University of South
Carolina
Department of
Biological Sciences
David L. Stray er
Institute of
Ecosystems Studies
NY
MD
OH
CA
TX
CA
SC
NY
9524305
9525573
9524398
9524755
9524481
9524703
9508981
EPA
R824787
EPA
R824766
EPA
R824769
EPA
R824780
EPA
R824775
+
R824770
EPA
R824776
346,500*
997,000*
445,000
198,000
700,000*
465,300*
900,000
A-3
-------
Scaling Up Spatially Distributed
Hydraulic Models of Semi-Arid
Watersheds
The Role of Colloidal Particles in the
Transport of Chemicals through an
Agricultural Watershed
The Role of Hg(II) Reduction and
Chemical Speciation in Controlling the
Concentration of Mercury and its
Methylation in Natural Waters
The Role of Ling-Lived Zooplankton
Diapausing Eggs Response and
Recovery of Impacted Lakes
The Role of Oyster Reefs in the
Structure and Function of Tidal Creeks
Towards a Model of the
Biogeochemistry of Large-Scale River
Basins: An Application to the Pacific
Rim
Tracing the Fate of Nitrogen Inputs from
Watersheds to Estuaries
Traveling Wave Behavior During
Subsurface Transport of Biologically
Reactive Contaminants: Implications for
In-Situ Bioremediation
Variability of Dissolved Trace Elements
in Rivers and Streams: Seasonal Redox
Effects
Water and Sustainable Development in
the Binational Lower Rio Grande/Bravo
Basin
David G. Tarboton
Utah State University
Department of Civil
and Environmental
Engineering
George M. Hornberger
University of
Virginia Department of
Environmental
Sciences
Francois M. M. Morel
Princeton University
Department of
Geological
and Geophysical
Sciences
Nelson G. Hairston, Jr.
Cornell University
Ecology and
Systemic s
Eric T. Koepfler
Coastal Carolina
University
Jeffrey E. Richey
University of
Washington,
Seattle
Linda A. Deegan
Marine Biology
Laboratory
Ecosystems Center
Albert J. Valocchi
University of Illinois,
Urbana-Champaign
Department of Civil
Engineering
Alan Shiller
University of
Southern Mississippi
Jurgen Schmandt
Houston Advanced
Research Center for
Global Studies
UT
VA
NJ
NY
SC
WA
MA
IL
MS
TX
9524405
9524352
9524644
9524583
9509057
9524524
9524297
9524432
9508199
9524748
EPA
R824784
EPA
R824772
EPA
R824778
EPA
R824771
EPA
R824767
EPA
R824785
EPA
R824799
330,000
500,000
349,950
350,000
405,000
93,260
230,000*
200,000
50,000
785,539*
A-4
-------
Watersheds and Wetlands: Large Scale
Disturbances and Small Scale
Responses
Charles Cole
Pennsylvania State
University
Environmental
Resources Research
Institute
PA
9524350
EPA
R824905
742,079*
A-5
-------
1996 STAR Water and Watersheds
PROJECT TITLE
An Integrated Approach to Assessing
Water Management Options in Major
Watersheds: Extending a
Hydrodynamic, Water Quality Model to
Include Biological and
Politico-Economic Components
Effectiveness of Regulatory Incentives
for Sediment Pollution Prevention:
Evaluation Through Policy Analysis
and Biomonitoring
Geochemical, Biological, and Economical
Effects of Arsenic and Other Oxyanions
on a Mining Impacted Watershed
Influence of Forest Fragmentation on
Watershed Functions in Northern
Vietnam
Integrated Urban Watershed Analysis:
The Los Angeles Basin and Coastal
Environment
Integrating Modeling and Management
of Agriculturally-Impacted Watersheds:
Issues of Spacial and Temporal Scale
Modeling Effects of Alternative
Landscape Design and Management on
Water Quality and Biodiversity in
Midwest Agricultural Watersheds
Strategic Renewal of Large Floodplain
Rivers
Principal
Investigate r(s),
Institution
Paul A. Sabatier
University of
California, Davis
Division of
Environmental Studies
Seth Reice
University of North
Carolina,
Chapel Hill
Department of Biology
Glenn C. Miller
University of Nevada,
Reno
Department of
Environmental and
Resource Sciences
A. Terry Rambo
East-West Center
Program on
Environment
Richard Turco
University of
California, Los
Angeles
Institute of the
Environment
Patrick Brezonik
University of
Minnesota Water
Resources Research
Center and
Department of
Civil Engineering
Mary Santelmann
Oregon State
University
Department of
Geosciences
JohnB. Braden
University of Illinois,
Urbana-Champaign
State
CA
NC
NV
HI
CA
MN
OR
IL
NSF
Number
9613472
9613613
9613562
EPA/
USDA
Number
EPA
R825285
EPA
R825286
EPA
R825289
EPA
R825381
EPA
R825290
EPA
R825335
Funding
($)
1,292,627
556,981
767,805
418,749
1,200,000
813,085
1,228,521
291,511
A-6
-------
Streamside Reforestation: An Analysis
of Ecological Benefits and Societal
Perceptions
Toward and Integrated Regional Model
of River Basins of the Pacific Rim
Urban Stream Rehabilitation in the
Pacific Northwest - Physical, Biological,
and Social Considerations
Watershed Protection in Agricultural
Environments: Integrated Social,
Geomorphological, and Ecological
Research to Support Ecosystem-Based
Stream Management
Bernard W. Sweeney
Academy of Natural
Sciences
Stroud Water
Research Center
Jeffrey E. Richey
University of
Washington
School of
Oceanography
Stephen Burges
University of
Washington, Seattle
Department of Civil
Engineering
Bruce L. Rhoads
University of Illinois,
Urbana-Champaign
Department of
Geography
PA
WA
WA
IL
9613588
9613370
9612958
EPA
R825284
EPA
R825306
940,000
50,000
663,020
350,000
A-7
-------
1997 STAR Water and Watersheds
PROJECT TITLE
A Study of Effects of Natural and
Anthropogenic Processes on Tillamook
Bay and its Watershed: An Integrated
Process Study and Land-Use
Perspective
An Integrated Ecological and
Socio-Economic Approach to Evaluating
and Reducing Agricultural Impacts on
Upper Mississippi River Watersheds
An Integrated Watershed Approach to
Evaluate and Model Ecosystem Effects
of Erosion and Pollutant Transport in
Urbanized Subalpine Landscapes
Community Values and the Long-Term
Ecological Integrity of Rapidly
Urbanizing Watersheds
Comprehensive Watershed
Management: A Spacial Water Quality
Assessment System (SWQAS)
Connecting Ecological and Social
Systems: Watershed Research Relating
Ecosystem Structure and Function to
Human Values and Socioeconomic
Behaviors
Development and Implementation of
Decision Support Systems for Predicting
Economic and Ecological Impacts of
Alternative Land and Water
Management Policies in Urbanized
Regions
Ecological Risks, Stakeholder Values
and River Basins: Testing Management
Alternatives for the Illinois River
Impact of Social Systems on Ecology
and Hydrology in Urban-Rural
Watersheds: Integration for Restoration
Principal
Investigate r(s),
Institution
James McManus
Oregon State
University
College of Oceanic
and Atmospheric
Sciences
Prasanna H. Gowda
University of
Wisconsin, La Crosse
Biology/Microbiology
Charles R. Goldman
University of
California, Davis
Environmental Studies
M. Bruce Beck
University of Georgia;
Warnell School of
Forest Resources
C. Gregory Knight
Pennsylvania State
University
Gaboury Benoit
Yale University
Daniel P. Loucks
Cornell University
Mark Meo
University of
Oklahoma
Science and Public
Policy Program
Steward T. A. Pickett
Institute of Ecosystem
Studies
State
OR
WI
CA
GA
PA
CT
NY
OK
NY
NSF
Number
9726863
9726861
9726860
EPA/
USDA
Number
EPA
R825751
EPA
R825761
EPA
R826282
EPA
R825758
EPA
R825791
EPA
R825792
Funding
($)
749,995
650,921*
879,376
849,999
475,106
795,000
258,292
849,996
999,932
A-8
-------
Integrated, Ecological-Economic
Modeling of Watersheds and Estuaries
at Multiple Scales
Landscapes and Waterscapes: An
Integrating Framework for Urbanizing
Watersheds
Linking Watershed-Scale Indicators of
Changes in Atmospheric Deposition to
Regional Response Patterns
Risk Based Urban Watershed
Management - Integration of Water
Quality and Flood Control Objectives
Social and Ecological Transferability of
Integrated Ecological Assessment
Models
Charles S. Hopkinson
Marine Biological
Laboratory
Woods Hole
Panos Diplas
Virginia Polytechnic
Institute and State
University
Department of Civil
Engineering
Jeffrey S. Kahl
University of Maine
Water Research
Institute
Vladimir Novotny
Marquette University
Department of Civil
Engineering
Linda A. Deegan
Marine Biological
Laboratory
Ecosystems Center
MA
VA
ME
WI
MA
9726862
EPA
R825760
EPA
R825762
EPA
R825759
EPA
R825757
815,000
849,266
623,395
827,745*
850,575
A-9
-------
1998 STAR Water and Watersheds
PROJECT TITLE
An Integrated Systems Approach to
Watershed Restoration with Community
Involvement Applied to a Small Rural
Watershed
Combining Economics and Ecological
Indicators to Prioritize Wetlands
Restoration Projects within a Spatial
GIS Framework
Developing Methods and Tools for
Watershed Restoration: Design,
Implementation, and Assessment in the
Willamette Basin, Oregon
Development and Testing of a Decision
Support System for River Restoration
Development of an Integrated Scientific
and Technological Framework for Stream
Naturalization
Development of an Urban Watershed
Rehabilitation Method Using
Stakeholder Feedback to Direct
Investigation and Restoration Planning
Integrating Models of Citizens
Perceptions, Metal Contaminants and
Wetlands Restoration in an Urbanizing
Watershed
Integrating Salmon Habitat Restoration
and Flood Hazard Initiatives:
Societal/Biophysical Estimators for the
Cedar River and Implications for
Regional Rivers
Restoring and Maintaining Riparian
Ecosystem Integrity in Arid
Watersheds: Meeting the Challenge
through Science
and Policy Analysis
Social Impact Assessment of Human
Exposure to Mercury Related to Land
Use and Physical Chemical Settings in
the Mobile-Alabama River Basin
Principal
Investigate r(s),
Institution
J. Boll
University of Idaho
James J. Opaluch
University of Rhode
Island
John Bolte
Oregon State
University
J. David Allan
University of
Michigan
Bruce L. Rhoads
University of Illinois
Marty Matlock
Texas A & M
University
Robert K. Tucker
Stony Brook-
Millstone Watershed
Association
Robert C. Wissmar
University of
Washington,
Seattle
Thomas Maddock
University of Arizona
Jean-Claude
Bonzongo
University of Alabama
State
ID
RI
OR
MI
IL
TX
NJ
WA
AZ
AL
NSF
Number
9900678
9900679
EPA/
USDA
Number
USDA
9801
EPA
R827146
EPA
R827148
EPA
R827147
EPA
R827288
EPA
R827149
EPA
R827150
EPA
R827168
Funding
($)
362,300*
475,000
809,993
9,645
881,913
838,767
749,954
749,991
849,638
804,534
A-10
-------
Understanding the Social Context of
Ecological Restoration in Multiple
Watersheds
When Do Stakeholder Negotiations
Work?
Whole Watershed Health and
Restoration: Applying the Patuxent and
Gwynns Falls Landscape Models to
Designing a Sustainable Balance
Between Humans and the Rest of Nature
Steve Kraft
University of
Southern Illinois
Paul Sabatier
University of
California, Davis
Robert Co stanza
University of
Maryland
IL
CA
MD
USDA
9802
EPA
R827145
EPA
R827169
878,360*
149,935
699,916
A-ll
-------
1999-2000 STAR Water and Watersheds
PROJECT TITLE
Alternative Urbanization Scenarios for
an Agricultural Watershed: Design
Criteria, Social Constraints, and Effects
on Groundwater and Surface Water
Systems
An Acre an Hour: Documenting the
Effects of Urban Sprawl on a Model
Watershed in Philadelphia,
Pennsylvania
Identification and Control of Non-Point
Sources of Microbial Pollution in a
Coastal Watershed
Integrating Coral Reef Ecosystem
Integrity and Restoration Options with
Watershed-based Activities in the
Tropical Pacific Islands and the Societal
Costs of Poor Land-use Practices
Linking Environmental and Social
Performance Measurement for
Management at National and Watershed
Levels: Modeling and Statistical
Approaches
Strategic Renewal of Large Floodplain
Rivers: Integrated Analysis
Targeting Decisions to Reduce Risk in
Agricultural Watersheds: Effective
Nutrient Management Through Local
Implementation
The Impact of Lawn Care Practices on
Aquatic Ecosystems in Suburban
Watersheds
Watershed Scale Assessments of E. coli
Contamination: Implications of Source
Identification for Public Policy Debate
Pulses - The Importance of Pulsed
Physical Events for Watershed
Sustainability in Coastal Louisiana
Principal
Investigator(s),
Institution
Richard C. Lathrop
Wisconsin
Department of
Natural Resources
Claire Welty
Drexel University
Synnove F.
Knutsen University
of California
Robert H. Richmond
University of Guam
Marine Laboratory
Scott Farrow
Carnegie Mellon
University
Richard Sparks
University of
Illinois,
Champaign-Urbana
D. J. Mulla
University of
Minnesota, St. Paul
Kevin Armbrust
University of
Georgia, Griffin
Ronald F. Turco
Purdue University
JohnW. Day
Louisiana State
University
Stat
e
WI
PA
CA
GU
PA
IL
MN
GA
IN
LA
NSF
Number
0001884
0003208
EPA/
USDA
Number
EPA
R828010
EPA
R828011
EPA
R828008
EPA
R828021
USDA
9901
EPA
R828007
USDA
0001
EPA
R82-8009
Funding
($)
886,105
464,012
895,234
795,249
649,864
1,090,000
75,377*
893,849
892,270*
899,995*
A-12
-------
The Spatial Patterning of Land Use
Conversion: Linking Economics,
Hydrology, and Ecology to Evaluate the
Effects of Alternative Future Growth
Scenarios on Stream Ecosystems
An Integrated GIS Framework for Water
Reallocation and Decision Making in the
Upper Rio Grande Valley.
Margaret A. Palmer
University of
Maryland/College
Park
Paul Olen Mattews
University of New
Mexico
MD
NM
EPA
R82-8012
EPA
R82-8070
1,125,212*
410,000*
A-13
------- |