awcl
Ecological Research
Program Review
f
Report of the Subcommittee on
Ecological Research
April 1, 2005
Revised August 16, 2005
r
-------
This report was written by the Ecological Research Subcommittee of
the Board of Scientific Counselors, a public advisory committee
chartered under the Federal Advisory Committee Act (FACA) that
provides external advice, information, and recommendations to the
Office of Research and Development (ORD). This report has not been
reviewed for approval by EPA, and therefore, the report's contents and
recommendations do not necessarily represent the views and policies of
., the EPA, or other agencies of the federal government. Further, the
content of this report does not represent information approved or
disseminated by EPA, and, consequently, it is not subject to EPA's
Data Quality Guidelines. Mention of trade names or commercial pro-
ducts does not constitute a recommendation for use. Reports of the
.Board of Scientific Counselors are posted on the Internet at http://
www.epa.gov/osp/bosc.
-------
REPORT OF THE
BOARD OF SCIENTIFIC COUNSELORS
ECOLOGICAL RESEARCH PROGRAM REVIEW
Office of Research and Development
U.S. Environmental Protection Agency
APRIL 1,2005
REVISED AUGUST 16,2005
SUBCOMMITTEE ON ECOLOGICAL RESEARCH
Dr. Michael T. Clegg-University of California, Irvine, Chair
Russel Frydenborg-Florida Department of Environmental Protection
Dr. John P. Giesy-Michigan State University
Dr. Richard Lowrance-USDA Agricultural Research Service
Dr. Sue A. Thompson-Pennsylvania Biodiversity Partnership
Dr. R. Eugene Turner-Louisiana State University
Dr. Jianguo Wu-Arizona State University
EPA CONTACT
Greg Susanke, Designated Federal Officer
U.S. Environmental Protection Agency
-------
TABLE OF CONTENTS
I. EXECUTIVE SUMMARY : i
Structure of the Program Review ; 1
Objectives of the Review 1
Conclusions and Recommendations 2
II. INTRODUCTION 3
III. RELEVANCE : ; 6
A. Charge Questions 6
B. Specifics for Program Elements 6
IV. SCIENTIFIC AND COMPETITIVE QUALITY 11
A. Charge Questions 11
B. Specifics for Program Elements 11
V. PERFORMANCE ;... 16
A. Charge Questions 16
B. Specifics for Program Elements 16
VI. LEADERSHIP 19
A. Charge Questions 19
B. Specifics for Program Elements 19
VII. COLLABORATIONS ! .' 22
A. Charge Questions 22
B. Specifics for Program Elements 22
VIII. RESOURCES 25
A. Charge Question 25
IX. INTEGRATION AND FUTURE DIRECTIONS '. 26
X. FINDINGS AND RECOMMENDATIONS 29
APPENDIX A—BOSC ECOLOGICAL RESEARCH SUBCOMMITTEE MEMBERS 32
APPENDIX B—BOSC ECOLOGICAL RESEARCH SUBCOMMITTEE MEETING DATES 33
APPENDIX C—LIST OF ACRONYMS 35
Ecological Research Program Review Report
-------
I. EXECUTIVE SUMMARY
The Executive Committee of the Board of Scientific Councilors (BOSC) of the Office of
Research and Development (ORD) within the U.S. Environmental Protection Agency (EPA) has
agreed to undertake a series of reviews of major ORD research programs. This report addresses
the review of the Ecological Research Program (ERP). The evaluation is based on the review of
a large volume of written materials describing the ERP Multi-Year Plan (MYP), individual,
research projects, and budgetary information.
The Ecological Research Subcommittee, chaired by Dr. Michael Clegg from the University of
California-Irvine, conducted this review. Other members included Russel Frydenborg, Florida
Department'of Environmental Protection; Dr. John P. Giesy, Michigan State University; Dr.
Richard Lowrance, U.S. Department of Agriculture (USDA) Agricultural Research Service; Dr.
Sue A. Thompson, Pennsylvania Biodiversity Partnership; Dr. R. Eugene Turner, Louisiana State
University; and Dr. Jianguo Wu, Arizona State University. A list of the Subcommittee members
is included in Appendix A.
The review was conducted from February through April 2005. The Subcommittee members met
twice via conference call prior to a site visit for orientation to the Federal Advisory Committee
requirements and to discuss the review procedures. A 3-day site visit was conducted at the EPA
facility in Research Triangle Park (RTP), North Carolina, from March 7-9,2005. The site visit
featured direct presentations by ERP program leaders and scientists and it provided time for the
BOSC Ecological Research Subcommittee to question EPA staff on program details.
Subsequent to the RTP meeting, a conference call was held on April 1, 2005, to finalize the draft
report. - .
The objectives of the review were to evaluate: (1) program relevance and quality; (2) program
design and implementation; (3) progress achieved towards meeting long-term goals (LTGs);
(4) stakeholder involvement and the degree to which research is consistent with needs articulated
at regional and local levels; and (5) the degree to which research "outputs" are being used by
stakeholders. The ERP's research is organized into three LTGs and this report provides an
analysis of the ERP in the context of each of the LTGs. The report then considers the ERP in
holistic terms and provides specific conclusions and recommendations to improve the program's
effectiveness and impact. A major purpose of this review, therefore, is to assist the ERP to adapt
to new requirements and expectations as it works to meet the needs of the nation.
Ecological Research Program Review Report
-------
As noted above, the review was focused on assessing progress towards meeting the three LTGs
of the ERP:
-$• LTG 1: By 2010, national policy makers will have the tools and technologies to develop
scientifically defensible assessments of the state of our nation's ecosystems and the
effectiveness of existing national programs and policies.
•$• LTG 2: By 2010, states and tribes apply improved tools and methods to protect and restore
their valued ecological resources.
•^ LTG 3: By 2012, decision-makers understand the importance of ecosystem services and
make informed, proactive management decisions that consider a range of alternative
outcomes.
In general, the Subcommittee found the ERP to be a high-quality scientific program that is
providing essential technical information to the regulatory offices within EPA as well as to state,
local, and tribal governments to assist these entities in addressing novel problems of
environmental management. In particular, the Subcommittee concluded that results of the
program's research are relevant and of direct use to states and tribes in protecting and restoring
ecological resources. The program is developing major tools for measuring environmental
health, and these tools are being adopted in the field. The quality of the ERP science is high and
compares favorably with ecological research in academic, non-governmental, and private
sectors. Program leadership appears to be very good.
There is a need for improved integration among the three LTGs. To gain additional investment
leverage, more emphasis should be placed on collaboration between EPA scientists and scientists
outside the Agency. Moreover, processes designed to facilitate communication between the ERP
and its stakeholders need to be improved. There also is a need for a formal performance
evaluation to assess the integration of the various research projects in the context of the LTGs of
the ERP. The articulation of research goals and priorities is not always clear to those inside or
outside the organization. In this context, it is crucial that a new MYP be developed that aligns
with current resource constraints and that better integrates the three LTGs.
The ERP is undergoing substantial budget reductions and short-term approaches to budget
reductions are being successfully implemented. The long-term impacts of these short-term
solutions, however, threaten to disproportionately reduce the ERP's effectiveness. In particular,
a budget reduction strategy focused almost entirely on extramural programs will have
disproportionate costs because the leveraging of outside resources may be compromised and
productive connections with academic, non-governmental, and private sector researchers may be
undermined. Plans need to be developed for a long-term equilibrium that balances the research
portfolio against expected resource constraints. This will entail a difficult priority-setting
process, but if done effectively, it will maximize the effectiveness of the ERP over the long term.
Ecological Research Program Review Report
-------
II. INTRODUCTION
EPA relies on expert external review to assess the scientific quality and performance of its
research programs. This report communicates the findings of a review of EPA's Ecological
Research Program. The ERP is one of nine research programs within ORD and it represents one
of ORD's largest budgetary components. ORD performs and funds research that provides
technical standards and scientific information to support EPA's broad mission of protecting and
restoring the environment and specifically to support EPA's various regulatory functions.
Many customers and stakeholders utilize ORD's research products. Foremost among these are
the regulatory offices of EPA that are mandated to provide technical standards for regulatory
decisions. These technical standards must be grounded in well-documented scientific knowledge.
Other important customers include state regulatory entities, tribes, and local governments. The
stakeholder population also includes non-governmental organizations (NGOs) that have a stake
in environmental conservation; the private sector that must adapt to and implement regulatory"
decisions which in turn require clear and understandable technical standards; and finally, the
general public that has a strong stake in EPA's mission of protecting, sustaining, and restoring
the health of the nation's ecosystems, to achieve this broad goal, EPA must devise scientifically
valid measurements of the health of the environment and this requires an intensive program of
ecological research. . .
The BOSC's Ecological Research Subcommittee conducted this review (see Appendix A for a
list of the Subcommittee members). Prior to the review, the Subcommittee met twice via
conference call (February 17 and March 3, 2005) for orientation to the Federal Advisory
Committee requirements and to discuss the review procedures. The review was conducted at the
EPA facility in RTF, North Carolina, from March 7-9, 2005. Subsequent to the RTP meeting, a
teleconference was held on April 1, 2005, to finalize the draft report. The review meeting at
RTP and the three teleconferences were conducted as open meetings under the guidelines of the
Federal Advisory Committee Act (FACA). A list of the Subcommittee conference calls and
meetings is provided in Appendix B.
The format of this review is relatively new. The BOSC has just begun an intensive review
process for the various ORD research programs and this is only the third review in that series.
An impetus for the BOSC review is the Office of Management and Budget's (OMB) Program
Assessment Rating Tool (PART) evaluation system. The ERP received a "Results Not
Demonstrated" rating in its recent PART evaluation. Based on this rating, an external review
appeared in order to more fully assess the strengths and weaknesses of the ERP and to identify
ways to improve the program. A major purpose of this review, therefore, is to assist the ERP to
adapt to new requirements and expectations as it works to meet the needs of the nation.
The charge to the Subcommittee was to review the ERP's work and products using the following
eight general questions as guidelines:
1. Is the focus of the program relevant to and consistent with EPA's strategic goals? Does it
develop a scientific foundation that will lead to attainment of the program's stated
Ecological Research Program Review Report • • • 3
-------
environmental outcomes? Are potential public benefits of the program clearly articulated?
What would be the minimum research program that would be both effective and successful?
2. Does the program have a logical and comprehensive design with clear goals, priorities, and
schedules to track progress toward those goals?
3. Do the design and implementation of the program's structure facilitate attainment of
outcomes through integration of research across the program?
4. Has the program made significant progress toward each of its long-term goals?
a. Do the research results address the key research questions?
b. Is the rationale to address the questions clearly articulated?
c. Does the program follow a long-term plan to address the logical sequence of the
questions?
d. Is progress to address the questions being made in a timely fashion?
e. Does the research reflect the current state-of-the-science, and meet the current and future
needs of the Agency, science, and program customers?
5. What is the scientific quality of the program's research products? Does the program ensure
high quality research through competitive, merit-based funding? If funds are not
competitively awarded, what process does the program use to allocate funds? Does this
process ensure that quality is maintained?
6. Is the stakeholder involvement in the development of the program clearly and adequately
articulated?
a. Does the program effectively engage stakeholders in its planning?
fa. Does the program have a process for using the results of the research, along with
stakeholders' feedback, to identify key research gaps and to update the program's
research agenda?
c. Are potential public benefits clearly articulated?
d. What are the impediments, if any, to collaboration with other organizations?
7. Are the program's research results being used by clients and stakeholders? Are these
research results consistent with the needs articulated by the Agency's program and regional
offices?
8. Will the program's completed and planned outputs lead to the intended outcomes that are
protective of our ecological resources?
This report discusses the above charge questions as they relate to the ERP's three revised LTGs:
<* LTG 1: By 2010, national policy makers will have the tools and technologies to develop
scientifically defensible assessments of the state of our nation's ecosystems and the
effectiveness of existing national programs and policies.
•$• LTG 2: By 2010, states and tribes apply improved tools and methods to protect and restore
their valued ecological resources.
. Ecological Research Program Review Report
-------
•$• LTG 3: By 2012, decision-makers understand the importance of ecosystem services and
make informed, proactive management decisions that consider a range of alternative
outcomes.
\
Each of the above LTGs is reviewed in the context of the charge questions, followed by a
discussion of the integration of all LTGs to assess the program holistically, and finally a list of
conclusions and recommendations.
Ecological Research Program Review Report
-------
III. RELEVANCE
1. Is the focus of the ERP relevant to and consistent with EPA's strategic goals? Does it
develop a scientific foundation that will lead to attainment of the ERP's stated environmental
outcomes? Are potential public benefits of the program clearly articulated? What would be
the minimum research program that would be both effective and successful?
4e. Does the research reflect the current state-of-the-science, and meet the current and future
needs of EPA, science, and the ERP's customers?
LONG-TERM GOAL 1
By 2010, national policy makers will have the tools and technologies to
develop scientifically defensible assessments of the state of our nation's
ecosystems and the effectiveness of existing national
programs and policies.
Question 1: The focus of the ERP is entirely relevant and consistent with EPA's strategic goals.
The Agency's strategic goals are: (1) clean air, (2) clean and safe water, (3) land preservation
and restoration, (4) healthy communities and ecosystems, and (5) compliance and environmental
stewardship. To achieve these strategic goals, it is critically important for national policy makers
to have accurate, reliable, and timely information on the status and trends of the nation's
environmental conditions, so that proper policies can be made and actions taken accordingly. To
this end, scientifically sound and practically effective tools and technologies must be made
available to national policy makers, and this will be accomplished through LTG 1. These tools
and technologies include a scientifically sound design framework, reliable ecological indicators,
and effective monitoring protocols and actions. Products from LTG 1 are most relevant to
environmental problems and policy issues at the national level.
Through LTG 1, the ERP has developed a probability-based design and sampling framework that
is solid in theory and efficient in practice. This framework has been successfully used for
measuring, assessing, and monitoring the status and trends of ecosystem conditions at the
national, regional, and local scales. A comprehensive scientific foundation that emphasizes the
integration between pattern and process across systems, scales, and levels of biological
organization continues to be built and refined. These salient characteristics of the scientific
foundation are reflected in a number of projects, including the Environmental Monitoring and
Assessment Program (EMAP), the Mid-Atlantic Integrated Assessment (MAIA), and landscape
Ecological Research Program Review Report
-------
studies. This developing scientific foundation is appropriate for achieving the ERP's expected
environmental outcomes.
The potential benefits to the public of the ERP's research are evident and clearly articulated.
The products of LTG 1 are crucial for national policy makers and also for public awareness of
the nation's environmental conditions.
It is difficult to define the minimum research program for protecting the environment. All of the
ERP's present research projects seem necessary to understand and deal with the nation's
different environmental issues across different scales. Removing any part of the program would
certainly not enhance the functionality and outputs/outcomes of the ERP.
Question 4e: The research of LTG 1 is evidently of high scientific quality, and reflective of the
state-of-the-art in broad-scale environmental research, particularly in developing statistical
design and sampling strategies for measuring and monitoring ecosystem conditions. Because
landscapes are large enough to incorporate multiple interacting ecosystems and major regimes of
human influences on them, while still small enough for insightful mechanistic explorations,
landscape-level indicators provide critically important information for legislators and policy-
makers at the national level and decision-makers and stakeholders at the local level. The ERP
has developed a suite of such indicators in recent years, and this research represents cutting-edge
science. Empirical testing of these indices as well as additional core research in this extremely
important area, however, are clearly needed.
LONG-TERM GOAL 2
By 2010, states and tribes apply improved tools and methods to protect
and restore their valued ecological resources.
Core research and problem-driven research undertaken by ERP staff, cooperators, and grantees
form the basis for many of the tools and methods employed by states and local entities, both
public and private, to protect and restore ecological resources. This function of the ERP is not
only consistent with, but is essential to, EPA's core mission to protect human health and
safeguard the natural environment—air, water, land—upon which life depends. Safeguarding
the nation's environment is intrinsically linked to activities at the state, tribal, and local levels.
The scientific research produced and supported by the ERP in support of LTG 2 ranges in quality
from good to excellent. Results from these projects have answered key questions on processes in
natural systems and have resulted in a greater understanding of the affects of anthropogenic
activities on the physical, chemical, and biological integrity of the nation's waters and air. The
ERP research represents major contributions to ecosystem science. Examples of this research
include stressor identification techniques for specific biological impairment determination,
advanced watershed stormwater models for pollutant reduction, and novel microbial and
..genetically based ecological assessment procedures.
Results of the ERP research are relevant and of direct use to states and tribes in protecting and
restoring ecological resources. Model case studies by collaborators (for example, Maryland's
Green Infrastructure assessment) have had "on-the-ground" impacts in land conservation. Most
of the ERP's research, however, has not yet translated into direct conservation and restoration
Ecological Research Program Review Report 7
-------
activities, other than the handful of collaborative examples. These outcomes are expected to
occur in more states and tribes over time.
Although there are numerous potential public benefits of the ERP's research, these benefits are
not always clearly articulated, even at the program level, and are rarely explicitly included as a
goal in individual research projects. This lack of an outcomes-based research paradigm hampers
the program from producing research that fully meets both the future and current needs of
clients, especially those outside of EPA (e.g., states and tribes). Although examples were given
in the onsite presentations of states routinely and effectively applying ERP-derived tools, no
formalized tracking system that quantifies the extent of this application was evident and no
specific uses by tribal entities were highlighted.
LONG-TERM GOAL 3
By 2012, decision-makers understand the importance of ecosystem
services and make informed, proactive management decisions that
consider a range of alternative outcomes.
LTG 3 is the ERP's outreach component with the goal of developing sophisticated tools to
enable decision-makers to make decisions that enhance ecosystem services. Although this goal.
has elements of more basic and applied research, 'scientists working on LTG 3 seek
people/institutions with specific problems that can be used as case studies for critical resources
management issues. To date, these site-specific projects have been used as proof-of-concept
exercises, coordinated through either regional or program offices, such as the Office of Water
(OW) and the Office of Air Quality Planning and Standards (OAQPS).
The research, tools, and analytical technologies developed under the ERP represent the most
comprehensive federal government research program examining the provision of ecosystem
services and the communication of these to decision-makers. LTG 3 is a highly relevant activity
that is central to EPA's mandate of improving environmental quality and protecting and restoring
the health of the nation's ecosystems. ORD and particularly the ERP are uniquely suited and
positioned to address these issues.
In addition to being relevant to the federal role, the major projects under LTG 3 address critical
complex questions facing local and state governments. The major projects described under LTG
3 and the questions framed for future research will provide both new scientific knowledge about
the provision and value of ecosystem services and new decision tools to allow decision-makers
and the public to address alternative futures with varying levels of ecosystem services.
The projects presented during the site visit demonstrate an excellent balance between
development of new information and conceptualizing new analytical methods for the future,
while also providing clients with tools for addressing current issues. Quantifying and
demonstrating future scenarios with varying levels of ecosystem services, will provide the
scientific foundation needed to make informed decisions. Many of these decisions are made at a
local level with tools mat currently are inadequate to consider cumulative effects of the loss or
gain of ecosystem services. The tools developed under LTG 3 will provide the scientific
foundation for better decision-making and thus help in attainment of environmental benefits.
Ecological Research Program Review Report
-------
All projects presented under LTG 3 have the specific objective of providing information on
ecosystem services and ecosystem functions to either specific or hypothetical decision-making
groups. For some major projects and tools, potential public benefits are clearly articulated. For
example, the Willamette Basin alternative futures research has lead to an atlas of the basin and to
a clear focus among regional governments on using the tools developed under LTG 3 to increase
the level of future ecosystem services within the large river (see Box ,1).
Box »1 sThe Willamette
historical.landluseTand
lamtrendftcurrent
•ipbpujau
-------
realized unless the decision support systems and analytical techniques to project and
communicate alternative future development scenarios are available and applied. Thus, the
entire research and technology development program is necessary.
10
Ecological Research Program Review Report
-------
IV. SCIENTIFIC AND COMPETITIVE QUALITY
2. Does the ERP have a logical and comprehensive design with clear goals, priorities, and
schedules to track progress toward those goals?
3. Do the design and implementation of the ERP's structure facilitate attainment of outcomes
through integration of research across the ERP?
4a. Do the research results address the key research questions?
4b. Is the rationale to address the questions clearly articulated?
4c. Does the ERP follow a long-term plan to address the logical sequence of the questions?
4d. Is progress to address the questions being made in a timely fashion?
5. What is the scientific quality of the ERP's research products? Does the ERP ensure high
quality research through competitive, merit-based funding? If funds are not competitively
awarded, what process does the ERP use to allocate funds? Does this process ensure that
quality is maintained?
8. Will the ERP's completed and planned outputs lead to the intended outcomes that are
protective of our ecological resources?
LONG-TERM GOAL 1
By 2010, national policy makers will have the tools and technologies to
develop scientifically defensible assessments of the state of our nation's
ecosystems and the effectiveness of existing national
programs and policies.
Question 2: LTG 1 activities are organized in terms of three related, but distinctive,
components: (1) statistically valid and scientifically defensible design frameworks, (2) a suite of
reliable ecological indicators for different ecosystems, and (3) environmental monitoring. Goals
for each component are clearly defined. The priorities of LTG 1 activities are clearly placed at
the national level, which is appropriate because LTG 2 and LTG 3 deal mainly with the state and
local levels, respectively. In other words, each goal has its own focal level of organization, and
corresponding spatial scale, and this scheme is, in principle, both scientifically sound and
practically feasible.
LTG 1 is particularly important because it provides an overall conceptual framework and
'technical guidelines for the other LTGs. To assure both high scientific quality and relevance to
EPA's mission, LTG 1 projects should encourage close collaborations between EPA's own
researchers and external high-caliber scientists who have broad visions and pertinent knowledge
and skills. Such collaborations can be accomplished through open grant competition with an
explicit requirement for collaboration with EPA researchers. In addition, although the design
Ecological Research Program Review Report J}
-------
principles of LTG 1 are sound, improved planning and better-coordinated cross-level integration
are needed. Also, more systematic plans and mechanisms for increased tracking of the outputs
and outcomes of research projects must be developed as part of the research cycle.
Question 3: The new structure of the ERP with three LTGs seems scientifically sound and
practically feasible. Products of LTG 1 are most relevant to environmental problems and policy
issues at the national level, whereas LTG 3 deals primarily with local-scale environmental issues
that may frequently involve local ecosystems (e.g., lakes, streams, or forest habitats) or local
landscapes (watersheds or mosaics of different ecosystems) (see Figure 1). LTG 2 is mainly
targeted at environmental problems and policy issues at the state level, often spanning over a
region. This suggests that in order to achieve desirable outcomes, each LTG must emphasize
interactions and collaborations with relevant agencies and stakeholder groups at its focal level.
For example, the expected outcomes of LTG 3 are unlikely to be accomplished without directly
involving local decision-makers and stakeholders. Having a focal level or scale for each LTG
facilitates use-inspired research, which is consistent with the mission of EPA, but does not, and
should not, imply that the three LTGs can be achieved separately without interactions or
coordination. On the contrary, a major advantage of such a hierarchical framework is both to
highlight the primary targeted user group and to provide pathways to effectively integrate across
scales and programs. Products of LTG 1 should influence and have influenced activities of LTG
2, which in turn have bearings on LTG 3. The same is true the other way around. To date, the
ERP has demonstrated such two-way interactions to some extent with some excellent examples.
The ERP has a logical and comprehensive design, which is adequate for ORD's planning process
and for demonstrating progress toward its overall goals.
Questions 4a and4b: The results of LTG 1 are quite relevant to its key research questions. The
probability-based design framework has served well for multiple-scale sampling and monitoring
activities at the national, state, and local levels. A suite of indicators for landscape integrity and
ecosystem health has been developed. These methods and tools have produced valuable
information on the nation's ecosystems that is critically important to policy makers and decision-
makers.
Question 4c: The three LTGs are well designed and follow a multiple-scale, hierarchical
framework that facilitates addressing environmental issues at the national, state, and local levels.
LTGs are well articulated and are not only relevant, but also crucial, to the overall mission of
EPA. The realization of these LTGs requires continuation of high-quality research conducted by
EPA researchers and outside scientists collaboratively as well as secure long-term funding.
Question 4d: Research under LTG 1 has made tremendous progress in terms of both research
output and environmental outcomes. Timely progress also has been made to address important
research questions. For example, a statistically sound, scientifically defendable, and practically
feasible design and sampling framework was developed early through EMAP, and has been
widely used by and provided guidance to a variety of projects at the national and regional scales
ever since. Based on this framework, extensive efforts have been made to develop methods and
tools to measure and monitor conditions of different ecosystems and landscapes across the nation
(e.g., EMAP-WEST, Coastal Ecosystem Assessment, West Rivers).
12
Ecological Research Program Review Report
-------
Figure 1. Subcommittee Interpretation of the Organizational Levels and
Spatial Scales at Which Each LTG Operates
Organizational Level/Spatial Scale
National
State/Tribe
Decision-
Makers/
Stakeholders
National
Region/
Landscape
Local
Ecosystem
& Landscape
long-Term
Goal #1
Lone-Term
Goal #2
Long-Term
Goal #3
LONG-TERM GOAL 2
By 2010, states and tribes apply improved tools and methods to protect
and restore their valued ecological resources.
Questions 5 and 8: Although the implementation of the ERP's structure has demonstrated the
capability to achieve beneficial environmental outcomes, the formal integration between
individual research components should be better. Cooperation exists among many projects, but
the need for integration between some projects, although implicit, should be better articulated by
ORD management. This would improve synergy between investigators and enhance the overall
body of research, which is generally of excellent quality.
Ecological Research Program Review Report
13
-------
Based on the quality of research, substantial advances toward achieving LTG 2 are apparent.
Examples of these efforts include the Modeling and Stressor Identification procedures, which
have played key roles in watershed restoration through the Total Maximum Daily Load (TMDL)
program and the Multimedia Pollutant studies, which have led to mercury reduction in the
Everglades and elsewhere. Several other projects, such as the recent development of microbial
and genetic tools, should reasonably expect to achieve outcomes when more mature. A key to
the success of all the LTG 2 research efforts will be educating state arid local decision-makers on
the benefits of the tools, as well as training appropriate end users.
The research questions are clearly formulated to address factors most responsible for degrading
our environment. The rationale behind the research is scientifically sound and appropriate,
including physical, chemical, biological, landscape, and modeling components. The research is
structured in a logical sequence to allow results from projects to build on others, although the
formal integration of individual projects should be better articulated and incorporated into the
strategic planning process.
The overall body of research generated by the ERP, in terms of quality and ability to produce
beneficial environmental outcomes, is superior. Without the ERP's leadership in this arena,
many important advances in environmental assessment and the development of strategies for
effective ecological management would not exist. Stringent quality assurance procedures are
followed throughout the ERP's research process, from sampling to data analysis. Peer review
and feedback, including input from academic and government scientists, is routinely followed,
resulting in a solid scientific program.
ORD's Science To Achieve Results (STAR) grants program is an excellent example of the
successful implementation of a competitive funding process, which when coupled with the
ERP's guidance and understanding of complex environmental issues, can lead to research results
that are directly applicable to environmental problems at the state and tribal level.
When funds are not competitively awarded, the ERP appears to use a "best professional
judgment" approach to allocate funds, coupled with a post-award assessment of the project's
success. Based on the successful results associated with these projects, quality appears to have
been maintained, although a more formal evaluation is warranted.
LONG-TERM GOAL 3
By 2012, decision-makers understand the importance of ecosystem
services and make informed, proactive management decisions that
consider a range of alternative outcomes.
Although the initiatives included as elements of LTG 3 are very ambitious, the goals are clearly
stated and achievable. Because LTG 3 has not been the subject of an MYP, the specifics of
schedules are difficult to evaluate. The three component questions around which the future
activities will be developed are clear and should lead to measurable outcomes. Some
connections between LTG 3 components and other LTGs are not clear. Although the potential
for these connections is obvious and necessary, integration of all three LTGs should be made
explicit in the revised MYP. For example, a clear connection between EMAP and the Regional
14
Ecological Research Program Review Report
-------
Vulnerability Assessment (ReVA) Program was not evident, although the two are clearly parallel
projects. The program description would be improved by providing clear linkages with the other
twoLTGs.
The research results that were presented have done an excellent job of addressing the key
research questions. The ERP research has made numerous advances in both integrative
ecosystem science that explicitly includes socioeconomic constraints (Willamette Basin, MAIA)
and in developing general tools for integrated assessments (ReVA). The rationale for the key
questions is clear, but it is not always clear how individual projects are initiated. As there are
many ways that projects might be initiated, some of which are beyond the control of the ERP,
clear documentation of the criteria for project selection is desirable.
It is understood that a revised MYP is now being developed. Because LTG 3 is proposed as a
completely new element in the overall research strategy, the Subcommittee can make only an
overall assessment and give general guidance on program direction. The research is clearly
long-term and will need to incorporate feedback from clients who help implement the pilot
programs. As the MYP is revised, an integrated program structure should be developed that
explicitly states the relationships between program elements and packages them in a way that
allows the tracking of results and progress toward longer-term goals and the reporting of short-
term outcomes. LTG 3 was not part of the 2003 MYP, but includes elements of the earlier LTG
3 and LTG 4 as well as new elements that focus on the development of forecasting models and
analytical tools that will allow the assessment of ecosystem services. The Subcommittee
understands the rationale for this proposed LTG and endorses the goals and approach.
The research currently being conducted and that which is proposed to be conducted under LTG 3
represents state-of-the-science in assessing complex systems and developing tools to understand
and enhance ecosystem services. The scientific quality of the program is very good to excellent.
We received varied information on how peer review is used within the organization. Peer review
and quality assurance for the STAR Program were mostly good, although some inconsistencies
occurred. If there is a program for internal peer review of projects, it was not evident. It would
strengthen the internal research program to have external peer review of projects.
Ecological Research Program Review Report . '15
-------
V. PERFORMANCE
2. Does the ERP have a logical and comprehensive design with clear goals, priorities, and
schedules to track progress toward those goals?
3. Do the design and implementation of the ERP's structure facilitate attainment of outcomes
through integration of research across the ERP?
4d. Is progress to address the questions being made in a timely fashion?
6b. Does the ERP have a process for using the results of the research, along with stakeholders'
feedback, to identify key research gaps and to update the ERP's research agenda?
7. Are the results of ERP research being used by clients and stakeholders? Are these research
results consistent with the needs articulated by EPA's program and regional offices?
8. Will the ERP's completed and planned outputs lead to the intended outcomes that are
protective of our ecological resources?
LONG-TERM GOAL 1
By 2010, national policy makers will have the tools and,technologies to
develop scientifically defensible assessments of the state of our nation's
ecosystems and the effectiveness of existing national programs and
policies.
Question 2: LTG 1 activities are organized in terms of three related, but distinctive,
components: (I) statistically valid and scientifically defensible design frameworks, (2) a suite of
reliable ecological indicators for different ecosystems, and (3) environmental monitoring. Goals
for each component are clearly defined. The priorities of LTG I activities are clearly placed at
the national level, which is appropriate because LTG 2 and LTG 3 are more focused on the state
and local levels, respectively.
*
Question 3: LTG 1 program maturation has been accompanied by standardization of the
conceptual approaches and the experienced solicitation and involvement of stakeholders. These
positive developments have drawn strong support from a core group of statisticians and
geographic information system (GIS) experts (among others) within the ERP that has an
institutional memory, experience, and professional skills that are nationally significant. Then-
activities have been a positive and significant influence on LTG 2 and LTG 3 activities, as well.
Question 4d: Progress to address the research questions is being made in a timely fashion, but
articulation of those goals and the planning involved in the processes'leading to these goals need
to more clearly permeate the science culture. The idea is not to homogenize an individual's
scientific talents and perspectives so communication skills are emphasized over science skills,
16 Ecological Research Program Review Report
-------
but that the programmatic elements are masterfully and more completely articulated on behalf of
the whole.
Question 6b: The Subcommittee believes that more effective processes and formal mechanisms
are needed that allow better communications between the ERP and its clients. The experience of
some members of the Subcommittee is that dissemination of important research results is slow to
reach the end users.
Question 7: The stakeholders and clients are using the results in a variety of ways, some of
which are obvious and others that are not so obvious. The National Coastal Assessments have
led to an increasing number of states either issuing their own reports using the protocols and
sampling strategies developed by the ERP, or joining together to issue regional summaries.
These are in addition to the EPA documents produced by the ERP staff and their collaborators.
This tangible outcome is monitored. For example, in 2000, no states were directly involved in
these reports (as state or regional reports), but by 2005, a minimum of 12 states will participate.
At this rate, the goal of reaching all coastal states should be nearly completed in a few years.
Question 8: In the opinion of the Subcommittee, the ERP has resulted in the desired outcomes,
and its present activities and future (planned) outputs are consistent with the conclusion that
protection of the nation's ecological resources is enhanced.
LONG-TERM GOAL 2
By 2010, states and tribes apply improved tools and methods to protect
and restore their valued ecological resources.
Performance measures of the ERP's planning process and measures for demonstrating progress
toward overall program goals do not appear to be strictly established, although they can be
extracted from the body of research.
There is a need to develop a formal performance evaluation that assesses the degree of
integration between various research projects to help improve cooperation between individual
research investigators.
Clear and substantial progress toward LTG 2 was demonstrated in much of the mature research,
although this could not readily be .evaluated for research at earlier stages of development. Each
research component should issue a periodic report that more clearly details the incremental
progress made toward expected outcomes. This would allow an adaptive management strategy
to keep projects "on track and internally integrated.
Stakeholder input to identify research gaps does occur, but the process of briefing decision-
makers on research results as well as on applicability of the research currently is informal or
..often lacking. The effectiveness of this aspect of the program could be improved by establishing
timely and regular communications with a broad array of stakeholders using an established
procedure in conjunction with appropriate EPA program offices.
The ultimate use of the products is dependent upon factors beyond the control of the ERP, but
the above outreach process would improve the utilization of pertinent results. States, tribes, and
Ecological Research Program Review Report 17
-------
local governments should be urged to seek the ERP tools appropriate for carrying out their
specific environmental protection missions. For example, nonpoint source pollution can be
mitigated through judicious land use decisions, which almost exclusively are made at the level of
local government. If utilized by local government resource managers, the ERP products, such as
landscape and predictive models, would more directly achieve desired outcomes.
The results of the ERP are thoroughly consistent with the expressed needs of EPA program and
regional offices. EPA program offices present the ERP with a prioritized list of research issues
developed through interactions with stakeholders. There is a reasonable expectation that the
environmental protection tools generated by the ERP will translate into beneficial outcomes
when the research matures and the tools are implemented at appropriate levels of government. •
LONG-TERM GOAL 3
By 2012, decision-makers understand the importance of ecosystem
services and make informed, proactive management decisions that
consider a range of alternative outcomes.
It is not clear how progress is tracked, but it is assumed that this is the primary responsibility of
program managers and lead scientists on the projects. Very good progress in this area has
occurred during the past 5 years and it is expected that this progress will continue in the future
with adequate funding and scientific resources.
The component of the ERP aggregated under LTG 3 is well positioned to make significant
contributions hi the near- and long-term. While it is understood that science is a continuing
process, it is evident that research products are being adopted and used. The review process
should be structured in such a manner that useful products and technology can be transferred to
clients in a timely and efficient manner while the tools are further refined and adapted to
additional situations and applications. To identify key research gaps and to update the projects,
the Subcommittee suggests reviews of individual projects by external scientists and stakeholders.
Currently, LTG 3 does not have a plan for internal-periodic assessment, but it is assumed that
this will be part of the MYP.
The results, products, and tools generated by LTG 3 are being used by clients. At this point, the
main clients are involved with the pilot projects. Other clients are being developed through
online use of the ReVA and Restoration Plus (RePlus) tool kits. Research results, such as the use
of ReVA with the Sustainable Environment for Quality of Life (SEQL) program and the use of
research results to inform decision-making by state agencies, such as the Wisconsin Department
of Natural Resources, demonstrate that the ERP is meeting needs of clients and stakeholders.
The research program will lead to the intended outcomes if the research products and tools are
applied by the regions, states, tribes, and local governments." There is good evidence that this is
occurring and with continued involvement of clients in the research through pilot projects, it
should continue. As with ReVA and some of the new technologies to enhance water
management for ecosystem services, there should be specific plans to make all tools and
technologies available to users such as state governments, planning agencies, and private
consultants. This will require commitment of resources to technology transfer through both in-
person and online training.
Ecological Research Program Review Report
-------
VL LEADERSHIP
4e. Does the research reflect the current state-of-the-science, and meet the current and future
needs of EPA, science, and the ERP's customers?
6a. Does the ERP effectively engage stakeholders in its planning?
6b. Does the ERP have a process for using the results of the research, along with stakeholders'
feedback, to identify key research gaps and to update the ERP's research agenda?
6c. Are potential public benefits clearly articulated?
6d. What are the impediments, if any, to collaboration with other organizations? •
7. Are the ERP's research results being used by clients and stakeholders? Are these research
results consistent with the needs articulated by EPA's program and regional offices?
LONG-TERM GOAL 1
By 2010, national policy makers will have the tools and technologies to
develop scientifically defensible assessments of the state of our nation's
ecosystems and the effectiveness of existing national
programs and policies.
Question 4e: The research of LTG 1 is evidently of high scientific quality and reflective of the
state-of-the-art in-broad-scale environmental research, particularly in developing statistical
design and sampling strategies for measuring and monitoring ecosystem conditions. Because
landscapes are large enough to incorporate multiple interacting ecosystems and major regimes of
human influences on them, while still small enough for insightful mechanistic explorations,
landscape-level indicators provide critically important information for legislators and policy
makers at the national level and decision-makers and stakeholders at the local level. The ERP
has developed a suite of such indicators in recent years, and this research represents cutting-edge
science. Empirical testing of these indices, as well as additional core research in this extremely
important area, however, are clearly needed.
Question 6a: The LTG 1 program initially tended to develop methodologies and approaches that
were independent of stakeholder involvement once the basic national needs were identified.
Stakeholders increasingly are more involved in the process, from implementation of the
..conceptual designs and sampling protocols to field-testing and regional-application, and then to
the stage where there'is a firm engagement of LTG 1 products within a formal framework. The
LTG 1 sub-program is now at a point where stakeholders from many regions are interested in
adopting the methodologies, in some cases before EPA has resources to assist them.
Ecological Research Program Review Report
19
-------
Question 6b: More effective processes and formal mechanisms appear to be needed that allow
better communication between the ERP and its clients. These processes and mechanisms may be
sufficient, but were not evident to the Subcommittee because of time constraints.
Question 6c: The public benefits are well appreciated within the ERP science culture, and
among many state agencies. The Subcommittee members heard several verbal assessments and
read others that led to the conclusion that the public benefits were appreciated by the states. The
Subcommittee was not able to assess whether the public benefits were articulated well across the
entire spectrum of stakeholders, probably because of the time constraints involved in this
assessment. The Subcommittee recognizes that the public benefits are numerous and that they
are articulated within the culture of EPA and state agencies, but perhaps not clearly articulated to
all stakeholders, including the general public, in the same way.
Question 6d: There are no obvious impediments to collaboration with other organizations. The
three LTGs, each focusing mainly, but not exclusively, on a relatively explicit level of
organization and corresponding geographic scales,, should facilitate the process of identifying
and working with collaborative organizations outside EPA.
Question 7: The Subcommittee believes that research results are being used by clients and stake-
holders as illustrated in the boxes included in this report.
LONG-TERM GOAL 2
By 2010, states and tribes apply improved tools and methods to protect
and restore their valued ecological resources.
The ERP scientists and collaborators frequently are leaders in their respective scientific fields
and ORD has clearly supported outstanding research in many areas of ecosystem science. The
ERP is a valuable and integral part of EPA and essential to carrying out EPA's mission. The
ERP's leadership within the scientific community is evidenced by the participation of its
scientists on editorial boards, review panels, conferences, and other academic or information
exchange gatherings. There is evidence that the ERP scientists take a leadership role in
disseminating research results in many states, but this is not uniformly true for a variety of
reasons, some of which are beyond their control.
The process for stakeholder engagement in research planning is unclear. In many cases, it
appears to be ad hoc with fortuitous partnerships formed based on requests from entities or
similar interests. Although these partnerships have produced results that may translate into
useful products for states and tribes in general, implementation of a strategic planning process
that involves broad stakeholder input will increase these opportunities.
20
Ecological Research Program Review Report
-------
LONG-TERM GOAL 3
By 2012, decision-makers understand the importance of ecosystem
services and make informed, proactive management decisions that
consider a range of alternative outcomes.
ORD scientists and collaborators working on LTG 3 are among the leaders in this research in the
United States. No other federal research agency has as extensive or advanced program in
transferring tools to assess the provision of ecosystem services. Although numerous
collaborators and stakeholders already are engaged, the process of identifying and engaging them
could be more transparent. It would be very useful if resources were available to fund
cooperative development of the pilot projects and if these could be focused on areas with a wide
variety of resource management and environmental quality issues. In particular, when dealing
with rural areas and insuring the provision of ecosystem services as rural areas develop,
stakeholders representing agriculture and forestry should be more clearly involved as there is a
significant amount of terrestrial ecosystem assessment, landscape ecology, and ecosystem
modeling being conducted to understand how agriculture and forest practices affect ecosystem
services. Although the Subcommittee assumes that some level of coordination exists among
these activities, this was not apparent in the information presented to the Subcommittee.
Increased interactions among LTG 3 research scientists with other elements of ORD focusing on
socioeconomic research may result in significant opportunities to further leverage resources. In
addition, there was no evidence of interactions with the international community. Programs
addressing the issues outlined in LTG 3 are the subject of research programs in other countries.
The Subcommittee recommends that review of other ongoing programs will strengthen elements
of LTG 3.
Ecological Research Program Review Report 21
-------
VII. COLLABORATIONS
5. What is the scientific quality of the ERP's research products? Does the ERP ensure high
quality research through competitive, merit-based funding? If funds are not competitively
awarded, what process does the ERP use to allocate funds? Does this process ensure that
quality is maintained?
6a. Does the ERP effectively engage stakeholders in its planning?
6b. Does the ERP have a process for using the results of the research, along with stakeholders'
feedback, to identify key research gaps and to update the ERP's research agenda?
6c. Are potential public benefits clearly articulated?
6d. What are the impediments, if any, to collaboration with other organizations?
7. Are the results of the ERP being used by clients and stakeholders? Are these research results
consistent with the needs articulated by EPA's program and regional offices?
8. Will the ERP's completed and planned outputs lead to the intended outcomes that are
protective of our ecological resources?
LONG-TERM GOAL 1
By 2010, national policy makers will have the tools and technologies to
develop scientifically defensible assessments of the state of our nation's
ecosystems and the effectiveness of existing national
programs and policies.
Question 5: The quality of the science is adequate to high, depending on the standards used and
the subject area. The results from the congressionally mandated atmospheric deposition program
are exceptionally good, and meet internationally recognized science quality standards among
independent scientists active in the field, and have grounded EPA's policies in science. The
monitoring and assessment program has an excellent statistical design and more than adequate
sampling protocols, and is well on its way to accomplishing its leadership and implementation
goals. Because it also is in the first decade of what is a very long-term monitoring program, the
scientific quality of the monitoring and assessment program is not yet fully evident. The
program has very effectively engaged some state agencies, but with an anticipated focus on
Agency-style documents (primarily reports) arid less so on peer-reviewed literature that is more
widely distributed and which receives greater scrutiny (and concurrently, expanded opportunities
for improvement). The emphasis on fuller disclosure of the science products and their
implementation will eventually gain greater prominence as monitoring continues beyond the
alpha phase and generates results extending over several decades. Much of the research is done
through noncompetitive funding directly to stakeholders, which are primarily states, as a direct
and intentional result of the integration of stakeholders into the field program.
22 ' Ecological Research Program Review Report
-------
Question 6a: LTG 1 research can be further improved by more systematically and proactively.
engaging decision-makers and stakeholders at local, state, and national levels.
Question 6b: More effective processes and formal mechanisms appear to be needed that allow
better communication between the ERP and its clients. These processes and mechanisms may be
sufficient, but were not evident to the Subcommittee because of time constraints.
Question 6c: The potential benefits to the public and stakeholders are clearly articulated.
Question 6d: There are no obvious impediments to collaboration with other organizations. The
three LTGs, each focusing mainly, but not exclusively, on a relatively explicit level of
organization and corresponding geographic scale, should facilitate the process of identifying and
working with collaborative organizations outside EPA.
Question 7: The research results of LTG 1 have been widely used by clients and stakeholders.
Numerous examples exist in the documents and presentations provided to the Subcommittee.
More formal mechanisms for facilitating communications with regional offices and keeping
track of outreach activities, however, appear to be needed, although the Subcommittee is willing
to accept that these mechanisms exist, but were not revealed to the members during the course of
the Subcommittee's deliberations.
Question 8: LTG 1 addresses some of the most pressing and challenging environmental research
questions today, and is extremely relevant to EPA's mission. The research activities under LTG
1 so far have been quite productive, resulting in important outcomes. Maintaining the strength of
LTG 1 is imperative for the continued success of the ERP. The accomplishments of the ERP are
critical building blocks to achieve EPA's overall goals and objectives of protecting and restoring
the nation's ecosystems. The ERP, however, cannot by itself accomplish all of EPA's stated
goals and objectives because of the size and complexity of the problems at hand. High-quality
involvement makes enhancement more likely, and demonstrably so where there has been enough
time to move from program beginnings to maturation and implementation.
LONG-TERM GOAL 2
By 2010, states and tribes apply improved tools and methods to protect
and restore their valued ecological resources.
ORD has a superb track record of collaboration with a variety of partners at the level of specific
research projects (see Box 2 and Box 3). Such collaborations, especially for LTG 2 projects,
seem typically to be the result of "willing and able" partners expressing an interest in working
with EPA. Targeted outreach to nontraditional partners should be pursued as part of a strategic
communication process involving stakeholders.
Ecological Research Program Review Report ' 23
-------
LONG-TERM GOAL 3
By 2012, decision-makers understand the importance of ecosystem
services and make informed, proactive management decisions that
consider a range of alternative outcomes.
Collaboration on specific pilot
projects has been excellent and the
Subcommittee expects this to
continue. Collaboration would be
enhanced by the ability to enter into
cooperative research projects that
address specific resource management
issues. Outcomes would be enhanced
by identifying and collaborating with
representatives of expected user
groups at the outset of pilot projects,
in addition to the specific project
collaborators. This could be
accomplished by convening broad-
based user groups to comment on
specific pilot projects at an early stage
of the projects.
tssessmei
,
BnvJronnaental
HH
^ISSE!
JutaM^imptonsib
cati'communiae
"
9MS!ffiS>!S
24
Ecological Research Program Review Report
-------
VIII. RESOURCES
Question 1: What would be the minimum research program that would be both effective and
successful?
Resources, both intramural and extramural (i.e., STAR), are adequate for measured progress, but
• are clearly inadequate to undertake other than a handful of extensive projects.'such as the
Willamette Basin study and ReVA/SEQL. Currently, there are many issues that are important to
clients, both intramural and extramural, that cannot be addressed because of resource limitations.
The ERP has done an excellent job of using cooperation with the programs of various groups
within and outside of government to extend the resources available. The ERP scientists often
provide expertise and obtain the necessary information for proof-of-concept or model
parameterization by their participation in ongoing projects.
The ERP has done an excellent job of leveraging resources, but a fairly poor job of documenting
the magnitude of these leveraged funds. The ERP should institute a mechanism to track these
contributions so that they can be articulated and reported in a quantitative manner. Additional
project funds are needed to develop more pilot projects.
The loss of the STAR Program would hurt the integrity of the entire ERP. Something similar to
STAR must be maintained in accordance with the three LTGs to assure the quality of the ERP's
collaboration, efficiency, leveraging of funds, and intellectual capital.
Ecological Research Program Review Report
25
-------
IX. INTEGRATION AND FUTURE DIRECTIONS
EPA must perform scientific research to provide the baseline knowledge to establish technical
standards for regulatory decisions and to provide appropriate tools and methodologies to those
who are charged with measuring and conserving the health of our national environment This
section addresses the appropriateness of the ERP's three LTGs and their integration; the ERP's
effectiveness in meeting its responsibilities to its customers; and, to the extent possible, an
assessment of the future directions of ERP research. In evaluating the program and its LTGs, it
is important to reiterate that ORD has a research mission with the primary responsibility of
developing scientific knowledge, tools, and methodologies to serve the mission of the regulatory
offices of EPA as well as to meet the technical and scientific needs of state governments and
tribes. In this sense, these EPA regulatory offices are the customers of ORD and its constituent
research programs. Other important customers include other federal agencies, states, tribal
governments, and decision-makers.
The ERP has many substantial accomplishments to its credit and the BOSC Ecological Research
Subcommittee rates the overall quality of the research as good to excellent. Variation exists
among the projects as is inevitably the case within any large and complex organization, but
overall, EPA and the nation are well served by the research accomplishments within the ERP. In
many cases, there is ample evidence of demonstrated outcomes, for example the Willamette
River Basin project (see Box 1) and the wide implementation of the results of EMAP across the
nation. Other programs that deserve specific mention include the successful adoption of bio-
indicators as measures of environmental health, the MALA, research program, and the various
programs focused on developing tools to assess the water and air quality of our nation.
Following the disappointing PART review, the ERP's LTGs were revised and reduced from four
to three. The new LTGs thus supersede, in part, the current MYP. This review took place in the
context of both budgetary changes and changes in the formulation of the ERP goals. Moreover,
the review contains elements that are new to the BOSC, so the situation is quite fluid and this
review needs to viewed within this very dynamic context. In responding to the charge questions,
the appropriateness of both the reformulated LTGs and the budget reduction strategy in the
context of the goals was discussed. Because the PART review challenged the ERP to explain
more effectively how its research leads to concrete outcomes in the field, this question received
considerable attention. Standard questions about program quality and performance that are a
normal review component also were included in this review.
This is a challenging time for the ERP in terms of budget. The past 2 years have witnessed
substantial budget reductions (see Figure 2) and the ERP recently has experienced a
disappointing PART review. The budget for Fiscal Year (FY) 2005 is not yet finalized (as of
March 2005 when the review materials were provided to the Subcommittee) and Congress has
not acted on the FY 2006 budget request and thus the trajectory may change, but current data
suggest a reduction of approximately 25 percent over a 2-year period (see Figure 2). Despite
major reductions in budget, almost no changes have occurred in full-time equivalent (FTE)
employees supporting the ERP (see Figure 3). This disjunction reflects the fact that virtually all
26 ' ' Ecological Research Program Review Report
-------
Figure 2. Total Ecological Research Program Resources,
2001-2006 (total dollars in millions)
Figure 3. Total Ecological Research Program Resources,
2001-2006 (full-time equivalents)
Ecological Research Program Review Report
27
-------
of the budget reductions were in extramural programs, particularly in the STAR Program. The
present and future budgetary uncertainties make it difficult to comment on future directions.
Maintaining the current research portfolio and extending it into the future is both essential and of
high priority in meeting EPA's mission of protecting, sustaining, and restoring the health of the
nation's ecosystems. Nevertheless, current budgetary constraints dictate a full re-evaluation of
the priorities of the ERP. Future directions must be re-evaluated in light of current and expected
future budgetary resources.
As interpreted by this Subcommittee based on the presentations, the three LTGs have a
consistent logic (see Figure 1). The LTGs appropriately capture the mandate to develop
scientific knowledge, tools, and methodologies that serve the mission of EPA's regulatory
offices and meet the information needs of state governments and tribes. LTG 3 imposes the
special challenge of reaching local and regional decision-makers. The ERP must depend on
other EPA offices to implement programs in the field and to work directly with decision-makers,
and the program should not be asked to take on an extension role when other components of
EPA are better equipped to assume such a responsibility. Thus, the ERP clearly is dependent on
other EPA offices to create demonstrated outcomes in the field. It certainly is appropriate to -
expect the entire Agency to function as a seamless organization even though the BOSC reviews
are piece-meal and are not intended to look at the total organizational context.
28
Ecological Research Program Review Report
-------
X. FINDINGS AND RECOMMENDATIONS
The ERP is of good to high quality and program leadership is very good. The Subcommittee is
impressed with the substantial accomplishments of the ERP as evidenced by the variety of
excellent tools, methods, and research outcomes documented in the materials provided for the
review. The findings and recommendations below suggest areas for improvement, but it should
be emphasized that the ERP is fulfilling its mission as well as providing good value to the nation.
The Subcommittee's findings and recommendations are presented below by LTG:
4* The integration of LTG 1 with the other LTGs can be further improved through designing
research projects specifically for cross-level integration and by reinforcing rules set by
research programs for close collaborations between EPA and outside researchers at the
national, regional, and local levels. This integration can be facilitated by working with other
federal agencies, such as the National Science Foundation, U.S. Department of Agriculture,
National Aeronautics and Space Administration, and National Oceanic and Atmospheric
Administration, to develop multiple-scale, interdisciplinary, place-based, and use-inspired
research programs.
<* Although the overall quality of research under LTG 1 is excellent, high transparency in ,
research design, implementation, and evaluation and close collaborations with external
scientists must be maintained to assure that this high-quality research will persist.
•$• Research for all three LTGs would be improved by collaborations with international
scientific communities. This is important because many environmental problems are either
physically connected or ecologically similar worldwide and because such scholarly exchange
. among countries will help improve the global environment in which we are embedded.
•$• Clear and substantial progress toward LTG 2 was demonstrated in much of the mature
research, although this could not be readily evaluated for research at earlier stages of
development. Stakeholder input to identify research gaps does occur, but the process to brief
decision-makers of results and accept feedback currently is informal or often lacking. The
effectiveness of this aspect of the program could be improved by establishing timely and
regular communications with a broad array of stakeholders using an established procedure.
•$• LTG 3 is a newly reorganized program element, and there is an opportunity to develop an
explicit research plan. The Subcommittee suggests that the plan include specific programs
and projects with specific deliverables and timelines such that, in the future, progress can be
tracked and the quality, efficiency, and impact of the program elements can be evaluated.
Furthermore, the BOSC should review the MYP when it is developed.
•$• LTG 3 requires better integration with, and articulation of, outcomes at local levels. This is
essential to achieve EPA's mandate, but the Agency must be aware of the dangers of asking a
very good research organization to take on responsibilities that it is not structured to
accomplish. Responsibilities for communication and dissemination of results certainly rest
Ecological Research Program Review Report 29
-------
with the ERP, but other elements of EPA also have responsibilities for client and stakeholder
communication. It is important to recognize that ORD has a primary research mission.
There is a danger in assigning other priorities to ORD because the research mission may be
compromised. The time and talents of ORD's research scientists need to be focused on the
research mission. At the same time, careful tracking of outcomes is essential to assure that
the research conducted by the ERP is appropriate and that it addresses customer priorities.
Currently, there is no plan for an extramural component to LTG 3. The Subcommittee
members believe that, historically, partnering with other federal, state, and tribal agencies;
academic institutions; and NGOs has been very successful. This has increased the
productivity of the program by leveraging the resources and creativity of these partners. The
Subcommittee understands that difficult decisions need to be made relative to resource
allocations, but suggests that the elimination of extramural programs will result in
disproportionately less productivity and creativity. The Subcommittee recommends that
some form of extramural cooperation be re-established to leverage resources and continue to
provide flexibility in the research program.
More rigorous program-wide mechanisms should be in place to maximize collaborations
between EPA researchers and external scientists. This assures that the ERP research is of the
highest scientific quality and of utmost relevance to EPA's mission. Close collaborations
between EPA researchers and external scientists are vital. Mechanisms for forging and
maintaining such collaborations need to be in place to assure intellectual vitality and
openness to new ideas. For example, open grant competitions, such as STAR, should be
slightly modified with an explicit requirement for collaboration with EPA researchers. Such
collaborations make the best use of intellectual and logistic resources within and outside
EPA; facilitate high-quality scientific output that is intimately tied with the goals of the ERP;
and promote scholarly exchanges and information sharing that enhance EPA's research
capacities. For the ERP to be most productive and relevant, it is crucial to create a research
environment in which the ERP's own researchers can feel excited about, and rewarded by,
external collaborations rather than feeling pressured for competing resources with outsiders.
EPA and ORD must increase involvement of stakeholders (especially external stakeholders)
in setting research priorities and targeting research efforts (adaptive research management).
Effective communication is vital and must be viewed as an essential step in turning research
results into outcomes. Moreover, those charged with administering the organization must
receive timely and appropriate measures of research effectiveness.
•' Clearly articulate specific outcomes of research projects, including linked performance
indicators.
• Institute a formal process for sharing and disseminating research results to stakeholders.
Critical scientific peer review is the standard measure of quality in judging among potential
research investments and this standard has served our nation very well. The ERP utilizes
peer review in judging many, but not all, aspects of its research portfolio. There is room for
improvement in the application of peer review to potential projects within the organization.
30
Ecological Research Program Review Report
-------
At the same time, EPA should be generally complimented on its dedicated efforts to
implement peer review across the organization.
The ultimate concern of EPA is to measure and improve the health of the nation's
environment, yet some things are missing from what might be viewed as a well-balanced
research portfolio.
* The ERP program is heavily oriented towards aquatic ecosystems and EPA's mission of
conserving the nation's water resources justifies this focus. Moreover, many terrestrial
issues are indirectly addressed through the aquatic program. Nevertheless, a balanced
research portfolio also requires attention to impacts on terrestrial ecosystems, especially
relative to clean water and nonpoint source pollution. It is important to better integrate
ERP research with assessments of terrestrial ecosystems done by other entities, especially
other federal agencies.
i
• Improving the health of the environment clearly requires a better understanding of human
motivations and behavior. Some projects such as the MAIA involve a social science
component, yet this dimension of science is largely absent from the ERP portfolio. It
may be that these responsibilities lie elsewhere in ORD and the Subcommittee's review
failed to see the full picture. Nevertheless, we urge the ERP to be mindful of the full
spectrum of research required to meet the goals of EPA.
More could be done to improve the tracking of outcomes. Specifically, EPA, ORD, and the
ERP need to work harder to foster better communication among EPA offices, with the other
elements of the Executive Branch of government, and with external stakeholders. The
Subcommittee urges that more be done to integrate new research paradigms into an
"outcomes-based mentality" and that these efforts include specific provisions for tracking on-
the-ground outcomes. Somewhere within EPA, but not necessarily within ORD, there needs
to be a focus on better integration of scientific results into stakeholder education and
decision-making.
The recent budget reductions have been difficult for the ERP, and the decision to take these
reductions in extramural programs is understandable when viewed from the perspective of
salvaging an effective research organization. This may appear to be a reasonable short-terni
response to weathering a storm, but it is dangerous. EPA and ORD should take a broader
view and consider how to achieve maximum results in a new permanent budgetary
environment. Clearly, abandoning extramural research grants is not a cost-effective strategy
in this circumstance. Competitive research grants, as embodied in STAR and other
programs, leverage considerable resources as well as bring new thinking and approaches into
the organization. They also connect the organization to one component of the external
customer base. The lost leveraging capability, reduced connection to other sources of
innovation, and weakening of communication channels, over the long term, will
disproportionately diminish the cost-effectiveness of EPA's research.
Ecological Research Program Review Report . . '31
-------
APPENDIX A: BOSC ECOLOGICAL RESEARCH
SUBCOMMITTEE MEMBERS
Dr. Michael T. Clegg1
Department of Ecology & Evolution
University of California
Irvine, CA 92697-1010
Tel: 949-824-4490 or 949-824-4489
E-mail: mclegg@uci.edu
Russel Frydenborg
Florida Department of Environmental Protection
2600 Blair Stone Road
Tallahassee, FL 32399-2400
Tel: 850-245-8063
E-mail: russel.frydenborg@dep.state. flus
Dr. John P. Giesy2
Department of Zoology .,
Michigan State University
East Lansing, MI 48824-1222
Tel: 517-353-2000
E-mail: jgiesy@aol.com
Dr. Richard Lowrance
Agricultural Research Service
USDA-ARS
P.O. Box 748
Tifton,GA31794
Tel: 229-386-3894
E-mail: lorenz@tifton.usda.gov
Dr. Sue A. Thompson
Pennsylvania Biodiversity Partnership
16 Terminal Way
Pittsburgh, PA 15219-1209
Tel: 412-481-4100
E-mail: thompson@pabiodiversity.org
Dr. R. Eugene Turner
.Coastal Ecology Institute
Louisiana State University
Baton Rouge, LA 70803
Tel: 225-578-6454
E-mail: euturne@lsu.edu
Dr. Jianguo (Jingle) Wu
School of Life Sciences
Arizona State University
P.O. Box 874501
Tempe, AZ 85287-4501
Tel: 480-965-1063
E-mail: jingle.wu@asu.edu
1 Member of BOSC Executive Committee and Chair of the Ecological Research Program Subcommittee.
2 Member of BOSC Executive Committee.
32
Ecological Research Program Review Report
-------
APPENDIX B: BOSC ECOLOGICAL RESEARCH
SUBCOMMITTEE MEETING DATES
17 February 2005 Teleconference
Participants:
Subcommittee Members: Michael T. Clegg, Russel Frydenborg, John P. Giesy, Richard Lowrance, Sue
A. Thompson, R. Eugene Turner, Jianguo Wu
EPA Attendees: Steven Hedtke, Jennifer Robbins, Kevin Summers, Greg Susanke
Other Participants: Stefanie Nelson, The Scientific Consulting Group (SCG)
3 March 2005 Teleconference
Participants:
Subcommittee Members: Michael T. Clegg, Russel Frydenborg, John P. Giesy, Richard Lowrance, Sue
A. Thompson, R. Eugene Turner, Jianguo Wu
EPA Attendees: Deb Gonima, Kevin Summers, Greg Susanke
Other Participants: Stefanie Nelson, SCG
7-9 March 2005 Site Visit at EPA Facilities, Research Triangle Park, North Carolina
Participants: • -. .
Subcommittee Members: Michael T. Clegg, Russel Frydenborg, John P. Giesy, Richard Lowrance, Sue
A. Thompson, R. Eugene Turner, Jianguo Wu
EPA Attendees: Rochelle Araujo, Mark Bagley, Joan Baker, Thomas Barnwell, Allen Basala, William
Benson, Roger Blair, Benjamin Blaney, Michael Blum, David Bolgrien, Patricia Bradley, Arden
Calvert, Timothy Canfield, Robin Dennis, Robert Dyer, Virginia Engle, Gary Foley, Tome Fontaine,
Jonathan Garber, Deborah Gonima, Iris Goodman, Richard Greene, Steven Hedtke, Jeffrey Herrick,
Robert Hetes, Virginia Houk, Laura Jackson, William Jenkins, John Johnston, Bruce Jones, Janet
Keough, Daniel Kluza, Lorelei Kowalski, Gail Lacy, James Lazorchak, Timothy Lewis, Deborah
Mangis, Michael McDonald, Megan Mehaffey, Bruce Mintz, Marirosa Molina, Jewel Morris, Wayne
Munns, Anne Neale, Susan Norton, Robert Olexsey, Tony Olsen, John Paul, Steve Paulsen, Gina
Perovich, Drew Pilant, Robert Puls, Jack Puzak, Mary Reiley, Denise Sailstad, Victoria Sandiford,
Gregory Sayles, Stephen Schmelling, Nathan Schumacker, Victor Serveiss, Mark Shanis, Michael
Slimak, John Stoddard, Elise Striz, Kevin Summers, Greg Susanke, Ginger Tennant, Jo Ann
Thompson, Paul Wagner, Randall Waite, Thomas Wall, David Walters, Jordan West, Joseph
-------
-------
APPENDIX B: BOSC ECOLOGICAL RESEARCH
SUBCOMMITTEE MEETING DATES
17 February 2005 Teleconference
Participants:
Subcommittee Members: Michael T. Clegg, Russel Frydenborg, John P. Giesy, Richard Lowrance, Sue
A. Thompson, R. Eugene Turner, Jianguo Wu
EPA Attendees: Steven Hedtke, Jennifer Robbins, Kevin Summers, Greg Susanke
Other Participants: Stefanie Nelson, The Scientific Consulting Group (SCG)
3 March 2005 Teleconference
Participants:
Subcommittee Members: Michael T. Clegg, Russel Frydenborg, John P. Giesy, Richard Lowrance, Sue
A. Thompson, R. Eugene Turner, Jianguo Wu
EPA Attendees: Deb Gonima, Kevin Summers, Greg Susanke
Other Participants: Stefanie Nelson, SCG
7-9 March 2005 Site Visit at EPA Facilities, Research Triangle Park, North Carolina
Participants: • -. .
Subcommittee Members: Michael T. Clegg, Russel Frydenborg, John P. Giesy, Richard Lowrance, Sue
A. Thompson, R. Eugene Turner, Jianguo Wu
EPA Attendees: Rochelle Araujo, Mark Bagtey, Joan Baker, Thomas Barnwell, Allen Basala, William
Benson, Roger Blair, Benjamin Blaney, Michael Blum, David Bolgrien, Patricia Bradley, Arden
Calvert, Timothy Canfield, Robin Dennis, Robert Dyer, Virginia Engle, Gary Foley, Tome Fontaine,
Jonathan Garber, Deborah Gonima, Iris Goodman, Richard Greene, Steven Hedtke, Jeffrey Herrick,
Robert Hetes, Virginia Houk, Laura Jackson, William Jenkins, John Johnston, Bruce Jones, Janet
Keough, Daniel Kluza, Lorelei Kowalski, Gail Lacy, James Lazorchak, Timothy Lewis, Deborah
Mangis, Michael McDonald, Megan Mehaffey, Bruce Mintz, Marirosa Molina, Jewel Morris, Wayne
•* Munns, Anne Neale, Susan Norton, Robert Olexsey, Tony Olsen, John Paul, Steve Paulsen, Gina
Perovich, Drew Pilant, Robert Puls, Jack Puzak, Mary Reiley, Denise Sailstad, Victoria Sandiford,
Gregory Sayles, Stephen Schmelling, Nathan Schumacker, Victor Serveiss, Mark Shanis, Michael
Slimak, John Stoddard, Elise Striz, Kevin Summers, Greg Susanke, Ginger Tennant, Jo Ann
Thompson, Paul Wagner, Randall Waite, Thomas Wall, David Walters, Jordan West, Joseph
Williams, Dorsey Worthy
Other Participants: Tom Atkeson (Florida Department of Environmental Protection), Roger Batterman
(Wisconsin! Department of Environmental Protection), Vicki Bott (Bowman) (University of North
Carolina-Charlotte Urban Institute), Stefanie Nelson (SCG), Kathryn Saterson (Duke University),
Robert Stevens (Florida Department of Environmental Protection), Scott Urquhart (Colorado State
University), Lexia Weaver (University of North Carolina-Chapel Hill)
Ecological Research Program Review Report 33
-------
------- |