UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                          WASHINGTON D.C. 20460
                                                     OFFICE OF THE ADMINISTRATOR
                                                      SCIENCE ADVISORY BOARD

                               September 30, 2008

EPA-SAB-08-011

The Honorable Stephen L. Johnson
Administrator
U.S. Environmental Protection Agency
1200 Pennsylvania Avenue, N.W.
Washington, D.C. 20460

       Subject:  SAB Advisory on the EPA Ecological Research Program Multi-Year
               Plan

Dear Administrator Johnson:

   EPA's Office of Research and Development (ORD) requested that the Science
Advisory Board (SAB) provide advice on the Agency's draft Ecological Research
Program Multi-Year Plan FY 2008 - 2014 (Plan). The Plan presents proposed goals,
objectives, and research questions for EPA's Ecological Research Program (ERP) and
also lays out an implementation strategy for the Program.  In response to the Agency's
advisory request, the SAB Ecological Processes and Effects Committee (Committee)
reviewed the draft Plan. To augment the expertise on the Committee for this advisory
activity, several  SAB committee members with expertise in valuation of ecosystem
services also participated in the review.  EPA sought the SAB's advice on:  1) the
appropriateness  and utility of the strategic direction articulated in the Plan;  2) the
adequacy of the  goals, objectives, and research questions in meeting the purpose of the
Program; 3) the  logic model and implementation approach in the Plan; 4) anticipated
challenges to achieving the overall goal of the Ecological Research Program; 5)
suggestions for measuring the progress, productivity, efficiency, and effectiveness of the
Program; and 6) recommendations to enhance EPA's ability to leverage available
resources within and outside the Agency. The enclosed advisory report provides the
advice and recommendations of the Committee.

   EPA's draft Plan articulates a new strategic direction that focuses on quantifying
ecosystem services and their contribution to human health and well-being.  The SAB
strongly supports this strategic direction and commends the Agency for developing a
research program that, if properly funded and executed, has the potential to be
transformative for environmental decision making as well as for ecological science.  The
SAB finds that the research focus on ecosystem services represents a suitable approach to

-------
integrate ecological processes and human welfare.  The ERP's focus on ecosystem
services can provide a sound foundation for environmental decisions and regulation
based on the dependence of humans on ecological conditions and processes.

   Although the SAB strongly supports the new strategic direction of the ERP, we have a
number of concerns about the draft Plan.  Most of these are related to the tension between
stating an important and ambitious vision and producing a practical implementation plan
for a future that includes a limited and uncertain budget. The SAB is extremely
concerned that the resource allocation for the ERP is too small to accomplish the
ambitious program goals. Studying ecosystem services is a new field and the ORD staff
skill set may be insufficient to conduct all of the research proposed in the Plan.  Most
notable is the lack of in-house expertise in ecosystem services valuation and
outreach/education. The Agency could be better served by acquiring outside expertise in
this area to supplement the research program. We therefore strongly encourage EPA to
provide additional intramural and extramural support (e.g., through Science to Achieve
Results [STAR] Program grants) for the ERP, not only for technical elements but also for
critical outreach/education efforts.  Furthermore, the Plan represents a considerable
change in research direction of the ERP, whose previous research made significant
contributions to the science of ecological monitoring and assessment. As these efforts are
moved to other parts of the Agency, it is essential that EPA's strength and leadership in
this area be maintained.

   The SAB also finds  that the decadal overview of proposed ecological research would
be most useful if it included more detailed information concerning the knowledge gaps,
research questions, variables, geographic scales, and sites to be investigated. Similarly,
clear identification of the Agency's research partners and clients would facilitate
collaborative interactions.  The SAB has  suggested improvements in the Plan to address
these and other issues.  In particular, the SAB recommends improvements in the
following areas.

Discussion of the Strategic Vision

   •   The justification of research priorities in the Plan should be strengthened.  The
       discussion of research priorities should include the rationale leading to
       accomplishing initial goals; selecting geographic locations for research; and
       identifying scales of efforts. This discussion should also more clearly articulate
       how the concept of ecosystem services will provide guidance to EPA programs.

   •   The Plan should describe the linkages between EPA's previous ecological risk
       assessment research and  the proposed new research direction of quantifying
       ecosystem services and their contribution to human health and well-being.

   •   The intended audience of the plan and clients of the Research Program should be
       clearly identified. Outreach efforts should focus on educating those clients to
       ensure that the research products will be used appropriately in decision making.
                                        11

-------
       The SAB notes that outreach and education has not historically been part of
       ORD's work and therefore additional expertise may be needed in this area.

    •   The Plan should clearly describe existing and planned interactions among
       proposed research program components, with other Agency Programs, and with
       other federal agencies involved in assessment of ecosystem services to avoid
       duplication of effort and promote coordination and synergy. In this regard, the
       SAB recommends collaborating with other federal agencies and academic
       scientists to conduct a scientific community assessment of status and trends in
       ecosystem services in the U.S.

    •   The Plan should explicitly  recognize the role that emerging new ideas will play in
       the development of an adaptive program to respond to a rapidly changing arena
       for environmental and human welfare.

Discussion of Research Goals and Questions

    •   The Plan should contain additional information describing research project design
       and uncertainty associated  with the research to be completed.  ORD has indicated
       that these critical details will be addressed in follow-up implementation plans.
       The SAB recommends that these implementation plans be peer reviewed.

    •   The Plan should describe how partnerships with non-governmental organizations,
       professional societies, private businesses, and foundations, including international
       partnerships, can be enhanced to accomplish stated goals and objectives.

    •   Research with international partners  should be incorporated into the Plan in order
       to understand transboundary  conditions and connections that extend across
       national borders;

    •   The Plan should contain a more transparent explanation of the process and criteria
       used to select sites for place-based demonstration projects, following the
       procedure recommended in the body of this report to assure a sufficient number
       and diversity of sites;

Program Implementation Strategy

    •   The Plan should explain how program success will be evaluated on the basis of
       progress toward specifying relevant ecological endpoints and production
       functions, not on the basis of achieving the ultimate goals of EPA's research and
       regulatory mission.
                                        in

-------
   Thank you for the opportunity to provide advice on this important topic.  The SAB
looks forward to receiving your response to this advisory.

                                  Sincerely,
       /Signed/                                      /Signed/

Dr. M. Granger Morgan, Chair                       Dr. Judith L. Meyer, Chair
Science Advisory Board                             Ecological Processes and Effects
                                                   Committee
Enclosures
                                       IV

-------
                                  NOTICE

   This report has been written as part of the activities of the EPA Science Advisory
Board, a public advisory group providing extramural scientific information and
advice to the Administrator and other officials of the Environmental Protection
Agency.  The Board is structured to provide balanced, expert assessment of scientific
matters related to the problems facing the Agency. This report has not been reviewed
for approval by the Agency and, hence, the contents of this report do not necessarily
represent the views and policies of the Environmental Protection Agency, nor of other
agencies in the Executive Branch of the Federal government, nor does mention of
trade names or commercial products constitute a recommendation for use. Reports of
the EPA Science Advisory Board are posted on the EPA website at
http://www.epa.gov/sab.

-------
                                  Enclosure 1
                   U.S. Environmental Protection Agency
                         Science Advisory Board
               Ecological Processes and Effects Committee

 Augmented for the Advisory on the EPA Ecological Research Program
                             Multi-Year Plan
CHAIR
Dr. Judith L. Meyer, Distinguished Research Professor Emeritus, Odum School of
Ecology, University of Georgia, Athens, GA
ECOLOGICAL PROCESSES AND EFFECTS COMMITTEE MEMBERS
Dr. Fred Benfield, Professor of Ecology, Department of Biological Sciences, Virginia
Tech, Blacksburg, VA

Dr. Ingrid Burke, Professor, Department of Forest, Rangeland, and Watershed
Stewardship, Colorado State University, Fort Collins, CO

Dr. G. Allen Burton, Professor and Director, Cooperative Institute for Limnology and
Ecosystems Research, School of Natural Resources and Environment, University of
Michigan, Ann Arbor, MI

Dr. Peter M. Chapman, Principal and Senior Environmental Scientist, Environmental
Sciences Group, Golder Associates Ltd., North Vancouver, BC, Canada

Dr. Loveday Conquest, Professor and Associate Director, School of Aquatic and
Fishery Sciences, University of Washington, Seattle, WA

Dr. Wayne Landis, Professor and Director, Institute of Environmental Toxicology,
Western Washington University, Bellingham, WA

Dr. James Oris, Professor, Department of Zoology, Miami University, Oxford, OH

Dr. Charles Rabeni, Leader of the Missouri Cooperative Fish and Wildlife Research
Unit, U.S. Geological Survey, University of Missouri, Columbia, MO

Dr. Amanda Rodewald, Associate Professor of Wildlife Ecology, School of
Environment and Natural Resources, The Ohio State University, Columbus, OH

Dr. James Sanders, Director and Professor, Skidaway Institute of Oceanography,
Savannah, GA
                                     VI

-------
Dr. Ivor van Heerden, Associate Professor and Director, Department of Civil and
Environment Engineering, LSU Hurricane Public Health Research Center, Louisiana
State University, Baton Rouge, LA
OTHER SAB COMMITTEE MEMBERS
Dr. Gregory Biddinger, Coordinator, Natural Land Management Programs, Toxicology
and Environmental Sciences, ExxonMobil Biomedical Sciences, Inc., Houston, TX

Dr. James Boyd, Senior Fellow, Director, Energy and Natural Resources Division,
Resources for the Future, Washington, DC

Dr. Terry Daniel, Professor of Psychology and Natural Resources, Department of
Psychology, Environmental Perception Laboratory, University of Arizona, Tucson, AZ

Dr. Otto C. Doering III, Professor, Department of Agricultural Economics, Purdue
University, W. Lafayette, IN

Dr. William Moomaw, Professor of International Environmental Policy and Director of
the Center for International Environment and Resource Policy, The Fletcher School of
Law and Diplomacy, Tufts University, Medford, MA

Dr. Kathleen Segerson, Professor, Department of Economics, University of
Connecticut, Storrs, CT
SCIENCE ADVISORY BOARD STAFF

Dr. Thomas Armitage, Designated Federal Officer, U.S. Environmental Protection
Agency, Washington, DC
                                      vn

-------
                                 Enclosure 2
                  U.S. Environmental Protection Agency
                          Science Advisory Board

CHAIR
Dr. M. Granger Morgan, Lord Chair Professor in Engineering, Department of
Engineering and Public Policy, Carnegie Mellon University, Pittsburgh, PA
SAB MEMBERS
Dr. Gregory Biddinger, Coordinator, Natural Land Management Programs, Toxicology
and Environmental Sciences, ExxonMobil Biomedical Sciences, Inc., Houston, TX

Dr. Thomas Burke, Professor, Department of Health Policy and Management, Johns
Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD

Dr. James Bus, Director of External Technology, Toxicology and Environmental
Research and Consulting, The Dow Chemical Company, Midland, MI

Dr. Deborah Cory-Slechta, Professor, Department of Environmental Medicine, School
of Medicine and Dentistry, University of Rochester , Rochester, NY

Dr. Maureen L. Cropper, Professor, Department of Economics, University of
Maryland, College Park, MD

Dr. Virginia Dale, Corporate Fellow, Environmental  Sciences Division, Oak Ridge
National Laboratory, Oak Ridge, TN

Dr. Kenneth Dickson, Regents Professor, Department of Biological Sciences, University
of North Texas, Aubrey, TX

Dr. David A. Dzombak, Walter J. Blenko Sr. Professor of Environmental Engineering,
Department of Civil and Environmental Engineering,  College of Engineering, Carnegie
Mellon University, Pittsburgh, PA

Dr. Baruch Fischhoff, Howard Heinz University Professor, Department of Social and
Decision Sciences, Department of Engineering and Public Policy, Carnegie Mellon
University, Pittsburgh, PA

Dr. James Galloway, Professor, Department of Environmental Sciences, University of
Virginia, Charlottesville, VA

Dr. James K. Hammitt, Professor, Center for Risk Analysis, Harvard University,
Boston, MA
                                     Vlll

-------
Dr. Rogene Henderson, Senior Scientist Emeritus, Lovelace Respiratory Research
Institute, Albuquerque, NM

Dr. James H. Johnson, Professor and Dean, College of Engineering, Architecture &
Computer Sciences, Howard University, Washington, DC

Dr. Bernd Kahn, Professor Emeritus and Director, Environmental Radiation Center,
Nuclear and Radiological Engineering Program, Georgia Institute of Technology,
Atlanta, GA

Dr. Agnes Kane, Professor and Chair, Department of Pathology and Laboratory
Medicine, Brown University, Providence, RI

Dr. Meryl Karol, Professor Emerita, Graduate School of Public Health, University of
Pittsburgh, Pittsburgh, PA

Dr. Catherine Kling, Professor, Department of Economics, Iowa State University,
Ames, IA

Dr. George Lambert, Associate Professor of Pediatrics, Director, Center for Childhood
Neurotoxicology, Robert Wood Johnson Medical School-UMDNJ, Belle Mead, NJ

Dr. Jill Lipoti, Director, Division of Environmental Safety and Health, New Jersey
Department of Environmental Protection, Trenton, NJ

Dr. Michael J. McFarland, Associate Professor, Department of Civil and
Environmental Engineering, Utah  State  University, Logan, UT

Dr. Judith  L. Meyer, Distinguished Research Professor Emeritus, Odum School of
Ecology, University of Georgia , Athens , GA

Dr. Jana Milford, Professor, Department of Mechanical Engineering, University of
Colorado, Boulder, CO

Dr. Rebecca Parkin, Professor and Associate Dean, Environmental and Occupational
Health, School of Public Health and Health Services, The George Washington University
Medical Center, Washington, DC

Mr. David  Rejeski, Director, Foresight and Governance Project, Woodrow Wilson
International Center for Scholars, Washington, DC

Dr. Stephen M. Roberts, Professor, Department of Physiological Sciences, Director,
Center for Environmental and Human Toxicology, University of Florida, Gainesville, FL

Dr. Joan B. Rose, Professor and Homer Nowlin Chair for Water Research, Department
of Fisheries and Wildlife, Michigan State University, East Lansing, MI
                                       IX

-------
Dr. James Sanders, Director and Professor, Skidaway Institute of Oceanography,
Savannah, GA

Dr. Jerald Schnoor, Allen S. Henry Chair Professor, Department of Civil and
Environmental Engineering, Co-Director, Center for Global and Regional Environmental
Research, University of Iowa, Iowa City, IA

Dr. Kathleen Segerson, Professor, Department of Economics, University of
Connecticut, Storrs, CT

Dr. Kristin Shrader-Frechette, O'Neil Professor of Philosophy, Department of
Biological Sciences and Philosophy Department, University of Notre Dame, Notre Dame,
IN

Dr. Philip Singer, Professor,  Department of Environmental Sciences and Engineering,
School of Public Health, University of North Carolina,  Chapel Hill, NC

Dr. V. Kerry Smith, W.P. Carey Professor of Economics , Department of Economics ,
W.P Carey School of Business , Arizona State University, Tempe, AZ

Dr. Deborah Swackhamer, Interim Director and Professor, Institute on the
Environment, University of Minnesota, St. Paul, MN

Dr. Thomas L. Theis, Director, Institute for Environmental Science and Policy,
University of Illinois at Chicago,  Chicago, IL

Dr. Valerie Thomas, Anderson Interface Associate Professor, School of Industrial and
Systems Engineering, Georgia Institute of Technology, Atlanta, GA

Dr. Barton H. (Buzz) Thompson, Jr., Robert E. Paradise Professor of Natural
Resources Law at the Stanford Law School and Perry L. McCarty Director, Woods
Institute for the Environment Director, Stanford University, Stanford, CA

Dr. Robert Twiss, Professor Emeritus, University of California-Berkeley, Ross, C A

Dr. Lauren Zeise, Chief, Reproductive and Cancer Hazard Assessment Branch, Office
of Environmental Health Hazard Assessment, California Environmental Protection
Agency, Oakland, CA
SCIENCE ADVISORY BOARD STAFF

Mr. Thomas Miller, Designated Federal Officer, U.S. Environmental Protection
Agency, Washington, DC

-------
                                  Enclosure 3
                            TABLE OF CONTENTS

1.   EXECUTIVE SUMMARY	xii

2.  INTRODUCTION	1

3.  CHARGE TO THE COMMITTEE	2

4.  RESPONSE TO CHARGE QUESTIONS	3

4.1     Question 1: New strategic direction	3

4.2     Question2: Adequacy of goals	7

4.3     Questions: Logic model and implementation	23

4.4     Question4: Challenges to achievement	24

4.5     Questions: Measuring progress	27

4.6     Question 6: Leveraging resources inside and outside of EPA	29

5.  CONCLUSION	32

6.  REFERENCES	34

APPENDIX A. SPECIFIC COMMENTS ON THE ECOLOGICAL
RESEARCH PROGRAM MULTI-YEAR PLAN	A-l
                                      XI

-------
1.     EXECUTIVE SUMMARY

   EPA's Office of Research and Development requested that the Science Advisory
Board (SAB) provide advice on the Agency's draft Ecological Research Program Multi-
Year Plan FY 2008 - 2014 (Plan). The draft Plan was reviewed by the SAB Ecological
Processes and Effects Committee (Committee). To augment the expertise on the
Committee for this advisory activity, several SAB committee members with expertise in
valuation of ecosystem services also participated in the review.  The draft Plan presents
proposed goals, objectives, and research questions for EPA's Ecological Research
Program (Program) and also lays out an implementation strategy for the Program.  The
Plan articulates a new strategic direction that focuses on quantifying ecosystem services
and their contribution to human health and well-being. EPA has stated that the overall
goal of the Program is to "change the way decision makers understand and respond to
environmental issues by making clear the ways in which policy and management choices
affect the type, quality, and magnitude of goods and services that are received from
ecosystems."

   EPA sought the SAB's advice on: 1) the appropriateness and utility of the new
strategic direction in offering meaningful contributions to ecological sciences and
providing research that will be useful to decision makers; 2) the adequacy of the goals,
objectives, and research questions in contributing significantly to meeting the overall
purpose of the Program; 3) the logic model  and implementation approach in the Plan;  4)
anticipated  challenges to achieving the overall goal of the Ecological Research Program;
5) suggestions for measuring annually over the next five years the progress, productivity,
efficiency, and effectiveness of the Ecological Research Program; and 6)
recommendations to enhance EPA's ability to leverage available resources within and
outside the Agency. In response to the charge questions, the Committee has provided
comments and recommendations to improve the Plan. Our recommendations are listed as
bullets throughout this advisory report.

Strategic direction and focus of the Program

   The Committee strongly supports the new strategic direction of the Ecological
Research Program. We commend the Agency for developing a research program that, if
properly funded and executed, has the potential to be transformative for environmental
decision making as well as for ecological science. In this regard, a number of important
research themes are proposed in the Plan. These include: developing tools to identify  and
manage trade-offs among ecosystem services over time; disseminating information on
ecosystem services in ways that make it useable by the public; conducting research on the
relationship between participatory decision-making processes and social, environmental,
and economic outcomes; designing a system for monitoring ecosystem services;
conducting research on management of ecosystem services across spatial scales;
developing a better scientific understanding of ecosystem threshold responses and regime
shifts; and delivery of tools to understand societal benefits of ecosystem services. The
research program's focus on ecosystem services advances the desirable integration of
ecological processes and human welfare and serves well the purposes of a public
                                       xn

-------
environmental management agency.  This research focus can, if properly funded and
executed, provide a sound foundation for environmental decisions and regulation based
on the dependence of humans on ecological conditions and processes.

   Although the Committee supports the overall strategic direction of the Program, we
have a number of concerns about EPA's draft Plan. Most of these are related to the
tension between stating an important and ambitious vision and producing a practical
implementation plan for a future that includes a limited and uncertain budget. The
Committee notes that as EPA continues to develop the Program, it will be important to
consider opportunity costs that may be associated with the shift in strategic direction.
The new utilitarian focus  on ecosystem services may pose the risk of losing potential
research contributions to improved understanding of ecosystem functions and responses
that are unrelated to recognized services to human health and well-being. Similarly, there
may be a risk that over-emphasis on ecosystem services that are too narrowly defined will
prove to be a disservice to decision makers in the long run. An analysis of these
opportunity costs would require detailed information on specific components of the ORD
research program that are being eliminated or changed. Such information was not made
available to the Committee for this advisory activity.  The following recommendations
are provided to improve the discussion of the strategic vision and how it will be
accomplished:

•  The Committee finds  that the long-term goals of the program are unlikely to be
   accomplished in the proposed time frame with current resources. We find the lack of
   grant support to be particularly worrisome given the limited EPA expertise available
   in such areas as valuation and benefit assessment and education and outreach, and the
   fact that ecosystem services is a relatively young and rapidly developing field of
   science; we therefore  strongly encourage EPA to provide additional funds for
   research on ecosystem services (e.g., through the Agency's Science to Achieve
   Results [STAR] Program).

•  To strengthen the justification of research priorities and clarify how work will be
   accomplished, we  recommend that the discussion of priorities in the Plan include the
   rationale leading to: a) accomplishing initial goals; b) selecting geographic locations
   for research; and c) identifying the scales of efforts.

•  The overarching goals of the Program cannot be accomplished without basic
   ecological research. In particular, empirical data are needed to test hypotheses
   regarding why changes in ecosystem services are occurring, and at which scales. We
   therefore recommend  that more information be provided in the plan to identify
   knowledge gaps along with the basic research needed to fill these gaps, and that
   completion of this basic research be encouraged (e.g., through grants to researchers).

•  The intended audience of the Plan and the range of decision types supported by the
   Ecological Research Program should be explicitly described "up front" in the Plan.
   The Committee suggests that decision makers are an important audience, and that
                                       Xlll

-------
   members of the general public might be considered to be the ultimate decision
   makers.

•  The Plan should provide greater detail on how EPA will accomplish intra- and inter-
   program coordination and inter-institutional collaboration on the proposed research.

•  The Plan would do well to recognize that the environment, institutions, and human
   welfare are changing at an unprecedented rate, and as new situations, new priorities,
   and new ideas develop, EPA should remain nimble enough to identify new
   "services," ask new questions, and apply new measurement techniques.

•  The ways in which the concept of ecosystem services could provide guidance to the
   Agency's regulatory and non-regulatory programs need to be more fully explored and
   more clearly articulated in the Plan.

•  The relationship between ecosystem service valuation and the application of
   ecological risk assessment should be described in the Plan. There is a strong
   connection between the current vision outlined in the Plan and EPA's long history of
   engagement in ecological risk assessment.

•  The Plan represents a  considerable change in the research direction for EPA's
   Ecological Research Program. Previous research has made significant contributions
   to the science of ecological monitoring and assessment.  As monitoring and
   assessment is moved to other parts of the Agency, it is essential that EPA's strength
   and leadership in this  area be maintained.

Research goals and questions

   In the Plan, EPA has identified five long-term goals to guide its research agenda.  The
Committee has provided comments and recommendations on the goals, related research
questions and objectives.

   Long-term Goal 1 envisages development of a decision support platform that offers
EPA, states, local  communities, and resource managers the ability to integrate, visualize,
and maximize the use of diverse data, models, and tools at multiple scales for decision
making.  As further discussed in Section 4.2  of this advisory report, Long-term Goal 1
has four research elements: 1) Human Health and Well-being, 2) Ecosystem services
valuation; 3) Outreach and Education; and 4) Decision Support Platform (DSP). The
Committee supports Long-term Goal 1  and offers the following recommendations for
improvement.

•  Long-term Goal  1 should be restructured to integrate the elements of human health
   and well-being and ecosystem services valuation into one effort that must rely heavily
   on individuals and agencies outside of the core ecological research proposed.  The
   Ecological Research Program should focus on developing the ecological production
   functions of the ecosystems services framework.  Similarly, outreach and education
                                       xiv

-------
   should be integrated with the decision support platform into one effort addressing
   how decision makers would be targeted for outreach and education. A more
   comprehensive outreach and education plan should be developed to address human
   capital and resource needs.  In addition, EPA should explicitly identify potential
   clients who will use the decision support platform. It is important to identify
   stakeholders in the Program and undertake targeted  outreach and education efforts to
   ensure that they can benefit from the process and the approaches used.

•  The discussion of Long-term Goal 1 does not clearly describe how EPA will find the
   expertise to accomplish valuation of ecosystem services, development of the decision
   support platform, and outreach and education, including coordination  and
   collaboration with other units in EPA and/or through outside cooperators.  In the
   Plan, the discussion of the key role of ecosystem services value information should be
   clarified to indicate what original valuation research will, and will not, be conducted
   within the ecological research plan.

•  The Committee recommends that EPA focus on research that will be conducted to
   predict changes in ecosystem services rather than evaluating alternative valuation
   methods.  This approach will take advantage of the available expertise within EPA's
   Office of Research and Development (ORD).

•  The Committee recommends that EPA more thoroughly describe how the decision
   platform would work. This description should indicate whether the decision support
   platform is intended to support actual decisions or to teach decision makers about the
   importance of ecosystem services using illustrative case studies.  EPA should also
   describe how mapping, monitoring, and modeling research accomplished in other
   components of the research plan would be coordinated with work to develop the
   decision support platform.

•  As further discussed in Section 4.2 of this advisory report, the Committee is
   concerned about the overall feasibility of accomplishing Long-term Goal 1. We
   therefore recommend that development of the decision support platform be identified
   as a long-term objective, not a short run test of the Ecological Research Program's
   effectiveness.

   Long-term Goal 2 envisages developing a publicly accessible, scalable national atlas,
an inventory system, and models for selected ecosystem services. The Committee finds
that the work to be conducted under this goal may be one of the strongest parts of the
Ecological Research Program given that EPA has extensive experience in mapping and
monitoring.  We note that more detailed information is needed to understand how the
maps and models developed under Long-term Goal 2 would be incorporated into the
decision support tool. We offer the following key recommendations concerning Long-
term Goal 2:

•  EPA's Ecological Research Program should plan to use information in available
   databases to develop ecological production functions and models that can be used to
                                       xv

-------
   forecast the effects of various stressors on ecosystem service flows. In this regard,
   the Committee notes that the scale of data provided must be appropriate to support
   decision making. Thus the suitability of various databases for use in developing
   Program products supporting decision making should be assessed as soon as possible
   and definitely before 2013.

•  The atlas of selected ecosystem services should be linked to models that can predict
   changes in ecosystem services. Monitoring data should lead directly into the atlas
   and support the forecasting models.

•  The Committee recommends that EPA coordinate with other federal agencies and
   academic scientists to conduct a review of all federal agency ecosystem and
   ecosystem  services inventory, mapping, and monitoring type projects.  This review
   should be undertaken in order to determine how such projects can provide data to
   meet the objectives of the Ecological Research Program.  The review could be
   conducted through a workshop, with the aim of coordinating all of the federal agency
   components to provide synergy and  avoid duplication of effort. Subsequent to the
   workshop,  EPA should collaborate with other federal agencies and academic
   scientists to conduct a scientific community assessment of status and trends of
   ecosystem  services in the U.S. (similar to the Intergovernmental Panel on Climate
   Change [IPCC] assessments). Such  an assessment would be an appropriate and very
   important output from the research that is  described in the Plan.  It would be a high
   impact, visible product from EPA that could have a large influence on decision
   makers.

•  Ecosystem services should not be defined so narrowly that they overemphasize
   human health and welfare goals and  fail to appropriately value nonhuman dominated
   landscapes.

   Long-term Goal 3 calls for an assessment of the positive and negative impacts on
ecosystem services resulting from changes in  nitrogen levels at select locations and
within select ecosystems. The Committee finds that this is an important area of
ecological research. However, given the relatively modest effort that can be undertaken
with available  resources, we have some  concern about what can be accomplished in this
important area, and how EPA's contribution will complement what is being done in other
agencies. The following recommendations are provided:

•  The fundamental question to be addressed by the nitrogen assessment is not clearly
   articulated. A more detailed description and justification of the research to be
   conducted  should be developed.

•  Opportunities for coordination and collaboration with research conducted in the
   place-based and wetlands components of the ecological research plan should be
   vigorously pursued, including systematic replications of nitrogen studies across the
   different places and systems.
                                       xvi

-------
•  The Committee recommends that EPA partner with other federal agencies conducting
   research on reactive nitrogen as related to human health issues so that research is
   complementary and not duplicative.

   Long-term Goal 4 of the Plan focuses on investigation of the dynamics of ecosystem
service flows in two priority ecosystems, wetlands and coral reefs.  The Committee finds
that the long-term goal of assessing ecosystem services in wetland ecosystems is
appropriate, but notes that it will be a challenge to address the complex spatial and
temporal issues of ecosystem processes and their linkages to ecosystem services (and
ultimately to valuation of those services).  We note that a key missing piece in the Plan is
research to develop an understanding of the linkage between multiple stressors and
ecosystem attributes and services.  To address these challenges, it will be important to
coordinate research activities across many research entities (e.g., EPA, universities, and
other federal agencies). Chances of success could be improved by initially undertaking
pilot projects where tangible products can be developed within a three-year period.

•  The Committee recommends that detailed implementation plans be developed by
   EPA to accomplish Long-term Goal 4 and that these plans receive outside peer
   review.  It is particularly important to undertake projects related to multi-stressor
   diagnosis and subsequent ranking and linkage to ecosystem attributes and services.

•  Initial projects to accomplish Long-term Goal 4 should focus on a small set of
   representative wetland systems and perhaps also include a national assessment.

•  Although coral reef ecosystems are globally important, the Committee finds that they
   are a relatively low priority in the U.S. We recommend that EPA consider
   undertaking projects in other more common "human dominated" ecosystems that
   provide services to more U.S. citizens, and greater opportunities for coordination and
   collaboration with other studies within the ecological research program. If the
   Program decides to retain the coral reef component, we recommend that in the Plan,
   EPA provide a better explanation of how studying the dynamics of ecosystem service
   flows in coral reefs will advance ecological sciences and ultimately help inform
   decision making.

   Long-term Goal 5 calls for place-based research to investigate ecosystem services.
The Committee finds that there is a lack of adequate and transparent explanation in the
Plan regarding the selection of areas where this research will be conducted, and that
additional issues should be considered in developing this part of the Program. We
therefore recommend that:

   •   The Plan should contain a transparent explanation of the process used to select
       sites for place-based demonstration projects. In Section 4.1 of this advisory report
       we have suggested principles that could guide selection of these sites.

   •   Transboundary issues should be explicitly considered in the place-based projects.
                                       xvn

-------
    •   The application of life cycle analysis in demonstration projects should be
       expanded to show the utility of this approach in future decision making.

Implementation Strategy

   The Plan contains a logic model that describes how the Ecological Research Program
will be designed, planned, implemented and managed.  The Committee has provided a
number of comments and recommendations concerning:  1) the logic model; 2)
anticipated challenges to achieving the overall program goal; 3) measuring program
progress, productivity, efficiency, and effectiveness; and 3) enhancing EPA's ability to
leverage available resources.

Logic model

  The Committee finds that the construct of the logic model in the Plan is a sensible way
to represent  program activities, products, and outputs.  A  similar approach has been
suggested in a recent National Research Council (NRC, 2008) report.

•   As discussed in Section 4.3 of this advisory report, the Committee recommends that
    EPA consider adapting some  of the terminology and structure of the NRC logic
    model and more clearly identify the role of partnerships in accomplishing research
    goals.

Challenges to achieving goals

   The Committee has identified  the following four broad categories of challenges facing
the Ecological Research Program: 1) the ambitious nature of the overarching research
questions and annual performance goals; 2) scientific and technical issues to be overcome
in developing specific methodological or tactical approaches; 3) difficulties that may be
encountered in extending program outputs to partners to support decision making
processes; and 4) availability of resources (including institutional capabilities).
Developing  strategies to deal with these inherent challenges will provide opportunities to
advance the  way that ecological research is conducted.

   The Committee finds that the most serious challenge facing the Ecological Research
Program is the limited availability of resources. The long-term goals of the program are
unlikely to be accomplished in the proposed time frame with current resources. The
ORD staff skill set may not be sufficient to address the issues and conduct all of the work
needed to  achieve long-term program goals. Valuation and benefit assessment is one
particular  area where additional expertise is needed. If ecosystem services are to be
properly evaluated, EPA will need expertise to ensure that well-being is parameterized in
an accurate multidimensional manner.  The parameterization of well-being should
include  consideration of a range of cultural value systems. Furthermore, assessing
ecosystem services is a new and rapidly developing area of research that will benefit from
the diversity of insights and approaches provided by independent investigators.  Given
these conditions, we find the lack of grant support to be particularly problematic, and
                                       xvin

-------
therefore strongly encourage EPA to provide additional funds for ecological research
through programs such as the Agency's STAR program.

    An additional challenge facing the Program is enlisting the support of key
stakeholders and clients. In this regard, immediate efforts are needed to enlist the input
and cooperation of potential users and clients of the Program to ensure that planned
research will address issues of greatest interest to them.

Suggestions for measuring progress, productivity, efficiency, and effectiveness

   The Committee notes that the recent NRC (2008) report cited above provides relevant
recommendations for evaluation of research and development programs at EPA. In
Section 4.5 of this advisory report we have offered some additional recommendations.
We generally find that, given the visionary intentions of the Plan and the current lack of
detailed research implementation plans, it is premature to prescribe specific measures to
evaluate annual performance and progress goals. However, we recommend that:

•  At this formative stage an assessment of the Plan as it develops should include
   monitoring, evaluation, and adjustment of objectives as partnerships and
   collaborations within and outside EPA evolve.  Such an adaptive management
   approach requires flexibility and vigilance to capitalize on opportunities that arise.

•  The stated goals, research objectives and performance measures of the Plan should be
   focused on the identification and articulation of the ecological processes and
   structures that contribute toward ecosystems services that have been identified in
   collaboration with ecological, medical, and social scientists in the Agency. Program
   performance should not be judged based on measures of the incorporation of concepts
   of ecosystem services into management and regulatory decisions; this is a long-term
   goal.

Recommendations for enhancing EPA 's ability to leverage available resources within
and outside the Agency

   The Committee finds that the success of the Ecological Research Program is likely to
depend in large measure upon its ability to leverage available resources within and
outside of EPA. In Section 4.6 of this advisory report we have offered a number of
specific recommendations in this regard, summarized below.

•  The Memoranda of Understanding to be developed with federal partners should be
   more than agreements to cooperate.  The memoranda  should state who will do
   specific work when there is overlap, and how resources will be shared.

•  ORD should use its available people, infrastructure, and data to leverage in-kind
   services and collaborate with other groups/agencies.  In this regard, there are ample
   partnership opportunities. ORD can partner with other agencies within the U.S. (e.g.,
   U.S. Fish and Wildlife Service, U.S. Forest Service, and National Park Service).
                                       xix

-------
ORD should consider working with professional societies to sponsor sessions or
symposia for: 1) presenting results of work to accomplish the goals in the Plan, and 2)
soliciting feedback from stakeholders and end users. In addition, ORD should
consider partnerships with private business, non governmental organizations (NGOs),
and organizations such as non-profit foundations to conduct research and
development activities.

ORD should make the STAR program a priority in efforts to leverage resources. The
following will help achieve the Plan's goals: enhancing the STAR Graduate
Fellowships program; providing funds for exploratory extramural ecological research
to develop tools and procedures to accomplish the goals of the Plan; and developing a
competitive grants program to run  summer credit workshops for teachers through
STAR that would support the outreach goals of the Plan.

ORD should partner with professional societies, publishing companies, media outlets,
and NGOs to develop and disseminate education and outreach materials to
professionals, teachers, and the lay public. Some suggested approaches that could be
developed in partnership with other organizations include: workshops, symposia, and
sessions at meetings, WIKI blogs, presentation materials for educators and public
forums, media resources including cable television educational networks, and 10-15
minute video clips that can be used in classroom settings.

 ORD should also incorporate into the Plan research with international partners to
understand transboundary conditions and connections that extend across national
borders. Examples of such systems include the coastal waters of British Columbia,
Canada and the Puget Sound/Georgia Basin in Washington and the prairie grassland
ecosystems of the Midwestern United States and central Canada. A successful model
of such an interaction is the long-standing research and management collaboration for
the Great Lakes of North America.
                                    xx

-------
2.     INTRODUCTION

   EPA's Office of Research and Development requested that the Science Advisory
Board (SAB) provide advice on the Agency's draft Ecological Research Program Multi-
Year Plan FY 2008 - 2014 (Plan). The draft Plan was reviewed by the SAB Ecological
Processes and Effects Committee (Committee). To augment the expertise on the
Committee for this advisory activity, several SAB  committee members with expertise in
valuation of ecosystem services also participated in the review.  The draft Plan presents
proposed goals, objectives, and research questions for EPA's Ecological Research
Program and also lays out an implementation strategy for the Program.

   For the past ten years the EPA Ecological Research Program has focused on: 1)
developing monitoring tools and indicators to determine the status of and trends in
ecological resources and the effectiveness of national programs and priorities; 2)
developing diagnostic  tools and methods to determine causes of ecological degradation;
3) developing tools and methods to forecast the ecological impacts of actions taken by
states, tribes, and EPA offices; and 4) developing environmental restoration tools and
methods to improve the ability of states, tribes, and EPA offices to protect and restore
ecological condition.  EPA's draft Ecological Research Program Multi-Year Plan FY
2008 - 2014 articulates a new strategic direction for the Program that focuses on
quantifying ecosystem services and their contribution to human health and well-being.
This new approach takes the focus of the Program beyond traditional ecological
endpoints such as biological, chemical, and physical condition.  EPA has stated that the
overall goal of the new Program is to change the way decision makers understand and
respond to environmental issues by making clear the ways in which policy and
management choices affect the type, quality, and magnitude of goods and services that
are received from ecosystems.

   The Committee strongly supports the new strategic direction of the Ecological
Research Program. EPA's Ecological Research Program Multi-Year Plan contains a
discussion of the importance of quantifying ecosystem services and their contribution to
human health and well-being in order to advance ecological science and improve decision
making. In addition, the SAB Committee on Valuing the Protection of Ecological
Systems and Services has identified benefits associated with strengthening EPA's
approaches for valuing the protection of ecological systems and services (U.S. EPA
Science Advisory Board, 2008a). We commend the Agency for developing a research
program that, if properly funded and executed, has the potential to be transformative for
environmental decision making as well as ecological science. In this regard, a number of
important research themes  are proposed in the Plan.  These include: developing tools to
identify and manage trade-offs among ecosystem services over time; disseminating
information on ecosystem services in ways that make it useable by the public; conducting
research on the relationship between participatory  decision-making processes and social,
environmental, and economic outcomes; designing a system for monitoring ecosystem
services; conducting research on management of ecosystem services across spatial scales;
developing a better scientific understanding of ecosystem threshold responses and regime
shifts; and delivery of tools to understand societal benefits of ecosystem services.  The

-------
research focus on ecosystem services advances the desirable integration of ecological
processes and human welfare and serves well the purposes of a public environmental
management agency. The research program's focus on ecosystem services can provide a
sound foundation for environmental decisions and regulation based on the dependence of
humans upon ecological condition and processes.  While the Committee supports the
overall strategic direction, we have a number of concerns about EPA's draft Plan.  These
concerns are further discussed in various sections of this advisory report.  The Committee
has provided comments and recommendations to improve the Plan in response to the
charge questions. Our recommendations are listed as bullets throughout this advisory
report.

3.     CHARGE TO THE COMMITTEE

   EPA's Office of Research and Development sought advice from the Science Advisory
Board on the strategic direction and focus of the Ecological Research Program, the
research goals and objectives in the Plan, and the Agency's strategy for implementation.
The following specific charge questions were provided to the Committee.

Focus of the Program

1.  The strategic direction of the Ecological Research Program (Program) is to: a)
    characterize and quantify the type, quality, and magnitude of services that ecosystems
    provide; b) develop new methods to quantify and forecast how services respond to
    stressors; and c) combine these and existing tools for assessing the benefits of
    alternative management decisions. Please comment on the appropriateness and utility
    of this strategic direction in: 1) offering meaningful contributions to the ecological
    sciences and  2) providing research that will be useful to decision makers at EPA and
    other levels of governance.

Research Goals and Questions

2.  The Ecological Research Program includes five long-term goals, associated
    objectives, and research questions.  Please comment on the adequacy of the goals,
    objectives, and questions in contributing significantly to meeting the overall purpose
    of the program.  In reviewing each research goal please consider the following:

    •   Are the research questions appropriate?  If changes are needed in the research
       questions, please indicate how they should be changed.

    •   Are the descriptions of planned research adequate to characterize the intended
       results, and is the planned research appropriate for accomplishing the goals?

    •   Please comment on needed improvements in and clarification of the goals and
       objectives as well as additions or eliminations to be considered in future program
       development.

-------
Implementation Strategy

3.  The Ecological Research Multi-Year Plan lays out the process by which ORD intends
   to accomplish research. Please comment on the logic model approach and provide
   any recommendations that should be considered in developing implementation plans.

4.  Please comment on anticipated challenges to achieving the overall goal of the
   Ecological Research Program Multi-Year Plan based on the Program as presented.
   What recommendations does the Committee have to overcome the most significant of
   these challenges?

5.  What suggestions does the committee have for measuring annually over the next five
   years the progress, productivity, efficiency, and effectiveness of the Ecological
   Research Program?

6.  Does the Committee have any recommendations on how EPA can better enhance its
   ability to leverage available resources within and outside the Agency?

4.    RESPONSE TO CHARGE QUESTIONS

4.1    Charge Question 1. Please comment on the appropriateness and utility of the
       strategic  direction of the Plan in: 1) offering meaningful contributions to the
       ecological sciences; and 2) providing research that will be useful to decision
       makers at EPA and other levels of governance.

   The Committee unanimously supports the conceptual framework of EPA's draft
Ecological Research Program Multi-year Plan. The conceptual framework of the Plan
focuses on creation of an integrated systems-based approach to identify, inventory,
monitor, map, and model ecosystem services. In addition, the conceptual framework
focuses on quantifying ecosystem services and their contribution to human health and
well-being. The research focus on ecosystem services represents a suitable approach to
the integration of ecological processes and human welfare. The Committee finds that
EPA's focus on ecosystem services provides an appropriate foundation  for environmental
decisions and regulation based upon the dependence of humans upon ecological
condition and processes.  The conceptual framework for the program is  thus tightly
linked to the mission and agenda of EPA, and represents the leading ideas of the
international ecological community.  The vision outlined by EPA is a plan to develop the
next generation of environmental management support technologies that build on risk
assessment.  The Committee finds that the resulting knowledge and tools will more
completely support effective evaluation of management alternatives and improved
communication of benefits to the public than is presently the case.

   However, the  Committee has a number of concerns about EPA's draft Plan. Most of
these are related to the tension between stating an important and ambitious vision and
producing a practical implementation plan for a future that includes a limited and
uncertain budget.  Our suggestions for improvement are related to maintaining the large

-------
and influential vision while appropriately defining the most pressing questions, scales,
variables, and geographic locations to be investigated.

  The Committee notes that as EPA continues to develop the Program, it will be
important to consider opportunity costs that may be associated with the shift in strategic
direction. The new utilitarian focus on ecosystem services may pose the risk of losing
potential research contributions to improved understanding of ecosystem functions and
responses that are unrelated to recognized services to human health and well-being.
Similarly, there may be a risk that over-emphasis on ecosystem services that are too
narrowly defined will prove to be a disservice to decision makers in the long run.  An
analysis of these opportunity costs would require detailed information on specific
components of the ORD research program that are being eliminated or changed. Such
information was not made available to the Committee for this advisory activity.

   We have eleven major recommendations related to the overall adequacy and
appropriateness of the strategic direction outlined in the Plan. These recommendations
are aimed at improving the potential for contribution to ecological science and providing
research that will be highly useful to decision makers.

Recommendations to improve the potential contribution of the ecological research
program to ecological science and decision making

•  The vision and direction described in the Plan are sufficiently important to merit
   substantial investment by EPA.  The long-term goals of the program cannot be
   accomplished with current resources (funding and personnel) dedicated to this effort.
   It is our understanding that EPA is dedicating approximately $68 million per year of
   Office of Research and Development staff time to support the ecological research
   program but is not providing  any grant funding or other additional extramural
   support.  We recommend that Science to Achieve Results (STAR) program funds and
   other EPA resources be directed toward the ecological research program.  The
   research program is advancing an area of ecological science that is new, where
   innovative and exploratory research will be needed to accomplish the important goals
   of the Program, and it is appropriate that extramural funding be focused there.  The
   Plan is closely related to all five of the strategic  goals defined in EPA's 2006 - 2011
   Strategic Plan (U.S. EPA, 2006), and the Committee recommends that those
   connections be communicated clearly in order to support substantially increased EPA
   investment in the Ecological Research Program.

•  The vision outlined in the Plan is ambitious and important, and we recommend that
   the title of the document reflect this vision.  In addition, as a challenge, we
   recommend that long-term goals (stretch goals) be clearly identified as such and
   presented in the Plan first, followed by a sequence of short-term priorities and
   measurable outcomes (i.e. an implementation plan).  These measurable outcomes
   should be the basis for program evaluation criteria and metrics. The discussion of
   priorities in the Plan should include the rationale leading to: a) accomplishing initial
   goals for first efforts at addressing ecosystem services; b) selecting geographic

-------
   locations for research; and c) identifying the scales of the planned efforts. The
   discussion of the priorities should be clear and honest about current resources and
   leveraging past investments.

•  The Program goals cannot be accomplished without ecological research to answer
   basic science questions. It is recommended that knowledge gaps be identified in the
   Plan, and that EPA plan and appropriately fund the basic ecological research needed
   to fill these gaps.  In particular, empirical data are needed to test hypotheses regarding
   why changes in ecosystem services are occurring, and at which scales. Identification
   of knowledge gaps will allow the key basic science questions to be elaborated in the
   separate sections of the Plan, and provide both the rationale and intellectual  construct
   for contributing to ecological science.

•  Among the most complex challenges facing EPA is the rate of change: new
   environmental problems, new socioeconomic situations, and new threats to ecosystem
   services arise. A 5-year plan that is assiduously held to is very likely to miss
   opportunities for making the largest impacts, unless it has a review cycle and adaptive
   management plan. We recommend that not only the progress, but the vision and
   implementation, be reviewed frequently enough to allow nimble responsiveness and
   maximal effectiveness. For example, EPA's research activities must advance at a
   rapid pace to respond to the threats posed by invasive species.  An adaptive
   management plan is needed to show how EPA and its partners  can effectively address
   this problem.

•  In the Plan, it is important for EPA to balance the need for research to answer
   questions for a particular decision (which suggests waiting until those questions are
   clear and them formulating specific research projects) vs. research to develop a set of
   ecosystem service values for a range of decisions. In the latter case, the available
   values may not quite fit questions to be answered, and the values can be
   misinterpreted or misused.  In the former case, the analysis needed may not be
   completed rapidly enough to be of use in making the decision.  The Committee notes
   that EPA should not value ecosystem services simply for the sake of doing so.
   Indeed, many decisions related to ecosystems will not need formal valuation to
   support good decision making. In other cases, explicit valuation will be a very key
   input to a decision process.

•  It is recommended that the intended audience of the Plan and the range of decision
   types supported by the Ecological Research Program be more clearly described "up
   front" in the document.  It would be helpful to include in the Plan a matrix or table of
   decision types (i.e., the types of choices being made at various  decision-making
   levels) vs. decision makers (i.e., governmental, industrial, private organizations, etc.).
   The Committee notes that it is particularly important to elaborate issues of scale
   (local vs. regional).

•  The Committee recommends that EPA collaborate with other federal agencies and
   academic scientists to conduct a scientific community assessment of status and trends

-------
   of ecosystem services in the U.S. (similar to the Intergovernmental Panel on Climate
   Change [IPCC] assessments). Such an assessment would be an appropriate and very
   important output from the research that is described in the Plan. This assessment
   would be a high impact, visible product from EPA that could have a large influence
   on decision-makers.

•  The Committee recommends that EPA include in the Plan an organizational plan for
   inter-institutional collaboration.  The importance of inter-institutional collaboration is
   an issue that arose repeatedly in the Committee's discussion of the Plan.  The
   Committee notes that the assessment of status and trends in ecosystem services could
   provide an opportunity for such collaboration. While we understand the challenges
   associated with developing a large collaborative research program, we find that if
   EPA were to lead an effort to undertake the assessment suggested above, the payoff
   would be large for science and management.  The effort would be a visible
   contribution to a national initiative. One venue for an assessment of status and trends
   in ecosystem services would be  collaboration with the National Center for Ecological
   Analysis and Synthesis (NCEAS, 2008), which could provide data analysis support,
   as well as support services for a series of workshops.

•  The research program described in the Plan represents the most current ideas about
   interactions between humans and the environment.  It has the potential to provide
   guidance and to stimulate innovation in the Agency's environmental management
   actions and policies.  To realize that potential, effort is needed to strengthen and
   articulate the connections between the concepts in the research plan and the
   regulatory and non-regulatory programs in the Agency.

•  The Committee notes that there  is a strong  connection between the current vision
   outlined in the Plan and EPA's long history of engagement in risk assessment. We
   recommend that this  connection be explicitly discussed in the plan. The relationship
   between ecosystem services valuation and the application of ecological risk
   assessment should be described  in the Plan. The Committee finds that ecosystem
   services assessment is an activity that will provide decision makers with information
   to translate ecological risk assessments into management strategies for achieving
   sustainable future environmental protection.

•  The Plan represents a considerable change  in the research direction for EPA's
   Ecological Research Program.  Previous research has made significant contributions
   to the science  of ecological monitoring and assessment.  As monitoring and
   assessment is moved to other parts of the Agency, it is essential that EPA's strength
   and leadership in this area be maintained.

-------
4.2    Charge Question 2.  Please comment on the adequacy of the goals, objectives,
       and questions in contributing significantly to meeting the overall purpose of
       the program.

   In the Plan, EPA has identified five long-term goals to guide its research agenda.
These five goals are: 1) by 2014, provide on-line decision support that offers EPA, states,
local communities, and resource managers the ability to integrate, visualize, and
maximize the use of diverse data, models, and tools at multiple scales to generate and
understand the consequences of alternative decision options on the sustainability of
ecosystem services and human well-being; 2) by 2013, deliver publicly accessible,
scalable national atlas, inventory system, and models for selected ecosystem services that
can be quantified directly or indirectly; 3) by 2013, provide an assessment of the positive
and negative impacts on ecosystem services  resulting from changes in nitrogen levels at
select locations and within select ecosystems; 4) by 2015, provide guidance and decision
support tools to target, prioritize, and evaluate policy and management actions that
protect, enhance, and restore ecosystem goods and services at multiple scales for two
specific ecosystem types,  wetlands and coral reefs; and 5) by 2013, complete four site-
specific demonstration projects that illustrate how regional and local managers can
proactively use alternative future scenarios to conserve and enhance ecosystem goods and
services in order to benefit human well-being and secure the integrity and productivity of
ecological systems.

   In the discussion of each long-term goal in the Plan, EPA has outlined the science
questions and objectives to be addressed. The Committee provides the following
comments on the long-term goals and related research questions and objectives. The
Committee has not explicitly commented on every goal, objective, and question in the
plan.  We have focused our comments on areas where we found that improvement was
needed and/or recommendations should be provided.

Long-term Goal 1 - Effective Decision Support

   The Committee commends EPA's Office of Research and Development (ORD) on
expanding its vision for an ecological research agenda to include  a component targeted
directly at ensuring that its products are useful for decision making. This goal is not only
appropriate but also essential if the Plan is to be part of a catalyst that helps to address the
concern that ecosystems are being degraded  because they are perceived as "free and
limitless," and their full value is not reflected in individual and policy decisions. In
addition,  the Committee agrees with ORD that it is important to recognize and
incorporate into the vision for this long-term goal the overall objectives of outreach and
education, valuation of ecosystem services, and estimation of ecological production
functions. All of these are important objectives that, if met, will enhance the Agency's
ability to accomplish its mission and contribute to improved decision making.

   Although the Committee supports Long-term Goal 1 and the overall research
objectives included under this goal, we have several concerns about EPA's proposed plan
to accomplish the goal. These concerns focus on: 1) how the plan is structured; 2)

-------
specific means to accomplish the goal; and 3) overall feasibility of accomplishing the
goal.

Structuring the Plan to accomplish Long-term Goal 1

   As reiterated throughout the Plan, some of the information needed to evaluate
tradeoffs regarding ecosystem services in the context of decision making concerns the
value or benefits of changes in service flows.  These values reflect the impact of service
flow changes on human health and well-being. In order to influence decisions,
information about these values in turn must be communicated to the public (through
outreach and education) and to decision makers (through the decision support platform).
EPA describes the following four research program elements to accomplish Long-term
Goal 1:  1) Human Health and Well-being (HHWB) (i.e., research to help decision makers
understand links between ecosystem services and human health and well-being); 2)
Ecosystem services valuation (ESV) (i.e., research to give decision makers constructs to
describe ecosystem values in a way that supports assessment of tradeoffs); 3) Outreach
and Education (OE) (i.e., outreach to decision makers to ensure that research will meet
their needs and be applied with confidence); and 4) Decision Support Platform (DSP)
(i.e., research to develop and make available tools for decision makers operating in
different circumstances, communities, spatial  scales, and levels of complexity  and
uncertainty).  The Committee finds that acknowledging the important roles of all of these
elements is appropriate to an ecological research program within the ecosystem services
framework, but they do not seem to be logically structured within Long-term Goal 1 and
many aspects of these program elements may be outside the purview of ecological
research per se. The following recommendations are provided to restructure this part of
the Plan:

•  The Committee recommends combining and integrating the HHWB and ESV
   elements of the Plan, clearly identifying which aspects of HHWB and ESV are to be
   accomplished within the Ecological Research Program, and which are to be
   accomplished through cooperation and collaboration with other units within and
   outside of EPA. The logic of separating HHWB and ESV elements is not clear.  The
   whole purpose of ecosystem service valuation is to determine the value of the impacts
   of changes in the flow of ecosystem services on human well-being (including changes
   in well-being stemming from changes in health outcomes).  Thus, these two elements
   should logically be combined and integrated.  On page 22 of the Plan it is suggested
   that they will be "closely coordinated," but an explicit plan for using the output of the
   HHWB health outcomes as an input into the ESV is needed.  In addition, explicitly
   linking the HHWB and ESV research will provide a conceptual basis for thinking
   about the linkage of ecological  systems and indicators of human well-being in the
   context of the ecosystems services framework, which is likely to be a difficult task.
   The separate treatment of human health under the current structure may also give it
   more prominence in the study of ecosystem services than is warranted, since it is not
   clear that this is a major component of the impact of ecosystem services on human
   well-being.  The relationship between ecosystem services and human health and well-
   being should be considered at multiple levels: individuals (especially susceptible

-------
   individuals), local communities, and the entire population.  Specific case studies or
   examples should be developed to illustrate potential or demonstrated human health
   impacts at each of these levels. The Ecological Research Program should explicitly
   rely upon cooperation with the various medical, economic and other social sciences
   (mostly residing in other EPA units and outside agencies) to help identify, define, and
   quantify the values to ecosystem services to human health.  The Ecological Research
   Program should focus on developing the ecological production functions of the
   ecosystems services framework.

•  The Committee recommends combining the DSP and OE elements.  If the purpose of
   the OE element is to reach out to decision makers to ensure that the  DSP meets their
   needs (as stated on page 21 of the Plan), then it would seem logical to  combine these
   two elements into a single coordinated and integrated element which would draw
   from the ESV work.  In fact, much of what is described as the means by which the
   OE objectives will be met (on page 34  of the Plan) appears to link closely to the DSP.
   The Committee also notes that many aspects of the DSP and OE sections of the Plan
   will require cooperation with scientists in other agencies and parts of EPA, rather than
   being totally (or even largely) developed by ORD Ecological Research Program staff.
   The need for such cooperation is discussed in other sections of this advisory report.

Means to accomplish key research under Long-term Goal 1

  The Committee is concerned that the Plan does not clearly describe how EPA will
provide the expertise to accomplish research in three key  areas: 1) valuation of ecosystem
services; 2) development of the decision support platform; and 3) outreach and education.

       Valuation of ecosystem services

   One of the overarching research questions articulated on pages 8- 9 of the Plan
concerns the impact of "changes in ecosystem services on human well-being and on the
services' monetary and non-monetary value." It is important to note that although
monetization can serve some purposes (e.g., regulatory proceedings), there are some
situations where monetization is not possible. Unless EPA accepts the use of non-
economic valuation approaches, resources  that cannot be  monetized will implicitly be
devalued.  However, the Committee notes that developing these ecosystem service values
is a major research undertaking by itself (EPA Science Advisory Board, 2008a) and,
despite the repeated reference in the Plan to ecosystem service values, it is not clearly
indicated how these values will be determined and used, for example, in the DSP. The
Plan mentions "partnering" with other EPA offices, organizations, or individuals to
determine values.  The Committee supports such partnering, but it is not clear what role
these partners would play. The Plan seems to recognize this as a potential problem (see
page 17 of the Plan), but does not articulate a strategy for addressing the problem. There
is reference on page 22 of the Plan to drawing on the expertise within EPA's National
Center for Environmental Economics (NCEE),  but it is not clear what is intended here.
The Committee questions whether NCEE will be doing original valuation  research that is
specifically related to the Ecological Research Program.  Information the Committee has

-------
received suggests that the NCEE commitment to this effort is limited.  The Committee
notes that, in general, NCEE has a strong focus on supporting regulatory impact analyses
and therefore cannot devote resources to the goals of the Plan commensurate with what is
required unless additional resources are provided. In addition, the recent SAB review of
the ORD budget suggests there is little, if any, funding available for valuation research
through external (STAR) grants (U.S. EPA Science Advisory Board, 2008b). The
Committee further notes that, even though valuation or benefits assessment is listed as
one of the Plan's overarching research goals, on page 16 (Figure 5) the Plan indicates that
valuation work will  receive a very small share (only 2%) of Ecological Research Program
resources (U.S. EPA Office of Research and Development, 2008).  Thus, it appears that
the Program will not generate much (if any) original valuation research, either through
ORD directly or through its partners in the Plan. If this is true, a statement to clearly
indicate such should be included at the very beginning of the Plan where the issue of
valuation is first introduced.  Throughout the Plan, there is discussion about the key role
of value information, but it is not clear what valuation research will be undertaken.
Therefore the Committee recommends that:

•  In the Plan, the discussion of the key role of ecosystem services value information
   should be clarified to indicate what original valuation research will, and will not, be
   conducted.

   The Committee finds that without additional resources ORD does not have the
expertise to conduct valuation itself or the capacity to fund this type of research by
others. However, ORD can benefit from and provide valuable input into valuation efforts
conducted (and funded) by others. All ecosystem services valuation exercises, regardless
of the specific valuation method used, require as input predicted changes in the flow of
ecosystem services.  EPA's Ecological Research Program can play a critical role in
estimating the ecological production functions that can be used to generate predicted
changes in service flows stemming from alternative decisions or management options
(and the associated changes in stressors). The Committee notes, however, that even this
will require interaction of a team comprised of ORD  scientists from biological, physical,
and social science disciplines.

   The identification of ecosystem services requires information not just about the
functions, processes, and bio-physical state of ecosystems but also about the (potential or
actual) human uses or the contributions to well-being associated with those systems. It
will be important to consider a range of cultural value systems to ensure that well-being
is parameterized in an accurate multidimensional  manner.  This suggests that the
identification, measurement and mapping of ecosystem services cannot be based solely
on bio-physical information but must also incorporate information relating to social,
economic, cultural or other population characteristics that affect the extent to which
ecosystems contribute to human well-being.  For example, maps and models of the
relevant characteristics (and projected future characteristics) of the humans/societies near
(and downstream from) a wetland are required to  translate  the particular water captured,
filtered and stored into a "service" that is of value to people. These same human/social
characteristics are frequently cited in the Plan as potential sources of stressors on
                                        10

-------
wetlands, reinforcing the need for measures and models (and maps) of relevant
human/social characteristics.

   Incorporating this information to identify and measure changes in services does not,
however, mean that the Plan must include an assessment of alternative valuation methods
(as currently articulated in the Plan). While such an assessment is important, given
ORD's expertise, the Committee recommends that:

•  In the Plan, EPA should focus on research that will be conducted to predict changes
   in the ecosystems that provide selected ecosystem services rather than on evaluating
   alternative valuation methods for those services. This research focus will take
   advantage of the expertise available within ORD.

The Committee notes that valuation is a complicated area requiring extensive
consideration of a number of issues (EPA Science  Advisory Board, 2008a), and there is
the potential for misinterpretation if not done very  carefully. For example, the plan
suggests that the  Science Advisory Board Committee on Valuing the Protection of
Ecological Systems and Services (CVPESS) has recommended the use of "donor-based"
methods of valuation based on stocks and flows of energy.  The Committee notes that this
assertion is incorrect. CVPESS did not recommend the use of "donor-based" methods.
This subject was debated by the CVPESS, but it is a controversial approach that is
rejected by many, if not most, economists, as well  as others on the Committee. This is an
important consideration because "buy-in" from economists, social scientists, and others
involved in the valuation and policy making process is essential to the success  of the
Plan.  The Committee notes that this is just one example of the issues that can arise in
valuation, but it illustrates why the Committee is concerned about this aspect of the Plan.

       Decision  Support Platform (DSP)

   The Committee finds that in the Plan, several aspects of the discussion concerning the
DSP are unclear. First, the Plan does not clearly identify the user community for the
DSP.  There are numerous references in the Plan to decision makers who are the intended
audience for the DSP. However, it is likely that in many cases the users of the DSP may
be analysts rather than decision makers.  These analysts, in turn, provide information to
the decision  makers.  It is important that the types  of decision makers comprising the
audience of the DSP be clearly identified. The Committee questions, for example,
whether the DSP audience includes decision makers in industry.  The Committee finds
that EPA will miss a major opportunity if the Plan  does not address how industry would
use this information and tool set to factor ecosystem services into their day-to-day project
designs and funding decisions. The Committee notes that clients (stakeholders) who will
use the DSP must be identified early in the process, and their involvement in the decision
process must be continuous.  In addition, it is important to note that the DSP could  divert
limited resources from ecological research to expensive computer exercises, and be of
limited value, unless members of an explicitly identified user community are involved in
all stages of its development so that the DSP has specific uses. As further discussed
                                       11

-------
below, the DSP should also be subjected to rigorous empirical evaluation of its usability.
The Panel therefore recommends that:

•  In the Plan, EPA should explicitly identify potential clients who will use the DSP.
   Members of the explicitly identified user community must be involved in all stages of
   its development, so that the DSP has some specific uses and not just an ill-defined set
   of conceivable uses.  This will allow outreach efforts to be targeted more specifically.
   The Panel notes that any computer-based environmental decision tool needs to be
   marketed to show its utility.  Achieving widespread use among a variety of clients
   will require a variety of approaches.

   A second concern about the discussion of the DSP in the Plan is that it does not clearly
describe how the DSP would work.  The Committee questions, for example, whether the
DSP is intended to provide support for actual decisions (in which case it must include
specific information relevant to the particular decision context), or simply to teach
decision makers about the importance of ecosystem services using illustrative case
studies. The Committee notes that it may be a relatively easy task to collect information
about ecosystem services in one place on an internet website for easy access by decision
makers.  Similarly, teaching tools can be easily developed and made available to decision
makers. However, it is much more difficult to develop a meaningful interactive decision
support tool for direct use in evaluating specific policy options.  The nature and scope of
the decisions relating to the provision of ecosystem services are likely to be varied in
scale (e.g., local, regional, national) and geography (e.g., consideration of sites at
different  locations). Therefore, development of a single decision support tool that could
simply be adapted (e.g., through re-parameterization) to specific contexts seems nearly
impossible. If EPA envisions a suite of tools in the DSP, it is not clear how they would
be designed (e.g., by ecosystem type or scale).  Again, it might be possible to put various
ecological models (with estimated ecological production functions) into the DSP, but in
order to evaluate tradeoffs, information about values  is needed.  The Committee suggests
that it could be useful for EPA to examine in depth one or more DSPs that have been
developed and implemented by EPA or other agencies in order to learn what approaches
have been effective. The Committee questions whether the DSP will contain specific
valuation information that can be combined with estimated  ecological production
functions for use in evaluating tradeoffs. The Committee notes that it can be quite
challenging to combine specific valuation information with separately estimated
ecological production functions since this will inevitably involve the difficult task of
transferring ecological values data and functions (including economic benefits) between
different  ecological and social contexts. The validity of such transfers hinges on a
number of complex issues relating to the structural and functional similarities between
the original ecological/social system (the study context) and the target ecological/social
system (the policy context). If not done carefully, such transfers can be problematic, and
are likely to be invalid. The Committee therefore recommends that:

•  In the Plan, EPA should more clearly describe how the DSP would actually work.
   This description should indicate whether the DSP is intended to provide support for
   actual decisions or to teach decision makers about the importance of ecosystem
                                        12

-------
    services using illustrative case studies. The Plan should describe the suite of tools
    envisioned in the DSP and how these tools would be designed. Furthermore, the DSP
    should be subjected to rigorous empirical evaluation of its usability with individuals
    drawn from that identified user population performing tasks like those for which the
    DSP is intended. These evaluations must meet the highest standards of human-
    computer interaction research and, as mentioned, begin with the earliest stages of
    system development so that usability is essential to the design, not an afterthought
    tacked on at the end.

  In the Plan, the DSP is often described as an instrument bringing together and making
available whatever models and measures are developed under any of the other four long-
term goals.  The Committee finds that the DSP could more effectively promote
coordination if it were used to encourage convergence among the separately developed
models and measures.  In this sense, a less flexible platform that required all
projects/investigators to negotiate in the direction of common mutually acceptable
models and measures might be more advantageous. There is also some  indication that
research to be completed under Long-term Goals 1 and 2 (Effective Decision Support and
National Inventory, Mapping and Monitoring) could conflict and compete over models
and measures. As discussed in the Plan, ORD's intention seems to be that the work
under these two goals would be complementary, with the maps and models developed
under Long-term Goal 2 being designed to be easily incorporated as both tools and
contents in the DSP. However, it is not clear in the Plan how the required collaboration
between research projects conducted under Long-term Goals 1 and 2 would be achieved
operationally.  Similarly, models and measures to be developed under the other goals are
destined for use in the DSP, but it is not clear that they are constrained in any way to
promote convergence across goals/projects.  Therefore, the Committee recommends that:

•   In the Plan, EPA should clearly describe how mapping, monitoring and modeling
    research conducted under Long-term Goal 2 (and modeling work proposed under
    other long-term goals) would be coordinated with work to develop the DSP.  EPA
    should describe how collaboration on these research projects would  be achieved
    operationally.

       Outreach and Education (OE)

   Long-Term Goal 1 of the Plan contains an OE component. The Committee notes,
however, that OE has not historically been a significant part of ORD's work and,
therefore, additional expertise may be needed in this area. It will be important for ORD
to coordinate outreach activities with other EPA Offices such as the Office of
Environmental Information. The Plan specifically alludes to the use of participatory,
deliberative processes.  This will require expertise in the use of these types of processes,
but there appears to be limited (if any) expertise in this area within ORD. Aside from
direct work on decision-aiding processes of this type, the OE component of the plan
could seek to educate the general public about ecosystem services, under the assumption
that one way to influence decision makers is to generate pressure from consumers and
voters. This suggests the need for a more comprehensive OE plan, which will require
                                       13

-------
human capital resources to provide necessary education. In particular, the Committee
finds that efforts to "teach the teachers" could be very useful.  The Committee
recommends that:

•  EPA should develop a more comprehensive OE plan addressing human capital
   resource needs to provide the education. The committee supports the Agency's plans
   to pursue  opportunities for partnering with outside groups for these types of activities.
   The partnership with National Geographic is a good example of the kinds of activities
   needed.  In addition, community and education outreach programs have been
   developed through external funding mechanisms (e.g., Superfund Basic Research
   Program grants and National Institute of Environmental health Sciences Center
   grants). EPA should consider utilizing the resources and expertise that have already
   been developed through these mechanisms. It is also important that all outreach
   activities be evaluated to determine their effectiveness. The data used for such
   evaluations should be  collected according to social science standards (i.e., not just
   using "web hit" or television view data).

Overall feasibility of accomplishing Long-term Goal 1

   A major concern of the Committee relates to the overall feasibility of accomplishing
Long-term Goal 1. The plan to accomplish this goal is ambitious, and the Committee
questions whether ORD can realistically achieve the objectives and accomplish the tasks
set forth here. The following factors (some of which have already been discussed)
contribute to this concern:

       The design of decision support tools that can adequately address specific decision
       contexts will be difficult, given the wide diversity of: 1) needs of specific decision
       makers;  2) types of ecosystem services being addressed; 3) relevant geographical
       scales; 4) relevant jurisdictions; and 5) specific locations of interest.

       Development of the DSP is likely to be very time-consuming and  costly.

       There is currently  insufficient expertise within ORD to conduct the proposed
       research. Much of the research requires social and decision science expertise,
       which is generally lacking in ORD.  Although the plan calls for partnerships with
       other units within EPA (e.g., NCEE) or outside, the nature and strength of these
       commitments is unclear. For example, the commitment articulated by NCEE is
       fairly  limited and certainly not sufficient to meet the research objectives regarding
       valuation included in the plan. EPA should draw upon the available expertise in
       the U.S. Department of the Interior (Bureau of Land Management), U.S.
       Department of Agriculture (U.S. Forest Service), and U.S. Department of Defense
       (test and training ranges). These agencies are required to conduct ecological
       assessments of property under their jurisdiction and they support well funded
       ecological research programs whose activities may be leveraged by ORD. The
       Committee notes, however, that relying on the good will of partners to meet the
       objectives and annual performance goals of a major part of the plan is risky.
                                        14

-------
       Although the ORD identifies decision support as a fundamental driving force for
       the Plan, the resources devoted to this part of the Plan constitute a small
       percentage of total resources available to the Ecological Research Program.

       The timing of the work related to this objective is unclear. While it may be useful
       to collect currently available information about ecosystem services and their
       value(s) in a central on-line location in the early years of the Plan, the main payoff
       from the decision support will come much later when new research results and
       decision tools are available and incorporated into this platform. Alternatively, the
       DSP could be designed and then "tested" using the place-based projects in the
       Multi-Year Plan. The Committee finds that in all of these cases, the objective of
       having a fully operational decision support platform in place within five years
       may be unrealistic.

   Concerns about the feasibility of this part of the Plan are particularly worrisome
because ORD has suggested that ultimately the success or failure of the Plan hinges on
the success or failure of the decision support platform. The Committee recognizes the
need to ultimately justify the ORD ecological research program based on its ability to
affect decisions.  However, we recommend that:

•   Development of the DSP should be a long-term objective and not a short run test of
    the program's effectiveness (based on metrics such as the number of users of the
    decision support platform). The committee believes that ORD can contribute to this
    long run objective through other parts of the Plan even if it does not produce the type
    of fully operational decision support platform envisioned in the plan within the next
    five years.

Long-term Goal 2 -National Inventory, Mapping, and Monitoring

   Long-term Goal 2 envisages developing a publicly accessible, scalable national atlas,
an inventory system, and models for selected ecosystem services.  The Plan states that
these research products will enable EPA, state and local governments, non-governmental
organizations, and other decision makers to assess the likely effects of management
actions on ecosystem services.  The Committee finds that the work to be conducted under
Long-term Goal 2 may be one of the strongest parts of the Ecological Research Program
Multi-year Plan because EPA has extensive experience in developing environmental
inventories, mapping, and monitoring.  The maps and resulting models developed under
Long-term Goal 2 should definitely be incorporated into the Decision Support Platform
of Long-term Goal 1. However, the Committee notes that more detailed information is
needed to completely understand how this would happen.  We presume that such
information will appear in an implementation plan to be developed by ORD. The
Committee is concerned that the plan not define ecosystem services too narrowly,
overemphasizing basic human health and welfare goals. For example, under a narrow
perspective, the Arctic National Wildlife Refuge would have no value other than its
ability to produce oil.  The use of valuation has merit in the management of human-
                                        15

-------
dominated landscapes, but a major aspect of resource management, namely non human-
dominated systems, should also be considered in research questions and objectives under
Long-term Goal 2. In this regard, the key for the Ecological Research Program is to be
sure that research addresses all ecological components and processes that are important to
the provision of any services identified as relevant to EPA mandates and responsibilities.
In addition, it is important that adequate attention is given to identifying all of the
services to which any given component or process contributes, including services not
explicitly targeted within a given policy or decision-making context. With regard to
Long-term Goal 2, the Committee provides the following specific comments and
recommendations concerning: 1) forecasting models, the atlas of ecosystem  services, and
modeling expertise; and 2) the need for coordination of federal agency monitoring
activities.

Forecasting models, the atlas of ecosystem services, and the need for modeling
expertise

   Considerable data have been accumulating from numerous federal monitoring
programs; Olsen et al. (1999) identify at least  15 of these programs.  Some of these
monitoring programs are based on probability sampling, others on site characteristics.
Sampling occurs at different spatial and temporal scales,  resulting in different lengths of
series.  Thus far, the monitoring  programs have been used largely to determine status and
trends. The Committee finds that EPA now needs to address questions such as: How and
why are ecosystems and ecosystem  services changing? How are ecosystems being
affected by humans?  and finally How might management decisions reduce negative
consequences, or even result in beneficial gains?  The Committee also finds  that the idea
of developing a scalable national atlas is a good one; the atlas can be an excellent
communication tool but it should be linked to modeling efforts. The Committee
specifically recommends that:

•   EPA's Ecological Research Program should plan to develop forecasting  models from
    the information in available databases.

•   The atlas should be linked to models that can predict  changes in ecosystem services.
    The monitoring data should lead directly into the atlas and the forecasting models; by
    doing so EPA will be capable of assessing the consequences of choices.  The
    demonstration projects are the places to try to forge the connections between the
    maps, models, and forecasting tools.

•   The Plan proposes development of an Ecological Research Program "Community of
    Practice for Modeling."  This is a laudable idea, but the Committee questions who
    will participate, and where these modelers will come  from. The Committee
    recommends that EPA invest in meeting the need for graduate education to produce
    the next generation of modelers, and notes that industry has apparently started to do
    so.
                                       16

-------
Review of monitoring projects by the "federal family"

   As previously mentioned, numerous federal agencies are conducting ecosystem
monitoring activities. Given resource constraints, it is important to ensure that these
activities are well planned and coordinated.  In this regard, the Committee provides a
number of recommendations.

•  EPA should collaborate with other federal agencies to conduct a review of all federal
   agency  ecosystem/ecosystem services inventory, mapping, and monitoring type
   projects. This review could be conducted through a workshop similar to the type
   conducted by the National Center for Ecological Analysis and Synthesis (NCEAS,
   2008).  This review should bring together all of the various federal agency
   components as a "federal family" to optimize coordination and synergy among these
   different monitoring programs. The Committee notes that significant advances in
   monitoring have been realized through work conducted by ORD's Environmental
   Monitoring and Assessment Program. This work should be continued as the
   responsibility for that monitoring is assumed by other programs in EPA.

•  The suitability of various databases for use in developing EPA's Ecological Research
   Program products should be assessed as  soon as possible and definitely before 2013.
   One of the goals of the workshop recommended above would be to determine
   whether the scales of sampling and measurement are small enough.  Programs like
   EPA's Environmental Monitoring and Assessment Program (EMAP) were set up for
   inference at regional scales that may be too large for what is desired by the EPA's
   proposed Ecological Research Program.

•  The Committee finds that, subsequent to the workshop mentioned above, a regular,
   high visibility assessment of ecosystem services in space and time could be the most
   important product to come out of EPA's Ecological Research Program. The
   Committee recommends that EPA conduct such an assessment. It could be patterned
   after the Intergovernmental Panel on Climate Change model, which has certainly
   garnered international attention.  EPA's Ecological Research Program has the
   mapping and landscape ecology expertise to carry  out this work.

•  The Committee recommends that EPA provide some examples in the Plan to illustrate
   the link between ecosystem structures/functions and ecosystem services.  For
   example, water provisioning is an ecosystem service that could be linked to a wide
   range of interconnected ecosystem structures and functions.

Long-term Goal 3- Nitrogen Assessment

   Long-term Goal 3 of the Plan calls for an assessment of the positive and negative
impacts on  ecosystem services resulting from changes in nitrogen levels at select
locations and within select ecosystems. The Committee commends ORD for providing in
the Plan a more than ample background discussion of the importance of reactive nitrogen
(Nr) to terrestrial and aquatic ecosystems. We agree with the assertion in the plan that
                                       17

-------
this is an important area of ecological research. The decision to study Nr instead of other
chemicals is justified because of its ubiquitous nature, and the scale of its impacts on
multiple ecosystems and human health (as is being explored in the SAB Integrated
Nitrogen Committee study). Nitrogen is often the limiting nutrient in eutrophication, and
hence responsible for algae blooms and other impacts. Nitrogen is deliberately spread
across the environment as fertilizer in massive amounts (12 million tons per year) in
addition to being inadvertently released from livestock production, fossil fuel
combustion, waste water treatment, and industrial processes.  However, given the
relatively modest available resources, we have some concern about what can be
accomplished in this area, and how EPA's contribution will complement what is being
done in other agencies.  The following comments and recommendations are provided to
improve this part of the Plan.

•  The Committee recommends that a more detailed description of the research
   proposed under Long-term Goal 3 be provided.  The Committee expects that it is
   EPA's intention to provide this in the implementation phase of the program. At this
   point, however, some Committee members find that the fundamental question to be
   addressed by the Nitrogen Assessment is not clearly presented. We suggest that this
   fundamental question might be, "How can Nr more effectively managed to lower its
   environmental, health, and economic costs?

•  The Committee recognizes the potential value of investigating Nr because it
   represents a cross media approach for evaluating ecosystem services and it also
   impinges on human health. However, there are a number of other agencies (e.g., U.S.
   Department of Agriculture, and National Oceanic and Atmospheric Administration)
   and some programs within EPA (e.g. Office of Air and Radiation) conducting
   scientific studies and research on Nr as related to human health issues.  The
   Committee therefore recommends that ORD reduce the chance of duplication of
   effort by partnering with other federal agencies, including the Department of
   Agriculture, Department of Energy, Department of Transportation and National
   Oceanic and Atmospheric Administration, and other EPA offices conducting
   scientific studies and research on Nr as related to human health issues. We also note
   that the National Science Foundation and other foundations are increasing funding for
   Nr research. Nr should be viewed in a multimedia context with major implications
   for human health as well as environmental quality. EPA is in the best position to
   coordinate this effort to better understand Nr and develop improved technologies and
   practices for Nr management and control. In partnership with other agencies and
   EPA Offices, ORD might eventually contribute to a better understanding of the
   significance of Nr to ecosystem services flows and human health and well-being.

•  The discussion of Long-term Goal 3  in the Plan should contain a clearer explanation
   of why Nr was chosen for study. The Plan clearly describes the importance of Nr to
   ecosystems, and the Committee recognizes that EPA intends to initially undertake a
   modest Nitrogen Assessment at specific locations and eventually expand this to a
   national effort. However, the Committee finds that the Plan does not clearly or
   convincingly state why EPA's Ecological Research Program should include a
                                       18

-------
   Nitrogen Assessment at the limited level proposed.  The Plan states that Nr can have
   both positive and negative effects on ecosystem services and that both the positive
   and negative ends of the spectrum must be examined.  We strongly agree with that
   conclusion and note that this departure from the "negative only" approach is
   commendable.  However we recommend that EPA more fully discuss the rationale
   for choosing to study N in the manner proposed.

•  The Plan states that the nitrogen assessment will take advantage of ongoing studies in
   wetlands and coral reefs.  The Committee finds that concentrating Nr research on
   wetlands would be profitable, but we note that it would also be profitable to
   concentrate on terrestrial  systems (e.g., in the western U.S. where N is often limiting
   productivity). Although coral reefs are important in many parts of the world, they do
   not have a high importance to the majority of U.S. citizens (see below).

Long-term Goal 4 — Ecosystem Assessments

   Long-term Goal 4 of the Plan focuses on investigation of the dynamics of ecosystem
service flows in two priority ecosystems, wetlands and coral reefs. The Plan states that
both of these ecosystems deliver a wide range of services  (e.g., fish and fiber production,
water supply support, water purification,  climate regulation, flood regulation,  coastal
protection, recreational opportunities, and tourism). Furthermore, the plan indicates that
these systems are in serious decline (Dahl, 2005; Wilkinson, 2004) and that efforts to
manage and protect them have been inadequate. The Committee finds that the long-term
goal of assessing ecosystem services in wetland ecosystems is entirely appropriate, but
notes that it will be a challenge to address the complex spatial and temporal issues of
ecosystem processes  and their linkage to  ecosystem services (and ultimately their
valuation). These areas will require significant resources  for research extending beyond
those currently identified (i.e., the availability of EPA ORD scientists). In addition, while
we recognize that the purpose of the Plan is to provide a visionary "big picture" of EPA's
goals and objectives for ecological research, we note the need to address many complex
issues concerning project design and uncertainty associated with the research to be
completed under Long-term Goal 4.  ORD has indicated that these critical details (some
of which are described below) will be addressed in follow-up implementation plans. The
Committee provides the following recommendations to further develop and implement
Long-term Goal 4:

•  The follow-up implementation plans that will describe many complex issues
   concerning project design and uncertainty associated with research to be completed
   under Long-term Goal 4,  and other long-term goals, should receive outside peer
   review.

•  The initial projects to be undertaken by EPA to accomplish Long-term Goal 4 should
   focus on a small set of representative wetland systems and perhaps also  include a
   national assessment.  This would produce useful examples for different user groups.
                                        19

-------
•  The Committee finds that, although coral reef systems are globally important, other
   more common "human dominated" ecosystems may provide services to more U.S.
   citizens, and greater opportunities for coordination and collaboration with other
   studies within the ecological research program. We therefore recommend that the
   Program consider undertaking  projects in other more common "human dominated"
   ecosystems. If coral reef research is retained in the Plan, it should provide a better
   explanation of how studying the dynamics of ecosystem service flows in coral reefs
   will advance ecological sciences and ultimately help inform decision making.

•  Research efforts under Long-term Goal 4 should be integrated with some of EPA's
   other multi-year programs to more efficiently utilize resources.

•  The Committee recommends that, as research on this exciting area is accomplished,
   ORD develop a strong, active, iterative adaptive management process that modifies
   the process and coordinates efforts across the many research entities (e.g., EPA ORD
   laboratories, universities, National Science Foundation, National Oceanic and
   Atmospheric Administration [NOAA], and Department of the Interior). It is critical
   that this process and the approaches used receive "buy-in" now from these potential
   partners to ensure the success of this effort.  Given today's funding climate, joint
   partnership is essential.

•  The Plan should acknowledge that this approach is an extension of the EPA
   Ecological Risk Assessment framework and relate the  process to the risk assessment
   framework of Problem Formulation, Exposure and Effects Characterization, Risk
   Characterization, and Risk Management. The many critical issues and
   recommendations identified in the 2007 U.S. EPA Science Advisory Board (2007)
   report on improving ecological risk assessment (EPA Science Advisory Board, 2007)
   should be incorporated into the Plan. In this regard, spatial and temporal issues are
   particularly important.

•  The Committee recommends that in the Plan, ORD acknowledge and tackle multi-
   stressor diagnosis and subsequent ranking/linkage to ecosystem attributes, and then to
   services. Understanding "why" (i.e., causality) ecosystem services are lost in multi-
   stressor systems is a key missing piece. This work is critical to the success  of the
   overall approach articulated in the Plan. If such work is not undertaken, there will be
   substantial uncertainty in the model predictions and thus in EPA's ability to validate
   the approach. For example, if databases do not effectively characterize the
   spatial/temporal components of "background" or "reference," then it will not be
   possible to link a stressor with an adverse effect (or service loss), nor evaluate the
   effectiveness of a Best Management Practice in restoring an ecosystem attribute (and
   service). It is critically important to establish sound linkages among biophysical
   processes.  Such work should be regularly reviewed by external experts. This could
   be done as part of the implementation plan.

•  As discussed above, funding this effort will be a challenge.  To improve the chances
   of success, the Committee recommends that ORD follow a strategy of undertaking
                                       20

-------
    one or two simpler pilot projects initially, where tangible products showing the
    process from beginning to end can be produced within a three-year period.  This
    approach would increase the likelihood of new and continued funding, allowing for
    "proof of concept" and additional stakeholder buy-in.  Simultaneously, long-term
    projects could be proceeding.  There will undoubtedly be continual advances in the
    tools being created and the ability to value services each year, so work under Long-
    term  Goal 4 should continue to advance for many years to come.

Long-term Goal 5 - Place Based Demonstration Projects

   Long-term Goal 5 of the Plan calls for place-based research to investigate ecosystem
services. ORE) has chosen to focus on four different areas for proposed place-based
demonstration projects: Tampa Bay, the Midwest (13 "breadbasket" states); the
Willamette River; and the coastal Carolinas. Figure 22 on page 94 of the Plan provides a
partial map of the United States showing the location of these areas. There was a
diversity of opinion among Committee members regarding the suitability of these four
different areas for place-based  demonstration projects.  During the Committee's
deliberations, it became clear that this diversity of opinion was due to a lack of adequate
and transparent explanation in the Plan regarding the specific choices. The Committee
recognizes that there are no 'perfect' choices, but notes that a high degree of acceptability
can be obtained by well rationalized, transparent choices.  We therefore recommend that:

•   The Plan should contain a transparent explanation of the process used to select sites
    for place-based demonstration projects.  To this end, we recommend that EPA
    consider using the following organizing principles (along with others as appropriate,
    so long as they are transparent) for selecting and justifying different areas for place-
    based demonstration projects. Whether more or less than four such areas will be
    chosen will be governed by these principles:

       The areas must be widely representative of the major ecological areas in the U.S.
       where humans live or on which they rely.

       Historic, current and projected future changes to ecosystem services in these areas
       must be documented/predicted (in this regard we support use of the concept of
       "ecosystem services districts and operational  management options" discussed on
       page 5 of the Plan).

       It must be possible to generalize/transfer the findings of place-based
       investigations to other geographic areas/systems in the U.S. (and also, where
       appropriate, outside of the U.S.)

       The selected areas as a  set should provide opportunities for systematic
       comparisons and contrasts in important ecosystem services, structures and
       functions, as well as opportunities for collaborative studies in concert with the
       wetland (and coral reef or alternative ecosystem) and the nitrogen study
       components of the Ecological Research Program. For each selected area,
                                        21

-------
       appropriate data must be available on the local ecology, ecosystem services, and
       changes in those services.

   -   Adequate local resources (EPA or other [partner] staff and facilities) must be
       available.

       Although not an organizing principle, it is also highly recommended that local
       decision makers be supportive of these efforts in their area.

•  When the choices are made, they should be shown on a map that includes all U.S.
   States and Territories, which is not presently the case in Figure 22 on page 94 of the
   Plan. This will provide transparency regarding key ecological areas excluded (e.g.,
   Alaska is presently  excluded but not included on the figure).

•  In the Plan, some clarification of the text that supports the final choices is needed.
   The Plan should indicate that:  a) scales differ for a purpose - large and small scales
   need to be chosen (both within and between component studies) to attempt to
   determine what scale is most tractable/useful, and b) biofuels are not the only focus in
   the Midwest. With regard to the latter point, we note that the only mention in the
   Plan of life-cycle assessment (LCA) is in the Long-term Goal 5 in relation to biofuels.
   LCA is a useful means for visualizing and assessing different alternative actions
   relative to management alternatives. We therefore provide the following
   recommendation concerning LCA:

•  We strongly urge EPA to consider expanding the application of LCA in the Plan
   beyond biofuels, at  least in the form of demonstration projects that could be used to
   show the utility and need for this approach relative to future decision making.

•  The Committee emphasizes the importance of coordination and attention to
   interrelationships across the place-based demonstration projects.  This is explicitly
   mentioned in the Plan: ORD apparently has a designated place-based coordinator, and
   there is specific mention in the Plan of relationships to the nitrogen theme and the
   wetlands ecosystems project. However, we find that the brief descriptions of the
   individual projects do not show how such coordination will be operationally
   achieved. The usefulness of the "quintain" approach discussed on page 93 of the Plan
   (i.e., a function or condition studied in multiple cases to evaluate similarities and
   differences in order to better understand the whole) (Stake, 2006) would be more
   evident if a strategy for cross examination of functions and services were explained in
   more detail.

•  The Committee strongly recommends that transboundary issues be explicitly
   considered in the place-based projects. Due to atmospheric transport,  such issues will
   apply to  all projects, even those geographically isolated from political borders. We
   were surprised that transboundary issues were not discussed or considered in the
   discussion of Long-term Goal 5, particularly since the proposed mid-Western place-
   based demonstration project includes the border with Canada and the Great Lakes,
                                        22

-------
   which are managed by Canada and the U.S. as one entity.  Similar transboundary
   issues exist elsewhere; e.g., conditions in the coastal waters of British Columbia,
   Canada influence management in the Puget Sound/Georgia basin in Washington.

4.3    Charge Question 3. Please comment on the logic model approach and
       provide any recommendations that should be considered in developing
       implementation plans.

   In the Plan, ORD has provided a logic model that describes how the Ecological
Research Program will be designed, planned, implemented, and managed. The model
also  summarizes: 1) how research results will be communicated to users, and 2) the types
of outcomes and specific environmental results that the research program is designed to
achieve.  This model is summarized in Figure 4 on page 14 of the Plan.  The Committee
finds that the logic model approach articulated by ORD is a reasonable way to represent
the research activities that comprise the Plan. The logic model construct of inputs and
activities focused on particular outputs and, more importantly, outcomes is sensible.
Indeed, the Plan states explicitly that, without appropriate outcomes, research efforts and
the results that will follow are  of little utility. A similar approach is shown for EPA
research in general in the recent National Research Council (NRC, 2008) report.  This
NRC report, Evaluating Research Efficiency in the U.S. Environmental Protection
Agency, discusses the difficulty of evaluating research programs in terms of results,
which are usually described as outputs and ultimate outcomes. NRC (2008) notes that
between outputs and ultimate outcomes are many kinds of "intermediate outcomes" that
have their own value as results and can therefore be evaluated. The logic models in the
Ecological Research Program Multi-Year Plan and in the NRC report both show the
sequence of research, including inputs, outputs, intermediate outcomes, and ultimate
outcomes. By placing efforts into the structure of this kind of logic model, the Ecological
Research Program can in essence work backward from desired outcomes, and can
improve the potential that research efforts will be appropriately framed.  The Committee
does, however, have the following comments and recommendations that ORD should
consider as it refines and implements this logic model.

•  The outputs and outcomes  listed in the model are generic; considerable thought and
   attention must be put into ensuring that the appropriate specific outcomes are
   formulated.

•  The Committee recommends that ORD consider adapting some of the terminology
   and structure of the NRC logic model, particularly when research outputs are
   formulated.  ORD should consider including intermediate outcome boxes in the
   model as shown in Figure 4-1  on page 37 of the NRC (2008) report (outcomes from
   the research itself, and outcomes from users of the research). In addition, it will be
   critical that careful analysis and oversight of these outputs and outcomes occurs
   through time, and that feedback from outcomes is used to reevaluate both the
   necessary inputs and the activities, thus completing the loop suggested in the Figure 4
   of the Plan.
                                       23

-------
•  The Committee recommends that feedback loops be explicitly incorporated into the
   logic model.  It is important to ensure that the outputs lead to useful outcomes; if they
   do not, then the Ecological Research Program must address and adjust its activities.
   Such feedback loops, while implied in the logic model structure, are not explicitly
   described.  In addition, this mechanism will be  an important way for the Ecological
   Research Program to get feedback on the quality and utility of the research and tools
   being provided.

•  The Committee recommends that the logic model explicitly identify linkages to
   partners that are collaborating in research activities. The model shown in Figure 4 of
   the Plan appears to be internal to the EPA Ecological Research Program, even though
   many partners will be collaborating in the research activities. Thus, it is important
   that the transfers to and from other users be collaborative in nature, and not passive.
   This is necessary for other offices within EPA,  other users of the data from a
   management perspective, and the outside research community.  These linkages need
   to be shown in the model.  As noted elsewhere, the Committee is very concerned that
   the relatively small investment in outreach and  education, only 1% of the total effort
   overall, is not likely to be large enough to ensure these collaborations and transfers.
   Therefore, the Ecological Research Program will have to find creative partnerships to
   ensure that these interactions occur and that they are collaborative.

•  In addition, the "Externalities" identified in Figure 4 of the Plan should not be defined
   as such, at least not within the terminology of economics.  It is recommended that a
   more appropriate term, such as external forcing functions, be used to identify these
   important drivers.

4.4    Charge Question 4. Please comment on anticipated challenges to achieving
       the overall goal  of the Ecological Research Program Multi-Year Plan based
       on the Program as presented.

   The Committee has identified a number of challenges and research opportunities that
the Ecological Research Program will face as it strives to  achieve program goals.  It is
important to clarify that the Committee does not view these challenges necessarily as
shortcomings, but rather inherent issues that will persist and must be explicitly addressed.
The Committee recognizes four broad categories of challenges that are associated with:
1) the nature of the overarching research questions  and annual performance goals; 2)
specific methodological  or tactical approaches;  3) efforts to extend program outputs to
partners and other user groups in order to support decision-making processes; and 4)
resources, including institutional capabilities. Many of these challenges were clearly
articulated in the Plan. The Committee has also identified a number of cross-cutting
ecological research opportunities to improve and contribute to a variety of EPA
programs. We provide the following comments on these challenges and opportunities.
                                        24

-------
Challenges associated with the nature of overarching research questions and
performance goals

   The Committee commends the authors of the Plan for articulating an ambitious and
exciting vision for the Ecological Research Program.  The Committee finds that the
vision is appropriately bold and far-reaching, but we find that it would be helpful to focus
the vision on the timeline in the Plan (i.e., articulate the specific pieces that can actually
be accomplished in the proposed timeframe).  Several members of the Committee felt
that the specific long-term and annual performance goals were particularly ambitious
given the limited resources and short time span of the Plan.  Achieving fewer or narrower
goals is generally preferable to falling short of overly-ambitious aims. The Committee
recommends that the organization of the Plan be altered to more clearly distinguish
between the long-term  goals of the Program and the short-term specific objectives that
might actually be accomplished. Separating the vision statements and long-term goals
into a separate section at the beginning of the Plan would make it clear that these are not
intended to be accomplished in full within the time and resources of the current Multi-
Year Plan. Subsequent sections of the document could focus on the short-term goals and
objectives intended to be accomplished within the current Plan. In light of the need to
focus the goals, the Committee notes that reducing possible redundancy and increasing
connection/interaction with previous or current work of other agencies is imperative.
Two other general areas of concern are related to the heavy emphasis on the utilitarian
values of ecosystem services, particularly as related to human health, and the
comparatively little attention given to understanding the effects of multiple stressors on
ecosystem services. As noted above, adequate attention should be given to identifying all
of the services to which any given ecosystem component or process contributes,
including services not explicitly targeted within a given policy or decision-making
context.  Consideration of the effects of multiple stressors will be important in developing
ecological production functions for targeted ecosystem services.

Challenges associated with specific methodological or tactical approaches

   Given the ambitious nature of the Plan, the Committee finds that there are a number of
methodological challenges EPA scientists are likely to encounter. Although some of
these challenges were explicitly recognized in the Plan, it seems useful to highlight them.
Several methodological challenges relate to the use of data.  Clearly, developing metrics
for appropriate ecosystem services and connecting those indicators to human health and
well-being is a subject  of tremendous debate and will not be easily resolved. Similarly,
identifying the appropriate spatial and temporal scales of analysis and application is
exceedingly difficult, yet the Program's  success ultimately depends on getting this right.
Data management itself will likely pose  challenges.  These challenges involve not only
data manipulation, storage, metadata documentation, and analysis, but also  acquisition
(i.e., dealing with data gaps) and validation of data. Quantifying and articulating
uncertainty is a clear research opportunity related to data collection, analysis and model
development.  The Committee also recognizes that certain perceived challenges and
opportunities may derive from the fact that operational/tactical plans and implementation
strategies are still under development.
                                        25

-------
Challenges associated with extending program outputs to partners and other user
groups to support decision making

   The Committee recognizes that the ultimate success of the Ecological Research
Program lies in the extent to which it can support decision-making and regulatory
processes. Notably, decision-making tools such as risk assessment, life cycle assessment,
and the Natural Resource Damage Assessment and Recovery process need to connect
seamlessly to the proposed research program.  While the Committee finds that the goals
of the Ecological Research Program  are relevant to decision makers, we are concerned
that implementation of a successful outreach and education program is likely to be a
serious challenge for a number of reasons.  Most notably, we find that fully engaging the
diverse group of stakeholders and users will be difficult due to the diversity of their needs
and their capabilities to participate in the development of and/or use of the decision
support platform.  Active engagement seems essential given the reality that few users are
likely to train themselves.  Clearly, meeting the needs of users is further complicated by
the conflicting jurisdictional responsibilities of agencies and  organizations. Therefore,
the Committee recommends that:

•  Efforts be made immediately to enlist the input and cooperation of potential
   users/clients of the Ecological Research Program to better insure that the planned
   research will address issues of greatest interest to them, and that research outcomes
   can be communicated in a way that meets the most important user needs.

•  Direct links should be established between outcomes of place-based demonstration
   project research and policy and regulatory processes.  This is necessary in order to
   demonstrate the relevance and applicability of the Ecological Research Program to its
   partners.

In addition, the Committee is concerned that only 1% of the total budgetary resources of
the program are allocated to outreach and education.  It is the opinion of the Committee
that this amount is likely to be insufficient to support effective outreach efforts.

Challenges associated with availability of resources, including institutional capabilities

  The Committee applauds the authors  and contributors to the Plan for seeking to tackle
some of the most important, cross-cutting questions that we face in environmental
protection. Moreover, we  see that, simply by virtue of working through and developing
strategies to deal with the inherent challenges, efforts to develop the Plan represent a
tremendous opportunity to advance the way that ecological research is conducted.
The limited availability of resources  is the most serious and potentially problematic
challenge to the Ecological Research Program.  With the absence of funding in
competitive grant programs, such as  STAR, to fund partner efforts, the program will face
challenges in funding the necessary work and providing incentives for partner
involvement.  The lack of grant support is particularly problematic for involving
academic partners. As recognized in the Plan, the current Ecological  Research Program
                                        26

-------
staff skill set will not, by itself, be sufficient to address the issues and conduct the work
needed to achieve program goals. Reliance on partners for work to accomplish particular
program objectives is risky but, given the available program resources, that would seem
to be unavoidable at this point. In this context, the Committee recommends:

•  Cooperators and collaborators, both within and outside of EPA be identified as soon
   as possible and explicit agreements be drafted that specify what work is to be
   accomplished when by each partner and how resources will be shared.

4.5    Charge Question 5. What suggestions does the committee have for
       measuring annually over the next five years the progress, productivity,
       efficiency, and effectiveness of the Ecological Research Program?

   The recent NRC (2008) report on evaluating research efficiency provides
recommendations for the evaluation of research and development programs at EPA. The
Committee notes the following key recommendations provided by the NRC in this
regard: 1) EPA and other agencies should only apply quantitative efficiency metrics to
measure process efficiency of research programs. Process efficiency can be measured in
terms of inputs, outputs, and some intermediate outcomes; it does not require ultimate
outcomes.  2) EPA and other agencies should use expert review panels to evaluate the
investment efficiency (i.e., an  indication of whether an agency is doing the right research
and doing it well) of research programs.  The process should begin by evaluating the
relevance, quality, and performance of the research.  3) The efficiency of research
programs at EPA should be evaluated according to the  same overall standards used at
other agencies. In fact, the Plan indicates that EPA does intend to use expert  peer review
panels (e.g., the Agency's Board of Scientific Counselors, and the Science Advisory
Board) for future evaluation of the program.

   The Committee provides the following more specific comments and recommendations
concerning measurement of progress, productivity, efficiency, and effectiveness of the
Ecological  Research Program. In some of our comments we have referred to specific and
quantitative measures of program accomplishment. We therefore preface these
comments by noting the NRC  recommendations that quantitative efficiency metrics
should only be used to measure the process efficiency of research programs, and that
process efficiency should be evaluated only after relevance, quality, and  effectiveness of
a research program have been  evaluated.  We suggest that measured progress toward the
visionary goals and objectives in the more detailed implementation plans should focus on
the ecological structures and processes that contribute toward the production  of goods
and services, that themselves contribute toward human health and well-being. Program
performance should not be judged based on measures of the incorporation of concepts of
ecosystem services into management and regulatory decisions; this is a long-term goal,
whose achievement rests in part with the actions of entities other than the ERP. The
following recommendations are provided in this regard:
                                       27

-------
•  Goals and objectives should be monitored, reevaluated and adjusted as needed to
   capitalize on evolving and emerging partnerships and other opportunities to leverage
   the limited resources of the Ecological Research Program.

•  The stated goals, research objectives and performance measures of the Plan should be
   focused on the identification and articulation of the ecological processes and
   structures that contribute toward ecosystems services that have been identified in
   collaboration with ecological, medical, and social scientists in the Agency.

•  Specific research objectives should be operationally defined so that progress and
   attainment can be clearly determined and quantified.

•  In the specification of ecological production functions for targeted ecosystem
   services, the Ecological Research Program should maintain a broader ecosystems
   perspective to assure that the effects of multiple stressors on the multiple services that
   arise from these systems  are adequately acknowledged and addressed.

   The Committee finds that, given the visionary intentions of the Plan and the lack as
yet of detailed research implementation plans, it is premature to prescribe specific
measures to evaluate annual performance/progress goals for the program. However, as
development of the research plan goes forward, the authors of the Plan should specify
goals and associated research objectives for the individual projects and for the program as
a whole that are within the purview,  expertise and control of the Ecological Research
Program. As noted above, specific objectives should be operationally defined in a way
that: 1)  allows  clear determination of whether they have been achieved and 2) can be
subjected to quantitative measures of the extent of accomplishment. The Committee
further recommends that:

•  At this formative stage of the new ecosystems services paradigm, the program
   assessment  should include monitoring,  evaluation and adjustment of objectives as
   partnerships and collaborations within and outside the Agency evolve. Such an
   adaptive management approach requires flexibility  and vigilance to capitalize on
   opportunities that arise as the program continues to develop, and an explicit plan for
   coordinating activities and products across the multiple projects and themes of the
   Ecological Research Program.

   The Committee finds that it is appropriate for the Ecological Research Program to set
research goals based on contributions to understanding ecological service flows, and
through those service flows protection of human health and well-being. However, the
program should not claim responsibility (or allow itself to be held responsible) for
achieving the ultimate goals of the entire EPA research and regulatory mission. As
illustrated in Figure 1 below, the identification of relevant ecological  services and effects
on these services must be based on a dialog between Ecological Research Program
ecologists and the medical and social scientists, regulators and decision makers
representing EPA programs that are responsible for determining and valuing
environmental and human health and well-being goals of the Agency.  The key role for
                                        28

-------
the Ecological Research Program in this context is to research and articulate the
appropriate ecological endpoints and the intermediate ecological structures and processes
(ecological production functions) that contribute to identified services. Thus, the

                                Policy Question
 Ecological Research Program

 Ecological Context (e.g.,
 ecosystems, ecological production
 functions, stressor effects,
 response to management actions)
 Effects on
^Exosystep
       itfes
Valuation/Human Health and
Well-being Programs
Social and cultural context (e.g.,
social and economic value
assessments, human health and
well-being, response to ecological
changes)
                      Policy Decision and Implementation

Figure 1. The role of EPA's Ecological Research Program in an Ecosystem Services
Paradigm

evaluation of the success of the Ecological Research Program should be gauged in terms
of progress toward effective specification of relevant ecological endpoints and production
functions, with special attention to the effects of individual and multiple stressors that
come under the purview and regulatory control of the EPA. The Ecological Research
Program has the further responsibility to the Agency and to citizens of the country and
the world to investigate and bring attention to ecological processes and structures that
contribute to additional, non-targeted ecological services and potential services.

4.6     Charge Question 6. Does the Committee have any recommendations on how
        EPA can better enhance its ability to leverage available resources within and
       outside the Agency?

  As stated above, the Committee finds that the success of the Ecological Research
Program is  likely to depend in large measure upon its ability to leverage available
resources within and  outside of EPA. Based on information received by the Committee,
and our deliberative discussions, we have separated our comments on ways to leverage
resources into three topical areas. These three areas of concern are: 1) practical aspects
of implementation; 2) financial support for implementation; and 3) outreach and
education.
                                        29

-------
Practical aspects of implementation

   Because the Plan lays out a new approach, the Committee finds that there is a need to
avoid the perception that the Plan is being imposed upon the user community by ORD.
Thus, the Committee finds that there is a need to articulate a multi-level approach to the
Plan (i.e., research products will be developed at different levels for various users). In
addition, more input is needed from the end-users (e.g., municipalities, land managers,
industry) to identify the research products that would be most useful.

   In the Plan, ORD has identified potential partners for the development of new
methods and has indicated that memoranda of understanding will be developed to provide
arrangements for collaborative partnerships.  For example, the Plan cites a memorandum
of understanding that has been developed with the Gund Institute for Ecological
Economics to allow the sharing of data from study sites. The Committee provides three
recommendations concerning collaborative partnerships:

•  The Committee recommends that the memoranda of understanding to be developed
   with federal partners need to be more  than agreements to cooperate.  Specifics should
   be provided concerning who will do specific work when there is overlap, and how to
   share resources. During the Committee's discussions with EPA it was made clear
   that this is indeed the intent, but this needs to be articulated more clearly in the Plan.

•  Because there will be a need for access to expertise that may not be available "in-
   house," the Committee also suggests that ORD utilize Special Government
   Employees as  part-time consultants or advisors to quickly bring expertise to particular
   issues. In addition, EPA should consider negotiating Intergovernmental Personnel
   Agreements to enable government employees (local, state, or federal) with specific
   skill sets to be detailed to ORD or other EPA offices to meet program needs.

•  The success of the Plan is largely dependent on developing an effective outreach and
   education program, but the plan to develop an outreach program is not well
   developed. The Committee recommends that in the Plan ORD provide a section in
   the "vision" paragraphs to  outline how the Agency will achieve outreach and
   education goals. As stated above, this has not historically been a significant part of
   ORD's work; therefore additional expertise may be needed in this area.

Financial support for implementation

   It was made clear during the Committee's discussions with ORD that there are limited
resources available to achieve the goals of the Plan.  Therefore, it is important that ORD
consider reallocation or redistribution of existing resources to take advantage of
opportunities for partnerships with other groups and agencies. We provide six
recommendations in this regard:

•  The Committee finds that ORD's available people, infrastructure, and data represent
   leverage opportunities. We suggest that ORD use these opportunities as leverage to
                                       30

-------
   offer in-kind services and collaborate with other groups/agencies.  In this regard,
   there are ample partnership opportunities. ORD can partner with other agencies
   within the U.S. (e.g., U.S. Fish and Wildlife Service, U.S. Forest Service, National
   Park Service). For example, if a terrestrial place-based or ecosystem project is added
   to the Ecological Research Program, ORD can take advantage of U.S. Fish and
   Wildlife Service resources and expertise in existing projects. In addition, funding
   incentives for cross-agency collaborations such as the scientific community
   assessment of status and trends of ecosystem services in the U.S. (discussed
   previously) could enhance these partnerships.

•  ORD should consider active partnerships with other agencies outside the U.S. and
   thus gain the ability to address transboundary issues (e.g., watershed or airshed
   issues).

•  The Plan proposes partnerships with a number of nongovernmental organizations
   (NGOs). Beyond partnering with nongovernmental organizations, the Committee
   recommends that ORD consider working with professional societies to sponsor
   sessions or symposia in order to present results of work to accomplish the Plan's
   goals and solicit feedback from stakeholders and end-users. For example,
   partnerships with the following organizations could be considered: Society of
   Environmental Toxicology and Chemistry; North American Benthological Society;
   Ecological Society of America; North American Association of Environmental
   Educators; Association of Environmental and Resource Economists; and International
   Society for Ecological Economics.

•  The Committee also suggests that ORD consider partnerships with private business,
   foundations, NGOs, and non-profit foundations to conduct research and development
   activities.

•  We strongly encourage ORD to make the STAR program a priority in efforts to
   leverage resources and achieve goals of the Ecological Research Program by:
   enhancing the STAR Graduate Fellowships program to support ecological research;
   providing funds for exploratory extramural research to develop tools and procedures
   to accomplish the goals of the Plan; and developing a competitive grants program to
   run summer credit workshops for teachers through STAR that would support the
   outreach and education goals of the Plan.

•  The Committee recommends that ORD consider requiring or expecting leverage from
   universities in order to obtain ORD funding. Leverage can come in the form of
   reduced indirect costs or tuition and fee waivers. ORD could also consider providing
   matching funds or supplements to existing graduate and teacher education programs.

Outreach and education

   As stated previously, the success of the Plan is largely dependent on outreach and
education activities. Unless the human capital needed to bring expertise into the

-------
valuation process is developed, and the stakeholders and end-users are provided the
education needed to use the information, the tools and techniques developed will likely
not be used. To accomplish this, the Committee provides the following two
recommendations:

•  We recommend that ORD partner with professional societies, publishing companies,
   media outlets, and NGOs to develop and disseminate education and outreach
   materials to professionals, teachers, and the lay public.  Some suggested approaches
   that could be developed in partnership with other organizations include: workshops,
   symposia, and sessions at meetings, WIKI blogs, presentation materials for educators
   and public forums, media resources including cable television educational networks,
   and 10-15 minute video clips that can be used in classroom settings.

•  We also recommend that ORD partner with community groups to enhance education
   and outreach activities. It will be important to take advantage of local traditional eco-
   knowledge to address the issue of "sense of place" to gain acceptance of the valuation
   approach by end-users.

5.   CONCLUSION

   EPA's draft Ecological Research Program Multi-Year Plan FY 2008 - 2014
articulates a new strategic direction that focuses on quantifying ecosystem services and
their contribution to human health and well-being. As stated above, the Committee
strongly supports this strategic direction and commends the Agency for developing a
research program that has the potential to be transformative for environmental decision
making as well as for ecological science. We find that the research focus on ecosystem
services represents a suitable approach to integration of ecological  processes and human
welfare for the purposes of a public environmental management agency. The Ecological
Research Program's focus on ecosystem services can therefore provide a sound
foundation for environmental decisions and regulation based on the dependence of
humans upon ecological condition and processes. While we support the strategic
direction taken by EPA, we have concerns about the Agency's draft Plan.  The most
serious challenge facing the Ecological Research Program is the limited availability of
resources. We find that the long-term goals of the program are unlikely to be
accomplished in the proposed time frame with current resources. Furthermore, the ORD
staff skill set may be insufficient to address the issues and conduct  all of the work needed
to achieve long-term program goals.  Given these concerns and the fact that studying
ecosystem services is a field in its infancy, the lack of grant support is particularly
worrisome. We strongly encourage EPA to provide additional intramural and extramural
support (e.g., through STAR grants) for the Program.

   We have provided a number of recommendations to improve the long-term goals,
research objectives, and implementation strategy in the Plan. Our recommendations
focus on: 1) providing additional information to clarify how various research products
will be developed and used; 2) identifying and engaging as soon as possible clients who
will use the research products and targeting outreach efforts to educate those clients; 3)
                                       32

-------
working with other federal agencies to avoid duplication of effort and promote
coordination and synergy; 4) retaining the important long-term visionary goals, but
clearly identifying some relatively narrow goals and objectives that can be accomplished
on schedule with limited resources; 5) providing a more transparent explanation of the
process used to select sites for place-based demonstration projects; 6) evaluating program
success on the basis of progress toward specifying relevant ecological endpoints and
production functions, not achieving the ultimate goals of EPA's research and regulatory
mission; and 7) effectively partnering with other federal agencies, NGOs, professional
societies, private businesses, and foundations to leverage available resources.
                                         33

-------
6.   REFERENCES

Carpenter, S.R., R. DeFries, T. Dietz, H.A. Mooney, S. Polasky, W.V. Reid, and RJ.
Scholes.  2006.  Millennium Ecosystem Assessment: Research Needs. Science 314:257-
258.

Chapman, P.M. 2007. Future environmental science:  "Status humana", man as the
measure. Human Health and Ecological Risk Assessment 13:702-712.

Dahl, T.E.. 2005. Status and Trends of Wetlands in the Coterminous United States 1998
to 2004. U.S. Fish and Wildlife Service, Washington, D.C. 116 p.

National Research Council.  2008.  Evaluating Research Efficiency in the U.S.
Environmental Protection Agency.  National Academies Press, Washington, D.C.

NCEAS. 2008.  National Center for Ecological Analysis and Synthesis.
http://www.nceas.ucsb.edu/overview

Olsen, A., R, J. Sedransk, D. Edwards, C.A. Gotway, W. Liggett, S. Rathbun, K.H.
Reckhow, and LJ. Young.  1999. Statistical issues for monitoring ecological and natural
resources in the United States.  Environmental Monitoring and Assessment 54,  1-45.

Oreskes, N., K.  Schrader-Frechette, and K. Belitz.  1994. Verification, validation, and
confirmation of numerical models in the earth sciences. Science 263:641-623.

Ridder, B.  2008.  Questioning the ecosystem services argument for biodiversity
conservation. Biodiversity Conservation 17:781-790

Stake, R.E., 2006. Multiple Case Study Analysis. Guilford Press, New York, NY. 339 p

U.S. EPA.  2006.  2006-2011 EPA Strategic Plan: Charting Our Course. EPA-190-R-
06-001. U.S, Environmental Protection Agency, Washington, D.C. [Available at:
http://www.epa.gov/cfo/plan/2006/entire_report.pdf]

U. S. EPA 2008.  Causal Analysis/Diagnosis Information System (CADDIS).
http ://cfpub. epa.gov/caddi s/

U.S, EPA Office of Research and Development.  2008. Ecological Research Program
Multi-Year Plan FY 2008 - 2014 - February 2008 Review Draft. U.S. Environmental
Protection Agency Office of Research and Development,
Washington, D.C. [Available at: http://www.epa.gov/ord/npd/pdfs/ERP-MYP-complete-
draft-v5.pdf]
                                       34

-------
U.S. EPA Science Advisory Board. 2007. Advice to EPA on Advancing the Science and
Application of Ecological Risk Assessment in Environmental Decision Making: A Report
of the U.S. EPA Science Advisory Board. EPA-SAB-08-02. U.S. Environmental
Protection Agency Science Advisory Board, Washington, D.C. [Available at:
http://yosemite.epa.gov/sab/sabproduct.nsf/7140DCOE56EB148A8525737900043063/$F
ile/sab-08-002.pdf]

U.S. EPA Science Advisory Board. 2008a. Draft Report - Valuing the Protection of
Ecological Systems and Services. U.S. Environmental Protection Agency Science
Advisory Board, Committee on Valuing the Protection of Ecological Systems and
Services, Washington, D.C. [Available at:
http://yosemite.epa.gov/sab/sabproduct.nsf/ea5d9a9b55cc319285256cbd005a472e/f4771
258f94fda8c8525740900671186!OpenDocument]

U.S. EPA Science Advisory Board. 2008b. Summary Minutes of the United States
Environmental Protection Agency (U.S. EPA) Science Advisory Board Meeting, February
28,  2008 & February 29,  2008. U.S. Environmental Protection Agency Science Advisory
Board, Washington, D.C. [Available at:
http://yosemite.epa.gov/sab/sabproduct.nsf/a84bfeel6cc358ad85256ccd006bOb4b/62F53
7D2FF746527852573B400441FD6/$File/Minutes+SAB+02_28-29_2008+w+Atts.pdf]

Wilkinson, C. (ed.).  2004. Status of Coral Reefs of the World. 2004 vol. I. Australian
Institute of Marine Science, Townsville, Queensland.

Wu, J.D. and O.L. Loucks.  1995.  From balance of nature to hierarchical patch
dynamics: A paradigm shift in ecology. The Quarterly Review of Biology, 70:439-466.
                                      35

-------
APPENDIX A.  SPECIFIC COMMENTS ON THE ECOLOGICAL RESEARCH
PROGRAM MULTI-YEAR PLAN

  The following specific comments on various parts of the draft Ecological Research
Program Multi-Year Plan are offered by individual Committee members.

Page ii:

-  Ecological Research Program personnel do not appear to include many social
   scientists.  The Plan refers to a valuation team, but the individual named as the lead is
   an ecologist, not a social scientist. Similarly, the person named as the human well-
   being lead  is a biologist. Without more direct involvement from other disciplines,
   and more expertise specifically related to valuation, it is not clear that ORD will have
   the capacity to develop a meaningful decision support platform that meets Long-term
   Goal 1.

Page 1, Introduction:

   This part of the Plan should indicate how EPA will use  lessons learned from other
   programs.  The U.S. Forest Service and others have been managing ecological
   services for many years with varying amounts of success. It is not clear how this
   experience base was or will be used in the creation of the Plan.

Page 3:

   The list of "pioneering examples" on this page is a bit hard to fit into the plan for the
   future.

Page 4:

   Ecosystem services are defined here as "the products of ecological functions or
   processes that directly or indirectly contribute to human well-being, or have the
   potential to do so in the future." A concern about this definition is that it emphasizes
   the products rather than the processes that are the foundation for those products.

Page 5:

   The third bullet on this page indicates that enhancing understanding of ecosystem
   impacts that emerge over longer time scales, including threshold responses or tipping
   points, is reflected in the Ecological Research Program's ongoing suite of grants
   investigating threshold behavior and regime shifts in  aquatic systems. Examples of
   these research efforts (and findings) should be provided. This is a critical area and it
   is not apparent that the agency has invested much to support it.
                                       A-l

-------
Page 6, Table 1:

   This table presents priority ecosystem services, but it is not clear why or how this list
   was generated from the full set. What was the rationale, for instance, for having
   cultural services or nitrification in this table?  The logic behind the selections should
   be clearly presented. The lists of examples in the right three columns (regulating
   services, provisioning services, cultural services) appear to be incomplete.

-  Habitat and biodiversity are not services.  Both are very important but neither is a
   supporting service as defined in this table. Human well-being is derived from
   habitats and from having a biologically diverse condition in  that habitat. Trying to use
   a structural measure such as acres of habitat as a measure of ecological service will
   lead to confusion and possibly double counting of benefits.  Clearly there is a need to
   define the set of services that flow generally from specific habitat types (e.g. low
   marsh, high marsh, freshwater marsh, tidally flushed marsh), but these would not be
   separate services. Biodiversity is another structural measure of condition and we all
   might agree that more diversity is better.  However, if an upper limit to biodiversity is
   exceeded the process relationships that under lie ecological communities degrade.

   The list of ecosystem services in this table should be prioritized. If (or when)
   resources become limiting, there should be a structure in place for deciding what is
   most important. This would mean making a priori value statements, but some  of
   those ecosystem services are directly related to current human physical well-being,
   others to future physical well-being. Some are related to apparent economic status or
   current human psychological  well-being.

Page 8:

-  The proposed approach to measuring achievement of goals (i.e., by considering how
   the information is used by decision makers) is asking a great deal from a science that
   is not yet developed.

-  A simpler statement of general research questions presented here might be, "how and
   why are ecosystem services changing, how are they being impacted by humans, what
   are the consequences for human health and welfare, and how might management
   decisions reduce negative consequences?" More specific questions could address the
   theories and hypotheses to be tested. For example, how are  different temporal  or
   spatial scales to be integrated? One of the leading models for doing this, the
   hierarchical patch dynamics paradigm (Wu and Loucks, 1995), or another framework
   could be presented as a starting point.

Page 9:

-  The mention of multiple  stressors here is a positive feature.
                                       A-2

-------
-  The top two bullets and paragraph on this page are good but the research questions
   will be very difficult to address.  Answering these questions will likely take more
   resources and time than envisioned. We know that ecological responses to identical
   stressors can differ widely across regions, landscape, and social context. Much more
   work in a variety of contexts will need to be done in order to sufficiently answer the
   broad questions of 1) what are the effects of multiple stressors on ecosystem services
   at multiple scales over time and 2) what is the impact of changes in these services on
   human well-being and on the services' monetary and non-monetary value.

   It is surprising here that two "priority ecosystems" leapt to the fore so quickly.  Does
   this mean that the rest of the long-term goals are not national in scope? What is the
   rationale for selecting priority ecosystem types and priority geographic regions?
   There is no mention of investigating multiple stressors.

   In the general approach provided here, how does "landscape characterization" fit with
   ecosystem services  discussed in the rest of the Plan?

   The usefulness of "maps"  as described here and on page 43, paragraph 2) is critical,
   but the examples given are complex issues that cannot be crudely modeled.  Good
   data and an understanding of interlinking processes are needed. This requires
   substantial research.

   With regard to research outputs, the focus seems to be on carbon and nitrogen.  How
   can one model these two biologically driven  cycles without knowing the impacts of
   other key stressors (e.g., habitat, metals, organics, temperature, and hydraulics)? Will
   these impacts be defined?

-  Output #2 "stressors" should have a clearly corresponding counterpart that reflects
   not just things that degrade services (stressors) but also our ability to restore, reclaim,
   enhance services. We want to be able to predict not just losses, but our ability to
   achieve gains.  Later in the document it is clear that gains are being considered, but it
   does not come through in this section.

Page 10:

-  The first two bullets on this page do not seem to be different from one another.

   The last paragraph showing incremental changes in services due to a management
   action or the effect of an environmental stressor is good but it will require years of
   study of pre and post monitoring of best management practices - or an in-depth
   understanding of interlinking ecosystem processes which are modeled. The time
   frame required to accomplish this is uncertain.
                                       A-3

-------
Page 11:

   It is necessary to establish ecological "baselines" in order to measure both losses and
   gains.  "Baselines" should be given more emphasis in the Plan. Very little progress
   can be demonstrated until the Ecological Research Program can make a case for the
   baselines it is using.

-  The tables on pages 11 and 12 refer to several specific examples of "services."
   These services include nutrient removal, temperature regulation, habitat, and food and
   goods. These  services are also those described in the Millennium Ecosystem
   Assessment. However, there is a conceptual inconsistency with these services that
   acts as a barrier to clarity. "Nutrient removal" and "temperature regulation" are
   processes.  Habitat and food and goods are outputs of processes.  How do you
   measure a process?  By measuring the inputs to and outputs of that process.  A more
   consistent focus on the desirable (and undesirable) outcomes would be preferable as
   the focus of measurement.

Page 12, Figure 3:

-  This figure represents a potentially misleading and easily abused approach.  Applying
   monetary values to each of these services can be very divisive and open to
   "interpretation".  How much social value is applied to rice farming for example
   compared to fishing?  This graph shows we should never farm food since the loss of
   natural services will always exceed the food production.

Page 13:

-  It would be useful to see where inputs from other agencies and partners enter the
   logic model on this page. What or who will drive the cooperation among the 7
   research laboratories? How will partners be enlisted into the program? How will
   research be funded?

-  Timing of the  long-term goal outputs (pages 13  and 15 and figure 5) makes it appear
   that the place-based demonstration projects would be running in parallel with the
   mapping and model development and be completed prior to the decision support
   tools. This seems out of order. One would expect the place-based projects  to be an
   opportunity to test the tools, models, and maps.

Page 14, Figure 4:

-  The logic model presented here appears to be a useful way to characterize the
   relationships among the planning and implementation components of the proposed
   research activities within the Ecological Research Program. The model is less useful
   as a way to clearly place the Program activities in the larger environmental policy,
   planning and management context.  The "Externalities" component in the model
   identifies a number of potential constraints coming into the Program, but it does not
                                       A-4

-------
provide sufficient representation of environmental and social "inputs" (triggers, goals,
etc) such as environmental changes (from local floods to global climate change) and
social changes (population and demographic shifts, land development, etc). Nor does
the model show where Program research outputs go, such as to support EPA policy
making to protect relevant ecosystems functions and structures, to improve and
sustain the levels of ecosystems services that are enjoyed by citizens, and providing
scientific information to help educate publics about ecosystems services to secure
support for the protection of important ecosystems.

Outputs like peer-reviewed publications that are intermediate between doing the
research and observing outcomes are also important because there is still widespread
scientific skepticism that the concept of ecosystem services can be made operational.
Publications in journals such as Science, Nature, and Ecological Applications will
lead to more widespread acceptance of the concept among skeptical scientists. It is
legitimate for the Plan to focus on the research enterprise, but some acknowledgment
(in text and/or in the logic model figure) of where the Program fits in the larger
context would be a useful addition. Figure 4 makes it appear that the Ecological
Research Program is internal to EPA and it also appears that the Program is isolated
from the EPA Program Offices, Regions, and other ORD research programs.
Relationships between the Ecological Research Program and other research plans
should be acknowledged. Interactions with global change would include
collaboration on issues of carbon sequestration; interactions with the Office of Water
could relate to development of nutrient criteria as well as wetland and mitigation
evaluation procedures.  Establishing a linkage with the Human Health Research
Program seems particularly important. Another potential health link would be with
the Centers for Disease Control and Prevention.

The logic  model does not include reference to the quality of the research. Users will
not adopt implementation of items developed in the first three steps unless they are
part of adequate quality for making decisions.  The model also needs feedback loops
in case the models, maps or tools do not work.  In addition, the cost of tools does not
seem to be part of the process for evaluating how good the tools are. The tools should
be cost effective relative to the resources being protected.

In the logic model, why are the management options research outputs? Typically,
one would specify some possible options or policies under consideration and the
research would evaluate the impacts.

The objective is not to ensure human well-being by conserving and  enhancing
ecosystem services. What if there are tradeoffs (as there inevitably will be), either
between different ecosystem services and/or between ecosystem services and other
things that contribute to human well-being? Is the long-term environmental outcome
goal separate from a goal of enhancing human well-being?
                                    A-5

-------
Page 15:

   The five goals that are proposed here are individually important, but it is less clear
   whether they are collectively sufficient or the most important goals for EPA's
   ecological research efforts.  The Plan points out that the Ecological Research Program
   is one of several research programs within and outside of EPA and that the stated
   goals are intended complement those of the other programs. However, the brief
   description in the Plan does not convincingly show how the five goals and the noted
   efforts to cooperate with the other programs combine to cover the most important
   research needs of the Agency. The EPA should make a more comprehensive study of
   the interrelationships among the research programs cited (and others) and work
   vigorously to secure effective interrelationships and coordination among them.

   Similarities between the decision support tool mentioned here and EPA's CADDIS
   system (U.S. EPA, 2008) should be mentioned.

-  Uncertainty should be addressed in Long-term Goal 2 -National Mapping, Inventory,
   and Modeling.

Page 16, Figure 5:

   The figure illustrating the planning and implementation framework is confusing.
   Coordination and integration among the five goals of the proposed program are
   within the control of the Program. Such coordination is rightly a stated intention of
   the Program and the organization of the goals and projects implies an effective
   structure for achieving that end. However, the Plan does not adequately  describe how
   the coordination implied by the intersecting cells in Figure 5 will be operationally
   achieved.  There should be budget to support activities such as bringing project and
   theme leads (the bottom row and last column of the matrix) together periodically to
   assure that useful  coordination is planned and implemented, that schedules are set and
   upheld (or revised) so that progress on the separate themes and projects allows for
   timely and mutually beneficial sharing and integration of data, methods,  models and
   other information that is developed. In the Plan, more emphasis should be placed on
   how coordination among the goals/themes/projects will be operationally achieved.  It
   might be useful in this regard to define coordination activities as a sixth goal of the
   Ecological Research Program.  In addition, the resources allocations for the years
   2008 - 2014 should be identified.  It would seem that some projects will need more
   resources at the start and others will need more towards the end. Furthermore, it is
   difficult to evaluate the Program if the laboratories  and leads are not identified.

Page 17:

   The rationale for allocation  of the resource percentages to each long-term goal should
   be provided here.
                                       A-6

-------
Page 18:
   In Table 2 it is not apparent how the "overarching issues" of sustainability and global
   change relate to the "high priority topics" of endocrine disrupters, Hg, and
   nanotechnology.  It is a concern that these high priority topics have a human health
   focus. There needs to be a focus on natural stressors (e.g., habitat, temperature, flow,
   meteorological events) that are linked directly to human activities and climate change
   and are front and center for stressors and local to global impacts.

   The challenge presented here for EPA laboratories is great. They are likely to be
   entrenched in institutional momentum and tradition which will be difficult to change.
   In the second paragraph on this page it is stated that the Ecological Research Program
   has a close working relationship with the Global Change and Water Quality
   Programs. This relationship should be documented.
Page 19:
   In the third paragraph on this page it is stated that the Ecological Research Program is
   developing new methods to enhance, maintain, or restore the full range of water-
   related ecosystem services. This should be documented.
Page 20:
-  The purpose of including Table 3 is not clear. The table requires some additional
   discussion.  The Ecological Research Program workforce is indicated as internal,
   which contradicts what has been stated elsewhere, namely that there will be
   considerable reliance on outside collaborators.

Page 21:

   It is stated here that accomplishing Long-term Goal 1 will be one of the biggest
   challenges and that EPA has the least ability and internal expertise to deal with this.
   EPA should look externally and enlist the help of the academic community in
   addition to expanding internal resources.

-  When creating a large multi-model system to be used in a decision making context as
   described in Section 1.0, some systematic across the board validation would appear to
   be prudent.

Page 22, Section  1.1.1:

   The projects identified here include "associations between the condition of stream
   habitat and sport fishing revenue."  That kind of study has been done before; what has
   not been included in those kinds of analyses are other forms of recreation and
   spiritual renewal that are also dependent on condition of stream habitat.
                                       A-7

-------
Page 23:
   The discussion of decision tools is a nice "capstone" for the Plan but, in many cases,
   the science questions are a bit artificial, and could be better stated as scientific
   objectives.

   It seems unusual to use the terms "homes protected from flooding" and "recreational
   user days" to describe "population and human health issues." Also, terms like "urban
   greenspace and indicators of mental function" should be avoided. Doesn't this mean
   that urban greenspace can be valuable for a variety of reasons?  "Mental function"
   sounds either too vague or too peculiar.
Page 24:

   In developing a classification system of ecosystem services (Section 1.2.1), some
   recognition of regulatory structure should be acknowledged if this approach is to be
   useful to managers.

   In Section 1.1.2 recommend considering the increasing incidence of asthma and its
   relationship with air pollution. This seems to be a high priority as compared to
   nitrogen.

Page 25, Section 1.1.3:

-  Collaboration with some National Science Foundation research programs (e.g., Long-
   term Ecological Research Program, Human and Natural Systems - formerly
   Biocomplexity) with social science expertise would help in Section 1.1.3.

-  In the first bullet on this page, proposed work to conduct a spatiotemporal analysis of
   disease with sale of medical supplies/pharmaceuticals requires further justification.

-  The Ecological Research Program should ensure that at least one of the demonstration
   projects described here and elsewhere focus on an ecosystem service that can be
   taken "all the way to the end product."  That is, define an ecosystem service that can
   indeed be characterized, quantified, valued and its relationship to human health and
   well-being made clear. For example, the Plan suggests endpoints such as "reduced
   flood insurance payments, recreational expenditures, and reduced costs of mosquito
   control measures per wetlands area as potential endpoints." Page 25 of the Plan
   mentions "estimates of morbidity and mortality from air pollution levels under
   alternative scenarios of urban design."  This should be feasible.

Page 26:

-  The section lacks identification of specific efforts to include and/or to coordinate with
   relevant social science on human health and well-being. All long-term goals adhere
   to the ecosystem services framework and have at least one "valuation" objective, but
                                       A-8

-------
   it is not clear where the required measures of health and well-being will be obtained.
   The service targets of the Ecological Research Program can generally safely be
   assumed to be associated with human health and well-being (or at least they are all
   things that people generally care about), but there is little or no indication of any
   explicit effort to quantify and confirm specific associations within or across the
   particular themes/projects. For example, research is proposed to identify the
   ecological processes and structures in wetlands that affect the quantity, quality,
   spatial distribution (and timing) of fresh water. But there is no reference to how the
   models and maps of this (potential) service will be related to (e.g., overlaid with)
   relevant measures and/or projected characteristics of human/social "consumers"
   (demanders) of this service or where measures of such social characteristics will be
   obtained. Among possible sources of relevant social value information are the many
   national surveys conducted regularly by the U.S. government (U.S. EPA Science
   Advisory Board, 2008a) and focused surveys conducted by other regional, state, and
   local  agencies.

-  The annual performance goals listed in Table 4, beginning with 2010 as a target data
   for development and testing of the preliminary human health and well-being
   indicators tied to ecosystem services, seem to be very ambitious. Development  in this
   area will have to occur before results can be communicated to the client base
   described in Table 7.

   One example of valuation of certain ecosystem services from the Willamette River
   Basin is the Willamette Ecosystem Marketplace (www.willamettepartnership.org).
   The Marketplace conceives of a multi-credit bank for the Willamette Basin.
   Associated with this, the Willamette Partnership is a water quality trading program to
   cool the Willamette River. The Partnership integrates elements of ecosystem services
   into a "mitigation bank site" where credits can be bought and sold. The existence of
   the Partnership and the Marketplace means that environmental consequences are
   viewed as part of the economic  system, rather than external to it.

-  The way valuation is described here raises the concern that exploitation and alteration
   of natural and wild lands could increase.

Page 27:

-  The plan includes the development of an Ecosystem Services Classification System
   comparable to that used by the Census Bureau for industrial goods.  However, it is
   not clear that this type of standardization will be feasible, given the place-specific
   nature of ecosystem services. Nevertheless, some recognition of regulatory structure
   should be acknowledged if this approach is to be useful to managers.

Page 28, Figure 7:

-  The very philosophical Long-term Goal 1 described here may be quite elusive.  Will
   the Program really address the question of what economic valuation methods are most
                                       A-9

-------
   "efficacious" for valuing ecosystem services (as shown on Figure 7, page 28)? The
   current staff within ORD does not appear to have the needed expertise for answering
   this science question, and there is no meaningful discussion of any external funding
   for this component of the research.

Pages 28-29:

-  While the development of ecological production functions is an important objective,
   the description of this component of the Plan suggests some confusion about the
   concept of production functions. For example, economic production functions
   provide information about technological possibilities for substitutability, they do not
   provide any information about scarcity or the availability of complementary services.
   Likewise, production functions are not used for describing human well-being.

Page 31:

   The Plan makes reference to the use of information from the market for carbon offsets
   as a source of valuation information, but prices from tradable permit markets do not
   provide value information (except under very limited conditions).

Page 33:

   The use of NGOs to quickly enhance outreach and education activities is novel,
   innovative and should be encouraged.  This is how NGOs make a living, so why not
   take advantage?

   Regarding the text on pages 33 and 35  (Sections 1.3.1  and 2.0), client groups that will
   be receptive to using an ecosystem services approach include local watershed groups
   and the national NGOs they work with (e.g., American Rivers, River Network,
   Waterkeepers). Another potential  interested client would be developers of
   conservation subdivisions. Assessing ecosystem services arising from those
   developments could be couples with analyses of house prices, etc.

Page 35:

-  A more comprehensive education and outreach plan is needed here.

Page 43, Section 2.1:

-  EPA has a good deal of experience in monitoring (e.g., Olsen et al., 1999).  What is
   proposed under Long-term Goal 2 is at a scale and effort far greater than any of the
   current monitoring programs.  Agency program scientists will need to devote a great
   deal of thought to deciding what variables will be monitored, and at what spatial and
   temporal scales.  The temporal  scales do not have to be the same, even within a single
   monitoring program. As an example, the Oregon Plan for Salmon and Watersheds
   (run by the Oregon Department of Fish and Wildlife) has various sets of sampling
                                      A-10

-------
    sites (called panels) sampled at different frequencies: every year, every three years,
    every nine years, and every twenty-seven years (the multiples of three were chosen to
    coincide with salmon return periods). Yet, at any given point in time, information
    from all the sites, even though the sampling frequencies are different, can be
    combined in a statistically valid manner (based on statistical modeling results). Thus,
    information from different temporal and spatial scales of monitoring may be
    combined, as long as temporal/spatial correlation or other models have been
    developed to tie the pieces of information together.
Page 44:
   On this page and also in Figure 13 on page 96 it is difficult to visualize concrete
   results from some of the general statements (e.g., "quantifying ecosystem services").
   More detail would be helpful.

   A concern here is that the definition of ecosystem services to be monitored explicitly
   excludes ecological processes and functions as services. By excluding processes and
   functions one is only monitoring current state and not the underlying processes that
   generate that state. It apparently excludes rate measures, which would not appear to
   make sense if one is trying to measure provision of a service. An additional concern
   is that defining ecosystem services as those that are directly used by humans does not
   represent the value of natural systems and communities for their own sake (i.e.,
   existence value).
Page 45:
   Table 9, identifying core ecosystem services, is incomplete. Will climate change and
   nonpoint source runoff be considered? More information should be provided to
   indicate how this table was developed. What were the criteria for selection of
   services? On the next page, it is stated that biodiversity is directly measurable. This
   is possible with diversity indices, but that is feasible only with certain taxonomic
   groups. Which components will be chosen? In streams, for example, diversity of
   algae, macroinvertebrates, and fish respond differently to stressors.

   The atlas idea (Fig. 11) is an excellent communications tool; people are very
   comfortable looking at maps. The Willamette Futures Project has used an atlas
   successfully to display different scenarios for land cover change and changes in
   certain  ecosystem services as part of its public  product. Figure 11  also mentions
   "responsive, low variability indicators for estimating ecosystem services". EPA
   experienced a fair amount of difficulty in developing appropriate ecological
   indicators for EMAP, so this is probably a tall  order for at least some of the
   indicators. (How does one derive a meaningful, low variability indicator out of
   responses that often exhibit high variability?) Because different ecosystem services
   will require development of different indicators, this will indeed complicate the
   framework for a monitoring design (e.g., require sampling at different spatial and
   temporal scales)
                                       A-ll

-------
   The last paragraph on the page leaves the reader hanging because there is no answer
   to the obvious question of how the Program addresses the data gaps identified by
   Carpenter et al. (2006).
Page 46:
   The long-term goal monitoring component in Figure 11 (also described in the second
   paragraph on page 48 and the first paragraph on page 49) will require much future
   research.
Page 47:
   The first 4 bullets on this page all are based on best professional judgment and thus
   need some outside critical review in the process to ensure quality science.

   This and other parts of the Plan would be strengthened by adding examples showing
   how relationships between direct measures of ecosystem structure and function have
   been quantifiably linked to ecosystem services. What services have been
   demonstrated to be measurable and mapable?  This proof of concept is a crucial piece
   that is missing from the Plan.
Page 49:
   The science questions identified on this page (as well as on pages 50, 86, 87, and 111)
   are very complex.  Given the state of the science, it is unlikely that these questions
   can be completely addressed within a period of several years.

   Regarding the issue of "census vs. sample" addressed on this page, given the place-
   specific nature of ecosystem services, it is inevitable that many resources will need to
   be sampled. Ecosystem attributes such as land cover, desertification, and wetlands
   (mentioned as data gaps in the 2006 Millennium Ecosystem Assessment) are
   examples of candidates for censusing, along with any ecosystem services derived
   from land cover measures that can be derived from satellite imagery. Where a census
   is not possible, only a probability sample can yield statistically valid estimates of
   uncertainty. Probability  sampling occurs in many, but not all, of the  various national
   monitoring programs described in Olsen et al. (1999). It must be added that
   probability sampling does not rule out having sites such as Long Term Ecological
   Research Program sites, which provide extremely useful information on biological
   and ecological processes for scientists. It would indeed be useful (as the Ecological
   Research Program proposes) to take the current national monitoring programs that are
   based on probability sampling (starting with the EPA Office of Water's national
   aquatic survey indicators) and see how responses presently recorded could be used to
   develop indicators of ecosystem services for a national inventory.
                                      A-12

-------
Page 52:

   The annual performance goals presented in Table 10 are ambitious and may be
   unrealistic given that there is little current infrastructure set up to monitor services. If
   resources are limited, what will be diverted to address these goals?

Page 53, Figure 12:

   Some of the research questions listed here are management questions. Where is it
   clearly expressed that the Program will establish cause-effect relationships that can
   reliably predict effects to ecological resources to support decision making?  The Plan
   should clearly indicate how parts of the Program support the development of
   establishing cause and effect and how these relationships are used at various levels of
   the environmental management process.

Page 56:

   The community of practice for ecosystem services modeling is not adequately
   described. Who will participate? How inclusive will it be?

Page 57:

   The modeling described here is a very large challenge. The annual performance goals
   presented here for modeling are unrealistic given the general approach.  Where will
   the modelers come from?  An education plan is needed to support this goal.  An
   investment in graduate education is needed to move forward on this goal.

Page 61:

   Why does the first bullet on this page focus on fecal coliform impairment?  EPA has
   established thatE. coli is a more useful indicator.

Page 62:

   Haven't landscape metrics as indicators of Great Lakes coastal wetland quality (first
   bullet on the page) already been developed?

   More detailed information should be provided in paragraph two on this page to
   indicate how EPA will collaborate with the U.S. Geological Survey and National
   Oceanic and Atmospheric Administration.  These collaborations have been
   problematic in the past.

   The Plan mentions research teams exploring mapping techniques for different
   services.  Reference to or examples of some products from these teams would provide
   greater confidence in the feasibility of what is being proposed.
                                      A-13

-------
Page 64:
   The annual performance goals presented here for mapping are tractable. EPA has the
   expertise to accomplish them. However, it will be a challenge to obtain the data
   needed for the maps.
Page 67:
   It is good that the N example on this page illustrates both positive and negative
   effects. It is surprising that there is no mention of hormesis.
Page 69:
   The U.S. Army Corps of Engineers Institute for Water Resources would appear to be
   a natural partner in the nitrogen and ecosystem assessments.
Page 70:
   A concern here is that a net benefits approach would yield management decisions
   such as allowing fertilization of oligotrophic systems to produce stronger recreational
   or commercial fisheries.

   The outcomes section of the goal provided in Figure 15 states that economists will
   convert ecosystem response functions to monetary values where possible. Are these
   in-house economists? If not, is there funding for this research?
Page 72:
   It is difficult to tell how the ecosystem assessments will be performed. There are
   numerous references in this section of the Plan to generating value or benefit
   estimates for wetlands and coral reefs (as well as for specific demonstration projects)
   but no indication of who will do this research. In addition, it is not clear whether data
   from the place-based assessments in Long-term Goal 5 will be used for the ecosystem
   assessments. If so, will data from other studies also be incorporated? This would
   seem to be necessary, particularly for the coral reef assessment.

   Answering the question posed in the first bullet on this page (What are the current
   spatial extent and condition of ecosystems?) will require very long-term research.
   Answering the other questions on this page will also be difficult and will require
   several years to address at a minimum.
Page 74:
   Much research on wetlands and coral reefs has already occurred at the local scale.
   For wetlands, modeling strategies have been developed for the Willamette Futures
   Project and the Tampa Bay watershed. Further research should be able to use these
                                      A-14

-------
   modeling strategies to map different wetland scenarios at scales larger than simply the
   local level. For coral reefs, it appears that first "landscape characterization" will
   occur at the level of the eastern Caribbean. Though it is not a trivial effort to build a
   model linking coral reefs to human health and well-being, just communicating
   information on projected declines associated with urban development may prove
   useful.  As previously noted in this advisory report, the decision to conduct research
   on coral reefs is not well justified.
Page 75:
   The SAB report on ecological risk assessment (U.S.EPA Science Advisory Board,
   2007) addresses multi-scale research needs.
Page 76:
   The importance of wetlands on hydrological connectivity should be mentioned in the
   first paragraph on this page.
Page 77:
-  It is surprising that storm surge protection was not included as an ecosystem service
   in "Figure 16. Does that mean that salt marshes are not included in the assessment?

Page 82:

   The first bullet on this page indicates that the proposed research will determine the
   best methods (monetary and non-monetary) to value wetland services at multiple
   scales. It will be difficult to determine the best methods to value wetlands if the
   extent of the importance of wetlands is not known.

Page 84 - 85:

-  It will be important to make sure that models mentioned for valuing, assessing, and
   forecasting ecosystem services can show predictive relationships. Adequate data will
   be needed to do this. In this regard, some of the models/frameworks in EPA's
   CADDIS system are not effective.

Page 92:

-  This section has not clearly indicated how selection of places will "make the concept
   of ecosystem services districts an operational management option." The concept of
   ecosystem services  districts is not mentioned.  How did that concept shape the way
   the places were selected?
                                      A-15

-------
Page 93:
   The research questions outlined here are good and they relate to testable hypotheses.
   One concern is that the research is focused only on temperate and tropical areas. The
   U.S. also includes arctic regions, and those regions are experience considerable
   changes as a result of global climate change.
Page 95:
   It should be clearly indicated here that, with the exception of humans and endangered
   species, the focus is not on effects to individual organisms, but rather on impacts to
   populations or communities of organisms.  Thus, although biodiversity is important, it
   is not necessarily the key issue (cf. Ridder, 2008).

Page 99:

   The choice of the Willamette here makes considerable sense because much work has
   already been done  on ecosystem services in this region.  In producing the impressive
   work visualizing future scenarios for the Willamette Basin, work with landscape
   architects proved particularly valuable.  Collaboration with this group should be
   explored.

Page 105:

-  The Midwestern landscapes and coastal Carolina components are less developed,
   which is somewhat of a concern, particularly for the Midwestern landscape since it is
   so much larger and potentially more complex than any of the other place-based
   activities. The problems being faced by coastal Carolinas are no different than are
   being faced by Georgia. Why was this project cut off at the Carolinas?  In many
   respects state protections on  coastal development are much stricter in the Carolinas
   than in Georgia, which provides considerable opportunities for useful comparisons.

Page 110, Section 6.0:

   There should probably be several layers of annual review of progress. Each ORD
   laboratory could meet at least twice during the year and review progress of internal
   research initiatives. An annual meeting of the ORD laboratories and partners to
   report research findings in symposia or workshops could promote stronger
   interactions and information exchange.

Page 111:

   Concerning interaction with organizations, a proven way for EPA and the Ecological
   Research Program to take advantage of all the ecological and other scientific
   expertise in the marketplace is to put out requests for proposals for investigator
   initiated research.  The EPA Environmental Monitoring and Assessment Program
                                      A-16

-------
   made good progress with the help of EPA STAR and other grants.  EPA should
   continue with this model of making research progress.

-  It is stated here that the Program has been developed with "less-than-usual input from
   stakeholders within the Agency." This is unfortunate because the Program has set as
   a goal decision maker acceptance of ecosystem services as a valid basis on which to
   make environmental decisions. Succeeding in this task requires input from decision
   makers as the program is being developed.

Page 117:

-  It is not possible to comment on performance measures since they have not yet been
   developed. However, as previously noted, to the extent that some of the annual
   performance goals are very ambitious, the Program runs a risk of low performance
   ratings.

PageC-1:

   Important outcomes from the previous multi-year plan are listed here for 2009 and
   beyond.  What happens to these outcomes with the new direction of the Program?

Other specific comments:

   A key issue will be delivering information to decision makers at the political level
   and ensuring  that this information is heard and appropriately acted upon.  To this end
   there is a need to develop short, effective briefing notes (similar to press releases) that
   can be delivered to Congress.

-  It is appropriate that EPA establish appropriate linkages with at least its neighbors,
   Canada (via Environment Canada) and Mexico. Further, there are similarities with
   the European Union Water Framework Directive and other similar measures that
   strongly  suggest linkages also be  established with the European Union.

   The new strategic direction is good in that it is less fragmented and more holistic. It
   recognizes the reality that human beings need to take responsibility for changes they
   are making to the environment and specifically determine what changes should occur
   and what should not (cf Chapman, 2007).

   The Plan lacks a clear discussion  of what will be done with monitoring data.  There is
   a need to identify specific questions to be answered and the specifications of how the
   data are to be collected. In this regard power calculations are needed.  This should be
   part of the more detailed implementation plan.

-  Time and space remain among the most difficult features of a system to analyze
   because of the lack of independence of each factor. Bayesian tools can be used for
                                      A-17

-------
dealing with spatial relationships. It is not clear that the Plan sets the stage for the
decadal long sampling programs that will be necessary for the Program.

The specific strategy to build conceptual models that are clearly causal should be
included in implementation plans. At  this point it is not clear how these models will
be built, tested, and applied. Oreskes et al. (1994) should be consulted for useful
information on this subject.
                                    A-18

-------