State of the Lakes Ecosystem
Conference
Peer Review Report
October 7th and 8th, 2003
Toronto, Ontario
-------
This page intentionally left blank
-------
SOLEC Peer Review Report
Acknowledgements
This report was prepared under the direction of Paul Horvatin and Paul Bertram of the United
States Environmental Protection Agency (US EPA) Great Lakes National Program Office
(GLNPO) and State of the Lakes Ecosystem Conference (SOLEC) Executive Committee,
Harvey Shear of Environment Canada and SOLEC Executive Committee, and Louis Blume,
GLNPO Work Assignment Manager. The report was prepared by Judy Schofield of Computer
Sciences Corporation (CSC) based on the panel member comments from the 2-day peer review
meeting. GLNPO and Environment Canada thank Christina Forst of GLNPO for contributing to
the report's introduction and Attachments 1 and 2 and Molly Middlebrook of CSC for assisting
in capturing and compiling the comments during the meeting. GLNPO and Environment Canada
specifically thank Peter Hardi, peer review team leader, for assisting with design of the report,
editing the draft report, and incorporating panel member reviews.
GLNPO and Environment Canada especially thank the following peer review panelists for their
time and effort throughout the peer review process, for their objective evaluations of SOLEC,
and for providing comments on the report:
* Ronald Colman of the Genuine Progress Index for Atlantic Canada,
> Peter Hardi of the International Institute for Sustainable Development,
> Hans Herrmann of the Commission for Environmental Cooperation,
> Robin O'Malley of the H. John Heinz III Center for Science,
> William E. Rees of the University of British Columbia,
> Risa B. Smith of Environment Canada, and
> Ben ten Brink of the Netherlands Environmental Assessment Agency.
November 2003
-------
SOLEC Peer Review Report
November 2003
-------
SOLEC Peer Review Report
Table of Contents
ACKNOWLEDGEMENTS I
TABLE OF CONTENTS Ill
SECTION 1.0 INTRODUCTION 1
SECTION 2.0 SOLEC PROCESS 2
Section 2.1 Biennial cycle 3
Section 2.1.1 Effectiveness 3
Section 2.1.2 Efficiency 3
Section 2.2 Indicator selection and development process 4
Section 2.2.1 How well does this process work to identify a suite of Great Lakes
indicators? 5
Section 2.2.2 Does the peer reviewer have advice to improve the indicator
selection/development process? 6
SECTION 3.0 SOLEC PRODUCTS 7
Section 3.1 Are the biennial conferences an effective and efficient venue for information
exchange? 9
Section 3.2 One of the stated objectives of SOLEC is to help managers contribute to
policy-making. Are the products likely to achieve this goal? 9
SECTION 4.0 ADVICE BASED ON THE PANEL'S OWN EXPERIENCE 10
SECTION 5.0 GENERAL COMMENTS 11
ATTACHMENT 1 PEER REVIEW PANELIST BIOGRAPHIES
ATTACHMENT 2 TECHNICAL CHARGE TO PEER REVIEWERS
November 2003
-------
SOLEC Peer Review Report
iv November 2003
-------
SOLEC Peer Review Report
Section 1.0 Introduction
In 1992, the governments of Canada and the United States established the State of the Lakes
Ecosystems Conferences (SOLEC) to fulfill the requirements of the Great Lakes Water Quality
Agreement (GLWQA), which called for coordinated, consistent and science-based reporting on
the state of the health of the Great Lakes basin ecosystem every two years. Environment Canada
and the United States Environmental Protection Agency (US EPA) are the lead agencies
supporting the SOLEC initiative and they work with many partner agencies and organizations to
successfully report on multiple components of the Great Lakes basin ecosystem.
SOLEC has evolved into one of the primary venues by which the governments of Canada and the
U.S. report progress on attaining the goals of the GLWQA. The SOLEC approach is comprised
of two main elements: the biennial conferences and the subsequent State of the Great Lakes
reports that are based on ecosystem health indicators. The biennial conferences provide a forum
for Great Lakes stakeholders to review the initial assessments provided by indicator reports, to
discuss their management implications, and to provide any additional information or
interpretation on the indicators. The State of the Great Lakes reports provide an overall
assessment of the health of the Great Lakes based on individual indicators.
In the interest of improving SOLEC, Environment Canada and US EPA are implementing two
peer reviews. The first peer review involves a panel of experts in indicator development and
implementation (see Attachment 1 for biographies of the panel). The overall goal of the peer
review is to enhance the quality and credibility of the SOLEC work products so that any future
decisions or positions have a sound, credible basis. A second SOLEC review, or review
workshop to assess the entire suite of SOLEC indicators, is planned for January 2004, for
managers and stakeholders within the Great Lakes basin. Participants will evaluate the
indicators' utility, success, and effectiveness in influencing decision makers, and make
recommendations for improvements. This document summarizes the findings from the first peer
review.
The first peer review meeting was conducted from October 7-8, 2003 at Environment Canada, in
Toronto, Ontario. The peer review panel was comprised of the following experts:
> Ronald Colman, Genuine Progress Index for Atlantic Canada
* Peter Hardi, International Institute for Sustainable Development
> Hans Herrmann, Commission for Environmental Cooperation
> Robin O'Malley, H. John Heinz III Center for Science
> William E. Rees, University of British Columbia
* Risa B. Smith, Environment Canada
> Ben ten Brink, Netherlands Environmental Assessment Agency
The peer review panel was asked to provide objective evaluations of the SOLEC process,
approach, and efficiency, based on comparisons to other national and international indicator
reporting systems, as well as to evaluate the products of SOLEC. US EPA contractor staff
facilitated the meeting.
November 2003
-------
SOLEC Peer Review Report
Prior to the meeting, each peer review panelist received the following documents on a CD-ROM:
> three journal articles,
> Terms of Reference for the peer review,
> State of the Great Lakes 1995: Standard Report,
> State of the Great Lakes 1997: Standard Report.,
> State of the Great Lakes: Standard Report,
> State of the Great Lakes 2001: Standard Report,
> State of the Great Lakes 2003: Standard Report,
> Implementing Indicators 2003: A Technical Report,
> Selection of Indicators for Great Lakes Basin Ecosystem Health,
> The ABCs of Indicators, and
> the SOLEC 2002 presentations.
At the meeting, panelists received a copy of the Peer Review of the State of the Lakes Ecosystem
Conferences Briefing Binder, containing:
* the meeting agenda,
* the peer review technical charge,
> a list of the purpose and objectives of SOLEC,
> a copy of the presentations being given at the peer review,
> an abridged version of the report State of the Great Lakes 2003,
> the peer review panelist biographies, and
> peer review participant contact information.
The meeting began with extensive presentations by the SOLEC Executive Committee describing
SOLEC's history, objectives, biennial cycle, conferences, products (e.g., three types of reports),
partnership process, audience, indicator selection and development, and societal indicators.
During the presentations, an open question and answer forum was encouraged to promote
discussions and to ensure the panelists had a complete understanding of SOLEC.
The SOLEC Executive Committee requested that reviewers specifically consider and address the
topics and questions provided in the peer review technical charge (Attachment 2). Sections 2.0
to 4.0 of this document contain a detailed summary of the panel's comments on the charge.
Finally, Section 5.0 discusses any comments provided by the panel that did not address the
questions in the charge.
Section 2.0 SOLEC process
The reviewers' comments on the effectiveness and efficiency of SOLEC are based on their
impressions and the information provided by the SOLEC Executive Committee and not on a
scientific survey or other systematic means of assessment.
November 2003
-------
SOLEC Peer Review Report
Section 2.1 Biennial cycle
The panel members agreed that the biennial cycle is adequate and important to SOLEC's
success. A five-year cycle would be too long. One panelist suggested that SOLEC might be
more effective if it employed separate cycles for different tasks. He noted that it is worthwhile
for SOLEC to be constantly reporting on the full suite of indicators on a regular cycle, even if the
assessments have not changed. "No change" may also be an important feedback to managers
and decision makers.
Section 2.1.1 Effectiveness
The panel members agreed that the SOLEC conference has great impact. The conference
provides an extensive and very successful networking forum for a multitude of stakeholders. If
mayors and other stakeholders, who are not typically in attendance at such events, participate in
the conference, then SOLEC is a model of public debate and is very effective. The panelists
emphasized how important it was that SOLEC expanded the scope of Great Lakes management
focuses from only chemical toxins in the early 90's to a variety of important and complex issues,
such as land use and invasive species. The State of the Lakes Ecosystem Conference is a regular
event on the calendars of many Great Lakes stakeholders.
All panelists felt that, from the outreach perspective, SOLEC is unique and effective, but noted
that the SOLEC process and products are the areas that could use improvements. They noted
that earlier reports were too narrowly focused. One panelist stated that SOLEC is one of the
most important indicator projects around.
A few panelists suggested that the reports would be more effective if they provided
recommendations on environmental management and if environmental managers and policy
makers would provide responses to the SOLEC reports. At least one panelist strongly disagreed,
noting that adding a major emphasis on management recommendations, without appropriate
safeguards and "firewalls" between SOLEC as an indicator and reporting effort and the making
of recommendations, is a potentially dangerous direction for SOLEC to take. This is discussed
further in Section 3.0. One panelist suggested that in order to judge the effectiveness of the
SOLEC conference, a questionnaire could be circulated to attendees and the answers could be
evaluated. One panelist mentioned the need to measure the SOLEC indicators against some
form of desired future state, such as targets or management objectives, and this would improve
the ability to assess effectiveness. SOLEC has begun to do this in some instances and should
continue these efforts.
Section 2.1.2 Efficiency
All panelists agreed that SOLEC is very cost-efficient, given that the report, conference, and
other SOLEC products, are developed by only a few individuals. They noted that SOLEC is
very efficient, due to the unique cooperation between the United States and Canada. In terms of
return on investment, SOLEC is highly efficient due to the collaboration with other agencies,
academia, and monitoring groups. The partnership with monitoring groups, especially, is a very
efficient component of the process. The panelists suggested that in order to judge SOLEC's
November 2003
-------
SOLEC Peer Review Report
efficiency, the labor of all SOLEC report authors, monitoring groups, and other contributors
must be accounted for. One panelist cautioned that the monitoring groups might be less likely to
contribute to SOLEC if a larger group administered SOLEC. Several panelists suggested that
additional funds could be used to do further analysis, improve data quality, frequency of data
collection, and data management. The reviewers highlighted that although SOLEC is obviously
run very efficiently, and both governments receive an excellent return on their investments,
additional funds could be used not just to improve data quality and management but also
communication products and the development of indicators. A specific example of a need for
improvement is in the easy access to "drill down" data.
Section 2.2 Indicator selection and development process
The panelists agreed that SOLEC has done a remarkable job in developing the current suite of
indicators. At the same time, the panel had many specific suggestions for improving the process.
Many panelists commented on the state-pressure-activities (SPA) model that is used for indicator
development by SOLEC. They noted that there is a worldwide trend to avoid the SPA model
because it is linear and does not allow linking multiple causes and multiple effects. The World
Health Organization, for example, changed to the multiple effects model (Gibson model with
driver) because it allows for linkages among the SPA indicators. One panelist suggested that the
SPA indicators as they are right now do not show the relationship among them, making the
analysis of the overall system more difficult; the model could be more effective by directly
linking the state indicators with the pressure and response indicators.
One panelist noted that it might be unwise for SOLEC to change in midstream, because the SPA
model is well known and respected; another panelist cautioned that the model can still be useful
if it is not viewed and used in the strict linear sense noted above. It may be useful to add a fourth
dimension, focusing on "Uses (U)." Additionally, the Organization for Economic Co-operation
and Development (OECD) is committed to a version of the SPA model, the "Driver-Pressure-
State-Impact-Response" (DPSIR) model. That model has the advantage over standard SPA
models of focusing on whether policy response (R) does or does not effectively influence the
driving force (D) that underlies and moves environmental impacts. Yet it was noted that SOLEC
should be keeping an eye on developments in this area to ensure that it is not out of step with
directions in other indicator reports.
One panelist believes that in an ideal world, the indicator development process should be based
on modeling; however, the costs associated with this effort are prohibitive. Another panelist
noted that still the best way to expand the work is to link the SPA and U indicators through
quantitative modeling. It would also allow linking of socio-economic scenarios to the state of
the ecosystem and considerably increase the usefulness of indicators as policy-making tools.
The panel discussed the indicator development process in the context of the human component.
They all agreed that SOLEC should integrate the lake ecosystem and human activities to
understand the overall system. This point was raised repeatedly by the panel in the context of
many topics that were discussed throughout the meeting.
November 2003
-------
SOLEC Peer Review Report
The panelists agreed that SOLEC should reduce the number of indicators. Some of the
indicators are unnecessary and redundant. SOLEC should determine which indicators are
missing and which are over-represented. The panelists all agreed that it is difficult to narrow the
number of indicators, but it is important: having too many indicators results in diluting the more
important indicators. One panelist noted that, in reading the 2003 report, six significant issues
emerged: contaminants, invasive species, fisheries, lake levels, eutrophication and nutrients, and
loss of habitat. Another panelist suggested to add a seventh issue: the extent of remaining
ecosystem types and quality. The panelist suggested that the full suite of indicators could be
collapsed to these six "headline" issues, thus greatly simplifying the lake assessments and
reporting; a hierarchical approach was suggested, in which a short set of headline indicators
would be complemented with a larger set of supporting measures. Another panelist
recommended clustering the indicators into 2 categories: pollution and resources, using
freshwater quality and waste generation for the pollution category, and freshwater resources, fish
resources, biodiversity (habitat and species) for the second category. The panelists noted that
indicator selection was currently driven more by which agencies or experts were already
collecting data rather than by prioritizing indicators according to importance or significance.
The panelists all agreed that to adequately affect decision makers, SOLEC must decide what are
the most important indicators and report on those. They suggested that perhaps SOLEC could
collapse and bundle several indicators into one. The use of a small number of themes (such as
the six or seven identified above) would be very useful in improving the readability and
accessibility of the written product as well.
The panelists noted that there are two levels/layers of indicators applying to the two major
audience groups: policy makers and environmental managers. Environmental managers would
still need specific indicator information at a smaller scale, as opposed to being aggregated, while
policy makers need a more aggregated and smaller suite of indicators.
The SOLEC report presents what at least one panelist would consider an "assessment approach"
as opposed to an "indicator approach." The latter has a discrete well-defined set of indicators
and these are monitored over time using standard methods. An assessment approach has a set of
endpoints that are of interest, with a process for obtaining data and information about the
endpoints and making judgments about them. SOLEC takes its set of endpoints and every two
years, finds pertinent experts to gather available information for them, make judgments about
them, and report on their state. The reports include detailed narratives of what the author knows
about a particular subject and then makes a judgment about it.
Section 2.2.1 How well does this process work to identify a suite of
Great Lakes indicators?
All panelists felt that the assessment of indicators is not transparent, not standardized and is too
subjective. They noted that SOLEC has accomplished important tasks, but it is time to make it
more credible and internally consistent. The science must be more rigorous, transparent, and
focused. Additionally, the indicator assessments are confusing for the SOLEC audience. For
example, if an indicator is listed as "good" one year and as "mixed improving" the next year, the
November 2003
-------
SOLEC Peer Review Report
category may not reflect deterioration to the majority of the audience. SOLEC should re-
evaluate the categories of indicator assessment.
All of the panelists noted that SOLEC should establish indicators that more accurately represent
ecosystem characteristics of interest. They provided specific examples of inappropriate
indicators and suggestions for more appropriate indicators. Most of the issues concerned the
societal indicators and the lack of relationship with the suite of ecosystem indicators. Several
panelists noted that the SOLEC development process is missing the human component.
Section 2.2.2 Does the peer reviewer have advice to improve the
indicator selection/development process?
The panel members recognized that the management of SOLEC is a developing process, but all
agreed that the biggest shortcoming in the SOLEC process is the lack of standard methodology
and harmonization among indicators over the years. SOLEC must establish standard protocols to
improve data comparability and reliability. SOLEC currently trusts the integrity of the data and
the assessment process from a multitude of monitoring agencies. All panelists agreed that some
type of data verification should be instituted. One option suggested by the panel is to review the
quality control (QC) protocols that are being used by each agency, instead of reviewing each data
set. Another more rigorous approach is that SOLEC could review the procedures that were used
to collect data and confirm that the QC procedures were followed.
The panel raised the concern that the SOLEC process has serious flaws regarding lack of
repeatability and transparency. SOLEC needs to include much more detail regarding how the
assessments of the indicators were done. This is especially true given that the present assessment
of the indicators is subjective. The reports should clarify how and why the indicators were
selected and should provide more justification of the assessment.
One panelist suggested that in order to be effective, the indicators must be defined by the actual
users, and that policy makers and environmental managers need to be involved in the early stages
of indicator development. SOLEC's indicator development process is too weighted by expert
opinion. There is a disconnect between the development of the indicators and the usefulness to
the policy makers. The indicator assessments should be presented in terms of baselines that are
consistent across indicators and relevant in regards to the ecosystem and policy. The baselines
should be realistic in terms of the ecosystem characteristics and also should consider
environmental policy; they would help establish policy targets and link the indicators to policy
objectives. It was also noted that there is currently no systematic monitoring of the degree to
which the SOLEC indicator process impacts management decisions and effects change. This is
essential in order to bridge the current gap between the excellent work of the scientists, experts,
and SOLEC core group on the one hand, and the management, policy actors on the other.
One panelist in particular (generally supported by others) felt that SOLEC was uniquely
positioned to identify the "load" imposed by human activities taking place within the Great
Lakes basin (particularly economic production/consumption) on other regions outside the basin
and on the global commons. This panelist argued that at least some of the stable or improving
environmental quality of the Great Lakes could be explained by the facts that many polluting
November 2003
-------
SOLEC Peer Review Report
industries have migrated to other areas of the world (which now must bear the pollution burden),
that the region dumps much of its waste into the global commons (e.g., the atmosphere and
oceans) outside the basin, and that many of the resources and commodities consumed in the
region are produced in "distant elsewheres." In short, the high quality of life enjoyed by many in
the Great Lakes basin is supported, in part, by carrying capacity imported from outside the basin.
This means that the study region grows and develops partially at the expense of environmental
quality in the rest of the world. The panelist noted that the extent of this "impact off-loading"
could be illustrated through ecological footprint analysis (EFA) and that developing such an
index would help raise to popular and political consciousness the Great Lakes basin's
responsibilities as a regional "global citizen," to tread lightly on the planet.
A second panelist cautioned that developing a regional "ecological footprint," while potentially
useful, would be a difficult process and that the results would be politically sensitive. Yet
another panelist agreed that SOLEC should address human economic activities in some way if
only to demonstrate how people have to tread more lightly on the Great Lakes system itself.
Panelists noted that these points are being missed because the SOLEC process involves scientific
experts and one cannot get an over-arching perspective from a group of experts where each
individual is responsible for assessment of specific indicators. One panelist noted that such a
significant policy change as addressing human impacts in this holistic sense might be dangerous
for SOLEC. The panelists agreed that the inclusion of specific indicators in the SOLEC process
could stimulate the environmental agenda; however, too many agendas could collapse the entire
process.
The panelists all agreed that socio-economic indicators are worth including in the suite of
indicators being developed by SOLEC because they address human activities, as long as they
relate with the suite of ecosystem indicators. The biggest challenge with societal indicators is
obtaining pertinent and consistent data across the Great Lakes communities and developing
meaningful indicators that influence people. The societal indicators are too scattered,
incomplete, unstructured and not as rigorous as the ecosystem indicators; they do not show a
cause-effect relation with the ecosystem indicators. The panel discussed the fact that for societal
indicators, a better evaluation needs to be made regarding what types of data should be collected
and tracked to obtain a more meaningful assessment.
Section 3.0 SOLEC products
The panel members agreed that the SOLEC 2003 Standard Report is a very good report. Most of
the panelists agreed that there were only a small number of relevant issues. Many of the
indicators in the report can be thought of as "signals" supporting the relevant issues. One
panelist suggested that SOLEC should revise the reports to include five pages up front detailing
these issues with representative indicators. The rest of the report would discuss the full suite of
signals.
The panel members agreed that, in the reports, SOLEC should clarify how the authors made their
judgments (mixed, mixed-declining, etc.). It should specify how the various data components
were weighed in each assessment and clarify the basis for the judgments. The background for
the derivation of the calculations also should be detailed in the reports.
November 2003
-------
SOLEC Peer Review Report
The report should more clearly state who is making the assessments. One panelist suggested that
in order to improve readability, SOLEC should impose "meta-metrics" by collapsing many
graphs of data on separate species into one graph that shows if the overall population of the
species group (such as several frog species) is declining or increasing. Further, there are
different formats and metrics being used for different lakes and this requires extra work in
interpretation. The metrics need to be harmonized over the entire process.
Several panelists agreed that the reports should include environmental management
recommendations, while at least one of them strongly disagreed. Those in support noted that the
management issues should to be organized and presented in much more detail in order to be
effective. The management challenge structure that occurs at the front of the report is a good
format for the beginning of the document. The groupings of indicators should follow this
structure. The final chapter could include management recommendations and responses. The
panelists who disagreed strongly recommended that there be a clearly demarcated distinction
between the management recommendations element of the report and the indicators and
reporting elements. A management report could be a valuable product especially if it is
developed by a group that does not have a scientific or political agenda. One panelist added that
if this were done - in a transparent, open, accessible, and inclusive manner, it could be an
important (national, maybe even global) model for regional decision-making.
One panelist suggested that SOLEC could request several independent scientists to develop a
companion document that provides analysis, interpretation, and policy recommendations that
may be beyond the scope of an internal government process. Such a report could be the product
of a SOLEC workshop, isolating the government agencies from any potential conflicts.
The panel noted that the SOLEC reports discuss effects and do not discuss what is driving the
effects, and therefore, the reports are directing management responses at indicators and not at the
causes. One panelist, however, expressed serious caution in pointing out causes. If SOLEC
intends to point out causes, it must be on very firm ground and be disciplined in making the links
between states and drivers and making judgments about causes. One panelist made a reference
to the earlier comment on OECD's DPSIR model, which provides a framework that is: (1) linked
to the existing pressure-state model but goes beyond it by adding the "driving force" component;
and (2) places judgments about "causes" within an accepted conceptual framework and thus
makes them potentially more palatable to a management and policy audience.
Another panelist thought that identifying causes is exactly what needs to happen because
controversy provides visibility and may precipitate change. This panelist felt that if SOLEC
does not point out the causes, the products are much less useful. Another suggestion was that it
would be worthwhile to establish a management response to the report. Many of the
environmental managers reading the reports will believe the management challenges are not in
their purview. For example, few of the managers will deal with an issue such as land use.
One panelist questioned the utility of the fact sheets. They tend to focus on one isolated issue,
which is difficult when the science being described is based on an ecosystem approach.
Managers can be misled that there is only one issue to address. The panelist suggested an option
November 2003
-------
SOLEC Peer Review Report
in which SOLEC could develop short trend reports that provide a broader picture. Another
suggestion was that SOLEC could produce two products: one technical report for environmental
managers and scientists, and one less technical report for decision makers. Policy makers tend
to want a reduced report, however, there is also an opportunity to educate the policy makers and
environmental managers. SOLEC should not lose the technical aspects of the process and it is
important to educate the policy makers. Fact sheets often lose both capabilities. One panelist
noted that the reporting function could take a dual track approach: Reporting on current state of
affairs with diagnoses and policy evaluation; and presenting scenario calculations and policy
outlooks.
All panelists agreed that SOLEC should put everything on the Internet. The web site should
contain levels or hierarchies of information. The site can have condensed and summary
information, but include the ability to drill down to more detailed levels and the specific data.
Section 3.1 Are the biennial conferences an effective and efficient
venue for information exchange?
This question is addressed in Sections 2.2.1 and 2.2.2.
Section 3.2 One of the stated objectives of SOLEC is to help
managers contribute to policy-making. Are the
products likely to achieve this goal?
The panel felt that the present structure of reports is not helpful if the ultimate goal is policy-
making. They suggested that the report respond to questions raised by environmental managers,
including: "What should we do to improve the state of the environment? How much will it
cost?" A model that includes the drivers provides a more direct connection to policy. It answers
the questions: What is changing? Why? and What can we do about it? There should be clear
distinctions between these types of data/indicators.
All panelists agreed that the report does not present the ecosystem as a whole. Ecosystem health
or ecosystem quality cannot be measured by one single indicator and not from one single
standpoint/assessment principle (such as economic, ecological or esthetic); this means that the
report needs a small, representative set of indicators for each point of view. In this report the
assessment principles for each indicator are often implicit, hybrid and differ by indicators,
making the interpretation of what is good or bad less transparent and the aggregation to an
overall judgment difficult. The next step for SOLEC is to provide the holistic understanding of
the ecosystem and enable managers to make recommendations that address the environmental
issues raised by the indicator evaluation. This will complete the loop from environmental
assessment to practical management decisions. One panelist suggested that SOLEC should
employ multiple baselines to make the reporting more neutral. If multiple baselines are used the
report will be better received by the variety of stakeholders. Similarly, SOLEC could describe
several management scenarios instead of setting specific targets. One panelist strongly
suggested that the SOLEC reports include separate sections for assessment and
recommendations. Another suggestion was that the conference could include a session where
experts make environmental management recommendations based on the indicator assessments.
November 2003
-------
SOLEC Peer Review Report
All panelists agreed that the growth of the human component of the basin ecosystem must be
addressed. In this light, one panelist argued that industrial society's policy preoccupation with
material growth was in conflict with a basic principle of complex systems science, namely that
we could maximize only one variable at a time. He further observed that in a stable non-human
ecosystem, all ecosystem compartments normally operate sub-optimally (under the influence of
negative feedback) which ensures the integrity of the whole. By contrast, human beings strive to
maximize their material "appropriations" (energy and resources) from the ecosystems that
support them. Human society is driven by positive feedback and, in general, this means that as
the human compartment expands, other local ecosystems compartments (species, bio-
communities) necessarily contract. (There may be an exception if the human population is
supporting itself largely on imported resources [see Section 2.2.2], in which case many of the
negative impacts of economic growth are exported.) In any event, the general principle stands:
attempting to maximize human performance (e.g., incomes, economic growth) will generally
result in the depletion and possible destruction of other essential compartments of supporting
ecosystems.
Another panelist did not view the process as a zero-sum game, but agreed on the importance of
putting more emphasis on the impact of human actions. SOLEC is missing the human drivers
and demographics (human ecosystem) affecting the basin.
Section 4.0 Advice based on the panel's own experience
The entire panel felt that SOLEC provides an extraordinarily important contribution. This is
especially true because SOLEC has done this for several cycles. As a result, there are now
greater expectations of SOLEC's future contributions. It is likely that SOLEC will be asked to
do many additional tasks. The panel cautioned that the Executive Committee must bear in mind
that SOLEC has its limitations and will have to refuse some of these requests in order to stay
focused on its objectives. One panelist felt that SOLEC should be an independent body to
maintain validity. It can still be funded by the government, but needs to be independent from the
governments.
The key thing that SOLEC needs to accomplish is closing the loop at the end of the biennial
cycle to include environmental management recommendations. SOLEC should evaluate what
types of information it needs to develop to provide groups with the material needed to affect
change.
The panel compared SOLEC to several similar initiatives including: The Living Planet, Global
Environment Outlook, the Georgia Basin Puget Sound Ecosystem Indicators initiative, and the
McKenzie River Basin initiative. They discussed how these initiatives are similar, how they are
different, and pointed out components of the initiatives that may be of interest to SOLEC.
The panel noted that in the State of the Great Lakes report, SOLEC should state where the data
are coming from and list the monitoring agencies. SOLEC also should detail the gaps in the
information and identify areas that need additional funds in order to fill those gaps. SOLEC also
should identify priorities for financial investments. One panelist suggested that SOLEC should
10 November 2003
-------
SOLEC Peer Review Report
go one step further and assist the monitoring groups in maintaining their funds. This can be done
by identifying the monitoring agencies that are supporting SOLEC and describing the effect of
losing each group.
The importance of environmental non-government organizations (NGOs) is of critical
importance for SOLEC; a panelist suggested that through the Commission for Environmental
Cooperation (CEC) a dialog with NGOs could be started and eventually that would build a more
inclusive partnership.
Section 5.0 General comments
Each member of the panel felt that SOLEC is an outstanding group of people with limited
resources that are developing outstanding products. They noted that it is rare to see a team with
this kind of integrity, cooperation, and friendship. They concluded that SOLEC is not the world
leader in indicator development, but it is a world leader. In particular, it is a leader in the
consultation process, which is one of SOLEC's greatest strengths.
SOLEC is at the stage of its development where database management is critical. The data need
to be publicly accessible and user-friendly so that users can see how their community compares
to the Great Lakes as a whole. SOLEC should maintain a centralized data repository. SOLEC
could be linked to the Canadian Information System on the Environment (CISE). A strong link
to CISE would enhance the data accessibility component of SOLEC.
SOLEC now has a visible profile. If it includes a specific indicator as necessary for the suite,
then this will highlight the need for specific data. If more money is spent now on environmental
monitoring and expanding SOLEC, the governments can save money by not making the wrong
environmental management decisions.
The panelists were most impressed with the SHEMO presentation, and felt it to be an
outstanding educational tool. One panel member suggested that SHEMO might have a greater
and deeper long-term impact than all the SOLEC reports
Finally, a comment on the review process itself:
"The group of reviewers [was found] to be extremely knowledgeable, prepared, and thoughtful,
and the EPA and Environment Canada participants to be equally knowledgeable (which one
would expect, since this is their bread-and-butter) but also quite open and receptive to our
comments (which is not always the case...)."
November 2003 11
-------
ATTACHMENT 1: PANELIST BIOGRAPHIES FOR THE
PEER REVIEW OF THE STATE OF THE LAKES
ECOSYSTEM CONFERENCE
OCTOBER 7-8,2003
TORONTO, ONTARIO
-------
RONALD COLMAN, Ph.D.
Executive Director, Genuine Progress Index for Atlantic Canada
Dr. Ronald Colman is founder and executive director of Genuine Progress Index for Atlantic
Canada (GPI Atlantic), a non-profit research group that is constructing an index of sustainable
development for Nova Scotia as a pilot project for Canada. He previously taught for 20 years at
the university level and was a researcher and speech-writer at the United Nations.
GPI Atlantic has produced environmental and natural resource accounts for forests, soils and
agriculture, fisheries and marine environment, water resources, air quality, greenhouse gases,
and solid waste management, as well as a wide range of social, economic, and health indicator
reports, and ecological footprint analyses. These reports are available on the GPI Atlantic web
site at http://www. gpiatlantic.org.
Dr. Colman sat on the sustainable development indicators steering committee of the National
Round Table on the Environment and the Economy and is editor of Reality Check: The
Canadian Review of Wellbeing.
SOLEC Peer Review Panelist Biographies 1-1 October 7 - 8,2003
-------
PETER HARDI, Ph.D.
Director and Senior Fellow, Measurement and Indicators Branch, International Institute for
Sustainable Development (USD)
Dr. Hardi is a Senior Fellow of the Measurement and Indicators strategic objective. He joined
the Institute in 1993 as Senior Fellow and Director of the Measurement and Indicators work. He
has a Masters of Science in chemistry and a Ph.D. in political science. Over the past decade, the
main focus of Dr. Hardi's research interest has been related to sustainable development
assessment and decision-making. He combines a comprehensive knowledge of the theoretical
and methodological issues of sustainable development with practical field experience in
designing and working with decision-making tools, including indicators in a variety of settings,
ranging from local communities through regions to international agencies and from highly
developed countries to developing nations and countries in transition.
Dr. Hardi has tested different frameworks and approaches, applied several innovative methods in
practice. Dr. Hardi brings valuable experience in performance evaluation to the area. He
participated in several project evaluations in Canada, in Central and Eastern Europe as well as in
Asia. Dr. Hardi is a highly experienced presenter both at academic and public fora; he has
designed organized and moderated workshops and public meetings in culturally diverse settings.
He sits on several Boards of Directors and Advisory Committees of international and Canadian
organizations dealing with sustainable development; he is a member of the editorial board of the
journal Ecological Indicators.
SOLEC Peer Review Panelist Biographies 1-2 October 7 - 8,2003
-------
HANS HERRMANN
Head, Conservation of Biodiversity, Commission for Environmental Cooperation
Mr. Herrmann is a marine ecologist with over 20 years experience in the fields of phytoplankton
ecology, primary productivity, biodiversity conservation and natural resource policy. Before
joining the Commission for Environmental Cooperation he was for 8 years the General Director
of Pronatura, a Mexican non-governmental organization devoted to the conservation of
biodiversity. Prior to that, Mr. Herrmann was the Science Director at the Scientific Research
Center of Quintana Roo (CIQRO) and responsible for the administration and management of the
Sian Ka'an Biosphere Reserve.
In Mexico, Mr. Herrmann has served in the National Advisory Councils of Protected Areas,
Sustainable Development, and Forestry. At the National Forestry Council he served as the
Chairman of International Affairs. At the international level, Hans was very active as head of
the Mesoamerican Delegation at IUCN, and as a Mesoamerican representative of the GEF Focal
Points Network.
Since 1998 he has worked at the North American Commission for Environmental Cooperation as
Head of the Biodiversity Conservation Division. He is responsible for implementing
trinational-multistakeholder marine and terrestrial biodiversity conservation projects, such as:
the North American Bird Conservation Initiative, Species of Common Conservation Concern,
North American Marine Protected Areas Network, B2B (Baja California to Bering Sea Marine
Conservation initiative), Aquatic Invasive Species, Mapping Marine and Estuarine Ecosystems
of North America and the Global Programme of Action for the Protection of the Marine
Environment from Land Based Activities.
SOLEC Peer Review Panelist Biographies 1-3 October 7 - 8,2003
-------
ROBIN O'MALLEY
Senior Fellow and Project Director, The H. John Heinz III Center for Science, Economics and
the Environment
Mr. O'Malley directs The Heinz Center's Environmental Reporting program, which recently
released The State of the Nation's Ecosystems: Measuring the Lands, Waters, and Living
Resources of the United States. Prior to joining The Heinz Center in November 1997, Mr.
O'Malley was employed at the Department of the Interior, where he led U.S. government efforts
to establish a biodiversity information network throughout the Americas. From 1993 to 1996, he
was Chief of Staff for the National Biological Survey, where he was responsible for numerous
program development, budgeting, implementation, and outreach activities. Mr. O'Malley has
also served as a Special Assistant to Interior Secretary Bruce Babbitt, Deputy Science Advisor
within the Interior Department; Associate Director for Natural Resources at the White House
Council on Environmental Quality (CEQ); senior environmental advisor to Governor Thomas H.
Kean of New Jersey, and in a variety of environmental positions involving financing of
environmental infrastructure, hazardous site remediation, and solid waste management, within
New Jersey's Department of Environmental Protection. He holds a Masters degree from
Harvard University's Kennedy School of Government and a Bachelor's degree from the State
University of New York.
SOLEC Peer Review Panelist Biographies 1-4 October 7 - 8,2003
-------
WILLIAM E. REES, Ph.D.
The University of British Columbia, School of Community and Regional Planning
William Rees received his Ph.D. in population ecology from the University of Toronto and has
taught at the University of British Columbia's School of Community and Regional Planning
(SCARP) since 1969-70. He founded SCARP's 'Environment and Resource Planning'
concentration and from 1994 to 1999 served as director of the School. Prof Rees' teaching and
research focus on the public policy and planning implications of global environmental trends and
the necessary ecological conditions for sustainable socioeconomic development. Much of this
work is in the realm of human ecology and ecological economics where Prof Rees is best known
for inventing 'ecological footprint analysis.' Dr Rees' book on the concept, Our Ecological
Footprint (co-authored with then Ph.D. student Mathis Wackernagel) was published in 1996 and
is now available in English, Chinese, French, German, Hungarian, Italian, Latvian and Spanish
with other translations underway. Prof Rees is a founding member and recent past-President of
the Canadian Society for Ecological Economics. He is also a co-investigator in the 'Global
Integrity Project,' aimed at defining the ecological and political requirements for biodiversity
preservation. Prof Rees has been invited to lecture on areas of his expertise across Canada and
the U.S., as well as in Australia, Austria, China, Finland, France, Germany, Hungary, Japan,
Mexico, the Netherlands, Norway, Indonesia, Italy, Korea, the former Soviet Union, Spain, Sri
Lanka, Sweden and the U.K. In 1997, University of British Columbia awarded William Rees a
Senior Killam Research Prize in acknowledgment of his research achievements and in 2000 The
Vancouver Sun recognized him as one of British Columbia's top "public intellectuals."
SOLEC Peer Review Panelist Biographies 1-5 October 7 - 8,2003
-------
RISA B. SMITH, Ph.D., P.Ag
Acting Director, Environmental Reporting Branch, Knowledge Integration Directorate,
Environment Canada
Risa Smith has been working on state of environment reporting and environmental indicators for
the past ten years. For the past year and a half she has been Acting Director of Environment
Canada's Environmental Reporting Branch, which includes the National Indicators and
Reporting Office and the Ecosystem Monitoring and Assessment Network. For the previous
nine years she worked in the State of Environment Reporting Office for the province of British
Columbia, first as the senior research scientist and from 1996 to 2001 as head of the program. In
her position with Environment Canada, Risa led the development of the recently released federal
report Environmental Signals 2003: Canada's National Indicators Series and is currently
leading challenging efforts to develop a Canadian Biodiversity Index and a National Indicators
and Reporting Strategy.
Risa is perhaps best known for her work in British Columbia, on the biennial reports,
Environmental Trends in British Columbia. She is currently or has been in the past, a leader,
advisor or team member on many sustainable development indicator initiatives including:
Commission for Environmental Cooperation, Children's Health and the Environment Indicators
Report; National Roundtable on Environment and Economy, Environment and Sustainable
Development Indicators Initiative; Canadian Council of Ministers of the Environment, State of
Environment Task Force; Convention on Biological Diversity Experts Working Group on
Biodiversity Indicators, Georgia Basin-Puget Sound Ecosystem Indicators; the Pacific Northwest
Environmental Indicators Working Group; Towards a Small but Powerful Set of Salmon Habitat
Indicators; and the environment section of the British Columbia Provincial Health Officer's
Annual Report.
Risa is an adjunct professor at the University of Victoria where she taught a 4th year
undergraduate course for five years entitled Canada in Transition: Ecological Challenges and
Societal Responses. Risa completed her Ph.D. in 1990 at the University of British Columbia,
Institute of Animal Resource Ecology. Upon completion of her graduate work, she headed up a
biological control program for a consortium of federal and provincial government departments
and private industry.
SOLEC Peer Review Panelist Biographies 1-6 October 7 - 8,2003
-------
BEN TEN BRINK
Netherlands Environmental Assessment Agency, National Institute for Public Health and the
Environment
Ben ten Brink worked in various Dutch ministries as policy maker in the fields of nature
conservation, water management and the environment in relation to sector policies. His special
interest is finding vehicles of communication between science and policy such as the
development of indicators, monitoring and assessments approaches. He developed new concepts
and calculation methods for quantification of the condition, trends and future impact on
biodiversity based on species abundance or pressure factors. He worked out biodiversity
indicators for the Convention on Biological Diversity (in the first and second Liaison Group on
indicators of biological diversity), the OECD, UNEP, and European Union, and made
quantitative biodiversity assessments on several spatial scales: the Netherlands, the North Sea,
Europe and the World. He is national focal point on biodiversity indicators, monitoring and
assessment. In the above context, he also participated in the first technical design workshop of
the Millennium Ecosystem Assessment in Bilthoven, the Netherlands, and is delegate of The
Netherlands in CBD and OECD meetings on indicators.
SOLEC Peer Review Panelist Biographies 1-7 October 7 - 8,2003
-------
ATTACHMENT 2: TECHNICAL CHARGE PROVIDED To
THE PANELISTS DURING THE TWO-DAY STATE OF THE
LAKES ECOSYSTEM CONFERENCE PEER REVIEW
MEETING
OCTOBER 7-8,2003
TORONTO, ONTARIO
-------
TECHNICAL CHARGE TO THE PEER REVIEWERS
STATE OF THE LAKES
ECOSYSTEM CONFERENCE (SOLEC)
Background
The State of the Lakes Ecosystems Conferences (SOLEC) were established by the governments of Canada
and the United States in 1992 to fulfill the requirements of the Great Lakes Water Quality Agreement
(GLWQA), which called for the coordinated, consistent and science-based reporting on the state of the health
of the Great Lakes basin ecosystem every two years. Environment Canada and the United States
Environmental Protection Agency are the lead agencies for the SOLEC initiative, but they work with many
partner agencies and organizations to successfully report on multiple components of the Great Lakes basin
ecosystem.
SOLEC has evolved into one of the primary venues for the governments of Canada and the U.S. to report
progress on attaining the goals of the GLWQA, and the SOLEC approach is comprised of two main elements:
the biennial conferences and the subsequent State of the Great Lakes reports based on ecosystem health
indicators. The biennial conferences provide a forum for Great Lakes stakeholders to review the initial
assessments provided by indicator reports, to discuss their management implications, and to provide any
additional information or interpretation on the indicators. The State of the Great Lakes reports provide an
overall assessment of the health of the Great Lakes based on individual indicators.
SOLEC participants generally respond favorably to the SOLEC approach and products, and while the
organizers feel that significant transfers of information have been achieved between the indicator reports,
environmental decision makers and the public, improvements could be implemented. Therefore, this initial
peer review session, October 7-8, 2003 in Toronto, Ontario, is a review of the SOLEC process and its
products. This peer review is anticipated to provide many benefits, including an overall evaluation on the
Parties' approach to science-based GLWQA reporting.
A second SOLEC review, or review workshop, is planned for January 2004 for managers and stakeholders
within the Great Lakes basin to assess the entire suite of SOLEC indicators. Participants will evaluate the
indicators' utility, success and effectiveness in influencing decision makers, and make recommendations for
improvements.
Technical Charge 2-1 October 7 - 8, 2003
-------
Peer Review Charge
The charge to this peer review panel is to provide objective evaluations of the SOLEC process, approach, and
efficiency, based on comparisons to other national and international indicator reporting systems, as well as
to evaluate the products of SOLEC. The overall goal of the peer review is to enhance the quality and
credibility of this work product so that any future decisions or positions have a sound, credible basis. The
SOLEC Executive Committee requests that reviewers specifically consider and address the topics and
questions listed below.
1) SOLEC process
Biennial Cycle
ป Effectiveness: Given the established objectives, is SOLEC effective? How does the
effectiveness of the SOLEC process compare to the peer reviewer's experiences?
ป Efficiency: Are there other ways to conduct SOLEC? Does SOLEC appear to be efficient?
How does the efficiency compare to the peer reviewer's experiences?
Indicator Selection and Development Process
* How well does this process work to identify a suite of Great Lakes indicators?
ป Does the peer reviewer have advice to improve the indicator selection/development process?
* Great Lakes indicators in general
ป Societal Indicators, in particular
2) SOLEC products
Are the products effective and efficient? How do the following products compare to other
projects that the peer reviewer has seen or is involved with?
ป Standard State of the Great Lakes Report
* Technical State of the Great Lakes Report (new)
ป Fact Sheets (new)
> Data CD
ป Website: http://www.binational.net
* Poster (NOTE: Please see the poster on display during the session as an example.)
Are the biennial conferences an effective and efficient venue for information exchange?
Influence
ป One of the stated objectives of SOLEC is to help managers contribute to policy-making. Are
the products likely to achieve this goal?
3) Advice based on the panel's own experience
Are the SOLEC process and the current suite of indicators an adequate approach?
From a global perspective, is SOLEC a leader in its field of indicator development or a
"wannabe?"
ป How does the SOLEC process compare to other large-scale indicator assessments in other
geographic areas?
Should SOLEC better incorporate indices in the reporting process and if so, how? What is the
peer reviewer's advice on using indices as a tool to group indicators?
How should SOLEC maintain long-term monitoring with numerous partners?
How should SOLEC maintain funding for the system?
Influence
* Does the peer reviewer have advice on how to become more influential in the policy arena?
* Does the peer reviewer have comments on the managers session?
Frequency (of reporting, not monitoring): Is the current biennial cycle inadequate, satisfactory or
excessive?
Technical Charge 2-2 October 7 - 8, 2003
-------
Additional and Supporting Information
The SOLEC Peer Review Briefing Book, available to each peer review panel member, contains the meeting
agenda, technical charge, SOLEC purpose and objectives, plenary presentations, State of the Great Lakes
2003 abridged report, select references, peer review panelist biographies, and peer review participant contact
information. During the meeting, the SOLEC Executive Committee will give a series of presentations that
provide background information regarding the SOLEC process and products. In addition, the SOLEC
Executive Committee will be available for questions during and after the meeting to ensure the reviewers have
the information they need in order to provide a thorough evaluation of the SOLEC process and products.
Should the reviewers feel that any additional information or referenced materials are necessary to complete
their review, they may contact the SOLEC Peer Review Coordinator, Christina Forst, listed below. Contact
information for the entire SOLEC Executive Committee can also be found in the briefing book.
Time Frame and Format for Review
Following this peer review, being held October 7-8, 2003 in Toronto, Ontario, peer reviewers should submit
an electronic-copy of comments (in Corel WordPerfectฎ or Microsoft Wordฎ) to fellow peer review
participants and team leader, Dr. Peter Hardi, of the International Institute for Sustainable Development. The
submission should include comments that address the questions raised in this charge, in addition to significant
comments discussed during the meeting. Dr. Hardi, with assistance from the CSC peer review coordinators,
will compile the peer reviewers' comments and produce a final version of the report no later than December
1,2003.
Peer Review Contacts
SOLEC Peer Review Coordinator
Christina Forst
U.S. EPA, Great Lakes National Program Office
77 West Jackson Boulevard, (G-17J)
Chicago, IL 60604
(312) 886-7472 (phone)
(312) 353-2018 (fax)
Forst. Christina(g),epa.gov
Peer Review Team Leader
Peter Hardi, Ph.D.
Director, Measurement and Indicators Program
International Institute for Sustainable Development
161 Portage Avenue East, 6th Floor
Winnipeg, Manitoba, Canada
R3B OY4
(204)958-7731 (phone)
(204) 958-7710 (fax)
phardi@,iisd.ca
Technical Charge 2-3 October 7 - 8, 2003
-------
CSC Peer Review Coordinators
Judy Schofield
Computer Sciences Corporation
6101 Stevenson Avenue
Alexandria, VA 22307
(703) 461-2027 (phone)
(703) 461-2020 (fax)
Judv.Schofield(g),DynCorp.com
Molly Middlebrook
Computer Sciences Corporation
6101 Stevenson Avenue
Alexandria, VA 22307
(703) 461-2245 (phone)
(703) 461-2020 (fax)
Molly. Middlebrook(@,DvnCorp .com
Technical Charge 2-4 October 7 - 8, 2003
------- |