September 17, 1999
EPA SAB-EC-99-018
Dr. Joan Daisey, Chair
Science Advisory Board
U.S. Environmental Protection Agency
401 M Street, SW
Washington, DC 20460
Subject: Review of the "Integrated Environmental Decision-Making in the Twenty-
First Century"
Dear Dr. Daisey:
The Integrated Risk Project Peer Review Subcommittee (TRPPRS) of the Science
Advisory Board, hereinafter referred to as the "Subcommittee", met July 1 and 2, 1999 to review
the draft "Integrated Environmental Decision-Making in the Twenty-First Century". This review
was carried out in order to help ensure that the IRP document facilitates EPA's ability to
understand and use science in context with other considerations needed to inform environmental
decision-making.
The review meeting was conducted in public session under the provisions of the Federal
Advisory Committee Act (FACA). The IRP Steering Committee provided the Subcommittee
with the report "Integrated Environmental Decision-Making in the Twenty-First Century" and its
companion document "Integrated Environmental Decision-Making in the Twenty-First Century:
Summary Recommendations". Selected members of the IRP Steering Committee, including its
Chair, Dr. Genevieve Matanoski, briefed the Subcommittee at the outset of the review. This
letter summarizes IRPPRS' key findings and recommendations. The attached report provides a
more complete description of the Subcommittee's advice.
The Subcommittee thought that the idea of pursuing a more integrated approach to
environmental decision-making is both sensible and constructive. Many of the concepts in the
SAB framework have scientific merit, and provide a good starting point for improving the way in
which EPA and other agencies charged with environmental risk management go about their
business. The report should encourage the Agency and other environmental risk managers to
pursue a more holistic and rational approach to analyzing problems and making decisions.
However, the Subcommittee notes that the report appears to be of two minds as to
whether it is recommending a single strategy, or a menu of approaches and tools that hold
promise for improved integration. Our reading is that while there are many promising ideas here
-------
that deserve research attention and experimental application, few of the concepts discussed are
ready for direct routine application by EPA and other federal agencies.
We recommend that Volume 1 be significantly revised and published as a stand-alone
document under a new title. Something like "Toward Integrated Environmental Decision-
Making", which points to a direction, but acknowledges that there is not yet a clear practical
blueprint for getting there, would be appropriate. We also recommend that Chapters 2-7 in
Volume 2 be released as a set of working papers prepared in support of the report, and that their
status be noted in a preface to the document. We suggest that Chapter 1 be removed from
Volume 2. If Chapter 1 is to be retained, it needs to be substantially rewritten to provide an
appropriate introduction to Chapters 2-7.
In reading the report, and discussing its contents with representatives of the authoring
committees, the review committee came to the conclusion that if the SAB is going to continue to
work on issues like this, that lie at the interface between science, values, and decision-making, it
needs to substantially increase its expertise in the behavioral and decision sciences. There are
large, highly relevant, literatures on many of the social science issues that the authoring
committees dealt with. In most cases the committees did not include members familiar with these
literatures and were not able to adequately draw upon available insights that could have improved
the report. Similarly, if the Agency is going to begin experimental applications and conduct
expanded research on issues of the sort discussed in this report, it will need to increase its
behavioral and decision science expertise.
The IRPPRS received comments from Barbara Harper of the Yakama Indian Nation and
Stuart Harris of the Confederated Tribes of the Umatilla Indian Reservation after it met to review
the Integrated Environmental Decision-Making report. Although we were not able to consider
their comments during our review, we have enclosed a set of their comments for consideration by
the IRP Steering Committee.
Sincerely,
/signed/
Dr. M. Granger Morgan, Chair
Integrated Risk Project Peer Review Subcommittee
Science Advisory Board
-------
NOTICE
This report has been written as part of the activities of the Science Advisory Board, a
public advisory group providing extramural scientific information and advice to the Administrator
and other officials of the Environmental Protection Agency. The Board is structured to provide
balanced, expert assessment of scientific matters related to problems facing the Agency. This
report has not been reviewed for approval by the Agency and, hence, the contents of this report
do not necessarily represent the views and policies of the Environmental Protection Agency, nor
of other agencies in the Executive Branch of the Federal government, nor does mention of trade
names or commercial products constitute a recommendation for use.
Distribution and Availability: This Science Advisory Board report is provided to the EPA
Administrator, senior Agency management, appropriate program staff, interested members of the
public, and is posted on the SAB website (www.epa.gov/sab). Information on its availability is
also provided in the SAB's monthly newsletter (Happenings at the Science Advisory Board).
Additional copies and further information are available from the SAB Staff.
-------
ABSTRACT AND EXECUTIVE SUMMARY
A Subcommittee comprised of some SAB Executive Committee members, and Board
consultants, reviewed the SAB "Integrated Environmental Decision-Making" report. To ensure
an independent peer review, EC members who had served on the IRP were not included on the
review subcommittee. The Subcommittee found the approach to be sensible and constructive.
Many of the concepts have scientific merit, and provide a good starting point for improving the
way in which EPA and other agencies charged with environmental risk management go about
their business. The report should encourage the Agency and other environmental risk managers in
the direction of a more holistic and rational approach to analyzing problems and making decisions.
However, the Subcommittee noted that the report appears to be of two minds as to whether it is
recommending a single strategy, or a menu of approaches and tools that hold promise for
improved integration. While the report contains many promising ideas that deserve research
attention and experimental application, the Subcommittee believes that few of the concepts
discussed are ready for direct routine application by EPA and other federal agencies. In most
cases such application will require: more solid theoretical and empirical foundations; better natural
and social scientific knowledge; and, Agency staff willing to and capable of applying ideas in a
critical and inventive way, since their complexity makes it unlikely that it will ever be possible to
reduce many of them to routine formulas or step-by-step instructions. Volume 1 should be
significantly revised and published as a stand-alone document under a new title which points to a
direction, but does not imply a firm strategy. The revised report needs to address the enormous
practical difficulties involved in coming to grips with the many different specific pieces of
incomplete and uncertain science that underlie the various parts that must be integrated. The
report also needs to more explicitly discuss the various objectives that underlie risk-ranking,
because even though the results from risk assessments are a useful input to decision-making, most
risk managers would not want to use them as the sole basis for setting risk management priorities.
If the SAB is going to continue to work on issues that lie at the interface between science, values
and decision-making, the peer review subcommittee believes it needs to substantially increase its
behavioral and decision science expertise. Similarly, if the Agency is going to begin experimental
applications and conduct expanded research on issues of the sort discussed in this report, it will
need to increase expertise in these areas.
-------
US ENVIRONMENTAL PROTECTION AGENCY
SCIENCE ADVISORY BOARD
INTEGRATED RISK PROJECT
PEER REVIEW SUBCOMMITTEE OF THE
EXECUTIVE COMMITTEE
CHAIR
Dr. M. Granger Morgan, Head, Department of Engineering & Public Policy, Carnegie Mellon
University, Pittsburgh, PA
MEMBERS AND CONSULTANTS
Dr. William E. Bishop, Associate Director, Global Technical Policy, The Procter & Gamble
Company, Cincinnati, OH
Dr. Donald F. Boesch, President, University of Maryland Center for Environmental Science,
Cambridge, MD
Dr. Richard J. Bull, Senior Staff Scientist, Battelle Pacific Northwest Division, Molecular
Biosciences, Richland, WA
Dr. Terry Davies, Director, Center for Risk Management, Resources for the Future, Washington,
DC
Dr. John D. Graham, Professor of Policy and Decision Sciences and Director of Harvard Center
for Risk Analysis, Harvard University, Boston, MA
Dr. Catherine Kling, Professor, Department of Economics, Iowa State University, Ames, IA
Dr. Debra Knopman, Director, Center for Innovation and the Environment, Progressive Policy
Institute, Washington, DC
Dr. Morton Lippmann, Professor, Nelson Institute of Environmental Medicine, New York
University School of Medicine, Tuxedo, NY
Dr. Warner North, President and Principal Scientist, North Works Inc., Belmont, CA
Dr. Richard Revesz, Professor of Law, New York University School of Law, New York, NY
Dr. Bruce Tonn, Senior Researcher, Oak Ridge National Laboratory, Oak Ridge, TN
Science Advisory Board Staff
Dr. John R. Fowle III, Deputy Staff Director/Designated Federal Officer, Environmental
Protection Agency, Science Advisory Board
in
-------
Ms. Wanda R. Fields, Management Assistant, US EPA/Science Advisory Board
IV
-------
TABLE OF CONTENTS
1. OVERVIEW COMMENTS 1
1.1 Introduction 1
1.2 Overview Comments and Observations 1
2.0 RESPONSE TO CHARGE 3
2.1 Charge Question 1: Does the integrated framework document as a whole
provide a useful and scientifically valid concept for the Agency to develop
processes and procedures for integrated environmental decision-making? 3
2.1.1 Recommendation-by-recommendation commentary 7
2.2 Charge Question 2: Do the chapters on ecology, health, risk reduction options,
report card, and the deliberative portion of the valuation chapter describe adequate
and useful methods for addressing/ranking risks? 10
2.2.1 Ecological Risks 10
2.2.2 Health Risks 13
2.2.3 Risk Reduction Options 15
2.2.4 Performance Evaluation 16
2.3 Charge Question 3: Does the document provide an adequate and useful description
for how this information might be linked in decision-making? 17
2.4 Charge Question 4: Does the economics chapter provide an adequate and
useful primer for economic analysis? 18
2..5 Charge Question 5: Does the chapter on valuation provide an adequate
and useful philosophy describing how to incorporate values into
decision-making, clearly articulating that more than science is needed in
the decision-making process? 20
2.6 Charge Question 6: Is the Document Clearly Written, Comprehensible
and Complete? 23
3.0 REFERENCES CITED R-l
APPENDIX A. SPECIFIC EDITORIAL COMMENTS A-l
-------
1. OVERVIEW COMMENTS
1.1 Introduction
The Integrated Risk Project Peer Review Subcommittee (TRPPRS) met July 1 and 2, 1999
to review the draft "Integrated Environmental Decision-Making in the Twenty-First
Century"written by the U.S. Environmental Protection Agency Science Advisory Board
Integrated Risk Project Steering Committee. The materials provided to the SAB for review
consisted of the draft "Integrated Environmental Decision-Making in the Twenty-First Century"
and the companion draft document "Integrated Environmental Decision-Making in the Twenty-
First Century: Summary Recommendations". The charge to the Subcommittee (Attachment A)
contained six questions focusing on the concepts and application of the white paper to facilitate
future Agency use of science in the context of other considerations needed to inform
environmental decision-making.
1.2 Overview Comments and Observations
The Subcommittee thought that the idea of pursuing a more integrated approach to
environmental decision-making is both sensible and constructive. Many of the concepts in the
SAB framework have scientific merit, and provide a good starting point for improving the way in
which EPA and other agencies charged with environmental risk management go about their
business. The report should encourage the Agency and other environmental risk managers to
pursue a more holistic and rational approach to analyzing problems and making decisions.
However, the Subcommittee notes that the report appears to be of two minds as to
whether it is recommending a single strategy, or a menu of approaches and tools that hold
promise for improved integration. It recommended that Volume 1 be significantly revised and
published as a stand-alone document under a new title. In addition, the Subcommittee felt that
the report needs to address the enormous practical difficulties involved in coming to grips with the
many different specific pieces of incomplete and uncertain science that underlie the various parts
that must be integrated. It also needs to more explicitly discuss the various objectives that
underlie risk-ranking, because even though the results from risk assessments are a useful input to
decision-making, most risk managers would not want to use them as the sole basis for setting risk
management priorities. When possible, ranking actual management options would give managers
more directly useful guidance.
In reading the report, and discussing its contents with representatives of the authoring
committees, the review committee came to the conclusion that if the SAB is going to continue to
work on issues like this, that lie at the interface between science, values and decision-making, it
needs to substantially increase its expertise in behavioral and decision science. There are large,
highly relevant, literatures on many of the social science issues that the authoring committees
dealt with. In most cases the committees did not include members familiar with these literatures
and were not able to adequately draw upon available insights that could have improved the report.
Similarly, if the Agency is going to begin experimental applications and conduct expanded
-------
research on issues of the sort discussed in this report, it will need to increase its behavioral and
decision science expertise.
-------
2.0 RESPONSE TO CHARGE
2.1 Charge Question 1: Does the integrated framework document as a whole
provide a useful and scientifically valid concept for the Agency to
develop processes and procedures for integrated environmental decision-
making?
The idea of pursuing a more integrated approach to environmental decision-making is
sensible and constructive. Many of the concepts in the SAB framework have scientific merit, and
provide a good starting point for improving the way in which EPA and other agencies charged
with environmental risk management go about their business. The report should encourage the
Agency and other environmental risk managers in the direction of a more holistic and rational
approach to analyzing problems and making decisions.
The conceptual strategy summarized in Figure 1 on page 13 of Volume 1 is very sensible.
It is reminiscent of similar guidance available in many books, reports, and papers on risk analysis
and decision-making. The arguments that the boundaries for analysis should not be drawn too
tightly, that problem formulation deserves the same careful attention as analysis and decision-
making, and that feedback and iteration are important elements of successful environmental
decision-making, are valid. EPA needs to do more such integration than it has in the past on all of
these fronts. An SAB report that reminds the Agency of this, and suggests a variety of strategies
by which it might improve, is clearly a useful contribution.
The report appears to be of two minds as to whether it is recommending a single strategy,
or a menu of approaches and tools that hold promise for improved integration. While some of the
language the report suggests the former, the table on page 4 of Volume 1, and the briefings that
the review committee received from members of the authoring committees, suggest the latter.
Our reading is that while there are many promising ideas here that deserve research attention and
experimental application, few of the concepts discussed are ready for direct routine application by
EPA and other federal agencies. In most cases such application will require: more solid
theoretical and empirical foundations; better natural and social scientific knowledge, and Agency
staff willing to and capable of applying the ideas in a critical and inventive way — since their
complexity makes it unlikely that it will ever be possible to reduce many of them to routine
formulas or step-by-step instructions.
We recommend that Volume 1 be significantly revised and published as a stand-alone
document under a new title. Something like "Toward Integrated Environmental Decision-
making", which points to a direction, but acknowledges that there is not yet a clear practical
blueprint for getting there, would be appropriate.
As explained below, the quality and status of the chapters in Volume 2 is very uneven.
Some chapters could be published as separate SAB-reviewed documents after appropriate editing
-------
and revision. Others have fundamental problems that make it unlikely that even with major
revision they will be suitable for publication as SAB approved documents. In later portions of this
review we provide chapter-by-chapter commentary.
In the short run, we recommend that Chapters 2-7 be released as a set of working papers
prepared in support of the report. If contained in a single volume, that volume should contain a
preface that clearly indicates that these are working papers prepared in support of the SAB report
on integrated environmental decision-making. It should indicate that they have not passed SAB
peer review and are provided as a set of ideas and explorations that can be used by interested
readers as a starting place and source of ideas for future experimental applications and research.
We suggest that Chapter 1 be removed from Volume 2. If Chapter 1 is to be retained, it needs to
be substantially rewritten to provide an appropriate introduction to chapters 2-7. Some of the
language of Chapter 1 and Chapter 8 from Volume 2 might be moved to Volume 1.
While the broad concepts advanced in the two volumes are not scientifically invalid, as
they are now articulated, a number go beyond what current science can support. For example, a
heavy focus on outcomes requires being able to connect a decision directly to an improvement in
health or the environment. This is not the same as saying that changing levels of emissions,
discharges, or even ambient concentrations of pollutants cannot be accurately measured. In many
cases, while science can make a good case that a management strategy will lead to improvement,
there often are not practical experiments that can be conducted to measure the improvement in
the face of all the natural variability that occurs due to other causes. The health improvements
induced by pollution control, though they are likely to be real and important, are often below the
current detection capabilities of public health and medicine.
Because many of the discussions of "processes and procedures" are abstract and diffuse,
and disconnected from many of the practical realties of Agency decision-making, including the
external constraints imposed on EPA, Agency decision makers are likely to find themselves at a
loss as to how to actually implement many of these ideas.
The report stresses the importance of adopting a more holistic approach. It talks relatively
little about the enormous practical difficulties involved in coming to grips with the many different
specific pieces of incomplete and uncertain science that underlie the various parts that must be
integrated. At the moment, even if the more general problems we discuss in this review could be
addressed, these practical analytical and data needs could stymie efforts by Agency staff to apply
these ideas.
The report needs to more explicitly discuss the various objectives that underlie risk-
ranking. If the objective of ranking is to obtain insight about relative level of concern that a group
has about a number of risks, then ranking risks is an appropriate activity. However, while the
results are a useful input to decision-making, most risk managers would not want to use them as
the sole basis for setting risk management priorities. This is because the marginal cost of reducing
different risks can be very different. It may be prohibitively expensive, or impossible, to do much
-------
about some high ranked risks. It may cost next to nothing to remove some low ranked risks. The
feasibility and cost of management action clearly must be considered in setting management
priorities. At the same time, a risk ranked high should not be ignored just because there is no
practical way to manage it today. A high rank means it should get high research priority to
identify feasible management options for the future. When possible, ranking actual management
options would give managers more directly useful guidance. There are typically many options for
each risk, one option can often impact several risks, and valuing all the costs and consequences of
options is a complex analytical task. Thus, a systematic and comprehensive ranking of options is
typically more difficult than ranking risks.
On page 24 the following paragraph needs work:
"The best strategy, by definition, is not necessarily one that reduces the worst risks to a de
minimis level, but the one that reduces the most risk with the resources available...In this respect
this methodology goes beyond the suggestions of the SAB's 1990 Reducing Risk report. While
ranking risks is important for gauging the relative magnitude of individual risks, it is more
important to focus on the reduction of total risks resulting from risk management decisions than it
is to focus on the reduction of any particular risk, per se. In short, we should target the most
first, not the worst first." This is a normative, not scientific judgment. If it is retained, the
language should make it clear that this is a value judgment (which, for example, might be read as
saying efficiency is more important than equity).
On page 5 the sentence:
"Some went beyond the scientific issues to incorporate community values" should be dropped.
Because the definition of risk is value-based, all risk-ranking involves the incorporation of values.
Thus, it is very rare that circumstances allow a strictly scientific risk-ranking. The implication that
there is pervades the document and should be changed.
More generally we suggest changes like the following:
page
7 col 2
8 coll
9 coll
Now reads:
'Scientific tools needed for integration are
secoming available..."
'Seen from a scientific perspective, integration
;an improve the environmental decision-
making process..."
'the IRP's conceptual framework begins to lay
out a scientifically -based approach for
;onsidering multifaceted environmental
jroblems..."
Better to read:
Tools needed for integration are becoming
available..."
'Better integration can improve the
environmental decision-making process..."
'the IRP's conceptual framework begins to lay
approaches for considering multifaceted
environmental problems..."
And so on.
The report, and especially Volume 1, needs to be more careful in its use of the word
"risk."
-------
There is a large psychological literature that demonstrates that in assessing how risky
some health or safety hazard is, people care about things other than the expected number of
deaths and injuries. Their assessment of riskiness is influenced by: how equitably risks are
distributed; how much control those at risk have over their exposure; how well the risk is
understood; and, by many other factors. While this literature is acknowledged in a single line in
Volume 2 which reads "we care about dimensions of risk other than sheer statistical magnitude"
(Volume 2, page 4-4, line 22), the complexities that this fact introduces into the problems of
assessing and comparing risks are not acknowledged in the discussions of Volume 1 nor in much
of Volume 2. The literature on public perceptions of ecological risks is less well developed, but
there is every reason to believe that here too, people perceive risk in multi-attribute terms.
The last paragraph on page 1-19 of Chapter 1, Volume 2, would be an excellent addition
to the Summary discussion in Volume 1, which needs more emphasis on the uncertainties that are
associated with estimates of risk. The discussion should be expanded to include greater
acknowledgment of methods for describing, analyzing, and evaluating uncertainly which have
been developed over the last several decades and have now seen wide and successful application
to environmental and other risk problems. The expanded discussion should also include
references to the very considerable progress that some portions of the Agency have made in
applying these methods in addressing EPA's problems. This point is further elaborated below.
How does one get from fragmented laws, organizations, and thinking to an integrated
approach? The report gives no answers, and minimizes the difficulties. Volume 1 argues that the
Federal approach to environmental regulation has been "piecemeal" at least in part because
"...federal laws and regulatory requirements...[are] neither systematic nor comprehensive..."
(Volume 1, page 1). In Chapter 1 of Volume 2, we read that "much of the fragmentation in
EPA's approach to the control of environmental problems has its roots in the statuary framework
that guides the Agency" (Volume 2, page 11, lines 30-31), "...in some instances, current statutes
and regulation prevent the Agency from considering all relevant risk, benefit/cost or other
information..." (Volume 2, page 12. lines 18-19).
Yet, in a summary to the end of Volume 1 we are informed that "the Agency should be
able to take...[all the various] steps [the report has proposed] without new legislative authority."
The report contains no legal analysis that documents that conclusion. Indeed, we read in Volume
2 that "SAB views the issue of statutory integration as a policy discussion and outside the bounds
of the present study" (Volume 2, page 12, lines 18-19). It is true that the Agency has argued
valiantly that statutory limitations have not impeded its various place-based and sector-based
experimental programs, or impeded a more integrated approach that focuses on the largest risks
first. However, many independent observers as well as participants in these experimental
programs, do not share this optimism. The categorical conclusion that new statutory authority is
not needed seems particularly questionable in light of the discussion of Chapter 6, which calls on
the Agency to systematically consider a wide range of basic alternative regulatory options
(ranging from information strategies to market-based approaches) before choosing how to
proceed in a given environmental decision-making task.
-------
We recommend that, particularly in Volume 1, the report more directly acknowledge and
discuss the many legal, institutional and cultural obstacles to greater integration in environmental
risk analysis and management, and note the consequent necessity for the Agency to take an
incremental approach to adopting the recommendations made in the report.
Finally, in reading the report, and discussing its contents with representatives of the
authoring committees, the review committee came to the conclusion that if the SAB is going to
continue to work on issues like this, that lie at the interface between science, values and decision-
making, it needs to substantially increase its expertise in behavioral and decision science. There
are large, highly relevant, literatures on many of the social science issues that the authoring
committees dealt with. In most cases the committees did not include members familiar with these
literatures and were not able to adequately draw upon available insights that could have improved
the report. Similarly, if the Agency is going to begin experimental applications and conduct
expanded research on issues of the sort discussed in this report, it will need to increase its social
science expertise.
2.1.1 Recommendation-by-recommendation commentary
Recommendation 1. EPA should accelerate the transition to integrated, outcome-based
environmental protection, and apply an integrated environmental decision-making
framework in selected cases while maintaining the safeguards afforded by the
current system
While desirable in principle, an outcomes-based strategy is not feasible for many health
and environmental risks. Many of the tools for improved integration need further
development. It is likely that different strategies will be needed for different problems.
Thus, this recommendation might better read: "EPA should develop and apply integrated
methods in support of environmental decision-making." The discussion of past Agency
practice and initiatives on page 15 of Volume 1 is overly harsh, and fails to credit the
Agency for a number of important initiatives to improve integration. The first paragraph
should be dropped. The paragraph that begins "In the next generation..." should be
reworked in light of the limited practicality of outcomes-based approaches. On page 17
the sentence "The Agency's agenda seems still to reflect a narrow interpretation of what
respective environmental laws require it to do rather than a practical interpretation of what
the laws allow it to do" understates the efforts that the Agency has already made, and
minimizes the constraints imposed by authorizing statutes.
Recommendation 2. Because science plays a critical role in protecting the environment,
EPA should commit the resources necessary to expand the scientific foundation for
integrated decision-making and outcomes-based environmental management.
The recommendation is appropriate if the phrase "outcomes based" is removed. On page
18 replace "hard science" with "natural science". On page 19 "...can realistically expect to
-------
get the answer we need" should read "...can realistically expect to get many of the answers
we need." We will never completely eliminate uncertainly.
Recommendation 3. EPA should apply and encourage the broader use of risk comparison
methodologies, such as those described in this document, that clearly identify how
scientific information and judgment are incorporated into risk comparisons
The recommendation is appropriate if the phrase "such as those described in this
document" is removed. In the first paragraph modify the sentence that reads "...provides
the essential underpinning which makes objective risk comparisons possible." to read
"...provides an essential underpinning which makes risk comparisons possible." Any risk
comparison that involves more than an attribute-by-attribute comparison necessarily
contains a normative element. The section should be edited to either remove the
references to the specific ideas developed in Volume 2, or more clearly indicate their
untried nature. Specific claims such as "The methodology is capable of..." (page 21)
should be removed. One strategy would be to remove text beginning with the paragraph
that reads " As part of this project..." on page 20 though the end of the first new
paragraph on page 22. The last two paragraphs might then be rewritten to read "The sub-
committees developed two prototype methods that illustrate how science, expert
judgments, and values, can lead to a rating and comparison of environmental risks. These
approaches may offer useful starting points for the Agency as it seeks to develop and use
science-based methods to compare relative environmental risks at various levels..." The
text in this section should clearly indicate the need for research and empirical testing
before such methods will be ready for serious application.
Recommendation 4. EPA should use a broader range of risk reduction options in
combination to manage environmental risks
In place of "use" this recommendation should read "evaluate and consider using". In the
second paragraph "risk reduction" should be broadened to "environmental protection",
since minimizing risk is often not the only objective. On page 24, the paragraph that starts
"Determining the best tools..." should note that empirical evidence on the past
performance of various regulatory strategies should be an important factor in choosing
future strategies. The paragraph that begins "The best strategy, by definition..." is
problematic as noted above.
Recommendation 5. When evaluating risk reduction options, EPA should weigh the full
range of advantages and disadvantages, both those measured in dollars as costs and
benefits and those for which there may not be a comprehensive dollar measure, such
as sustainability and equity
The recommendation should be modified to read "should strive to weigh the full range...".
The subsequent write up should acknowledge that in some cases enabling legislation does
-------
not allow the Agency to actually include such considerations in their final decision. On
page 25 in the paragraph that starts "Some of society's environmental values..." it is noted
that "tools are available". The paragraph should note that these tools are controversial, of
varying quality, and need further development. While developing better methods to
support deliberation, especially structured deliberation, is important, so too is the
development of methods to help parties understand and deal with the substance of the
issues under consideration. Deliberation without knowledge and understanding is of
limited value. The objective should be to obtain the well-informed opinions of individuals
and groups, not their seat-of-the-pants reactions. It is not apparent that stakeholders, in
the sense of interest groups, are the only groups that should be involved. EPA acts on
behalf of the public as a whole, and as such should seek methods to obtain and consider
the well informed views of members of the general public.
Recommendation 6. EPA should make fuller use of the scientific methods available to
characterize public values and incorporate those values into goal setting and
decision-making
This recommendation might better read "EPA should seek methods, such as those in the
field of formal preference measurement, that obtain, develop and make fuller use of
methods to characterize..." The discussion that follows should reflect this change. Add at
least a paragraph on research needs. Delete the sentence "For a more complete
discussion...Chapter 5." from the next to last paragraph on page 29.
Recommendation 7. EPA should identify, collect, and disseminate scientifically-based
environmental metrics organized in new ways to support a more integrated
approach to managing environmental risk
This recommendation might better read "EPA, by itself and in concert with other agencies,
should collect and disseminate scientifically-based environmental metrics to support a
more integrated approach to managing environmental risks." Given that there is an infinite
number of potential metrics, this effort should be matched to the needs of environmental
management.
Recommendation 8. EPA should develop a system of "report cards" to organize and
disseminate information on the status of ecological and human health and the
quality of life in order to assess the effectiveness of its environmental decisions and
to guide future environmental management
Again, the phrase "by itself and in concert with other agencies" should be added. A
paragraph should be added that discusses the efforts of other agencies, the Heinz Center
and others. On page 32 the sentence "Because the overarching goal...health declines"
should be removed. Remove the references to "outcomes-based". On page 33 modify the
-------
sentence that begins "This new kind of reporting system..." to read "This new kind of
reporting system will emphasize outcomes and reportability"
Recommendation 9. EPA should expand and develop new collaborative working
relationships with other federal and non-federal agencies and others who also will be
involved in integrated environmental decision-making
This recommendation is acceptable. On page 36, the last paragraph should refer to
"human health and ecological problems, and their management...".
Recommendation 10. EPA should aggressively explore options for reducing risks from
significant stressors that currently are addressed inadequately by the nation's
environmental institutions
This recommendation is acceptable. The box on page 37 should be dropped, or moved to
the section of the revised Volume 2 where the ecological risk-ranking procedure is
discussed and represented as the results of an illustrative application of the method. The
final paragraph on page 39 should be dropped.
2.2 Charge Question 2: Do the chapters on ecology, health, risk reduction options,
report card, and the deliberative portion of the valuation chapter describe adequate
and useful methods for addressing/ranking risks?
We address the question as related to: ecological risks; health risks; risk reduction options;
and performance evaluation separately below.
2.2.1 Ecological Risks
While it needs some refinement, and should be better grounded in multi-attribute utility
theory and practice, the procedure outlined for ranking ecological risks is very interesting and
holds considerable promise.
Evaluating relative risks among stressors and ecosystems is a challenging task because it is
a complex, multifaceted problem of analysis, involves comparisons of the essentially
incomparable, and can be heavily value-laden. The basic methodology developed and articulated
here provides a classification scheme relating stressor intensity and frequency, behavior of
ecosystems, and relevant influencing factors such as recoverability and species depletion. This
productive and informative approach offers significant advances over the less explicit approaches
to risk-ranking used in Unfinished Business and Reducing Risks. The chapter is generally well
written and builds logically on the Agency's previous ecological risk framework and guidelines. It
contains a significant part of what is new and exciting in the overall report. With modest revision
to avoid over-reaching, to better reveal limitations, and to relate the work to relevant literatures,
this chapter should be suitable for SAB approved publication.
10
-------
Our main concerns are that: (1) the methodology developed is as yet experimental,
heuristic, and but one of several possible approaches, (2) it embodies normative assumptions as
well as scientific appraisals but does not make this fact explicit. For these reasons, the stressor
risk-rankings produced are not unique or definitive. While some of these limitations are
acknowledged in the draft report, we believe that the methodology and results require further
qualification in the final report to avoid uncritical application and simplistic interpretation by users
of the report. In essence, we feel that while the approach developed moves the bar, it is not at the
end of the line. The numerical approach needs the constructive experimentation and evaluation of
the scientific community in a number of realistic applications.
The following is an elaboration of our main concerns and associated suggestions:
Methodology
The multi-attribute model for risk-ranking is based on a heuristic procedure and is not the
only one that could be developed. There is no significant discussion provided of the
implications of choosing this as opposed to another model form (e.g. multiplicative versus
additive, etc.). There is no indication that it has been significantly informed by the large
literature on multi-attribute decision-making (half of the references are to papers by
Harwell and none of the others are among the classic papers in multi-attribute utility or
behavioral decision theory). We suggest adding a brief discussion of alternative strategies
(with references), and an explanation of why the multiplicative approach was selected, and
the potential pitfalls of this approach. There should also be brief mention of the limitations
of the expert-opinion and/or consensus approaches and elaboration of the suggestion
(Volume 2, page 2-15) to use an expert opinion survey to provide a "verification" of the
multiplicative factors.
It is good to have such an explicit, transparent methodology to assess relative risks,
particularly one that can be applied on different scales. While it is true that assumptions
and scoring on specific components can be changed and debated, the multiplicative
approach results in exponential dynamic range and compounded influences of (potentially
related) factors. For example in the national risk-ranking, the rank order among
pesticides, nutrients and sediment and turbidity could easily be reversed by small and
defensible adjustments in the weights applied to proportion of resource, recovery and
ecological significance. Our point is not to suggest the ratings should be rescored in the
report, but that relatively small differences can affect the relative and categorical rankings
of the mid-range stressors, i.e. those in the dynamic response domain for the methodology.
Some criteria are related to others, e.g. proportion of resource and special ecological
significance or recovery potential and duration of the stress effect. Thus, because of the
multiplicative approach, their influences are compounded.
11
-------
While sources of uncertainty in the ranking process were identified, there were no
suggestions on how to assess their importance. More discussion of the attempted
"sensitivity analysis" (pages 2-22 and 2-30) would have been helpful. Estimates of
individual confidence in the "availability of the data and in their own grasp of this
information" (page 3-25) could provide added credibility to the outcome.
Environmental stressors on ecosystems are often interconnected. The approach taken
here is integrated in so far as it makes comparisons among stressors and ecosystems, but
does not specifically address integration in the sense of interaction among stressors, except
through the factor of secondary stress induction (narrowly constrained between 1 and
1.1). Some potentially important interactions between "low risk" stressors and others may
be neglected, e.g. the increased loading of fixed nitrogen that affects NOX, acid deposition,
tropospheric ozone, contaminated groundwater and nutrient stress in aquatic and
terrestrial ecosystems. This whole problem, rapidly expanding in the late 20th century, is
lost in this assessment.
Normative assumptions
The impression created by the current text is that risk comparisons such as this are done
on strictly scientific bases and that human values come in only in the decision-making
process. This is clearly not the case, because there are many normative assumptions
embedded both within the methodology (e.g., in the structure of the model) and in the
expert applications of the methodology (e.g., in the choice of weights). As currently
proposed, the procedure produces a ranking based on the normative judgments of
ecologists. Members of the general public do not have significant understanding of many
ecological processes and systems so unmediated public participation in these value
judgments is probably not practical. However, the IRP report does note that the ratings
by ecologists would have to be supplemented by value judgments provided by lay people
and with some background information, it might be possible to involve the public. It is an
open question whether the resulting preferences would be similar. The public might, for
example, care more about keeping landscapes looking verdant than about their ecological
composition. There should be greater recognition and discussion of this point.
The overall framework suggests that the goal should be reduction of the aggregate risks.
This too is a normative goal. One could easily imagine how society may focus on
reducing some risks that are not the "greatest" but may be the most achievable or socially
important.
Stressor risk rankings
Given the genesis of, and expectations for, this SAB assessment, there will undoubtedly
be much attention given to the National-Scale Ecological Risk Ranking. There should be
a clearer disclaimer in the narrative (page 2-23) about the limitations of this ranking
12
-------
(methodological, informational, uncertainty). For reasons stated under Methodology,
above, we are not convinced that the rankings are as robust as suggested on page 2-30.
Different scientific participants, with different information, experiences, and viewpoints,
could produced different rankings, probably not turned on end, but well across the
boundaries of high to medium and medium to low. The question of how robust the
current ranks are is one of the issues that should be explored in future research.
Appendix 2A containing the ecological risk profiles was not available but would be useful
in understanding how the "stressor-intensity profile" (i.e. the "estimated actual distribution
of the stressor in the environment") was considered for endpoints such as climate change
and should be peer-reviewed as suggested for the health risk characterization data sheets.
To what extent, if any, does the information format/content influence the subsequent
profile development?
The risk-ranking breakdown for "harvesting-coastal" is not included in Table 2-4. It
should be, given the high ranking for this risk, which is likely to be controversial.
The observation that the highest ranked ecological risks at the national level relate
predominately to physical and biological, rather than chemical, stressors, needs to be
underscored by the observation that these risks do not clearly fall within the purview of
any single regulatory/resource agency (most fall outside the direct regulatory authority of
EPA), and cut across local, state regional and national levels of government. There is no
guidance on how to link the rankings to any management decision framework.
While identified above as a potential problem, the stated goal of LED is to provide the
greatest reduction in "total aggregate risk" (1-29). In light of this, should ecosystems be
ranked as well as stressors? Can the methodologies be used to determine which of the
ecological systems (Table 2-2) is at highest risk due to the combination of stressors, and
how would such an analysis inform decision-making?
2.2.2 Health Risks
The objective of this method is to combine available data with expert (or other) judgment
to produce rankings of health risks. While the review committee finds this general objective
appropriate, the specific method proposed falls far short of meeting the need.
The authors accurately point out the difficulties in making comparative risk assessments
between health "endpoints". These relate to a number of specific factors:
a) There are limited data that are directly applicable, i.e. demonstrated effects in humans
at environmentally relevant exposure levels.
13
-------
b) It is frequently necessary to depend upon data from experimental animals to make
judgments about human risks.
c) Measures made in experimental systems do not readily translate into health effects that
are recognizable by lay persons.
d) While much of the available data can be and are being used for establishing standards
for many chemicals, their comparability has not been systematically approached in
a probabilistic way.
It is unfortunate that the health subcommittee did not wrestle more directly with the
question of how experimental data in animals or in vitro systems might be more effectively
integrated as predictors of disease processes. Even if a methodology could not be developed, a
more extended discussion could serve to highlight the problems. This could serve as a focus of
further biomedical research into these processes. Indeed, this direction is embedded in the
research agendas of many governmental and non-governmental research agencies. By masking
these issues under a general call for experts (and others) to exercise their judgments, the chapter
fails to provide advice or assistance on the really difficult parts of the health ranking problem.
Expert judgment has an important role when scientific understanding is incomplete. But to be
useful, it must be obtained with careful consideration of the information needs and cognitive
limitations of those making judgments.
The proposed approach has several practical limitations. Foremost among these are: 1)
the paucity of sufficient and comparable data on exposure or dose to populations of interest; 2)
the general lack of sufficient or comparable data on hazard posed by environmental levels; and 3)
the reliance on judgments made by either technically-trained or lay public panel members.
"Experts" from specific scientific disciplines such as toxicology, epidemiology and
radiation biology will differ on the respective merits of exposure-response data from human
populations vis-a-vis controlled exposure studies. Panels of non-scientists will be hard pressed to
evaluate the merits and limitations of any of these data. How can they be expected to make
judgments of the relative risks of stressors when some data sets will have apparently conflicting
data from epidemiology and toxicology, while others will have only one kind of data or the other?
Will they be able to distinguish between the effects of chronic and acute exposures or chronic and
acute outcomes?
The proposed "Stressor Risk Characterization Data Sheets" could potentially be very
valuable but pose obvious questions. Who will construct them? How will they reduce the
equivalent of multivolume "criteria documents" to a page or two? How will they be reviewed for
issues such as selection bias (e.g., listing some specific data elements and not others, etc.)?
14
-------
How will data on toxic substances be presented? Would unit risk factors and/or reference
concentrations be relied upon? If so, would the data sheet also contain best estimates of the
nature and extent of safety factors that are built into them?
Much of the latter portion of the chapter discusses the possible use of fuzzy sets in this
application. The chapter does not succeed in demonstrating the value of these tools in this
application. The theory does not appear to be used by the health subcommittee for any significant
manipulation to demonstrate its utility in this context. Given that probability has superior
analytical properties it is unclear why the simplification is desirable. For many years, EPA's
Office of Air Quality Planning and Standards has been using expert elicitation of subjective
probability distributions to characterize uncertainties about health damage functions for Criteria
Air Pollutants. None of this work is mentioned or discussed in the chapter. The section contains
no references to the literature on expert elicitation (except for the book by Morgan and Henrion
which includes a chapter on this topic) nor to the previous work of OAQPS.
In view of the inherent data limitations, the limited focus on the cognitive demands
imposed on the participants, problems in unbiased panel selection and similar issues, the
methodology proposed should not be endorsed as a viable option for risk ratings. These concerns
raise serious questions about whether the chapter should be included, even in unreviewed form, in
Volume 2 in its present form. A major rewrite would be needed.
2.2.3 Risk Reduction Options
Chapter 6, Risk Reduction Options outlines in detail some useful tools for considering
alternative risk reduction approaches. Many EPA choices of regulatory approach are determined
by the relevant legislation, but some are not and, in any case, Congress, EPA, and state and local
agencies have an obligation to consider whether the approach chosen is the most appropriate and
satisfactory. The chapter outlines a ten-step strategy for identifying and selecting a risk reduction
strategy from the various available options, but does little to guide the selection in relation to the
other chapters in the background volume being reviewed here. Further, there is a large literature
on alternative regulatory strategies. Chapter 6 ignores most of it.
The discussion appears to place too little value on previous experience including empirical
evaluation of the performance of alternative regulatory formulations in the US and elsewhere. It
also appears to be formulated in terms of finding the single best strategy, as opposed to an
adaptive approach to developing regulatory strategy in which the strategy evolves with experience
and learning.
The chapter talks frequently in terms of "optimum" choice of strategy. Given the complex
nature of the task, thinking in terms of "optima" as opposed to "good or better" strategies, is
probably unrealistically ambitious.
15
-------
The Framework in Figure 6-1 does not differentiate between choosing a broad regulatory
strategy and choosing particular options within that strategy. This ambiguity about the two levels
of decision-making often makes the text hard to follow.
The text reads as if EPA has the freedom to rethink the broad approach it will take to
managing all environmental risks. As argued in the discussion above about enabling legislation,
this is often not true. While the authors might argue that is simply something to be dealt with
when considering constraints, it is sufficiently fundamental that, unless it is explicitly addressed, it
leaves much of the discussion of the chapter in an uncertain state.
The text talks a bit about iteration (e.g., page 6-11, lines 29-34), but there is none shown
in the internal structure of Figure 6-1.
The chapter proposes the following screening criteria for evaluating regulatory strategies
(Section 6.5):
a) Environmental Effectiveness
b) Cost
c) Equity
d) Workability
e) Flexibility
This list needs further thought. For example, should "adaptive", "democratic", and
"scientifically sound" be added? Should each criterion be elaborated? For example, "equity"
might be expanded to: distributes costs and benefits fairly; gives all affected parties standing;
provides all parties equal access to key information.
The document makes a powerful point by noting that ranking of risks can be misleading if
the same intervention opportunity can reduce more than one "stressor". For example, greater
reliance on nuclear power or solar energy for electricity generation (instead of fossil fuels) can
reduce emissions of both greenhouse gases and particulates. Yet the document does not take this
point to its logical conclusion: that rankings of decision options could be more useful than
rankings of risks.
2.2.4 Performance Evaluation
The goals outlined in this chapter for the consistent reporting of performance measures on
environmental risk management decisions are highly commendable. They may be easily
implemented in some domains of ecological impact, but in other domains, such as low probability
health impacts, practical implementation may be difficult or impossible. There is no practical way,
for example, that one will be able to observe the removal of a 10"5 lifetime mortality risk in many
populations. In the case where effects measures are infeasible, exposure measures that exceed
standards or guidelines serve as early warning signs that further exposures are likely to cause
16
-------
disease or disability in sensitive or target populations. To the extent that the standards and
guidelines are conservative, or that actions can be taken to reduce such exposures, there should be
few, if any, adverse effects to be measured.
While EPA should continue to examine health and/or environmental surveillance and
monitoring data leading to the development of effects measures, its ability to employ them on a
consistent basis will likely be limited, requiring concentration on process, stressor and exposure
measures that prevent exposures that cause adverse effects.
This chapter ignores entirely the problem of confounding variables, a major problem if one
tries to use performance indicators as part of a decision-making process. For acute (short-term)
exposures, stressor levels are often much more influenced by meteorological, economic, lifestyle
and other variables than by EPA regulations. Thus, it is necessary to statistically control for these
variables if the indicators are to provide useful information. The problem is less severe when
evaluating long-term changes in exposure.
The chapter also pays little attention to other relevant efforts dealing with report cards and
performance indicators. It gives only passing mention to the Vice President's report card effort
(currently being developed by the Heinz Center) and to the Governmental Performance and
Results Act, a major force for performance indicators. It deals not at all with EPA's new
information office, the National Academy of Sciences' Committee on Environmental Indicators
and Board on Sustainable Development, EDF's web-based efforts to summarize TRI data, or
several Congressional acts that deal directly with the subject.
The chapter should also acknowledge that report cards are difficult to construct in a
comprehensive way, cannot be done frequently enough for some environmental decision-making,
and are most helpful on large-scale, strategic assessments. Environmental management, however,
is often on a shorter time step and is more tactical than strategic. In this vein, the ideas of
adaptive management, already embraced in various federal initiatives for ecosystem management,
gets scant attention. Adaptive environmental management places a premium on monitoring and
learning through doing. Thus, monitoring (observing) systems on regional scales (the most
effective scale for integrated environmental decision-making) need to include interactive
environmental monitoring and modeling components, as well as periodic report cards based on
these technical tools.
Thus, the chapter needs fairly extensive revision if it is to serve its intended functions as a
stimulus to EPA to prepare credible and effective report cards as documentation of effective
program management and environmental cleanup.
2.3 Charge Question 3: Does the document provide an adequate and useful description
for how this information might be linked in decision-making?
17
-------
The appearance of an integrated framework from the text in Chapter 1 and Figure 1-3 is
simply not supported in the remaining chapters of the report nor does it discuss how information
from Chapters 2-7 should be linked to Agency decision-making. There is no set of linkages that
provide an integrated quantitative or semi-quantitative methodology based on Figure 1-3 to
integrate health risk, ecological risk, benefit-cost, valuation issues and regulatory option
development so as to aid in Agency decision-making. Rather, what the report presents is a
collection of ideas and approaches. The task of matching the ideas and approaches to specific
Agency needs is not addressed in either volume. Rather, they deal with broad generalities about
environmental health and ecosystems, with no attempt made to assess the needs of individual
Agency Program Offices to make improvements in existing decision processes.
Some of the ideas and approaches are well-known, such as benefit-cost analysis. Some are
innovative and hold great promise, such as the ecological risk-ranking system developed in
Chapter 2. Some, such as the health ranking system proposed in Chapter 3, are not likely to be
useful. The Report provides little specific guidance on how to use the tools or put these building
blocks together, and how to incorporate them into existing Agency processes and procedures. It
may be difficult for Agency managers to evaluate which of the ideas and approaches fit their
specific needs.
The discussion in Chapter 1 suggests that there are four key components to integrated
environmental decision-making: 1) comparative risk assessment for multiple sources, stressors,
and effects; 2) decision options for multiple sources and stressors, accompanied by benefit-cost
analysis; 3) output-oriented progress measures and goals; and 4) public input, especially on
values.
The risk assessment parts of the report deal mostly with comparisons of risks for broadly
categorized stressors. Comparative Risk Assessment (CRA) is not a new idea. CRA exercises
may generate insights in highlighting environmental problems that deserve more attention from the
Agency. Such insights could be important for Agency top management and stakeholders.
The report is not very helpful about how to realistically identify a menu of decision
options. It doesn't help much at the general level (market mechanisms, enforcement, etc.) and it
helps not at all at the more specific level where the dialogue about real decisions is conducted.
The benefit-cost chapter is a useful primer, as the following pages acknowledge, but it doesn't
add anything either to benefit cost analysis or to how to link it with other parts of decision-
making.
2.4 Charge Question 4: Does the economics chapter provide an adequate and useful
primer for economic analysis?
This is a well-written and coherent chapter which provides a solid review of current
literature and practice in benefit-cost analysis. Unlike Chapters 2 and 3 which, despite their
methodological limitations, break important new ground, Chapter 4 is a very straightforward
18
-------
tutorial of standard economic ideas and methods. It does a better job than some such reviews of
stating some underlying assumptions and limitations of the standard methods. For example, it
notes (page 4-4, lines 21-22) that risk is a multi-attribute concept and that simply estimating
expected mortality or morbidity is not the same thing as estimating level of risk. In contrast to
some of the proceeding chapters, this chapter does a good job of summarizing relevant literature
and reflecting at least some of its limitations.
It does a good job in debunking some erroneous beliefs about benefit-cost analysis and of
explaining many of the legitimate limitations of the technique. The paragraphs that follow offer
recommendations to strengthen the chapter, particularly in the context of the integrated risk
assessment focus of the overall volume.
The economics chapter is completely unintegrated with the rest of the volume. This is
unfortunate as the demonstration of how economics could be more effectively integrated into
environmental decision-making would have been a substantive contribution. The lack of
integration is apparent in several dimensions. Chapter 4 contains no discussion of problems
identified in other chapters and other chapters contain almost no mention of the tool of benefit-
cost analysis or how it should be used in accomplishing the objectives of integration. For
example, consider the discussion of ecological risk-ranking in Chapter 2 and Table 2-4. There is
no indication of whether or how benefit-cost measures could be integrated with the risk rating to
generate an overall ranking based on both scientific risk information and public values. The term
"value" as used here is not always consistent with its use in Chapter 5, where it takes on several
different definitions.
Given the subject matter of the report, Chapter 4 needs a discussion of particular issues
concerning the preparation of benefit-cost analyses under integrated environmental decision-
making. For example, it is important for those who might use the results of a benefit-cost analysis
to fully comprehend the degree and sources of uncertainty associated with the numbers in such an
analysis. Thus, discussion of methods for incorporating and evaluating uncertainly in benefit-cost
analysis should be included.
Time preference and discounting is another topic which needs further discussion. The
issue of discounting in the inter-generational context is quite complicated and the discussion on
page 4-24 does not do it justice. A summary of the treatment of this topic in "Guidelines for the
Preparation of Economic Analyses" found on pages 6-21 to 6-25 of the June 11, 1999 draft
(EPA, 1999) would be more appropriate. More generally, the SAB's Environmental Economics
Advisory Committee has reviewed a draft of EPA's "Guidelines for Preparing Economic
Analyses." This document sets forth EPA's approach toward benefit-cost analyses. Efforts
should be undertaken to ensure that Chapter 4 is consistent with this document. The literature in
behavioral decision-making indicates that actual revealed time preference is often significantly
different than conventional economic theory. This issue deserves discussion.
19
-------
A primary focus of the report as a whole is the need for EPA to do integrated assessment
of multiple consequences and actions. Conceptually, this is easily accommodated in a carefully
done benefit-cost analysis; all of the affected benefits of a policy are compared with all of the
costs. Practically however, many of the cost and benefit components to such an overarching
benefit-cost analysis will come from different sources (different researchers using different
assumptions and different data, etc). In such a study, the different assumptions and different
quality of data and quality of analysis used in the different components should be acknowledged
and perhaps evaluated; that is, some pieces of benefits are likely to be more reliably estimated than
others. Discussion of the practical difficulties to addressing such issues would strengthen the
chapter.
Another troubling practical problem involves using the correct path of integration in
welfare assessment. When taking benefit estimates from different sources, it will often not be the
case that a correct path is being followed. Thus, a brief discussion of this topic would be
appropriate.
Benefit-cost analysis must be performed in a dynamic context where the value of waiting
to gain further information is weighed against the costs of potential irreversibilities in choosing
when to regulate and the scope of regulation. This is more difficult than period-by-period static
analysis, but in some cases (greenhouse gases, CFC's, etc.) it may be important. Once these
dynamic aspects are considered, a third question "When should risks be reduced?" should be
added to the two that are raised at the beginning of section 4.2 (lines 14 and 15).
The discussion of whether the public understands environmental risks (lines 21-29 on page
4-7) well enough to meaningfully value them, dismisses the issue too easily. Later in the chapter,
the authors themselves recognize that the public needs to be given information before they can
reasonably assess benefits (in the stated preference section) and that the methods "are least
successful when the protection measure to be valued has little-understood, amorphous, or very
long-term consequences" (page 4-17, lines 8-9). The question is what level of understanding is
required before useful assessments can be made? Clearly, a benefit-cost analysis should not
exclude harmful effects from a highly carcinogenic food additive just because the public has not
been informed of the harmful effects and at the moment has a zero willingness-to-pay to eliminate
it.
As the chapter's discussion indicates, it is very important that the lack of "expert"
knowledge on the part of the general public not be used to dismiss or invalidate public values.
However, there are some cases where the public probably does not have enough information and
understanding of the issue to reasonably value it. Ecological benefits of diverse ecosystems and
small probability risks are two other examples of where this might be a problem.
The discussion of equity concerns with benefit-cost analysis in connection with
willingness-to-pay being sensitive to income, needs some attention. The way the issue is currently
introduced implies that there is something "wrong" with benefit-cost analysis. Or something is
20
-------
"wrong" with willingness-to-pay (or willingness-to-accept) as a welfare measure because it is
sensitive to income. It's not that there's something wrong with it, it's just that it's about efficiency
and not equity. The U.S. is a wealthier country and thus is willing and able to spend more on
clean air than Mexico. However, one feels about it, that is a fact. If one wants to be concerned
with equity one has to go beyond willingness-to-pay. However, willingness-to-pay and benefit-
cost analysis can at least provide information to do some equity comparisons.
When the costs of a policy affect one group, the benefits accrue to another and when
compensation is not actually paid (which is, of course, almost always) then equity can be at odds
with efficiency. This is the case with the example presented in the "distributional considerations"
section (on page 4-28).
In several places, the chapter raises questions regarding how to do an aspect of benefit-
cost analysis, but doesn't answer them (e.g., lines 14-15 and 19-22 on page 4-21). These are
tough questions, but the presence of questions without answers is problematic.
The bottom-line paragraph on this issue (lines 16-24, page 4-28) is excellent, but could be
stated in a stronger and more positive way. Benefit-cost analysis is about efficiency. Trying to
add welfare weights is likely to complicate the efficiency message without solving the equity issue
satisfactorily.
2..5 Charge Question 5: Does the chapter on valuation provide an adequate and useful
philosophy describing how to incorporate values into decision-making, clearly
articulating that more than science is needed in the decision-making process?
The chapter correctly makes the case that values are an important element of integrated
environmental decision-making and that more than science is needed to support the decision-
making process. While there is a considerable literature on the subject, some of which is
referenced in the chapter, many scientists and others who play important roles in the
environmental decision-making community are not aware of this literature. Thus, this chapter
provides an overview of the important valuation issues for these people. The chapter, through its
breadth of topics, demonstrates that values is a diverse and complex area of study.
The chapter is, however, not adequate or useful in helping readers understand how to
incorporate values into decision-making for the following reasons.
Both Chapters 4 and 5 make the classical economic assumption "that people know their
preferences" (pages 5-2, line 11 and 5-24, line 18) and that the problem is "eliciting" (page 5-10,
line 11) those values. For many environmental risks, people probably do not have well-articulated
values (see for example Fischoff, 1991). Thus, a process of value synthesis or construction is
probably more appropriate than value elicitation. While there is brief acknowledgment of this
(e.g., page 5-18, line 13), it needs more development. In contrast to value elicitation, for which
there is a large literature, relatively little theoretical and experimental work has been done on
21
-------
methods for value synthesis, although several investigators are now working on this problem. The
authors of Chapter 5 do propose the use of "deliberative methods" (e.g., page 5-18, line 22),
although these methods are unspecified.
While deliberation can be an important part of a value synthesis strategy, it is not
sufficient. An equally important element is providing participants with an adequate understanding
of the issues at hand, and a decision-making context which is supportive, but leaves participants
free to reach their own conclusions.
The document highlights the important role of values (as well as science) in environmental
decision-making. Yet the roles of the social and decision sciences in the assessment of citizen
values are not clearly defined in the framework. At some points, the document seems to suggest
that value-laden issues are not the province of science and should be addressed by a broad group
of the public or stakeholder representatives. At other points, the document seems to suggest more
rigorous deployment of scientific methods to support citizen decision-making and characterize
citizen values, such as might be accomplished by formal tools of preference measurement.
The chapter is not integrated with other chapters in the report. For example, the
approaches set out in Chapters 2 and 3 do not make use of any formal valuation techniques to
create their heuristic risk-ranking methods even though value judgments were necessary in several
components of the approaches. Chapter 6, on risk reduction option selection, makes no reference
to any material in Chapter 5. Much of what is said duplicates points made in Chapter 4. For
example, pages 5-15 to 5-20 sounds like a prescription of how to do contingent valuation
methods.
The chapter is also not internally integrated. It does not provide a coherent framework for
the integration of all of the different "values" that are discussed. The organization of the chapter
lacks logical flow and coherence. Part of the problem arises from the difficulty of integrating
economic values and non-economic and non-monetary values within the economic valuation
paradigm. Since the chapter starts out with this discussion, it is hard to recover. Since 'values' is
such an important aspect of environmental decision-making, the environmental decision-making
community needs an integrated approach to the topic. Instead, the chapter suggests that there are
choices that can be made about values but that it is not possible to integrate the various
approaches. Perhaps because different people wrote different sections of the chapter, there are
different definitions in the various sections.
The chapter is weak in its descriptions about process issues. The chapter honestly admits
that there is no silver bullet for valuation methods. However, by going on to say that "expanded,
rich, and complex processes must be employed to fully characterize environmental values" (page
5-4, lines 3-4), it confuses rather than clarifies the practical challenges facing the Agency. The
reader never really finds out exactly what these expanded, rich and complex processes entail and
whether they're going to be worthwhile or relevant in a decision-making context.
22
-------
The discussion of deliberation makes some headway on process issues. However, it can
be improved by expanding its focus to include different types of decision-making processes or
modes to complement who needs to be involved in the processes. For example, two processes
that are important to environmental decision-making are conflict management and collaborative
learning and could be integrated within the deliberative process framework. Also, the discussion
needs to address how to coordinate deliberation across scales, time, institutions and cultures. The
impact of deliberative group processes on valuation is an interesting concept that should be the
subject of research (e.g., how should information be conveyed to the group, how should
individuals in the group be educated prior to participating in a valuation exercise).
The terminology around the concept of value needs to be improved. It is not helpful to
the environmental decision-making community for the word to have many different and legitimate
meanings. One example of a possible taxonomy has been proposed by Tom Brown, an economist
with the U.S. Forest Service, who distinguishes between "held values", "instrumental values", and
"assigned values" (Brown, 1984). In his view, "held values" are a person's core values, which are
essentially non-trade-off-able. Instrumental values are means to the ends of held values. People
can choose different means to ends. Assigned values arise when people commit resources of
some sort to achieve their goals. Thus, assigned values are in the realm of economics. This is not
to say that this is the best or only way to improve the semantics around values, but it helps to
illustrate the need for clearer definition.
Section 5.6 on additional approaches to valuation could be one of the most important
sections of this chapter, if not the entire document, but it is neither well-organized nor sufficiently
focused to have much impact. There are important and relatively new (to the public) ideas that
should be better described, again with some real examples, and cast in terms that might better
indicate their usefulness to EPA.
A looming question raised by this chapter is whether EPA is properly equipped to play out
the "values debate" or whether it should be playing the role of framing the technical questions and
turning back to Congress for guidance on how to balance competing values. This question is of
course even more relevant in light of the EPA's recent decision on the ozone and PM standards,
the role of CASAC in setting these standards, and the recent decision by the appellate court.
Lastly, does EPA have the regulatory flexibility to adequate consider values as discussed in the
chapter?
Numerous difficult values issues are not addressed or are addressed without indicating to
the reader what the important accompanying issues are. For instance, the chapter does note that
"changes in the perceptions of the value of actions toward reaching goals is part of adaptive
management..." (page 5-14 lines 4-5). However, the report does not fully note the dimensions
surrounding the role of values in goal-setting. For instance, March writes: "The argument that
goal development and choice are independent behaviorally seems clearly false. It seems...perfectly
obvious that a description that assumes that goals come first and action comes later is frequently
23
-------
radically wrong. Human choice behavior is at least as much a process of discovering goals as
acting on them..." (March, 1976).
Lastly, the title of this chapter is misleading. Although there is some good discussion of
issues in the valuation of natural resources/ecology (5-37 to 5-47), the bulk of the chapter lacks
the intended focus on natural resources and instead attempts a highly ambitious discussion of
everything from philosophy to stakeholder management in democracy.
2.6 Charge Question 6: Is the Document Clearly Written, Comprehensible and Complete?
As is clearly explained in the preceding pages, the document is often not clear. Some of
the arguments are incomplete or difficult to understand. There are a number of gaps in the
coverage. There is insufficient integration of the various pieces.
24
-------
3.0 REFERENCES CITED
Brown, T. 1984. "The Concept of Value in Resource Allocation." Land Economics. Vol. 60(3):
231-246.
EPA, 1999. Draft Guidelines for the Preparation of Economic Analyses. Office of Policy, US
EPA, Washington, DC. June, 1999
Fischoff, B. 1991. "Value Elicitation: Is There Anything There?" American Psychologist. Vol. 46:
835-847.
March, J.G. 1976. "The Technology of Foolishness" in Ambiguity and Choice in Organizations.
J.G. March and J.P. Olsen, eds. Universitetsforlaget, Oslo, Norway
R-l
-------
APPENDIX A. SPECIFIC EDITORIAL COMMENTS
1. Comments on Volume 1
Page 3, Col. 1, line 3: insert "environmental" before "risks".
Page 7, Col. 2, line 7: Donora is in the "Monongahela River Valley" and the episode was in Oct.
1948.
Page 10, Col. 1,1| 3, line 3: insert "Integrated" before "Environmental"
2. Comments on Volume 2
Page 1-1, lines 9-28: This first paragraph contains better summary language than that contained in
Volume 1.
Page 1-1, line 10: replace "powerful" with "responsive to national needs".
Page 1-1, line 14: replace "offered broad"; with "created broadened".
Page 1-1, line 18: replace "results" with "impacts"
Page 1-12, line 7: delete "requires"
Page 1-12, line 8: replace "and" with "are required for the".
Page 1-12, line 26: replace "have unique" with "each have important".
Page 1-14, line 15: should "...about what needs to be done by whom and why..." read "...about
what needs to be done by who and when..."
Page 1-15 box: The second paragraph neglects or at least downplays the multi-attribute nature of
risk. The sentence "...the SAB did not specifically propose a method for ranking QOL
risks because the selection of QOL ranking criteria is largely a value-driven, rather than
scientific, process..." Be careful not to imply that the risk-ranking that the SAB did was
not a value-based process.
Page 1-15, Figure 1-4, ]f 4, line 2: delete "appropriately".
Page 1-17, lines 7-13: It might be worth adding a few words on how one makes decisions in
Phase 1 stick, in light of likely subsequent legal, legislative and other factors.
Page 1-17, line 29: replace "reduction in" with "impact on".
Page 1-18, Figure 1-5, :]f 1, line 8: insert new sentence after "well."as follows: "It may also
involve increases in risks associated with solid or liquid waste disposal of the ash
collected." Also, after "benefits" insert "and risks."
-------
Page 1-25, lines 12-16: This language is overly categorical. How about "...is often not a
realistic..."
Page 1-25, line 17: Drop the word "over".
Page 1-25, lines 26-35: There are nice things said here about the integrated nature of the Food
Quality Protection Act. In discussions at the last SRA annual meeting, EPA staff were
unable to make the case that all the integrated analysis they have been doing has actually
done anything to improve health and safety. Can the report authors add anything on this
question?
Page 1-27, line 20: replace "our" with "EPAs".
Page 2-5, lines 32: "where appropriate." Categorizing risks or impacts into a finite number is a
critical part of risk-ranking. What are the rules that determine when one should
"appropriately" aggregate or desegregate.
Page 2-7, lines 7-18: Again, categorization is potentially very important in determining the
outcome. Can the report authors give any guidance on how it thinks EPA should do this
task?
Page 2-9, lines 5-6: The literature on multi-attribute utility stresses the importance of paying
attention to the actual range of values assumed by the attributes. In the method proposed,
how does one keep those ranges in decision-makers minds as they are asked to make the
various evaluative judgments that follow?
Page 2-10, Figure 2-3: With three data points (L, M, H) it is unclear how 1 differs from 7 or 19.
Page 2-14, lines 30-31, also Table 2-3: "...multiplicative factors...were developed by expert
judgment..." This is, of course, a value judgment not a strict scientific judgment. Why
should ecologists make these judgments? Did the IRP give consideration to a strategy by
which lay representatives could be educated in the underlying science and then provide the
value judgments? While it is interesting to know the value judgments of experts, there is
no reason to privilege their values (as opposed to their expert scientific knowledge). Lay
people may value different things than ecologists, even once they are fully informed.
Page 2-10, lines 21-22: "7...i.e. (1.17)" logic unclear.
Page 2-34, lines 16-30: Some discussion of whether and how this method might deal with
synergistic effect would be a useful addition.
Page 3-3, line 9: "elicitation" not "solicitation."
A-2
-------
Page 3-6, Figure 3-2: Treatment of exposure appears to be more simplistic than current OAQPS
EPA practice.
Page 3-11, line 3: clarify meaning of "high end". Following EPA's Exposure Guidelines, this has
the explicit meaning of exposures above the 90th percentile of the population distribution
of exposure.
Page 3-11, line 30: change ":half life in humans" with "half-times in human organs and tissues".
Page 3-12, line 8: change "often" with "sometimes". Most TLVs are PELs are based on animal
toxicology and have large safety factors built into them.
Page 3-13, line 7: either explain the asterisk or delete it.
Page 3-13, line 8a: insert "Lung airway" before "doses".
Page 3-13, line 13: change "COPD" to "cardiopulmonary".
Page 3-13, line 13a: insert "ischemic disease, and heart failure" after "bronchitis".
Page 3-13, line 16a: insert "daily mortality" after "values".
Page 3-13, line 16b: change "RR is" to "RRs are".
Page 3-13, line 18: change "COPD" to "cardiopulmonary diseases".
Page 3-13, line 19: change bulleted text to "Reduced lung growth in children".
Page 3-13, line 29: Is U.S. EPA (1992) really an appropriate reference for this Table?
Page 3-15, line 10: change "but" to "or".
Page 3-15, line 11: change "accumulate" to "persist". Ozone and radon daughters don't
accumulate, but persist because of continuous formation
Page 3-19, lines 14-15: Morgan and Keith actually elicited full subjective probability distributions.
They simply used box plots to provide a compact characterization of their results.
Page 3-19, lines 29-32: This reference turns the argument of Morgan and Henrion on its head.
They express a preference to not use qualitative probability words because their meaning
is so different in different context and for different people. For a more elaborated
argument of the same point, see M. Granger Morgan, "Uncertainty Analysis in Risk
Assessment," Human and Ecological Risk Assessment, 4(1), 25-39, February 1998.
A-3
-------
Page 3-20, line 13: should "truth value" read "membership value"?
Page 3-21, line 30: There is no obvious vertical line in Figure 3-7.
Page 3-22, lines 1-6: As noted earlier, Morgan and Keith did not elicit standard deviations. They
elicited full probability density functions (as CDFs) and then used them to compute the
various statistics on the distribution.
The preface uses the language "benefit/cost" (Volume 2, page Ill-i, line 20 and ff). Chapter 4
uses the language "benefit-cost". In as much as the difference is a more appropriate
formulation than the ratio in most of the applications being discussed (see the various
books by Mishan such as EJ. Mishan, Economics for Social Decisions: Elements of
cost-benefit analysis, George Allen and Unwin, 1972), the preface should be changed to
"benefit-cost."
Page 4-3, lines 13-25: the discussion basically assumes that "riskyness" is measurable, or perhaps
that it is the same as some single attribute such as expected mortality. See the discussion
above about multi-attribute nature of risk
Page 4-15, line 24, should "complementary" be "substitute"?
Page 4-16, first paragraph. Option value is based on risk aversion and is generally not considered
a separate component of value; rather it's the difference between an ex ante and ex post
measure. Providing some of the motives for preservation or existence value might help
people understand why these are legitimate values, but giving them specific names (e.g.,
option demand, bequest value, etc.) may imply an additive and exhaustive feature to them.
Page 4-21, lines 24-29. It is likely true that many ex ante abatement cost estimates have been too
large (although the citation provided here appears to only cover CFC's), it is also probably
true that the costs of permit schemes have probably been understated (i.e., more cost
savings are predicted than actually occur). There are likely other types of biases that often
rear their head, but are case specific.
In section 4.5.4, If the approach described in the concluding paragraph (on page 4-27) is the
preferred alternative of the committee, the report should be written with a stronger
statement to that effect.
The statement on page 4-8, line 3, that a wealthier person "will be able to pay more" is
incomplete. It's not only that they're "able," it's also that they are willing to pay more.
Willingness-to-pay captures both willingness and ability to pay.
Page 4-2, line 6. Change "effect" to "affect".
A-4
-------
Page 5-3, lines 18-19: Not clear this is true. It is sometimes possible to agree on policy while not
sharing the same goals. This is acknowledged on page 5-15, lines 6-7.
Page 5-4, lines 5-6: How about "the public." Isn't it general public values that should ultimately
carry the most weight?
Page 5-7, line 9: The sentence reads: "Equally important is the consideration of the value ascribed
to alternative outcomes." Outcomes of what?
Page 5-8, lines 2-8: The example could add to understanding if it included some pertinent fact
about lead removal technologies.
Page 5-9, lines 2-6 and 19-29: Examples would add to understanding.
Page 5-10, lines 4-12: This paragraph is riddled with jargon and ultimately unclear as to its
message.
Page 5-10, line 11: See previous discussion of elicitation versus synthesis.
Page 5-11, lines 6-10: This statement reflects the near total disconnection of the chapter from the
current baseline of Agency stakeholder and deliberative processes.
Page 5-11, lines 25-26: "...to be socially significant, a change caused by a stressor must exceed
the natural variability of the environmental system." This needs to be stated more
carefully. As stated it is not true. Consider a natural system that has high variability which
often puts it within 0.2 sigma of serious damage. Then an environmental loading of 0.3
sigma, which is less than the natural variability, could wipe it out.
Page 5-11, lines 29-32: This is repetitive.
Page 5-12: Figure 5-2 adds nothing.
Page 5-14, line 1: delete "recently"
Page 5-17, lines 6-7: "Some important preferences for protecting and managing environmental
resources are constructed in the elicitation process..." This is an important point worth
some more elaboration in the text. See article by Fischhoff on value elicitation cited above.
Page 5-18, lines 32-33: These lines contain a point that deserves much more explicit discussion.
Some people view some environmental features (such a minimally clean air) as a right, not
to be traded off against other things. And in the case of air, the Clean Air Act adopts this
formulation. This is at odds with the b-c formulation and deserves more discussion than it
gets here.
A-5
-------
Page 5-19, lines 8-12: This is a very important paragraph.
Page 5-24, lines 30-33: This is a much tougher problem than these few lines suggest. See the
discussion above about expert versus lay values.
Section 5.4.5 The "Conclusions" section (page 5-25, line 25) adds very little.
Figure 5-3 is useful as is the accompanying discussion. However, this section would be improved
significantly if it could cite a few examples in each of the four categories of decision-
making.
Page 5-32, lines 11-18: Will this be news to anyone at the Agency? There are probably hundreds
of people at the Agency in the Congressional and public affairs as well as the media offices
who believe that asking (and answering) these questions has always been part of their job.
Page 5-33: It would be appropriate to reference Terry Yosie's recent review of stakeholder
methods.
Page 5-33, lines 33-35: These are very important lines. Similar lines stressing the importance of
further research should appear in several other chapters of this report including Chapters 2
and 3.
Section (i) beginning on page 5-34 has some promise as does section (j). These sections would
benefit from further concrete elaboration.
Page 5-37, lines 10-11: "one can not state a value without stating the goal to be served." Is there
general agreement among philosophers on this statement?
Page 5-37, Section 5.6.2: The "Findings" are not really findings. Some are statements with no
explanation; others have further discussion attached.
Page 5-37, lines 28-35: The discussion here is redundant.
The concepts of fairness and sustainability (e.g., page 5-37) (as alternative goals to the
maximization of aggregate individual welfare) are not defined.
Page 5-39, lines 33-35: Isn't it more the case that people describe environmental systems in terms
of nested hierarchies (see, for example, discussion by Herb Simon in Science of the
Artificial).
Page 5-40, lines 19-25: This paragraph begins a line of thought about governance of the commons
and then drops it. (Also Ostrom is spelled incorrectly)
A-6
-------
Page 5-41, lines 9-13: Ecosystem management and community-based environmental protection
are not interchangeable terms.
Page 5-42, lines 18-29: Paragraph (i) is jargon-laden and does not convey a clear message on
research directions.
Page 5-43, lines 4-5: A new heading is needed. The discussion that follows is not related to
systems analysis in any common meaning of the term.
Page 5-44, lines 26-27: Doesn't this argument ignore consumer surplus? This section appears to
be verging on an entropy theory of value.
Pages 5-44 to 5-46, The discussion of biophysical measures of value is bizarre (it seems like a
prescription for furthering inefficiency) and should be eliminated.
Page 6-3, line 1: "knowledge experts", is there some other kind?
Page 6-7, line 17: How about replacing "will be ineffective" with "may be ineffective"
Page 6-7, line 21: "optimum" is probably too strong
Page 6-20, lines 1-13: This is interesting speculation. Isn't there literature and empirical evidence
on such issues? Similarly, there certainly is a significant literature on green labeling.
Page 6-23, lines 8-12: Engineered solutions are not necessarily end of pipe solutions. What about
pollution prevention?
Page 6-26, lines 31-34: This problem is not unique to this regulatory strategy. How, for example,
does one choose an emission tax rate, or fix the number of tradable permits to allow
without an "understanding of the sources contributing to the pollution load..."
Page 6-27, lines 11-17: Same point.
Page 6-32, Table 6-1: This table should either be consistently filled in or be dropped.
Page 6-41, lines 10-11: "The application of multi-criteria, multi-objective analysis to a large
number of identified options may unduly complicate the options selection process." This
is true, but appears to be at odds with some of the rhetoric of Volume 1.
Page 6-43 Table 6-2: Improve the examples under "education" (TRI, green labels, etc.). Add
emissions control technology and process redesign explicitly.
Page 6-49, line 17: "these data."
A-7
-------
Page 6-49, lines 23-26: Be careful not to imply that data on how a strategy has worked in the past
is not useful to choosing a strategy for the future.
Page 6-19, line 32: change "toxins" to "toxicants"
Page 6-28, line 11: insert "of after "Disadvantages"
Page 6-28, line 24: delete "?"
Page 6-44, line 22: What is "River Brown"?
Page 6-62, line 11: define "lexicographic". Many readers will be unfamiliar with the concept.
Page 7-5, Figure 7-1: where is "concentration measures"?
Page 7-12, line 15: insert "sometimes" after "are".
Page 7-18, line 27: delete "Human".
Page 7-19, line 17: change "expansive" to "extensive"
Page 7-20, lines 10-12: report cards have value even absent a more complex "integrated form of
environmental decision-making"
Page 8-2, line 2: change "science plays a critical role" to "the application of scientific knowledge
and technology play critical roles"
Page 8-2, line 15: change "also is" to "and technology also are"
Page 8-7, line 28: at end of line, add "to do what it can to"
A-8
-------
United States Science Advisory EPA-SAB-EC-99-018
Environmental Board (1400A) September 1999
Protection Agency Washington DC ivww.epa.gov/sab
oEPA Review of the SAB Report
"Integrated Environmental
Decision-Making in the
Twenty- First Century"
A Report By the Integrated
Environmental Decision-Making
Peer Review Subcommittee of the
SAB's Executive Committee
------- |