United States
Environmental Protection
Agency
Science Advisory Board
(1400A)
Washington, DC
EPA-SAB-EC-00-011
August 2000
www.epa.gov/sab
SEPA
Toward
is ion-Making
-------
The Science Advisory Board (SAB) of the U.S. Environmental Protection Agency is a body of
independent experts who provide advice to the EPA Administrator on scientific and engi-
neering issues. The SAB was established in its present form by the Congress in 1978. The
SAB's approximately 100 members and more than 300 consultants include scientists, engineers, and
other specialists drawn from a broad range of disciplines—physics, chemistry, biology, mathematics,
engineering, ecology, economics, medicine, and other fields. Members are appointed by the
Administrator to two-year terms. The SAB meets in public session, and its committees and review
panels are designed to include a diverse and technically balanced range of views, as required by the
Federal Advisory Committee Act (FACA).
The Board's principal mission is to review the quality and relevance of the scientific information
being used to support Agency decisions, review research programs and strategies, and provide broad
strategic advice on scientific and technological matters. In addition, the Board occasionally conducts
special studies at the request of the Administrator to examine comprehensive issues, such as antici-
pating future environmental problems and developing new approaches to analyze and compare risks
to human health and the environment.
-------
U.S. Environmental Protection Agency
Science Advisory Board
J
Integrated Risk Project
August 2000
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, DC 20460
August 18, 2000
OFFICE OF
THE ADMINISTRATOR
SCIENCE ADVISORY BOARD
EPA-SAB-EC-00-011
Honorable Carol M. Browner
Administrator
U.S. Environmental Protection Agency
Washington, DC 20460
Subject: Final SAB Report, Toward Integrated Environmental Decision-making
Dear Ms. Browner:
It is our pleasure to send you the accompanying final report, Toward Integrated Environmental
Decision-making, from the Science Advisory Board's (SAB) Integrated Risk Project. This report marks the
completion of the most complex and challenging activity ever undertaken by the Board, involving more
than 50 natural scientists, economists, and social scientists working together over more than three years.
The report has been subjected to a public peer review process in accordance with the Agency's Peer
Review Policy, and we acknowledge the inputs and insights of the peer reviewers.
Originally conceived by some as a simple "updating" of the SAB's Reducing Risk report, the Integrated
Risk Project (IRP) challenged the SAB to extend far beyond that 1990 product. Reducing Risk effectively
legitimized the notion that larger environmental risks can be distinguished from smaller risks on the basis
of scientific criteria. Some advocates of the IRP on Capitol Hill and in the Agency envisioned that the SAB
would go further in today's report and develop a ranked list of environmental risks, based not only on
consensus scientific insights but on consensus social values as well! Suffice it to say that this grand view
was neither the SAB's intention nor its product.
Instead, the SAB has developed a conceptual framework or vision for "the next step" in environmental
protection in this country. The first step was taken in the 1970s when there was broad public consensus
regarding environmental problems and their sources. The second step was taken when the Agency adopted
the risk assessment/risk management paradigm in the 1980s to support risk-based decision-making.
Building on Agency actions and several external reports generated in the 1990s, the SAB now proposes a
-------
Framework for Integrated Environmental Decision-making that describes how a broader array of consid-
erations and participants can and should affect environmental decision-making in the future.
One of the principle features of "the next step" is the involvement of a wider range of people — and
their perspectives/values — in the decision-making process. The Framework also emphasizes use of the
best science (both natural and social sciences) to assess cumulative, aggregate risks; to consider a broader
range of options for managing or preventing risks; to make clear the role of societal (public) values in
deciding what to protect; to clarify the trade-offs (including costs and benefits) associated with choosing
some management scenarios and not others; and to evaluate progress toward desired environmental
outcomes.
To be sure, the Framework is not a "turn-key" operation that needs only to be implemented. In fact,
much work remains to be done. The IRP took the SAB, and the Agency, into unfamiliar territory, involv-
ing research literatures in behavioral decision science and decision theory with which we have had limited
past experience. The effort has emphasized the importance of expanding the scope of expertise—both in
the SAB and the Agency—into these important domains. The project also has emphasized the importance
of adopting an interdisciplinary approach that combines deep understanding of environmental science with
theory and empirical methods in behavioral and decision science.
If the journey toward more integrated environmental decision-making is to be successful, the Agency
will need to undertake a significantly expanded effort in developing improved tools and guidance that have
been vetted with real problems. Specific focused research is needed on problems that range from improving
methods for informed synthesis and elicitation of public environmental values, to tools and procedures that
support improve characterization and treatment of uncertainty, reasoned science-based deliberative
processes, and the evaluation of multi-dimensional risks.
We note that the seeds of integrated environmental decision-making as described in the report already
have been planted in the Agency. New technologies and systems are being instituted at EPA that can work
together to nourish and encourage those seeds to sprout and grow. Prototypic tools have been devised as a
part of this SAB exercise that should be explored and, as appropriate, further developed through research
and applications to real problems in environmental decision-making. While the report charts a valuable
future direction for the Agency, a number of specific paths can lead to the desired destination. There can be
no substitute for a thoughtful strategy of experimentation worked out in the specific settings of different
environmental problems.
As noted in the final section of the report, "In some ways, the Board has been true to the old adage that
one's goal should exceed one's grasp. That is, the goal of this project — to articulate a complete and rational
method for including all aspects of integrated environmental decision-making in a single process — has
exceeded the SAB's grasp. In fact, it is likely that that goal will never be reached to everyone's satisfaction.
iii
-------
And yet, the Framework that was developed during the course of the project, clearly points to the direc-
tion in which "the next step" of environmental decision-making should go. The efforts of the individual
Subcommittees can be examined for further insights about how the Agency might — or might not —
make additional progress in that direction. In any event, there is enough direction and more than enough
challenge in this report to keep the Agency, and others interested in the next step of environmental deci-
sion-making, productively active for some time to come." The challenges of improving and better integrat-
ing environmental decision-making are considerable, but the end result should be worth the effort.
We look forward to discussing this report and your reaction to it at an upcoming SAB Executive
Committee meeting.
Sincerely,
Dr. M. Granger Morgan, Acting Chair Dr. Genevieve M. Matanoski, Chair
Science Advisory Board Integrated Risk Project Steering Committee
Science Advisory Board
IV
-------
NOTICE
This report has been written as part of the activities of the Science Advisory Board, a public advisory
group providing extramural scientific information and advice to the Administrator and other
officials of the Environmental Protection Agency. The Board is structured to provide balanced,
expert assessment of scientific matters related to problems facing the Agency. This report has not been
reviewed for approval by the Agency and, hence, the contents of this report do not necessarily represent
the views and policies of the Environmental Protection Agency, nor of other agencies in the Executive
Branch of the Federal government, nor does mention of trade names or commercial products constitute a
recommendation for use. The SAB members and consultants who participated in the activities that resulted
in this report did so as individuals, rather than as representatives of their employing organizations.
This particular project was conducted at the request of the EPA Administrator and addresses a broader
range of issues and concerns than most SAB reports. Consequently, many of the recommendations in this
report have more of a policy orientation than is usually the case.
Distribution and Availability: This Science Advisory Board report is provided to the EPA Administrator,
senior Agency management, appropriate program staff, interested members of the public, and is posted on
the SAB website (www.epa.gov/sab). Information on its availability is also provided in the SAB's monthly
newsletter (Happenings at the Science Advisory Board). Additional copies and further information are
available from the SAB staff.
-------
VI
-------
INTEGRATED RISK PROJECT COMMITTEES
STEERING COMMITTEE
Dr. Genevieve Matanoski (Chair)
School of Hygiene and Public Health
The Johns Hopkins University
Baltimore, MD
Dr. Joan M. Daisey1
Lawrence Berkeley National Laboratory
Berkeley, CA
Dr. Paul Deisler
Austin, TX
Dr. Mark A. Harwell
University of Miami
Miami, FL
Dr. Wayne Rachel
MELE Associates
Brook Air Force Base, TX
Dr. Alan Maki
Exxon Company, USA
Houston, TX
Dr. Paul R. Portney
Resources for the Future
Washington, DC
Dr. Milton Russell
Joint Institute for Energy & Environment
and University of Tennessee
Knoxville, TN
Dr. Ellen R. Silbergeld
University of Maryland
Baltimore, MD
Dr. Paul H. Templet
Louisiana State University
Baton Rouge, LA
Dr. Valerie Thomas
Princeton University
Princeton, NJ
Dr. Bernard Weiss
University of Rochester Medical Center
Rochester, NY
Dr. Marcia Williams
Putman, Hayes, & Bartlett, Inc.
Los Angeles, CA
Dr. Terry F. Yosie2
Ruder and Finn, Inc.
Washington, DC
Dr. Terry F. Young
Environmental Defense
Oakland, CA
Science Advisory
Board Staff
Mr. Thomas O. Miller
Designated Federal Officer
EPA Science Advisory Board
Washington, DC
Ms. Stephanie Sanzone
Designated Federal Officer
EPA Science Advisory Board
Washington, DC
Ms. Wanda Fields
Management Assistant
EPA Science Advisory Board
Washington, DC
Mr. Thomas Super
EPA Science Advisory Board
Washington, DC
Dr. Donald G. Barnes
Director
EPA Science Advisory Board
Washington, DC
ECOLOGICAL RISKS
SUBCOMMITTEE
Dr. Mark Harwell (Chair)
University of Miami
Miami, FL
Dr. William Adams
Kennecott Utah Copper Corp
Magna, UT
Dr. Steven M. Bartell^
SENES Oak Ridge, Inc
Oak Ridge, TN
Dr. Renneth W Cummins4
South Florida Water Management District
Sanibel, FL
Dr. Virginia Dale
Oak Ridge National Laboratory
Oak Ridge, TN
Dr. Carol Johnston
University of Minnesota
Duluth, MN
Dr. Frederick R. Pfaender
University of North Carolina
Chapel Hill, NC
Dr. William H. Smith
Yale University
New Haven, CT
Dr. Terry F. Young
Environmental Defense
Oakland, CA
1 Deceased.
2 Now with Chemical Manufacturers
Association.
3 Now with Cadmus Group, TN.
4 Now with Humboldt State Univ., CA.
Vll
-------
Advisory
Ms. Stephanie San zone
Designated Federal Officer
EPA Science Advisory Board
Washington, DC
Ms. Wanda R. Fields
Management Assistant
EPA Science Advisory Board
Washington, DC
AND
Dr. Joan Daisey5 (Co-Chair)
Lawrence Berkeley Laboratory
Berkeley, CA
Dr. Bernie Weiss (Co-Chair)
University of Rochester Medical Center
Rochester, NY
Dr. Stephen Ay res
School of Medicine Virginia
Commonwealth University
Richmond, VA
Dr. Paul Bailey-
Mobil Business Resources Corp.
Paulsboro, NJ
Dr. George Daston
Procter and Gamble Co.
Ross, OH
Dr. Curtis Klaussen
University of KS Medical Center
Kansas City, KS
Dr. Paul Lioy
Rutgers University
Piscataway, NJ
Dr. William Pease
Environmental Defense
Oakland, CA
Dr. Henry Pitot
McArdle Laboratory for Cancer Research
University of Wisconsin
Madison'W!
Dr. Jonathan Samet
Department of Epidemiology
Johns Hopkins University
Baltimore, MD
Dr. Valerie Thomas
Princeton University
Princeton, NJ
Dr. Lauren Zeise
California EPA
Berkeley, CA
Science Advisory
Mr. Samuel Rondberg
Designated Federal Officer
EPA Science Advisory Board
Washington, DC
Ms. Mary L. Winston
Management Assistant
EPA Science Advisory Board
Washington, DC
SUBCOMMITTEE
Dr. Paul R. Portney (Chair)
Resources for the Future
Washington, DC
Dr. Nancy E. Bockstael
Dept. of Agricultural and Resource
Economics
University of Maryland
College Park, MD
Dr. Trudy Ann Cameron
Dept. of Economics
University of California
Los Angeles, CA
Dr. Maureen L. Cropper
The World Bank
Washington, DC
Dr. A. Myrick Freeman
Dept. of Economics
Bowdoin College
Brunswick, ME
Dr. Charles D. Kolstad
Dept. of Economics
University of California
Santa Barbara, CA
Dr. Robert Repetto6
World Resources Institute
Washington, DC
Dr. Robert Stavins
JFK School of Government
Harvard University
Cambridge, MA
Dr. Thomas H. Tietenberg
Dept. of Economics
Colby College , Waterville, ME
Dr. W. Ep Viscusi
Harvard Law School
Cambridge, MA
Advisory
Mr. Thomas Miller
Designated Federal Officer
EPA Science Advisory Board
Washington, DC
Ms. Diana L. Pozun
Management Assistant
EPA Science Advisory Board
Washington, DC
VALUATION
SUBCOMMITTEE
Dr. Alan W. Maki (Co-Chair)
Exxon Company, USA
Houston, 'IX
Dr. Milton Russell (Co-Chair)
Joint Institute for Energy & Environment
and University of Tennessee
Knoxville, TN
Dr. Stephen M. Ayres
Medical College of Virginia
Virginia Commonwealth University
Richmond, VA
•^ Deceased.
6 Now with Stratus Consulting, Inc.,
Boulder, CO.
Vlll
-------
Dr. Caron Chess
Center for Environmental
Communications
Rutgers University
New Brunswick, NJ
Dr. Virginia Dale
Oak Ridge National Laboratory
Oak Ridge, TN
Dr. William H. Desvousges
Triangle Economic Research
Durham, NC
Dr. Thomas Dietz
Dept. of Sociology and Anthropology
George Mason University
Fairfax, VA
Dr. A. Myrick Freeman
Dept. of Economics
Bowdoin College
Brunswick, ME
Dr. Mark A. Harwell
Center for Marine and Environmental
Assessment
University of Miami
Miami, FL
Professor Jerry A. Hausman
Dept. of Economics
Massachusetts Institute of Technology
Cambridge, MA
Dr. Douglas E. MacLean
Dept. of Philosophy-
University of Maryland
Baltimore, MD
Dr. John W. Payne
Fuqua School of Business
Duke University
Durham, NC
Dr. Edella Sch lager
School of Public Administration and
Policy
University of Arizona
Tucson, AZ
Dr. Margaret Shannon
Center for Envir. Policy and
Administration
Syracuse University
Syracuse, NY
Dr. Paul Templet
Institute for Environmental Studies
Louisiana State University
Baton Rouge, LA
Dr. Terry F. Young
Environmental Defense
Oakland, CA
Dr. James Wilson
Dept. of Resource Economics and Policy
University of Maine
Orono, ME
Science Advisory
Mr. Thomas Miller
Designated Federal Officer
EPA Science Advisory Board
Washington, DC
Ms. Diana Pozun
Management Assistant
EPA Science Advisory Board
Washington, DC
RISK
SUBCOMMITTEE
Dr. Wayne M. Rachel (Co- Chair)
MELE Associates
Brooks AFB, TX
Ms. Marcia Williams (Co-Chair)
Putman, Hayes & Bartlett, Inc
Los Angeles, CA
Dr. Ann Bostrom
School of Public Policy
Georgia Institute of Technology
Atlanta, GA
Ms. Dorothy Bowers
Merck and Co., Inc
Whitehouse Station, NJ
Mr. Robert Frantz
General Electric Company
Cincinnati, OH
Dr. Nina Bergan
French SKY+
Oakland, CA
Ms. Mary A. Gade
Illinois EPA
Springfield, IL
Mr. Bradford Gentry
Center for Environmental Law & Policy
Yale University
New Haven, CT
Dr. Ricardo R. Gonzalez
Dept. of Radiological Sciences School of
Medicine
University of Puerto Rico
San Juan, PR
Dr. Michael Greenberg
Dept. of Urban Studies & Community Health
State University of New Jersey
Rutgers, NJ
Dr. Linda E. Greer
Natural Resources Defense Council
Washington, DC
Dr. Hilary I. Inyang
Center for Envir. Engineering & Sciences
Technologies
University of Massachusetts
Lowell, MA
Dr. Charles D. Kolstad
Dept. of Economics
University of California
Santa Barbara, CA
Mr. Terrence J. McManus
Intel Corporation
Chandler, AZ
Dr. Wm. Randall Seeker
Energy & Environmental Research
Corporation
Irvine, CA
Advisory
Ms. Kathleen White Con way
Designated Federal Officer
EPA Science Advisory Board
Washington, DC
Ms. Dorothy M. Clark
Management Assistant
EPA Science Advisory Board
Washington, DC
-------
INTEGRATED RISK PROJECT PEER REVIEWERS
IRP PEER REVIEW
COMMITTEE
Dr. M. Granger Morgan (Chair)
Dept. of Engineering and Public Policy
Carnegie Mellon University
Pittsburgh, PA
Dr. Henry A. Anderson
Wisconsin Bureau of Public Health
Madison, WI
Dr. William E. Bishop
Procter and Gamble Co.
Cincinnati, OH
Dr. Donald F. Boesch
Univ. of Maryland Center for
Environmental Science
Cambridge, MD
Dr. Richard J. Bull
MoBull Consulting, Inc.
Kennewich, WA
Dr. Terry Davies
Resources for the Future
Washington, DC
Dr. John D. Graham
Harvard Center for Risk Analysis
Harvard University
Boston, MA
Dr. Catherine Kling
Dept. of Economics
Iowa State University
Ames, IA
Dr. Debra Knopman
Progressive Policy Institute
Washington, DC
Dr. Morton Lippmann
New York University School of
Medicine
Tuxedo, NY
Dr. Warner North
NorthWorks Inc.
Belmont, CA
Dr. Richard Revesz
New York University School of Law
New York, NY
Dr. Bruce Tonn
Oak Ridge National Laboratory
Oak Ridge, TN
Science Advisory
Board Staff
Dr. John R. Fowle, III
Deputy Staff Director and Designated
Federal Officer
EPA Science Advisory Board
Washington, DC
Ms. Wanda R. Fields
Management Assistant
EPA Science Advisory Board
Washington, DC
-------
TABLE OF CONTENTS
1. Integrated Environmental
Decision-making 1
1.1 Environmental Integration: The Next Step 1
1.2 The Call for Integrated Decision-Making 2
1.3 Signs of Progress 4
1.4 Scope of the Project 5
2. A Proposed Conceptual
Framework 8
2.1 Overview of the Framework 8
2.2 Problem Formulation (Phase I) 11
2.3 Analysis and Decision-Making (Phase II) 12
2.4 Implementation and Performance
Evaluation (Phase III) 14
2.5 Building on Previous Frameworks 15
2.5.1 Integrated Aspects of Single
Stressors/Risks 15
2.5.2 Integrated Aspects of Multiple
Stressors/Risks 16
2.5.3 Considering Environmental Values 17
3. Working Toward
Implementation 18
3.1 Comparative Risk for Problem
Formulation (Phase I) 18
3.1.1 What We Have 18
3.1.2 What We Need 20
3.2 Risk Assessment for Analysis and
Decision-Making 22
3.2.1 What We Have 22
3.2.2 What We Need 22
3.3 The Analysis of Benefits and Costs
in Decision-making (Phases I and II) 23
3.3.1 What We Have 23
3.3.2 What We Need 24
3.4 Forming, Eliciting, and Considering
Public Values (Phases I and II) 25
3.4.1 What We Have 25
3.4.2 What We Need 27
3.5 Options Analysis (Phases I and II) 28
3.5.1 What We Have 28
3.5.2 What We Need 30
3.6 Performance Measures 31
3.6.1 What We Have 31
3.6.2 What We Need 35
4. Recommendations to the
Agency 37
5. Lessons Learned 43
6. References Cited 45
XI
-------
Xll
-------
1. INTEGRATED ENVIRONMENTAL
DECISION-MAKING
1.1 Environmental Integration:
The Next Step
Environmental protection in the United States
has progressed through at least two major steps.
The first step characterized the early days of the
1970s when the Environmental Protection Agency
(Agency) enjoyed both a public consensus and
Congressional mandates about what needed to be
done; e.g., clean up the air and make the waters
fishable and swimable. A second, more sophisti-
cated step was taken in the mid-1970s and honed
in the 1980s and 1990s when the "risk assess-
ment/risk management (RA/RM) paradigm,"
first proposed by the National Research Council
(NRC, 1983), was adopted and implemented by
the Agency. The RA/RM approach has been
especially useful in those instances in which the
damage and/or danger is not immediately evident
and some data and analysis are necessary around
which to form a consensus for action.
In this document, the Science Advisory Board
(SAB) describes the outlines of "the next step" in
environmental decision-making. The SAB presents
here a broad conceptual way of thinking about
environmental problems that responds to criti-
cisms of the fragmented approaches that dominate
current national strategies for protecting human
health and the environment. At the core of this
concept is integrated thinking about complex
environmental problems, integrated resources and
analyses to address the problems as they occur in
the real world, and integrated input from the
public and interested and affected parties.
Integrated decision-making is a natural next step
from both prior SAB and Agency projects that
considered individual risks in relation to each
other. In addition, it builds upon the significant
success that has grown from many of the reinven-
tion and reorganization activities in the Agency
during the 1990s.
As a result of conducting the Integrated Risk
Project (IRP), the SAB is proposing a conceptual
Framework for Integrated Environmental
Decision-making (IED) to guide the Agency in
the continuing evolution of environmental deci-
sion-making. In addition, IRP subcommittees
have described a number of tools and approaches
that, with further development and application,
may provide a means of developing some of the
information and analyses suggested by the
Framework. However, the SAB is not presenting
a "turn-key" procedure that will solve all of the
problems associated with the current decision-
making environment. Rather, the report points to
new directions that will broaden the decision-
making process in terms of both the factors that
should be considered and the parties who should
be involved in making and evaluating environmen-
tal decisions. It is left to the Agency and others
to apply the conceptual Framework to specific
problem sets, and thus to develop and implement
practical strategies and operations that will realize
the potential benefits of integrated environmental
thinking.
The evolution in environmental decision-
making called for in this report is not meant to
detract from past environmental accomplishments
nor to replace existing regulatory processes and
requirements. The concept of an integrated
1
-------
Prioritizing and
managing risks
pollutant by
pollutant and
medium by
medium can be
inefficient both
in reducing
the major
burdens of
environmental
impacts on
human health
and
ecosystems
and in
allocating
society's
resources..
Framework is intended to be further developed,
tested, and used in conjunction with existing envi-
ronmental management approaches to strengthen
the risk reduction programs now being imple-
mented and improve environmental management
in the future.
1.2 The Call for Integrated
Decision-Making
Environmental decision-makers currently
draw upon an eclectic mixture of data, tools, and
analyses to inform their decisions: ecological and
human health risk assessment; benefit-cost and
cost-effectiveness models; expanded risk commu-
nication and public participation; and measures for
monitoring the results of the decisions themselves.
Although these contributions are essential inputs
for decision-making, they have often been applied
unevenly and to relatively narrow issues. In the
area of human health risks, for example, assess-
ments often have been framed around single
stressors or classes of stressors in relatively specific
exposure situations. The deficiencies of such
highly focused assessments are increasingly appar-
ent; they sidestep the complexities, interrelation-
ships, and subtleties of environmental problems
as they actually confront society. Thus, a number
of recent studies have urged the Agency to begin
to address environmental issues in a more inte-
grated way (e.g., NAPA, 1995; Presidential and
Congressional Commission on Risk Assessment
and Risk Management, 1997).
Much of the fragmentation in EPA's approach
to the control of environmental problems has its
roots in the statutory framework that guides the
work of the Agency. From its formation in 1970,
the EPA has been given responsibility for imple-
menting a number of environmental statutes that
mandate targeted actions to control specific pollu-
tants in specific media (cf., Clean Air Act language
regarding particulates in air) or specific routes of
exposure (cf., Safe Drinking Water Act language
regarding priority pollutants in drinking water).
The focus on assessing and controlling chemical
contaminants pollutant by pollutant in single
media has resulted in a regulatory system that
is neither systematic nor comprehensive.
Nonetheless, the system has been largely success-
ful in controlling many of the targeted pollutants
and has provided a strong national underpinning
for an effective environmental protection program
comprised of federal, state, and local controls.
Despite these successes, there is a growing
consensus, both within and outside the Agency,
that a more integrated approach to environmental
management is needed. Prioritizing and managing
risks pollutant by pollutant and medium by
medium can be inefficient both in reducing the
major burdens of environmental impacts on
human health and ecosystems and in allocating
society's resources in the face of multiple demands
on limited budgets. Further, that piecemeal
approach ignores the integrated manner in which
hazards usually occur. Of still greater concern is
the possibility that such a fragmented approach
may cause us to overlook significant environmen-
tal problems while we busy ourselves with
comparatively minor improvements that
contribute little to the overall protection of human
health and ecosystems.
In some instances, current statutes and regula-
tions prevent the Agency from considering all
relevant risk, benefit-cost, or other information.
Further, as pointed out in a recent NAPA report,
there are "no established criteria that the Agency
might use to set priorities that cut across statutory
lines" (NAPA, 1995). The SAB views the issue of
statutory integration as a policy discussion and
outside the bounds of the present study. However,
the Board believes that even within the current
statutory framework, there are numerous oppor-
tunities for a more holistic assessment of risks and
risk management options. There are also opportu-
nities for more inclusive decision-making
approaches, both in terms of participation in the
-------
process by groups with diverse perspectives on the
issues and in terms of the focus of such reviews,
e.g., sectors, communities, industries, ecosystems,
and special groups within the population. In addi-
tion, there is a growing sense that a broader range
of factors should influence a given environmental
decision and that such factors should be more
systematically considered and articulated. These
factors include economic consequences (often
expressed in terms of benefits and costs) and
concerns of the public as expressed in their
underlying values.
Answers now are needed to questions that were
not asked when the predominantly pollutant-by-
pollutant regulatory system was first established.
These new questions focus on, for example,
cumulative risks to sub-populations or particular
geographic areas, the role of public involvement in
environmental decision-making, and the balance
between present and future consequences of deci-
sions. The proposed Framework for making inte-
grated environmental decisions should help direct
research and thinking about these issues in a way
that will generate answers to such questions.
New questions require new approaches to technical analyses and
public policy-making.
What are the most serious environmental risks facing children, or the elderly, across the nation?
What are the factors that pose an aggregate set of risks to everyone living in a particular geographic area?
What are the interrelated risks to human and ecological health associated with a particular industrial
sector?
Of all the risks affecting a geographic area or subset of the population, which are most serious, and
which are we most capable — economically, technologically, and politically — of limiting?
How can we achieve the right balance between protecting the health and the welfare of present and
future generations and assure ecological security for the long term?
What combinations of risk management tools — regulatory and non-regulatory, technological and
non- technological — can be used in concert to achieve environmental goals in particular communities
or ecosystems?
How can we measure our progress in solving integrated environmental problems and reaching our
long-term goals: protection of ecological integrity, human health, and quality of life?
How can worthy, but competing, individual and group goals be accommodated?
What is the distribution across the population of benefits and costs associated with environmental risks
and their possible management?
What is the appropriate role for public involvement in the assessment and management of
environmental risks?
... even within
the current
statutory
framework,
there are
numerous
opportunities
for a more
holistic
assessment of
risks and risk
management
options.
-------
1.3 Signs of Progress
In a 1983 publication entitled Risk Assessment
in the Federal Government: Managing the Process
(NRC, 1983), commonly referred to as the "Red
Book," an NRC panel laid out the elements of
RA/RM using terminology that came to be the
standard. These concepts were adopted by the
Agency in 1984 and have formed the basis for
much of the Agency's action to this day. In
summary, the NRC panel described the four steps
of risk assessment as hazard identification, dose-
response assessment, exposure assessment, and
risk characterization, where the latter was defined
as "the estimated incidence of the adverse effect in
a given population." In addition, the panel stressed
the scientific basis for risk assessment and the need
for both quantitative and qualitative expressions of
risk. Risk management was viewed as "a decision-
making process that entails consideration of politi-
cal, social, economic, and engineering information
with risk-related information to develop, analyze,
and compare regulatory options and to select the
appropriate regulatory response."
The Red Book was extremely useful in articu-
lating the risk assessment process and its relation-
ship to risk management. The paradigm was
expressed, however, in terms of single agents and
single health effects in humans. Since that time, the
Agency has developed risk assessment guidelines
to address a number of human health endpoints
(i.e., cancer, reproductive and developmental toxi-
city, and neurotoxicity), as well as guidelines for
exposure assessment. In addition, the Agency has
taken steps to consider more integrated exposure
scenarios; e.g., multi-route exposures to a)
mixtures of chemical agents associated with
Superfund sites (U.S. EPA, 1989), b) a fuller range
of combustion emissions (EPA, 1990; 1993a), and
c) multiple pesticides, in response to the Food
Quality Protection Act.
The Agency also has made significant progress
in adapting the Red Book paradigm to ecological
risk assessment. In 1992, the Agency released its
Framework for Ecological Risk Assessment (U.S.
EPA, 1992) which used the term "characterization
of ecological effects" to include both hazard identi-
fication and exposure assessment. The Framework
also added a) an explicit Problem Formulation
phase prior to the analysis of exposure and effects
in order to emphasize the importance of articulat-
ing the problem and b) a plan for analyzing and
characterizing risk prior to conducting specific
risk analyses. The resulting framework contained
three phases: Problem Formulation, Analysis, and
Risk Characterization. An expanded discussion
of ecological risk assessment principles and
approaches was subsequently provided by the
Agency in final Guidelines for Ecological Risk
Assessment (U.S. EPA, 1998). The guidelines note
that "although ecological risk assessments provide
critical information to risk managers, they are
only part of the environmental decision-making
process" (U.S. EPA, 1998). In addition to assessing
the relationship between a particular stressor and a
particular effect, the ecorisk guidelines set the stage
for considering multiple effects (including cascad-
ing effects) associated with a single stressor or
source, as well as multiple causes of an observed
effect or change in ecological condition. The
Agency has already applied the ecological risk
assessment paradigm to five watershed cases,
which included developing a conceptual model
for each that relates various stressors and effects
(for discussion, see SAB, 1997).
Four additional Agency developments, though
not reviewed by the SAB for scientific content,
merit mention here: a) guidance and support for
comparative risk analysis; b) extra-statutory
approaches to environmental protection; c) guid-
ance on planning and scoping for cumulative risk
assessment; and d) inclusion of interested and
affected parties in decision-making.
First, Comparative Risk Analysis (CRA) has
been defined by the Agency as "both an analytical
process and a set of methods used to systemati-
-------
cally measure, compare, and rank environmental
problems" (U.S. EPA, 1993b). The Agency, in its
Unfinished Business report (U.S. EPA, 1987), and
the SAB, in Reducing Risk (SAB, 1990), engaged
in comparative risk analyses. In its 1990 report,
the Board concluded that it was possible, on a
scientific basis, to distinguish between large risks
and small risks using a set of technical criteria. In
the years that followed, the Agency promoted the
wide use of CRA as a process for setting priorities
by considering multiple stressors and multiple
types of risks within specific regions, such as cities
or states, or for the nation as a whole.
Comparative risk analysis is intended principally
as a policy-development and broad resource-allo-
cation tool. In contrast to Unfinished Business
and Reducing Risk, however, state and local-level
comparative risk analyses have highlighted the
role of the public and stakeholder groups, in
addition to the scientific/technical community, in
defining risk priorities. Support for broader
inclusion of public values in decision-making
is a theme that has been echoed by a number of
recent reports (e.g., NRC, 1996; Presidential/
Congressional Commission on Risk Assessment
and Risk Management, 1997).
Second, during the 1990s, the Agency
experimented with a number of approaches to
re-inventing environmental protection, including
greater use of community-based decision-making,
voluntary cross-media emissions reductions, inte-
grated environmental agreements with states, and
voluntary regulatory reform efforts with an array
of stakeholders (U.S. EPA, 1999).
Third, the Agency has issued cumulative risk
guidance that directs program offices to "consider
a broader scope that integrates multiple sources,
effects, pathways, stressors, and populations for
cumulative risk analyses in all cases for which rele-
vant data are available" (U.S. EPA, 1997a). The
cumulative risk guidance also notes that on-going
Agency efforts to involve stakeholders "will
provide the solid basis for engaging interested and
affected parties in risk assessment and risk
management issues."
Fourth, the Agency has made progress in
recent years in including interested and affected
parties in the decision-making process. A recent
description of one of these processes— regulatory
negotiation in the microbial disinfection and
disinfection byproducts rules—can be found in
Pontius (1999).
Although the SAB has not reviewed the role
and adequacy of the science being used in the
planning and evaluation of these activities, these
endeavors signal movement in the Agency toward
more integrated and inclusive methods of environ-
mental decision-making. These experiments,
however, do not yet represent the mainstream of
EPA's efforts.
1.4 Scope of the Project
It is in this atmosphere that the SAB under-
took the task of revisiting its 1990 report,
Reducing Risk, to update and extend the thinking
about how science can best inform the decision-
making process. The Charge to the SAB from the
Agency included requests to update the risk rank-
ings in Reducing Risk using explicit scientific
criteria and the judgments of SAB panel members;
identify risk reduction opportunities and strate-
gies; identify uncertainties and data quality issues
associated with the risk rankings; assess costs and
benefits of risk reduction options; and propose a
new framework for assessing the value of natural
resources to society.
The initial charge also included a request
that the SAB explore techniques and criteria for
identifying emerging risks. However, the SAB
concluded that its report, Beyond the Horizon:
Using Foresight to Protect the Environmental
Future (SAB, 1995) provided criteria and sugges-
tions germane to this charge question and so did
not elaborate further on future risks as part of the
project.
...the SAB
concluded that
it could best
assist the
Agency not by
developing lists
of risk priorities
with associated
risk reduction
options, but by
investigating
approaches
that the Agency
might use for
developing
such priorities.
-------
After careful consideration of the Charge and
discussion with Deputy Administrator Fred
Hansen, the SAB concluded that it could best
assist the Agency not by developing lists of risk
priorities with associated risk reduction options,
but by investigating approaches that the Agency
might use for developing such priorities. In addi-
tion, the SAB hoped to place the various technical
analyses within the broader decision-making
context. Subcommittee Charges
The project, known as the Integrated Risk
Project (IRP), was guided by a Steering Com-
mittee (SC) and five specialized Subcommittees
who began work in Fiscal Year 1996. The Sub-
committees and their respective charges were as
follows:
a) The Steering Committee (SC), chaired by
Dr. Genevieve Matanoski, set the overall direc-
tion for the project by defining scope and
timetables. The SC met periodically over the
course of the project to: (1) assess the progress
and direction of the subcommittees; (2) ensure
that the results could be integrated into a
comprehensive decision process for identifying
current and future environmental risks; and (3)
review options for reducing risks in a holistic
context. The SC's efforts were designed specifi-
cally to illustrate the relationship among the
various factors influencing risk management
decisions; e.g., technical assessment of the risks
and risk reduction options, economic consider-
ations, equity considerations, and so forth.
b) The Ecological Risks Subcommittee (ERS),
chaired by Dr. Mark Harwell, was charged
with assessing and ranking risks to ecosystems
at the national scale, and suggesting ways in
which the risk ranking methodology could be
applied at smaller geographical scales; e.g.,
regional, state, or local. The group was also
asked to explore commonalities and differences
with the Human Exposure and Health
Subcommittee (HEHS) methodology, with the
aim of integrating the two ranking schemes.7
c) The Human Exposure and Health Sub-
committee (HEHS), co-chaired by Drs. Joan
Daisey and Bernard Weiss, was charged with
developing a methodology for assessing and
ranking risks to human health, considering
ways in which an integrated risk ranking could
be produced that includes both cancer and
non-cancer risks, and test the methodology for
a limited set of environmentally mediated
health issues. The Subcommittee was also
asked to explore commonalities and differences
with the ERS methodology.8
d) The Risk Reduction Options Subcommittee
(RROS), co-chaired by Dr. Wayne Rachel and
Ms. Marcia Williams, was charged with devel-
oping a methodology for selecting an optimal
set of risk reduction options with due regard
for the human health and ecological risks
(defined in terms of risks associated with
environmental stressors, locations, or expo-
sure/transport media). The Subcommittee
illustrated the methodology by applying it
to a small set of example problems.9
e) The Economic Analysis Subcommittee
(EAS), chaired by Dr. Paul Portney, was
charged with assessing current methods for
estimating costs and benefits associated both
with the implementation of risk reduction
7 The work of the ERS will result in a separate SAB report
from the Board's Ecological Processes and Effects
Committee.
8 The work of the HEHS is summarized in a Working Paper.
In addition, the Agency is exploring further development of
the prototypic Internet-based tool created in the course of the
project.
9 The work of the RROS will result in a separate SAB report
from the Board's Environmental Engineering Committee.
-------
strategies and with allowing risks to go unad-
dressed. The EAS was also asked to consider
those aspects of the "net benefits" equation
that cannot easily be monetized.10
The Valuation Subcommittee (VS), co-
chaired by Drs. Alan Maki and Milton Russell,
was charged to consider a new framework for
assessing the value of ecosystems to humans,
including ecological services and environmen-
tally mediated health and quality of life values.
The work of the VS was intended to provide a
wider societal view of risk and risk reduction
options than that derived from science-based
risk assessments and current methods of
economic
anal
Over the course of the project, the SC and
subcommittees held over 25 public meetings
and teleconference calls. Although most of these
meetings were held in Washington, D.C., public
sessions were also held in Berkeley and San
Francisco, CA; Atlanta, GA; New Orleans, LA;
and Baltimore, MD. While the bulk of the
Subcommittee work was completed in FY 1998,
the SC has continued its work of producing a
synthesis report, subjecting it to peer review, and
responding to comments from the reviewers.
This report contains several sections. Section 2
synthesizes much of the SAB's deliberations into a
conceptual Framework for making integrated
environmental decisions. Section 3 describes six
of the major elements that contribute to the
Framework, identifying specifically the informa-
tion/tools that are already available and the areas
in which additional work is particularly needed.
The major recommendations of the SAB are laid
out in Section 4. In the closing Section 5, the SAB
briefly reviews some of the "lessons learned" in
the course of carrying out the IRP.
10 The work of the EAS will result in a separate SAB
Backgrounder that provides an introduction to benefit-cost
analysis, its strengths, and its limitations.
11 The work of the VS is summarized in a Working Paper that
could serve as input to an SAB/Agency Workshop to
explore the issue further with a broader range of participants.
7
-------
2. A PROPOSED CONCEPTUAL FRAMEWORK
2.1 Overview of the Framework
The approach to integrated environmental
decision-making that emerged from the SAB's
deliberations is captured in the conceptual
Framework for Integrated Environmental
Decision-making (IED) in Figure 1. The IED
Framework recognizes that risks often are experi-
enced simultaneously and are cumulative (i.e.,
additive, synergistic, and/or antagonistic); that
efforts to manage one risk may have impacts —
positive or negative — on other risks; that benefit-
cost scenarios may be affected by the scope of the
problem definition; and that values deliberation is
essential to the development of environmental
decisions.
As illustrated in Figure 1, the Framework lays
out a series of straightforward questions. What are
the most important environmental risks ? What are
our environmental goals? What are the best risk
reduction opportunities? How can we achieve our
goals and objectives? How will we know whether
or not we are meeting our goals? What modifica-
tions in our approach are needed to improve envi-
ronmental results? Are we meeting our objectives?
Is the nature of the problem changing? Finding
answers to these fundamental questions requires
application of scientific and technical assessment
and analysis techniques, as well as political, policy,
and values-driven choices. By fostering a more
integrated look at environmental goals and priori-
ties, the Framework should help us to make
choices not only about which actions to take, but
also about which actions are not worth taking.
The explicit acknowledgment of the trade-offs
required to achieve multiple, often competing,
goals is an important part of setting priorities.
The proposed Framework consists of three
interacting phases:
Phase I—Problem Formulation, which involves
preliminary analysis and comparison of risks,
establishment of goals, and preliminary analysis of
risk reduction options. The "problem" may be
defined in terms of either a single risk (e.g., one
stressor, but associated with multiple sources with
multiple routes of exposure) or multiple risks (e.g.,
environmental stressors associated with a particular
geographic location, possibly including multiple
sources, pathways, types of receptors, and effects).
Phase II—Analysis and Decision-Making, which
includes in-depth analysis of risks and projected
risk reduction under possible management scenar-
ios and selection of a preferred option or set of
options, based on criteria such as feasibility, cost-
effectiveness, seriousness of the risks addressed,
and equity.
Phase III—Implementation and Performance
Evaluation, in which preferred management options
are implemented and the environmental results are
monitored and evaluated, thereby providing impor-
tant feedback so that management approaches can
be modified ("adapted"), as necessary.
The various activities that take place during the
three phases of decision-making are discussed
further below, and in the outputs from the IRP
subcommittees.
8
-------
Figure 1. Framework for Integrated Environmental Decision-making
- Information
— Expert Judgment
— Values
- Information
- Expert Judgment
- Values
- Legal and
Institutional Milieu
PHASE I
PROBLEM FORMULATION
(What are the most important environmental risks?
What are our environmental goals?)
Risk Comparisons Goal Setting
Preliminary Options Analysis
I
PHASE II
ANALYSIS AND DECISION-MAKING
(What are the best risk reduction opportunities?
How can we achieve our goals and objectives?)
Risk Assessment Screening/Selection
Options Analysis Performance Measures
REPORT
CARD
(Is the nature
of the problem
changing?)
REPORT
CARD
(Are we meeting
our objectives?)
PHASE III
IMPLEMENTATION and
PERFORMANCE EVALUATION
(How are we doing?)
Implementation Monitoring and Reporting
Information Evaluation
-------
The straightforward structure of the Framework
diagram belies the complexities involved in putting
the concept of integrated decision-making into prac-
tice. For example, the Framework envisions itera-
tive, analytic/deliberative interactions (NRC, 1996)
involving and eliciting values from the Agency, the
public, and interested and affected parties. There has
been an evolution in the thinking and practice at the
Agency about who should be involved in the assess-
ment and management of risks, with a recognition
that public values underlie the selection of specific
goals and risk reduction options. Although it is not
always clear how best to include a broader range of
participants in decision-making, the Agency has
begun to implement various aspects of integrated
decision-making and is "learning by doing."
Characteristics of Integrated Environmental Decision-making
Transparency
All parties should be able to follow and understand how the decision was reached.
Flexibility
Integrated decision-making approaches should be applied in a flexible manner depending on the
specific circumstance; i.e., where appropriate, to permit valid short-cuts, to eliminate unnecessary
procedures, and so to expedite the process of decision-making and implementation.
Dynamic process design
The technical analyses required to implement integrated decision-making should be iterative and
interconnected. For example, during the Problem Formulation Phase, problem scoping and definition
and preliminary analysis of options will affect the development of goals, and vice versa. Some iteration
is also required between Problem Formulation and Analysis, since preliminary analyses will often
point out missing elements in the problem definition or inconsistencies in goals.
Explicit feedback, interaction, and cooperation
Integrated decision-making approaches should involve cooperation and open, continuing
communication among scientists, managers, members of the public, and others involved in the
different phases of the Framework.
The use of information from many sources
Integrated decision-making approaches should draw upon concepts and methods originating in many
different scientific, technical, and scholarly fields (e.g., physical and biological sciences, public health,
environmental engineering, political science, social science, philosophy, and economics), as appropriate
for any given case.
A way of thinking about environmental problems
Integrated environmental decision-making is not just a series of methodologies, but rather is a way of
thinking, in a whole and complete way, about any environmental problem in order to maximize the
efficient reduction of aggregate risk to populations or ecological systems.
10
-------
There are also new technical tools available
(e.g., improved computer hardware, increasingly
powerful computer models, and greater availabil-
ity of geographic information system (GIS) soft-
ware) that have improved the Agency's ability to
collect, analyze, and disseminate information in
ways that will enhance integrated decision-
making. In addition, advances in the social
sciences are providing improved techniques for
helping people develop considered values, for elic-
iting and using public values in decision-making,
for communicating technical information in ways
that are helpful to non-expert participants, and for
understanding people's choices and preferences.
Although integrated decision-making requires
the involvement of a broad spectrum of partici-
pants (e.g., scientists, engineers, economists, deci-
sion-makers, and the public), the different groups
have unique roles to play. In other words, the
Framework does not imply that "everyone must
be involved in everything all the time." For exam-
ple, just as scientists are not expected to provide
the perspective of the general public, members of
the general public are not expected to conduct the
technical analyses of scientists. Decision-makers,
after considering the various sorts of information
(data, views, and judgments) generated in Problem
Formulation (Phase I) and Analysis and Decision-
making (Phase II), must ultimately make the deci-
sion. At the same time, it is important that the
various groups maintain effective communication
with each other, informing one another of their
perspectives and insights, so that the collective
wisdom of all of the participants is brought to
bear on the problem.
2.2 Problem Formulation (Phase I)
Integrated decision-making should begin with
Problem Formulation, in which risk assessors, risk
managers, and interested and affected parties seek
agreement through extensive dialogue and discus-
sion on what analytical and deliberative steps need
to be taken by whom, by when, and why — if not
how. This initial Phase includes three related tasks:
a) Risk Comparisons, in which sets of risks—
including risks to ecological systems, human
health, and/or quality of life—are ranked or
rated, and a set of risks selected for detailed
consideration in the second phase;
b) Goal Setting, in which the participants agree
on goals relating to the risks of concern and
measures that will be used to evaluate progress
towards those goals; and
c) Preliminary Options Analysis, in which an
initial range of risk reduction options is identi-
fied and considered in terms of the estimated
total reduction of risk and likely benefit-cost
of each.
Risks to Quality of Life
Although "quality of life (QOL) risks" is an
imprecise term that means different things to
different people, examples of risks often
included in this category are aesthetic,
economic, and equity impacts, as well as
effects on peace of mind, cultural or
community identity, and recreational
opportunities. In most of the state
comparative risk projects, risks to human
welfare or quality of life were considered
separately from human health and ecological
risks by a non-technical subcommittee that
developed criteria and produced a ranked list
of QOL risks. The EPA (1993) developed a
guidebook for assessing quality of life risks
that provides a starting point for such
assessments.
11
-------
While scientists
can help to
characterize
risks, they are
not uniquely
qualified to set
priorities
among them
and broader
deliberation is
essential.
The dialogue among the participants in Phase I is
designed to focus the problem and to ensure that
there is a logical consistency between the stressors
and risk being considered, on the one hand, and
the risk reduction opportunities that may be avail-
able, on the other. In this initial Phase, the discus-
sions are at the level of planning, scoping, and
screening, rather than the detailed analyses
conducted in Phase II.
Scientists play an important role in Phase I by
collecting, analyzing, and presenting data in such a
way that all parties can appreciate the type and
magnitude of the problem(s) under discussion. This
activity will generally involve all four parts of risk
assessment, including assessment of exposures expe-
rienced by special populations and/or ecological
resources. Planning, scoping, and screening —
including selection of endpoints of concern — also
requires explicit input of societal values and stake-
holder participation. For instance, while some of
the ecological endpoints may be chosen because of
their role in a valued ecosystem, there may also be
ecological endpoints chosen because of their direct
significance to society. Examples of the latter include
both economically important species and "charis-
matic" species. Similarly, in integrated decision-
making, judgments may have to be made about
diverse health endpoints, such as cancer risks in the
general population and the risk of reproductive/
developmental risks in children. While scientists
can help to characterize such risks, they are not
uniquely qualified to set priorities among them and
broader deliberation is essential. Finally, decision-
makers also play an important role during Problem
Formulation; in addition to bringing the scientific
and other resources of the Agency to bear on the
problem, they also should help to identify the range
of potential decisions and viable management
options, while examining economic, political, or
other constraints on those options. Decision-makers
also serve as managers of the overall process.
During the development of the IED frame-
work, the SAB emphasized that, although infor-
12
mation on the nature and extent of risks is critical
to Problem Formulation, the ranking of risk
reduction opportunities is equally important
when establishing risk priorities. One means of
describing the relationship between risk rankings
and possible priorities is pictured in Figure 2.
In summary, participants in the initial Phase —
scientists, decision-makers, and interested and
affected parties nshould seek agreement through
an open, yet structured, exchange of information,
concerns, opinions, and values (i.e., iterative,
deliberative-analytic dialogue as described in
NRC, 1996) on a series of issues that will define
the problem so that more in-depth analyses can
be conducted in the subsequent phase of decision-
making.
2.3 Analysis and Decision-Making
(Phase II)
In Phase II of the IED Framework, the analysts
take the information and general directions gained
in Phase I and generate more detailed, more fully
supported assessments of risks and risk reduction
options. For integrated decision-making, options
analysis should include consideration of risk
reduction opportunities with regard to their tech-
nical feasibility, aggregate risk reduction to be
obtained (e.g., reductions in "target" risks and
collateral reduction in all affected risks), full
economic consequences of various risk reduction
scenarios, and so forth. Decision-makers also
should request analysis of potential options with
regard to sustainability, equity, and other potential
decision criteria.
Options analysis generally is more "analytic"
than "deliberative" (NRC, 1996) although a
continued level of interaction between the
participants in the overall process (scientists, risk
managers, and interested and affected parties) is
important. Options Analysis is also more
resource-intensive than Problem Formulation.
In the decision-making portion of Phase II, the
-------
Figure 2. The Relationship Between Risk Rankings and Possible Priorities
Agency or other decision-makers should a) utilize
outputs from the analyses of risk and risk reduc-
tion options, b) consider widely-held public
values, as well as the views of participating stake-
holders, c) consider the legal, economic, and insti-
tutional constraints, and d) ultimately, make the
decision. Clearly, this process is not totally scien-
tific. However, the best science should inform and
contribute to decision-making. Developments in
the social and decision sciences, for example, are
providing improved methods for value elicitation
and multi-attribute decision-making. The docu-
mentation supporting the decision should make
explicit a) the implications of the chosen manage-
ment option(s) to the health of ecological or
human systems, b) the economic costs and bene-
fits associated with the selected option, and c) the
societal values that both influence and are affected
by the decision, including values relating to
economic efficiency, sustainability, equity, and
quality of life. Integrated decision-making requires
explicit consideration of the trade-offs involved in
pursuing multiple environmental goals and/or in
simultaneously pursuing environmental and non-
environmental goals. In some cases, analysis may
indicate that a particular management option is
not worth doing because of the greater good that
might be achieved by investing those resources
toward the achievement of another goal.
It is important that the scientific and technical
analyses prepared during Phase II articulate clearly
the uncertainties associated with the estimates of
risk, the estimates of risk reduction that may be
achieved by different management options, and
the economic assessments of various risk manage-
ment scenarios. Integrated decision-making does
not eliminate the uncertainties associated with
making decisions. However, by encouraging an
13
Integrated
decision-making
requires
explicit
consideration
of the
trade-offs
involved in
pursuing
multiple
goals...
-------
The SAB
advocates the
use of
Environmental
Report Cards
to document
performance
and outcomes
of risk
reduction
activities..
open and comprehensive examination of environ-
mental problems, integrated decision-making
should lead to a clearer identification of the
nature, extent, and consequences of the uncertain-
ties associated with the available information.
In any event, environmental decision-making
must proceed in the presence of uncertainties, and
nothing in the proposed Framework should be
construed as precluding environmental decisions
simply because uncertainties remain.
2.4 Implementation and Performance
Evaluation (Phase III)
In the third phase of integrated decision-making,
the chosen risk reduction option or set of options
is implemented and evaluated over time to ascer-
tain the extent to which it is achieving the desired
environmental outcomes. The specific activities
required to implement an environmental decision
will depend on the suite of management options
selected for any particular problem or set of prob-
lems. The Agency has considerable experience
with many risk reduction options (e.g., adopting
best available technology, imposing permit limits,
and regulatory negotiation), and is gaining valu-
able new experiences with others (e.g., National
Environmental Performance Partnership System).
In contrast to implementation, the perform-
ance evaluation process is fundamentally rooted in
science because it is science that can translate the
public's overarching environmental goals (e.g.,
improved health, sustainable ecosystems) into
discrete, measurable components. Accordingly,
science is essential in deciding what to monitor,
i.e., specifying the endpoints of concern for the
systems at risk and identifying the specific meas-
ures that need to be monitored in order to charac-
terize the status and trends for those endpoints
with respect to the environmental goals. Further,
Desired Outputs from Problem Formulation:
The initial goals for the decision-making exercise, including environmental goals to be achieved;
Which environmental problems/stressors/systems will be included and which will not, and the reasons
for these decisions;
The health, ecological, and quality-of-life effects of concern;
The spatial, temporal, and organizational dimensions of the problem;
Relevant data and models, and possible approaches to data analysis;
Scoping of the uncertainties involved and research needed to significantly reduce critical uncertainties;
Initial review of the range of options available to reduce risks, considering likely economic, political, or
other constraints;
The endpoints upon which the condition of the ecological, human health, or societal systems ultimately
will be judged; and
The types of factors that will be considered when reaching a decision.
14
-------
issues such as spatial and temporal variability,
measurement error, and time lags must be
addressed explicitly in order to demonstrate envi-
ronmental conditions and to separate signal from
noise. And finally, reference conditions and
benchmarks or milestones along the way to the
desired system conditions must be defined scien-
tifically so that meaningful and measurable
performance criteria for success or failure can be
defined.
The SAB advocates the use of Environmental
Report Cards to document performance and
outcomes of risk reduction activities at several
levels and for different audiences. As noted in
Figure 1, a successfully implemented performance
evaluation system will generate several important
feedbacks affecting the first and second phases.
One feedback loop is to the Analysis and
Decision-Making Phase, reporting on how well
the selected risk reduction options and strategies
are achieving the environmental goals. This feed-
back loop allows for adaptive management and
changes in implementation activities, including the
possible need to identify and analyze additional
options to further reduce risks. Information also
feeds back into Phase I allowing re-examination,
as needed, of the initial goals, risk rankings or
other aspects of Problem Formulation. As risk
reduction options are put into place, for example,
particular risks should be reduced, and a new
comparison of risks may be appropriate. In addi-
tion, there may be a shift in or redefinition of soci-
etal values over time, requiring different sets of
environmental goals and, therefore, different envi-
ronmental decisions. In short, integrated decision-
making emphasizes the need for performance
information at several points in the process and
for systematic review of environmental decisions
in light of new scientific understanding, shifts in
societal values, changes in stakeholder preferences
and available resources, and/or responses of the
environment to previous decisions.
2.5 Building on Previous Frameworks
The FED Framework builds upon several
previous efforts, in particular the risk assessment/
risk management model described by the National
Research Council (NRC, 1983), the update to that
report (NRC, 1994), the ecological risk assessment
framework (U.S. EPA, 1992), the report of the
Presidential/Congressional Commission on Risk
Assessment/Risk Management (1997), and the risk
characterization process described by the NRC
(1996), which focused on the interaction between
analytic and deliberative processes in decision-
making.
Building on this base of information, the FED
Framework is intended to integrate a range of
factors that are important in modern risk assess-
ment and risk management (see box). In particular,
the process envisioned in the Framework moves
beyond these earlier efforts in the three areas
discussed below.
2.5.1 Integrated Aspects of Single
Stressors/Risks
Although the SAB emphasizes that the Agency
should consider multiple stressors in an integrated
manner, it recognizes that integrated thinking also
can enhance the decision-making process for
single stressors, which have typified many Agency
decisions in the past. When a single stressor is
considered, integrated thinking should expand the
previous approaches by:
a) Characterizing stress-effects relationships
across all systems and populations;
b) Exploring a broader range of risk reduction
options, some of which may be qualitatively new;
c) Assessing the benefits and costs of each option,
including explicit consideration of non-mone-
tary benefits and costs;
15
-------
d) Assessing the magnitude and nature of the
aggregate risk reduction associated with each
option;
e) Involving scientists, options analysts, stake-
holders, and risk managers collectively at vari-
ous points throughout the process; and
f) Establishing a performance evaluation "report
card" to characterize the efficacy of the imple-
mented risk reduction option and to signal
both the need and opportunity for adapting
the original management decision.
2.5.2 Integrated Aspects of Multiple
Stressors/Risks
One of the most important extensions of inte-
grated decision-making is its focus on multiple
stressors. The SAB suggests prototypic structured
approaches that can be used to begin to explore
multiple ecological risks and multiple human
health risks. In some cases these approaches will
lead to consideration of combinations or groups
of stressors; e.g., all organophosphate pesticides,
all automobile emissions, or all factors leading to
local loss of biodiversity.
Types of Integration in the IED Framework
Risk Comparisons
Consider a wide range of environmental risks simultaneously so that the seriousness of risks can be
characterized relative to one another.
Integrated Risk Assessment
Develop scientific data and analytical methods for assessing risks from multiple stressors, from multiple
sources, by multiple exposures, resulting in multiple effects/outcomes, in order to represent real world
situations more accurately.
Integrated Analysis of Management Options
Investigate options to reduce subsets of ranked risks, rather than considering single risks in isolation, in
order to achieve greater aggregate risk reduction.
Integrated Analysis of Economic Consequences
Identify the full range of benefits and costs, both monetized and non-monetized, associated with
reduction of multiple risks.
Integrated Performance Information
Use performance evaluation measures to better characterize conditions and to adapt implemented
actions appropriately.
Integrating Multiple Disciplines and Points of View
Understand and utilize information from all concerned parties in decision-making.
16
-------
Initial steps in integrated decision-making
should produce relative rankings of risks to
human health and ecosystems, independently.
In the context of integrated decision-making,
however, it is also important to go further and
develop integrated health and ecological risk rank-
ings. A merger of such qualitatively different risks
into a single scheme requires even larger value
judgments than the ranking of similar risks and
should be informed by a broader societal valua-
tion process.
Qualitative differences in human health and
ecological risks should be considered during
Problem Formulation. For example, the focus of
health and ecological risk assessment is different;
that is, the focus of health risk assessment is an
individual within a single species, while the major-
ity of ecological risk assessments focus on entire
populations of one or many species. More gener-
ally, ecological risk assessments often address the
integrated risks to a prescribed region, such as a
watershed, versus health risk assessments that are
generally more national in scope. This difference is
reflected in the different types of stressors of
concern; cf., carcinogens for humans vs. habitat
fragmentation. However, in those instances in
which there are common stressors of concern (e.g.,
chlorinated pesticides or climate change) or where
effects of a stressor on an ecological system
produce effects on human health or quality of life
(e.g., habitat alteration that affects the range and
activity of disease vectors and infective parasites,
or changes in the abundance of commercially
important or endangered species), there is an
opportunity for some merging of concerns to take
place. Integrated decision-making should combine
risks into logical groupings, e.g., those with a
common source or pathway, in order to identify
risk reduction opportunities across stressors. In
order to be successful, this analysis requires open,
publicly-accessible, and frequent dialogue among
those who assess and compare risks, those who
determine methods for reducing risks, and those
who make the final decisions. In summary, as inte-
grated thinking is developed to address multiple
stressors, it should:
a) Lead to a more realistic comparison of risks to
humans and to ecosystems, where some of
those risks may be posed by combinations of
related stressors;
b) Lead the Agency to consider in a systematic
fashion all of the appropriate factors related to
risks in a given circumstance, including aggre-
gate risk, economic factors, and societal values;
and
c) Lead to action that will increase the reduction
in aggregate risk posed by a combination of
stressors in a given circumstance.
2.5.3 Considering Environmental Values
The SAB's integrated environmental decision-
making Framework highlights the analytic/delib-
erative process (NRC,1996). It is through such a
process that societal values intersect with the
scientific risk characterization and risk reduction
analyses. The integrated decision-making concept
emphasizes the role and timing of stakeholder and
decision-maker inputs to the analytic processes. It
explores more deeply the valuation of environ-
mental outcomes and risks and the need to include
not only the concepts of economic efficiency and
willingness-to-pay, but also issues such as environ-
mental sustainability and equity. Societal values
constitute the milieu in which integrated environ-
mental decision-making occurs, forming the basis
of goals for improved social welfare, improved
ecological/health conditions, and long-term
sustainability and equity. Ultimately, it is in the
realm of social values that the success or failure of
environmental decisions will be judged.
17
Integrated
decision-
making should
combine risks
into logical
groupings,
e.g., those with
a common
source or
pathway, in
order to identify
risk reduction
opportunities
across
stressors.
-------
3. WORKING TOWARD IMPLEMENTATION
The SAB
suggests
prototypic
structured
approaches
for rating
multiple
ecological
risks or
multiple human
health risks.
The elements within the Phases of the
Framework and how they interact with each other
are at the heart of an integrated approach to deci-
sion-making. This section of the report elaborates
upon many of those elements and points the way
for further development of each in order to meet
the needs of integrated decision-making.
3.1 Comparative Risk for Problem
Formulation (Phase I)
The term Risk Comparisons is used in this
report to denote the characterization and ranking of
risks posed by environmental stressors, where an
environmental stressor is any physical, chemical, or
biological change or agent that could affect ecologi-
cal or human health systems. The objective of risk
comparisons is to determine how members of an
array of risks compare in magnitude and scope to
each other, so that a subset of risks can be identified
for simultaneous evaluation and decision-making.
As depicted in the Framework, the resulting infor-
mation on comparative risk should be integrated
with other relevant technical information (including
expected costs and benefits), expert judgment, and
values brought into the process through delibera-
tion with interested and affected parties. This inter-
action in Phase I should lead to agreement on goals,
limitations on analysis, and a tentative identification
of possible risk reduction options.
3.1.1 What We Have
One approach to Risk Comparisons, devel-
oped by the Ecological Risks Subcommittee (ERS)
18
and applied to the comparison of ecological risks,
utilizes an expert group to develop and weight
risk ranking factors in order to produce the
group's consensus judgment of the relative risks
associated with various environmental stressors.
The ERS approach considers the two fundamental
components of ecological risk—the stress or expo-
sure regime and the response or ecological effects
regime— for each stressor of concern. Because
each stressor may result in multiple effects on
different entities in an ecosystem, the goal of the
risk comparisons is to characterize the dominant
relationships between the environmental stressors
and ecological effects. Ecosystem-specific stress-
effect relationships are then transformed into a
relative ranking of risk at a specific spatial scale
(e.g., regional, national, or global) by applying a
series of numerical factors that, in the Subcom-
mittee's view, reflect the relative impact of the
effects on the ecosystem(s) at risk.
The ERS categorized ecological risks from
chemical, biological, and physical stressors into
five narrative categories: Highest, High, Medium,
and Low Ecological Risks, and Unknown But
Potentially Important Risks. Stressors ranked by
the ERS as posing Very High ecological risks are
listed in Table 1. These conclusions represent the
consensus expert judgment of the Subcommittee,
based on general ecological principles. The trans-
parency of the ERS methodology allows others to
evaluate the scientific bases for the risk rankings
and to reach their own conclusions. A different
group of experts, using a different set of risk
factors or weightings, might come to different risk
rankings. Nevertheless, the conclusions of the
-------
Table 1. Environmental Stressors Posing
the Highest Risk at the National Scale:
Conclusions of the ERS*
Hydrologic Alterations
Harvesting Living Marine Resources
Habitat Conversion
Climate Change
Introduction of Exotic Species
* Given current efforts to manage risks
ERS are consistent with other previous national
rankings of ecological risks (e.g., EPA, 1987; SAB,
1990) and suggest that the present greatest risks to
the nation's environment result from physical and
biological, rather than chemical, Stressors. Further
detail on the ERS method and findings will be
provided in a separate SAB report.
A second approach to Risk Comparisons, less
fully developed by the Human Exposure and
Health Subcommittee (HEHS) and illustrated
using human health risk issues, employs Internet-
based polling of individuals (experts) for their
(professional) judgment of the relative degree of
risk associated with various Stressors, and solicits
information on which factors most influenced this
judgment and the degree of confidence each indi-
vidual (expert) assigns to his/her assessments (see
Figure 3). Thus, this method for developing risk
comparisons also provides information on the
range of recorded opinions.
The HEHS developed a prototype for an
Internet-based system for polling and characteriz-
ing expert judgments about the risks posed by
various environmental Stressors. The prototype
restricted-access Internet site includes an Entry
Page, where survey participants register, followed
by a Ranking Page (Figure 3), where participants
are asked to rate a series of Stressors as Very High,
High, Medium, Low, Very Low, or Unknown
Risk. The Ranking Page also asks for the factors
that influenced the rating and a confidence rating
for each risk ranking. The Ranking Page contains
an Information Window, which allows partici-
pants to access summary information for each
stressor. The HEHS recommends that information
be provided on relevant exposure routes and path-
ways, population exposed, average dose, toxico-
logical and health effects data, and so forth. An
example of the output data that would result from
the polling approach is shown in Figure 4.
While there were some differences, the two
subcommittees proposed quite similar sets of
factors that they felt were important in evaluating
the comparative risks associated with various envi-
ronmental Stressors (see box). Both subcommit-
tees also recommended that information on each
of the factors be assembled in a summary
format—a risk data sheet— to assist non-expert
participants in Problem Formulation to assess the
relative priority that should be accorded each
stressor.
The consensus approach or the individual
polling approach could be applied to either human
health or ecological risk comparisons using Stres-
sors and rating factors specific to each group. The
methodologies also could be used to elicit stake-
holder or public views on risk priorities by
expanding the composition of the surveyed or
empaneled group. Clearly, there will be variation
among the results of the various groups — even
among groups of experts — because of differences
in level of expertise and knowledge about a partic-
ular stressor or because of differences in the
degree of concern that individual participants
attach to various health or environmental
outcomes associated with a stressor.
While the examples developed by the ERS and
the HEHS provide illustrations of the type of new
tools we believe need to be developed, these
19
-------
Figure 3. Internet-based risk rating form. The respondent selects a stressor, then
accesses the data sheet by selecting the Information Window.
Stressor Risk Characterization and Health Risk Ranking
Data Sheet
Edit/Update Risk Date
For
Or. Bernard
Urlv {/ Rochester
WB s s^cnvmcd . roc hosier . edu
Loch DM* *nd *•**<• YH31W «3m* v~4 P-,iss-wrj*e,
Update your current entry on Stressor: Halogenated Hydrocarbons
Relative Rlbk
Get INFORMATION
Haiogenaled Hydrocarbons
. . . . . .!'_ . :.--,:.
specific examples were created with limited
consideration of the literature in modern social
and decision science. As the Agency develops and
refines such tools for future use, it will be impor-
tant to involve experts from these fields.
3.1.2 What We Need
During the design of the Framework, the SAB
participants acknowledged that technical risk
rankings, in isolation, offered insufficient guidance
for policy decisions. Given the multitude of prob-
20
lems and issues to be addressed, a more compre-
hensive and systematic framework for analyzing
and reducing environmental health, ecological, and
quality of life risks appeared necessary. During
Problem Formulation, the Agency needs methods
for comparing risks that are robust, transparent,
effective, and inexpensive. As noted, some initial
steps have been taken for ranking risks within
categories; e.g., human health, quality of life, or
ecosystem risks.
Additional work, however, needs to be done
on methods to incorporate non-expert values in
-------
risk comparisons and priority-setting, including
inputs from the general public as well as interested
and affected parties, and to integrate the analysis
of multiple risks.
The assessment and comparison of risks and
the definition of environmental goals are inter-
related. In other words, scientific information on
the nature and extent of various risks influences
the relative priority that society places on those
risks. In addition, other factors (e.g., dread, previ-
ous experiences, the degree to which exposure is
voluntary, and underlying values) also influence
the relative priority assigned by the public to envi-
ronmental risks. For this reason, the ultimate
priorities for action that emerge during Problem
Formulation should be a product of the interac-
tion between risk comparisons by experts and the
more inclusive goal-setting processes, supple-
mented by a preliminary assessment of risk reduc-
tion potential of the various options. The purpose
of ranking risks and ranking risk reduction oppor-
tunities is to assist decision-makers to allocate
resources in Phase II.
The interaction between risk comparisons and
goal-setting could be enhanced if the risk compar-
isons are structured so as to identify linkages
between health, ecological, and quality of life
risks; e.g., common stressors or root causes, indi-
rect affects on health of an ecological impact.
Defining risk problems using a common dimen-
sion (e.g., in terms of stressors) will facilitate the
identification of both direct and indirect effects of
stressors, and of those stressors that affect both
humans and natural systems. Approaches for
achieving this sort of integrated analysis will
require further development.
Figure 4. Synthetic plots of risk versus confidence (12 respondents): A) high
variability; B) low risk, mid to high confidence; C) high risk, low confidence;
D) high risk, high confidence.
A
VL
VH
Risk Racing
•
•
•
«
«•
«
•
«
•
«
*
VH
H
M
I,
VL
Confidence
B
VL
VH
Risk Racing
• ••
*•**«
*•**
VH
H
M
I,
VL
Confidence
C
VI,
VH
••*•
**
VH
H
M
L
VL
Confidence
D
VI,
H VH
• •*
• ••
• ••
• ••
VH
H
M
L
VL
Confidence
21
-------
3.2 Risk Assessment for Analysis and
Decision-Making (Phase II)
3.2.1 What We Have
The Agency has been a leader in developing
guidelines for risk assessment. Beginning with the
first version of cancer risk assessment guidelines in
1976, the Agency now has risk assessment guide-
lines in place for six human health endpoints. In
addition, the Agency broke new ground with the
publication of ecological risk assessment guide-
lines in 1998. These guidance documents —
and their updates — provide state-of-the-art
information on how to perform detailed analytical
assessments of risk that can inform the decision-
making process.
In 1997, the Administrator signed a
Cumulative Risk Policy that committed the
Agency to move in the direction of assessing the
multiple effects of multiple stressors through
multiple routes of exposure, which is certainly
consistent with the Framework. This Policy,
however, promises a level of sophistication in risk
analysis that currently does not exist.
3.2.2 What We Need
Most of the risk assessment guidelines that are
in place generally address single stressors resulting
in single endpoints; e.g., the cancer risk from
arsenic. However, the public is increasingly asking
more holistic questions; e.g., "What is my total
risk of living in this environment?" Implicit in the
question is consideration of the total risks from
varying, complex mixtures of stressors, as well as
the cumulative impacts of multiple exposures,
experienced over time and space, to populations
who might have differing degrees of susceptibility
and resources to address the consequences of
those risks. In addition, current guidelines imply
that the goal of reducing risk to a specified level
(e.g., ID-4 or 1O6 lifetime cancer risk) is to be
pursued regardless of the associated benefits and
Correspondence of Human Health and Ecological Risk Comparison Factors
Human Health Risk Factors
Size of population affected
Particular subpopulations at risk
Severity and persistence of effects
Persistence in environment and/or human body
Percent of attributable incidence
Potential future risk
Ecological Risk Factors
Proportion of resource at risk
Distribution of "hot spots"
Recovery potential, species depletion
Duration of stress-effects co-occurrence
Secondary stress induction
Special ecological significance
22
-------
costs, including opportunity costs. When benefit-
cost and other information is considered, however,
the public may set a different priority among
multiple public welfare goals. The scientific
community has taken the initial steps in scientific
risk assessment by applying simplifying assump-
tions; cf., a generic receptor exposed to a single
stressor under one set of conditions. This is analo-
gous to the first approximation methods used in
the physical sciences, such as assuming no atmos-
phere in gravity experiments. But now the scien-
tists appear to be more willing — even insistent —
to ask the more "real world" questions associated
with complex mixtures and diverse populations.
However, the need for such information and the
willingness to work on the problem have yet to
yield suitable and effective methods. Much work
remains to be done and much experience is yet to
be gained before we will able to assess cumulative
risks posed by multiple stressors and/or multiple
exposures over time.
3.3 The Analysis of Benefits and Costs in
Decision-making (Phases I and II)
In deciding what to do about a particular envi-
ronmental problem, it is impossible to escape the
question of values. At its heart environmental deci-
sion-making is about how people, acting as indi-
viduals or through their government, can make
themselves best off through actions that affect the
environment directly and indirectly. Judging which
choice, among alternative actions, will produce the
most "well being" requires information not only
on risks, but also on how "well being" (i.e., goals
and the relationships among goals and among
alternative choices or actions) is defined. Integrated
decisions on whether and how to address certain
environmental risks or sets of risks requires the
consideration of both a full range of risks and the
full economic consequences associated with possi-
ble decisions. An understanding of the tradeoffs
implied by these choices is crucial both when
problems are formulated for consideration (in
Phase I) and when they are being analyzed in detail
to support decision-making (in Phase II).
3.3.1 What We Have
The work of the Economic Analysis
Subcommittee (EAS) is presented in a separate
SAB document that describes the strengths and
limitations of benefit-cost analysis. In short, risk
information, by itself, is inadequate to guide deci-
sions because there are a near-infinite number of
substances and conditions that can impose harm,
and it is impossible to act to eliminate them all. The
ability to reduce risks is limited by the resources of
labor, capital, knowledge (technology), and physical
endowment available to devote to the task. The task
of environmental decision-makers is to provide —
for now and for the future — the healthiest, safest,
most ecologically secure set of conditions for which
the American people are willing to pay, in the face
of other competing goals. Given these conditions,
the decision-maker can use a number of approaches
(within the limits of enabling legislation) to deter-
mine whether and by how much to reduce risk.
Prominent among these approaches is benefit-cost
analysis that seeks to strike a balance between the
benefits associated with reducing risks and the costs
associated with taking a particular action. In ideal
terms, benefit-cost analysis measures the good and
bad effects of a proposed action in some common
term so that decision-makers can more directly
determine whether the associated gains to society
will outweigh the losses.
In real terms, the application of benefit-cost
analysis to environmental decision-making is chal-
lenging. On the one hand, assessing benefit is
difficult. Often the goods and services that come
from ecosystems are not traded in markets.
Further, even if real markets exist for the goods
and services, the changes in human health and
ecosystem functions associated with a particular
action are difficult to predict or measure. Where
23
The task of
environmental
decision-
makers is to
provide — for
now and for the
future — the
healthiest,
safest, most
ecologically
secure set of
conditions for
which the
American
people are
willing to pay...
-------
markets exist, economists have well-established
methods for determining peoples' willingness to
trade one resource use for another. Where markets
do not exist, however, economists must use tech-
niques that are more controversial in order to esti-
mate the values people place on an ecosystem
good or service.
Cost is also difficult to assess, especially when
viewed as economists do as opportunity costs,
government administration costs, transaction
costs, general equilibrium effects, and social
impacts, in addition to direct compliance costs.
(Opportunity costs are the foregone social bene-
fits associated with applying scarce resources to
environmental protection instead of other possible
uses of those resources.)
Other challenges for the benefit-cost analyst
include consideration of time and equity. Costs
and benefits associated with an action may not be
realized over the same time period (e.g., benefits
may accrue to present generations, but costs to a
future generation, or vice versa). This difference in
the time horizon often raises equity concerns. In
addition, future costs and benefits cannot be
compared directly with current costs and benefits
without taking into account the potential earnings
that may be associated with deferred expenditures
and consumer preferences for consumption in the
present vs. in the future. Thus, economists have
developed a system of discounting the stream of
benefits and costs as they are realized over time so
that future costs and benefits can be compared
with present costs and benefits. Although the
discounting process can be controversial, many
economists believe that discounting of the streams
of benefits and costs in a consistent way is essential
to show current decision-makers the full implica-
tions (present and future) of alternative choices.
In the matter of equity, while benefit-cost
analysis generally provides a societal level view of
the benefits and costs associated with a proposed
action, the technique is less adept at accounting for
the uneven distribution of those benefits and costs
24
across different individuals and groups in society.
Another complicating factor is that individuals
affected by the environmental action are often
unequally endowed in wealth and income. This
difference influences both the benefits and costs as
they enter the evaluation framework, and the
sense of fairness that pervades the results.
Although benefit-cost analysis can shed light on
these distributional issues, such analysis cannot
answer the societal question regarding the desired
distribution of benefits and costs. The definition
of what is "equitable" is left to the decision-maker,
in conjunction with the other participants in the
decision process.
3.3.2 What We Need
As noted above, more work needs to be done
to enable decision-makers to assess the impact of
potential decisions on the distribution of benefits
and costs across the population and on society at
different points in the future. In addition, the
Framework suggests a role for benefit-cost analy-
ses on the impacts of potential management
options to help formulate risk reduction priorities.
"Rough-cut" benefit-cost analyses should be suffi-
cient as first approximations of whether the
options are economically feasible or not.
In summary, benefit-cost analysis is a consis-
tent, coherent, and transparent tool for looking at
trade-offs among competing goals. When well
done, the products of specific benefit-cost analyses
can generally be relied upon as useful inputs for
decision-making purposes, but are always limited
by data and sometimes by methodology. In some
cases and for some questions (e.g., equity), the
inherent uncertainties limit benefit-cost analysis to
providing only indicative information, which,
nonetheless, can be useful. As with risk assessment,
risk comparison, and other elements in the
Framework, there is a recognition that even the
best and most complete benefit-cost analysis is
inadequate, by itself, to yield "the answer" in a
-------
particular case. Additional work is needed to refine
existing approaches for forming and eliciting the
values that individuals place on possible environ-
mental or health outcomes and on their preferences
among possible risk reduction options. The use of
deliberation, as part of or as a complement to more
traditional benefit-cost analyses, holds promise for
characterizing difficult-to-monetize values and
thus bringing those values to the table. The next
section discusses these issues in greater detail.
3.4 Forming, Eliciting, and Considering
Public Values (Phases I and II)
The term "values" takes on two meanings in
integrated decision-making, and it is important to
distinguish between the two in order to avoid
confusion and mis-communication. In the first
instance, "values" refers to the set of underlying
factors that, taken together, cause people to hold
the opinions that they hold and to make the
choices that they make when presented with real
situations. There are many such value systems that
contribute to the normative makeup of people in a
diverse society. The diverse nature of peoples'
values contributes to the complexity and difficulty
responsible authorities face when making deci-
sions on whether or not to act to reduce risks.
These values must be considered, along with the
statutory framework and other factors, when
goals are identified during Problem Formulation.
In the second instance, "values" is used in an
operational sense as a measure of what one
outcome is worth in comparison with alternative
outcomes. This is the sense in which values are
reflected in benefit-cost analysis and is the sense in
which the term in used in this section.
3.4.1 What We Have
The Valuation Subcommittee (VS) engaged in a
series of intensive discussions that explored the
nature of natural resource valuation and the basis
behind the public construct of values. The general
themes that emerged during the Subcommittee's
discussions are listed in the box on page 26.
The VS discussions affirmed the notion that
integrated environmental decision-making
requires a process within which the decision-
maker can meld the results of science and the goals
of the people served and formulate acceptable
decisions. In principle, economic analysis provides
one such approach to valuing the human health
and ecosystem benefits of proposed actions in
terms of the change in economic well-being asso-
ciated with the action. Economic values from this
approach are said to be derived from the vector of
all underlying abstract values held by individuals
in concert when a situation is presented.
By their nature, changes in human health and
ecological conditions often are not easily observed
in quantifiable terms that can be rendered in the
monetary units that most often are used to
compare possible outcomes in benefit-cost analy-
ses. For this reason, some people assert that the
benefits received from environmental protection
are systematically under-estimated and, conse-
quently, that ecological systems and human health
are under-protected as compared to other goals
and desires of people.
An example of the difficulty faced by decision-
makers can be seen in the case of determining
ecosystem values. In principle ecosystem values
are not different from other values, and they need
not enter into the decision process in any unique
way. At the same time, however, measuring and
incorporating the values ascribed to anthropogenic
changes in ecological conditions does present seri-
ous difficulties that require special care.12
Ecosystem value can come from direct use,
from indirect use, or simply from the value that the
public places on keeping the ecosystem viable
12 Although the VS focused on ecosystem valuation, valuation
of human health conditions presents similar challenges in
environmental decision-making.
25
... integrated
environmental
decision-
making
requires a
process within
which the
decision-maker
can meld the
results of
science and
the goals of
the people
served..
-------
Deliberative
processes play
an important
role in eliciting
values of
people, and in
obtaining
stakeholder
participation
in decision-
making.
("existence value"). Eliciting these values is difficult
for several reasons. For example, people doing the
valuing often have insufficient knowledge of how
changes in ecological factors affect the things they
care about. They need expert scientific assistance in
making the important connections. Also, many of
the benefits from ecological systems are subtle and
do not enter into conscious consideration in the
normal course of events, in contrast to market
goods and services which are much more evident.
To account for these benefits, special techniques
for eliciting preferences are required. In addition,
some of the values ascribed to ecological outcomes
(e.g., equity, sustainability, and stewardship) arise
in a social context and may not surface from
commonly used individual preference measures.
Most importantly, because ecological services
often do not enter into markets or enter them
incompletely, it is difficult — if not impossible —
to provide robust and dependable monetized
measures of the benefits they deliver. Often it is
not possible to even determine quantitative meas-
ures for differences in outcomes. Qualitative
measures of benefits and costs, therefore, must be
arrived at and then incorporated into decision
processes. In short, the valuation of ecological
costs and benefits is prone to error because
elements valued by people may be omitted or
incorrectly specified and because measurement is
inherently more difficult than with goods and
services for which market and market-like meas-
ures are available.
Because of these limitations, there is a need to
improve existing methods for estimating the value
that society places on various aspects of human
health and ecological condition. One approach to
assure that all relevant elements are included in
decisions, and that they are valued properly, is to
expand the use of deliberative processes.
Deliberative processes play an important role in
eliciting values of people, and in obtaining stake-
holder participation in decision-making. If such
Environmental Valuation Themes
Ecological valuation is an anthropocentric exercise; i.e., people's wishes count; there is no external set
of values waiting to be discovered for application to decision-making.
The value of anything reflects its contribution toward the achievement of some goal. The process of
valuation cannot be separated from the need to reach agreement on goals.
Environmental valuation requires interaction and deliberation among scientists, decision-makers, and
other stakeholders to identify goals and to define endpoints to characterize those goals.
Existing economic approaches, broadly considered, are consistent and coherent frameworks for
valuation because they organize a system of trade-offs. However, they are not mechanisms for
producing "the answer" because they may omit trans-economic values that may be important, may
include some elements that are difficult or impossible to estimate, and may employ preference
elicitation processes that are incomplete.
An expanded, rich, and complex process using multiple approaches is required to fully encompass
ecological valuation.
26
-------
processes are used discriminately and tailored to
the situation, they should lead not only to more
satisfactory results, but also should not delay —
and indeed may even speed the delivery of— envi-
ronmental protection because they lessen post-
decision controversy.
However, incorporating people's values and
preferences into decision-making is not an easy
task. For instance, not all values, and therefore the
preferences they reflect and the choices they
direct, are equal within or across people. Some
values are held more tightly than others, and thus
would be expected to be traded-off differently —
if at all — from those of lesser importance to indi-
viduals. Further, stated preferences may be
constructed during the elicitation process itself.
Knowledge of decision theory, cognitive
processes, and how people react to complex situa-
tions in reaching decisions can help inform delib-
erative processes that are intended not only to
elicit values information but also to obtain stake-
holder participation in the decision-making
process. Finally, the deliberative processes must be
open and transparent so that people can under-
stand how decisions are made. Through such
openness and the use of procedures that are
common to facilitation and mediation, institutions
can build trust and thereby help to legitimize the
processes used in integrated decision-making.
The VS identified four classes of decisions and
the type of deliberative process that would be
appropriate for each (Figure 5).
When agreement about values (economic and
non-economic) is high and the state of knowledge
(relevant science, economics, and social science) is
sufficient and/or non-controversial, Agency deci-
sion-making is likely to be routine and delibera-
tion will only be needed periodically, if at all, for
oversight (oversight deliberation). At the other
extreme, when agreement about values is low and
the state of knowledge is insufficient and/or
controversial, decision-making is likely to require
multi-dimensional tradeoffs based on insufficient
Figure 5. Typology of Deliberation
Processes with Stakeholders and
Experts (adapted from Chess et al, 1998)
Insufficient
STATE OF
KNOWLEDGE
Sufficient
Expert
Deliberation
Oversight
Deliberation
Integrated
Deliberation
Stakeholder
Deliberation
High
STATE OF VALUE
AGREEMENT
Low
knowledge. In this situation, integrated delibera-
tion, involving both experts and outside stake-
holders, is needed throughout the decision-
making process. Between these two extremes lie
intermediate intensities of deliberation to fit inter-
mediate states of knowledge and agreement on
underlying values.
3.4.2 What We Need
The Working Paper that resulted from the VS
effort offers recommendations for a future work-
shop to explore the topic of natural resource
valuation more fully with a larger group of partici-
pants, including interested and affected parties.
The SAB experience testifies to both the impor-
tance and the difficulty of such open discussion,
frank exchange of views, and search for common
ground. A workshop that reaches out to a larger
audience could be beneficial. While it is not certain
27
-------
that such a workshop will settle the remaining
questions regarding natural resource valuation,
such a workshop should further the Agency's
understanding of recent valuation developments,
additional methods for eliciting public values, and
the role of valuation in integrated environmental
decision-making.
In addition, there is a need for further research
on and experimentation with approaches to valu-
ing benefits of environmental systems. Existing
approaches are inadequate in their treatment of
such values as fairness and sustainability. They
also have difficulty in incorporating the systemic
benefits of ecosystem attributes such as biodiver-
sity and are incomplete in their treatment of
dynamic responses of ecosystems to change. More
holistic approaches that take into account the web
of interactions among decisions surrounding
ecological systems and related production and
consumption activities would be helpful.
3.5 Options Analysis (Phases I and II)
In both Phase I and Phase II, after iteratively
assessing and comparing environmental risks and
considering the implications of various risk reduc-
tion possibilities on the grounds of economics or
other measures of welfare, the IED Framework
calls for an overall design and selection of risk
management scenarios to address the environmen-
tal problem at hand. Such Options Analysis
during Problem Formulation (Phase I) informs
goal-setting, thereby ensuring that goals are not
defined solely on the basis of risk, but also take
into account possibilities to reduce risks within
likely constraints and trade-offs that will be
required among goals. During Phase II, when
the environmental problem(s) and associated envi-
ronmental goals have been defined, Options
Analysis becomes a more in-depth consideration
of possible risk reduction options. In both Phases,
the possible options should be analyzed with
regard to their potential to reduce single and
28
multiple risks of concern, associated costs, sustain-
ability, equity, and other criteria specified by the
participants.
3.5.1 What We Have
The Risk Reduction Options Subcommittee
(RROS) developed a process for identifying the
most effective risk management approaches for a
variety of types of risk problems that might
confront a decision-maker. Specifically, they
considered options analysis from three different
perspectives: a) for a stressor-based problem (e.g.,
ozone); b) for a geographically based problem
(e.g., risks associated with an urban area); and c)
for a media-based problem (e.g., contaminated
groundwater). Their efforts, which are outlined
here, will be detailed in a separate SAB report.
In short, the RROS found that approaches for
developing and evaluating options to reduce single
risks are fairly well developed. However, the LED
Framework emphasizes the importance of examin-
ing a broader array of potential options than might
typically have been done in the past. Criteria
should be developed to screen potential options,
aggregate or disaggregate options, and, through an
iterative process, converge on a set of options that
analysis indicates would best meet the goals. As
noted in the previous sections, analysis of the
economic and societal consequences of various
options is an important aspect of options analysis.
The results of the Subcommittee's deliberations
on identifying, screening, and selecting risk reduc-
tion options, and the relationship of these steps to
the phases of the FED Framework, are captured in
the ID-step process depicted in Figure 6. The first
step in the method is to define the problem.
Articulation of the environmental problem(s),
including specific goals for what/whose risk is to
be reduced and by how much, is critical to subse-
quent steps. Clear environmental goals, with
explicit statement of the relationships and the
potential tradeoffs among goals, are the founda-
-------
Figure 6. Risk Reduction Options Selection Methodology
Define Problem
Develop Background Information
Indcntify All Risk Reduction Options
-------
ranking, and multi-attribute decision procedures)
that may be useful in the screening and selection
process.
The RROS also analyzed seven different cate-
gories of options (as well as subcategories) (Table
2) and described situations in which they felt each
category of options would be most effective and
least effective.
3.5.2 What We Need
The task of developing and selecting risk
reduction options is most easily envisioned in
terms of reduction of a single risk, e.g., one associ-
ated with a particular stressor or source.
Integrated environmental decision-making,
Table 2. Risk Reduction Option
Categories Evaluated by the RROS
1. Communication/Education
a. Information Reporting
b. Technical Assistance
2. Enforcement
3. Engineering
4. International and Intergovernmental
Cooperation
5. Management Systems
6. Market Incentives
a. Tradable Emissions
b. Pollution Charges
c. Subsidies
7. Regulation
a. Standards
(i) Harm-based
(ii) Technology-forcing
(iii) Design
b. Product Bans
c. Challenge Regulations
however, requires a more complex analysis of
options. Although it is often best to address risks
with a combination of risk reduction tools, this
often is not done because of inadequate informa-
tion on the multiple sources of a stressor and their
relative contribution to total risk.
Extending the methodology to an integrated
problem set containing risks from multiple stres-
sors (e.g., those experienced by a particular
community) will further increase the complexity
of the analysis. Thus, it will be important to aggre-
gate or disaggregate the problem set (e.g., using
"root cause" or "common source/common path-
way" analysis) so that analysis of risk reduction
options is more manageable. Although it may not
be possible to group all risks of concern on the
basis of their technical attributes, a scientific
analysis of the risks may well reveal commonali-
ties that indicate which risks will be affected by
the same risk reduction option.
The likely effect of this integrated view is that
the option(s) selected to reduce a group of risks
might differ from that which would be selected to
reduce the top ranked risk, if it were to be consid-
ered in isolation. The complexity of the analysis
also increases greatly as the number of screening
and selection criteria increase.
One objective of options analysis in an inte-
grated decision-making context might be to iden-
tify those options that may simultaneously reduce,
directly or indirectly, risks posed by more than
one stressor. Such analysis would be important if
the management goal were to maximize the reduc-
tion of the total aggregate risk from multiple stres-
sors, rather than to maximize the reduction of
risks posed by any single stressor. This approach,
requiring as it may the simultaneous consideration
of risks from quite different types of stressors, has
not yet been fully developed; and it will not be
trivial to implement. Nevertheless, the SAB
believes that its development and implementation
offer tremendous potential for improving environ-
mental health overall.
30
-------
Examples of challenges inherent in applying
options analysis to an integrated problem set,
consisting of multiple stressors, sources, or
endpoints, include the following:
a) Aggregation is helpful in identifying multiple
stressors that may have a common set of risk
reduction options with the objective of select-
ing a set of options that will provide the most
risk reduction for the set of risks being
analyzed. Practically speaking, it will not be
possible to optimize risk reduction over all
stressors of concern considered at once. For
this reason, screening and aggregation of stres-
sors into manageable subsets should be driven
by analysis of common aspects of stressors,
root causes, and/or activities.
b) In order to compare risk reduction across sets
of options and sets of risks, and to evaluate risk
reduction per unit cost, it is critical to have a
common measurement of risk or common
denominator for all risks. In many cases,
comparisons of risks and risk reduction under
different scenarios will involve unlike risks
(e.g., cancer risk in humans vs. chronic health
effects vs. effects on wildlife populations), even
where those risks have a common "root cause"
(e.g., a single stressor or source). While models
can be developed to weight and combine the
different attributes in disparate types of risk,
getting wide-spread social agreement on such
weightings may be quite difficult.
c) Uncertainty associated with options analysis
is likely to increase as a broader set of options
and an array of stressors are considered
simultaneously. Sources of this uncertainty
include the relative contribution of different
stressors/ sources to the total aggregate risk;
the effectiveness of combined options for reduc-
ing aggregate risk; and the benefit-cost, equity,
or other tradeoffs involved in addressing groups
of risks. The extent of uncertainty associated
with a decision may affect the balance of
options selected. When uncertainty is high, for
example, it may be easier to gain support for
education/communication or management
system approaches rather than for regulation.
3.6 Performance Measures
In recent years there has been increased inter-
est in tracking, measuring, and reporting on the
performance of governmental actions. Most
notably, the Government Performance and
Results Act (GPRA) requires that federal agencies
report annually on measures of the results of their
various programs and activities. In addition, there
has been discussion and activity in the area of
"environmental report cards" as a mechanism for
succinctly summarizing the state of the environ-
ment in easily comprehensible terms.
3.6.1 What We Have
To evaluate performance, one should have a
good set of measures and a system for evaluating
and reporting data on those measures.
In the area of measures, the SAB identified
four types of performance measures that can be
used in concert to evaluate progress toward
environmental goals, whether broad goals that
relate to a number of management and regulatory
programs or specific goals defined for a particular
integrated environmental decision:
a) Process Measures are measures of administra-
tive effort or program actions that are
presumed to result in environmental or health
improvements (e.g., number of permits issued,
number of enforcement cases pursued, number
of contaminated sites cleaned up to standards).
b) Stressor Measures (levels) are measures
(levels) of stressors in the environment used to
31
GPRA requires
that federal
agencies report
annually on
measures of
the results of
their various
programs and
activities.
-------
determine attainment or non-attainment of
desired reductions in stressor levels; e.g., total
emissions of a pollutant, concentrations of
particulates in ambient air, levels of dissolved
oxygen or turbidity in a stream, and density of
roads in a watershed.
c) Exposure Measures are measures of the co-
occurrence or contact between an individual or
population and environmental stressor(s) over
a defined time period. The term "exposure" is
traditionally associated with chemical stressors
(e.g., contaminant levels in food, concentra-
tions of contaminants in tissues, time-activity
measures, and total exposure to a contaminant
via all routes), whereas the term co-occurrence
is often used as a broader term applicable to
chemical, physical, and biological stressors.
Exposure measures are more directly related to
effects than ambient levels of stressors in envi-
ronmental media because they address direct
contact with the stressor. These measures may
also be more readily linked to risk manage-
ment decisions than effects measures since
causes of adverse effects are often multi-facto-
rial and difficult to relate to a single environ-
mental variable. The use of biological markers
of exposure as a measure of environmental
insults is a rapidly expanding field.
d) Effects Measures are measures of human
and/or ecological effects (e.g., asthma rates,
deaths from acute poisoning from household
products or pesticides, deaths from cancer,
acres of wetlands gained or lost, local extinc-
tions of important species) the changes in
which can be used in one of two ways: (1) to
assess the impact of an environmental risk
reduction program; and/or (2) for condition
assessment, in which a suite of effects meas-
ures are evaluated and reported in combination
to characterize the health or condition of an
entire population or ecosystem. Condition
assessment provides a baseline against which to
evaluate the success of broad policies or multi-
ple decisions impacting a population or
geographic region.
Although it is difficult to relate changes in
human health condition to a single environmental
factor, recent studies are improving our ability to
identify precursors to chronic diseases so that the
detection of early effects occurs closer in time to
the environmental exposure(s). Examples of
precursor measures include changes in the P53
gene associated with exposure to sunlight (as an
early marker of damage that may develop into
skin cancer), changes in lung function associated
with exposures to air pollutants, and changes in
IQ associated with exposure to contaminants
such as lead.
The Committee considers measures of effects
or condition to be environmental outcome
measures. We note that this definition differs
from the Agency's definition of "environmental
outcomes," which also includes measures of
stressor levels (EPA, 1997b). We recommend,
however, that stressor measures be kept distinct
from environmental outcome measures because a)
changes (increases or decreases) in stressor levels
do not necessarily translate into changes (increases
or decreases) in risk, and b) the public's environ-
mental goals are typically in terms of desired
states of health or condition, rather than desired
stressor levels.
The different types of measures can be thought
of in terms of a spectrum of measures ranging
from least directly to most directly related to
actual environmental outcomes, which are of
primary interest to the public. In many cases, the
spectrum of measures will also correspond to a
spectrum of response times, ranging from shorter
term to longer term measures. For example, it is
relatively quick and easy to document the number
of permits issued in a year, but it is more difficult
and time-consuming to determine the actual
32
-------
Evaluating the Success of an Integrated Environmental Decision:
A Watershed Example
To illustrate the use of each of the types of performance measures to report on a specific
Integrated Environmental Decision, consider a watershed management or restoration program. In
this example, the ultimate goal of the program is to maintain or re-establish an ecosystem that
supports a diversity of habitat types along with their resident communities of plants and animals,
supports essential ecological functions, and is self-sustaining. Here, the Integrated Environmental
Decision will consist of a set of interrelated actions, many of which will be designed to address
multiple stressors in order to achieve a reduction in the aggregate risk from those stressors. Other
IED program actions will focus on restoring damage from past stressors (such as the restoration of
riparian zones damaged by livestock operations in order to decrease sedimentation downstream,
provide shade and cooler temperatures for aquatic species, and provide additional nutrients to the
system from dropping leaves).
The evaluation criteria used to judge overall IED program results will include measures of habitat
quality (such as length of intact corridors of natural riparian vegetation), water quality and tempera-
ture, hydrology that mimics natural variations, the extent of connectivity between floodplains and
the river, nutrient balance, presence of sustainable native populations, and the like. Taken together,
these effects measures effectively describe the condition of the watershed, i.e., whether the water-
shed, in fact, can sustainably support native populations and their habitats and maintain ecological
functions — and they therefore report on the aggregate results of the IED program.
Each of these effects measures can also be used as to evaluate the success of individual actions
within the IED program. Following the example above, measures of length of intact riparian corri-
dor, water temperature, and nutrient balance would be used to assess the success of specific actions
taken to restore riparian zones.
The time frame required to see changes in the effects measures will vary. Some, such as population
levels of short-lived species of interest, may be detected after only one year of a management regime
that alters pollutant inputs and water releases to the system. Other environmental responses may
take more than ten years to be detectable.
In addition to the effects measures and condition assessment, direct reporting on the decreased
pressure from various stressors will be useful. In the example above, such stressor measures might
include the decrease in the rate of new riparian damage and increases in the release of cool water
from dams in the summer. Stressor measures relating to other actions that are also part of the IED
program might include reductions in ambient pollution levels, decreases in the number of unscreened
pumps that injure fish, and the restoration of periodic flood flows.
Finally, process measures will provide insight into the level of effort expended and provide a
shorter-term indicator that the program is proceeding as planned. Examples of process measures
might include conservation easement contracts signed for the management of riparian corridors,
changes in the regulations governing water releases from dams, and numbers of water pollution
permits updated with new effluent limits.
33
-------
An
environmental
results
reporting
system should
include a mix
of process,
stressor,
exposure, and
outcome
measures.
impact of that process measure on the condition
of the environment. Figure 7 illustrates this spec-
trum and relates the Committee's terminology to
other commonly used terms for environmental
performance measures.
In the area of reporting systems, the SAB
recommends that feedback be provided on the
extent and distribution of environmental risks to
determine whether the relative seriousness of risks
was accurately characterized in the first place and
whether specific risks have changed as a result of
an implemented risk reduction program. Report
cards also should provide the basis for evaluating
the performance of specific risk reduction
programs or decisions, as judged against decision
criteria such as efficacy at reducing aggregate risk,
cost, equity, and time required to achieve risk
reduction goals. In summary, environmental
report cards should provide the information
needed to a) identify opportunities for course
correction and adaptive management, i.e., modifi-
cation of risk reduction approaches in light of
performance information or new information on
risks; and b) assign specific accountability—to
individuals, programs, or organizations—for envi-
ronmental results.
Report cards for evaluating integrated environ-
mental decisions should contain specific mile-
stones that can be used to measure progress
towards achieving the environmental goal(s)
agreed upon by the participants. Each of the
selected endpoints defined during Problem
Formulation (Phase I) should be a part of the
report card, as well as the specific measures or
indicators that are monitored to characterize those
endpoints. The frequency of the reporting should
be commensurate with the nature of the risk and
the expected time frame for system response.
Figure 7. Spectrum of Performance Measures
Environmental Outcomes
Process
Measures
Stressor
Measures
Exposure
Measures
activity measures1 pressure indicators3 co-occurrence4
output measures2 release measures
response indicators3 emission measures
"beans"
Effects/Condition
Measures
adverse effects*
health outcomes6
state indicators3
Least directly related
to Environmental Outcomes
Most directly related
to Environmental Outcomes
'EPA, 1997b; ^GPRA; 3QECD, 1998; *ERS Method; >EPA, 1998; WRC, 1996
34
-------
3.6.2 What We Need
Progress is being made in the area of environ-
mental report cards. For example, there is an effort
underway, initiated by the Office of Science and
Technology Policy and jointly funded by eight
federal agencies, to design a national environmen-
tal report card. This multi-year project, which is
being conducted by the Heinz Center, has already
produced a prototype design for a report card on
the state of the nation's ecosystems and specific
indicators of ecosystem condition for 3 types of
ecological systems: coastal/ocean, croplands, and
forest systems (Heinz Center, 1999). Over the
next several years, the Heinz Center plans to
develop a list of indicators for an additional 3
types of systems: freshwater, arid lands/range-
lands, and cities/suburbs. Although the SAB has
not reviewed the appropriateness of the report
card design or the specific proposed measures, the
effort is a clear indication of growing federal
agency interest in environmental report cards. A
somewhat analogous reporting effort that focuses
on human health is Healthy People 2000—and
now Healthy People 2010—which defined a series
of health objectives and established a series of
health indicators that are monitored and reported
annually (e.g., see National Center for Health
Statistics, 1999).
An environmental results reporting system
should include a mix of process, stressor, exposure,
and outcome measures. Historically, the Agency
has focused primarily on process and stressor
measures. Therefore, more emphasis should be
given to outcome measures, developing new ones
where required. Environmental outcome meas-
ures—whether of health, ecological condition, or
quality of life—may not exist, may be subtle or
difficult to measure, or may be observable only
over long time frames. Evaluating the performance
of risk prevention programs may be particularly
problematic. Thus, process, stressor, and exposure
measures will continue to be important in environ-
mental decision-making and management.
Outcome measures, however, are an important
means of verifying the theoretical basis for the
control or abatement options chosen. In other
words, if the postulated relationship between stres-
sors and effects is inaccurate, then stressor or expo-
sure reductions might not produce the expected
improvements in adverse effects or condition.
Decisions on the design of a report card
system have implications for information collec-
tion systems, both those that track administrative
processes and those that collect environmental
data (e.g., ambient monitoring programs). In
selecting performance measures, there will often
be a gap between what it would be desirable to
know and what it may be feasible to know. This
gap may exist as a result of several factors, includ-
ing the limited state of knowledge about the rela-
tionship between stressors and effects, the costs
involved in obtaining certain types of information,
and the willingness of affected people to provide
information. Nonetheless, it is important that
careful consideration of the most desirable
performance measures, including those based on
an established chain from process and stressor
measures to exposure and outcome measures,
influence the types of information that are actually
collected.
The need for data to assess performance of
integrated environmental decisions emphasizes
again the importance of monitoring programs that
can measure both ecological and human health
exposure and outcomes. Implementation of a
reporting system for human health risk reduction
will be hampered in many cases by a lack of data
in key areas; e.g., exposures, differential suscepti-
bilities among subpopulations, and incidence of
non-fatal diseases. In the ecological arena, assess-
ments of ecological integrity require information
not only on water quality, air quality, soil quality,
water flow, and populations of certain species,
which are commonly monitored today, but also
measurements of biological community structure,
35
...the design
of a report card
system has
implications for
information
collection
systems, both
those that track
administrative
processes
and those
that collect
environmental
data..
-------
the presence of successional states, diversity of
habitat types across the landscape, connectivity,
altered topography, hydrodynamic patterns, and
so forth. This latter set of parameters is not typi-
cally monitored by EPA, but has received greater
attention by other federal agencies (e.g., USGS,
US'FS). This fact emphasizes the importance of
strengthening and maintaining the collaboration
among the many agencies that conduct monitor-
ing, an effort begun several years ago under the
auspices of the Committee on Environment and
Natural Resources (CENR, 1997).
Even the Agency's traditional list of process
measures will need to be expanded in order to
evaluate the performance of some of the new
approaches to environmental management. In
recent years, the Agency has added more tools to
the environmental toolbox, bringing to bear such
new approaches as economic incentives, negoti-
ated agreements, and the like. Therefore, the
collection of data to evaluate these new
approaches should evolve as well. There are
several important inputs currently missing from
most reporting systems that would provide
valuable information for assessing how well
environmental programs have worked and what
changes or adjustments might be made. In the
case of a marketable permit system, for example,
systematic reporting about the number of transac-
tions, the average price per unit of the traded
commodity, the number of market participants,
the net reduction in pollution output, and
average marginal cost per unit of pollution
reduced should be compared with initial predic-
tions. To the extent that a risk management
program involves more than one tool, the entire
panoply of actions should be reviewed for efficacy,
cost, and other measures of performance. Separate
information should also be provided on the "non-
quantitative" inputs to decision-making, including
the effects of the risk reduction program on
sustainability and equity.
36
-------
4. RECOMMENDATIONS TO THE AGENCY
As a result of conducting the Integrated Risk
Project, the SAB has developed a set of 10 recom-
mendations that should help the Agency take "the
next step" in improving environmental decision-
making in this country.
Recommendation 1: EPA should continue
development of integrated, outcomes-
based environmental protection, while
maintaining the safeguards afforded by
the current system.
Previous environmental management
approaches, both regulatory and non-regulatory,
have resulted in substantial improvements in
human health and ecological condition, particu-
larly in regard to chemical risks. While maintain-
ing the current regulatory and management
structure, the Agency should continue its develop-
ment of more broadly integrated approaches to
environmental decision-making. In particular, the
Agency should use the concepts inherent in the
Framework laid out in this report to seek oppor-
tunities for going beyond current protections by
taking a truly integrated look at risks, opportuni-
ties for risk reduction, and the consequences of
addressing (or failing to address) those risks. To
the degree possible, the focus of this next step of
environmental decision-making should be on
environmental results; that is, on demonstrable
outcomes (improvements) in the environment
resulting from integrated action. As a part of this
effort, the Agency will need to develop additional
tools to implement the approach and to monitor
the results of actions taken.
Recommendation 2: Because science plays
a critical role in protecting the
environment, EPA should commit the
resources necessary to expand the
scientific foundation for integrated
decision-making and outcomes-based
environmental management.
An important theme of the IED Framework is
the need to integrate scientific and technical infor-
mation on risks and risk reduction opportunities
with information on societal values, aspirations,
and goals. The call for greater inclusion of multi-
ple disciplines and points of view, however, must
not obscure the fact that science and scientific
methods are critical to sound environmental deci-
sion-making. Science has a unique and critical role
to play in protection of the environment. It is
through scientific investigations that many envi-
ronmental problems are first discovered; e.g., the
depletion of stratospheric ozone, hypotheses
about environmental endocrine disrupters. Science
also is instrumental in developing, testing, and
evaluating risk reduction options, and social
sciences offer techniques for assessing societal
preferences and wants. Implementation of the
IED Framework, particularly its application to
multiple-stressor problems, will in many cases
require new science, both empirical and theoreti-
cal. In order to gain the greatest benefit from
this next step in environmental decision-making,
the Agency will need to invest in research
designed to support integrated risk assessment
and management.
37
-------
Recommendations to Foster Integrated Environmental Decision-making
1. EPA should continue development of integrated, outcomes-based environmental protection,
while maintaining the safeguards afforded by the current system.
2. Because science plays a critical role in protecting the environment, EPA should commit the
resources necessary to expand the scientific foundation for integrated decision-making and
outcomes-based environmental management.
3. EPA should apply and encourage the broader use of risk comparison methodologies that clearly
identify how scientific information and judgment are incorporated into risk comparisons.
4. EPA should explore a broader range of risk reduction options in combination to manage
environmental risks.
5. When evaluating risk reduction options, EPA should strive to weigh the full range of advantages
and disadvantages, both those measured in dollars as costs and benefits and those for which there
may not be a comprehensive dollar measure, such as sustainability and equity.
6. EPA should seek and develop methods to characterize public values and incorporate those values
into goal-setting and decision-making.
7. EPA, by itself and in concert with others, should identify, collect, and disseminate scientifically-
based environmental metrics organized in new ways to support a more integrated approach to
managing environmental risk.
8. EPA, by itself and in concert with others, should develop a system of "report cards" to organize
and disseminate information on the status of ecological and human health and the quality of life
in order to assess the effectiveness of its environmental decisions and to guide future
environmental management.
9. EPA should expand and develop new collaborative working relationships with other federal and
non-federal governmental agencies and others who also will be involved in integrated
environmental decision-making.
10. EPA should explore options for reducing risks from significant stressors that currently are
addressed inadequately by the nation's environmental institutions.
38
-------
Recommendation 3: EPA should apply
and encourage the broader use of risk
comparison methodologies that clearly
identify how scientific information and
judgment are incorporated into risk
comparisons.
Scientific information on risk, such as quanti-
fied risk assessments and scientifically demon-
strated relationships between stressors and effects,
provides an essential basis for making objective
risk comparisons. However, desired scientific
information often is incomplete or absent, and
scientists have to use their best professional judg-
ment to bridge important gaps in the data.
Further, scientific information and analyses, by
themselves, are not sufficient for comparing unlike
risks; e.g., when comparing human healths risks
vs. ecosystem risks or cancer risk in adults vs.
neurologic risk to children. Thus, public values
also must come into play in making the compar-
isons. For these reasons, methodologies used to
compare environmental risks—whether human
health risks, ecosystem risks, or both—should
incorporate not only the most up-to-date scien-
tific information, but also should identify explic-
itly where professional judgment and values have
influenced the results. In the course of this project
the SAB developed, to varying extents, two proto-
type methodologies for risk ratings by technical
experts. These and similar approaches should be
further explored as possible points of departure
for the Agency as it seeks to develop and use
methods for comparing relative environmental
risks that incorporate public values.
Recommendation 4: EPA should explore a
broader range of risk reduction options in
combination to manage environmental risks.
In 1990, the SAB recommended in Reducing
Risk that the nation make greater use of all the
tools, including market forces, information, and
product specifications, available to reduce risk.
Now, almost a decade later, many of those tools
are being used to a greater extent than ever before.
The challenge for EPA is not only to expand the
use of those various tools, but also to use them in
creative, coordinated ways to reduce multiple risks
to multiple receptors in communities and ecosys-
tems across the country. In the course of this proj-
ect, the SAB described a prototype methodology
for identifying and selecting risk reduction options
to address single or multiple stressors. This
methodology should be evaluated by the Agency
as a possible approach for better identifying multi-
dimensional strategies to control complex envi-
ronmental problems that involve many sources,
stressors, and receptors.
Recommendation 5: When evaluating risk
reduction options, EPA should strive to
weigh the full range of advantages and
disadvantages, both those measured in
dollars as costs and benefits and those for
which there may not be a comprehensive
dollar measure, such as sustainability
and equity.
The benefit-cost paradigm that underlies many
environmental decisions is the simple formulation
of whether the gains that accrue from protective
actions are worth what is given up to attain them.
There is a subsidiary question: Would other possi-
ble actions be preferable? This question highlights
the importance of taking all effects, including
long-term effects, into account. It also raises the
sometimes hidden issue of how people value
different aspects of the environment. Some of
society's environmental values can be measured
directly in monetizable terms, and others can be
inferred and translated into monetizable terms
with some confidence. But other things that
people value, such as sustainability and equity,
often may be expressed only qualitatively, yet are
39
-------
of no less importance. As a part of this project, the
SAB has described the applicability and limita-
tions of the benefit-cost approach, and suggested
areas where new approaches to characterizing
values are essential. The SAB also has attempted to
define better the full range of relevant questions
that must be considered in environmental valuation.
Recommendation 6: EPA should seek and
develop methods to characterize public
values and incorporate those values into
goal-setting and decision-making.
Community and national values have been and
will continue to be a primary driver of commu-
nity-level and national efforts to protect the envi-
ronment. However, values usually are not weighed
transparently in the decision-making process.
Rather, they usually are implicit in the judgments
made by decision-makers. Thus, they often influ-
ence decisions in ways that are not clear to, or
reviewable by, the public.
Because public values undoubtedly help shape
environmental decisions, it is important to under-
stand and document their role in and influence on
decision-making. It is also important to elicit
public values systematically, differentiate values
from technical information as a part of decision-
making, and include their effects on decisions as
part of the public record. In this way, value judg-
ments will be neither disregarded nor disguised.
Individual and community values should be
solicited systematically by social scientists and
other appropriately trained individuals. The delib-
erative processes that are used in arriving at deci-
sions should involve professionals trained in fields
like consensus-building and dispute resolution.
EPA should make more extensive use of experts in
the areas of behavioral and decision science so that
a more complete representation of community
values is incorporated into the Agency's decisions.
Research likely will be necessary to develop
improved methods for helping people develop
40
considered values and for eliciting and using
public values in decision-making.
Recommendation 7: EPA, by itself and
in concert with others, should collect
and disseminate scientifically based
environmental metrics to support a more
integrated approach to managing
environmental risks.
The transition to and effectiveness of integrated
environmental protection will depend to a large
extent on the availability and utilization of appro-
priate information in the areas of exposure, human
health, ecological health, and quality of life.
Current information collection efforts, however,
often are insufficient, inadequately organized, or
focused on inappropriate endpoints. In the area of
human health, for example, data on non-fatal
outcomes of environmental exposures, such as
asthma, are not being collected except in a rudi-
mentary manner or at certain sites. Only mortality
information is collected systematically at all loca-
tions and can be related to some limited informa-
tion on selected exposures. With regard to
ecological data, more comprehensive information
is needed on the current status of ecosystems such
as wetlands, lakes, forests, and grasslands. This
information should include the extent to which
each ecosystem is exposed to and affected by non-
chemical stressors such as habitat conversion, habi-
tat fragmentation, and invasions of exotic species.
The challenge facing EPA and the nation is not
only one of collecting the right environmental
data, but of finding new ways to manage and use
those data. Working with other federal and state
agencies, EPA should take a leadership role in
identifying critical environmental data gaps,
including data on exposures and health and
ecological outcomes; integrating the largely frag-
mented data collection efforts already underway;
and disseminating integrated environmental infor-
mation to decision-makers and the public.
-------
Recommendation 8: EPA, by itself and in
concert with others, should develop a
system of "report cards" to organize and
disseminate information on the status of
ecological and human health and the
quality of life in order to assess the
effectiveness of its environmental decisions
and to guide future environmental
management.
One of the most valuable uses of environmen-
tal data is to measure the results of the actions
society takes to reduce environmental risk.
However, even if such data did exist, through a
vigorous implementation of Recommendation 7,
there would still be a need to develop widely
accepted and commonly used methods for evalu-
ating a) the state of our environment and b) the
success of national environmental protection
efforts — where, over the long term, success is
measured in terms of demonstrable outcomes in
the health of humans and ecosystems.
Reporting on measures of progress towards
ecological, human health, and quality of life goals
will provide a number of benefits. First, the exer-
cise of defining performance measures and report-
ing on them will bring more focus and discipline
to the Agency by expressing the relationship
between investments (measured in time, money, or
information) and environmental results. Second,
shorter term measures of progress (including
process measures or measures of stressor reduc-
tions) will be useful for accountability and course
correction. Third, longer term measures of
progress (such as improvements in overall human
and ecosystem health and quality of life) will be
most helpful in determining whether goals are
being met, whether further actions are needed to
control well-recognized stressors, or whether new
actions are needed to control new stressors.
To strengthen their credibility and utility, envi-
ronmental report cards should contain informa-
tion derived from objective measurements, be
transparent and clearly documented, and provide
integrated information on progress towards multi-
ple inter-related environmental goals.
Recommendation 9: EPA should expand
and develop new collaborative working
relationships with other federal and non-
federal governmental agencies and others
who also will be involved in integrated
environmental decision-making.
Inherent in integrated decision-making is the
idea that problem formulation and decision-
making must match in scale and location. In some
cases, decisions may be most effective when local
or state governments play the primary role. In
others, coordinated action across several levels of
government or among a number of state and/or
local governments will be required. In still others,
national decision-making may be the preferred
approach. In any event, integrated thinking about
environmental problems will tend to drive deci-
sion-making to the agency or level of government
where decisions are most appropriately made. As
a consequence, EPA's role will evolve to one in
which the depth of control gives way to broader
involvement in partnership with other agencies.
EPA will continue to be responsible for imple-
menting and enforcing federal environmental laws
and statutes, conducting environmental research
and development, and conducting stressor-specific
risk assessments. At the same time, the Agency
can exert national leadership to bring together
appropriate agencies and stakeholders to explore
integrated approaches to environmental problems.
Recommendation 10: EPA should explore
options for reducing risks from significant
stressors that currently are addressed
inadequately by the nation's
environmental institutions.
41
-------
Over the course of the Integrated Risk Project,
it became clear to the SAB that a number of
important human health and ecological risks are
not being addressed adequately by the nation's
environmental institutions. In many cases this is
because risk management responsibility is not
clearly assigned to any one government entity, or
is scattered over many agencies and/or levels or
government. This fragmented approach results in
uncoordinated and incomplete efforts to identify
cause-and-effect linkages and to manage risks.
With regard to ecological risks, the SAB has
concluded that many of the highest ranking risks
(e.g., hydrologic alterations, harvesting of living
marine resources, habitat conversion, climate
change, and introduction of exotic species) are
associated with physical and biological, rather
than chemical, stressors, which do not fall clearly
within the purview of any single federal Agency.
Important human health risks that remain unad-
dressed include those for which the environmental
exposure link is suspected but not certain (e.g.,
asthma, brain cancer, and non-Hodgkins
lymphoma) and risks associated with susceptible
and/or compromised human populations.
To control many of these inadequately
addressed risks will require the kind of integrated
decision-making envisioned in this report. It will
also require a new kind of integrated institutional
leadership. When EPA determines that serious
risks are not being addressed effectively by exist-
ing environmental institutions or decision-making
systems, the Agency has a responsibility to inform
the public about those risks and bring together the
appropriate federal, state, and local agencies to
address them.
42
-------
5. LESSONS LEARNED
The Integrated Risk Project (IRP) has been
unlike any other project undertaken by the SAB.
Conceived by some as a simple updating to the
1990 Reducing Risk report, it soon became evident
that the IRP was far more complex and far more
difficult than earlier undertakings. Issues of science
(e.g., risk assessment, cumulative risk, and econom-
ics) merged with even less tractable — but
arguably even more important — issues of social
sciences and ethics (e.g., personal and group choice,
trans-generational equity, and the role of govern-
ment vis a vis the public). More than 50 partici-
pants from various fields of science and social
science worked together to share their respective
experiences and insights in a collaborative effort to
illuminate a path that would lead to "the answer".
In the process, the participants gained a new appre-
ciation for the benefits from and the barriers to
interdisciplinary efforts of this sort. Differences in
background, nomenclature, and academic culture
of the participants both enhanced and hindered
progress.
Recent advances in science and technology
legitimize asking some of the "tough questions"
that were necessarily simplified just a few years
ago. Questions of total cumulative risk of multiple
endpoints, from multiple pollutants, from multiple
sources, by multiple routes of exposure are now
being confronted directly. More powerful methods
of data collection and analysis, more comprehen-
sive models, and broader consideration of manage-
ment options are evidence of the growing scientific
support for a more integrated approach to environ-
mental decision-making. Inclusion of interested
and affected parties in addressing environmental
problems, the emphasis on an iterative analytical/
deliberative process, and continual grappling with
the ethical aspects of decision-making (e.g., the
meanings of "values", the uneven distribution of
benefits and costs, and explicit consideration of
effects on future generations) point to a recogni-
tion of the fact that today's problems include
important aspects that must be addressed by social
sciences and ethics.
It is important to recognize that successful
application of the IED Framework will require
some adjustments in the manner and degree to
which the many participants interact over the
course of the decision-making process. Integrated
environmental decision-making requires the shar-
ing of information, ideas, approaches, and manage-
ment deliberations to a degree now seldom
practiced among individuals of very different back-
grounds. Although this sharing is a positive aspect
of the Framework, it may require significant adap-
tation on the part of individual policy-makers and
institutions. For example, decision-makers will
need to interact more extensively with scientific
and technical analysts and the public in the course
of developing integrated approaches to environ-
mental risks. Likewise, scientific and technical
experts will need to recognize the role (and limita-
tions) of science in decision-making, and also to
recognize the legitimate role of values in establish-
ing environmental goals and selecting management
approaches. This culture change will be assisted by
familiarity and experience with integrated environ-
mental decision-making. Experience, combined
with discipline, will also be needed to apply the
Framework with discretion to the depth of detail
43
-------
necessary for a particular problem, and no more;
i.e., the Framework should not become a barrier to
decision-making.
Additional challenges likely to be encountered
when applying the Framework include the follow-
ing:
a) Problems of understanding arising from differ-
ences in terminology and outlook imbedded in
the different disciplines and backgrounds of
the participants in the process;
b) Difficulties of using both qualitative and quan-
titative measures concurrently in the decision
process;
c) The need to compare different types of risks
(e.g., health, ecological and quality of life risks)
within a common decision framework and to
discern and define the inter-relationships
among risks so as to define common goals
across the different risk types; and
d) Time-lags between implementation of risk
reduction plans and the detection of results
and effects, which make the selection of appro-
priate performance measures particularly
important.
In some ways, the Board has been true to the
old adage that "One's goal should exceed one's
grasp." That is, the goal of this project — to artic-
ulate a complete and rational method for including
all aspects of integrated environmental decision-
making in a single process — has exceeded the
SAB's grasp. In fact, it is likely that that goal will
never be reached to everyone's satisfaction. And
yet, the Framework for Integrated Environmental
Decision-making that was developed during the
course of the project clearly points to the direction
in which "the next step" of environmental deci-
sion-making should go. The efforts of the individ-
ual Subcommittees can be examined for further
insights about how the Agency might — or might
not — make additional progress in that direction.
In any event, there is enough direction and more
than enough challenge in this report to keep the
Agency — and others interested in the next step
of environmental decision-making — productively
active for some time to come.
44
-------
6. REFERENCES CITED
Chess, C, T. Dietz, and M. Shannon. 1998. "Who
Should Deliberate When?" Human Ecology
Review 5(1): 45-48.
Morgenstern, R. D. (Ed.). 1997. Economic
Analyses at EPA: Assessing Regulatory Impact.
Resources for the Future, Washington, D.C.
National Academy of Public Administration.
1995. Setting Priorities, Getting Results: A
New Direction for EPA.
National Center for Health Statistics. 1999.
Healthy People 2000 Review, 1998-1999.
Hyattsville, Maryland: Public Health Service.
National Research Council. 1983. Risk
Assessment in the Federal Government:
Managing the Process. National Academy
Press, Washington, DC.
National Research Council. 1994. Science and
Judgment in Risk Assessment. National
Academy Press, Washington, DC.
National Research Council. 1996. Stern, PC. and
H.V. Fineberg (Ed.s). Understanding Risk:
Informing Decisions in a Democratic Society.
National Academy Press, Washington, DC.
Organization for Economic Co-operation and
Development (OECD). 1998. Towards
Sustainable Development -Environmental
Indicators. 130p.
Pontius,W 1999. Negotiating Regulations for
Microorganisms and Disinfection By-Products.
Journal of American Water Works Association
91:14.
Presidential/Congressional Commission on Risk
Assessment and Risk Management. 1997. Risk
Assessment and Risk Management in
Regulatory Decision-Making.
Sandman, Peter M. 1990. Getting to Maybe: Some
Communications Aspects of Siting Hazardous
Waste Facilities. In: Readings in Risk. T.S.
Glickman and M. Gough, (Ed.s). Resources for
the Future. Washington, DC. Pp. 233-245.
Science Advisory Board. 1990. Reducing Risk:
Setting Priorities and Strategies for
Environmental Protection
(EPA-SAB-EC-90-021).
Science Advisory Board. 1995. Beyond the
Horizon: Using Foresight to Protect the
Environmental Future (EPA-SAB-EC-95-007).
Science Advisory Board. 1997. Advisory on the
Problem Formulation Phase of EPA's
Watershed Ecological Risk Assessment Case
Studies (EPA-SAB-EPEC-ADV-97-001).
U.S. Environmental Protection Agency. 1987.
Unfinished Business: A Comparative
Assessment of Environmental Problems.
Washington, DC.
45
-------
U.S. Environmental Protection Agency. 1982.
Framework for Ecological Risk Assessment
(EPA/630/R-92/001). Office of Research and
Development, Washington, DC.
U.S. Environmental Protection Agency. 1989.
Risk Assessment Guidance for Superfund, Vols.
1 and 2 , Interim Final (EPA/540/1-89/001 and
002).
U.S. Environmental Protection Agency. 1990.
Methodology for Assessing Health Risks
Associated with Indirect Exposure to
Combustor Emissions—Interim Final
(EPA/600/6-90/003).
U.S. Environmental Protection Agency. 1993a.
Draft Addendum to the Indirect Exposure
Document.
U.S. Environmental Protection Agency. 1993b. A
Guidebook to Comparing Risks and Setting
Environmental Priorities (EPA 230-B-93-003).
Office of Policy, Planning, and Evaluation,
Washington, DC.
U.S. Environmental Protection Agency. 1997a.
Guidance on Cumulative Risk Assessment,
Part 1: Planning and Scoping. July 3,1997.
U.S. Environmental Protection Agency. 1997b.
EPA Strategic Plan (EPA/190-R-97-002).
Office of the Chief Financial Officer,
Washington, DC.
U.S. Environmental Protection Agency. 1998.
Guidelines for Ecological Risk Assessment
(EPA/630/R-95/002Fa).
U.S. Environmental Protection Agency. 1999.
Reinventing Environmental Protection: 1998
Annual Report (EPA100-R-99-002).
46
-------
47
-------
48
-------
-------
------- |