oEPA
United States
Environmental Protection
Agency
EPA/1 OO/R-14/004 July 2014
www.epa.gov/raf
Risk Assessment Forum White Paper:
Probabilistic Risk Assessment Methods
and Case Studies
Equation 2. Probabilistic Risk Assessment
[Concentration! fExposurel F Ingestion or ' Fjoxicity
Concentration I Inhalation Rate X Factor
in enwonmentl i ' '
= RISK
Mean risk
95th %
population
level risk
Office of the Science Advisor
Risk Assessment Forum
-------
EPA/100/R-14/004
Risk Assessment Forum White Paper:
Probabilistic Risk Assessment Methods and Case Studies
July 25, 2014
U.S. Environmental Protection Agency
Office of the Science Advisor
Risk Assessment Forum
Probabilistic Risk Analysis Technical Panel
Washington, D.C. 20460
-------
Disclaimer
This document has been reviewed in accordance with U.S. Environmental Protection Agency (EPA)
policy and approved for publication. Mention of trade names or commercial products does not
constitute endorsement or recommendation for use.
This document was produced by a Technical Panel of the EPA Risk Assessment Forum (RAF). The
authors drew on their experience in doing probabilistic assessments and interpreting them to
improve risk management of environmental and health hazards. Interviews, presentations and
dialogues with risk managers conducted by the Technical Panel have contributed to the insights
and recommendations in this white paper and the associated document titled Probabilistic Risk
Assessment to Inform Decision Making: Frequently Asked Questions.
U.S. Environmental Protection Agency (USEPA). 2014. Risk Assessment Forum White Paper:
Probabilistic Risk Assessment Methods and Case Studies. EPA/100/R-09/001A. Washington, D.C.
Risk Assessment Forum, Office of the Science Advisor, USEPA.
-------
Foreword
Throughout many of the U.S. Environmental Protection Agency's (EPA) program offices and
regions, various forms of probabilistic methods have been used to answer questions about
exposure and risk to humans, other organisms and the environment. Risk assessors, risk managers
and others, particularly within the scientific and research divisions, have recognized that more
sophisticated statistical and mathematical approaches could be utilized to enhance the quality and
accuracy of Agency risk assessments. Various stakeholders, inside and outside of the Agency, have
called for a more comprehensive characterization of risks, including uncertainties, to improve the
protection of sensitive or vulnerable populations and lifestages.
The EPA identified the need to examine the use of probabilistic approaches in Agency risk
assessments and decisions. The RAF developed this paper and the companion document,
Probabilistic Risk Assessment to Inform Decision Making: Frequently Asked Questions, to provide a
general overview of the value of probabilistic analyses and similar or related methods, as well as
provide examples of current applications across the Agency. Drafts of both documents were
released, with slightly different titles, for public comment and external peer review in August 2009.
An external peer review was held in Arlington, Virginia in May 2010.
The goal of these publications is not only to describe potential and actual uses of these tools, but
also to encourage their further implementation in human, ecological and environmental risk
analysis and related decision making. The enhanced use of probabilistic analyses to characterize
uncertainty in assessments will not only be responsive to external scientific advice (e.g.,
recommendations from the National Research Council) on how to further advance risk assessment
science, but also will help to address specific challenges faced by managers and increase the
confidence in the underlying analysis used to support Agency decisions.
Robert Kavlock
Interim Science Advisor
U.S. Environmental Protection Agency
in
-------
Probabilistic Risk Analysis Technical Panel
Haluk Ozkaynak, EPA Office of Research and Development (Co-Chair)
Robert Hetes, EPA Office of Research and Development (Co-Chair)
Kathryn Gallagher, EPA Office of Water (Co-Chair)
Chris Frey, North Carolina State University (while on temporary assignment at EPA) (White
Paper Co-Lead)
John Paul, EPA Office of Research and Development (White Paper Co-Lead)
Pasky Pascual, EPA Office of Research and Development (White Paper Co-Lead)
Marian Olsen, EPA Region 2 (Case Study Lead)
Mike Clipper, EPA Office of Solid Waste and Emergency Response
Michael Messner, EPA Office of Water
Keeve Nachman, currently at The Johns Hopkins Bloomberg School of Public Health
Zachary Pekar, EPA Office of Air and Radiation
Rita Schoeny, EPA Office of Research and Development
Cynthia Stahl, EPA Region 3
David Hrdy, EPA Office of Pesticide Programs
John Langstaff, EPA Office of Air and Radiation
Elizabeth Margosches, EPA Office of Chemical Safety and Pollution Prevention (retired)
Audrey Galizia, EPA Office of Research and Development
Nancy Rios-Jafolla, EPA Region 3
Donna Randall, EPA Office of Chemical Safety and Pollution Prevention
Khoan Dinh, EPA Office of Chemical Safety and Pollution Prevention (retired)
Harvey Richmond, EPA Office of Air and Radiation (retired)
Other Contributing Authors
Jonathan Chen, EPA Office of Chemical Safety and Pollution Prevention
Lisa Conner, EPA Office of Air and Radiation
Janet Burke, EPA Office of Research and Development
Allison Hess, EPA Region 2
Kelly Sherman, EPA Office of Chemical Safety and Pollution Prevention
Woodrow Setzer, EPA Office of Research and Development
Valerie Zartarian, EPA Office of Research and Development
EPA Risk Assessment Forum Science Coordinators
Julie Fitzpatrick, EPA Office of the Science Advisor
Gary Bangs, EPA Office of the Science Advisor (retired)
External Peer Reviewers
Scott Person (Chair), Applied Biomathematics
Annette Guiseppi-Elie, DuPont Engineering
Dale Hattis, Clark University
Igor Linkov, U.S. Army Engineer Research and Development Center
John Toll, Windward Environmental LLC
IV
-------
Table of Contents
Foreword iii
List of Figures and Tables viii
List of Acronyms and Abbreviations ix
EXECUTIVE SUMMARY 1
1. INTRODUCTION: RELEVANCE OF UNCERTAINTY TO DECISION MAKING: HOW
PROBABILISTIC APPROACHES CAN HELP 4
1.1. EPA Decision Making 4
1.2. The Role of Probabilistic Risk Analysis in Characterizing Uncertainty and
Variability 4
1.3. Goals and Intended Audience 5
1.4. Overview of This Document 5
1.5. What Are Common Challenges Facing EPA Risk Decision Makers? 5
1.6. What Are Key Uncertainty and Variability Questions Often Asked by Decision
Makers? 6
1.7. Why Is the Implementation of Probabilistic Risk Analysis Important? 9
1.8. How Does EPA Typically Address Scientific Uncertainty and Variability? 10
1.9. What Are the Limitations of Relying on Default-Based Deterministic Approaches? 11
1.10. What Is EPA's Experience with the Use of Probabilistic Risk Analysis? 12
2. PROBABILISTIC RISK ANALYSIS 15
2.1. What Are Variability and Uncertainty, and How Are They Relevant to Decision
Making? 15
2.1.1. Variability 15
2.1.2. Uncertainty 15
2.2. When Is Probabilistic Risk Analysis Applicable or Useful? 17
2.3. How Can Probabilistic Risk Analysis Be Incorporated Into Assessments? 17
2.4. What Are the Scientific Community's Views on Probabilistic Risk Analysis, and
What Is the Institutional Support for Its Use in Performing Assessments? 18
2.5. Additional Advantages of Using Probabilistic Risk Analysis and How It Can
Provide More Comprehensive, Rigorous Scientific Information in Support of
Regulatory Decisions 19
2.6. What Are the Challenges to Implementation of Probabilistic Analyses? 19
2.7. How Can Probabilistic Risk Analysis Support Specific Regulatory Decision
Making? 20
2.8. Does Probabilistic Risk Analysis Require More Resources Than Default-Based
Deterministic Approaches? 21
2.9. Does Probabilistic Risk Analysis Require More Data Than Conventional
Approaches? 21
2.10. Can Probabilistic Risk Analysis Be Used to Screen Risks or Only in Complex or
Refined Assessments? 22
2.11. Does Probabilistic Risk Analysis Present Unique Challenges to Model Evaluation? 23
2.12. How Do You Communicate the Results of Probabilistic Risk Analysis? 23
2.13. Are the Results of Probabilistic Risk Analysis Difficultto Communicate to Decision
Makers and Stakeholders? 24
-------
3. AN OVERVIEW OF SOME OF THE TECHNIQUES USED IN PROBABILISTIC RISK
ANALYSIS 26
3.1. What Is the General Conceptual Approach in Probabilistic Risk Analysis? 26
3.2. What Levels and Types of Probabilistic Risk Analyses Are There and How Are
They Used? 26
3.3. What Are Some Specific Aspects of and Issues Related to Methodology for
Probabilistic Risk Analysis? 29
3.3.1. Developing a Probabilistic Risk Analysis Model 29
3.3.2. Dealing With Dependencies Among Probabilistic Inputs 29
3.3.3. Conducting the Probabilistic Analysis 30
4. SUMMARY AND RECOMMENDATIONS 33
4.1. Probabilistic Risk Analysis and Related Analyses Can Improve the Decision-
Making Process at EPA 33
4.2. Major Challenges to Using Probabilistic Risk Analysis to Support Decisions 34
4.3. Recommendations for Enhanced Utilization of Probabilistic Risk Analysis at EPA 34
GLOSSARY 37
REFERENCES 42
BIBLIOGRAPHY 48
Probabilistic Risk Analysis Methodology—General 48
Probabilistic Risk Analysis and Decision Making 49
Probabilistic Risk Analysis Methodology—Specific Aspects 49
Sensitivity Analysis 50
Case Study Examples of Probabilistic Risk Analysis—EPA (See Also the PRA Case
Studies Appendix) 51
Case Study Examples of Probabilistic Risk Analysis—Other 52
APPENDIX: CASE STUDY EXAMPLES OF THE APPLICATION OF PROBABILISTIC RISK
ANALYSIS IN U.S. ENVIRONMENTAL PROTECTION AGENCY REGULATORY DECISION
MAKING 54
A. OVERVIEW 55
B. INTRODUCTION 55
C. OVERALL APPROACH TO PROBABILISTIC RISK ANALYSIS AT THE U.S.
ENVIRONMENTAL PROTECTION AGENCY 56
C.I.U.S. Environmental Protection Agency Guidance and Policies on Probabilistic Risk
Analysis 56
C.2.Categorizing Case Studies 56
Group 1 Case Studies 57
Group 2 Case Studies 57
Group 3 Case Studies 58
VI
-------
D. CASE STUDY SUMMARIES 63
D.I. Group 1 Case Studies 63
Case Study 1: Sensitivity Analysis of Key Variables in Probabilistic Assessment
of Children's Exposure to Arsenic in Chromated Copper Arsenate Pressure-
Treated Wood 63
Case Study 2: Assessment of the Relative Contribution of Atmospheric
Deposition to Watershed Contamination 64
D.2. Group 2 Case Studies 65
Case Study 3: Probabilistic Assessment of Angling Duration Used in the
Assessment of Exposure to Hudson River Sediments via Consumption of
Contaminated Fish 65
Case Study 4: Probabilistic Analysis of Dietary Exposure to Pesticides for Use
in Setting Tolerance Levels 65
Case Study 5: One-Dimensional Probabilistic Risk Analysis of Exposure to
Polychlorinated Biphenyls via Consumption of Fish From a Contaminated
Sediment Site 66
Case Study 6: Probabilistic Sensitivity Analysis of Expert Elicitation of
Concentration-Response Relationship Between Fine Particulate Matter
Exposure and Mortality 69
Case Study 7: Environmental Monitoring and Assessment Program: Using
Probabilistic Sampling to Evaluate the Condition of the Nation's Aquatic
Resources 70
D.3. Group 3 Case Studies 71
Case Study 8: Two-Dimensional Probabilistic Risk Analysis of
Cryptosporidium in Public Water Supplies, With Bayesian Approaches to
Uncertainty Analysis 71
Case Study 9: Two-Dimensional Probabilistic Model of Children's Exposure to
Arsenic in Chromated Copper Arsenate Pressure-Treated Wood 72
Case Study 10: Two-Dimensional Probabilistic Exposure Assessment of Ozone 74
Case Study 11: Analysis of Microenvironmental Exposures to Fine Particulate
Matter for a Population Living in Philadelphia, Pennsylvania 76
Case Study 12: Probabilistic Analysis in Cumulative Risk Assessment of
Organophosphorus Pesticides 77
Case Study 13: Probabilistic Ecological Effects Risk Assessment Models for
Evaluating Pesticide Use 78
Case Study 14: Expert Elicitation of Concentration-Response Relationship
Between Fine Particulate Matter Exposure and Mortality 80
Case Study 15: Expert Elicitation of Sea-Level Rise Resulting From Global
Climate Change 81
Case Study 16: Knowledge Elicitation for a Bayesian Belief Network Model of
Stream Ecology 82
CASE STUDY REFERENCES 85
vn
-------
List of Figures and Tables
Figures
Figure 1. General Phases of the Risk Assessment Process 8
Figure 2. Tiered Approach for Risk Assessment 10
Figure 3. Graphical Description of the Likelihood (Probability) of Risk 25
Figure 4. Diagrammatic Comparison of Three Alternative Probabilistic Approaches for the
Same Exposure Assessment 28
Appendix Figures
Figure A-l. Monte Carlo Cancer Summary Based on a One-Dimensional Probabilistic Risk
Analysis of Exposure to Polychlorinated Biphenyls 72
Figure A-2. Monte Carlo Noncancer Hazard Index Summary Based on a One-Dimensional
Probabilistic Risk Analysis of Exposure to Polychlorinated Biphenyls 73
Figure A-3. Uncertainty Distribution Model Results 80
Tables
Table 1. Selected Examples of EPA Applications of Probabilistic Risk Assessment Techniques 14
Appendix Table
Table A-l. Case Study Examples of EPA Applications of Deterministic and Probabilistic Risk
Assessment Techniques 59
vni
-------
List of Acronyms and Abbreviations
l-D MCA One-Dimensional Monte Carlo Analysis
2-D MCA Two-Dimensional Monte Carlo Analysis
APEX Air Pollutants Exposure Model
BBN Bayesian Belief Network
CAA Clean Air Act
CASAC Clean Air Scientific Advisory Committee
CCA Chromated Copper Arsenate
CPSC Consumer Product Safety Commission
CRA Cumulative Risk Assessment
CSFII Continuing Survey of Food Intake by Individuals
CWA Clean Water Act
DEEM Dietary Exposure Evaluation Model
DRA Deterministic Risk Assessment
ORES Dietary Risk Evaluation System
ERA Ecological Risk Assessment
EMAP Environmental Monitoring and Assessment Program
EPA U.S. Environmental Protection Agency
FACA Federal Advisory Committee Act
FAQ Frequently Asked Questions
FDA U.S. Food and Drug Administration
FFDCA Federal Food, Drug, and Cosmetic Act
FIFRA Federal Insecticide, Fungicide, and Rodenticide Act
FQPA Food Quality Protection Act
GAO Government Accountability Office
HHRA Human Health Risk Assessment
HI Hazard Index
IPCC Intergovernmental Panel on Climate Change
IRIS Integrated Risk Information System
LHS Latin Hypercube Sampling
LOAEL Lowest-Observed-Adverse-Effect Level
LT Long-Term
LT2 Long-Term 2 Enhanced Surface Water Treatment Rule
MCA Monte Carlo Analysis
MCS Monte Carlo Simulation
MEE Microexposure Event Analysis
MOEs Margins of Exposure
NAAQS National Ambient Air Quality Standards
NAS National Academy of Sciences
NERL National Exposure Research Laboratory
NOAEL No-Observed-Adverse-Effect Level
NRC National Research Council
OAQPS Office of Air Quality Planning and Standards
OAR Office of Air and Radiation
OCSPP Office of Chemical Safety and Pollution Prevention
OERR Office of Emergency and Remedial Response
OGWDW Office of Groundwater and Drinking Water
OMB Office of Management and Budget
OP Organophosphorus Pesticide
IX
-------
OPP Office of Pesticide Programs
ORD Office of Research and Development
OSA Office of the Science Advisor
OSTP Office of Science and Technology Policy
OSWER Office of Solid Waste and Emergency Response
OW Office of Water
PAH Polycyclic Aromatic Hydrocarbon
PCB Polychlorinated Biphenyl
PCCRARM Presidential/Congressional Commission on Risk Assessment and Risk Management
PDF Pesticide Data Program
PM Particulate Matter
PRA Probabilistic Risk Assessment
RAF Risk Assessment Forum
RfC Reference Concentration (Inhalation)
RfD Reference Dose (Oral)
RIA Regulatory Impact Analysis
RME Reasonable Maximum Exposure
SAB Science Advisory Board
SAP Scientific Advisory Panel
SHEDS Stochastic Human Exposure and Dose Simulation Model
SHEDS-PM Stochastic Human Exposure and Dose Simulation Model for Particulate Matter
STPC Science and Technology Policy Council
TRIM.Expo Total Risk Integrated Methodology/Exposure Model
UI Uncertainty Interval
USDA U.S. Department of Agriculture
USGS U.S. Geological Survey
WHO World Health Organization
-------
EXECUTIVE SUMMARY
Probabilistic risk assessment (PRA), in its simplest form, is a group of techniques that incorporate
uncertainty and variability into risk assessments. Variability refers to the inherent natural
variation, diversity and heterogeneity across time, space or individuals within a population or
lifestage, while uncertainty refers to imperfect knowledge or a lack of precise knowledge of the
physical world, either for specific values of interest or in the description of the system (USEPA
20lie). Variability and uncertainty have the potential to result in overestimates or underestimates
of the predicted risk.
PRA provides estimates of the range and likelihood of a hazard, exposure or risk, rather than a
single point estimate. Stakeholders inside and outside of the Agency have recommended a more
complete characterization of risks, including uncertainties and variability, in protecting more
sensitive or vulnerable populations and lifestages. PRA can be used to support risk management by
assessment of impacts of uncertainties on each of the potential decision alternatives.
Numerous advisory bodies, such as the Science Advisory Board (SAB) and the National Research
Council (NRC) of the National Academy of Sciences (NAS), have recommended that EPA incorporate
probabilistic analyses into the Agency's decision-making process. EPA's Risk Assessment Forum
(RAF) formed a Technical Panel, consisting of representatives from the Agency's program and
regional offices, to develop this white paper and its companion document, titled Probabilistic Risk
Assessment to Inform Decision Making: Frequently Asked Questions (FAQ). The RAF is recommending
the development of Agency resources, such as a clearinghouse of PRA case studies, best practices,
resources and seminars, to raise general knowledge about how these probabilistic tools can be
used.
The intended goal of this white paper is to explain how EPA can use probabilistic methods to
address data, model and scenario uncertainty and variability by capitalizing on the wide array of
tools and methods that comprise PRA. This white paper describes where PRA can facilitate more
informed risk management decision making through better understanding of uncertainty and
variability related to Agency decisions. The information contained in this document is intended for
both risk analysts and managers faced with determining when and how to apply these tools to
inform their decisions. This document does not prescribe a specific approach but, rather, describes
the various stages and aspects of an assessment or decision process in which probabilistic
assessment tools may add value.
Probabilistic Risk Assessment
PRA is an analytical methodology used to incorporate information regarding uncertainty and/or
variability into analyses to provide insight regarding the degree of certainty of a risk estimate and
how the risk estimate varies among different members of an exposed population, including
sensitive populations or lifestages. Traditional approaches, such as deterministic analyses, often
report risks as "central tendency," "high end" (e.g., 90th percentile or above) or "maximum
anticipated exposure," but PRA can be used to describe more completely the uncertainty
surrounding such estimates and identify the key contributors to variability or uncertainty in
predicted exposures or risk estimates. This information then can be used by decision makers to
achieve a science-based level of safety, to compare the risks related to different management
options, or to invest in research with the greatest impact on risk estimate uncertainty.
To support regulatory decision making, PRA can provide information to decision makers on specific
questions related to uncertainty and variability. For example, in the context of a decision analysis
that has been conducted, PRA can: identify "tipping points" where the decision would be different if
-------
the risk estimates were different; estimate the degree of confidence in a particular decision; and
help to estimate trade-offs related to different risks or management options. PRA can provide
useful (even critical) information about the uncertainties and variability in the data, models,
scenario, expert judgments and values incorporated in risk assessments to support decision making
across the Agency.
PRA is applicable to both human health risk assessment (HHRA) and ecological risk assessment
(ERA); however, there are differences between how PRA is used for the two. Both HHRA and ERA
have a similar structure and use the same risk assessment steps, but HHRA focuses on individuals, a
single species, morbidity and mortality, but ERA is more concerned with multiple populations of
organisms (e.g., individual species offish in a river) or ecological integrity (e.g., will the types of
species living in the river change over time). In ERA, there also is a reliance on indicators of impacts
(e.g., sentinel species and other metrics).
Risk Assessment at EPA
PRA began playing an increasingly important role in Agency risk assessments following the 1997
release of EPA's Policy for Use of Probabilistic Analysis in Risk Assessment at the U.S. Environmental
Protection Agency (USEPA 1997a) and publication of the Guiding Principles for Monte-Carlo Analysis
(USEPA 1997b). PRA was a major focus in an associated review of EPA risk assessment practices by
the SAB (USEPA 2007b). The NRC recommended that EPA adopt a "tiered" approach for selecting
the level of detail used in uncertainty and variability assessment (NRC 2009). Furthermore, the NRC
recommended that a discussion about the level of detail used for uncertainty analysis and
variability assessment should be an explicit part of the planning, scoping and problem formulation
step in the risk assessment process. Both this white paper and the companion FAQ document take
into account recommendations on risk assessment processes described in the NRC's report Science
and Decisions: Advancing Risk Assessment (NRC 2009) and Environmental Decisions in the Face of
Uncertainty (IOM 2013).
EPA's recent risk assessment publications, including the document titled Framework for Human
Health Risk Assessment to Inform Decision Making (UAEPA 2014b) as well as this white paper,
emphasize the importance of communicating the results of a PRA because it provides the range and
likelihood estimates for one or more aspects of hazard, exposure or risk, rather than a single point
estimate. Risk assessors are responsible for sharing information on probabilistic results so that
decision makers have a clear understanding of quantitative assessments of uncertainty and
variability, and how this information will affect the decision. Effective communication between the
risk assessor and decision maker is key to promote understanding and use of the results from the
PRA.
PRA generally requires more resources than standard Agency default-based deterministic
approaches. Appropriately trained staff and the availability of adequate tools, methods and
guidance are essential for the application of PRA. Proper application of probabilistic methods
requires not only software and data, but also guidance and training for analysts using the tools, and
for managers and decision makers tasked with interpreting and communicating the results. In most
circumstances, probabilistic assessments may take more time and effort to conduct than
conventional approaches, primarily because of the comprehensive inclusion of available
information on model inputs. The potentially higher resource costs may be offset, however, by a
more informed decision than would be provided by a comparable deterministic analysis.
Content of the White Paper and Frequently Asked Questions Companion Documents
These two documents describe how PRA can be applied to enhance the scientific foundation of
EPA's decision making across the Agency. This white paper describes the challenges faced by EPA
-------
decision makers, defines and explains the basic principles of probabilistic analysis, briefly
highlights instances where these techniques have been implemented in EPA decisions, and
describes criteria that may be useful in determining whether and how the application of
probabilistic methods may be useful and/or applicable to decision making. This white paper also
describes commonly employed methods to address uncertainty and variability, including those
used in the consideration of uncertainty in scenarios and uncertainty in models. Additionally, it
addresses uncertainty and variability in the inputs and outputs of models and the impact of these
uncertainties on each of the potential management options. A general description of the range of
methods from simple to complex, rapid to more time consuming and least to most resource
intensive is provided, as well as uses of these methods.
Both documents address issues such as uncertainty and variability, their relevance to decision
making and the PRA goal to provide quantitative characterization of the uncertainty and variability
in estimates of hazard, exposure, or risk. The difference between the white paper and the FAQs
document is the level of detail provided about PRA concepts and practices, and the intended
audience (e.g., risk assessors vs. decision makers). Detailed examples of applications of these
methods are provided in the Appendix of this white paper, which is titled "Case Study Examples of
the Application of Probabilistic Risk Analysis in U.S. Environmental Protection Agency Decision
Making." The white paper Appendix includes 16 case studies—11 HHRA and 5 ERA examples—that
illustrate how EPA's program and regional offices have used probabilistic techniques in risk
assessment. To aid in describing how these tools were applied, the 16 case studies are subdivided
among 3 categories for purposes of this document. Group 1 includes 2 case studies demonstrating
point estimate, including sensitivity analysis; Group 2 is comprised of 5 case studies demonstrating
probabilistic risk analysis, including one-dimensional Monte Carlo analysis and probabilistic
sensitivity analysis; and Group 3 includes 9 case studies demonstrating advanced probabilistic risk
analysis, including two-dimensional Monte Carlo analysis with micro exposure (micro
environments) modeling, Bayesian statistics, geostatistics and expert elicitation.
The FAQ document provides answers to common questions regarding PRA, including key concepts
such as scientific and institutional motivations for the use of PRA, and challenges in the application
of probabilistic techniques. The principal reason for including PRA as an option in the risk
assessor's toolbox is its ability to support the refinement and improvement of the information
leading to decision making by incorporating known uncertainties.
-------
1. INTRODUCTION: RELEVANCE OF UNCERTAINTY TO
DECISION MAKING: HOW PROBABILISTIC APPROACHES
CAN HELP
1.1. EPA Decision Making
To discuss where probabilistic approaches can aid EPA's decision making, it is important to
generally describe the Agency's current decision-making processes and how better understanding
and improving elements within these processes can clarify where probabilistic approaches might
provide benefits. The enhanced use of PRA and characterization of uncertainty would allow EPA
decision makers opportunities to use a more robust and transparent process, which may allow
greater responsiveness to outside comments and recommendations. Such an approach would
support higher quality EPA assessments and improve confidence in Agency decisions.
There are two major areas in the decision-making process that might be improved with PRA.
Scientists currently are generally focused on the first area—the understanding of data, model and
scenario uncertainties and variability. The second area is one that has not, until recently and only in
a limited fashion, been used by EPA decision makers. This area is formal decision analysis. With
decision analytic techniques, decision makers can weigh the relative importance of risk information
compared to other information in making the decision, understand how uncertainty affects the
relative attractiveness of potential decision alternatives, and assess overall confidence in a decision.
In addition to data, model and scenario uncertainty, there is a separate category of uncertainties
specifically associated with how the decision criteria relate to the decision alternatives. Although it
is quite relevant to risk management decisions, the topic and decision analysis in general are
outside of the scope of this report This white paper focuses on technical information that would
allow better understanding of the relationships among alternative decisions in assessing risks.
1.2. The Role of Probabilistic Risk Analysis in Characterizing
Uncertainty and Variability
Probabilistic analyses include techniques that can be applied formally to address both uncertainty
and variability, typically arising from limitations of data, models or adequately formulating the
scenarios used in assessing risks. Probability is used in science, business, economics and other
fields to examine existing data and estimate the chance of an event, from health effects to rain to
mental fatigue. One can use probability (chance) to quantify the frequency of occurrence or the
degree of belief in information. For variability, probability distributions are interpreted as
representing the relative frequency of a given state of the system (e.g., that the data are distributed
in a certain way); for uncertainty, they represent the degree of belief or confidence that a given
state of the system exists (e.g., that we have the appropriate data; Cullen and Frey 1999). PRA often
is defined narrowly to indicate a statistical or thought process used to analyze and evaluate the
variability of available data or to look at uncertainty across data sets.
For the purposes of this document, PRA is a term used to describe a process that employs
probability to incorporate variability in data sets and/or the uncertainty in information (such as
data or models) into analyses that support environmental risk-based decision making. PRA is used
here broadly to include both quantitative and qualitative methods for dealing with scenario, model
and input uncertainty. Probabilistic techniques can be used with other types of analysis, such as
benefit-cost analysis, regulatory impact analysis and engineering performance standards; thus, they
can be used for a variety of applications and by experts in many disciplines.
-------
1.3. Goals and Intended Audience
The primary goals of this white paper are to introduce PRA, describe how it can be used to better
inform and improve the decision-making process, and provide case studies where it has been used
in human health and ecological analyses at EPA (see the Appendix for the case studies). A secondary
goal of this paper is to bridge communication gaps regarding PRA among analysts of various
disciplines, between these analysts and Agency decision makers, and among affected stakeholders.
This white paper also is intended to serve as a communication tool to introduce key concepts and
background information on approaches to risk analysis that incorporate uncertainty and provide a
more comprehensive treatment of variability. Risk analysts, decision makers and affected
stakeholders can benefit from understanding the potential uses of PRA. PRA and related
approaches can be used to identify additional research that may reduce uncertainty and more
thoroughly characterize variability in a risk assessment This white paper explains how PRA can
enhance the decision-making processes faced by managers at EPA by better characterizing data,
model, scenario and decision uncertainties.
1.4. Overview of This Document
This white paper provides an overview of EPA's interest and experience in addressing uncertainty
and variability using probabilistic methods in risk assessment; identifies key questions asked or
faced by Agency decision makers; demonstrates how conventional deterministic approaches to risk
analysis may not answer these questions fully; provides examples of applications; and shows how
and why "probabilistic risk analysis" (broadly defined) could provide added value, compared to
traditional methods, with regard to regulatory decision making by more fully characterizing risk
estimates and exploring decision uncertainties. For the purposes of this white paper, PRA and
related tools for both human health and ecological assessments include a range of approaches, from
statistical tools, such as sensitivity analysis, to multi-dimensional Monte Carlo models, geospatial
approaches and expert elicitation. Key points addressed by this document include definitions and
key concepts pertaining to PRA, benefits and challenges of PRA, a general conceptual framework for
PRA, conclusions regarding products and insights obtained from PRA, and examples where EPA has
used PRA in human health and ecological analyses. A Glossary and a Bibliography also are provided.
1.5. What Are Common Challenges Facing EPA Risk Decision
Makers?
EPA operates under statutory and regulatory constraints that often limit the types of criteria that
can be considered (including whether the use of PRA is appropriate) and impose strict timeframes
in which decisions must be made. Typically, the decision begins with understanding (1) who or
what will be protected; (2) the relationship between the data and decision alternatives; and (3) the
impact of data, model and decision uncertainties related to each decision alternative. These are
among the considerations of the planning and scoping and problem formulation phases of risk
assessment (US EPA 2014). EPA decision makers need to consider multiple decision criteria, which
are informed by varying degrees of confidence in the underlying information. Decision makers need
to balance the regulatory/ statutory requirements and timeframes, resources (i.e., expertise, costs
of the analysis, review times, etc.) to conduct the assessment, management options, and
stakeholders while at the same time keeping risk assessment and decision making separate.
Uncertainty can be introduced into any assessment at any step in the process, even when using
highly accurate data with the most sophisticated models. Uncertainty can be reduced or better
characterized through knowledge. Variability or natural heterogeneity is inherent in natural
systems and therefore cannot be reduced, but can be examined and described. Uncertainty in
decisions is unavoidable because real-world situations cannot be perfectly measured, modeled or
-------
predicted. As a result, EPA decision makers face scientifically complex problems that are
compounded by varying levels of uncertainty and variability. If uncertainty and variability have not
been well characterized or acknowledged, potential complications arise in the process of decision
making. Increased uncertainty can make it more difficult to determine, with reasonable confidence,
the balance point between the costs of regulation and the implications for avoiding damages and
producing benefits. Characterization facilitated by probabilistic analyses can provide insight into
weighing the relative costs and benefits of varying levels of regulation and also can assist in risk
communication activities.
Decision makers often want to know who is at risk and by how much, the tradeoffs between
alternative actions and the likely or possible consequences of decisions. To this end, it is
particularly useful for decision makers to understand the distribution of risk across potentially
impacted populations and ecological systems. It can be important to know the number of
individuals experiencing different magnitudes of risk, the differences in risk magnitude experienced
by individuals in different lifestages or populations or the probability of an event that may lead to
unacceptable levels of risk. Given the limitations of data, traditional methods of risk analyses are
not well suited to produce such estimates. Probabilistic analytical methods are capable of
addressing these shortcomings and can contribute to a more thorough recognition of the impact of
data gaps on the projected risk estimates. Although PRA can be used to characterize the uncertainty
and variability in situations with limited data, currently there is not extensive experience using PRA
to characterize the range of effects or dose-response relationships for populations, including
sensitive populations and lifestages.
Other challenges facing EPA decision makers include the need to consider multiple decision criteria,
which are informed by varying degrees of confidence in the underlying information, understanding
the relationship between and among those decision criteria (including multi-pollutant and multi-
media effects) and the decision alternatives, and the timeliness of the decision making.
Furthermore, even when PRA is used, EPA decision makers must be mindful of potential misuses
and obfuscations when conducting or presenting PRA results. Decision makers also need to
consider the evolving science behind PRA. As the use of PRA increases decision makers will become
more familiar with the techniques and their application.
A risk assessment process needs to consider uncertainties, variability and the rationale or factors
influencing how they may be addressed by a decision maker. Decision makers need a foundation for
estimating the value of collecting additional information to allow for better informed decisions.
There are costs associated with ignoring uncertainty (McConnell 1997 and Toll 1999), and a focus
by decision makers on the information provided by uncertainty analysis can strengthen their
choices.
1.6. What Are Key Uncertainty and Variability Questions Often Asked
by Decision Makers?
As described above, determining the decision-making context and specific concerns is a critical first
step toward developing a useful and responsive risk assessment that will support the decision. For
example, the appropriate focus and level of detail of the analysis should be commensurate with the
needs of the decision maker and stakeholders, as well as the appropriate use of science. Analyses
often are conducted at a level of detail dictated by the issue being addressed, the breadth and
quality of the available information upon which to base an analysis, and the significance
surrounding a decision. The analytical process tends to be iterative. Although a guiding set of
questions may frame the initial analyses, additional questions can arise that further direct or even
reframe the analyses.
-------
Based on a series of discussions with Agency decision makers and risk assessors, some typical
questions about uncertainty and variability relevant to risk analyses including:
n Factors influencing decision uncertainty:
• Would my decision be different if the data were different, improved or expanded?
Would additional data collection and research likely lead to a different decision?
How long will it take to collect the information, how much would it cost, and would
the resulting decision be significantly altered?
• What are the liabilities and consequences of making a decision under the current
level of knowledge and uncertainty?
• How do the alternatives and their associated uncertainty and variability affect the
target population or lifestage?
n Considerations for evaluating data or method uncertainty:
• How representative or conservative is the estimate due to data or method
uncertainty (also incorporating variability)?
• What are the major gaps in knowledge, and what are the major assumptions used in
the assessment? How reasonable are the assumptions?
n Issues arising when addressing variability:
• Can a probabilistic approach (e.g., to better characterize uncertainties and
variability) be accomplished in a timely manner?
• What is the desired percentile of the population to be protected? By choosing this
percentile, who may not be protected?
The questions that arise concerning uncertainty and variability change depending on the stage and
nature of the decision-making process and analysis. General phases of the risk assessment process
are illustrated in Figure 1. For further information on the process of decision making, we suggest
referring to the description provided by EPA Region 3 on the Multi-Criteria Integrated Resource
-------
1
1:
Planning and Scoping
• Acs- 0« h
• Resources
• Past Experiences
Problem Formulation
Conceptual Model
•Sources 'Receptors
•Stressors -Endpoinls
• Pathway/floutes
Analysis Plan
• Methods
• Models
* Data Gaps
• Uncertainties
Integration of Exposure, Hazard and Dose-Response Information Considering:
Time Related Aspects, Vulnerability and Subpopulations with Special exposure
Single Strcssor Information
• Tomcologic independence
* Toxicoiof JC Similarity
Multiple Strcv,or Information
• Stressor Interactions
• Joint Chemical Ton'Oty
Measures and metrics
• Decijion Indices • Common Metric
• Probabilistic Approaches
• Quantitative Approaches
Risk Description
Central Tendency and High-End Individual Risk
Population flish
a.jK to important Sub&oputations
Uncertainty Analysts
• &ein| Expert About Uncertainty
- Unc«rt«intV and Variability
* Sensitivity Analysts
Information Provided by Risk Assessment
Figure 1. General Phases of the Risk Assessment Process. Risk assessment is an iterative process
comprised of planning, scoping and problem formulation; analysis (e.g., hazard identification, dose-
response assessment and exposure assessment); interpretation and risk characterization; and risk
communication. The highlighted boxes explain how PRA fits into the overall process.
Assessment Internet page at http://www.epa.gov/reg3esdl/data/mira.htm. The utility of various
levels of analysis and sophistication in answering these questions is illustrated in the case studies
described in Section 1.10 and presented in the Appendix of this white paper. References to
examples beyond these EPA case studies can be found in the Bibliography. Additionally, Lester et al.
(2007) identified more than 20 PRA application case studies (including EPA examples) performed
since 2000; these case study examples are categorized as site-specific applications and regional risk
assessments.
-------
1.7. Why Is the Implementation of Probabilistic Risk Analysis
Important?
The principal reason for the inclusion of PRA as an option in the risk assessor's toolbox is PRA's
ability to support refinement and improvement of the information leading to decision making by
incorporating known uncertainties. Beginning as early as the 1980s, expert scientific advisory
groups, such as the National Research Council (NRC), recommended that risk analyses include a
clear discussion of the uncertainties in risk estimation (NRC 1983). The NRC stated the need to
describe uncertainty and to capture variability in risk estimates (NRC 1994). The Presidential/
Congressional Commission on Risk Assessment and Risk Management (PCCRARM) recommended
against a requirement or need for a "bright line" or single-number level of risk (PCCRARM 1997).
See Section 2.4 for more information regarding the scientific community's opinion on the use of
PRA.
Regulatory science often requires selection of a limit for a contaminant, yet that limit always
contains uncertainty as to how protective it is. PRA and related tools quantitatively describe the
very real variations in natural systems and living organisms, how they respond to stressors, and the
uncertainty in estimating those responses.
Risk characterization became EPA policy in 1995 (USEPA 1995b), and the principles of
transparency, clarity, consistency and reasonableness are explicated in the 2000 Risk
Characterization Handbook (USEPA 2000a). Transparency, clarity, consistency and reasonableness
criteria require decision makers to describe and explain the uncertainties, variability and known
data gaps in the risk analysis and how they affect the resulting decision-making processes (USEPA
1992, 1995a, 2000a).
The use of probabilistic methods also has received support from some decision makers within the
Agency, and these methods have been incorporated into a number of EPA decisions to date.
Program offices, such as the Office of Pesticide Programs (OPP), Office of Solid Waste and
Emergency Response (OSWER), Office of Air and Radiation (OAR), and Office of Water (OW), as well
as the Office of Research and Development (ORD), have utilized probabilistic approaches in
different ways and to varying extents, for both human exposure and ecological risk analyses. In
addition, OSWER has provided explicit guidance on the use of probabilistic approaches for
exposure analysis (USEPA 2001). Some program offices have held training sessions on Monte Carlo
simulation (MCS) software that is used frequently in probabilistic analyses.
The NRC recommended that EPA should adopt a tiered approach for selecting the level of detail
used in uncertainty and variability assessment (NRC 2009). Furthermore, NRC recommended that a
discussion about the level of detail used for uncertainty analysis and variability assessment should
be an explicit part of the planning, scoping and problem formulation step in the risk assessment
process. The way that PRA fits into a graduated hierarchical (tiered) approach is more fully
described in Section 2.10 and illustrated in Figure 2.
When it is beneficial to refine risk estimates, the use of PRA can help in the characterization and
communication of uncertainty, variability and the impact of data gaps in risk analyses for assessors,
decision makers and stakeholders (including the target population or lifestage).
-------
O 42
if =
II
I
cr
e
Tier 3: Advanced PRA (High Complexity)
Probabilistic techniques (e.g., 2-D Monte Carlo Analysis)
•f Probabilistic exposure assumptions
+ Detailed, site-specific analysis and/or modeling
Decision-making cycle: Evaluating the
adequacy of the risk assessment and the
value of additional complexity/level of
effort
Tier 2: PRA (Moderate Complexity)
•f Realistic exposure assumptions
•f More detailed analysis and/or modeling
Decision-making cycle: Evaluating the
adequacy of the risk assessment and
the value of additional complexity/level
of effort
Tier 1: Deterministic (Point Estimate) RA
(Screening Level)
"f Conservative exposure assumptions
+ Simple analysis and/or modeling
1
f fe
in H
ra re
Planning and Scoping/Problem Formulation
Figure 2. Tiered Approach for Risk Assessment. The applicability of a probabilistic approach depends on
the needs of decision makers and stakeholders. Assessments that are high in complexity and regulatory
significance benefit from the application of probabilistic techniques.
Source: Adapted from USEPA 2004a and WHO 2008.
1.8. How Does EPA Typically Address Scientific Uncertainty and
Variability?
Environmental assessments can be complex, such as covering exposure to multiple chemicals in
multiple media for a wide-ranging population. The Agency has developed simplified approaches to
characterize risks associated with such complex assessments through the use of point estimates for
model variables or parameters. Such an approach typically produces point estimates of risks (e.g.,
10-5 or a lifetime probability of cancer risk of one individual in 100,000). These often are called
"deterministic" assessments. As a result of the use of point estimates for variables in model
algorithms, deterministic risk results usually are reported as what are assumed to be either average
or worst-case estimates. They do not contain any quantitative estimate of the uncertainty in that
estimate, nor report what percentile of the exposed population the estimate applies. The methods
typically used in EPA risk assessments rely on a combination of point values with potentially
varying levels of conservatism and certainty, yielding a point estimate of exposure at some point in
the range of possible risks.
Because uncertainty is inherent in all risk assessments, it is important that the risk assessment
process enable handling uncertainties in a logical way that is transparent and scientifically
defensible, consistent with the Agency's statutory mission and responsive to the needs of decision
makers (NRC 1994). Uncertainty is a factor in both ecological and human health risk assessments.
10
-------
For human health risk assessments, uncertainties arise for both noncancer and cancer endpoints.
Thus, when data are missing, EPA often uses several options to provide boundaries on uncertainty
and variability in an attempt to avoid risk underestimation; attempting to give a single
quantification of how much confidence there is in the risk estimate may not be informative or
feasible.
In exposure assessment, for example, the practice at EPA is to collect new data where they are
needed and where time and resources allow. Alternative approaches to address uncertainty include
narrowing the scope of the assessment; using screening-level default assumptions that include
upper-end values and/or central tendency values that are generally combined to generate risk
estimates that fall within the higher end of the population risk range (USEPA 2004b); applying
models to estimate missing values; using surrogate data (e.g., data on a parameter that come from a
different region of the country than the region being assessed); or applying professional judgment.
The use of individual assumptions can range from qualitative (e.g., assuming that one is secured to
the residence location and does not move through time or space) to more quantitative (e.g., using
the 95th percentile of a sample distribution for an ingestion rate). This approach also can be
applied to the practice of hazard identification and dose-response assessment when data are
missing. Identifying the sensitivity of exposure or risk estimates to key inputs can help focus efforts
to reduce uncertainty by collecting additional data.
Current EPA practices to address uncertainty and variability are focused on the evaluation of data,
model, and scenario uncertainty and variability. In addition, decision makers are faced with
combining many different decision criteria that may be informed by science and PRA as well as by
expert judgment or the weighting of values to choose a decision alternative. Data, model, and
scenario uncertainties and variability (including their probability distributions), as well as expert
judgment, can be important considerations in the selection of one alternative over another
(Costanza etal. 1997; Morgan etal. 2009; Stahl and Cimorelli 2005; Wright eta/. 2002).
1.9. What Are the Limitations of Relying on Default-Based
Deterministic Approaches?
Default-based deterministic approaches are applied to data, model and scenario uncertainties.
Deterministic risk assessment (DRA) often is considered a traditional approach to risk analysis
because of the existence of established guidance and procedures regarding its use, the ease with
which it can be performed, and its limited data and resource needs. The use of defaults supporting
DRA provides a procedural consistency that allows for risk assessments to be feasible and tractable.
Decision makers and members of the public tend to be relatively familiar with DRA, and the use of
such an approach addresses assessment-related uncertainties primarily through the incorporation
of predetermined default values and conservative assumptions. It addresses variability by
combining input parameters intended to be representative of typical or higher end exposure (i.e.,
considered to be conservative assumptions). The intention often is to implicitly provide a margin of
safety (i.e., more likely to overestimate risk than underestimate risk) or construct a screening-level
estimate of high-end exposure and risk (i.e., an estimate representative of more highly exposed and
susceptible individuals).
DRA provides an estimation of the exposures and resulting risks that addresses uncertainties and
variabilities in a qualitative manner. The methods typically used in EPA DRA rely on a combination
of point values—some conservative and some typical—yielding a point estimate of exposure that is
at some unknown point in the range of possible risks. Although this conservative bias aligns with
the public health mission of EPA (USEPA 2004b), the degree of conservatism in these risk estimates
(and in any concomitant decision) cannot be estimated well or communicated (Hattis and
Burmaster 1994). Typically, this results in unquantified uncertainty in risk statements.
11
-------
Quantitative information regarding the precision or potential systematic error and the distribution
of exposures, effects and resulting risks across different members of an exposed population are
usually not provided with estimates generated using default approaches. Although DRA may
present qualitative information regarding the robustness of the estimates, the impact of data and
model limitations on the quality of the results cannot be quantified. Reliance on deterministically
derived estimations of risk can result in decision making based solely on point estimates with an
unknown degree of conservatism, which can complicate the comparison of risks or management
options.
In risk assessments of noncancer endpoints, metrics such as an oral reference dose (RfD) and an
inhalation reference concentration (RfC) are typically used. The use of conservative defaults long
has been the target of criticism (Finkel 1989) and has led to the presumption by critics that EPA
assessments are overly conservative and unrealistic. The use of PRA would be advantageous in
eliminating a single value and might be less likely to imply undue precision and lessen the need for
conservative assumptions, thereby reducing bias in the estimate. In the probabilistic framework, a
probability distribution would be used to express the belief that any particular value represents the
dose or exposure concentration that would pose no appreciable risk of adverse effects (NRC 2009).
EPA is investigating the use of PRA to derive risk values for RfD and RfC in EPA's Integrated Risk
Information System (IRIS) Database [www.epa.gov/IRIS/).
EPA commissioned a white paper (Hattis and Lynch 2010) presented at the Hazardous Air Pollutant
Workshop, 2009, illustrating the implementation of probabilistic methods in defining RfDs and
assessing the benefits for reducing exposure to toxicants that act in part through traditional
individual threshold processes. The use of PRA, among other things, makes provision for
interactions with background pathological processes, as recommended by the NRC (2009), and
shows how the system can inform assessments for "data-poor" toxicants.
PRA may be more suitable than DRA for complex assessments, including those of aggregate and
cumulative exposures and time-dependent individual exposure, dose and effects analyses.
Identification and prioritization of contributory sources of uncertainty can be difficult and time
consuming when using deterministic methods, leading to difficulties in model evaluation and the
subsequent appraisal of risk estimates (Cullen and Frey 1999). Quantitative analyses of model
sensitivities are essential for the prioritization of key uncertainties—a critical process in identifying
steps for data collection or research to improve exposure or risk estimates.
1.10. What Is EPA's Experience with the Use of Probabilistic Risk
Analysis?
EPA's experience with PRA has, to date, primarily been limited to the evaluation of data, model and
scenario uncertainties. To assist with the growing number of probabilistic analyses of exposure
data in these uncertainty areas, EPA issued Guiding Principles for Monte Carlo Analysis (USEPA
1997b). Given adequate supporting data and credible assumptions, probabilistic analysis
techniques, such as Monte Carlo analysis, can be viable statistical tools for analyzing uncertainty
and variability in risk assessments. EPA's policy for the use of probabilistic analysis in risk
assessment, released in 1997, is inclusive of human exposure and ecological risk assessments and
does not rule out probabilistic health effects analyses (USEPA 1997a). Subsequently, EPA's SAB and
Scientific Advisory Panel (SAP) have reviewed PRA approaches to risks used by EPA offices such as
OAR, OPP and others. Several programs have developed specific guidance on the use of PRA,
including OPP and OSWER (USEPA 1998a, 2001).
To illustrate the practical application of PRA to problems relevant to the Agency, several example
case studies are briefly described here. The Appendix titled Case Study Examples of Application of
12
-------
Probabilistic Risk Analysis in U.S. Environmental Protection Agency Regulatory Decision Making,
discusses these and other case studies in greater detail, including the procedures and outcomes.
The Appendix includes 16 case studies—11 HHRA and 5 ERA examples—that are intended to
illustrate how some of EPA's programs and offices currently utilize PRA. To aid in describing how
probabilistic analyses were used, the 16 case studies are subdivided among 3 categories of PRA
tools: Group 1—point estimate, including sensitivity analysis; Group 2—probabilistic risk analysis,
including one-dimensional Monte Carlo analysis (1-D MCA) and probabilistic sensitivity analysis;
and Group 3—advanced probabilistic risk analysis, including two-dimensional Monte Carlo analysis
(2-D MCA) with microexposure (microenvironments) modeling, Bayesian statistics, geostatistics
and expert elicitation.
It is useful to note that the NRC (2009) recommended a tiered approach to risk assessment using
both qualitative and quantitative (deterministic and probabilistic) tools, with the complexity of the
analysis increasing as progress is made through the tiers. The use of PRA tools to address issues of
uncertainty and variability in a tiered approach is described more completely in Section 2.10 and
was illustrated in Figure 2. The three tiers illustrated in that figure approximately correspond to the
three groups of EPA case studies described in the Appendix that provide examples of the use of
various PRA tools.
Table A-1 in the Appendix offers a summary of the 16 case studies based on the type of risk
assessment, the PRA tools used in the assessment, and the EPA program or regional office
responsible for the assessment Some of the approaches that are profiled in these case studies can
be used in the planning and scoping phases of risk assessments and risk management. Other, more
complex PRA approaches are used to answer more specific questions and provide a richer
description of the risks. Most studies show that PRA can improve or expand on information
generated by deterministic methods. In some of the case studies, the use of multiple PRA tools is
illustrated. For example, Case Study 1 describes the use of a point estimate sensitivity analysis to
identify exposure variables critical to the analysis summarized in Case Study 9. Both of these case
studies focus on children's exposure to chromated copper arsenate (CCA)-treated wood. In Case
Study 9. an MCA was used as an example of a two-dimensional (i.e., addressing both variability and
uncertainty) probabilistic exposure assessment.
Overall, the case studies illustrate that the Agency already has applied the science of PRA to
ecological risk and human exposure estimation and has begun using PRA to describe health effects.
Some of the applications have used existing "off-the-shelf" software, whereas others have required
significant effort and resources. Once developed, however, some of the more complex models have
been used many times for different assessments. All of the assessments have been validated by
internal and external peer review. Table 1 gives some highlights the case studies from deterministic
to more complex assessments, which are described in more detail in the Appendix.
13
-------
Table 1. Selected Examples of EPA Applications of Probabilistic Risk Assessment Techniques
Case
Study No.
2
5
7
9
13
14
Description
Atmospheric Deposition to Watershed
Contamination: The Office of Research and
Development (ORD) developed an analysis of
nitrogen, mercury and polycyclic aromatic
hydrocarbons (PAHs) depositions toward
watershed contamination in the Casco Bay
Estuary in southwestern Maine.
Hudson River Polychlorinated Biphenyl
(PCB)-Contaminated Sediment Site: Region
2 evaluated the variability in risks to anglers
who consume recreationally caught fish
contaminated with PCBs from sediment
contamination in the Hudson River.
Environmental Monitoring and Assessment
Program (EMAP): ORD developed and the
Office of Water (OW) applied probabilistic
sampling techniques to evaluate the Nation's
aquatic resources under the Clean Water Act
(CWA) Section 305(b).
Chromated Copper Arsenate (CCA) Risk
Assessment: ORD and the Office of Pesticide
Programs (OPP) conducted a probabilistic
assessment of children's exposure
(addressing both variability and uncertainty) to
arsenic and chromium from contact with CCA-
treated wood play sets and decks.
Evaluating Ecological Effects of Pesticide
Uses: OPP developed a probabilistic model,
which evaluates acute mortality levels in
generic and specific ecological species for
user-defined pesticide uses and exposures.
Fine Particulate Matter Health Impacts:
ORD and the Office of Air and Radiation
(OAR) used expert elicitation to more
completely characterize, both qualitatively and
quantitatively, the uncertainties associated
with the relationship between reduction in fine
particulate matter (PlVks) and benefits of
reduced PM2.s-related mortality.
Group
Group 1: Point
Group 2:
1-D Monte Carlo
Analysis
Group 2:
Probabilistic
Sensitivity
Analysis
Group 3:
2-D Monte Carlo
Analysis
Group 3:
Probabilistic
Analysis
Group 3: Expert
Elicitation
Type of Risk
Assessment
Ecological
Human Health
Ecological
Human Health
Ecological
Human Health
Office/Region
ORD
Superfund/
Region 2
(New York)
ORD/OW
ORD/OPP
OPP
ORD/OAR
14
-------
2. PROBABILISTIC RISK ANALYSIS
2.1. What Are Uncertainty and Variability, and How Are They Relevant
to Decision Making?
The concepts of uncertainty and variability are introduced here, and the relevance of these
concepts to decision making is discussed.
2.1.1. Variability
Variability refers to real differences over time, space or members of a population and is a property
of the system being studied (e.g., drinking water consumption rates for each of the many individual
adult residents living in a specific location or differences in body lengths or weights for humans or
ecological species) (Cullen and Frey 1999; USEPA 201 Ic). Variability can arise from inherently
random processes, such as variations in wind speed over time at a given location or from true
variation across members of a population that, in principle, could be explained, but which, in
practice, may not be explainable using currently available models or data (e.g., the range of lead
levels in the blood of children 6 years old or younger following a specific degree of lead exposure).
Of particular interest in both HHRA and ERA is inter-individual variability, which typically refers to
differences between members of the same population in either behavior related to exposure (e.g.,
dietary consumption rates for specific food items), or biokinetics related to chemical uptake (e.g.,
gastrointestinal uptake rates for lead following intake) or toxic response (e.g., differences among
individuals or species in the internal dose needed to produce a specific amount of neurological
impairment).
Inter-individual variability is illustrated in Case Study 5 in the Appendix, which assesses a PCB-
contaminated sediment site in the Hudson River. In this case study, the quantification of variability
is illustrated through the use of a PRA tool—1-D MCA—to describe the variability of exposure as a
function of individual exposure factors (i.e., young children's fish ingestion).
2.1.2. Uncertainty
Uncertainty is the lack of knowledge of the true value of a quantity or relationships among
quantities (USEPA 2011c). For example, there may be a lack of information regarding the true
distribution of variability between individuals for consumption of certain food items. There are a
number of types of uncertainties for both risk analysis. The following descriptions of the types of
uncertainty (adapted from Cullen and Frey 1999) addresses uncertainties that arise during risk
analyses. These uncertainties can be separated broadly into three categories: (1) scenario
uncertainty; (2) model uncertainty; and (3) input or parameter uncertainty. Each of these is
explained in the paragraphs that follow.
Scenario uncertainty refers to errors, typically of omission, resulting from incorrect or incomplete
specification of the risk scenario to be evaluated. The risk scenario refers to a set of assumptions
regarding the situation to be evaluated, such as: (1) the specific sources of chemical emissions or
exposure to be evaluated (e.g., one industrial facility or a cluster of varied facilities impacting the
same study area); (2) the specific receptor populations and associated exposure pathways to be
modeled (e.g., indoor inhalation exposure, track-in dust or consumption of home-produced dietary
items); and (3) activities by different lifestages to be considered (e.g., exposure only at home, or
consideration of workplace or commuting exposure). Mis-specification of the risk scenario can
result in underestimation, overestimation or other mischaracterization of risks. Underestimation
may occur because of the exclusion of relevant situations or the inclusion of irrelevant situations
with respect to a particular analysis. Overestimation may occur because of the inclusion of
15
-------
unrealistic or irrelevant situations (e.g., assuming continuous exposure to an intermittent airborne
contaminant source rather than accounting for mobility throughout the day).
Model uncertainty refers to limitations in the mathematical models or techniques that are
developed to represent the system of interest and often stems from: (1) simplifying assumptions;
(2) exclusion of relevant processes; (3) mis-specification of model boundary conditions (e.g., the
range of input parameters); or (4) misapplication of a model developed for other purposes. Model
uncertainty typically arises when the risk model relies on missing or improperly formulated
processes, structures or equations. Sources of model uncertainty are defined in the Glossary.
Input or parameter uncertainty typically refers to errors in characterizing the empirical values used
as inputs to the model (e.g., engineering, physical, chemical, biological or behavioral variables).
Input uncertainty can originate from random or systematic errors involved in measuring a specific
phenomenon (e.g., biomarker measurements, such as the concentration of mercury in human hair);
statistical sampling errors associated with small sample sizes (e.g., if the data are based on samples
selected with a random, representative sampling design); the use of surrogate data instead of
directly measured data; the absence of an empirical basis for characterizing an input (e.g., the
absence of measurements for fugitive emissions from an industrial facility); or the use of summary
measures of central tendency rather than individual observations. Nonlinear random processes can
exhibit a behavior that, for small changes in input values, produces a large variation in results.
Input or parameter uncertainty is illustrated in Case Study 3 in the Appendix titled "Probabilistic
Assessment of Angling Duration Used in the Assessment of Exposure to Hudson River Sediments via
Consumption of Contaminated Fish." In this case study, a probabilistic analysis of one parameter in
an exposure assessment—the time an individual spends fishing in a large river system—was
assessed using sensitivity analysis. This analysis was conducted because there was uncertainty that
the individual exposure duration based on residence duration may underestimate the time spent
fishing (i.e., angling duration). The full distribution of the calculated values was used in conducting
the 1-D MCA for the fish consumption pathway, which is presented in Case Study 5.
Decision uncertainty refers to a decision analysis that would include not only the impact of scenario,
model and input uncertainties on the relative attractiveness of potential decision alternatives, but
also would include the degree to which specific choices (such as selecting input data, models, and
scenarios, and even how the problem or decision analysis is framed) impact the relative
attractiveness of potential decision alternatives. In decision making, analysts use data to represent
decision criteria that decision makers and other stakeholders believe will help them to answer their
decision question(s). These questions might include which policy alternative best meets Agency
goals (that must be articulated) or which risk assessment scenario best describes the observed
effects. Data, model and scenario uncertainties will influence the risk assessment results and those,
in turn, will influence the risk management options. Decision makers who understand the
uncertainty associated with their specific choices can be more confident that the decision will
produce the results that they seek. In addition, these decision makers will be able to defend their
decisions better and explain how the decision meets Agency and stakeholder goals.
While this is beyond the scope of this document, Stahl and Cimorelli (2005 and 2012) illustrate how
uncertainty throughout the decision making process can be assessed. These case studies explored
the assessment of ozone monitoring networks and air quality management policies that seek to
minimize the adverse impacts from ozone, fine particulate matter and air toxics simultaneously.
These case studies demonstrate the importance and feasibility of better understanding the
uncertainty introduced by specific choices (e.g., selecting input data, models, and scenarios) when
making public policy decisions.
16
-------
2.2. When Is Probabilistic Risk Analysis Applicable or Useful?
PRA may be particularly useful, for example, in the following (Cooke 1991; Cullen and Frey 1999;
NRC 2009; USEPA 2001):
n When a screening-level DRA indicates that risks are possibly higher than a level of concern
and a more refined assessment is needed.
n When the consequences of using point estimates of risk are unacceptably high.
n When significant equity or environmental justice issues are raised by inter-individual
variability.
n To estimate the value of collecting additional information to reduce uncertainty.
n To identify promising critical control points and levels when evaluating management
options.
n To rank exposure pathways, sites, contaminants and so on for the purposes of prioritizing
model development or further research.
n When combining expert judgments on the significance of the data.
n When exploring the impact of the probability distributions of stakeholder and decision-
maker values on the attractiveness of potential decision alternatives (Fischhoff 1995; Illing
1999; Kunreuther and Slovic 1996; USEPA 2000b).
n When exploring the impact of the probability distributions of the data, model and scenario
uncertainties, and variability together to compare potential decision alternatives.
PRA may add minimal value to the assessment in the following types of situations (Cullen and Frey
1999; USEPA 1997a):
n When a screening-level deterministic risk assessment indicates that risks are negligible,
presuming that the assessment is known to be conservative enough to produce
overestimates of risk.
n When the cost of averting the exposure and risk is smaller than the cost of a probabilistic
analysis.
n When there is little uncertainty or variability in the analysis (this is a rare situation).
2.3. How Can Probabilistic Risk Analysis Be Incorporated Into
Assessments?
As illustrated in the accompanying case studies in the Appendix, probabilistic approaches can be
incorporated into any stage of a risk assessment, from problem formulation or planning and
scoping to the analysis of alternative decisions. In some situations, PRA can be used selectively for
certain components of an assessment. It is common in assessments that some model inputs are
known with high confidence (i.e., based on site-specific measurements), whereas values for other
inputs are less certain (i.e., based on surrogate data collected for a different purpose). For example,
an exposure modeler may determine that relevant air quality monitoring data exists, but there is a
lack of detailed information on human activity patterns in different microenvironments. Thus, an
assessment of the variability in exposure to airborne pollutants might be based on direct use of the
monitoring data, whereas assessment of uncertainty and variability in the inhalation exposure
component might be based on statistical analysis of surrogate data or use of expert judgment The
uncertainties are likely to be larger for the latter than the former component of the assessment;
17
-------
efforts to characterize uncertainties associated with pollutant exposures would focus on the latter.
PRA also deals with dependency issues; a description of these issues is available in Section 3.3.2.
2.4. What Are the Scientific Community's Views on Probabilistic Risk
Analysis, and What Is the Institutional Support for Its Use in
Performing Assessments?
The NRC and IOM recently emphasized their long-standing advocacy for PRA (NRC 2007a and b;
IOM 2013). Dating from its 1983 Risk Assessment in the Federal Government: Managing the Process
(NRC 1983)—which first formalized the risk assessment paradigm—through reports released from
the late 1980s through the early 2000s, various NRC panels have maintained consistently that
because risk analysis involves substantial uncertainties, these uncertainties should be evaluated
within a risk assessment These panels noted that:
1. When evaluating the total population risk, EPA should consider the distribution of exposure
and sensitivity of response in the population (NRC 1989).
2. When assessing human exposure to air pollutants, EPA should present model results along
with estimated uncertainties (NRC 1991).
3. When conducting ERA, EPA should discuss thoroughly uncertainty and variability within
the assessment (NRC 1993).
4. "Uncertainty analysis is the only way to combat the 'false sense of certainty/ which is caused
by a refusal to acknowledge and [attempt to] quantify the uncertainty in risk predictions,"
as stated in the NRC report, Science and Judgment in Risk Assessment (NRC 1994).
5. EPA's estimation of health benefits was not wholly credible because EPA failed to deal
formally with uncertainties in its analyses (NRC 2002).
6. EPA should adopt a "tiered" approach for selecting the level of detail used in uncertainty
and variability assessment Furthermore, the NRC recommended that a discussion of the
level of detail used for uncertainty analysis and variability assessment should be an explicit
part of the planning, scoping and problem formulation phase of the risk assessment process
(NRC 2009).
7. EPA should develop methods to systematically describe and account for uncertainties in
decision-relevant factors in addition to estimates of health risk in its decision-making
process (IOM 2013).
Asked to recommend improvements to the Agency's HHRA practices, EPA's SAB echoed the NRC's
sentiments and urged the Agency to characterize uncertainty and variability more fully and
systematically and to replace single-point uncertainty factors with a set of distributions using
probabilistic methods (Parkin and Morgan 2007). The key principles of risk assessment cited by the
Office of Science and Technology Policy (OSTP) and the Office of Management and Budget (OMB)
include "explicit" characterization of the uncertainties in risk judgments; they proceed to cite the
National Academy of Science's (NAS) 2007 recommendation to address the "variability of effects
across potentially affected populations" (OSTP/OMB 2007).
18
-------
2.5. Additional Advantages of Using Probabilistic Risk Analysis and
How It Can Provide More Comprehensive, Rigorous Scientific
Information in Support of Regulatory Decisions.
External stakeholders previously have used the Administrative Procedure Act and the Data Quality
Act to challenge the Agency for a lack of transparency and consistency or for not fully analyzing and
characterizing the uncertainties in risk assessments or decisions (Fisher et ctl. 2006). The more
complete implementation of PRA and related approaches to deal with uncertainties in decision
making would address stakeholder concerns in regard to characterizing uncertainties.
The results of any assessment, including PRA, are dependent on the underlying methods and
assumptions. Accompanied by the appropriate documentation, PRA may communicate a more
robust representation of risks and corresponding uncertainties. This characterization may be in the
form of a range of possible estimates as opposed to the more traditionally presented single-point
values. Depending on the use of the assessment, ranges can be derived for variability and
uncertainty (or a combination of the two) in both model inputs and resulting estimations of risk.
PRA quantifies how exposures, effects and risks differ among human populations or lifestages or
target ecological organisms. PRA also provides an estimation of the degree of confidence with
which these estimates may be made, given the current uncertainty in scientific knowledge and
available data. A 2007 NRC panel stated that the objective of PRAs is not to decide "how much
evidence is sufficient" to adopt an alternative but, rather, to describe the scientific bases of
proposed alternatives so that scientific and policy considerations may be more fully evaluated (NRC
2007a). EPA's SAB similarly noted that PRAs provide more "value of information" through a
quantitative assessment of uncertainty and clarify the science underlying Agency decisions (USEPA
2007b).
The SAB articulated a number of advantages for EPA decision makers from the utilization of
probabilistic methods (Parkin and Morgan 2007):
n A probabilistic reference dose could help reduce the potentially inaccurate implication of
zero risk below the RfD.
n By understanding and explicitly accounting for uncertainties underlying a decision, EPA can
estimate formally the value of gathering more information. By doing so, the Agency can
better prioritize its information needs by investing in areas that yield the greatest
information value.
n Strategic use of PRA would allow EPA to send the appropriate signal to the intellectual
marketplace, thereby encouraging analysts to gather data and develop methodologies
necessary for assessing uncertainties.
2.6. What Are the Challenges to Implementation of Probabilistic
Analyses?
Currently, EPA is using PRA in a variety of programs to support decisions, but challenges remain
regarding the expanded use of these tools within the Agency. The challenges include:
n A lack of understanding of the value of PRA for decision making. PRA helps to improve the
rigor of the decision-making process by allowing decision makers to explore the impacts of
uncertainty and variability on the decision choices.
n A clear institutional understanding of how to incorporate the results of probabilistic
analyses into decision making is lacking.
19
-------
n PRA typically requires a different skill set than used in current evaluations, and limited
resources (staff, time, training or methods) to conduct PRA are available.
n Communicating probabilistic analysis results and the impact of those results on the
decision/policy options can be complex.
n Communication with stakeholders is often difficult and results in the appearance of
regulatory delays due the necessity of analyzing numerous scenarios using various models.
n PRA complicates decision making and risk communication in instances where a more
comprehensive characterization of the uncertainties leads to a decrease in clarity regarding
how to estimate risk for the scenario under consideration. These challenges are discussed in
more detail in Sections 2.7 through 2.13.
2.7. How Can Probabilistic Risk Analysis Support Specific Regulatory
Decision Making?
Decision makers sometimes perceive that the binary nature of regulatory decisions (e.g., Does an
exposure exceed a reference dose or not? Do emissions comply with Agency standards or not?)
precludes the use of a risk range developed through PRA. Generally, it is necessary to explain the
rationale underlying a particular decision. PRA's primary purpose is to provide information to
enhance the ability to make transparent decisions based on the best available science. By
conducting a sensitivity analysis of the influence of the uncertainty on the decision-making process,
it can be determined how or if PRA can help to improve the process.
PRA can provide information to decision makers on specific questions related to uncertainty and
variability. For questions of uncertainty and to minimize the likelihood of unintended
consequences, PRA can help to provide the following types of information:
n Characterization of the uncertainty in estimates (i.e., What is the degree of confidence in the
estimate?). Could the prediction be off by a factor of 2, a factor of 10 or a factor of 1,000?
n Critical parameters and assumptions that most affect or influence a decision and the risk
assessment.
O "Tipping points" where the decision would be altered if the risk estimates were different, or
if a different assumption was valid.
n Estimate the likelihood that values for critical parameters will occur or test the validity of
assumptions.
n Estimate the degree of confidence in a particular decision and/or the likelihood of specific
decision errors.
n The possibility of alternative outcomes with additional information, or estimate tradeoffs
related to different risks or risk-management decisions.
n The impact of additional information on decision making, considering the cost and time to
obtain the information and the resulting change in decision (i.e., the value of the
information).
For the consideration of variability, PRA can help to provide the following types of information for
exposures:
n Explicitly defined exposures for various populations or lifestages (i.e., Who are we trying to
protect?). That is, will the regulatory action keep 50 percent, 90 percent, 99.9 percent or
some other fraction of the population below a specified exposure, dose or risk target?
20
-------
n Variability in the exposures, among various populations or lifestages, and information on
the percentile of the population that is being evaluated in the risk assessment (e.g.,
variations in the number of liters of water per kilogram [kg] body weight per day consumed
by the population). This information is helpful in addressing comments:
• On the conservatism of EPA's risk assessments;
• Concerns about whether their particular exposures were evaluated in the risk
assessment;
• Whom or what is being protected by implementing a decision; and
• Whether and what additional research may be needed to reduce uncertainty.
PRA helps to inform decisions by characterizing the alternatives available to the decision maker
and the uncertainty he or she faces, and by providing evaluation measures of outcomes.
Uncertainties often are represented as probabilities or probability distributions numerically or in
graphs. As part of a decision analysis, stakeholders can more fully examine how uncertainties
influence the preference among alternatives.
2.8. Does Probabilistic Risk Analysis Require More Resources Than
Default-Based Deterministic Approaches?
PRA generally can be expected to require more resources than standard Agency default-based
deterministic approaches. There is extensive experience within EPA in conducting and reviewing
DRA. These assessments tend to follow standardized methods that minimize the effort required to
conduct them and to communicate the results. Probabilistic assessments often entail a more
detailed analysis, and as a result, these assessments require substantially more resources, including
time and effort, than do deterministic approaches.
Appropriately trained staff and the availability of adequate tools, methods and guidance are
essential for the application of PRA. Proper application of probabilistic methods requires not only
software and data, but also guidance and training for analysts using the tools and for managers and
decision makers tasked with interpreting and communicating the results.
An upfront increase in resources needed to conduct a probabilistic assessment can be expected, but
development of standardized approaches and/or methods can lead to the routine incorporation of
PRA in Agency approaches (e.g., OPP's use of the Dietary Exposure Evaluation Model [DEEM;
http://www.epa.gov/pesticides/science/deem/]. a probabilistic dietary exposure model). The
initial and, in some cases, ongoing resource cost (e.g., for development of site-specific models for
site assessments) may be offset by a more informed decision than a comparable deterministic
analysis. Probabilistic methods are useful for identifying effective management options and
prioritizing additional data collection or research aimed at improving risk estimation, ultimately
resulting in decisions that enable improved environmental protection while simultaneously
conserving more resources.
2.9. Does Probabilistic Risk Analysis Require More Data Than
Conventional Approaches?
There are differences of opinion within the technical community as to whether PRA requires more
data than other types of analyses. Although some emphatically believe that PRA requires more data,
others argue that probabilistic assessments make better use of all of the available data and
information. Stahl and Cimorelli (2005) discuss when and how much data are necessary for a
decision. PRA can benefit from more data than might be used in a DRA. For example, where DRA
21
-------
might employ selected point estimates (e.g., the mean or 95th percentile values) from available data
sets for use in model inputs, PRA facilitates the use of frequency-weighted data distributions,
allowing for a more comprehensive consideration of the available data. In many cases, the data that
were used to develop the presumptive 95th percentile can be employed in the development of
probabilistic distributions.
Restriction of PRA to principally data-rich situations may prevent its broader application where it is
most useful. Because PRA incorporates information on data quality, variability and uncertainty into
risk models, the influence of these factors on the characterization of risk can become a greater focus
of discussion and debate.
A key benefit of using PRA is its ability to reveal the limitations as well as the strengths of data that
often are masked by a deterministic approach. In doing so, PRA can help to inform research
agendas, as well as support regulatory decision making, based on the state of the best available
science. In summary, PRA typically requires more time for developing input assumptions than a
DRA, but when incorporated into the relevant steps of the risk assessment process, PRA can
demonstrate added benefits. In some cases, PRA can provide additional interpretations that
compensate for the extra effort required to conduct a PRA.
2.10. Can Probabilistic Risk Analysis Be Used to Screen Risks or Only
in Complex or Refined Assessments?
Probabilistic methods typically are not necessary where traditional default-based deterministic
methods are adequate for screening risks. Such methods are relatively low cost, intended to
produce conservatively biased estimates, and useful for identifying situations in which risks are so
low that no further action is needed. The application of probabilistic methods can be targeted to
situations in which a screening approach indicates that a risk may be of concern or when the cost of
managing the risk is high, creating a need for information to help inform decision making. PRA fits
directly into a graduated hierarchical approach to risk analysis. This tiered approach, depicted in
Figure 2. is a process for a systematic informed progression to increasingly more complex risk
assessment methods, depending on the decision-making context and need. Higher tiers reflect
increasing complexity and often will require more time and resources. An analysis might typically
start at a lower tier and only progress to a higher tier if there is a need for a more sophisticated
assessment commensurate with the importance of the problem. Higher tiers also reflect increasing
characterization of variability and/or uncertainty in the risk estimate, which may be important for
risk-management decisions. The case studies described in the Appendix are presented in three
groups that generally correspond to the tiers identified in Figure 2. Group 1 case studies are point
estimate (sensitivity analysis) examples (Tier 1); Group 2 case studies include most moderate-
complexity PRA examples (Tier 2); and Group 3 case studies are advanced (high complexity) PRA
examples (Tier 3).
The tiered approach in Figure 2 depicts a continuum from screening level point estimate that is
done with little data and conservative assumptions to PRA that requires an extensive data set and
more realistic (less conservative) assumptions. In between, there can be a wide variety of tiers of
increasing complexity, or there may be only a few reasonable choices between screening methods
and highly refined analyses (USEPA 2004a). A similar four-tiered approach for characterizing the
variability and/or uncertainty in the estimated exposure or risk analysis (WHO 2008) has been
adapted by EPA in the risk and exposure assessments conducted for the National Ambient Air
Quality Standards (NAAQS).
PRA also could be used to examine more fully the existing default-based methods based on the
current state of information and knowledge to determine if such methods are truly conservative
22
-------
and adequate for screening (e.g., in dose-response analyses dealing with hazard characterization)
(Swartouteta/. 1998; Hattiseta/. 2002).
The use of a spectrum of data should be employed both in determining screening risks and in more
complex assessments. For HHRA, data from human, animal, mechanistic and other studies should
be used to develop a probabilistic characterization of cancer and noncancer risks and to identify
uncertainties. The NRC recommended that EPA facilitate this approach by redefining RfD and RfC
within the probabilistic framework to take into accountthe probability of harm (NRC 2009). It is
likely that both DRA and PRA will be part of this framework.
2.11. Does Probabilistic Risk Analysis Present Unique Challenges to
Model Evaluation?
The concept of "validation" of models used for regulatory decision making has been a topic of
intense discussion. In a recent report on the use of models in environmental regulatory decision
making, the NRC recommended using the notion of model "evaluation" rather than "validation,"
suggesting that use of a process that encompasses the entire life cycle of the model and
incorporates the spectrum of interested parties in the application of the model often extends
beyond the model builder and decision maker. Such a process can be designed to ensure that
judgment of the model application is based not only on its predictive value determined from
comparison with historical data, but also on its comprehensiveness, rigor in development,
transparency and interpretability (NRC 2007b).
Model evaluation is important in all risk assessments. In the case of PRA, there is an additional
question as to the validity of the assumptions regarding probability and frequency distributions for
model inputs and their dependencies. Probabilistic information can be accounted for during
evaluation analyses by considering the range of uncertainty in the model prediction and whether
such a range overlaps with the "true" value based on independent data. Thus, probabilistic
information can aid in characterizing the precision of the model predictions and whether a
prediction is significantly different from a benchmark of interest. For example, comparisons of
probabilistic model results and monitoring data were performed for multiple models in developing
the cumulative pesticide exposure model. Concurrent PRA model evaluations using a Bayesian
analysis also have been published (Clyde 2000).
When risk assessors develop models of risk, they rely on two predominant statistical methods. Both
methods arise from axioms of probability, but each applies these axioms differently. Under the
frequentist approach, one develops and evaluates a model by testing whether the model—as
applied to the observations—conforms to idealized distributions. Under the Bayesian approach,
one develops and evaluates a model by testing which—among alternative models—best yields the
underlying distribution describing the data. The practical differences between these two
approaches can perhaps best be appreciated when considering the structural uncertainty in models
(Section 3.3.3). Because Bayesians estimate model parameters with the expectation that these
parameters—or even model structures—will be updated as new data become available, they have
developed formal techniques to provide uncertainty bounds around these parameter estimates,
select models that best explain the given data, or combine the results of alternative models.
2.12. How Do You Communicate the Results of Probabilistic Risk
Analysis?
Effective communication makes it easier for regulators and stakeholders to understand the decision
criteria driving the decision-making process. In other words, communication of PRA results within
the decision-making context facilitates understanding. The specific approaches for reporting results
23
-------
from PRA vary depending on the assessment objective and the intended audience. Beyond the basic
1997 principles and the policy from the same year (USEPA 1997a and b), the Risk Assessment
Guidance for Superfund: Volume HI—Part A, Process for Conducting Probabilistic Risk Assessment also
provides some guidance on the quality and criteria for acceptance as well as communication basics
(USEPA 2001). There have been limited studies of how information from PRA regarding uncertainty
and variability can or should be communicated to key audiences, such as decision makers and
stakeholders (e.g., Morgan and Henrion 1990; Bloom et al. 1993; Krupnick et al. 2006). Among the
analyst community, there often is an interest in visualization of the structure of a scenario and
model using influence diagrams and depiction of the uncertainty and variability in model inputs
and outputs using probability distributions in the form of cumulative density functions or
probability distribution functions (Figure 3). Sensitivity of the model output to uncertainty and
variability in model inputs can be depicted using graphical tools.
In some cases, these graphical methods can be useful for those less familiar with PRA, but in many
cases there is a need to translate the quantitative results into a message that extracts the key
insights without burdening the decision maker with obscure technical details. In this regard, the
use of ranges of values for a particular metric of decision-making relevance (e.g., the range of
uncertainty associated with a particular estimate of risk) may be adequate. The presentation of PRA
results to a decision maker may be conducted best as an interactive discussion, in which a principal
message is conveyed, followed by exploration of issues such as the source, quality and degree of
confidence associated with the information. There is a need for the development of
recommendations and a communication plan regarding how to communicate the results of PRA to
decision makers and stakeholders, building on the experience of various programs and regions.
2.13. Are the Results of Probabilistic Risk Analysis Difficult to
Communicate to Decision Makers and Stakeholders?
Research has shown that the ability of decision makers to deal with concepts of probability and
uncertainty varies. Bloom eta/. (1993) surveyed a group of senior managers at EPA and found that
many could interpret information about uncertainty if it was communicated in a manner
responsive to decision-maker interests, capabilities and needs. In a more recent survey of ex-EPA
officials, Krupnick et al. (2006) concluded that most had difficulty understanding information on
uncertainty with conventional scientific presentation approaches. The findings of these studies
highlight the need for practical strategies for the communication of results of PRA and uncertainty
information between risk analysts and decision makers, as well as between decision makers and
other stakeholders. The Office of Emergency and Remedial Response (OERR) has compiled
guidance to assist analysts and managers in understanding and communicating the results of PRA
(USEPA 2001).
Risk analysts need to focus on how to use uncertainty analysis to characterize how confident
decision makers should be in their choices. As Wilson (2000) explained, "... uncertainty is the bane
of any decision maker's existence. Thus, anyone who wants to inform decisions using scientific
information needs to assure that their analyses transform uncertainty into confidence in
conclusions." Hence, although environmental risk assessments are complicated and it is easy to get
lost in the details, presenting and discussing these results within the context of the decision
facilitates understanding. The translation of uncertainty into confidence statements forces a "top-
down" perspective that promotes accounting for whether and how uncertainties affect choices (Toll
eta/. 1997).
24
-------
Figure 3. Graphical Description of the Likelihood (Probability) of Risk. Hypothetical fitted data
distribution with upper and lower confidence intervals are depicted for the output of a 2-D MCA
model.
25
-------
3. AN OVERVIEW OF SOME OF THE TECHNIQUES USED IN
PROBABILISTIC RISK ANALYSIS
3.1. What Is the General Conceptual Approach in Probabilistic Risk
Analysis?
PRA includes several major steps, which parallel the accepted environmental health risk
assessment process. These include: (1) problem and/or decision criteria identification;
(2) gathering information; (3) interpreting the information; (4) selecting and applying models and
methods for quantifying variability and/or uncertainty; (5) quantifying inter-individual or
population uncertainty and variability in metrics relevant to decision making; (6) sensitivity
analysis to identify key sources of variability and uncertainty; and (7) interpreting and reporting
results.
Problem formulation entails identifying the assessment endpoints or issues that are relevant to the
decision-making process and stakeholders, and that can be addressed in a scientific assessment
process. Following problem formulation, information is needed from stakeholders and experts
regarding the scenarios to evaluate. Based on the scenarios and assessment endpoints, the analysts
select or develop models, which in turn leads to identification of model input data requirements
and acquisition of data or other information (e.g., expert judgment encoded as the result of a formal
elicitation process) that can be used to quantify inputs to the models. The data or other information
for model inputs is interpreted in the process of developing probability distributions to represent
variability, uncertainty or both for a particular input Thus, steps (1) through (4) listed above are
highly interactive and iterative in that the data input requirements and how information is to be
interpreted depend on the model formulation, which depends on the scenario and that in turn
depends on the assessment objective. The assessment objective may have to be refined depending
on the availability of information.
Once a scenario, model and inputs are specified, the model output is estimated. A common
approach is to use Monte Carlo Analysis (MCA) or other probabilistic methods to generate samples
from the probability distributions of each model input, run the model based on one random value
from each probabilistic input, and produce one corresponding estimate of the model outputs. This
process is repeated typically hundreds or thousands of times to create a synthetic statistical sample
of model outputs. These output data are interpreted as a probability distribution of the output of
interest. Sensitivity analysis can be performed to determine which model input distributions are
most highly associated with the range of variation in the model outputs. The results may be
reported in a wide variety of forms depending on the intended audience, ranging from qualitative
summaries to tables, graphs and diagrams.
Detailed introductions to PRA methodology are available elsewhere, such as Ang and Tang (1984),
Cullen and Frey (1999), EPA (2001), and Morgan and Henrion (1990). A few key aspects of PRA
methodology are briefly mentioned here. Readers who seek more detail should consult these
references and see the Bibliography for additional references.
3.2. What Levels and Types of Probabilistic Risk Analyses Are There
and How Are They Used?
There are multiple levels and types of analysis used to conduct risk assessments (illustrated in
Figure 2 and Table 1. respectively). Graduated approaches to analysis are widely recognized (e.g.,
USEPA 1997a, 2001; WHO 2008). The idea of a graduated approach is to choose a level of detail and
refinement for an analysis that is appropriate to the assessment objective, data quality, information
available and importance of the decision (e.g., resource implications).
26
-------
As discussed in section 1.8, there is a variety of approaches to risk assessment that differ in their
complexity and the manner in which they address uncertainty and variability. In DRA one does not
formally characterize uncertainty or variability but rather typically relies on using default-based
assumptions and factors to generate a single estimate of risk. In PRA there is a variety of
approaches to explicitly address or characterize uncertainty or variability in risk estimates and
these differ in terms of how they accomplish this, the data used, and the overall complexity. Some
examples are:
n Sensitivity analysis
n Monte Carlo analysis of variability in exposure data
n Human health or ecological effects data
n Monte Carlo analysis of uncertainty
n "Cumulative" PRA—multi-pathway or multi-chemical
n Two-dimensional PRA of uncertainty and variability
n Decision uncertainty analysis
n Geospatial analysis
n Expert elicitation
The DRA approaches described in Section 1.8 are examples of lower levels in a graduated approach
to analysis. Risk at the lower levels of analysis is assessed by conservative, bounding assumptions.
If the risk estimate is found to be very low despite the use of conservative assumptions, then there
exists a great deal of certainty that the actual risks to the population of interest for the given
scenario are below the level of concern and no further intervention is required, assuming that the
scenario and model specifications are correct. When a conservative DRA indicates that a risk may
be high, it is possible that the risk estimate is biased and the actual risk may be lower. In such a
situation, depending on the resource implications of the decision, it may be appropriate to proceed
with a more refined or higher level of analysis. The relative costs of intervention versus further
analysis should be considered when deciding whether to proceed with a decision based on a lower
level analysis or to escalate to a higher level of analysis. In some deterministic assessments (e.g.,
ecological risks), the assumptions are not well assured of conservatism, and the estimated risks
might be biased to appear lower than the unseen actual risk.
A more refined analysis could involve the application of DRA methods, but with alternative sets of
assumptions intended to characterize central tendency and reasonable upper bounds of exposure,
effects and risk estimates, such that the estimates could be for an actual individual in the population
of interest rather than a hypothetical maximally exposed individual. Such analyses are not likely to
provide quantification regarding the proportion of the population at or below a particular exposure
or risk level of concern, uncertainties for any given percentile of the exposed population, or
priorities among input assumptions with respect to their contributions to uncertainty and
variability in the estimates.
To more fully answer the questions often asked by decision makers, the analysis can be further
refined by incorporating quantitative comparisons of alternative modeling strategies (to represent
structural uncertainties associated with scenarios or models), quantifying ranges of uncertainty
and variability in model outputs, and providing the corresponding ranges for model outputs of
interest. When performing probabilistic analyses, choices are made regarding whether to focus on
the quantification of variability only, uncertainty only, both variability and uncertainty together
(representing a randomly selected individual), or variability and uncertainty independently (e.g., in
27
-------
a two-dimensional depiction of probability bands for estimates of inter-individual variability; see
Figure 4). The simultaneous but distinct propagation of uncertainty and variability in a two-
dimensional framework enables quantification of uncertainty in the risk for any percentile of the
population. For example, one could estimate the range of uncertainty in the risk faced by the
median member of the population or the 95th percentile member of the population. Such
information can be used by a decision maker to gauge the confidence that should be placed in any
particular estimate of risk, as well as to determine whether additional data collection or
information might be useful to reduce the uncertainty in the estimates. The OPP assessment of
Chromated Copper Arsenate-treated wood used such an approach. (See Case Study 9 in the
Appendix.]
100
0
^
CO
c
o
o
0
80
60
jjj 40
0
Q_
20
0
Option 3.
Separating
• variability &
uncertainty J. /— ^ /
(2DMC) \\. /
Option 2.
Combining
variability &
uncertainty
(1D MC)
Option 1.
Only variability
10
100
1000
Exposure (jug/day)
Figure 4. Diagrammatic Comparison of Three Alternative Probabilistic Approaches for the Same
Exposure Assessment. In Option 1 (one dimensional Monte Carlo analysis), only variability is quantified.
In Option 2 (one dimensional Monte Carlo analysis), both uncertainty and variability are combined. In
Option 3 (two dimensional Monte Carlo analysis), variability and uncertainty are analyzed separately.
Source: WHO 2008.
When conducting an analysis for the first time, it may not be known or clear, prior to analysis,
which components of the model or which model inputs contribute the most to the estimated risk or
its uncertainty and variability. As a result of completing an analysis, however, the analyst often
gains insight into the strengths and weaknesses of the models and input information. Probabilistic
analysis and sensitivity analysis can be used together to identify the key sources of quantified
uncertainty in the model outputs to inform decisions regarding priorities for additional data
collection. Ideally, time should be allowed for collecting such information and refining the analysis
to arrive at a more representative and robust estimate of uncertainty and variability in risk. Thus,
the notion of iteration in developing and improving an analysis is widely recommended.
28
-------
The notion of iteration can be applied broadly to the risk assessment framework. For example, a
first effort to perform an analysis may lead to insight that the assessment questions might be
impossible to address, or that there are additional assessment questions that may be equally or
more important. Thus, iteration can include reconsideration of the initial assessment questions and
the corresponding implications for definition of scenarios, selection of models and priorities for
obtaining data for model inputs. Alternatively, in a time-limited decision environment, probabilistic
and sensitivity analyses may offer insight into the effect of management options on risk estimates.
3.3. What Are Some Specific Aspects of and Issues Related to
Methodology for Probabilistic Risk Analysis?
This section briefly describes a few key aspects of PRA, model development and associated
uncertainties. Detailed introductions to PRA methodology are available elsewhere, such as Ang and
Tang (1984), Morgan and Henrion (1990), Cullen and Frey (1999) and EPA (2001). For more
detailed information, consult these references and see the Bibliography for additional sources.
3.3.1. Developing a Probabilistic Risk Analysis Model
There are two key issues that should be considered in developing a PRA model; as discussed below.
Structural Uncertainty in Scenarios
A potentially key source of uncertainty in an analysis is the scenario, which includes specification of
pollutant sources, transport pathways, exposure routes, timing and locations, geographic extent
and related issues. There is no formalized methodology for dealing quantitatively with uncertainty
and variability in scenarios. Decisions regarding what to include or exclude from a scenario could
be recast as hypotheses regarding which agents, pathways, microenvironments, etc., contribute
significantly to the overall exposure and risk of interest In practice, however, the use of qualitative
methods to frame an assessment tends to be more common, given the absence of a formal
quantitative methodology.
Coupled Models
For source-to-outcome risk assessments, it often is necessary to work with multiple models, each of
which represents a different component of a scenario. For example, there may be separate models
for emissions, air quality, exposure, dose and effects. Such models may have different spatial and
temporal scales. When conducting an integrated assessment, there may be significant challenges
and barriers to coupling such models into one coherent framework. Sometimes, the coupling is
done dynamically in a software environment In other cases, the output of one model might be
processed manually to prepare the information for input to the next model. Furthermore, there may
be feedback between components of the scenario (e.g., poor air quality might affect human activity,
which, in turn, could affect both emissions and exposures) that are incompletely captured or not
included. Thus, the coupling of multiple models can be a potentially significant source of structural
uncertainty (Ozkaynak 2009).
3.3.2. Dealing With Dependencies Among Probabilistic Inputs
When representing two or more inputs to a model as probability distributions, the question arises
as to whether it is reasonable to assume that the distributions are statistically independent If there
is a dependence, it could be as simple as a linear correlation between two inputs, or it could be
more complicated, such as nonlinear or nonmonotonic relationships. Dependencies typically are
not important if the risk estimate or other model output is sensitive to one or none of the
probabilistic inputs that might have interdependence. Furthermore, dependencies typically are not
of practical importance if they are weak. When dependencies exist and might significantly influence
29
-------
the risk estimate, they can be taken into account using a variety of statistical simulation methods or,
perhaps more appropriately, by modeling the dependence analytically where possible. Details on
methods for assessing the importance of possible dependencies and of quantifying them when
needed are described in Person et al. (2004 and 2009).
For some types of models, such as air quality models, it is not possible to introduce a probability
distribution to one input (e.g., ambient temperature at a particular location) without affecting
variables at other locations or times (e.g., temperatures in other locations at the same times or
temporal trends in temperature). In such cases, it is better to produce an "ensemble" of alternative
temperature fields, each of which is internally consistent. Individual members of an ensemble
usually are not interpreted as representing a probability sample; however, comparison of multiple
ensembles of meteorological conditions, for example, can provide insight into natural sources of
variability in ambient concentrations.
3.3.3. Conducting the Probabilistic Analysis
Quantifying Uncertainty and Variability in Model Inputs and Parameters
After the models are selected or developed to simulate a scenario of interest, attention typically
turns to the development of input data for the model. There is a substantial amount of literature
regarding the application of statistical methods for quantifying uncertainty and variability in model
inputs and parameters based on empirical data (e.g., Ang and Tang 1984; Cullen and Frey 1999;
Morgan and Henrion 1990; USEPA 2001). For example, a commonly used method for quantifying
variability in a model input is to obtain a sample of data, select a type of parametric probability
distribution model to fit to the data (e.g., normal, lognormal or other form), estimate the
parameters of the distribution based on the data, critique the goodness-of-fit using graphical (e.g.,
probability plot) and statistical (e.g., Anderson-Darling, Chi-Square or Kolmogorov-Smirnov tests)
methods and choose a preferred fitted distribution. This methodology can be adjusted to
accommodate various types of data, such as data that are samples from mixtures of distributions or
that contain non-detected (censored) values. Uncertainties can be estimated based on confidence
intervals for statistics of interest, such as mean values, or the parameters of frequency distributions
for variability. Various texts and guidance documents, both Agency and programmatic, describe
these approaches, including the Guiding Principles for Monte Carlo Analysis (USEPA 1997b).
The most common method for estimating a probability distribution in the output of a model, based
on probability distributions specified for model inputs, is MCS (Cullen and Frey 1999; Morgan and
Henrion 1990). MCS is popular because it is very flexible. MCS can be used with a wide variety of
probability distributions as well as different types of models. The main challenge for MCS is that it
requires repetitive model calculations to construct a set of pseudo-random numbers for model
inputs and the corresponding estimates for model outputs of interest. There are alternatives to MCS
that are similar but more computationally efficient, such as Latin Hypercube Sampling (LHS).
Techniques are available for simulating correlations between inputs in both MCS and LHS. For
models with very simple functional forms, it may be possible to use exact or approximate analytical
calculations, but such situations are encountered infrequently in practice. There may be situations
in which the data do not conform to a well-defined probability distribution. In such cases,
algorithms (such as Markov Chain Monte Carlo) can estimate a probability distribution by
calculating a mathematical form describing the pattern of observed data. This form, called the
likelihood function, is a key component of Bayesian inference and, therefore, serves as the basis for
some of the analytical approaches to uncertainty and variability described below.
The use of empirical data presumes that the data comprise a representative, random sample. If
known biases or other data quality problems exist, or if there is a scarcity or absence of relevant
data, then na'ive reliance on available empirical data is likely to result in misleading inferences in
30
-------
the analysis. Alternatively, estimates of uncertainty and variability can be encoded, using formal
protocols, based on elicitation of expert judgment (e.g., Morgan and Henrion 1990, USEPA 2011a).
Elicitation of expert judgment for subjective probability distributions is used in situations where
there are insufficient data to support a statistical analysis of uncertainty, but in which there is
sufficient knowledge on the part of experts to make an inference regarding uncertainty. For
example, EPA conducted an expert elicitation study on the concentration-response relationship
between the annual average ambient less than 2.5 micrometer ([im) diameter particulate matter
(PM2.s) exposure and annual mortality (IEC 2006; see also Case Studies 6 and 14 in the Appendix].
Subjective probability distributions that are based on expert judgment can be "updated" with new
data as they become available using Bayesian statistical methods.
Structural Uncertainty in Models
There maybe situations in which it proves useful to evaluate not just the uncertainties in inputs
and parameter values, but also uncertainties regarding whether a model adequately captures—in a
hypothesized, mathematical, structured form—the relationship under investigation. A qualitative
approach to evaluating the structural uncertainty in a model includes describing the critical
assumptions within a model, the documentation of a model or the model quality, and how the
model fits the purpose of the assessment Quantitative approaches to evaluating structural
uncertainty in models are manifold. These include parameterization of a general model that can be
reduced to alternative functional forms (e.g., Morgan and Henrion 1990), enumeration of
alternative models in a probability tree (e.g., Evans et al 1994), comparing alternative models by
evaluating likelihood functions (e.g., Royall 1997; Burnham and Anderson 2002), pooling results of
model alternatives using Bayesian model averaging (e.g., Hoeting et al. 1999) or testing the causal
relationships within alternative models using Bayesian Networks (Pearl 2009).
Sensitivity Analysis: Identifying the Most Important Model Inputs
Probabilistic methods typically focus on how uncertainty or variability in a model input affect [or
result in] with respect to uncertainty or variability in a model output After a probabilistic analysis
is completed, sensitivity analysis typically takes the perspective of looking back to evaluate how
much of the variation in the model output is attributable to individual model inputs (e.g., Frey and
Patil2002; Mokhtari et al. 2006; Saltelli et al. 2004).
There are many types of sensitivity analysis methods, including simple techniques that involve
changing the value of one input at a time and assessing the effect on an output, and statistical
methods that evaluate which of many simultaneously varying inputs contribute the most to the
variance of the model output. Sensitivity analysis can answer the following key questions:
n What is the impact of changes in input values on model output?
n How can variation in output values be apportioned among model inputs?
n What are the ranges of inputs associated with best or worst outcomes?
n What are the key controllable sources of variability?
n What are the critical limits (e.g., the emission reduction target)?
n What are the key contributors to the output uncertainty?
Thus, sensitivity analysis can be used to inform decision making.
Iteration
There are two major types of iteration in risk assessment modeling. One is iterative refinement of
the type of analysis, perhaps starting with a relatively simple DRA as a screening step in an initial
31
-------
level of analysis and proceeding to more refined types of assessments as needed in subsequent
levels of analysis. Examples of more refined levels of assessment include application of sensitivity
analysis to DRA; the use of probabilistic methods to quantify variability only, uncertainty only, or
combined variability and uncertainty (to represent a randomly selected individual); or the use of
two-dimensional probabilistic methods for distinguishing and simultaneously characterizing both
uncertainty and variability.
The other type of iteration occurs within a particular level and includes iterative efforts to
formulate a model, obtain data and evaluate the model to prioritize data needs. For example, a
model may require a large number of input assumptions. To prioritize efforts of specifying
distributions for uncertainty and variability for model inputs, it is useful to determine which model
inputs are the most influential with respect to the assessment endpoint Therefore, sensitivity can
be used based on preliminary assessments of ranges or distributions for each model input to
determine which inputs are the most important to the assessment Refined efforts to characterize
distributions then can be prioritized to the most important inputs.
32
-------
4. SUMMARY AND RECOMMENDATIONS
4.1. Probabilistic Risk Analysis and Related Analyses Can Improve
the Decision-Making Process at EPA
PRA can provide useful (even critical) information about the uncertainties and variability in the
data, models, scenario, expert judgments and values incorporated in risk assessments to support
decision making across the Agency. As discussed in this paper PRA is an analytical methodology
capable of incorporating information regarding uncertainty and/or variability in risk analyses to
provide insight on the degree of certainty of a risk estimate and how the risk estimate varies within
the exposed population. Traditional approaches such as DRA, often report risks using descriptors
such as "central tendency," "high end" (e.g., 90th percentile or above) or "maximum anticipated
exposure". By contrast PRA can be used to describe more completely the uncertainty surrounding
such estimates, as well as to identify the key contributors to uncertainty and variability in predicted
exposures or risk estimates. This information then can be used by decision makers to weigh
alternatives, or to make decisions on whether to collect additional data, or to conduct additional
research in order to reduce the uncertainty and further characterize variability within the exposed
population. Information on uncertainties and variability in exposure and response can ultimately
improve the risk estimates.
PRA can be used to obtain insight on whether one management alternative is more likely to reduce
risks compared to another. In addition, PRA can facilitate the development of modeling scenarios
and the simultaneous consideration of multiple model alternatives. Probabilistic methods offer a
number of tools designed to increase confidence in decision making through the incorporation of
input uncertainty and variability characterization and prioritization in risk analyses. For example,
one PRA tool, sensitivity analyses can be used to identify influential knowledge gaps in the
estimation of risk; this improves transparency in the presentation of these uncertainties and
improves the ability to communicate the most relevant information more clearly to decision makers
and stakeholders. PRA allows one to investigate potential changes in decisions that could result
from the collection of additional information. However, the additional resources (e.g., time, costs,
or expertise) to undertake need to be weighed against the potential improvements in the decision
making process. Ultimately, PRA may enhance the scientific foundation of the EPA's approach to
decision making.
The various tools and methods discussed in this white paper can be utilized at all stages of risk
analysis and also can aid the decision-making process by, for example, characterizing inter-
individual variability and uncertainties.
PRA and related methods are employed in varying degrees across the Agency. Basic guidance exists
at EPA on the use and acceptability of PRA for risk estimation, but implementation varies greatly
within programs, offices and regions .The use of Monte Carlo or other probability-based techniques
to derive a range of possible outputs from uncertain inputs is a fairly well-developed approach
within EPA. Although highly sophisticated human exposure assessment and ecological risk
applications have been developed, the use of PRA models to assess human health effects and dose-
response relationships has been more limited at the Agency.
The evaluation of the application of PRA techniques under specific laws and regulations varies by
program, office and region. Moving forward, it is important to broaden discussions between risk
assessors and risk managers regarding how PRA tools can be used to support specific decisions and
how they can be used within the regulatory framework used by programs, offices, and regions to
make decisions. This can be accomplished by expanding the dialogue between assessors and
33
-------
manages at all levels regarding how the PRA tools have been used and how they have enhanced
decision making.
Increased use of PRA and consistent application of PRA tools in support of EPA decision making
requires enhanced internal capacity for conducting these assessments, as well as improved
interpretation and communication of such information in the context of decisions. Improvements of
Agency capacity could be accomplished through sharing of experiences, knowledge and training
and increased availability of tools and methods.
4.2. Major Challenges to Using Probabilistic Risk Analysis to Support
Decisions
The challenges for EPA are two-fold. As an Agency responsible for protecting human health and the
environment, EPA makes regulatory and policy decisions, even in the presence of conflicting
stakeholder positions and the inevitable uncertainties in the science. The first challenge for EPA is
to determine how to conduct its decision-making responsibilities, weighing determinations of what
constitutes too much uncertainty to make a decision, against potential adverse consequences of
postponing decisions.
The second challenge, is that although current PRA techniques are available that would help to
inform EPA decision-making processes, research and guidance are needed to improve these
methods for a more complete implementation of PRA in HHRA and ERA. In particular, additional
guidance is needed to help analysts and decision makers better understand how to incorporate PRA
approaches into the decision-making process. This includes, guidance on which statistical tools to
use and when to use them, and how probabilistic information can help to inform the scientific basis
of decisions. Both DRA and PRA as well as appropriate statistical methods may be useful at any
stage of the risk analysis and decision-making process, from planning and scoping to characterizing
and communicating uncertainty.
n As noted in Section 3.3. there are significant challenges in properly accounting for
uncertainty and variability when multiple models are coupled together to represent the
source-to-outcome continuum. Moreover, the coupling of multiple models might need to
involve inputs and corresponding uncertainties that are incorporated into more than one
model, potentially resulting in complex dependencies. Integrative research on coupled
model uncertainties will be quite valuable.
n There may be mismatches in the temporal and spatial resolution of each model that
confound the ability to propagate uncertainty and variability from one model to another.
For some models, the key uncertainties may be associated with inputs, whereas for other
models, the key uncertainties may be associated with structure or parameterization
alternatives. Model integration and harmonization activities will be important to addressing
these technical issues.
4.3. Recommendations for Enhanced Utilization of Probabilistic Risk
Analysis at EPA
Some examples of areas where new or updated guidance would be helpful are these:
n Identification of different types of information required for the various Agency decision-
making processes, such as data analysis, tools, models, and use of experts.
n Use of probabilistic approaches to evaluate health effects data.
n Use of probabilistic approaches for ERA.
34
-------
n Integrating probabilistic exposure and risk estimates and communicating uncertainty and
variability.
In order to support the development of guidance on these or related topics, following studies or
research are recommended:
n The use of PRA models to evaluate toxicity data has been very limited. Scientific, technical
and policy-based discussions are needed in this area.
n Additional research on formal methods for treating model uncertainties will be valuable.
Some steps to improve implementation include these:
n Informing decision makers about the advantages and disadvantages of using PRA
techniques in their decision-making processes through lectures, webinars and
communications regarding the techniques and their use in EPA.
n Incorporating a discussion of PRA tools during Planning and Scoping for HHRAs and ERAs.
n Continuing the dialogue between assessors and managers on how to use PRA within the
regulatory decision making process.
n Conducting meetings and discussions of PRA techniques and their application with both
managers and assessors to aid in providing greater consistency and transparency in EPA's
risk assessment and risk management process and in developing EPA's internal capacity.
n Developing a "Community of Practice" for further discussion regarding the application of
PRA techniques and the use of these tools in decision making.
Risk assessors and risk managers need information and training so that they can better utilize these
tools. Education and experience will generate familiarity with these tools, which will help analysts
and decision makers better understand and consider more fully utilizing these techniques within
their regulatory programs. Increased training is needed to facilitate understanding on all levels and
may include the following:
n Providing introductory as well as advanced training to all EPA offices.
n Training risk assessors and risk managers in the PRA techniques so that they can learn
about the various tools available, their applications, software and review considerations,
and resources for additional information (e.g., experts and support services within the
Agency).
n Providing easily available, flexible, modular training for all levels of experience to
familiarize EPA employees with the menu of tools and their capacities.
n Providing live and recorded seminars and webinars for introductory and supplemental
education, as well as periodic, centralized hands-on training sessions demonstrating how to
utilize software programs.
Training is critical both for an improved understanding but also to build increased capacity in the
Agency and explicit steps could include these:
n Demonstrating, through informational opportunities and resource libraries, the various
tools and methods that can be used at all stages of risk analysis to aid the decision-making
process by characterizing inter-individual variability and uncertainties.
n Promoting the sharing of experience, knowledge, models and best practices via meetings of
risk assessors and managers; electronic exchanges, such as the EPA Portal Environmental
35
-------
Science Connector [https://ssoprod.epa.gov/sso/jsp/obloginESCNew.jsp]: and more
detailed discussions of the case studies.
As EPA works toward the more integrated evaluation of environmental problems, this will include
not just the improved understanding of single pollutants/single media, but multi-pollutant, multi-
media and multi-receptor analysis within a decision analytic framework. EPA is beginning to build
such integrated capability into analytical tools like PRA (Babendreier and Castleton 2005; Stahl et
al. 2011).
The RAF will be taking a leadership role through the Uncertainty and Variability Workgroup to
more fully evaluate the application and use of PRA tools and broadening the dialogue between
assessors and managers. Updates on the progress of this Technical Panel will be provided on the
RAF webpage at: www.epa.gov/raf.
36
-------
GLOSSARY
Analysis. Examination of anything complex to understand its nature or to determine its essential
features (WHO 2004).
Assessment. A determination or appraisal of possible consequences resulting from an analysis of
data(2011b).
Assessment endpoint. An explicit expression of the environmental value that is to be protected,
operationally defined by an ecological entity and its attributes. For example, salmon are valued
ecological entities; reproduction and age class structure are some of their important attributes.
Together, salmon "reproduction and age class structure" form an assessment endpoint (USEPA
1998b).
Bayesian probability. An approach to probability, representing a personal degree of belief that a
value of random variable will be observed. Alternatively, the use of probability measures to
characterize the degree of uncertainty (Gelman et al 2004).
Bayesian Analysis. Bayesian analysis is a method of statistical inference in which the knowledge of
prior events is used to predict future events (USEPA 2011b).
Correlation. An estimate of the degree to which two sets of variables vary together, with no
distinction between dependent and independent variables. Correlation refers to a broad class of
statistical relationships involving dependence (USEPA 2012).
Critical control point. A controllable variable that can be adjusted to reduce exposure and risk. For
example, a critical control point might be the emission rate from a particular emission source. The
concept of critical control point is from the hazard assessment and critical control point concept for
risk management that is used in space and food safety applications, among others (USEPA 2006c).
Critical limit. A numerical value of a critical control point at or below which risk is considered to be
acceptable. A criterion that separates acceptability from unacceptability (USEPA 2006c).
Deterministic. A methodology relying on point (i.e., exact) values as inputs to estimate risk; this
obviates quantitative estimates of uncertainty and variability. Results also are presented as point
values. Uncertainty and variability may be discussed qualitatively or semi-quantitatively by
multiple deterministic risk estimates (USEPA 2006b).
Deterministic risk assessment (DRA). Risk evaluation involving the calculation and expression of
risk as a single numerical value or "single point" estimate of risk, with uncertainty and variability
discussed qualitatively (USEPA 2012).
Ecological risk assessment. The process that evaluates the likelihood that adverse ecological
effects may occur or are occurring as a result of exposure to one or more stressors (USEPA 1998b).
Ecosystem. The biotic community and abiotic environment within a specified location in space and
time (USEPA 1998b).
Ensemble. A method for predictive modeling based on multiple measures of the same event over
time (e.g., the amount of carbon dioxide present in the atmosphere at selected time points). The
collection of data input is known as an ensemble and can be used to develop a quantification of
prediction variability within the model. Ensemble modeling is used most commonly in atmospheric
prediction in forecasting, although ensemble modeling has been applied to biological systems to
better quantify risks of events or perturbations within biological systems (Fuentes and Foley 2012).
Environment. The sum of all external conditions affecting the life, development and survival of an
organism (USEPA 2010a).
37
-------
Expert elicitation. A systematic process of formalizing and quantifying, typically in probabilistic
terms, expert judgments about uncertain quantities (USEPA 201 la).
Frequentist (or frequency) probability. A view of probability that concerns itself with the
frequency with which an event occurs given a long sequence of identical and independent trials
(USEPA 1997b).
Hazard identification. The risk assessment process of determining whether exposure to a stressor
can cause an increase in the incidence or severity of a particular adverse effect, and whether an
adverse effect is likely to occur (USEPA 2012).
Human health risk assessment. 1. The process to estimate the nature and probability of adverse
health effects in humans who may be exposed to chemicals in contaminated environmental media,
now or in the future (USEPA 2010b). 2. The evaluation of scientific information on the hazardous
properties of environmental agents (hazard characterization), the dose-response relationship
(dose-response assessment), and the extent of human exposure to those agents (exposure
assessment). The product of the risk assessment is a statement regarding the probability that
populations or individuals so exposed will be harmed and to what degree (risk characterization)
(USEPA 2006a).
Inputs. Quantities that are applied to a model (WHO 2008).
Likelihood Function. An approach to modeling exposure in which long-term exposure of an
individual is simulated as the sum of separate short-term exposure events (USEPA 2001).
Microenvironment. Well-defined surroundings such as the home, office, automobile, kitchen, store,
etc., that can be treated as homogenous (or well characterized) in the concentrations of a chemical
or other agent (USEPA 1992).
Microexposure event (MEE) analysis. An approach to modeling exposure in which long-term
exposure of an individual is simulated as the sum of separate short-term exposure events (USEPA
2001).
Model. A mathematical representation of a natural system intended to mimic the behavior of the
real system, allowing description of empirical data, and predictions about untested states of the
system (USEPA 2006b).
Model boundaries. 1. Decisions regarding the time, space, number of chemicals, etc., used in
guiding modeling of the system. Risks can be understated or overstated if the model boundary is
mis-specified. For example, if a study area is defined to be too large and includes a significant
number of low-exposure areas, then a population-level risk distribution can be diluted by including
less exposed individuals, which can, in turn, result in a risk-based decision that does not protect
sufficiently the most exposed individuals in the study area. 2. Designated areas of competence of
the model, including time, space, pathogens, pathways, exposed populations, and acceptable ranges
of values for each input and jointly among all inputs for which the model meets data quality
objectives (WHO 2008).
Modeling. Development of a mathematical or physical representation of a system or theory that
accounts for all or some of its known properties. Models often are used to test the effect of changes
of components on the overall performance of the system (USEPA 2010a).
Model uncertainty (sources of):
n Model structure. A set of assumptions and inference options upon which a model is based,
including underlying theory as well as specific functional relationships [WHO 2008).
38
-------
n Model detail. Level of simplicity or detail associated with the functional relationships
assumed in the model compared to the actual but unknown relationships in the system being
modeled (WHO 2008).
n Extrapolation. Use of models outside of the parameter space used in their derivation may
result in erroneous predictions. For example, a threshold for health effects may exist at
exposure levels below those covered by a particular epidemiological study. If that study is
used in modeling health effects at those lower levels (and it is assumed that the level of
response seen in the study holds for lower levels of exposure), then disease incidence may be
overestimated (USEPA 2007a).
Monte Carlo analysis (MCA) or simulation (MCS). A repeated random sampling from the
distribution of values for each of the parameters in a generic exposure or risk equation to derive an
estimate of the distribution of exposures or risks in the population (USEPA 2006b).
One-dimensional Monte Carlo analysis (1-D MCA). A numerical method of simulating a
distribution for an endpoint of concern as a function of probability distributions that characterize
variability or uncertainty. Distributions used to characterize variability are distinguished from
distributions used to characterize uncertainty (WHO 2008).
Parameter. A quantity used to calibrate or specify a model, such as 'parameters' of a probability
model (e.g., mean and standard deviation for a normal distribution). Parameter values often are
selected by fitting a model to a calibration data set (WHO 2008).
Probability. A frequentist approach considers the frequency with which samples are obtained
within a specified range or for a specified category (e.g., the probability that an average individual
with a particular mean dose will develop an illness) (WHO 2008).
Probabilistic risk analysis (PRA). Calculation and expression of health risks using multiple risk
descriptors to provide the likelihood of various risk levels. Probabilistic risk results approximate a
full range of possible outcomes and the likelihood of each, which often is presented as a frequency
distribution graph, thus allowing uncertainty or variability to be expressed quantitatively (USEPA
2012).
Problem formulation. The initial stage of a risk assessment where the purpose of the assessmentis
articulated, exposure and risk scenarios are considered, a conceptual model is developed, and a
plan for analyzing and characterizing risk is determined (USEPA 2004a).
Reference concentration (RfC). An estimate (with uncertainty spanning approximately an order of
magnitude) of a continuous inhalation exposure to the human population (including sensitive
subgroups) that is likely to be without an appreciable risk of deleterious effects during a lifetime. It
can be derived from a No-Observed-Adverse-Effect Level (NOAEL), Lowest-Observed-Adverse-
Effect Level (LOAEL), or benchmark concentration, with uncertainty factors generally applied to
reflect limitations of the data used. It is generally used in EPA's noncancer health assessments
(USEPA 2007a).
Reference dose (RfD). An estimate (with uncertainty spanning approximately an order of
magnitude) of a daily oral exposure to the human population (including sensitive subgroups) that is
likely to be without an appreciable risk of deleterious effects during a lifetime. It can be derived
from a NOAEL, LOAEL or benchmark dose, with uncertainty factors generally applied to reflect
limitations of the data used. It is typically used in EPA's noncancer health assessments (USEPA
2011c).
Risk. 1. Risk includes consideration of exposure to the possibility of an adverse outcome, the
frequency with which one or more types of adverse outcomes may occur, and the severity or
39
-------
consequences of the adverse outcomes if such occur. 2. The potential for realization of unwanted,
adverse consequences to human life, health, property or the environment. 3. The probability of
adverse effects resulting from exposure to an environmental agent or mixture of agents. 4. The
combined answers to: What can go wrong? How likely is it? What are the consequences? (USEPA
2011c).
Risk analysis. A process for identifying, characterizing, controlling and communicating risks in
situations where an organism, system, subpopulation or population could be exposed to a hazard.
Risk analysis is a process that includes risk assessment, risk management and risk communication
(WHO 2008).
Risk assessment. I. A process intended to calculate or estimate the risk to a given target organism,
system, subpopulation or population, including the identification of attendant uncertainties
following exposure to a particular agent, taking into account the inherent characteristics of the
agent of concern, as well as the characteristics of the specific target system (WHO 2008). 2. The
evaluation of scientific information on the hazardous properties of environmental agents (hazard
characterization), the dose-response relationship (dose-response assessment), and the extent of
human exposure to those agents (exposure assessment) (NRC 1983). The product of the risk
assessment is a statement regarding the probability that populations or individuals so exposed will
be harmed and to what degree (risk characterization; USEPA 2000a). 3. Qualitative and quantitative
evaluation of the risk posed to human health or the environment by the actual or potential presence
or use of specific pollutants (USEPA 2012).
Risk-based decision making. A process through which decisions are made according to the risk
each posed to human health and the environment (USEPA 2012).
Risk management. A decision-making process that takes into account environmental laws;
regulations; and political, social, economic, engineering and scientific information, including a risk
assessment, to weigh policy alternatives associated with a hazard (USEPA 2011c).
Scenario. A set of facts, assumptions and inferences about how exposure takes place that aids the
exposure assessor in evaluating, estimating or quantifying exposures (USEPA 1992). Scenarios
might include identification of pollutants, pathways, exposure routes and modes of action, among
others.
Sensitivity analysis. The process of changing one variable while leaving the others constant to
determine its effect on the output This procedure fixes each uncertain quantity at its credible lower
and upper bounds (holding all others at their nominal values, such as medians) and computes the
results of each combination of values. The results help to identify the variables that have the
greatest effect on exposure estimates and help focus further information-gathering efforts (USEPA
2011b).
Tiered approach. Refers to various hierarchical tiers (levels) of complexity and refinement for
different types of modeling approaches that can be used in risk assessment. A deterministic risk
assessment with conservative assumptions is an example of a lower level type of analysis (Tier 0)
that can be used to determine whether exposures and risks are below levels of concern. Examples
of progressively higher levels include the use of deterministic risk assessment coupled with
sensitivity analysis (Tier 1), the use of probabilistic techniques to characterize either variability or
uncertainty only (Tier 2), and the use of two-dimensional probabilistic techniques to distinguish
between but simultaneously characterize both variability and uncertainty (Tier 3) (USEPA 2004a
and WHO 2008).
Two-dimensional Monte Carlo analysis (2-D MCA). An advanced numerical modeling technique
that uses two stages of random sampling, also called nested loops, to distinguish between
40
-------
variability and uncertainty in exposure and toxicity variables. The first stage, often called the inner
loop, involves a complete 1-D MCA simulation of variability in risk. In the second stage, often called
the outer loop, parameters of the probability distributions are redefined to reflect uncertainty.
These loops are repeated many times resulting in multiple risk distributions, from which
confidence intervals are calculated to represent uncertainty in the population distribution of risk
(WHO 2008).
Uncertainty. Uncertainty occurs because of a lack of knowledge. It is not the same as variability.
For example, a risk assessor may be very certain that different people drink different amounts of
water but may be uncertain about how much variability there is in water intakes within the
population. Uncertainty often can be reduced by collecting more and better data, whereas
variability is an inherent property of the population being evaluated. Variability can be better
characterized with more data but it cannot be reduced or eliminated. Efforts to clearly distinguish
between variability and uncertainty are important for both risk assessment and risk
characterization, although they both may be incorporated into an assessment (USEPA 201 Ic).
Uncertainty analysis. A detailed examination of the systematic and random errors of a
measurement or estimate; an analytical process to provide information regarding uncertainty
(USEPA 2006b).
Value of information. An analysis that involves estimating the value that new information can have
to a risk manager before the information is actually obtained. It is a measure of the importance of
uncertainty in terms of the expected improvement in a risk management decision that might come
from better information (USEPA 2001).
Variability. Refers to true heterogeneity or diversity, as exemplified in natural variation. For
example, among a population that drinks water from the same source and with the same
contaminant concentration, the risks from consuming the water may vary. This may result from
differences in exposure (e.g., different people drinking different amounts of water and having
different body weights, exposure frequencies and exposure durations), as well as differences in
response (e.g., genetic differences in resistance to a chemical dose). Those inherent differences are
referred to as variability. Differences among individuals in a population are referred to as inter-
individual variability, and differences for one individual over time are referred to as intra-
individual variability (USEPA 20lie).
41
-------
REFERENCES
Ang, A. H-S., and W. H. Tang. 1984. Probability Concepts in Engineering Planning and Design, Volume
2: Decision, Risk, and Reliability. New York: John Wiley & Sons.
Babendreier, J. E., and K. J. Castleton. 2005. "Investigating Uncertainty and Sensitivity in Integrated,
Multimedia Environmental Models: Tools for FRAMES-3MRA." Environmental Modeling & Software
20:1043-1055.
Bloom, D. L, D. M. Byrne, and J. M. Andreson. 1993. Communicating Risk to Senior EPA Policy-
Makers: A Focus Group Study. Prepared by Bloom Research and the Office of Air Quality Planning
and Standards. Research Triangle Park, NC: USEPA.
Burnham, K. P., and D. R. Anderson. 2002. Model Selection and Inference: A Practical Information-
Theoretic Approach. 2nded. New York: Springer-Verlag.
Clyde, M., P. Guttorp, and E. Sullivan. 2000. Effects of Ambient Fine and Coarse Particles on Mortality
in Phoenix, Arizona. National Research Center for Statistics and the Environment Technical Report
Series, No. 40.
Cooke, R. M. 1991. Experts in Uncertainty: Opinion and Subjective Probability in Science. New York:
Oxford University Press.
Costanza, R., S. 0. Funtowicz, and J. R. Ravetz. 1997. "Assessing and Communicating Data Quality in
Policy-Relevant Research." In Frontiers in Ecological Economics: Transdisciplinary Essays, by Robert
Costanza, 283-308. Cheltenham, UK: E. Elgar.
Cullen, A. C., and H. C. Frey. 1999. Probabilistic Exposure Assessment: A Handbook for Dealing with
Variability and Uncertainty in Models and Inputs. New York: Plenum Press.
Evans, J. S., J. D. Graham, G. M. Gray, and R. L. Sielken. 1994. "A Distributional Approach to
Characterizing Low-Dose Cancer Risk." Risk Analysis 14 (1): 25-34.
Person, S., R. Nelsen, J. Hajagos, D. Berleant, J. Zhang, W. T. Tucker, L. Ginzburg, and W. L.
Oberkampf. 2004. Dependence in Probabilistic Modeling, Dempster-Shafer Theory, and Probability
Bounds Analysis. SAND20043072. Albuquerque, NM: Sandia National Laboratories.
www.ramas.com/depend.pdf.
Person, S., W. L. Oberkampt, and L. Ginzburg. 2009. "Validation of Imprecise Probability Models."
International Journal of Reliability and Safety 3 (1-3): 3-22.
Finkel, A. M. 1989. "Is Risk Assessment Really Too Conservative?: Revising the Revisionists."/ourna/
of Environmental Law 14: 427-67.
Fisher, L., P. Pascual, and W. Wagner. 2006. Mapping the Role of Models in U.S. and E.U. Risk
Regulation: A Legal Background Paper. Washington, D.C.: The Association of Union Constructors
Workshop.
Fischhoff, B. 1995. "Ranking Risks." Risk: Health, Safety & Environment 6: 191-202.
Frey, H. C., and S. R. Patil. 2002. "Identification and Review of Sensitivity Analysis Methods." Risk
Analysis 22 (3): 553-78.
Fuentes, M. and K. Foley. 2012. "Ensemble Models." In Abdel, H. and Walter W. El-Shaarawai (eds.)
Encyclopedia of Environmetrics. Indianapolis: John Wiley and Sons.
Gelman, A, J. B. Carlin, H. S. Stern, and D. B. Rubin. 2004. Bayesian Data Analysis. 2nd ed. Boca Raton,
FL: Chapman and Hall/CRC.
42
-------
Hattis, D., S. Baird, and R. Goble. 2002. "A Straw Man Proposal for a Quantitative Definition of the
RfD." Drug and Chemical Toxicology 25 (4): 403-36.
Hattis, D. B., and D. E. Burmaster. 1994. "Assessment of Variability and Uncertainty Distributions for
Practical Risk Analyses." Risk Analysis 14 (5): 713-30.
Hattis, D., and M. K. Lynch. 2010. Alternatives to Pollutant-by-Pollutant Dose-Response Estimation for
Air Toxics (Final Report}. EPA-W-05-022, WA 3-80. Washington, D.C.: USEPA.
Hoeting, J. A, D. Madigan, A. E. Raftery, and C. T. Volinsky. 1999. "Bayesian Model Averaging: A
Tutorial." Statistical Science 14 (4): 382-417.
IEC (Industrial Economics, Inc.). 2006. Expanded Expert Judgment Assessment of the Concentration-
Response Relationship Between PM2.sExposure and Mortality (Final Report). Research Triangle Park,
NC: USEPA. http://www.epa.gov/ttn/ecas/regdata/Uncertainty/pm ee reportpdf.
Institute of Medicine. 2013. Environmental Decisions in the Face of Uncertainty. Washington, D.C.:
The National Academies Press.
Illing, H. P. A. 1999. "Are Societal Judgments Being Incorporated Into the Uncertainty Factors Used
in Toxicological Risk Assessment?" Regulatory Toxicology and Pharmacology 29 (3): 300-308.
Krupnick, A, R. Morgenstern, M. Batz, P. Nelson, D. Burtraw, J. Shih, and M. McWilliams. 2006. Not a
Sure Thing: Making Regulatory Choices Under Uncertainty. Washington, D.C.: Resources for the
Future. http://www.rff.org/rff/Documents/RFF-Rpt-RegulatoryChoices.pdf.
Kunreuther, H. C., and P. Slovic. 1996. "Science, Values and Risk." Annals of the American Academy of
Political and Social Science 545: 116-25.
Lester, R. R, L. C. Green, and I. Linkov. 2007. "Site-Specific Applications of Probabilistic Health Risk
Assessment: Review of the Literature Since 2000." Risk Analysis 27: 635-58.
McConnell, K. E. 1997. "Using Cost-Benefit Analysis in the Management of Contaminated Sediments
(Appendix E)." In Contaminated Sediments in Ports and Waterways: Cleanup Strategies and
Technologies, by the Commission on Engineering and Technical Systems of the National Research
Council, 239-56. Washington, D.C.: The National Academies Press.
Mokhtari, A., H. C. Frey, and J. Zheng. 2006. "Evaluation and Recommendation of Sensitivity Analysis
Methods for Application to Stochastic Human Exposure and Dose Simulation (SHEDS) Models."
Journal of Exposure Science and Environmental Epidemiology 16 (6): 491-506.
Morgan, G., H. Dowlatabadi, M. Henrion, D. Keith, R. Lempert, S. McBrid, M. Small, and T. Wilbanks,
eds. 2009. Best Practice Approaches for Characterizing, Communicating, and Incorporating Scientific
Uncertainty in Decision Making. Final Report Washington, D.C.: U.S. Climate Change Science
Program, National Oceanic and Atmospheric Administration, http://library.globalchange.gov/sap-
5-2-best-practice-approaches-for-characterizing-communicating-and-incorporating-scientific-
uncertainty-in-decisionmaking.
Morgan, M. G., and M. Henrion. 1990. Uncertainty: A Guide to Dealing with Uncertainty in
Quantitative Risk and Policy Analysis. New York: Cambridge University Press.
NRC (National Research Council). 1983. Risk Assessment in the Federal Government: Managing the
Process. Committee on the Institutional Means for Assessment of Risks to Public Health.
Washington, D.C.: The National Academies Press.
NRC. 1989. Risk Assessment in the Federal Government: Managing the Process. Washington, D.C.: The
National Academies Press.
43
-------
NRC. 1991. Rethinking the Ozone Problem in Urban and Regional Air Pollution. Washington, D.C.: The
National Academies Press.
NRC. 1993. Issues in Risk Assessment. Washington, D.C.: The National Academies Press.
NRC. 1994. Science and Judgment in Risk Assessment. Washington, D.C.: The National Academies
Press.
NRC. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. Washington,
D.C.: The National Academies Press.
NRC. 2007a. Scientific Review of the Proposed Risk Assessment Bulletin From the Office of
Management and Budget. Washington, D.C.: The National Academies Press.
NRC. 2007b. Models in Environmental Regulatory Decision Making. Washington, D.C.: The National
Academies Press.
NRC. 2009. Science and Decisions: Advancing Risk Assessment. Washington, D.C.: The National
Academies Press.
OSTP (Office of Science and Technology Policy)/OMB (Office of Management and Budget). 2007.
Memorandum (M-07-24) for the Heads of Executive Departments and Agencies; from Susan E.
Dudley, Administrator, Office of Information and Regulatory Affairs, Office of Management and
Budget, to Sharon L. Hays, Associate Director and Deputy Director for Science, Office of Science and
Technology Policy: Updated Principles for Risk Analysis. September 19, 2007.
Ozkaynak H., H.C. Frey, J. Burke, and R.W. Finder. 2009. "Analysis of Coupled Model Uncertainties in
Source-to-Dose Modeling of Human Exposures to Ambient Air Pollution: A PM2.5 Case-Study."
Atmospheric Environment 43: 1641-49.
Parkin, R. T., and M. G. Morgan. 2007. Consultation on Enhancing Risk Assessment Practices and
Updating EPA's Exposure Guidelines. EPA/SAB-07/003. Washington, D.C.: USEPA. USEPA. 2007.
Letter From M. G. Morgan and R. T. Parkin, Science Advisory Board, to S. Johnson, U.S.
Environmental Protection Agency. February 28, 2007. EPA/SAB-07/003.
http://yosemite.epa. gov/sab/sabproductnsf/55ElB2C78C6085EB8525729C00573A3E/$File/sab-
07-003.pdf.
PCCRARM (Presidential/Congressional Commission on Risk Assessment and Risk Management).
1997. Framework for Environmental Health Risk Management Final Report, Volume 1. Washington,
D.C.: PCCRARM. http://www.riskworld.com/nreports/1997/risk-rpt/pdf/EPAJAN.PDF.
Pearl, J. 2009. Causality: Models, Reasoning, and Inference. 2nd ed. New York: Cambridge University
Press.
Royall, R. M. 1997. Statistical Evidence: A Likelihood Paradigm. London: Chapman & Hall.
Saltelli, A., S. Tarantola, F. Campolongo, and M. Rattoet 2004. Sensitivity Analysis in Practice. New
York: John Wiley & Sons.
Stahl, C. H., and A. J. Cimorelli. 2005. "How Much Uncertainty Is Too Much and How Do We Know? A
Case Example of the Assessment of Ozone Monitor Network Options." Risk Analysis 25 (5): 1109-20.
Stahl, C. H., and A. J. Cimorelli. 2012. "The Demonstration of the Necessity and Feasibility of Using a
Clumsy Decision Analytic Approach in Wicked Environmental Problems." Integrated Environmental
Assessment and Management 9 (1):17.
44
-------
Stahl, C. H., A. J. Cimorelli, C. Mazzarella, and B. Jenkins. 2011. "Toward Sustainability: A Case Study
Demonstrating Transdisciplinary Learning." Integrated Environmental Assessment and Management
7 (3): 483-98.
Swartout, J. C., P. S. Price, M. L. Dourson, H. L. Carlson-Lynch, and R. E. Keenan. 1998. "Probabilistic
Framework for the Reference Dose (Probabilistic RfD)." Risk Analysis 18 (3): 271-82.
Toll, J., S. Pavlou, D. Lee, L. Zaragosa, and P. Shelley. 1997. "Using Decision Analysis in the
Management of Contaminated Sediments (Appendix E)." In Contaminated Sediments in Ports and
Waterways: Cleanup Strategies and Technologies, by the Commission on Engineering and Technical
Systems of the National Research Council, 257-84. Washington, D.C.: The National Academies Press.
Toll, J. E. 1999. "Elements of Environmental Problem-Solving." Human and Ecological Risk
Assessments (2): 275-80.
USEPA (U.S. Environmental Protection Agency). 1992. Guidelines for Exposure Assessment.
EPA/600/Z-92/001. Washington, D.C.: Risk Assessment Forum, USEPA.
http://cfpub.epa. gov/ncea/cfm/recordisplay.cfm?deid=l 5263.
USEPA. 1995a. Guidance for Risk Characterization. Washington, D.C.: Science Policy Council, USEPA.
http://www.epa.gov/spc/pdfs/rcguide.pdf.
USEPA. 1995b. Policy for Risk Characterization. Washington, D.C.: USEPA.
http://www.epa.gov/oswer/riskassessment/pdf/1995 0521 risk characterization program.pdf.
USEPA. 1997 a. Policy for Use of Probabilistic Analysis in Risk Assessment at the U.S. Environmental
Protection Agency. Washington, D.C.: USEPA. http://www.epa.gov/spc/pdfs/probpol.pdf.
USEPA. 1997b. Guiding Principles for Monte Carlo Analysis. EPA/630/R-97/001. Washington, D.C.:
USEPA. http://www.epa.gov/raf/publications/pdfs/montecar.pdf.
USEPA. 1998a. Guidance for Submission of Probabilistic Human Health Exposure Assessments to the
Office of Pesticide Programs (Draft). Washington, D.C.: USEPA. http://www.epa.gov/fedrgstr/EPA-
PEST/1998/November/Day-05/6021.pdf.
USEPA. 1998b. Guidelines for Ecological Risk Assessment. EPA/630/R-95/002F. Washington, D.C.:
Risk Assessment Forum, USEPA. http://www.epa.gov/raf/publications/pdfs/ECOTXTBX.PDF.
USEPA. 2000a. EPA Science Policy Council Handbook: Risk Characterization. EPA/100/B-00/002.
Washington, D.C.: USEPA. http://www.epa.gov/spc/pdfs/rchandbkpdf.
USEPA. 2000b. Toward Integrating Environmental Decision-Making. EPA/SAB/EC-00/011.
Washington, D.C.: Science Advisory Board, USEPA.
USEPA. 2001. Risk Assessment Guidance for Superfund: Volume III—Part A, Process for Conducting
Probabilistic Risk Assessment. EPA/540/R-02/002. Washington, D.C.: USEPA.
http://www.epa.gov/oswer/riskassessment/rags3adt/pdf/rags3adt complete.pdf.
USEPA. 2004a. Air Toxics Risk Assessment Reference Library. Volume I: Technical Resource Manual.
EPA/453/K-04/001A. Research Triangle Park, NC: USEPA.
http://www.epa.gov/ttn/fera/data/risk/vol I/title page volume l.pdf.
USEPA. 2004b.,4n Examination of EPA Risk Assessment Principles and Practices. EPA/100/B-04/001.
Washington, D.C.: USEPA. http://www.epa.gov/osa/pdfs/ratf-final.pdf.
USEPA. 2006a. A Framework for Assessing Health Risks of Environmental Exposures to Children.
EPA/600/R-05/093F. Washington, D.C.: USEPA.
http://ofmpub.epa.gov/eims/eimscomm.getfile7p download id=459047.
45
-------
USEPA. 2006b. Air Toxics Risk Assessment Reference Library, Volume 3: Community-Scale Assessment.
EPA/452/K-06/001C. Washington, D.C.: USEPA. http://www.epa.gov/ttn/fera/risk atra vol3.html.
USEPA. 2006c. Hazard Analysis Critical Control Point Strategies for Distribution System Monitoring,
Hazard Assessment and Control. Washington, D.C.: USEPA.
http://www.epa.gov/ogwdw/disinfection/tcr/pdfs/issuepaper tcr haccp-strategies.pdf.
USEPA. 2007a. Concepts, Methods and Data Sources for Cumulative Health Risk Assessment of
Multiple Chemicals, Exposures and Effects: A Resource Document. EPA/600/R-06/013F. Washington,
D.C.: USEPA. http://cfpub.epa.gov/ncea/risk/recordisplay.cfm?deid= 190187.
USEPA. 2007b. Letter From M. G. Morgan and R. T. Parkin, Science Advisory Board, to S. Johnson,
U.S. Environmental Protection Agency. February 28, 2007. EPA/SAB-07/003.
http://yosemite.epa.gov/sab/sabproduct. nsf/55ElB2C78C6085EB8525729C00573A3E/$File/sab-
07-003.pdf.
USEPA. 2010a. Environmental Monitoring and Assessment Program Master Glossary. Last modified
November 8. http://www.epa.gov/emap
2/html/pubs/docs/resdocs/mglossary.html#ee.
USEPA. 2010b. Human Health Risk Assessment Onondaga Lake, Lake Bottom Subsite: Sediment
Consolidation Area, Camillus, NY. Washington, D.C.: USEPA.
http://www.epa.gov/region2/superfund/npl/onondagalake/healthriskassess.pdf.
USEPA. 2011a. Expert Elicitation Task Force White Paper. Washington, D.C.: USEPA.
http://www.epa.gov/stpc/pdfs/ee-white-paper-final.pdf.
USEPA. 2011b. Exposure Factors Handbook: 2011 Edition. EPA/600/R-09/052F. Washington, D.C.:
National Center for Environmental Assessment, Office of Research and Development, USEPA.
http://cfpub.epa. gov/ncea/risk/recordisplay.cfm?deid=23 625 2.
USEPA. 2011c. Integrated Risk Information System Glossary. Last modified August 21.
http://ofmpub.epa.gov/sor internet/registry/termreg/searchandretrieve/glossariesandkeywordli
sts/search.do?details=&glossaryName=IRIS%20Glossary.
USEPA. 2012. Waste and Cleanup Risk Assessment Glossary. Office of Solid Waste and Emergency
Response, USEPA. Last modified December 24.
http://www.epa.gov/oswer/riskassessment/glossary.htm.
USEPA. 2014a. Probabilistic Risk Assessment to Inform Decision Making: Frequently Asked Questions.
EPA/100/R-09/001B. Washington, D.C.: Risk Assessment Forum, Office of the Science Advisor,
USEPA.
USEPA. 2014b. Framework for Human Health Risk Assessment to Inform Decision Making.
EPA/100/R-14/001. Washington, D.C.: Risk Assessment Forum, Office of the Science Advisor,
USEPA.
Wilson, J. D. 2000. "Risk Management Cannot Abide Uncertainty." In J. H. Exner and M. L. Trehy
(eds.) Symposia papers presented before the Division of Environmental Chemistry, American
Chemical Society, August 20-24, 2000. Preprints of Extended Abstracts 40 (2): 268-69. Washington,
D.C.: American Chemical Society.
WHO (World Health Organization). 2004. International Programme on Chemical Safety Risk
Assessment Terminology. Geneva: WHO.
http://www.who.int/ipcs/methods/harmonization/areas/ipcsterminologypartsland2.pdf.
46
-------
WHO. 2008. Uncertainty and Data Quality in Exposure Assessment. Geneva: WHO.
http://www.who.int/ipcs/publications/methods/harmonization/exposure assessmentpdf.
Wright, G., F. Bolger, and G. Rowe. 2002. "An Empirical Test of the Relative Validity of Expert and
Lay Judgments of Risk." Risk Analysis 2 2 (6): 1107-22.
47
-------
BIBLIOGRAPHY
Probabilistic Risk Analysis Methodology—General
Cullen, A. C., and H. C. Frey. 1999. The Use of Probabilistic Techniques in Exposure Assessment: A
Handbook for Dealing with Variability and Uncertainty in Models and Inputs. New York: Plenum.
Dakins, M. E., J. E. Toll, M. J. Small, and K. P. Brand. 1996. "Risk-Based Environmental Remediation:
Bayesian Monte Carlo Analysis and the Expected Value of Sample Information." Risk Analysis 16 (1):
67-79.
Person, S., andV. Kreinovich. 2006. Modeling Correlation and Dependence Among Intervals.
University of Texas at El Paso Departmental Technical Reports (CS), Paper 131.
http://digitalcommons.utep.edu/cs techrep/131.
Finkel, A. M. 1990. Confronting Uncertainty in Risk Management: A Guide for Decision-Makers.
Washington, D.C.: Resources for the Future.
Frey, H. C., D. Crawford-Brown, J. Zheng, and D. Loughlin. 2003. Hierarchy of Methods to
Characterize Uncertainty: State of Science of Methods for Describing and Quantifying Uncertainty
[Draft). Research Triangle Park, NC: USEPA.
Frey, C., J. Penman, L. Hanle, S. Monni, and S. Ogle. 2006. Chapter 3, "Uncertainties." In S. Eggleston,
L. Buendia, K. Miwa, T. Ngara, and K. Tanabe (eds.) 2006IPCC Guidelines for National Greenhouse
Gas Inventories, Volume I: General Guidance and Reporting. Kanagawa, Japan: National Greenhouse
Gas Inventories Programme, Inter-Governmental Panel on Climate Change.
Hattis, D. 1996. "Human Interindividual Variability in Susceptibility to Toxic Effects—From
Annoying Detail to a Central Determinant of Risk." Toxicology III: 5-14.
Hattis, D., and E. Anderson. 1999. "What Should Be the Implications of Uncertainty, Variability, and
Inherent 'Biases'/'Conservatism' for Risk Management Decision Making?" Risk Analysis 19: 95-107.
Hattis, D., and M. K. Lynch. 2007. "Empirically Observed Distributions of Pharmacokinetic and
Pharmacodynamic Variability in Humans—Implications for the Derivation of Single Point
Component Uncertainty Factors Providing Equivalent Protection as Existing RfDs." In J. C. Lipscomb
and E. V. Ohanian (eds.) Toxicokinetics in Risk Assessment, 69-93. New York: Informa Healthcare
USA, Inc.
Hattis, D. 2009. "Failure to Communicate—Comment: Regulator/Risk Perspective on Burzala and
Mazzuchi's Paper." In R.M. Cooke (ed.). Uncertainty Modeling in Dose Response: Bench Testing
Environmental Toxicity, 153-59. New York: John Wiley & Sons, Inc.
Loughlin, D., H. C. Frey, K. Hanisak, and A. Eyth. 2003. Implementation Requirements for the
Development of a Sensitivity/Uncertainty Analysis Tool for MIMS (Draft). Research Triangle Park, NC:
USEPA.
Morgan, M. G., and M. Henrion. 1990. Uncertainty: A Guide to Dealing with Uncertainty in
Quantitative Risk and Policy Analysis. New York: Cambridge University Press.
NARSTO (North American Research Strategy for Tropospheric Ozone). 2005. Improving Emission
Inventories for Effective Air Quality Management Across North America, A NARSTO Assessment.
NARSTO-05-001. Pasco, WA: NARSTO.
NRC (National Research Council). 1994. Science and Judgment in Risk Assessment. Washington, D.C.:
The National Academies Press.
48
-------
USEPA (U.S. Environmental Protection Agency). 1997. Guiding Principles for Monte Carlo Analysis.
EPA/630/R-97/001. Washington, D.C.: USEPA.
http://www.epa.gov/raf/publications/pdfs/montecar.pdf.
Probabilistic Risk Analysis and Decision Making
Bernardini, A., and F. Tonon. 2010. Bounding Uncertainty in Civil Engineering: Theoretical
Background. New York: Springer.
Bloom, D. L, D. M. Byrne, and J. M. Andreson. 1993. Communicating Risk to Senior EPA Policy-
Makers: A Focus Group Study. Research Triangle Park, NC: USEPA.
Krupnick, A, R. Morgenstern, M. Batz, P. Nelson, D. Burtraw, J.-S. Shih, and M. McWilliams. 2006. Not
a Sure Thing: Making Regulatory Choices Under Uncertainty. Washington, D.C.: Resources for the
Future.
McGarity, T. 0., and W. E. Wagner. 2003. "Legal Aspects of the Regulatory Use of Environmental
Modeling." Environmental Law Reporter 33: 10751-74.
NRC. 1996. Understanding Risk: Informing Decisions in a Democratic Society. Washington, D.C.: The
National Academies Press.
NRC. 2009. Science and Decisions: Advancing Risk Assessment. Washington, D.C.: The National
Academies Press.
Pascual, P., N. Steiber, and E. Sunderland. 2003. Draft Guidance on the Development, Evaluation, and
Application of Regulatory Environmental Models. Washington, D.C.: USEPA.
Thompson, K. M., and J. D. Graham. 1996. "Going Beyond the Single Number: Using Probabilistic
Risk Assessment to Improve Risk Management." Human and Ecological Risk Assessment 2 (4): 1008-
34.
USEPA. 1997. Memo from Fred Hansen. Use of Probabilistic Techniques (including Monte Carlo
Analysis) in Risk Assessment, dated May 15,1997. Washington, D.C.: USEPA.
Vick, S. G. 2002. Degrees of Belief: Subjective Probability and Engineering Judgment. Reston, VA: ASCE
Press.
Probabilistic Risk Analysis Methodology—Specific Aspects
Abdel-Aziz, A, and H. C. Frey. 2003. "Development of Hourly Probabilistic Utility NOX Emission
Inventories Using Time Series Techniques: Part I—Univariate Approach." Atmospheric Environment
37:5379-89.
Abdel-Aziz, A., and H.C. Frey. 2003. "Development of Hourly Probabilistic Utility NOX Emission
Inventories Using Time Series Techniques: Part II—Multivariate Approach." Atmospheric
Environments?: 5391-401.
Abdel-Aziz, A., and H. C. Frey. 2004. "Propagation of Uncertainty in Hourly Utility NOX Emissions
Through a Photochemical Grid Air Quality Model: A Case Study for the Charlotte, NC Modeling
Domain." Environmental Science & Technology 38 (7): 2153-60.
Gasman, E. A., M. G. Morgan, and H. Dowlatabadi. 1999. "Mixed Levels of Uncertainty in Complex
Policy Models." Risk Analysis 19 (1): 33-42.
49
-------
Crawford-Brown, D.}., and K. G. Brown. 1987. "A Framework for Assessing the Rationality of
Judgments in Carcinogenicity Hazard Identification." Risk: Health, Safety and Environment 8: 307-
37.
Draper, D. 1995. "Assessment and Propagation of Model Uncertainty." Journal of the Royal Statistical
Society 57: 45-97.
Evans, J. S., J. D. Graham, G. M. Gray, and R. L. Sielken. 1994. "A Distributional Approach to
Characterizing Low-Dose Cancer Risk." Risk Analysis 14 (1): 25-34.
Frey, H. C., and D. E. Burmaster. 1999. "Methods for Characterizing Variability and Uncertainty:
Comparison of Bootstrap Simulation and Likelihood-Based Approaches." Risk Analysis 19 (1): 109-
30.
Frey, H. C., and D. S. Rhodes. 1996. "Characterizing, Simulating, and Analyzing Variability and
Uncertainty: An Illustration of Methods Using an Air Toxics Emissions Example." Human and
Ecological Risk Assessment: An International Journal 2 (4): 762-97.
Frey, H. C., and Y. Zhao. 2004. "Quantification of Variability and Uncertainty for Air Toxic Emission
Inventories With Censored Emission Factor Data." Environmental Science & Technology 38 (22):
6094-100.
Landis, W. G. 2005. Regional Scale Ecological Risk Assessment Using the Relative Risk Model. Boca
Raton, FL: CRC Press.
Maginnis, C. M. 2006. A Screening Level Integrated Ecological and Human Health Risk Assessment for
Lake Whatcom, Whatcom County, Washington. Master's Thesis, Western Washington University.
Shlyakhter, A. 1.1994. "An Improved Framework for Uncertainty Analysis: Accounting for
Unsuspected Errors." Risk Analysis 14: 441-47.
Zheng, J., and H. C. Frey. 2004. "Quantification of Variability and Uncertainty Using Mixture
Distributions: Evaluation of Sample Size, Mixing Weights and Separation between Components."
Risk Analysis 24 (3): 553-71.
Zheng, J., and H. C. Frey. 2005. "Quantitative Analysis of Variability and Uncertainty with Known
Measurement Error: Methodology and Case Study." Risk Analysis 25 (3): 663-76.
Sensitivity Analysis
Agro, K. E., C. A. Bradley, and N. Mittmann. 1997. "Sensitivity Analysis in Health Economic and
Pharmacoeconomic Studies—an Appraisal of the Literature." Pharmacoeconomics 11 (1): 75-88.
Cukier, R.I., H.B. Levine, and K.E. Shuler. 1978. "Nonlinear Sensitivity Analysis of Multi-parameter
Model Systems." Journal of Chemical Physics 26(1): 1-42.
Frey, H. C., and R. Patil. 2002. "Identification and Review of Sensitivity Analysis Methods." Risk
Analysis22 [3]: 553-77.
Frey, H. C., A. Mokhtari, and J. Zheng. 2004. Recommended Practice Regarding Selection, Application,
and Interpretation of Sensitivity Analysis Methods Applied to Food Safety Process Risk Models.
Washington, D.C.: U.S. Department of Agriculture. http://www.ce.ncsu.edu/risk/Phase3Final.pdf
Helton, J. C., and F. J. Davis. 2002. "Illustration of Sampling-Based Methods for Sensitivity Analysis."
Reliability Engineering & System Safety 81 (1): 23-69.
50
-------
Mokhtari, A., and H. C. Frey. 2005. "Recommended Practice Regarding Selection of Sensitivity
Analysis Methods Applied to Microbial Food Safety Process Risk Models." Human and Ecological
Risk Assessment 11 (3): 591-605.
Mokhtari, A., H. C. Frey, and J. Zheng. 2006. "Evaluation and Recommendation of Sensitivity Analysis
Methods for Application to EPA-Stochastic Human Exposure and Dose Simulation (SHEDS) Models."
Journal of Exposure Science and Environmental Epidemiology 16 (6): 491-506.
Saltelli, A., K. Chan, and M. Scott 2000. Sensitivity Analysis, Probability and Statistics Series. New
York: John Wiley & Sons.
Sobol, I. M. 1993. "Sensitivity Estimates for Nonlinear Mathematical Models." Mathematical
Modeling and Computation 1 (4): 407-14.
Xue, J., V. G. Zartarian, H. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. 2006. "A
Probabilistic Arsenic Exposure Assessment for Children Who Contact Chromated Copper Arsenate
(CCA)-Treated Playsets and Decks, Part 2: Sensitivity and Uncertainty Analyses." Risk Analysis 26
(2): 533-41.
Case Study Examples of Probabilistic Risk Analysis—EPA (See Also
the PRA Case Studies Appendix)
Babendreier, J. E., and K. L. Castleton. 2005. "Investigating Uncertainty and Sensitivity in Integrated
Multimedia Environmental Models: Tools for FRAMES-3MRA." Environmental Modeling & Software
20:1043-55.
Blancato, J. N., F. W. Power, R. N. Brown, and C. C. Dary. 2006. Exposure Related Dose Estimating
Model (ERDEM): A Physiologically-Based Pharmacokinetic and Pharmacodynamic (PBPK/PD) Model
for Assessing Human Exposure and Risk. EPA/600/R-06/061. Research Triangle Park, NC: USEPA.
http://www.epa.gov/heasd/products/erdem/237edrb05-Reportpdf
Burke, J. 2005. EPA Stochastic Human Exposure and Dose Simulation for Particulate Matter (SHEDS-
PM) User's Guide. Research Triangle Park, NC: USEPA.
Frey, H. C. 2003. Evaluation of an Approximate Analytical Procedure for Calculating Uncertainty in
the Greenhouse Gas Version of the Multi-Scale Motor Vehicle and Equipment Emissions System. Ann
Arbor, MI: USEPA.
Frey, H. C., and Y. Zhao. 2003. Development of Probabilistic Emission Inventories of Benzene,
Formaldehyde and Chromium for the Houston Domain. Research Triangle Park, NC: USEPA.
Frey, H. C., R. Bharvirkar, andj. Zheng. 1999. Quantitative Analysis of Variability and Uncertainty in
Emissions Estimation. Research Triangle Park, NC: USEPA.
Frey, H. C., J. Zheng, Y. Zhao, S. Li, and Y. Zhu. 2002. Technical Documentation oftheAuvTool
Software for Analysis of Variability and Uncertainty. Research Triangle Park, NC: USEPA.
Ozkaynak H., H. C. Frey, J. Burke, and R. W. Finder. 2009. "Analysis of Coupled Model Uncertainties
in Source-to-Dose Modeling of Human Exposures to Ambient Air Pollution: A PM2.s Case-Study."
Atmospheric Environment 43: 1641-49.
Zartarian, V. G., J. Xue, H. A. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. 2005. A
Probabilistic Exposure Assessment for Children Who Contact CCA-Treated Playsets and Decks Using
the Stochastic Human Exposure and Dose Simulation Model for the Wood Preservative Scenario
(SHEDS-WOOD), Final Report. EPA/600/X-05/009. Washington, D.C.: USEPA.
http://www.epa.gov/heasd/sheds/CCA all.pdf
51
-------
Zartarian, V. G., J. Xue, H. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. 2006. "A
Probabilistic Arsenic Exposure Assessment for Children Who Contact CCA-Treated Playsets and
Decks, Part 1: Model Methodology, Variability Results, and Model Evaluation." Risk Analysis 26 (2):
515-32.
Zheng, J., and H. C. Frey. 2002. AuvTool User's Guide. Research Triangle Park, NC: USEPA.
Case Study Examples of Probabilistic Risk Analysis—Other
Bloyd, C., J. Camp, G. Conzelmann, J. Formento, J. Molburg, J. Shannon, M. Henrion, etctl. 1996.
Tracking and Analysis Framework (TAP) Model Documentation and User's Guide. ANL/DIS/TM-36.
Washington, D.C.: Argonne National Laboratory.
Frey, H. C., and S. Li. 2003. "Quantification of Variability and Uncertainty in AP-42 Emission Factors:
Case Studies for Natural Gas-Fueled Engines." Journal of the Air & Waste Management Association 53
(12): 1436-47.
Hanna, S. R., Z. Lu, H. C. Frey, N. Wheeler, J. Vukovich, S. Arunachalam, M. Fernau, and D. A. Hansen.
2001. "Uncertainties in Predicted Ozone Concentrations Due to Input Uncertainties for the UAM-V
Photochemical Grid Model Applied to the July 1995 OTAG Domain." Atmospheric Environment 35
(5): 891-903.
Munns, W. R, Jr., H. A. Walker, and J. F. Paul. 1989. "An Ecological Risk Assessment Framework for
Examining the Impacts of Oceanic Disposal Status and Trends Program." In Oceans '89, Volume 2:
Ocean Pollution, by the Marine Technology Society and the Oceanic Engineering Society (U.S.), 664-
69. IEEE Publication No. 89, CH2780-5. New York: IEEE Press.
Munns, W. R, Jr., H. A. Walker, J. F. Paul, and J. H. Gentile. 1996. "A Prospective Assessment of
Ecological Risks to Upper Water Column Populations from Ocean Disposal at the 106-Mile
Dumpsite."/oun?a/ of Marine Environmental Engineering 3: 279-97.
Nocito, J. A., H. A. Walker, J. F. Paul, and C. Menzie. 1988. "Application of a Risk Assessment for
Marine Disposal of Sewage Sludge at Midshelf and Offshelf Sites." In Proceedings of the 7th
International Ocean Disposal Symposium, 644-63. Ottawa: Environment Canada.
Nocito, J. A., H. A. Walker, J. F. Paul, and C. A. Menzie. 1989. "Application of a Risk Assessment
Framework for Marine Disposal of Sewage Sludge at Midshelf and Offshelf Sites." In G. W. Suter II
and M.A. Lewis (eds.). Aquatic Toxicology and Environmental Fate: Eleventh Volume, 101-20. ASTM
STP 1007. Philadelphia, PA: American Society for Testing and Materials.
Paul, J. F., H. A. Walker, and V. J. Bierman, Jr. 1983. "Probabilistic Approach for the Determination of
the Potential Area of Influence for Waste Disposed at the 106-Mile Ocean Disposal Site." In J. B.
Pearce, D. C. Miller, and C. Berman (eds.). 106-Mile Waste Disposal Site Characterization Update
Report, A-l toA-8. NOAA Technical Memorandum NMFS-F/NEC-26. Woods Hole, MA: National
Marine Fisheries Service.
Paul, J. F., V. J. Bierman, Jr., W. R. Davis, G. L. Hoffman, W. R. Munns, C. E. Pesch, P. F. Rogerson, and S.
C. Schimmel. 1988. "The Application of a Hazard Assessment Research Strategy to the Ocean
Disposal of a Dredged Material: Exposure Assessment Component" In D. A. Wolfe and T. P.
O'Connor (eds.). Oceanic Processes in Marine Pollution, Volume 5,123-35. Melbourne, FL: Kreiger
Publishing Co.
Prager, J. C., V. J. Bierman, Jr., J. F. Paul, and J. S. Bonner. 1986. "Sampling the Oceans for Pollution: A
Risk Assessment Approach to Evaluating Low-Level Radioactive Waste Disposal at Sea." Dangerous
Propertiesof Industrial Materials Report 6 (3): 2-26.
52
-------
Walker, H. A, J. F. Paul, J. A. Nocito, and J. H. Gentile. 1988. "Ecological and Human Health Risks for
Sewage Sludge Disposal at the 106-Mile Site." In Proceedings of the 7th International Ocean Disposal
Symposium, 625-43. Ottawa: Environment Canada.
53
-------
APPENDIX: CASE STUDY EXAMPLES OF THE APPLICATION
OF PROBABILISTIC RISK ANALYSIS IN U.S.
ENVIRONMENTAL PROTECTION AGENCY REGULATORY
DECISION MAKING
54
-------
A. OVERVIEW
This Appendix focuses on examples of how probabilistic risk analysis (PRA) approaches have been
used at EPA to inform regulatory decisions. The Appendix was prepared by representatives from
various EPA program offices and regions currently involved in the development and application of
PRA techniques. The Technical Panel selected the case study examples based on the members'
knowledge of the specific PRA procedures, the types of techniques demonstrated, the availability to
the reader through the Internet and the condition of having been peer reviewed; they also were
selected to be illustrative of a spectrum of PRA used at EPA. The case studies are not designed to
provide an exhaustive discussion of the wide variety of applications of PRA used within the Agency,
but to highlight specific examples reflecting the range of approaches currently applied within EPA.
This Appendix is intended to serve as a resource for managers faced with decisions regarding when
to apply PRA techniques to inform environmental decisions, and for exposure and risk assessors
who may not be familiar with the wide variety of available PRA approaches. The document outlines
categories of PRAs classified by the complexity of analysis to aid the decision-making process. This
approach identifies various PRA tools, which include techniques ranging from a simple sensitivity
analysis (e.g., identification of key exposure parameters or data visualization) requiring limited
time, resources and expertise to develop (Group 1); to probabilistic approaches, including Monte
Carlo analysis, that provide tools for evaluating variability and uncertainty separately and that
require more resources and specialized expertise (Group 2); to sophisticated techniques of expert
elicitation that generally require significant investment of employee time, additional expertise and
external peer review (Group 3).
The case studies in this Appendix used PRA techniques within this ranked framework to provide
additional information for managers. The case study summaries are provided in a format designed
to highlight how the results of the PRAs were considered in decision making. These summaries
include specific information on the conduct of the analyses as an aid in determining what tools
might be appropriate to develop specific exposure or risk assessments for other sites.
The case studies range from examples of less resource-intensive analyses, which might assist in
identifying key exposure parameters or the need for more data, to more detailed and resource-
intensive approaches. Examples of applications in human health and ecological risk assessment
include the exposure of children to chromated copper arsenate (CCA)-treated wood, the relation
between particulates in air and health, dietary exposures to pesticides, modeling sea level change,
sampling watersheds, and modeling bird and animal exposures.
B. INTRODUCTION
Historically, EPA has used deterministic risk assessments, or point estimates of risk, to evaluate
cancer risks and noncancer health hazards to high-end exposed individuals (90th percentile or
higher) and the average exposed individual (50th percentile) and, where appropriate, risks and
hazards to populations, as required by specific environmental laws (USEPA 1992a). The use of
default values for exposure parameters in risk assessments provides a procedural consistency that
allows risk assessments to be feasible and tractable (USEPA 2004). The methods typically used in
EPA deterministic risk assessments (DRA) rely on a combination of point values—some
conservative and some typical—yielding a point estimate of exposure that is at some unknown point
in the range of possible risks (USEPA 2004).
This Appendix presents case studies of PRA conducted by EPA over the past 10 to 15 years.
Table A-1 summarizes the case studies by title, technique demonstrated, classification as a human
health risk assessment (HHRA) or ecological risk assessment (ERA), and the program or regional
55
-------
office responsible for developing the case studies. This Appendix, provides a "snapshot" of the
utilization of PRA across various programs in EPA.
C. OVERALL APPROACH TO PROBABILISTIC RISK
ANALYSIS AT THE U.S. ENVIRONMENTAL PROTECTION
AGENCY
C.1. U.S. Environmental Protection Agency Guidance and Policies on
Probabilistic Risk Analysis
The case studies presented here build on the principles of PRA outlined in EPA's 1997 Policy for Use
of Probabilistic Analysis in Risk Assessment at the U.S. Environmental Protection Agency (USEPA
1997 a) and Guiding Principles for Monte Carlo Analysis (USEPA 1997b), as well as subsequent
guidance documents on developing and using PRA. Guidance has been developed for the Agency
and individual programs. Specific documents that refer to the use of PRA include the Risk
Assessment Guidance for Superfund: Volume III (USEPA 2001); Risk Assessment Forum (RAF)
Framework for Ecological Risk Assessment (USEPA 1992b); Guidelines for Ecological Risk Assessment
(USEPA 1998); Guidance for Risk Characterization (USEPA 1995a); Policy on Evaluating Health Risks
to Children (USEPA 1995b); Policy for Use of Probabilistic Analysis in Risk Assessment (USEPA
1997a); Guidance on Cumulative Risk Assessment, Part 1: Planning and Scoping (USEPA 1997c); and
Risk Characterization Handbook (USEPA 2000a); and Framework for Human Health Risk Assessment
to Inform Decision Making (USEPA 2014).
As shown in the individual case studies, the range and scope of the PRA will depend on the overall
objectives of the decision that the analysis will inform. The Guiding Principles for Monte Carlo
Analysis (USEPA 1997b) lay out the general approach that should be taken in all cases, beginning
with defining the problem and scope of the assessment to selecting the best tools and approach.
The Guiding Principles also describe the process of estimating and characterizing variability and
uncertainty around risk estimates. Stahl and Cimorelli (2005) and the Risk Assessment Guidance for
Superfund: Volume HI (USEPA 2001) highlight the importance of communication between the risk
assessor and manager. Stahl and Cimorelli (2005) and Jamieson (1996) indicate that it is important
to determine whether a particular level of uncertainty is acceptable or not. The authors also suggest
that this decision depends on context, values and regulatory policy. The Risk Assessment Guidance
for Superfund: Volume III (Chapter 2 and Appendix F in USEPA 2001) describes a process for
determining the appropriate level of PRA using a ranked approach from the less resource- and
time-intensive approaches to more sophisticated analyses. Furthermore, the Risk Assessment
Guidance for Superfund: Volume III outlines a process for developing a PRA work plan and a
checklist for PRA reviewers (Chapter 2 and Appendix F in USEPA 2001). This guidance also
provides information regarding how to communicate PRA results to decision makers and
stakeholders (Chapter 6 in USEPA 2001).
C.2. Categorizing Case Studies
The ranked approach used for categorization is a process for a systematic, informed progression to
increasingly complex risk assessment methods of PRA, which is outlined in the Risk Assessment
Guidance for Superfund (USEPA 2001). The use of categories provides a framework for evaluating
the various techniques of PRA. Higher categories reflect increasing complexity and often will
require more time and resources. Higher categories also reflect increasing characterization of
variability and uncertainty in the risk estimate, which may be important for making specific risk
management decisions. Central to the approach is a systematic, informed progression using an
56
-------
iterative process of evaluation, deliberation, data collection, planning and scoping, development,
and updates to the work plan and communication. All of these steps focus on deciding:
1. Whether or not the risk assessment, in its current state (e.g., DRA) is sufficient to support
decisions (i.e., a clear path to exiting the process is available at each step).
2. If the assessment is determined to be insufficient, whether or not progression to a higher
level of complexity (or refinement of the current analyses) would provide a sufficient
benefit to warrant the additional effort of performing a PRA.
This Appendix groups case studies according to level of effort and complexity of the analysis and
the increasing sophistication of the methods used (Table A-l). Although each group generally
represents increasing effort and cost, this may not always be true. The groups also are intended to
reflect the progression from simple to complex analysis that is determined by the interactive
planning and scoping efforts of the risk assessors and managers. The use of particular terms to
describe the groups, including "tiers," was avoided due to specific programmatic and regulatory
connotations.
Group 1 Case Studies
Assessments within this group typically involve a sensitivity analysis and serve as an initial
screening step in the risk assessment Sensitivity analyses identify important parameters in the
assessment where additional investigation may be helpful (Kurowicka and Cooke 2006). Sensitivity
analysis can be simple or involve more complex mathematical and statistical techniques, such as
correlation and regression analysis, to determine which factors in a risk model contribute most to
the variance in the risk estimate.
Within the sensitivity analyses, a range of techniques is available: simple, "back-of-the-envelope"
calculations, where the risk parameters are evaluated using a range of exposure parameters to
determine the parameter that contributes most significantly to the risk (Case Study 1): analyses to
rank the relative contributions of variables to the overall risk (Case Study 2): and data visualization
using graphical techniques to array the data or Monte Carlo simulations (e.g., scatter plots).
More sophisticated analyses may include sensitivity ratios (e.g., elasticity); sensitivity scores (e.g.,
weighted sensitivity ratios); correlation coefficient or coefficient of determination; r2 (e.g., Pearson
product moment, Spearman rank); normalized multiple regression coefficients; and goodness-of-fit
tests for subsets of the risk distribution (USEPA 2001).
The sensitivity analyses typically require minimal resources and time. Results of the sensitivity
analyses are useful in identifying key parameters where additional Group 2 or Group 3 analyses
may be appropriate. Sensitivity analyses also are helpful in identifying key parameters where
additional research will have the highest impact on the risk assessment.
Group 2 Case Studies
Case studies within this group include a more sophisticated application of probabilistic tools,
including PRA of specific exposure parameters (Case Studies 3. and 4), one-dimensional analyses
(Case Study 5) and probabilistic sensitivity analysis (Case Studies 6 and 7).
The Group 2 case studies require larger time commitments for development, specialized expertise
and additional analysis of exposure parameter data sources. Depending on the nature of the
analysis, peer involvement or peer review may be appropriate to evaluate the products of the
analysis.
57
-------
Group 3 Case Studies
Assessments within this group are the most resource- and time-intensive analyses of the three
categories. Risk analyses include two-dimensional Monte Carlo analysis (2-D MCA) that evaluates
model variability and uncertainty (Case Studies 8, 9 and 10]: microexposure event analysis (MEE),
in which long-term exposure of an individual is simulated as the sum of separate short-term
exposure events [Case Study 11): and probabilistic analysis (Case Studies 12 and 13).
Other types of analyses within this group include the expert elicitation method that is a systematic
process of formalizing and quantifying, in terms of probabilities, experts' judgments about
uncertain quantities (Case Studies 14 and 15]: Bayesian statistics, which is a specialized branch of
statistics that views the probability of an event occurring as the degree of belief or confidence in
that occurrence (Case Study 16): and geostatistical analysis, which is another specialized branch of
statistics that explicitly takes into account the geo-referenced context of the data and the
information (e.g., attributes) attached to the data.
The Group 3 analyses require additional time and expertise in the planning and analysis of the
assessment Within this group, the level of expertise and resource commitments may vary with the
techniques. Expert elicitation, for example, requires significantly more time for planning,
identification of experts and meetings, when compared with the other techniques.
58
-------
Table A-l. Case Study Examples of EPA Applications of Deterministic and Probabilistic Risk
Assessment Techniques
Case Study
Number
Title and Case Study Description
Type of Risk
Assessment
Office/Region
Group 1: Point Estimate— Sensitivity Analysis
1
2
Sensitivity Analysis of Key Variables in Probabilistic
Assessment of Children's Exposure to Arsenic in
Chromated Copper Arsenate (CCA) Pressure-Treated
Wood. This case study demonstrates use of a point estimate
sensitivity analysis to identify exposure variables critical to
the analysis summarized in Case Study 9. The sensitivity
analysis identified critical areas for future research and data
collection and better characterized the amount of
dislodgeable residue that exists on the wood surface.
Assessment of the Relative Contribution of Atmospheric
Deposition to Watershed Contamination. An example of a
workbook that demonstrates how "back-of-the-envelope"
analysis of potential exposure rates can be used to target
resources to identify other inputs before further analysis of air
inputs in watershed contamination. Identification of key
variables aided in identifying uncertainties and data gaps to
target resource expenditures for further analysis. A case
study example of the application of this technique also is
identified.
Human Health
Ecological
Office of
Research and
Development
(ORD) and
Office of
Pesticide
Programs
(OPP)
ORD
Group 2: Probabilistic Risk Analysis, One-Dimensional Monte Carlo Analysis (1-D MCA) and
Probabilistic Sensitivity Analysis
Group 2: Probabilistic Risk Analysis
3
4
Probabilistic Assessment of Angling Duration Used in
the Assessment of Exposure to Hudson River Sediments
via Consumption of Contaminated Fish. A probabilistic
analysis of one parameter in an exposure assessment— the
time an individual fishes in a large river system. Development
of site-specific information regarding exposure, with an
existing data set for this geographic area, was needed to
represent this exposed population. This information was used
in the one-dimensional PRA described in Case Study 5.
Probabilistic Analysis of Dietary Exposure to Pesticides
for Use in Setting Tolerance Levels. The probabilistic
Dietary Exposure Evaluation Model (DEEM) provides more
accurate information on the range and probability of possible
exposures.
Human Health
Human Health
Superfund/
Region 2
(New York)
OPP
59
-------
Table A-l. Case Study Examples of EPA Applications of Deterministic and Probabilistic Risk
Assessment Techniques
Case Study
Number
Title and Case Study Description
Type of Risk
Assessment
Office/Region
Group 2: One-Dimensional Monte Carlo Analysis (1-D MCA)
One-Dimensional Probabilistic Risk Analysis of
Exposures to Polychlorinated Biphenyls (PCBs) via
Consumption of Fish From a Contaminated Sediment
Site. An example of a one-dimensional PRA (1-D MCA) of
the variability of exposure as a function of the variability of
individual exposure factors to evaluate the risks to anglers
who consume recreationally caught fish from a PCB-
contaminated river.
Human Health
Superfund/
Region 2
(New York)
Group 2: Probabilistic Sensitivity Analysis
Probabilistic Sensitivity Analysis of Knowledge
Elicitation of the Concentration-Response Relationship
Between PIV^.s Exposure and Mortality. An example of how
the probabilistic analysis tools can be used to conduct a
probabilistic sensitivity analysis following an expert elicitation
(Group 3) presented in Case Study 14.
Human Health
Office of Air
and Radiation
(OAR)
Environmental Monitoring and Assessment Program
(EMAP): Using Probabilistic Sampling Techniques to
Evaluate the Nation's Ecological Resources. A probability-
based sampling program designed to provide unbiased
estimates of the condition of an aquatic resource over a large
geographic area based on a small number of samples.
Ecological
ORD
Group 3: Advanced Probabilistic Risk Analysis— Two-Dimensional Monte Carlo Analysis (2-D MCA)
Including Microexposure Modeling, Bayesian Statistics, Geostatistics and Expert Elicitation
Group 3: Two-Dimensional Probabilistic Risk Analysis
Two-Dimensional Probabilistic Risk Analysis of
Cryptosporidium in Public Water Supplies, With
Bayesian Approaches to Uncertainty Analysis. An
analysis of the variability in the occurrence of
Cryptosporidium in raw water supplies and in the treatment
efficiency, as well as the uncertainty in these inputs. This
case study includes an analysis of the dose-response
relationship for Cryptosporidium infection.
Human Health
Office of
Water (OW)
Two-Dimensional Probabilistic Model of Children's
Exposure to Arsenic in Chromated Copper Arsenate
(CCA) Pressure-Treated Wood. A two-dimensional model
that addresses both variability and uncertainty in the
exposures of children to CCA pressure-treated wood. The
analysis was built on the sensitivity analysis described in
Case Study 2.
Human Health
OPP/ORD
60
-------
Table A-l. Case Study Examples of EPA Applications of Deterministic and Probabilistic Risk
Assessment Techniques
Case Study
Number
10
Title and Case Study Description
Two-Dimensional Probabilistic Exposure Assessment of
Ozone. A probabilistic exposure assessment that addresses
short-term exposures to ozone. Population exposure to
ambient ozone levels was evaluated using EPA's Air
Pollutants Exposure (APEX) model, also referred to as the
Total Risk Integrated Methodology/Exposure (TRIM. Expo)
model.
Type of Risk
Assessment
Human Health
Office/Region
OAR
Group 3: Microexposure Event Analysis
11
Analysis of Microenvironmental Exposures to Fine
Participate Matter (PM2 5) for a Population Living in
Philadelphia, Pennsylvania. A microexposure event
analysis to simulate individual exposures to PIVh.s in specific
microenvironments, including the outdoors, indoor
residences, offices, schools, stores and a vehicle.
Human Health
Region 3
(Philadelphia)
and ORD
Group 3: Probabilistic Analysis
12
13
Probabilistic Analysis in Cumulative Risk Assessment of
Organophosphorus Pesticides. A probabilistic computer
software program used to integrate various pathways, while
simultaneously incorporating the time dimensions of the input
data to calculate margins of exposure.
Probabilistic Ecological Effects Risk Assessment Models
for Evaluating Pesticide Uses. A multimedia exposure/
effects model that evaluates acute mortality levels in generic
or specific avian species over a user-defined exposure
window.
Human Health
Ecological
OPP
OPP
Group 3: Expert Elicitation and Bayesian Belief Network
14
15
Expert Elicitation of Concentration-Response
Relationship Between Fine Particulate Matter (PIVhs)
Exposure and Mortality. A knowledge elicitation used to
derive probabilistic estimates of the uncertainty in one
element of a cost-benefit analysis used to support the PMzs
regulations.
Expert Elicitation of Sea-Level Change Resulting From
Global Climate Change. An example of a PRA that
describes the probability of sea level rise and parameters
that predict sea level change.
Human Health
Ecological
ORD/
OAR
Office of
Policy,
Planning, and
Evaluation
61
-------
Table A-l. Case Study Examples of EPA Applications of Deterministic and Probabilistic Risk
Assessment Techniques
Case Study
Number
16
Title and Case Study Description
Knowledge Elicitation for Bayesian Belief Network Model
of Stream Ecology. An example of a Bayesian belief
network model of the effect of increased fine-sediment load
in a stream on macroinvertebrate populations.
Type of Risk
Assessment
Ecological
Office/Region
ORD
62
-------
D.CASE STUDY SUMMARIES
D.1. Group 1 Case Studies
Case Study 1: Sensitivity Analysis of Key Variables in Probabilistic
Assessment of Children's Exposure to Arsenic in Chromated Copper
Arsenate Pressure-Treated Wood
This case study provides an example of the application of sensitivity analysis to identify important
variables for population exposure variability for a Group 2 assessment [Case Study 9] and to
indicate areas for further research. Specifically, EPA's Office of Research and Development (ORD), in
collaboration with the Office of Pesticide Programs (OPP), used sensitivity analyses to identify the
key variables in children's exposure to CCA-treated wood.
Approach. The sensitivity analyses used two approaches. The first approach estimated baseline
exposure by running the exposure model with each input variable set to its median (50th
percentile) value. Next, alternative exposure estimates were made by setting each input to its 25th
or 75th percentile value while holding all other inputs at their median values. The ratio of the
exposure estimate calculated when an input was estimated at its 25th or 75th percentile to the
exposure estimate calculated when the input was at its median value provided a measure of that
input's importance to the overall exposure assessment. The second approach applied a multiple
stepwise regression analysis to the data points generated from the first approach. The correlation
between the input variables and the exposure estimates provided an alternative measure of the
input variable's relative importance in the exposure assessment These two approaches were used
in tandem to identify the critical inputs to the exposure assessment model.
Results of Analysis. The two sensitivity analyses together identified six critical input variables that
most influenced the exposure assessment. The critical input variables were: wood surface residue-
to-skin transfer efficiency, wood surface residue levels, fraction of hand surface area mouthed per
mouthing event, average fraction of nonresidential outdoor time spent playing on a CCA-treated
playset, frequency of hand washing and frequency of hand-to-mouth activity.
Management Considerations. The results of the sensitivity analyses were used to identify the
most important input parameters in the treated wood risk assessments. The process also identified
critical areas for future research. In particular, the assessment pointed to a need to collect data on
the amount of dislodgeable residue that is transferred from the wood surface to a child's hand upon
contact, and to better characterize the amount of dislodgeable residue that exists on the wood
surface.
Selected References. The final report on the probabilistic exposure assessment of CCA-treated
wood:
Zartarian, V. G., J. Xue, H. A. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. A Probabilistic
Exposure Assessment for Children Who Contact CCA-Treated Playsets and Decks Using the Stochastic
Human Exposure and Dose Simulation Model for the Wood Preservative Scenario (SHEDS-WOOD),
Final Report. EPA/600/X-05/009. Washington, D.C.: USEPA.
See also: Xue, J., V. G. Zartarian, H. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. 2006. "A
Probabilistic Exposure Assessment for Children Who Contact Chromated Copper Arsenate (CCA)-
Treated Playsets and Decks, Part 2: Sensitivity and Uncertainty Analyses." Risk Analysis 26:533-41.
63
-------
Case Study 2: Assessment of the Relative Contribution of
Atmospheric Deposition to Watershed Contamination
Watershed contamination can result from several different sources, including the direct release of
pollution into a water body, input from upstream water bodies and deposition from airborne
sources. Efforts to control water body contamination begin with an analysis of the environmental
sources to identify the parameters that provide the greatest contribution and determine where
mitigation and/or analysis resources should be directed.
Approach. This case study provides an example of a "back-of-the-envelope" deterministic analysis
of the contribution of air deposition to overall watershed nitrogen? Nutrient? contamination to
identify uncertainties and/or data gaps, as well as to target resource expenditures. Nitrogen inputs
have been studied in several east and Gulf Coast estuaries due to concerns about eutrophication.
Nitrogen from atmospheric deposition is estimated to be as high as 10 to 40 percent of the total
input of nitrogen to many of these estuaries, and perhaps higher in a few cases. For a watershed
that has not been studied yet, a back-of-the-envelope calculation could be prepared based on
information about the nitrogen deposition rates measured in a similar area. To estimate the
deposition load directly to the water body, one would multiply the nitrogen deposition rate by the
area of the water body. The analyst then could estimate the nitrogen load from other sources (e.g.,
point source discharges and runoff) to estimate a total nitrogen load for the water body. The
estimate of loading due to atmospheric deposition then could be divided by the total nitrogen load
for the water body to estimate the percent of contribution directly to the water body from
atmospheric deposition.
The May 2003 report by the Casco Bay Air Deposition Study Team titled Estimating Pollutant
Loading From Atmospheric Deposition Using Casco Bay, Maine as a Case Study is an analysis using the
methodology described above. The Casco Bay Estuary, located in southwestern Maine, is used as a
case study. The paper also includes the results of a field air deposition monitoring program
conducted in Casco Bay from 1998 to 2000 and favorably compares the estimates developed for the
rate of deposition of nitrogen, mercury and polycyclic aromatic hydrocarbons (PAHs) to the field
monitoring results. The estimation approach is a useful starting point for understanding the
sources of pollutants entering water bodies that cannot be accounted for through runoff or point
source discharges.
Results of Analysis. The approach outlined above was applied to the Casco Bay Estuary in Maine.
Resources, tools and strategies for pollution abatement can be effectively targeted at priority
sources if estuaries are to be protected. Understanding the sources and annual loading of
contaminants to an estuary facilitates good water quality management by defining the range of
controls of both air and water pollution needed to achieve a desired result The cost of conducting
monitoring to determine atmospheric loading to a water body can be prohibitively high. Also,
collection of monitoring data is a long-term undertaking because a minimum of 3 years of data is
advisable to "smooth out" inter-annual variability. The estimation techniques described in this
paper can serve as a useful and inexpensive "first-cut" at understanding the importance of the
atmosphere as a pollution source and can help to identify areas where field measurements are
needed to guide future management decisions.
Management Considerations. If a review of information on air deposition available for the
analysis indicates a wide range of potential deposition rates, further study of this input would lead
to better characterization of the air contribution to overall contamination. If the back-of-the
envelope analysis suggests that air deposition is very small relative to other inputs, then resources
should be targeted at studying or reducing other inputs before proceeding with further analysis of
the air inputs.
64
-------
Selected References. The back-of-the-envelope calculation is outlined in Frequently Asked
Questions about Atmospheric Deposition: A Handbook for Watershed Managers.
http://www.epa.gov/air/oaqps/gr8water/handbook/airdep septpdf.
Further analysis is available in Deposition of Air Pollutants to the Great Waters—Third Report to
Congress, http://www.epa.gov/air/oaqps/gr8water/3rdrpt/index.html.
The Casco Bay Estuary example is available at http://epa.gov/owow/airdeposition/cascobay.pdf.
D.2. Group 2 Case Studies
Case Study 3: Probabilistic Assessment of Angling Duration Used in
the Assessment of Exposure to Hudson River Sediments via
Consumption of Contaminated Fish
In assessing the health impact of contaminated Superfund sites, exposure duration typically is
assumed to be the same as the length of time that an individual lives in a specific area
(i.e., residence duration). In conducting the HHRA for the Hudson River Polychlorinated Biphenyl
(PCB) Superfund Site, however, there was concern that exposure duration based on residence
duration may underestimate the time spent fishing (i.e., angling duration).
Risk Analysis. An individual may move from one residence to another and continue to fish in the
same location, or an individual may choose to stop fishing irrespective of the location of his or her
home. EPA Region 2 developed a site-specific distribution of angling duration using the fishing
patterns reported in a New York State-wide angling survey (Connelly et al. 1990) and migration
data for the five counties surrounding more than 40 miles of the Upper Hudson River collected as
part of the U.S. Census.
Results of Analysis. The 50th and 95th percentile values from the distribution of angling durations
were higher than the default values based on residence duration using standard default exposure
assumptions for residential scenarios. These values were used as a base for the central tendency
and reasonable maximum exposure point estimates, respectively, in the deterministic assessment.
Management Considerations. The information provided in this analysis was used in the point
estimate analysis. The full distribution was used in conducting a Group 2 PRA for the fish
consumption pathway, which is presented as Case Study 5.
Selected References. The final risk assessment was released in November 2000 and is available at
http://www.epa.gov/hudson/reports.htm.
Further information, including EPA's January 2002 response to comments on the risk assessment,
is available at http://www.epa.gov/hudson/ResponsivenessSummary.pdf.
Case Study 4: Probabilistic Analysis of Dietary Exposure to
Pesticides for Use in Setting Tolerance Levels
Under the Federal Food, Drug, and Cosmetic Act (FFDCA), EPA may authorize a tolerance or
exemption from the requirement of a tolerance to allow a pesticide residue in food, only if the
Agency determines that such residues would be "safe." This determination is made by estimating
exposure to the pesticide and comparing the estimated exposure to a toxicological benchmark dose.
Until 1998, the OPP used a software program called the Dietary Risk Evaluation System (ORES) to
conduct its acute dietary risk assessments for pesticide residues in foods. Acute assessments
conducted with DRES assumed that 100 percent of a given crop with registered use of a pesticide
65
-------
was treated with that pesticide and all such treated crop items contained pesticide residues at the
maximum legal (tolerance) level, matching this to a reasonably high consumption value (around the
95th percentile). The resulting DRES acute risk estimates were considered "high-end" or
"bounding" estimates. It was not possible, however, to know where the pesticide exposure
estimates from the DRES software fit in the overall distribution of exposures due to the limits of the
tools being used.
To address these deficiencies, OPP developed an acute probabilistic dietary exposure guidance to
use a model that would estimate the exposure to pesticides in the food supply. Rather than the
crude "high-end," single-point estimates provided by deterministic assessments, the probabilistic
Dietary Exposure Evaluation Model (DEEM) provides specific information about the range and
probability of possible exposures. Depending on the characterization of the input, this could include
the 95th percentile regulation—generally for lower tiers that do not include the percent of crop
treated—to the 99.9th percentile for the more refined assessments, which would include the
percent of crop treated information.
Probabilistic Analysis. This case study provides an example of a one-dimensional PRA of dietary
exposure to pesticides (Group 2). The DEEM generates acute, probabilistic dietary exposure
assessments using data on: (1) the distribution of daily consumption of specific commodities (e.g.,
wheat, corn and apples) by specific individuals; and (2) the distribution of concentrations of a
specific pesticide in those food commodities. Data on commodity consumption are collected by the
U.S. Department of Agriculture (USDA) in its Continuing Survey of Food Intake by Individuals
(CSFII). Pesticide residue concentrations on food commodities are generally obtained from crop
field trials, USD As Pesticide Data Program (POP), U.S. Food and Drug Administration (FDA)
monitoring data, or market basket surveys. Using these data, DEEM is able to calculate an estimate
of the risk to the general U.S. population, in addition to 26 population subgroups, including 5
subgroups for infants and children (infants less than 1, children 1 to 2, children 3 to 5, youths 6 to
12 and teens 13 to 19 years of age).
Results of Analysis. DEEM has been used in risk assessments to support tolerance levels for
several pesticides (e.g., phosalone) and as part of cumulative risk assessments for
organophosphorus compounds (see Case Study 12) and other pesticides.
Management Considerations. Using the DRES, decisions were being made without a complete
representation of the distribution of risk among the population and without full knowledge of
where in the distribution of risk the DRES risk estimate lay. This was of concern not only for
regulators interested in public health protection, but also for the pesticide registrants who could
argue that the Agency was arbitrarily selecting the level at which to regulate. For most cases
reviewed by OPP to date, estimated exposure at the 99.9th percentile calculated by DEEM
probabilistic techniques is significantly lower than exposure calculated using DRES-type
deterministic assumptions at the unknown percentile.
Selected References. A link to the DEEM model is available at
http://www.epa.gov/pesticides/science/deem/index.html.
Case Study 5: One-Dimensional Probabilistic Risk Analysis of
Exposure to Polychlorinated Biphenyls via Consumption of Fish From
a Contaminated Sediment Site
EPA Region 2 conducted a preliminary deterministic HHRA at the Hudson River PCBs Superfund
site. The DRA demonstrated that consumption of recreationally caught fish provided the highest
66
-------
exposure among relevant exposure pathways, which resulted in cancer risks and noncancer health
hazards that exceeded regulatory benchmarks.
Probabilistic Analysis. Because of the size, complexity and high level of public interest in this site,
EPA Region 2 implemented a Group 2 probabilistic assessment to characterize the variability in
risks associated with the fish consumption exposure pathway. The analysis was a one-dimensional
Monte Carlo analysis (1-D MCA) of the variability of exposure as a function of the variability of
individual exposure factors. Uncertainty was assessed using sensitivity analysis of the input
variables. Data to characterize the distributions of exposure parameters were drawn from the
published literature (e.g., fish consumption rate) or from existing databases, such as the U.S. Census
data (e.g., angling duration, see Case Study 3). Mathematical models of the environmental fate,
transport and bioaccumulation of PCBs in the Hudson River previously developed were used to
forecast changes in PCB concentration over time.
Results of Analysis. The results of the PRA were consistent with the deterministic results. For the
central tendency individual, point estimates were near the median (50th percentile). For the
reasonable maximum exposure (RME) individual, point estimate values were at or above the 95th
percentile of the probabilistic analysis. The DRA and PRA were the subject of a formal peer review
by a panel of independent experts.
The Monte Carlo-based case scenario is the one from which point estimate exposure factors for fish
ingestion were drawn; thus, the point estimate RMEs and the Monte Carlo-based case estimates can
be compared. Similarly, the point estimate central tendency (average) and the Monte Carlo-based
case midpoint (50th percentile) are comparable. For cancer risk, the point estimate RME for fish
ingestion (1 x 1Q-3) falls approximately at the 95th percentile from the Monte Carlo-based case
analysis. The point estimate central tendency value (3 x 1Q-5) and the Monte Carlo-based case 50th
percentile value (6 x 1Q-5) are similar. For noncancer health hazards, the point estimate RME for
fish ingestion (104 for a young child 1 to 6 years of age) falls between the 95th and 99th percentiles
of the Monte Carlo-based case. The point estimate central tendency hazard index (HI; 12 for a
young child) is approximately equal to the 50th percentile of the Monte Carlo-based case HI of 11.
Figures A-1 and A-2 provide a comparison of results from the probabilistic analysis with that of the
DRA for cancer risks and noncancer health hazards. Figures A-1 and A-2 plot percentiles for 72
combinations of exposure variables (e.g., distributions from creel angler surveys' residence
duration, fishing locations and cooking losses) of the noncancer HI values and the cancer risks,
respectively. In each of these figures, the variability of cancer risk or noncancer His for anglers
within the exposed population is plotted on the y-axis for particular percentiles within the
population. This variability is a function of the variations in fish consumption rates, fishing
duration, differences in fish species ingested and so forth. The uncertainty in the estimates is
indicated by the range of either cancer risk or noncancer HI values plotted on the x-axis. This
uncertainty is a function of the 72 combinations of the exposure factor inputs examined in the
sensitivity analysis. This analysis provides a semi-quantitative confidence interval for the cancer
risks and HI values at any particulate percentile. As these figures show, the intervals span
somewhat less than two orders of magnitude (e.g., < 100-fold). The vertical lines indicate the
deterministic endpoints.
Management Considerations. Early and continued involvement of the community improved
public acceptance of the results. In addition, careful consideration of the methods used to present
the probabilistic results to the public lead to greater understanding of the findings.
Selected References. The final risk assessment was released in November 2000 and is available at
http://www.epa.gov/hudson/reports.htm.
67
-------
Further information, including EPA's January 2002 response to comments on the risk assessment,
is available at http://www.epa.gov/hudson/ResponsivenessSun arv.pdf.
100%
Fraction
of Anglers
With
Risk<
Indicated
Value
75% H
50%
25% +
0%
Central Tendency
Value
RME
Value
o.i i 10 100 1000 10000 100000
Estimated Cancers in Population of 1 Million
Figure A-l. Monte Carlo Cancer Summary Based on a One-Dimensional Probabilistic Risk Analysis of
Exposure to Polychlorinated Biphenyls. The estimated cancer rate was calculated based on the
consumption offish from a contaminated sediment site. Source: USEPA 2000b.
Fraction
of Anglers
With
Hl<
Indicated
Value
100% -i
75%
50%
25%
0%
Central
Tendency
Value «
RME Value
0.1 1 10 100 1000 10000
Incremental Individual Hazard Index
Figure A-2. Monte Carlo Noncancer Hazard Index Summary Based on a One-Dimensional Probabilistic
Risk Analysis of Exposure to Polychlorinated Biphenyls. The incremental individual hazard index (HI)
was calculated based on the consumption offish from a contaminated sediment site. Source: USEPA
2000b.
68
-------
Case Study 6: Probabilistic Sensitivity Analysis of Expert Eli citation
of Concentration-Response Relationship Between Fine Particulate
Matter Exposure and Mortality
In 2002, the National Research Council (NRC) recommended that EPA improve its characterization
of uncertainty in the benefits assessment for proposed regulations of air pollutants. NRC
recommended that probability distributions for key sources of uncertainty be developed using
available empirical data or through formal elicitation of expert judgments. In response to this
recommendation, EPA conducted an expert elicitation evaluation of the concentration-response
relationship between fine particulate matter (PIVb.s) exposure and mortality, a key component of
the benefits assessment of the PIVb.s regulation. Further information on the expert elicitation
procedure and results is provided in Case Study 14. To evaluate the degree to which the results of
the assessment depended on the judgments of individual experts or on the methods of expert
elicitation, a probabilistic sensitivity analysis was performed on the results.
Probabilistic Analysis. The expert elicitation procedure used carefully constructed interviews to
elicit from each of 12 experts an estimate of the probabilistic distribution for the average expected
decrease in U.S. annual, adult, all-cause mortality associated with a 1 microgram per cubic meter
(H.g/m3) decrease in annual average PM2.s levels. This case study provides an example of the use of
probabilistic sensitivity analysis (Group 2) as one element of the overall assessment. For the
sensitivity analysis, a simplified health benefits analysis was conducted to assess the sensitivity of
the results to the responses of individual experts and to three factors in the study design: (1) the
use of parametric or nonparametric approaches by experts to characterize their uncertainty in the
PM2.5 mortality coefficient; (2) participation in the Pre-Elicitation Workshop; and (3) allowing
experts to change their judgments after the Post-Elicitation Workshop. The individual quantitative
expert judgments were used to estimate a distribution of benefits, in the form of the number of
deaths avoided, associated with a reduction in ambient, annual average PM2.s concentrations from
12 to 11 |ig/m3. The 12 individual distributions of estimated avoided deaths were pooled using
equal weights to create a single overall distribution reflecting input from each expert. This
distribution served as the baseline for the sensitivity analysis, which compared the means and
standard deviations of the baseline distribution with several variants.
Results of Analysis. The first analysis examined the sensitivity of the mean and standard deviation
of the overall mortality distribution to the removal of individual experts' distributions. In general,
the results suggested a fairly equal division between those experts whose removal shifted the
distribution mean up and those who shifted it down. There were relatively modest impacts of
individual experts. The standard deviation of the combined distribution also was not affected
strongly by the removal of individual experts. The second analysis evaluated whether the use of
parametric or nonparametric approaches affected the overall results. The results suggested that the
use of parametric distributions led to distributions with similar or slightly increased uncertainty
compared with distributions provided by experts who offered percentiles of a nonparametric
distribution. The last analysis evaluated whether participation in the Pre- or Post-Elicitation
Workshops affected the results. Participation in either workshop did not appear to have a
significant effect on experts' judgments based on measures of change in the baseline distribution.
Overall, the sensitivity analyses demonstrated that the assessment was robust, with little
dependence on individual experts' judgments or on the specific elicitation methods evaluated.
Management Considerations. The sensitivity analysis demonstrated the robustness of the PM2.s
expert elicitation-based assessment by showing that the panel of experts was generally well
balanced and that alternative elicitation methods would not have markedly altered the overall
results.
69
-------
Selected References. The details of this analysis are provided in the Industrial Economics, Inc.,
document titled: Expanded Expert Judgment Assessment of the Concentration-Response Relationship
Between PM2.s Exposure and Mortality, Final Report, September 21, 2006. This document is available
at http://www.epa.gov/ttn/ecas/regdata/Uncertainty/pm ee reportpdf.
The expert elicitation assessment, along with the Regulatory Impact Analysis (RIA) of the PM2.s
standard, is available at http://www.epa.gov/ttn/ecas/ria.html.
Case Study 7: Environmental Monitoring and Assessment Program:
Using Probabilistic Sampling to Evaluate the Condition of the Nation's
Aquatic Resources
Monitoring is a key tool used to identify the locations where the environment is in a healthy
biological condition and requires protection, and where environmental problems are occurring and
need remediation. Most monitoring, however, is not performed in a way that allows for statistically
valid assessments of water quality conditions in unmonitored waters (USGAO 2000). States thus
cannot adequately measure the status and trends in water quality as required by the Clean Water
Act (CWA) Section 305(b).
The Environmental Monitoring and Assessment Program's (EMAP) focus has been to develop
unbiased statistical survey design frameworks and sensitive indicators that allow the condition of
aquatic ecosystems to be assessed at state, regional and national scales. A cornerstone of EMAP has
been the use of probabilistic sampling to allow representative, unbiased, cost-effective condition
assessments for aquatic resources covering large areas. EMAP's statistical survey methods are very
efficient, requiring relatively few sample locations to make valid scientific statements about the
condition of aquatic resources over large areas (e.g., the condition of all of the wadeable streams in
the western United States).
Probabilistic Analysis. This research program provides multiple case studies using probabilistic
sampling designs for different aquatic resources (estuaries, streams and rivers). An EMAP
probability-based sampling program delivers an unbiased estimate of the condition of an aquatic
resource over a large geographic area from a small number of samples. The principal
characteristics of a probabilistic sampling design are: the population being sampled is
unambiguously described; every element in the population has the opportunity to be sampled with
a known probability; and sample selection is conducted by a random process. This approach allows
statistical confidence levels to be placed on the estimates and provides the potential to detect
statistically significant changes and trends in condition with repeated sampling. In addition, this
approach permits the aggregation of data collected from smaller areas to predict the condition of a
large geographic area.
The EMAP design framework allows the selection of unbiased, representative sampling sites and
specifies the information to be collected at these sites. The validity of the overall inference rests on
the design and subsequent analysis to produce regionally representative information. The EMAP
uses the approach outlined in the EPA's Generalized Random Tessellation Stratified Spatially-
Balanced Survey Designs for Aquatic Resources (Olsen 2012). The spatially balanced aspect spreads
out the sampling locations geographically, but still ensures that each element has an equal chance
of being selected.
Results of Analysis. Data collected using the EMAP approach has allowed the Agency to make
scientifically defensible assessments of the ecological condition of large geographic areas for
reporting to Congress under CWA 305(b). The EMAP approach has been used to provide the first
reports on the condition of the nation's estuaries, streams, rivers and lakes, and it is scheduled to be
70
-------
used for wetlands. EMAP findings have been included in EPA's Report on the Environment and the
Heinz Center's The State of the Nation's Ecosystems. Data collected through an EMAP approach
improve the ability to assess ecological progress in environmental protection and restoration, and
provide valuable information for decision makers and the public. The use of probabilistic analysis
methods allows meaningful assessment and regional comparisons of aquatic ecosystem conditions
across the United States. Finally, the probabilistic approach provides scientific credibility for the
monitoring network and aids in identifying data gaps.
Management Considerations. Use of an EMAP approach addresses criticisms from the
Government Accountability Office (GAO), the National Academy of Sciences (NAS), the Heinz Center
(a nonprofit environmental policy institution), and others that noted the nation lacked the data to
make scientifically valid characterizations of water quality regionally and across the United States.
The program provides cost-effective, scientifically defensible and representative data, to permit the
evaluation of quantifiable trends in ecosystem condition, to support performance-based
management and facilitate better public decisions regarding ecosystem management EMAP's
approach now has been adopted by EPA's Office of Water (OW) to collect data on the condition of
all the nation's aquatic resources. OW, Office of Air and Radiation (OAR) and Office of Chemical
Safety and Pollution Prevention (OCSPP; formerly the Office of Prevention, Pesticides, and Toxic
Substances) now have environmental accountability endpoints using EMAP approaches in their
Agency performance goals.
Selected References. General information concerning EMAP is available at
http://www.epa.gov/emap/index.html.
Information on EMAP monitoring designs is available at
http://www.epa.gov/nheerl/arm/designpages/monitdesign/monitoring design info.htm.
EPA's Generalized Random Tessellation Stratified Spatially-Balanced Survey Designs for Aquatic
Resources document is available at
http://www.epa.gov/nheerl/arm/documents/presents/grts ss.pdf.
USGAO (U.S. Government Accountability Office). 2000. Water Quality: Key EPA and State Decisions
Limited by Inconsistent and Incomplete Data. GAO/RCED-00-54. Washington, D.C.: USGAO.
http://www.environmental-auditing.Org/Portals/0/AuditFiles/usengOOar ft key epa.pdf.
USEPA (U.S. Environmental Protection Agency). 2002. EMAP Research Strategy. Research Triangle
Park, NC: Environmental Monitoring and Assessment Program, National Health and Environmental
Effects Research Laboratory (NHEERL), USEPA.
http://www.epa.gov/nheerl/emap/files/emap research strategy.pdf.
D.3. Group 3 Case Studies
Case Study 8: Two-Dimensional Probabilistic Risk Analysis of
Cryptosporidium in Public Water Supplies, With Bayesian
Approaches to Uncertainty Analysis
Probabilistic assessment of the occurrence and health effects associated with Cryptosporidium
bacteria in public drinking water supplies was used to support the economic analysis of the final
Long-Term 2 Enhanced Surface Water Treatment Rule (LT2). EPA's Office of Ground Water and
Drinking Water (OGWDW) conducted this analysis and established a baseline disease burden
attributable to Cryptosporidium in public water supplies that use surface water sources. Next, it
modeled source water monitoring and finished water improvements that will be realized as a result
71
-------
of the LT2. Post-Rule risk is estimated and the LT2's health benefit is the result of subtracting this
from the baseline disease burden.
Probabilistic Analysis. Probabilistic assessment was used for this analysis as a means of
addressing the variability in the occurrence of Cryptosporidium in raw water supplies, the
variability in the treatment efficiency, and the uncertainty in these inputs and in the dose-response
relationship for Cryptosporidium infection. This case study provides an example of a PRA that
evaluates both variability and uncertainty at the same time and is referred to as a two-dimensional
PRA. The analysis also included probabilistic treatments of uncertain dose-response and
occurrence parameters. Markov Chain Monte Carlo samples of parameter sets filled this function.
This Bayesian approach (treating the unknown parameters as random variables) differs from
classical treatments, which would regard the parameters as unknown, but fixed (Group 3:
Advanced PRA). The risk analysis used existing datasets (e.g., the occurrence of Cryptosporidium
and treatment efficacy) to inform the variability of these inputs. Uncertainty distributions were
characterized based on professional judgment or by analyzing data using Bayesian statistical
techniques.
Results of Analysis. The risk analysis identified the Cryptosporidium dose-response relationship as
the most critical model parameters in the assessment, followed by the occurrence of the pathogen
and treatment efficiency. By simulating implementation of the Rule using imprecise, biased
measurement methods, the assessment provided estimates of the number of public water supply
systems that would require corrective action and the nature of the actions likely to be implemented.
This information afforded a realistic measure of the benefits (in reduced disease burden) expected
with the LT2. In response to Science Advisory Board (SAB) comments, additional Cryptosporidium
dose-response models were added to more fully reflect uncertainty in this element of the
assessment.
Management Considerations. The LT2 underwent external peer review, review by EPA's SAB and
intense review by the Office of Management and Budget (OMB). Occurrence and dose-response
components of the risk analysis model were communicated to stakeholders during the Rule's
Federal Advisory Committee Act (FACA) process. Due to the rigor of the analysis and the signed
FACA "Agreement in Principle," the OMB review was straightforward.
Selected References. The final assessment of occurrence and exposure to Cryptosporidium was
released in December 2005 and is available at
http://www.epa.gov/safewater/disinfection/lt2/regulations.html.
Case Study 9: Two-Dimensional Probabilistic Model of Children's
Exposure to Arsenic in Chromated Copper Arsenate Pressure-Treated
Wood
Probabilistic models were developed in response to EPA's October 2001 Federal Insecticide,
Fungicide, and Rodenticide Act (FIFRA) Scientific Advisory Panel (SAP) recommendations to use
probabilistic modeling to estimate children's exposures to arsenic in CCA-treated playsets and
home decks.
Probabilistic Analysis. EPA's ORD, in collaboration with the Office of Pesticide Programs (OPP),
developed and applied a probabilistic exposure assessment of children's exposure to arsenic and
chromium from contact with CCA-treated wood playsets and decks. This case study provides an
example of the use of two-dimensional (i.e., addressing both variability and uncertainty)
probabilistic exposure assessment (Group 3: Advanced PRA). The two-dimensional assessment
employed a modification of ORD's Stochastic Human Exposure and Dose Simulation (SHEDS) model
72
-------
to simulate children's exposure to arsenic and chromium from CCA-treated wood playsets and
decks, as well as the surrounding soil. Staff from both ORD and OPP collaborated in the
development of the SHEDS-Wood model.
Results of Analysis. A draft of the probabilistic exposure assessment received SAP review in
December 2003; the final report was released in 2005. The results of the probabilistic exposure
assessment were consistent with or in the range of the results of deterministic exposure
assessments conducted by several other organizations. The model results were used to compare
exposures under a variety of scenarios, including cold versus warm weather activity patterns, use
of wood sealants to reduce the availability of contaminants on the surface of the wood, and different
hand-washing frequencies. The modeling of alternative mitigation scenarios indicated that the use
of sealants could result in the greatest exposure reduction, while increased frequency of hand
washing also could reduce exposure.
OPP used the SHEDS-Wood model exposure results in its probabilistic children's risk assessment
for CCA (USEPA 2008). This included recommendations for risk reduction (use of sealants and
careful attention to children's hand washing) to homeowners with existing CCA wood structures. In
addition, the exposure assessment was used to identify areas for further research, including: the
efficacy of wood sealants in reducing dislodgeable contaminant residues, the frequency with which
children play on or around CCA wood, and transfer efficiency and residue concentrations. To better
characterize the efficacy of sealants in reducing exposure, EPA and the Consumer Product Safety
Commission (CPSC) conducted a 2-year study of how dislodgeable contaminant residue levels
changed with the use of various types of commercially available wood sealants.
Management Considerations. The OPP used SHEDS results directly in its final risk assessment for
children playing on CCA-treated playground equipment and decks. The model enhanced risk
assessment and management decisions and prioritized data needs related to wood preservatives.
The modeling product was pivotal in the risk management and re-registration eligibility decisions
for CCA, and in advising the public how to minimize health risks from existing treated wood
structures. Industry also is using SHEDS to estimate exposures to CCA and other wood
preservatives. Some states are using the risk assessment as guidance in setting their regulations for
CCA-related playground equipment.
Selected References. The final probabilistic risk assessment based on the SHEDS-Wood exposure
assessment is available at
http://www.epa.gov/oppad001/reregistration/cca/final cca factsheethtm.
The model results were included in the final report on the probabilistic exposure assessment of
CCA-treated wood surfaces: Zartarian, V.G., J. Xue, H. A. Ozkaynak, W. Dang, G. Glen, L. Smith, and C.
Stallings. 2006. A Probabilistic Exposure Assessment for Children Who Contact CCA-Treated Playsets
and Decks Using the Stochastic Human Exposure and Dose Simulation Model for the Wood
Preservative Scenario (SHEDS-Wood), Final Report. EPA/600/X-05/009. Washington, D.C.: USEPA.
Results of the sealant studies were released in January 2007 and are available at
http://www.epa.gov/oppad001/reregistration/cca/index.htmtfreviews.
The results of the analysis were published as: Zartarian, V.G., J. Xue, H. Ozkaynak, W. Dang, G. Glen,
L. Smith, and C. Stallings. 2006. "A Probabilistic Arsenic Exposure Assessment for Children who
Contact CAA-Treated Playsets and Decks, Part 1: Model Methodology, Variability Results, and Model
Evaluation." Risk Analysis 26: 515-31.
More information on the analysis can be found by consulting the following resource:
73
-------
USEPA (U.S. Environmental Protection Agency). 2008. Case Study Examples of the Application of
Probabilistic Risk Analysis in U.S. Environmental Protection Agency Regulatory Decision-Making (In
Review). Washington, D.C.: Risk Assessment Forum, USEPA
Case Study 10: Two-Dimensional Probabilistic Exposure Assessment
of Ozone
As part of EPA's recent review of the ozone National Ambient Air Quality Standards (NAAQS), the
Office of Air Quality Planning and Standards (OAQPS) conducted detailed probabilistic exposure
and risk assessments to evaluate potential alternative standards for ozone. At different stages of
this review, the Clean Air Scientific Advisory Committee (CASAC) Ozone Panel (an independent
scientific review committee of EPA's SAB) and the public reviewed and provided comments on
analyses and documents prepared by EPA. A scope and methods plan for the exposure and risk
assessments was developed in 2005 (USEPA 2005). This plan was intended to facilitate
consultation with the CASAC, as well as public review, and to obtain advice on the overall scope,
approaches and key issues in advance of the completion of the analyses. This case study describes
the probabilistic exposure assessment, which addresses short-term exposures to ozone. The
exposure estimates were used as an input to the HHRA for lung function decrements in all children
and asthmatic school-aged children based on exposure-response relationships derived from
controlled human exposure studies.
Probabilistic Analysis. Population exposure to ambient ozone levels was evaluated using EPA's
Air Pollutants Exposure (APEX) model, also referred to as the Total Risk Integrated
Methodology/Exposure (TRIM.Expo) model. Exposure estimates were developed for recent ozone
levels, based on 2002 to 2004 air quality data, and for ozone levels simulated to just meet the
existing 0.08 ppm, 8-hour ozone NAAQS and several alternative ozone standards, based on
adjusting the 2002 to 2004 air quality data. Exposure estimates were modeled for 12 urban areas
located throughout the United States for the general population, all school-age children and
asthmatic school-age children. This exposure assessment is described in a technical report
(USEPA 2007b). The exposure model APEX is documented in a user's guide and technical document
(USEPA 2006). A Monte Carlo approach was used to produce quantitative estimates of the
uncertainty in the APEX model results, based on estimates of the uncertainties for the most
important model inputs. The quantification of model input uncertainties, a discussion of model
structure uncertainties, and the results of the uncertainty analysis are documented in
Langstaff(2007).
Results of Analysis. Uncertainty in the APEX model predictions results from uncertainties in the
spatial interpolation of measured concentrations, the microenvironment models and parameters,
people's activity patterns, and to a lesser extent, model structure. The predominant sources of
uncertainty appear to be the human activity pattern information and the spatial interpolation of
ambient concentrations from monitoring sites to other locations. The primary policy-relevant
finding was that the uncertainty in the exposure assessment is small enough to lend confidence to
the use of the model results for the purpose of informing the Administrator's decision on the ozone
standard.
Figure A-3 illustrates the uncertainty distribution for one model result, the percent of children with
exposures above 0.08 ppm, 8-hour while at moderate exertion. The point estimate of 20 percent is
the result from the APEX simulation using the best estimates of the model inputs. The
corresponding result from the Monte Carlo simulations ranges from 17 to 26 percent, with a
95 percent uncertainty interval (UI) of 19 to 24 percent. Note that the UIs are not symmetric
because the distributions are skewed.
74
-------
Management Considerations. The exposure analysis also provided information on the frequency
with which population exposures exceeded several potential health effect benchmark levels that
were identified based on the evaluation of health effects in clinical studies.
The exposure and risk assessments are summarized in Chapters 4 and 5, respectively, of the Ozone
Staff Paper (USEPA 2007a). The exposure estimates over these potential health effect benchmarks
were part of the basis for the Administrator's March 27, 2008, decision to revise the ozone NAAQS
from 0.08 to 0.075 ppm, 8-hour average (see the final rule for the ozone NAAQS1).
450
400
~ 350 -
JS
I 300
'in
| 250 ^
O
S 200
o
S 150
o
£
3
100 -
50 -
0
17%
18% 19% 20% 21% 22%
Percent of children
23%
24%
25%
26%
Figure A-3. Uncertainty Distribution Model Results. The estimated percentage of children with 8-hour
exposures above 0.08 ppm at moderate exertion (the point estimate is 20%).
Selected References. More information on the analysis can be found by consulting the following
resources:
Langstaff, J. E. 2007. Analysis of Uncertainty in Ozone Population Exposure Modeling. Office of Air
Quality Planning and Standards Staff Memorandum to Ozone NAAQS Review Docket OAR-2005-
0172. http://www.epa.gOV/ttn/naaqs/standards/ozone/s ozone cr td.html
USEPA (U.S. Environmental Protection Agency). 2005. Ozone Health Assessment Plan: Scope and
Methods for Exposure Analysis and Risk Assessment. Research Triangle Park, NC: Office of Air Quality
Planning and Standards, USEPA. http://www.epa.gOV/ttn/naaqs/standards/ozone/s o3 cr pd.html
USEPA. 2006. Total Risk Integrated Methodology (TRIM)—Air Pollutants Exposure Model
Documentation (TRIM.Expo/APEX, Version 4) Volume I: User's Guide; Volume II: Technical Support
Document Research Triangle Park, NC: Office of Air Quality Planning and Standards, USEPA. June
2006. http://www.epa.gov/ttn/fera/human apex.html
USEPA. 2007a. Review of National Ambient Air Quality Standards for Ozone: Policy Assessment of
Scientific and Technical Information—OAQPS Staff Paper. Research Triangle Park, NC: Office of Air
Quality Planning and Standards, USEPA.
http://www.epa.gOV/ttn/naaas/standards/ozone/s ozone cr sp.html
National Ambient Air Quality Standards for Ozone, Final Rule. 73 Fed. Reg. 16436 (Mar. 27, 2008).
75
-------
USEPA. 2007b. Ozone Population Exposure Analysis for Selected Urban Areas. Research Triangle
Park, NC: Office of Air Quality Planning and Standards, USEPA.
http://www.epa.gOV/ttn/naaqs/standards/ozone/s ozone cr td.html
Case Study 11: Analysis of Microenvironmental Exposures to Fine
Particulate Matter for a Population Living in Philadelphia,
Pennsylvania
This case study used the Stochastic Human Exposure and Dose Simulation model for Particulate
Matter (SHEDS-PM) developed by EPA's National Exposure Research Laboratory (NERL) to prepare
a probabilistic assessment of population exposure to PM2.s in Philadelphia, Pennsylvania. This case
study simulation was prepared to accomplish three goals: (1) estimate the contribution of PM2.s of
ambient (outdoor) origin to total PM2.5 exposure; (2) determine the major factors that influence
personal exposure to PM2.s; and (3) identify factors that contribute the greatest uncertainty to
model predictions.
Probabilistic Analysis. The two-dimensional probabilistic assessment used a microexposure event
technique to simulate individual exposures to PM2.5 in specific microenvironments (outdoors,
indoor residence, office, school, store, restaurant or bar, and in a vehicle). The population for the
simulation was generated using demographic data at the census-tract level from the U.S. Census.
Characteristics of the simulated individuals were selected randomly to match the demographic
proportions within the census tract for gender, age, employment status and housing type. The
assessment used spatially and temporally interpolated ambient PM2.s measurements from 1992 to
1993 and 1990 U.S. Census data for each census tract in Philadelphia. This case study provides an
example of both two-dimensional (variability and uncertainty) probabilistic assessment and
microexposure event assessment (Group 3: Advanced PRA).
Results of Analysis. Results of the analysis showed that human activity patterns did not have as
strong an influence on ambient PM2.s exposures as was observed for exposure to indoor PM2.s
sources. Exposure to PM2.5 of ambient origin contributed a significant percent of the daily total PM2.5
exposures, especially for the segment of the population without exposure to environmental tobacco
smoke in the residence. Development of the SHEDS-PM model using the Philadelphia PM2.s case
study also provided useful insights into data needs for improving inputs into the SHEDS-PM model,
reducing uncertainty and further refinement of the model structure. Some of the important data
needs identified from the application of the model include: daily PM2.s measurements over multiple
seasons and across multiple sites within an urban area, improved capability of dispersion models to
predict ambient PM2.s concentrations at greater spatial resolution and over a 1-year time period,
measurement studies to better characterize the physical factors governing infiltration of ambient
PM2.s into residential microenvironments, further information on particle-generating sources
within the residence, and data for the other indoor microenvironments not specified in the model.
Management Considerations. The application of the SHEDS-PM model to the Philadelphia
population gave insights into data needs and areas for model refinement. The continued
development and evaluation of the SHEDS-PM population exposure model are being conducted as
part of ORD's effort to develop a source-to-dose modeling system for PM and air toxics. This type of
exposure and dose modeling system is considered to be important for the scientific and policy
evaluation of the critical pathways, as well as the exposure factors and source types influencing
human exposures to a variety of environmental pollutants, including PM.
76
-------
Selected References. The results of the analysis were published in:
Burke,}., M. Zufall, and H. Ozkaynak. 2001. "A Population Exposure Model for Particulate Matter:
Case Study Results for PM2.5 in Philadelphia, PA" Journal of Exposure Analysis and Environmental
Epidemiology 11 (6): 470-89.
Georgepoulos, P. G., S. W. Wang, V. M. Vyas, Q. Sun, J. Burke, R. Vedantham, T. McCurdy, and H.
Ozkaynak. 2005. "A Source-to-Dose Assessment of Population Exposure to Fine PM and Ozone in
Philadelphia, PA, During a Summer 1999 Episode." Journal of Exposure Analysis and Environmental
Epidemiology 15 (5): 439-57.
Case Study 12: Probabilistic Analysis in Cumulative Risk Assessment
of Organophosphorus Pesticides
In 1996, Congress enacted the Food Quality Protection Act (FQPA), which requires EPA to consider
"available evidence concerning the cumulative effects on infants and children of such residues and
other substances that have a common mechanism of toxicity" when setting pesticide tolerances (i.e.,
the maximum amount of pesticide residue legally allowed to remain on food products). FQPA also
mandated that EPA completely reassess the safety of all existing pesticide tolerances (those in effect
since August 1996) to ensure that they are supported by current scientific data and meet current
safety standards. Because organophosphorus pesticides (OPs) were assigned priority for tolerance
reassessment, these pesticides were the first "common mechanism" group identified by EPA's OPP.
The ultimate goal associated with this cumulative risk assessment (CRA) was to provide a basis for
the decision maker to establish safe tolerance levels for this group of pesticides, while meeting the
FQPA standard for protecting infants and children.
Probabilistic Analysis. This case study provides an example of an advanced probabilistic
assessment (Group 3). In 2006, EPA analyzed exposures to 30 OPs through food consumption,
drinking water intake, and exposure via pesticide application. Distributions of human exposure
factors, such as breathing rates, body weight and surface areas used in the assessment, came from
the Agency's Exposure Factors Handbook (USEPA 1997d). EPA used Calendex, a probabilistic
computer software program (available at http://www.epa.gov/pesticides/science/deem/) to
integrate various pathways, while simultaneously incorporating the time dimensions of the input
data. Based on the results of the exposure assessment, EPA calculated margins of exposure (MOEs)
for the total cumulative risk from all pathways for each age group (infant less than 1; children 1-2,
3-5, 6-12; youth 13-19; and adults 20-49 and 50+ years of age).
The food component of the OPs CRA was highly refined, as it was based on residue monitoring data
from the USDA's POP and supplemented with information from the FDA's Surveillance Monitoring
Programs and Total Diet Study. The residue data were combined with actual consumption data
from USDA's Continuing Survey of Food Intakes by Individuals (CSFII) using probabilistic
techniques. The CRA evaluated drinking water exposures on a regional basis. The assessment
focused on areas where combined OP exposure is likely to be highest within each region. Primarily,
the analysis used probabilistic modeling to estimate the co-occurrence of OP residues in water.
Monitoring data were not available with enough consistency to be the sole basis for the assessment;
however, they were used to corroborate the modeling results. Data sources for the water
component of the assessment included USD A Agricultural Usage Reports for Field Crops, Fruits and
Vegetables; USDA Typical Planting and Harvesting Dates for Field Crops and Fresh Market and
Processing Vegetables; local sources for refinements; and monitoring studies from the U.S.
Geological Survey (USGS) and other sources. Finally, exposure via the oral, dermal and inhalation
routes resulting from applications of OPs in and around homes, schools, offices and other public
areas were assessed probabilistically for each of the seven regions. The data sources for this part of
77
-------
the assessment included information from surveys and task forces, special studies and reports from
published scientific literature, EPA's Exposure Factors Handbook (USEPA 1997d), and other sources.
Results of Analysis. The OPs CRA presented potential risk from single-day (acute) exposures
across 1 year and from a series of 21-day rolling averages across the year. MOEs at the 99.9th
percentile were approximately 100 or greater for all populations for the 21-day average results.
The only exception is a brief period (roughly 2 weeks) in which drinking water exposures
(identified from the Exposure Factors Handbook, USEPA 1997d) attributed to phorate use on
sugarcane resulted in MOEs near 80 for children ages 1 to 2 years. Generally, exposures through the
food pathway dominated total MOEs, and exposures through drinking water were substantially
lower throughout most of the year. Residential exposures were substantially smaller than
exposures through both food and drinking water.
The OPs CRA was very resource intensive. Work began in 1997 with the preparation of guidance
documents and the development of a CRA methodology. Over 2 to 3 years, more than 25 people
spent 50 to 100 percent of their time working on the assessment, with up to 50 people working on
the CRA at critical periods. EPA has spent approximately $1 million on this assessment (e.g., for
computers, models and contractor support).
Management Considerations. The OPs CRA was a landmark demonstration of the feasibility of a
regulatory-level assessment of the risk from multiple chemicals. Upon completion, EPA presented
the CRA at numerous public technical briefings and FIFRA SAP meetings, and made all of the data
inputs available to the public. The OPP's substantial effort to communicate methodologies,
approaches and results to the stakeholders aided in the success of the OPs CRA. The stakeholders
expressed appreciation for the transparent nature of the OPs CRA and recognized the innovation
and hard work that went into developing the assessments.
Selected References. The 2006 assessment and related documents are available at
http://www.epa.gov/pesticides/cumulative/common mech groups.htm#op.
USEPA (U.S. Environmental Protection Agency). 1997d. Exposure Factors Handbook. Washington,
D.C.: National Center for Environmental Assessment, USEPA.
http://cfpub.epa. gov/ncea/cfm/recordisplay.cfm?deid=12464.
Case Study 13: Probabilistic Ecological Effects Risk Assessment
Models for Evaluating Pesticide Use
As part of the process of developing and implementing a probabilistic approach for ERA, an
illustrative case was completed in 1996. This case involved both DRA and PRA for the effects of a
hypothetical chemical X (ChemX) on birds and aquatic species. Following the recommendations of
the SAP and in response to issues raised by OPP risk managers, the Agency began an initiative to
refine the ERA process for evaluating the effects of pesticides to terrestrial and aquatic species
within the context of FIFRA, the main statutory authority for regulating pesticides at the federal
level. The key goals and objectives of EPA's initiative were to:
n Incorporate probabilistic tools and methods to provide an estimate on the magnitude and
probability of effects.
n Build on existing data requirements for registration.
n Utilize, wherever possible, existing databases and create new ones from existing data
sources to minimize the need to generate additional data.
n Focus additional data requirements on reducing uncertainty in key areas.
78
-------
After proposing a four-level risk assessment scheme, with higher levels reflecting more refined risk
assessment techniques, the Agency developed pilot models for both terrestrial and aquatic species.
Refined risk assessment models (Level II) then were developed and used in a generic chemical case
study that was presented to the SAP in 2001.
Probabilistic Analysis. This case study describes an advanced probabilistic model for the
ecological effects of pesticides (Group 3). The terrestrial Level II model (version 2.0) is a
multimedia exposure/effects model that evaluates acute mortality levels in generic or specific avian
species over a user-defined exposure window. The spatial scale is at the field level, which includes
the field and surrounding area. Both areas are assumed to meet the habitat requirements for each
species, and contamination of edge or adjacent habitat from drift is assumed to be zero. For each
individual bird considered in a run of the Level II model, a random selection of values is made for
the major exposure input parameters to estimate an external oral dose equivalent for that
individual. The estimated dose equivalent is compared to a randomly assigned tolerance for the
individual preselected from the dose-response distribution. If the dose is greater than the tolerance,
the individual is scored "dead," if not, the individual is scored "not dead." After multiple iterations of
individuals, a probability density function of percent mortality is generated.
From May 29 to 31,1996, the Agency presented two ERA case studies to the SAP for review and
comment Although recognizing and generally reaffirming the utility of EPA's current deterministic
assessment process, the SAP offered a number of suggestions for improvement Foremost among
their suggestions was a recommendation to move beyond the existing deterministic assessment
approach by developing the tools and methodologies necessary to conduct a probabilistic
assessment of effects. Such an assessment would estimate the magnitude and probability of the
expected impact and define the level of certainty and variation involved in the estimate; risk
managers within EPA's OPP also had requested this information in the past
The aquatic Level II model is a two-dimensional Monte Carlo risk model consisting of three main
components: (1) exposure, (2) effects and (3) risk. The exposure scenarios used at Level II are
intended to provide estimates for vulnerable aquatic habitats across a wide range of geographical
conditions under which a pesticide product is used. The Level II risk evaluation process yields
estimates of likelihood and magnitude of effects for acute exposures. For the estimate of acute risks,
a distribution of estimated exposure and a distribution of lethal effects are combined through a 2-D
MCA to obtain a distribution of joint probability functions. For the estimate of chronic risks, a
distribution of exposure concentrations is compared to a chronic measurement endpoint The risk
analysis for chronic effects provides information only on the probability that the chronic endpoint
assessed will be exceeded, not on the magnitude of the chronic effect expected.
Results of Analysis. As part of the process of developing and implementing a probabilistic
approach for ERA, a case study was completed. The case study involved both DRAs and PRAs for
effects of ChemX on birds and aquatic species. The deterministic screen conducted on ChemX
concluded qualitatively that it could pose a high risk to both freshwater fish and invertebrates and
showed that PRA was warranted. Based on the probabilistic analysis, it was concluded that the use
of ChemX was expected to infrequently result in significant freshwater fish mortalities but routinely
result in reduced growth and other chronic effects in exposed fish. Substantial mortalities and
chronic effects to sensitive aquatic invertebrates were predicted to occur routinely after peak
exposures.
Management Considerations. In its review of the case study, the FIFRA SAP congratulated the
Agency on the effort made to conduct PRA of pesticide effects in ecosystems. The panel commented
that the approach had progressed greatly from earlier efforts, and that the intricacy of the models
was surprisingly good, given the time interval in which the Agency had to complete the task.
79
-------
Following the case study, EPA refined the models based on the SAP comments. In addition, the
terrestrial Level II model was refined to include dermal and inhalation exposure.
Selected References. An overview of the models is available at
http://www.epa.gov/oppefedl/ecorisk/fifrasap/rra exec sum.htm#Terrestrial.
Case Study 14: Expert Elicitation of Concentration-Response
Relationship Between Fine Particulate Matter Exposure and Mortality
In 2002, the NRC recommended that EPA improve its characterization of uncertainty in the benefits
assessment for proposed regulations of air pollutants. NRC recommended that probability
distributions for key sources of uncertainty be developed using available empirical data or through
formal elicitation of expert judgments. A key component of EPA's approach for assessing potential
health benefits associated with air quality regulations targeting emissions of PM2.s and its
precursors is the effect of changes in ambient PM2.s levels on mortality. Avoided premature deaths
constitute, on a monetary basis, between 85 and 95 percent of the monetized benefits reported in
EPA's retrospective and prospective Section 812A benefit-cost analyses of the Clean Air Act (CAA;
USEPA 1997e and 1999) and in Regulatory Impact Analysis (RIA) for rules such as the Heavy Duty
Diesel Engine/Fuel Rule (USEPA 2000c) and the Non-Road Diesel Engine Rule (USEPA 2004). In
response to the NRC recommendation, EPA conducted an expert elicitation evaluation of the
concentration-response relationship between PM2.s exposure and mortality.
Probabilistic Analysis. This case study provides an example of the use of expert elicitation (Group
3) to derive probabilistic estimates of the uncertainty in one element of a cost-benefit analysis.
Expert elicitation uses carefully structured interviews to elicit from each expert a best estimate of
the true value for an outcome or variable of interest, as well as their uncertainty about the true
value. This uncertainty is expressed as a subjective probabilistic distribution of values and reflects
each expert's interpretation of theory and empirical evidence from relevant disciplines, as well as
their beliefs about what is known and not known about the subject of the study. For the PM2.s
expert elicitation, the process consisted of development of an elicitation protocol, selection of
experts, development of a briefing book, conduct of elicitation interviews, the use of expert input
prior to and following individual elicitation of judgments and the expert judgments themselves. The
elicitation involved personal interviews with 12 health experts who had conducted research on the
relationship between PM2.s exposures and mortality.
The main quantitative question asked each expert to provide a probabilistic distribution for the
average expected decrease in U.S. annual, adult and all-cause mortality associated with a 1 [ig/m3
decrease in annual average PM2.s levels. When answering the main quantitative question, each
expert was instructed to consider that the total mortality effect of a 1 [ig/m3 decrease in ambient
annual average PM2.s may reflect reductions in both short-term peak and long-term average
exposures to PM2.s. Each expert was asked to aggregate the effects of both types of changes in their
answers. The experts were given the option to integrate their judgments about the likelihood of a
causal relationship or threshold in the concentration-response function into their own distributions
or to provide a distribution "conditional on" one or both of these factors.
Results of Analysis. The project team developed the interview protocol between October 2004 and
January 2006. Development of the protocol was informed by an April 2005 symposium held by the
project team, where numerous health scientists and analysts provided feedback; detailed pretesting
with independent EPA scientists in November 2005; and discussion with the participating experts
at a pre-elicitation workshop in January 2006. The elicitation interviews were conducted between
January and April 2006. Following the interviews, the experts reconvened for a post-elicitation
80
-------
workshop in June 2006, in which the project team anonymously shared the results of all experts
with the group.
The median estimates for the PM2.s mortality relationship were generally similar to estimates
derived from the two epidemiological studies most often used in benefits assessment However, in
almost all cases, the spread of the uncertainty distributions elicited from the experts exceeded the
statistical uncertainty bounds reported by the most influential epidemiologic studies, suggesting
that the expert elicitation process was successful in developing more comprehensive estimates of
uncertainty for the PM2.s mortality relationship. The uncertainty distributions for PM2.s
concentration-response resulting from the expert elicitation process were used in the RIA for the
revised NAAQS standard for PM2.s (promulgated on September 21, 2006). Because the NAAQS are
exclusively health-based standards, this RIA played no part in EPA's determination to revise the
PM2.s NAAQS. Benefits estimates in the RIA were presented as ranges and included additional
information on the quantified uncertainty distributions surrounding the points on the ranges,
derived from both epidemiological studies and the expert elicitation results. OMB's review of the
RIA was completed in March 2007.
Management Considerations. The NAAQS are exclusively health-based standards, so these
analyses were not used in any manner by EPA in determining whether to revise the NAAQS for
PM2.s. Moreover, the request to engage in the expert elicitation did not come from the CASAC, the
official peer review body for the NAAQS; a decision to conduct the analyses does not reflect CASAC
advice that such analyses inform NAAQS determinations. The analyses addressed comments from
the NRC that recommended that probability distributions for key sources of uncertainty be
addressed. The analyses were used in EPA's retrospective and prospective Section 812A benefit-
cost analyses of the CAA (USEPA 1997e and 1999) and in RIAs for rules such as the Heavy Duty
Diesel Engine/Fuel Rule (USEPA 2000c) and the Non-Road Diesel Engine Rule (USEPA 2004). In
response to the NRC recommendation, EPA conducted an expert elicitation evaluation of the
concentration-response relationship between PM2.s exposure and mortality.
Selected Reference. The assessment is available at http://www.epa.gov/ttn/ecas/ria.html.
Case Study 15: Expert Elicitation of Sea-Level Rise Resulting From
Global Climate Change
The United Nations Framework Convention on Climate Change requires nations to implement
measures for adapting to rising sea level and other effects of changing climate. To decide on an
appropriate response, coastal planners and engineers weigh the cost of these measures against the
likely cost of failing to prepare, which depends on the probability of the sea rising a particular
amount The U.S. National Academy of Engineering recommended that assessments of sea level rise
should provide probability estimates. Coastal engineers regularly employ probability information
when designing structures for floods, and courts use probabilities to determine the value of land
expropriated by regulations. This 1995 case study describes the development of a probability
distribution for sea level rise, using models employed by previous assessments, as well as the
expert opinions of 20 climate and glaciology reviewers about the probability distributions for
particular model coefficients.
Probabilistic Analysis. This case study provides an example both of an analysis describing the
probability of sea level rise, as well as an expert elicitation of the likelihood of particular models
and probability distributions of the coefficients used by those models to predict future sea level rise
(Group 3). The assessment of the probability of sea level rise used existing models describing the
change in five components of sea level rise associated with greenhouse gas-related climate change
(thermal expansion, small glaciers, polar precipitation, melting and ice discharge from Greenland
81
-------
and ice discharge from Antarctica). To provide a starting point for the expert elicitation, initial
probability distributions were assigned to each model coefficient based on the published literature.
After the initial probabilistic assessment was completed, the draft report was circulated to expert
reviewers considered most qualified to render judgments on particular processes for revised
estimates of the likelihood of particular models and the model coefficients' probability
distributions. Experts representing both extremes of climate change science (those who predicted
trivial consequences and those who predicted catastrophic effects; individuals whose thoughts had
been excluded from previous assessments) were included. The experts were asked to provide
subjective assessments of the probabilities of various models and model coefficients. These
subjective probability estimates were used in place of the initial probabilities in the final model
simulations. Different reviewer opinions were not combined to produce a single probability
distribution for each parameter; instead, each reviewer's opinions were used in independent
iterations of the simulation. The group of simulations resulted in the probability distribution of sea
level rise.
Results of Analysis. The analysis, completed with a budget of $100,000, provided a probabilistic
estimate of sea level rise for use by coastal engineers and regulators. The results suggested that
there is a 65 percent chance that the sea level will rise 1 millimeter (mm) per year more rapidly in
the next 30 years than it has been rising in the last century. Under the assumption that nonclimatic
factors do not change, the projections suggested that there is a 50 percent chance that the global
sea level will rise 45 centimeters (cm), and a 1 percent chance of a 112 cm rise by the year 2100.
The median prediction of sea level rise was similar to the midpoint estimate of 48 cm published by
the Intergovernmental Panel on Climate Change (IPCC) shortly thereafter (IPCC 1996). The report
also found a 1 percent chance of a 4 to 5 meter rise over the next 2 centuries.
Management Considerations. There are two reports (USEPA 1995c; Titus and Narayanan 1996)
that discuss several uses of the results of this study. By providing a probabilistic representation of
sea level rise, the assessment allows coastal residents to make decisions with recognition of the
uncertainty associated with predicted change. Rolling easements that vest when the sea rises to a
particular level can be properly valued in both "arms-length" transaction sales or when calculating
the allowable deduction for a charitable contribution of the easement to a conservancy. Cost-benefit
assessments of alternative infrastructure designs—which already consider flood probabilities—
also can consider sea level rise uncertainty. Assessments of the benefits of preventing an
acceleration of sea level rise can include more readily low-probability outcomes, which can provide
a better assessment of the true risk when the damage function is nonlinear, which often is the case.
Selected References.
USEPA (U.S. Environmental Protection Agency). 1995c. The Probability of Sea Level Rise.
EPA/230/R-95/008. Washington, D.C.: Climate Change Division, USEPA.
http://nepis.epa.gov/Exe/ZyPURL.cgi?Dockey=20011G10.txt
IPCC (Intergovernmental Panel on Climate Change). 1996. Climate Change 1995: The Science of
Climate Change. Contribution of Working Group I to the Second Assessment of the Intergovernmental
Panel on Climate Change. Cambridge: Cambridge University Press.
Titus, J. G., and V. Narayanan. 1996. "The Risk of Sea Level Rise." Climatic Change 33(2): 151-212.
Case Study 16: Knowledge Elicitation fora Bayesian Belief Network
Model of Stream Ecology
The identification of the causal pathways leading to stream impairment is a central challenge to
understanding ecological relationships. Bayesian belief networks (BBNs) are a promising tool for
82
-------
modeling presumed causal relationships, providing a modeling structure within which different
factors describing the ecosystem can be causally linked and calculating uncertainties expressed for
each linkage.
BBNs can be used to model complex systems that involve several interdependent or interrelated
variables. In general, a BBN is a model that evaluates situations where some information already is
known, and incoming data are uncertain or partially unavailable. The information is depicted with
influence diagrams that present a simple visual representation of a decision problem, for which
quantitative estimates of effect probabilities are assigned. As such, BBNs have the potential for
representing ecological knowledge and uncertainty in a format that is useful for predicting
outcomes from management actions or for diagnosing the causes of observed conditions. Generally,
specification of a BBN can be performed using available experimental data, literature review
information (secondary data) and expert elicitation. Attempts to specify a BBN for the linkage
between fine sediment load and macroinvertebrate populations using data from literature reviews
have failed because of the absence of consistent conceptual models and the lack of quantitative data
or summary statistics needed for the model. In light of these deficiencies, an effort was made to use
expert elicitation to specify a BBN for the relationship between fine sediment load resulting from
human activity and populations of macroinvertebrates. The goals of this effort were to examine
whether BBNs might be useful for modeling stream impairment and to assess whether expert
opinion could be elicited to make the BBN approach useful as a management tool.
Probabilistic Analysis. This case study provides an example of expert elicitation in the
development of a BBN model of the effect of increased fine sediment load in a stream on
macroinvertebrate populations (Group 3). For the purpose of this study, a stream setting
(a Midwestern, low-gradient stream) and a scenario of impairment (introduction of excess fine
sediment) were used. Five stream ecologists with experience in the specified geographic setting
were invited to participate in an elicitation workshop. An initial model was depicted using influence
diagrams, with the goals of structuring and specifying the model using expert elicitation. The
ecologists were guided through a knowledge elicitation session in which they defined factors that
described relevant chemical, physical and biological aspects of the ecosystem. The ecologists then
described how these factors were connected and were asked to provide subjective, quantitative
estimates of how different attributes of the macroinvertebrate assemblage would change in
response to increased levels of fine sediment Elicited input was used to restructure the model
diagram and to develop probabilistic estimates of the relationships among the variables.
Results of Analysis. The elicited input was compiled and presented in terms of the model as
structured by the stream ecologists and their model specifications. The results were presented both
as revised influence diagrams and with Bayesian probabilistic terms representing the elicited input
The study yielded several important lessons. Among these were that the elicitation process takes
time (including an initial session by teleconference as well as a 3-day workshop), defining a
scenario with an appropriate degree of detail is critical and elicitation of complex ecological
relationships is feasible.
Management Considerations. The study was considered successful for several reasons. First, the
feasibility of the elicitation approach to building stream ecosystem models was demonstrated. The
study also resulted in the development of new graphical techniques to perform the elicitation. The
elicited input was interpreted in terms of predictive distributions to support fitting a complete
Bayesian model. Furthermore, the study was successful in bringing together a group of experts in a
particular subject area for the purpose of sharing information and learning about expert elicitation
in support of model building. The exercise provided insights into how best to adapt knowledge
elicitation methods to ecological questions and informed the assembled stream ecologists on the
elicitation process and on the potential benefits of this modeling approach. The explicit
83
-------
quantification of uncertainty in the model not only enhances the utility of the model predictions,
but also can help guide future research.
Selected References.
Black, P., T. Stockton, L. Yuan, D. Allan, W. Dodds, L. Johnson, M. Palmer, B. Wallace, and A. Stewart.
2005. "Using Knowledge Elicitation to Inform a Bayesian Belief Network Model of a Stream
Ecosystem." Eos, Transactions, American Geophysical Union 86 (18), Joint Assembly Supplement,
Abstract #NB41E-05.
Yuan, L. 2005. "TI: A Bayesian Approach for Combining Data Sets to Improve Estimates of Taxon
Optima." Eos, Transactions, American Geophysical Union 86 (18), Joint Assembly Supplement,
Abstract #NB41E-04.
84
-------
CASE STUDY REFERENCES
Black, P., T. Stockton, L. Yuan, D. Allan, W. Dodds, L. Johnson, M. Palmer, B. Wallace, and A. Stewart.
2005. "Using Knowledge Elicitation to Inform a Bayesian Belief Network Model of a Stream
Ecosystem." Eos, Transactions, American Geophysical Union 86 (18), Joint Assembly Supplement,
Abstract #NB41E-05.
Burke, J., M. Zufall, and H. Ozkaynak. 2001. "A Population Exposure Model for Particulate Matter:
Case Study Results for PM2.s in Philadelphia, PA." Journal of Exposure Analysis and Environmental
Epidemiology 11(6]: 470-89.
Connelly, N. A., T. L. Brown, and B. A. Knuth. 1990. New York Statewide Angler Survey 1988. Albany,
NY: Bureau of Fisheries, New York Department of Environmental Conservation.
Georgepoulos, P. G., S. W. Wang, V. M. Vyas, Q. Sun, J. Burke, R. Vedantham, T. McCurdy, and H.
Ozkaynak. 2005. "A Source-to-Dose Assessment of Population Exposure to Fine PM and Ozone in
Philadelphia, PA, During a Summer 1999 Episode." Journal of Exposure Analysis and Environmental
Epidemiology 15 (5): 439-57.
IPCC (Intergovernmental Panel on Climate Change). 1996, Climate Change 1995: The Science of
Climate Change. Contribution of Working Group I to the Second Assessment of the IPPC. Cambridge:
Cambridge University Press.
Jamieson, D. 1996. "Scientific Uncertainty and the Political Process." Annals of the American
Academy of Political and Social Science 545: 35-43.
Kurowicka, D., and R. Cooke. 2006. Uncertainty Analysis With High Dimensional Dependent Modeling.
Wiley Series in Probability and Statistics. New York: John Wiley & Sons.
Langstaff, J. E. 2007. Analysis of Uncertainty in Ozone Population Exposure Modeling. Office of Air
Quality Planning and Standards Staff Memorandum to Ozone NAAQS Review Docket OAR-2005-
0172. Washington, D.C.: USEPA.
http://www.epa.gOV/ttn/naaqs/standards/ozone/s ozone cr td.html.
Olsen AR, Kincaid TM, Payton Q. 2012. Spatially balanced survey designs for natural resources. In:
Gitzen RA, Millspaugh JJ, Cooper AB, Licht DS (eds) Design and Analysis of Long-Term Ecological
Monitoring Studies. Cambridge University Press, Cambridge, UK, pp 126-150
Stahl, C. H., and A. J. Cimorelli. 2005. "How Much Uncertainty Is Too Much and How Do We Know? A
Case Example of the Assessment of Ozone Monitor Network Options." Risk Analysis 25 (5): 1109-20.
Titus, J. G. and V. Narayanan. 1996. "The Risk of Sea Level Rise: A Delphic Monte Carlo Analysis in
Which Twenty Researchers Specify Subjective Probability Distributions for Model Coefficients
Within Their Respective Areas of Expertise." Climatic Change 33 (2): 151-212.
USEPA (U.S. Environmental Protection Agency). 1992a. Guidelines for Exposure Assessment.
EPA/600/Z-92/001. Washington, D.C.: Risk Assessment Forum, USEPA.
http://cfpub.epa. gov/ncea/cfm/recordisplay.cfm?deid=15 263.
USEPA. 1992b. Framework for Ecological Risk Assessment. EPA/630/R-92/001. Washington, D.C.:
Risk Assessment Forum, USEPA. http://www.epa.gov/raf/publications/framework-eco-risk-
assessmenthtm.
USEPA. 1995a. Guidance for Risk Characterization. Washington, D.C.: Science Policy Council, USEPA.
http://www.epa.gov/spc/pdfs/rcguide.pdf.
85
-------
USEPA. 1995b. Policy on Evaluating Health Risks to Children. Washington, D.C.: Science Policy
Council, USEPA. http://www.epa.gov/spc/pdfs/memohlth.pdf.
USEPA. 1995c. The Probability of Sea Level Rise. EPA/230/R-95/008. Washington, D.C.: Office of
Policy, Planning, and Evaluation, USEPA.
http://nepis.epa. gov/Exe/ZyPURL.cgi?Dockey=20011G10.txt.
USEPA. 1997a. Policy for Use of Probabilistic Analysis in Risk Assessment at the U.S. Environmental
Protection Agency. Washington, D.C.: Science Policy Council, USEPA.
http://www.epa.gov/spc/pdfs/probpol.pdf.
USEPA. 1997b. Guiding Principles for Monte Carlo Analysis. EPA/630/R-97/001. Washington, D.C.:
Risk Assessment Forum, USEPA. http://www.epa.gov/raf/publications/pdfs/montecar.pdf.
USEPA. 1997c. Guidance on Cumulative Risk Assessment. Part 1: Planning and Scoping. Washington,
D.C.: Science Policy Council, USEPA. http://www.epa.gov/spc/pdfs/cumrisk2.pdf.
USEPA. 1997d. Exposure Factors Handbook. Washington, D.C.: National Center for Environmental
Assessment, USEPA. http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=12464.
USEPA. 1997e. Final Report to Congress on Benefits and Costs of the Clean Air Act, 1970 to 1990.
EPA/410/R-97/002. Cincinnati, OH: Office of Air and Radiation, USEPA.
USEPA. 1998. Guidelines for Ecological Risk Assessment. EPA/630/R-95/002F. Washington, D.C.:
Risk Assessment Forum, USEPA. http://www.epa.gov/raf/publications/pdfs/ECOTXTBX.PDF.
USEPA. 1999. The Benefits and Costs of the Clean Air Act 1990 to 2010: EPA Report to Congress.
EPA/410/R-99/001. Washington, D.C.: Office of Air and Radiation, USEPA.
http://www.epa.gov/oar/sect812/1990-2010/chapll30.pdf.
USEPA. 2000a. EPA Science Policy Council Handbook: Risk Characterization. EPA/100/B-00/002.
Washington, D.C.: Science Policy Council, USEPA. http://www.epa.gov/spc/pdfs/rchandbk.pdf.
USEPA. 2000b. Phase 2 Report: Further Site Characterization and Analysis. Volume 2F—Revised
Human Health Risk Assessment, Hudson River PCBs Reassessment RI/FS. New York: Region 2, USEPA.
http://www.epa.gov/hudson/revisedhhra-textpdf.
USEPA. 2000c. Final Heavy Duty Engine/Diesel Fuel Rule: Air Quality Estimation, Selected Health and
Welfare Benefits Methods, and Benefit Analysis Results. Research Triangle Park, NC: Office of Air
Quality Planning and Standards, USEPA.
http://www.epa.gov/ttnecasl/regdata/Benefits/tsdhddv8.pdf.
USEPA. 2001. Risk Assessment Guidance for Superfund: Volume HI — Part A, Process for Conducting
Probabilistic Risk Assessment. EPA/540/R-02/002. Washington, D.C.: Office of Emergency and
Remedial Response, USEPA.
http://www.epa.gov/oswer/riskassessment/rags3adt/pdf/rags3adt complete.pdf.
USEPA. 2002. EMAP Research Strategy. EPA 620/R-02/002. Research Triangle Park, NC:
Environmental Monitoring and Assessment Program, NHEERL, USEPA.
http://www.epa.gov/nheerl/emap/files/emap research strategy.pdf.
USEPA. 2004. An Examination of EPA Risk Assessment Principles and Practices. EPA/100/B-04/001.
Washington, D.C.: Office of the Science Advisor, USEPA. http://www.epa. gov/osa/pdfs/ratf-
final.pdf.
USEPA. 2005. Ozone Health Assessment Plan: Scope and Methods for Exposure Analysis and Risk
Assessment. Research Triangle Park, NC: Office of Air Quality Planning and Standards, USEPA.
http://www.epa.gov/ttn/naaas/standards/ozone/data/o3 health assessment plan april05.pdf.
86
-------
USEPA. 2006. Total Risk Integrated Methodology (TRIM)—Air Pollutants Exposure Model
Documentation (TRIM.Expo/APEX, Version 4) Volume I: User's Guide; Volume II: Technical Support
Document. Research Triangle Park, NC: Office of Air Quality Planning and Standards, USEPA.
http://www. epa. gov/ttn/fera/human apex.html.
USEPA. 2007a. Review of National Ambient Air Quality Standards for Ozone: Policy Assessment of
Scientific and Technical Information—OAQPS Staff Paper. Research Triangle Park, NC: Office of Air
Quality Planning and Standards, USEPA.
http://www.epa.gov/ttn/naaqs/standards/ozone/data/2007 07 ozone staffpaper.pdf.
USEPA. 2007b. Ozone Population Exposure Analysis for Selected Urban Areas. EPA/452/R-07-010.
Research Triangle Park, NC: Office of Air Quality Planning and Standards, USEPA.
http://www.epa.gov/ttn/naaqs/standards/ozone/data/2007 07 o3 exposure tsd.pdf.
USEPA. 2014. Framework for Human Health Risk Assessment to Inform Decision Making. EPA/100/R-
14/001. Washington, D.C.: Risk Assessment Forum, Office of the Science Advisor, USEPA.
http://www.epa.gov/raf/frameworkhhra.htm
USGAO (U.S. Government Accountability Office). 2000. Water Quality: Key EPA and State Decisions
Limited by Inconsistent and Incomplete Data. GAO/RCED-00-54. Washington, D.C.: USGAO.
http://www.environmental-auditing.Org/Portals/0/AuditFiles/usengOOar ft key epa.pdf.
Xue,}., V. G. Zartarian, H. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. 2006. "A
Probabilistic Arsenic Exposure Assessment for Children Who Contact Chromated Copper Arsenate
(CCA)-Treated Playsets and Decks, Part 2: Sensitivity and Uncertainty Analyses." Risk Analysis 26
(2): 533-41.
Yuan, L. 2005. "TI: A Bayesian Approach for Combining Data Sets to Improve Estimates of Taxon
Optima." Eos, Transactions, American Geophysical Union 86 (18), Joint Assembly Supplement,
Abstract #NB41E-04.
Zartarian, V. G., J. Xue, H. A. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. 2006. A
Probabilistic Exposure Assessment for Children Who Contact CCA-Treated Playsets and Decks Using
the Stochastic Human Exposure and Dose Simulation Model for the Wood Preservative Scenario
[SHEDS-Wood), Final Report. EPA/600/X-05/009. Washington, D.C.: USEPA.
Zartarian, V. G., J. Xue, H. A. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. 2006. "A
Probabilistic Arsenic Exposure Assessment for Children Who Contact Chromated Copper Arsenate
(CAA)-Treated Playsets and Decks, Part 1: Model Methodology, Variability Results, and Model
Evaluation." Risk Analysis 2 6 (2): 515 -31.
87
------- |