oEPA
United States
Environmental Protection
Agency
EPA/1 OO/R-14/003 July 2014
www.epa.gov/raf
Probabilistic Risk Assessment to Inform
Decision Making: Frequently Asked Questions
Question B: What is the risk for an
individual experiencing high-end risk
impacts in the population (e.g., 95%ile)?
Question A: What is the risk
for a typical (median or average)
individual in the population?
19 high
Figure 4. What Questions Can PRA Address? PRA can be used to determine the likelihood of
exposure or risk in a specific fraction of the population.
Office of the Science Advisor
Risk Assessment Forum
-------
EPA/100/R-14/003
Probabilistic Risk Assessment to Inform Decision Making:
Frequently Asked Questions
July 25, 2014
U.S. Environmental Protection Agency
Office of the Science Advisor
Risk Assessment Forum
Probabilistic Risk Analysis Technical Panel
Washington, D.C. 20460
-------
DISCLAIMER
This document has been reviewed in accordance with U.S. Environmental Protection Agency policy.
Mention of trade names or commercial products does not constitute endorsement or
recommendation for use.
Preferred Citation:
U.S. Environmental Protection Agency (USEPA). 2014. Probabilistic Risk Assessment to Inform
Decision Making: Frequently Asked Questions. EPA/100/R-09/001B. Washington, D.C.: Risk
Assessment Forum, Office of the Science Advisor, USEPA.
http://epa.gov/raf/prawhitepaper/index.htm.
-------
This document was produced by a Technical Panel of the Risk Assessment Forum to summarize the
Risk Assessment Forum White Paper: Probabilistic Risk Assessment Methods and Case Studies for
managers and agency scientist to gain a better understanding of the principles of probabilistic risk
assessment The authors drew on their experience in performing and interpreting probabilistic
assessments to improve the management of environmental and health hazards. Interviews,
presentations and dialogues with risk managers conducted by the Technical Panel have contributed
to the insights and recommendations in this FAQ document and the associated white paper.
Authors
Robert Hetes, EPA Office of Research and Development (Document Lead)
Kathryn Gallagher, EPA Office of Water (formerly with the Office of the Science Advisor)
Marian Olsen, EPA Region 2
Rita Schoeny, EPA Office of Research and Development (formerly with the Office of Water)
Cynthia Stahl, EPA Region 3
Probabilistic Risk Analysis Technical Panel
Haluk Ozkaynak, EPA Office of Research and Development (Co-Chair)
Robert Hetes, EPA Office of Research and Development (Co-Chair)
Kathryn Gallagher, EPA Office of Water (Co-Chair)
Christopher Frey, North Carolina State University (while on temporary assignment at EPA)
(White Paper Co-Lead)
John Paul, EPA Office of Research and Development (White Paper Co-Lead)
Pasky Pascual, EPA Office of Research and Development (White Paper Co-Lead)
Marian Olsen, EPA Region 2 (Case Study Lead)
Michael Clipper, EPA Office of Solid Waste and Emergency Response
Michael Messner, EPA Office of Water
Keeve Nachman, currently at The Johns Hopkins Bloomberg School of Public Health
Zachary Pekar, EPA Office of Air and Radiation
Rita Schoeny, EPA Office of Research and Development
Cynthia Stahl, EPA Region 3
David Hrdy, EPA Office of Pesticide Programs
John Langstaff, EPA Office of Air and Radiation
Elizabeth Margosches, EPA Office of Chemical Safety and Pollution Prevention (retired)
Audrey Galizia, EPA Office of Research and Development
Nancy Rios-Jafolla, EPA Region 3
Donna Randall, EPA Office of Chemical Safety and Pollution Prevention
Khoan Dinh, EPA Office of Chemical Safety and Pollution Prevention (retired)
Harvey Richmond, EPA Office of Air and Radiation (retired)
Risk Assessment Forum Science Coordinator
Julie Fitzpatrick, EPA Office of the Science Advisor
External Peer Reviewers
Scott Person (Chair), Applied Biomathematics
Annette Guiseppi-Elie, DuPont Engineering
Dale Hattis, Clark University
Igor Linkov, U.S. Army Engineer Research and Development Center
John Toll, Windward Environmental LLC
in
-------
TABLE OF CONTENTS
EXECUTIVE SUMMARY 1
FREQUENTLY ASKED QUESTIONS 2
Why should I care about PRA? Why is it important to risk managers? 2
As a manager, what do I need to know about PRA? 2
How could a risk manager approach a PRA? What would a manager consider? 2
What is PRA? How does it compare with current approaches? 3
What are common challenges facing EPA decision makers? 3
What are variability and uncertainty? What is their relevance in risk assessment and risk
management? 4
How is risk assessment conducted currently at EPA? 7
Do human health risk assessment and ecological risk assessment treat uncertainty and
variability differently? 7
How does EPA typically address variability and uncertainty? 7
How does PRA address variability and uncertainty? 9
How can enhanced characterization of uncertainty and variability help inform decisions? 10
What is the impact of variability and uncertainty on making decisions? 11
Is uncertainty and variability limited only to risk assessment? 11
What key questions related to uncertainty and variability are asked or considered by decision
makers? 12
What is the desired percentile of the population to be protected? By choosing this percentile,
who may not be protected? How can addressing uncertainty and variability using PRA help
inform decisions? 12
Whatare some of the limitations of and challenges for PRA? 14
Whatis EPA's experience with PRA? 14
What are other EPA publications on PRA? 17
When should one consider using PRA? 17
Whatis the right level of analysis? 18
If one is go ing to use PRA, what factors should be considered? 20
Whatare the resources needed to conducta PRA? 20
Does PRA require more data than conventional approaches? 20
Communication of PRA results to the manager and community: does the presentation of results
matter? 21
How can I get more information on PRA? 21
GLOSSARY 22
REFERENCES 26
IV
-------
LIST OF FIGURES, TABLES AND EQUATIONS
Figures
Figure 1. Source-to-Effect Continuum 5
Figure 2. Ingestion of Drinking Water for Adults 6
Figure 3. General Phases of the Risk Assessment Process 8
Figure 4. What Questions Can PRA Address? 13
Figure 5. Tiered Approach for Risk Assessment 19
Tables
Table 1. Selected Examples of EPA Applications of Risk Assessment Techniques from Deterministic
to more Complex Analysis 15
Equations
Equation 1. Deterministic Risk Assessment 7
Equation 2. Probabilistic Risk Assessment 9
-------
EXECUTIVE SUMMARY
Probabilistic risk assessment (PRA), in its simplest form, is a group of techniques that incorporate
variability and uncertainty into risk assessments. Variability refers to the inherent natural
variation, diversity and heterogeneity across time, space or individuals within a population or
lifestage, while uncertainty refers to imperfect knowledge or a lack of precise knowledge of the
physical world, either for specific values of interest or in the description of the system (USEPA
20lie). Variability and uncertainty have the potential to result in overestimates or underestimates
of the predicted risk.
PRA provides estimates of the range and likelihood of a hazard, exposure or risk, rather than a
single point estimate. Stakeholders inside and outside of the Agency have recommended a more
complete characterization of risks, including uncertainties and variability, in protecting more
sensitive or vulnerable populations and lifestages. PRA can be used to support decision-making risk
management by assessment of impacts of uncertainties on each of the potential decision
alternatives. The aim of this is documentis to summarize the Risk Assessment Forum White Paper:
Probabilistic Risk Assessment Methods and Case Studies (USEPA 2014a) for managers and agency
scientists to gain a better understanding of the principles of PRA without the more detailed
discussion presented in the White Paper.
Numerous advisory bodies, such as the Science Advisory Board (SAB) and the National Research
Council (NRC) of the National Academy of Sciences (NAS), have also recommended that EPA
incorporate probabilistic analyses into the Agency's decision-making process. EPA's Risk
Assessment Forum (RAF) formed a Technical Panel, consisting of representatives from the Agency's
program and regional offices, to develop this Frequently Asked Questions (FAQ) document and its
companion publication titled Risk Assessment Forum White Paper: Probabilistic Risk Assessment
Methods and Case Studies (USEPA 2014a). The RAF is also recommending the development of
Agency resources, such as a clearinghouse of PRA case studies, best practices, resources and
seminars, to raise general knowledge about how these probabilistic tools can be used.
The purpose of this FAQ document is to present general concepts and principles of PRA, describe
how PRA can improve the bases of Agency decisions, and provide illustrations of how PRA has been
used in risk estimation and in describing the uncertainty in decision making. This document is a
companion to the more detailed white paper, which also includes a compendium of 16 relevant case
studies ranging in their level of complexity. Both documents describe the potential and actual uses
of probabilistic tools in the risk decision process and facilitate their implementation in human,
ecological and environmental risk analysis and related decision-making at EPA. The documents do
not prescribe a specific approach to risk assessment; rather, they describe the various stages and
aspects of an assessment or decision process in which probabilistic tools may add value. These are
not regulatory guidance documents, but they are designed to be helpful handbooks and/or
references for risk assessors, risk managers and decision-makers interested in improving the risk
assessment and decision-making process.
Overall, this FAQ document answers more than 20 questions commonly asked about PRA,
beginning with "Why should I care about PRA, and why is it important to risk managers?" and
ending with "How can I get more information on PRA?" These questions cover a wide range of PRA
issues, and risk managers, decision-makers and other interested readers will find a plethora of PRA
information in this document. Further detailed information, including recommendations for future
activities, are provided in the companion white paper (USEPA 2014a).
-------
FREQUENTLY ASKED QUESTIONS
Why should I care about PRA? Why is it important to risk managers?
Uncertainty is unavoidable, even when using highly accurate data with the most sophisticated
models. EPA makes decisions in the presence of uncertainty. Therefore, understanding the decision
context is critical for Agency decision making. Various stakeholders inside and outside of the
Agency have requested a more comprehensive characterization of risks, including uncertainties, in
protecting more sensitive or vulnerable populations and lifestages. Probabilistic risk assessment
(PRA) is one way to characterize the uncertainty associated with any risk assessment As part of a
decision analysis, the enhanced use of PRA and characterization of uncertainty would allow EPA
decision-makers opportunities to assess uncertainty pertaining to its effect on decisions and
explore the defensibility of the available risk management options through a more robust and
transparent process. Most often, risk managers want to know if better understanding of
uncertainties might support a different decision alternative or provide further support for the
selected decision.
The use of PRA in addressing uncertainty and variability at EPA is not systematically practiced, and
the lack thereof is often a major recommendation from both internal and external reviews of EPA
products and procedures (e.g., the SAB review of EPA practices in 2007 [USEPA 2007b], the NRC
review of the Dioxin Reassessment [NRC 2006], the Office of Management and Budget's Circular A-4
[OMB 2003] and Updated Principles for Risk Analysis [OMB 2007]). The Agency has published basic
guidance as well as some program-specific procedures and applications for PRA (USEPA 2001). As
the science of PRA continues to evolve, the enhanced use of this approach will facilitate better
characterization of uncertainty and improve the overall transparency and quality of EPA
assessments. PRA approaches provide additional tools to address specific challenges faced by
managers and improve confidence in Agency decisions. PRA can inform decision-makers about
specific segments of the population who are at risk, as opposed to the population as a whole. A PRA
also can add confidence to the conclusions of a deterministic risk estimate. This information can be
important to managers when a different decision might be made if the upper or lower limits of the
range of estimated exposures, doses or risks were presented in an analysis as described more fully
in Risk Assessment Forum White Paper: Probabilistic Risk Assessment Methods and Case Studies (EPA
2014a).
As a manager, what do I need to know about PRA?
There are probability, uncertainty and variability in many fields, such as weather forecasting,
political polls or climate change predictions. PRA can be used in each of these examples and others,
at many levels or degrees of sophistication, to support or improve decisions in a variety of different
decision contexts. The following sections describe the basics of PRA in more detail, how it can be
used to support decisions, and what to consider when pursuing a PRA or interpreting PRA results.
How could a risk manager approach a PRA? What would a manager consider?
PRA results are likely just some of the information that risk managers might use to inform their
decision-making. If the risk manager is trying to answer a question that requires the use of many
different criteria (where risk might be just one criterion), then they might want to contemplate the
use of formal decision analytic techniques. With decision analytic techniques, risk managers can
explore the relative importance of risk information compared to other information in making the
decision, and explore how uncertainty affects potential decisions. PRA provides information or
input to decision analysis. Decision analysis is a logical procedure for the balancing of factors that
influence a decision. The procedure incorporates uncertainties, values, and preferences in a basic
-------
structure that models the decision (Howard, 1966). Decision analysis is a broader activity than PRA,
and formal decision analytic techniques are outside of the scope of this FAQ document and
companion PRA white paper. There is a range of PRA techniques that might be useful to support
environmental decisions, and they vary in complexity and resource requirements. Communication
between the risk managers and risk assessors is critical to clearly define the specific needs of the
decision-maker and the questions to be addressed by the risk assessment, so that one can focus the
PRA on the information most likely to inform the decision. As part of Planning and Scoping of the
risk assessment the risk assessor and risk manager should evaluate the types of techniques
appropriate to meet the goals of the assessment and establish a process for completing and
reviewing the PRA in a cost-effective and timely manner. The dialogue should continue throughout
the process (U.S. EPA 2014b).
What is PRA? How does it compare with current approaches?
Current approaches to risk assessment typically rely on single point (often called "deterministic")
estimates of risk, with brief qualitative descriptions of uncertainty. PRA provides risk managers
with information about the uncertainties in the data, models, assumptions and results. A basic
characteristic of PRA is that it does not generate a single point estimate (like conventional
approaches), but rather produces a likelihood and range that a particular exposure, dose or effect
will occur. PRA is a group of computational techniques that allows the analysis of variability and
uncertainty to be incorporated into exposure and/or risk assessments. PRA also can be used to
support risk management using sensitivity and uncertainty analyses to assess the impact of
uncertainties on each of a set of potential decision alternatives. As mentioned above, this type of
information can be critical in decisions, especially in determining the likelihood of exposure or risk
in a specific portion of the population. Unlike conventional approaches, PRA also is suited to
providing decision-makers with a better understanding of the impact of uncertainties on the
relative merits of each of the decision alternatives, providing the risk manager with a clearer
rationale for his or her decision choice.
What are common challenges facing EPA decision makers?
The challenges facing EPA decision-makers include the need to make complex, science-based
decisions, informed by considering the impact of uncertainties, where conflicting stakeholder
perspectives and short timeframes are involved. Attributes critical to decision-making include:
(1) understanding whom or what will be protected; and (2) knowing the appropriate degree of
confidence in the estimated protection provided by a particular decision. Decisions are often time-
sensitive and need to be made based on the current state of knowledge. They may involve the
comparison of several alternatives. In most cases, the health and environmental impacts of
environmental exposures cannot be isolated and measured directly; therefore, risk assessment
methods to estimate the health and ecological risks based on available data and information can be
used. Because EPA decision makers may face challenges in decision making that are not completely
related to deciding whether or not to use PRA, they must achieve a balance in the decision-making
process, trading off the costs and benefits of more detailed analyses like PRA with their overall
decision goals.
Making a risk decision may also consider other factors, such as the economic impacts or costs of
alternatives to address differential risks in a vulnerable population. EPA decision makers need to
consider multiple decision criteria informed by varying degrees of confidence in the underlying
information, understanding the relationship between and among those decision criteria (including
multi-pollutant and multi-media effects) and decision alternatives.
-------
What are variability and uncertainty? What is their relevance in risk assessment and risk
management?
Variability refers to the inherent natural variation, diversity and heterogeneity across time, space
or individuals within a population or lifestage, while uncertainty refers to imperfect knowledge or a
lack of precise knowledge of the physical world, either for specific values of interest or in the
description of the system (USEPA 2011c). Variability and uncertainty have the potential to result in
overestimates or underestimates of the predicted risk.
Variability refers to natural, inherent variation, variability is unavoidable, and cannot be reduced
in the way uncertainty can. Data and information can be used to provide a better description and
understanding of variability in the world or a particular system therein. Variability is present in all
aspects of the source-to-effect continuum [Figure 1), including the following:
n How pollutants are released (e.g., the effectiveness of emission controls).
D How pollutants are influenced by environmental conditions once released
(e.g., meteorology—temperature, wind and precipitation).
D The exposure of pollutants to receptors (e.g., inhalation or ingestion rates).
n The effects of pollutants (e.g., endpoint, health status, genetic susceptibility).
An example of variability is the amount of water consumed by an adult population. For example, a
survey of 1,000 people on daily water consumption might result in the distribution depicted in
Figure 2A. An alternative presentation of the same distribution using cumulative probability is
given by Figure 2B. Both figures show that 5 percent of the adult population consumes 2.96 liters of
water per day or more, whereas the mean (average) individual consumption is 1.04 liters of water
per day (based on studies identified in the 1997 edition of the Exposure Factors Handbook [USEPA
1997a], which was updated in 2011 [USEPA 2011b]).
Uncertainty is the lack of understanding of the world; while unavoidable it can be reduced through
additional investigation or collection of better information. Numerous schemes for classifying
uncertainty have been proposed, most focus on three broad technical categories:
n Input or parameter uncertainty refers to uncertainties in specific estimates or values used in
a model, such as the average drinking water intake rate.
D Model uncertainty refers to gaps in the scientific knowledge or theory that is required to
make accurate predictions, such as how two correctly specify exposures to water.
D Scenario uncertainty refers to errors, typically of omission, resulting from incorrect or
incomplete specification of the risk scenario to be evaluated such as errors in the problem
formulation omitting indirect water ingestion exposure pathways. The risk scenario is a set
of assumptions for the situation to be evaluated.
A decision analysis would include not only the impact of the input, model and scenario
uncertainties on the relative attractiveness of potential decision alternatives, but also would
include the degree to which specific choices (such as selecting input data, models and scenarios,
and even how the problem or decision analysis is framed) also may impact the relative
attractiveness of potential decision alternatives.
From a risk manager's perspective, understanding both uncertainty and variability are important.
Variability relates to our understanding of whom or what we are protecting, and uncertainty relates
-------
SOURCE/STRESSOR
FORMATION
HEALTH OUTCOME
TRANSPORT/FATE
ALTERED STRUCTURE/
FUNCTION
ENVIRONMENTAL
CONTAMINATION
EARLY BIOLOGICAL
EFFECT
DOSE
EXPOSURE
Figure 1. Source-to-Effeet Continuum. Variability exists within each aspect of the
source-to-effect continuum, the path from an environmental source to resulting
exposures and ultimate health outcomes (adapted from NRC 2009, Figure 4-1, page 95).
to our confidence in the estimate and the level of protection afforded by the options. The Appendix
in the companion document to this FAQ document titled Risk Assessment Forum White Paper:
Probabilistic Risk Assessment Methods and Case Studies (USEPA 2013), describes in greater detail
some specific issues related to the methodology for PRA.
Whether it is appropriate or useful to consider variability and uncertainty in decisions made by EPA
programs and regions may be affected by statutes, precedents and input provided by stakeholders.
Congress establishes legal requirements that generally describe the level of protectiveness that EPA
regulations must achieve. Infrequently, Congress imposes specific risk assessment or other
requirements that may minimize the regulatory necessity to address uncertainty or variability.
Individual statutes identify varying risks to evaluate and protect against (e.g., establish a margin of
safety, protect sensitive resources, or reduce overall risks). In addition, they may mandate different
levels of protection (e.g., protect public welfare, prevent unreasonable risk, reduce overall risks, or
function without adverse effects) (USEPA 2004b). These differences have implications for whether
or how one analyzes or considers uncertainty (e.g., the degree of confidence in protection) or
variability within a population or lifestage (e.g., evaluating the "expected risk or central estimate of
risk for the specific populations"1 vs. risks to "sensitive or susceptible individuals or groups"2 or
"individual most exposed"3).
The 2004 EPA staff paper titled An Examination of EPA Risk Assessment Principles and Practices
(USEPA 2004b) and the PRA white paper titled Risk Assessment Forum White Paper: Probabilistic
Risk Assessment Methods and Case Studies (USEPA 2014a; companion publication to this FAQ
document) describe uncertainty and variability in some detail and can be referred to for more
information.
1 Safe Drinking Water Act Amendments of 1996. 42 U.S.C. § 300g-l (b)(3) (1996).
2 Clean Air Act Amendments of 1990. 42 U.S.C. § 108 (f)(l)(C) (1990).
3 Clean Air Act Amendments of 1990. 42 U.S.C. § 112 (c)(9)(B)(i) (1990).
-------
I
.9 0.75 -
B
^0.50 -
'w
A. Displayed as Probability Density (Lognormal Density)
o.oo -
Mean: 1.04
95 %-ile: 2.96
1234
Drinking Water Intake (L/day)
"81.00 -
B. Displayed as Cumulative Probability (Lognormal Distribution)
95%-ile: 2.96
234
Drinking Water Intake (L/day)
Figure 2. Ingestion of Drinking Water for Adults. Variability can be demonstrated by the
amount of water consumed by a population. In this example, the distribution of daily
water consumption for adults (ages > 21 years) is depicted in part A, while the
cumulative probability is presented in part B.
-------
How is risk assessment conducted currently at EPA?
General phases of the risk assessment process are illustrated in Figure 3. There are many
similarities between ecological and human health risk assessment, and EPA is working to increase
integration between these two processes.
Do human health risk assessment and ecological risk assessment treat uncertainty and
variability differently?
Human health risk assessment (HHRA) addresses the protection of one species. Ecological risk
assessment (ERA) generally addresses the protection of a large number of species, using a few
species as surrogates for hundreds to thousands of species present in the environment HHRA
usually addresses the evaluation of risk to individuals or groups of susceptible human
populations or lifestages. ERA generally addresses the evaluation of risk to populations (not
individuals, except in the case of endangered species), communities of organisms or ecosystems.
HHRA generally focuses on sublethal effects or endpoints. ERA frequently focuses on lethality,
reproduction and growth as major assessment endpoints.
How does EPA typically address variability and uncertainty?
Because EPA cannot perform a time- and resource-intensive risk assessment for every situation and
decision, the Agency must be strategic in determining when more intensive assessments are
needed. When EPA does not explicitly quantify the degree of confidence in a risk estimate, the
Agency attempts to increase confidence that risk is not being underestimated by applying various
assumptions to address uncertainty and variability. As depicted in Equation 1. methods that rely on
a combination of point estimates—some conservative (high parameter values that are more likely
to overestimate risk) and some that are typical or average—usually multiply the point estimates to
achieve a single estimate (e.g., conservative point estimates yield a conservative estimate, average
point estimates yield an estimate for an average or typical individual).
Equation 1. Deterministic Risk Assessment
_ — p ^ p 1 F 1
F Concentration I I Exposure I I Ingestion or I I Toxicity I = otcif
Concentration Inhalation Rate X Factor
I in environment! ^ [_ JL JL J
Yields a reasonable estimate for
average or typical individual
Central tendency (average) r^i v Tel v HI X Ttl =
values for all parameters LCJ X L J X LJ LJ
Risk
High-end values for FQI X [El X [|1 X [T] =
L J
some or all parameters L J Yje|ds estimate that js |ike|y
biased high (conservative)
-------
Planning and Scoping
• Purpose • Approach
• Scope • Resources
• Participants • Past Experiences
->
<-
A
>
Problem Formulation
Conceptual Model
• Sources
« Stressors
• Pathway/Routes
Analysis Plan
• Methods
* Models
• Receptors
• End points
• Data Gaps
• Uncertainties
f
Integration of Exposure, Hazard and Dose-Response Information Considering:
Time-Related Aspects, Vulnerability and Subpopulations with Special exposure
Single Stressor Information
• Toxicologic Independence
• Toxicologic Similarity
>
f
Multiple Stressor Information
• Stressor Interactions
• Joint Chemical Toxicity
Measures and metrics
• Decision indices * Common Metric
• Probabilistic Approaches • Biomarkers
* Quantitative Approaches
Risk Description
• Central Tendency and High-End Individual Risk
• Population Risk
• Risk to Important Subpopulations
Uncertainty Analysis
• Being Explicit About Uncertainty
• Uncertainty and Variability
• Sensitivity Analysis
A
\
f
Information Provided by Risk Assessment
Figure 3. General Phases of the Risk Assessment Process. Risk assessment is an iterative
process comprised of planning, scoping and problem formulation; analysis (e.g., hazard
identification, dose-response assessment and exposure assessment); interpretation and risk
characterization; and risk communication.
-------
This approach typically produces a single estimate of risk (e.g., a 1 x 10-6 excess cancer risk, which
is an increased risk of one in 1 million additional cancers) along with a qualitative discussion of
uncertainty. These are generally referred to as "deterministic assessments" or "point estimates" of
risk because all the elements of the calculation are assumed to be determined without variation at
single values. Point estimates are useful, particularly in screening assessments with either cancer or
noncancer endpoints, but the inherent variability and uncertainties are not quantified fully. These
inherent limitations can affect EPA decisions in the following ways:
n The inability to characterize explicitly the basic elements of EPA decisions—whom or what
is being protected or with what degree of confidence.
n The inability to capture more realistically the risk range instead of a point estimate.
n The inability to compare accurately the options (e.g., cleanup levels, permit levels,
regulations, actions) due to the differing levels of conservatism implicit in the individual
estimates.
n The decreased ability to make tradeoffs or reach an appropriate balance between benefits
and costs.
n Use of a point estimate draws criticism and engenders debate for being conservative and
unrealistic or for providing inadequate protection. These criticisms may not be answered
readily due to limitations in either the methods or data used.
n Point estimates do not permit measurement of uncertainty because they are presented as a
single "true" value, rather than a range of possible values.
How does PRA address variability and uncertainty?
PRA uses distributions of values that, for variability, reflect variations in the real world, and for
uncertainty, reflect lack of knowledge regarding parameters, models and/or scenarios. The result is
an overall probability statement of the risk; for example, what the risks are to the average or mean
individual and the high-end individual, such as the 95th percentile, illustrated in Equation 2. The
hypothetical example illustrates the approach for estimating individual risk. In some cases,
exposure and health risk are estimated for an entire population or among susceptible populations
and lifestages.
Equation 2. Probabilistic Risk Assessment
[Concentration! v [Exposure] [ "nation or 1 [joxicity 1
in environment] X l^"*0" J X [ J L **** \
= RISK
Mean risk
1 ,
X
_L
X
1,
X
^ —
population
95th %
population
level risk
risk
-------
PRA could be applied to one or more phases of the risk assessment paradigm, including hazard
characterization as well as exposure, toxicity and/or risk assessment. In the past EPA has focused
probabilistic techniques on predictions of exposures. Probabilistic techniques can also be used
during the effects assessment phase to address variability/uncertainty in data and activities that
include the following:
n Hazard identification;
n Assessment of dose-response relationships (e.g., a probabilistic reference dose (RfD) could
help reduce potential inaccuracies in the estimation of risk close to or below the RfD);
n Evaluation of health effects data; and
n Consideration of differential vulnerabilities and sensitivities within a population.
How can enhanced characterization of uncertainty and variability help inform decisions?
Pertaining to input parameter, model and scenario uncertainties, there are several ways by which
various types of PRA can inform risk management decision making. First, a sensitivity analysis can
determine if more refined information about the distribution and range of data would have a
substantial effect on the assessment of risk and management options. This analysis will help
determine how robust a decision may be with current information and/or identify which
parameters having further refined information would most benefit the decision. Second, PRA can be
used to describe uncertainty in the risk assessment, increase the transparency of the inputs to the
decision and assist in selecting among various management options. In addition, PRA can do the
following:
n Enhance EPA decisions by providing more information about the possible impacts of
alternative decisions.
n Provide clarity on whom or what we are protecting and the degree of confidence that can be
placed on the estimates of protection provided by a given decision.
D Allow for a more detailed comparison of alternative risk management options in terms of
the estimated impacts on both protection and costs.
D Characterize the inherent uncertainties and impact of those uncertainties on the decision
and improve the overall confidence in specific decisions.
n Support the understanding of the cost, time and feasibility of obtaining improved
information on critical uncertainties.
The NRC recommended that EPA adopt a "tiered" approach to select the level of detail used in
uncertainty analysis and variability assessments (NRC 2009). Furthermore, it recommended that a
discussion of the level of detail needed for uncertainty analysis and variability assessment should
be an explicit part of the planning, scoping and problem formulation phase of the risk assessment
process. The companion document titled Risk Assessment Forum White Paper: Probabilistic Risk
Assessment Methods and Case Studies (USEPA 2014a) provides a detailed discussion of PRA and
EPA's experience with risk assessments, highlighting the steps in the process during which PRA can
be applied to characterize uncertainty and variability, including:
D Planning, scoping and problem formulation (e.g., developing an environmental sampling
plan that considers cost, timeliness and the elimination of sampling bias).
D Analysis (e.g., considering uncertainty and variability in exposure factors, exposure
scenarios, environmental concentrations of agents and dose-response relationships;
10
-------
assessing who is exposed; modeling and evaluating whether further planning, scoping and
problem formulation is required).
n Interpretation and risk characterization (e.g., describing variability in risk estimates
because of variability of geographic conditions and exposure within a population,
conducting an uncertainty analysis by performing a sensitivity analysis, estimating
uncertainty and identifying sources of uncertainty).
n Risk communication (e.g., being explicit about variability and uncertainty; evaluating
whether further risk characterization is required).
n Decision making (e.g., prioritizing data needs, evaluating whether further risk
communication or analysis is required).
What is the impact of variability and uncertainty on making decisions?
Variability and uncertainty are inherent in risk assessments. Where data and information are
incomplete or are inadequate, making informed decisions is more difficult, and without considering
uncertainty in the state of knowledge or the impact of choices made (such as in selecting data,
models, scenarios or framing the problem), there exists greater potential for making poorly
informed decision choices. In the case of environmental regulations, specific decisions may lead to
over- or under-regulation compared with decisions that could be made with perfect information.
Setting an environmental standard that is too lax may threaten public health, whereas a standard
that is unnecessarily stringent may impose a significant economic cost for a marginal gain in public
health and environmental protection.
The use of point estimates in decision making might convey an unfounded high degree of precision.
A probabilistic approach can be applied rather than defining a single value and might be less likely
to imply undue precision. The display of results as a distribution can convey that no particular
value carries specific risks of adverse effects; it also can support the understanding of the
magnitude and likelihood of risk of adverse impact for a range of exposures, not just an individual
point estimate of dose or exposure.
Is uncertainty and variability limited only to risk assessment?
PRA can also help to inform decisions by forcing delineation of the uncertainty present and
providing evaluation measures of outcomes (often referred to as a decision analysis). Uncertainties
often are represented as probabilities or probability distributions in graphs or numerical formats.
This can be very useful in evaluating and communicating the risk management alternatives
available to the decision maker.
In decision making, uncertainty and variability are not limited to data, models and scenario
construction and how these may affect the preference of one decision alternative over another. A
decision maker may be faced with combining many different decision criteria requiring the use of
weights or values (e.g., what is more important and by how much?) to choose a decision alternative.
The use and combination of decision criteria may introduce other uncertainties and variability. The
combined impact of data/model/scenario selection, expert judgment and weighting/values on the
preference of decision alternatives falls under the broad umbrella of the field of decision analysis.
By fully understanding the ways in which all types of uncertainty influence the information used in
developing a decision, managers can be more confident that their decision will produce the results
that they seek as well as be better able to defend their decisions and explain how the chosen
alternative meets Agency and stakeholder goals. This summary is focused primarily on addressing
uncertainty and variability related to data/inputs, models and scenarios. Uncertainty and
11
-------
associated with decision criteria values and weights are beyond its scope. Although work in this
aspect of decision making at EPA has been limited, some examples are available at EPA's Multi-
Criteria Integrated Resource Assessment (MIRA) website
[http://www.epa.gov/reg3esdl/data/mira.htm].
What key questions related to uncertainty and variability are asked or considered by
decision makers?
The PRA Technical Panel conducted several dialogues with EPA decision makers, asking them what
questions arise when they are faced with the task of making decisions in the presence of
uncertainty. The following questions represent typical concerns when addressing different types of
uncertainty and variability:
n Would my decision be different if the data were different, improved or expanded? Would
additional data collection and research likely lead to a different decision? How long will it
take to collect the information, how much would it cost, and would the resulting decision be
significantly altered?
n What are the liabilities and consequences of making a decision under the current level of
knowledge and uncertainty?
n How do the alternatives and their associated uncertainty and variability affect the target
population or lifestage?
n How representative or conservative is the estimate due to data, method or scenario
uncertainty?
D What are the major gaps in knowledge, and what are the major assumptions used in the
assessment? How reasonable are the assumptions?
D Can a probabilistic approach (e.g., to better characterize variability and uncertainty) be
accomplished in a timely manner?
What is the desired percent!le of the population to be protected? By choosing this
percentile, who may not be protected? How can addressing uncertainty and variability
using PRA help inform decisions?
PRA can provide information to decision makers on specific questions related to variability and
uncertainty.
n Characterization of the uncertainty in estimates (i.e., what is the degree of confidence in an
estimate?). Could the estimate be off by a factor of 2, a factor of 10 or a factor of 1,000?
D Critical parameters and assumptions that most affect or influence a decision and the risk
assessment.
O "Tipping points" where the options chosen would be altered if the risk estimates were
different, or if a different assumption was valid.
n Estimate the likelihood that values for critical parameters will occur.
n Test the validity of assumptions.
D Estimate the degree of confidence in a particular decision and/or the likelihood of specific
decision errors.
n The possibility of alternative outcomes with additional information, or estimate tradeoffs
related to different risks or decisions.
12
-------
PRA can provide information on the impact of additional information on decision making,
considering the cost and time to obtain the information and the potential for change in the decision
(i.e., the value of the information). For consideration of variability, PRA can provide the following
types of information for exposures:
n Explicitly defined exposures for various populations or lifestages (i.e., who are we trying to
protect?). That is, will the regulatory action keep 50 %, 90 %, 99.9 % or some other fraction
of the population below a specified exposure, dose or risk target?
n Variability in the exposures, among various populations or lifestages, and information on
the percentile of the population that is being evaluated in the risk assessment (e.g.,
variations in the number of liters of water per kilogram [kg] body weight per day consumed
by the population). This information is helpful in addressing:
• The conservatism of EPA's risk assessments;
• Concerns about whether particular exposures were evaluated in the risk
assessment;
• Whom or what is being protected by a management action; and
• Whether and what additional research may be needed to reduce uncertainty.
A few hypothetical examples of the types of risk assessment questions that can be addressed
explicitly through PRA are illustrated in Figure 4.
Question B: What is the risk for an
individual experiencing high-end risk
impacts in the population (e.g., 95%ile)?
Question A: What is the risk
for a typical (median or average)
individual in the population?
Question C: What fraction of the
population is likely to experience at
least a specific degree of risk
(e.g., 1 in a million cancer)?
low
Risk Level
high
Figure 4. What Questions Can PRA Address? PRA can be used to determine the likelihood of
exposure or risk in a specific fraction of the population.
13
-------
What are some of the limitations of and challenges for PRA?
Data and/or resources may not be available to support probabilistic techniques at all stages in the
same assessment, requiring the risk assessor to continue to apply some deterministic science-
policy assumptions and conversions. If science-policy assumptions or default values for parameters
are applied to an assessment that includes PRA, they should be articulated clearly in the
dissemination of results. PRA typically requires more time to develop than a deterministic
assessment, but these techniques can fit into a graduated or tiered approach to risk analysis.
Additional limitations/challenges include the following:
n PRA may be more data intensive than deterministic approaches, requiring additional time
and financial and analytic resources to obtain the necessary statistical distribution input
data for each aspect of the risk assessment More routine incorporation of probabilistic
designs in risk assessment and its supporting research could reduce this cost differential.
n To date, PRA techniques have been used by EPA scientists most successfully in the exposure
aspect of human health risk assessment Although PRA can be used to characterize the
uncertainty and variability in situations with limited data, experience is needed to use PRA
to characterize the range of effects or dose-response relationships for populations,
including sensitive populations and lifestages. More work is required to develop these
methods for other types of analyses.
D The dissemination of a statistical distribution or probability output number should be
related carefully to the quality and coverage of the input data; otherwise, the PRA results
could lead to a false sense of precision.
D PRA assessments that are not transparent may be used inappropriately to manipulate
results and obfuscate the bases of decisions. It should be noted that this may occur with
deterministic assessments as well.
What is EPA's experience with PRA?
Recent EPA experience with PRA is primarily limited to analyzing data, model and scenario
uncertainties and variability. In the past, however, EPA usually—but not always—relied on
deterministic or point estimates to evaluate risk (e.g., cancer risk of 1 xlO6 excess cancers or one in
1 million). The use of PRA to evaluate uncertainty and variability in risk assessments is increasing.
These efforts are varied across program offices and regions, both in complexity and in applications.
Many PRA applications focused on specific elements of a risk assessment (e.g., exposure), variability
or uncertainty. Examples are provided in the Appendix of the Risk Assessment Forum White Paper:
Probabilistic Risk Assessment Methods and Case Studies (USEPA 2014a), the companion publication
to this FAQ document The white paper contains 16 case studies (11 HHRAs and 5 ERAs) that detail
PRAs that have been conducted to support regulatory decisions and/or regulatory impact analyses.
Examples of EPA's use of deterministic risk assessment and PRA are illustrated in Table 1. which
highlights 6 of the 16 case studies in the companion document These examples apply different PRA
tools in risk assessment. The case studies range from less resource-intensive analyses to more
detailed and resource-intensive approaches.
14
-------
Table 1. Selected Examples of EPA Applications of Risk Assessment Techniques from Deterministic to
More Complex Analysis
Case
Study No.
2*
5t
7t
9t
Description
Atmospheric Deposition to Watershed
Contamination: EPA's Office of Research and
Development (ORD) developed an analysis of nitrogen,
mercury and polynuclear aromatic hydrocarbons (PAHs)
in the Casco Bay Estuary in southwestern Maine.
Hudson River Polychlorinated Biphenyl (PCB)-
Contaminated Sediment Site: EPA Region 2 evaluated
the variability in risks to anglers who consume
recreationally caught fish contaminated with PCBs from
sediments in the Hudson River.
Environmental Monitoring and Assessment Program
(EMAP): ORD developed and the Office of Water (OW)
applied probabilistic sampling techniques to evaluate the
nation's aquatic resources under the Clean Water Act
(CWA) Section 305(b).
Chromated Copper Arsenate (CCA) Risk Assessment:
ORD and the Office of Pesticide Programs (OPP)
conducted a probabilistic exposure assessment of
children's exposure (addressing both variability and
uncertainty) to arsenic and chromium from contact with
CCA-treated wood play sets and decks.
Type of Analysis
(Application)
Deterministic
Analysis (Nitrogen
Deposition Rate)
1 -Dimensional
Monte Carlo
Analysis (Exposure
Factors)
Probabilistic
Sensitivity Analysis
(Environmental
Sampling Plan)
2-Dimensional
Monte Carlo
Analysis (Exposure
Scenarios)
Factors Informing
Decisions
Cost-Effectiveness,
Timeliness
Exposure Variability
Within a Population
Representative
Results, Cost-
Effectiveness,
Elimination of
Sampling Bias
Exposure Variability,
Sensitivity Analysis,
Prioritization of Data
Needs
Type of Risk
Assessment
Ecological
Human Health
Ecological
Human Health
Program/
Region
ORD
Superfund/
Region 2
(New York)
ORD/OW
ORD/OPP
U.S. EPA Internal Review Draft
15
Do Not Cite - Quote - or Distribute
-------
Table 1. Selected Examples of EPA Applications of Risk Assessment Techniques from Deterministic to
More Complex Analysis
Case
Study No.
13t
14T
Description
Evaluating Ecological Effects of Pesticide Uses: OPP
developed a probabilistic model that evaluates acute
mortality levels in generic and specific ecological species
for user-defined pesticide uses and exposures.
Fine Particulate Matter (PM2.s) Health Impacts: ORD
and the Office of Air and Radiation (OAR) used expert
elicitation to characterize more completely, both
qualitatively and quantitatively, the uncertainties
associated with the relationship between the reduction in
particulate matter (PIVb.s) and benefits of reduced PlVb.s-
related mortality.
Type of Analysis
(Application)
Probabilistic
Analysis (Exposure,
Dose-Response)
Expert Elicitation
(Dose-Response)
Factors Informing
Decisions
Variability of
Geographic
Conditions,
Prioritization of Data
Needs
Fcitim^tinn nf
Uncertainty,
Identification of
Sources of Uncertainty
Type of Risk
Assessment
Ecological
Human Health
Program/
Region
OPP
ORD/OAR
* Deterministic risk assessment
T Probabilistic risk assessment
U.S. EPA Internal Review Draft
16
Do Not Cite - Quote - or Distribute
-------
What are other EPA publications on PRA?
EPA's experience with PRA includes not only individual assessments or applications, but also the
development of general guidance and policies related to data, model and scenario uncertainty, such
as the following documents:
n Policy for Use of Probabilistic Analysis in Risk Assessment at the U.S. Environmental Protection
Agency (\JSEPA1997c);
d Guiding Principles for Monte Carlo Analysis (USEPA 1997b); and
n Risk Assessment Guidance for Superfund, Volume HI—Part A, Process for Conducting
Probabilistic Risk Assessment (USEPA 2001).
When should one consider using PRA?
It is appropriate to consider PRA any time there is an expectation that the quantification of
uncertainty and/or variability might influence or impact a decision. A sensitivity analysis for the
decision can help managers determine whether having such information is critical and whether the
time and resources needed for PRA are warranted. PRA may not be needed when the decision is
routine, legislatively mandated or if a standard methodology is prescribed. Furthermore, PRA may
not be needed when there is high confidence in the data and models used to support the decision.
Planning and scoping discussions are needed between the risk assessors and the risk managers;
these are described in EPA's Framework for Human Health Risk Assessment to Inform Decision
Making (USEPA,2014b), Ecological Risk Assessment Framework (USEPA 1992), Guidelines for
Ecological Risk Assessment (USEPA 1998), and Framework for Cumulative Risk Assessment (USEPA
2003). An analysis plan developed during the planning and scoping phase may indicate that
information from a PRA could significantly inform the decision. Some examples are:
D A specified target level of protection in a population is identified by the manager (e.g., the
95th percentile), and it is necessary to demonstrate that this goal is met
n Significant equity or environmental justice issues are raised by variation in risks among the
exposed population of concern.
n Screening-level point estimates of risk are higher than an accepted level of concern.
D Uncertainty in some aspect of the risk assessment is high, and decisions are contentious or
have large resource implications.
D Specific critical risk estimates and assumptions point to different management options.
n The scientific rigor and quality of the assessment is critical to the credibility of the EPA
decision.
D When a screening-level DRA indicates that risks are possibly higher than a level of concern
and a more refined assessment is needed.
D When the consequences of using point estimates of risk are unacceptably high.
n When significant equity or environmental justice issues are raised by interindividual
variability.
n When exploring the impact of the probability distributions of the data, model and scenario
uncertainties as well as variability together to compare potential decision alternatives.
U.S. EPA Internal Review Draft 17 Do Not Cite - Quote - or Distribute
-------
PRA is useful especially if the decision process is iterative and adaptive, allowing for detailed and
targeted analysis over time. If the decision has a single decision point or separate single steps in
making a decision that will not be repeated or revised, however, PRA should be used with caution
as it may be used to manipulate results if it is not designed properly and transparently.
PRA is not necessarily appropriate when regulatory and legislative mandates define the assessment
(e.g., "individual most exposed"), which may minimizes the issue of variability. An uncertainty
analysis would still be appropriate.
What is the right level of analysis?
The NRC (2009) recommends a tiered approach to risk assessment using both qualitative and
quantitative (deterministic and probabilistic) tools, with the complexity of the analysis increasing
as progress is made through the tiers. The way that PRA fits into a graduated hierarchical (tiered)
approach is described more fully below and is illustrated in Figure 5. The three tiers in the figure
roughly correspond to the three groups of EPA case studies that use PRA tools, described in Table
A-l of the Appendix of this publication's companion document titled Risk Assessment Forum White
Paper: Probabilistic Risk Assessment Methods and Case Studies (USEPA 2014a).
There is no single "right" level of analysis. Instead, the appropriate level of analysis is guided by the
nature of the decision and applicable regulations. As is the case for risk assessment in general,
approaches to PRA and specific analytical methods may vary dramatically in their complexity and
resource implications. The concept of iterative or tiered analyses, depicted in Figure 5. is widely
accepted in risk assessment, and this applies to PRA as well. Higher tiers could reflect increasing
complexity and generally will require more time and resources. An analysis typically might start at
a lower tier (such as screening for prioritization) and only progress if there is a need for a more
sophisticated assessment commensurate with the importance of the problem. Higher tiers also
reflect increasing characterization of variability and/or uncertainty in the risk estimate, which may
be important for decision making. There is a wide range of PRA methods and approaches of varying
complexity and rigor that can be applied for different purposes, ranging from sensitivity analysis to
integrated analysis of uncertainty and variability. The goal is to choose a level of detail and
refinement for an analysis appropriate to the overall objectives of the decision and types of
available data and analyses needed to support decisions. In the planning and scoping and problem
formulation phases, it is critical to have early and continued dialogue between the manager and risk
assessor to develop a clear understanding of the overall project objectives, the needs of the decision
maker, the timing of the decision and how PRA may play a role in the decision-making process. The
use of formal decision analytic approaches, although not often used in the past, can provide insight
to decision makers about the type and level of analyses required to answer their questions.
U.S. EPA Internal Review Draft 18 Do Not Cite - Quote - or Distribute
-------
I
c
1
tf
*1 1
"x * 5
(_j
f"
s
sf
k Tier 3: Advanced PRA (High Complexity) J
Probabilistic techniques (e.g., 2-D Monte Carlo Analysis)
+ Probabilistic exposure assumptions
•*• Detailed, site-specific analysis and/or modeling
i
k
Decision-
^ ari&quaci
value of:
effort
Tier 2: PRA (Moderate Complexity)
+ Realistic exposure assumptions
+ More detailed analysis and/or modeling
i
making cycle: Evaluating the
of the risk assessment and the
dditional complexity/level of
k
Decision-making cycle: Evaluating the
t adequacy of the risk assessment and
the value of additional complexity/level
of effort
Tier 1 : Deterministic (Point Estimate) RA
(Screening Level)
+ Conservative exposure assumptions
+ Simple analysis and/or modeling
i
k
Planning and Scoping/Problem Formulation
Increasing Regulatory Significance
Greater Decision-Making Needs
Figure 5. Tiered Approach for Risk Assessment. The applicability of a probabilistic
approach depends on the needs of decision makers and potential stakeholders.
Assessments that are high in complexity and regulatory significance benefit from the
application of probabilistic techniques. Source: Adapted from USEPA 2004a and WHO
2008.
Some of the considerations in determining the type and level of analyses that should be performed
include:
1. Whether or not the risk assessment in its current state is sufficient to discriminate among
options (i.e., a clear path to exiting the process is available).
2. If the assessment is determined to be insufficient, whether or not progression to a higher
level of complexity would provide a sufficient benefit to warrant the additional effort of
performing a PRA.
3. Whether there are significant differences in costs and/or benefits between the alternatives.
Some of the EPA case studies provided in the Appendix of this summary's companion document
titled Risk Assessment Forum White Paper: Probabilistic Risk Assessment Methods and Case Studies
(USEPA 2014a) offer examples of how PRA tools can be used in an iterative or tiered approach in a
U.S. EPA Internal Review Draft
19
Do Not Cite - Quote - or Distribute
-------
risk assessment Both Case Studies 1 and 9 deal with children's exposure to arsenic in chromated
copper arsenate (CCA) pressure-treated wood. Case Study 1 provides an example of the application
of point estimate sensitivity analysis (Tier 1) to identify important variables for population
exposure variability for Case Study 9. In Case Study 9, a two-dimensional Monte Carlo analysis (2-D
MCA) was conducted to address the variability and uncertainty in the exposure assessment Two
stages of random sampling were conducted to distinguish between variability and uncertainty in
exposure and toxicity variables.
If one is going to use PRA, what factors should be considered?
If one decides that the use of PRA would provide valuable information in support of a decision,
some other things to consider in moving forward include the following:
D Resources needed to develop the PRA and review the document, including:
• Expertise of EPA staff to develop a PRA or review a PRA submitted by a contractor or
member of the regulated community.
• Time needed for the development and review of the analyses.
• Funding, either intramural or extramural, that may be necessary for development and
review of the PRA.
n Data availability and format (e.g., electronic or paper copy) for developing distributions to
include in the PRA.
D Peer review, including either internal and/or external review, which has time and cost
implications.
D Communication of results to the scientific community, Agency executives, stakeholders
and the public. PRA may often increase the complexity of the risk information to be
communicated to various audiences.
What are the resources needed to conduct a PRA?
PRA generally requires more time, effort and resources than deterministic assessments with
standard science-policy assumptions. The resources will vary depending on the tool or approach
that is selected. There is a continuum of PRA methodologies to choose from, ranging from simple
sensitivity analyses to complex approaches such as 2-D Monte Carlo Analysis. In some cases, simple
sensitivity analyses, which may require limited time and risk assessor resources, can be conducted
"in-house." More sophisticated analyses may require specific expertise or the use of specific tools or
models not available at EPA; maintaining transparency will be important in these efforts. Proper
application of probabilistic methods requires not only software and data, but also guidance and
training for analysts using the tools as well as for managers and decision makers tasked with
interpreting and communicating the results. The development of standardized approaches and/or
methods can lead to the routine incorporation of PRA in Agency approaches and greatly reduce
costs in future applications. Greater development and understanding of the decision context to
which potential PRA might contribute will allow decision makers to make a more informed decision
about the benefits versus the costs of conducting a PRA relative to their decision-making goals.
Does PRA require more data than conventional approaches?
In general, PRA requires more data than conventional approaches because distributions of values
rather than single values are used. Minimum data requirements currently are a topic of debate in
the broader risk assessment community. Minimum data needs vary depending on the analytical
U.S. EPA Internal Review Draft 20 Do Not Cite - Quote - or Distribute
-------
approach used; empirical-based (observational or frequentist) methods have significant data
requirements compared with subjective methods. Some of the data that would be applied in a
frequentist approach may be available already as part of the underlying data set used in standard
deterministic analyses. As a result, PRA can be applied in most cases provided that the methods
used are appropriate for the available body of evidence and data.
Communication of PRA results to the manager and community: does the presentation of
results matter?
The lack of familiarity with PRA presents a challenge in effectively presenting results to decision
makers, stakeholders and the public. Many view PRA as a highly technical discipline that uses
sophisticated mathematics and requires extensive training to apply and understand. Single point
estimates are easier to grasp for most people, based in part on familiarity with this approach over
the history of EPA. Although some people initially have difficulty interpreting probability
distributions of values, everyone has a common baseline experience with probability, uncertainty
and variability from everyday life (e.g., weather forecasting, odds of winning a lottery), and this
experience could be used to frame the discussion of results. It is not necessary to understand the
underlying mathematics or even to include results as full distributions. Results can be distilled
down to the critical essence or decision-meaningful input of interest.
The different audiences and their range of knowledge and expertise must be considered in
developing materials for effective communication. When a decision is made to conduct a PRA, it is
helpful to consider early orientation of the community, managers and others in the basic principles
before the final decision is presented. Alternatively, it may be helpful to present the results of the
PRA along with the point estimate to provide a contextual frame for the results.
How can I get more information on PRA?
This document provides a general overview and basic concepts to establish some familiarity and a
foundation for further education on PRA. The companion publication to this FAQ document titled
Risk Assessment Forum White Paper: Probabilistic Risk Assessment Methods and Case Studies (USEPA
2014a) provides a detailed discussion of PRA and EPA's experience with it. There are numerous
additional resources to provide more detail about PRA, including EPA's Risk Assessment website
(www.epa.gov/risk]. which is the Agency's source for links to risk assessment methods and
policies.
The RAF continues to have ongoing activities addressing the uncertainty and variability needs of
the Agency; details can be found on the RAF's website (www.epa.gov/rafj.
U.S. EPA Internal Review Draft 21 Do Not Cite - Quote - or Distribute
-------
GLOSSARY
Analysis. Examination of anything complex to understand its nature or to determine its essential
features (WHO 2004).
Assessment. Interpretation and evaluation of Environmental Monitoring and Assessment Program
(EMAP) results for the purpose of answering policy-relevant questions about ecological resources,
including (1) determination of the fraction of the population that meets a socially defined value, and
(2) association among indicators of ecological conditions and stressors (USEPA 2010a).
Assessment endpoint. An explicit expression of the environmental value that is to be protected,
operationally defined by an ecological entity and its attributes. For example, salmon are valued
ecological entities; reproduction and age class structure are some of their important attributes.
Together, salmon "reproduction and age class structure" form an assessment endpoint (USEPA
1998).
Cumulative Distribution Function. In probability theory and statistics, the cumulative distribution
function (CDF) describes the probability that a real-valued random variable X with a given
probability distribution will be found to have a value less than or equal to x.
Deterministic. A methodology relying on point (i.e., exact) values as inputs to estimate risk; this
obviates quantitative estimates of uncertainty and variability. Results also are presented as point
values. Uncertainty and variability may be discussed qualitatively or semi-quantitatively by
multiple deterministic risk estimates (USEPA 2006b).
Deterministic risk assessment (DRA). Risk evaluation involving the calculation and expression of
risk as a single numerical value or "single point" estimate of risk, with uncertainty and variability
discussed qualitatively (USEPA 2012).
Ecological risk assessment. The process that evaluates the likelihood that adverse ecological
effects may occur or are occurring as a result of exposure to one or more stressors (USEPA 1998).
Ecosystem. The biotic community and abiotic environment within a specified location in space and
time (USEPA 1998).
Environment. The sum of all external conditions affecting the life, development and survival of an
organism (USEPA 2010a).
Expert elicitation. A systematic process of formalizing and quantifying, typically in probabilistic
terms, expert judgments about uncertain quantities (USEPA 2011a).
Frequentist (or frequency) probability. A view of probability that concerns itself with the
frequency with which an event occurs given a long sequence of identical and independent trials
(USEPA 1997b).
Hazard identification. The risk assessment process of determining whether exposure to a stressor
can cause an increase in the incidence or severity of a particular adverse effect, and whether an
adverse effect is likely to occur (USEPA 2012).
Human health risk assessment (HHRA). I. The process to estimate the nature and probability of
adverse health effects in humans who may be exposed to chemicals in contaminated environmental
media, now or in the future (USEPA 2010b). 2. The evaluation of scientific information on the
hazardous properties of environmental agents (hazard characterization), the dose-response
relationship (dose-response assessment), and the extent of human exposure to those agents
(exposure assessment). The product of the risk assessment is a statement regarding the probability
U.S. EPA Internal Review Draft 22 Do Not Cite - Quote - or Distribute
-------
that populations or individuals so exposed will be harmed and to what degree (risk
characterization) (USEPA2006a).
Inputs. Quantities that are applied to a model (WHO 2008).
Model. A mathematical representation of a natural system intended to mimic the behavior of the
real system, allowing description of empirical data, and predictions about untested states of the
system (USEPA 2006b).
Modeling. Development of a mathematical or physical representation of a system or theory that
accounts for all or some of its known properties. Models often are used to test the effect of changes
of components on the overall performance of the system (USEPA 2010a).
Monte Carlo analysis (MCA) or simulation (MCS). A repeated random sampling from the
distribution of values for each of the parameters in a generic exposure or risk equation to derive an
estimate of the distribution of exposures or risks in the population (USEPA 2006b).
One-dimensional Monte Carlo analysis (1-D MCA). A numerical method of simulating a
distribution for an endpoint of concern as a function of probability distributions that characterize
variability or uncertainty. Distributions used to characterize variability are distinguished from
distributions used to characterize uncertainty (WHO 2008).
Parameter. A quantity used to calibrate or specify a model, such as "parameters" of a probability
model (e.g., mean and standard deviation for a normal distribution). Parameter values often are
selected by fitting a model to a calibration data set (WHO 2008).
Probability. A frequentist approach considers the frequency with which samples are obtained
within a specified range or for a specified category (e.g., the probability that an average individual
with a particular mean dose will develop an illness) (WHO 2008).
Probability Density Function. In probability theory, a probability density function (pdf) of a
continuous random variable is a function, often denoted as f(x), that describes the relative
likelihood for this random variable to take on a given value.
Probabilistic risk analysis (PRA). Calculation and expression of health risks using multiple risk
descriptors to provide the likelihood of various risk levels. Probabilistic risk results approximate a
full range of possible outcomes and the likelihood of each, which often is presented as a frequency
distribution graph, thus allowing uncertainty or variability to be expressed quantitatively (USEPA
2012).
Problem formulation. The initial stage of a risk assessment where the purpose of the assessmentis
articulated, exposure and risk scenarios are considered, a conceptual model is developed, and a
plan for analyzing and characterizing risk is determined (USEPA 2004a).
Reference concentration (RfC). An estimate (with uncertainty spanning approximately an order of
magnitude) of a continuous inhalation exposure to the human population (including sensitive
subgroups) that is likely to be without an appreciable risk of deleterious effects during a lifetime. It
can be derived from a No-Observed-Adverse-Effect Level (NOAEL), Lowest-Observed-Adverse-
Effect Level (LOAEL), or benchmark concentration, with uncertainty factors generally applied to
reflect limitations of the data used. It generally is used in EPA's noncancer health assessments
(USEPA 2007a).
Reference dose (RfD). An estimate (with uncertainty spanning approximately an order of
magnitude) of a daily oral exposure to the human population (including sensitive subgroups) that is
likely to be without an appreciable risk of deleterious effects during a lifetime. It can be derived
from a NOAEL, LOAEL or benchmark dose, with uncertainty factors generally applied to reflect
U.S. EPA Internal Review Draft 23 Do Not Cite - Quote - or Distribute
-------
limitations of the data used. It typically is used in EPA's noncancer health assessments (USEPA
2011c).
Risk. 1. Risk includes consideration of exposure to the possibility of an adverse outcome, the
frequency with which one or more types of adverse outcomes may occur, and the severity or
consequences of the adverse outcomes if such occur. 2. The potential for realization of unwanted,
adverse consequences to human life, health, property or the environment. 3. The probability of
adverse effects resulting from exposure to an environmental agent or mixture of agents. 4. The
combined answers to: What can go wrong? How likely is it? What are the consequences? (USEPA
2011c).
Risk analysis. A process for identifying, characterizing, controlling and communicating risks in
situations where an organism, system, subpopulation or population could be exposed to a hazard.
Risk analysis is a process that includes risk assessment, risk management and risk communication
(WHO 2008).
Risk assessment. I. A process intended to calculate or estimate the risk to a given target organism,
system, subpopulation or population, including the identification of attendant uncertainties
following exposure to a particular agent, taking into account the inherent characteristics of the
agent of concern, as well as the characteristics of the specific target system (WHO 2008). 2. The
evaluation of scientific information on the hazardous properties of environmental agents (hazard
characterization), the dose-response relationship (dose-response assessment), and the extent of
human exposure to those agents (exposure assessment) (NRC 1983). The product of the risk
assessment is a statement regarding the probability that populations or individuals so exposed will
be harmed and to what degree (risk characterization; USEPA 2000). 3. Qualitative and quantitative
evaluation of the risk posed to human health or the environment by the actual or potential presence
or use of specific pollutants (USEPA 2012).
Risk-based decision making. A process through which decisions are made according to the risk
each posed to human health and the environment (USEPA 2012).
Risk management. A decision-making process that takes into account environmental laws,
regulations, and political, social, economic, engineering and scientific information, including a risk
assessment, to weigh policy alternatives associated with a hazard (USEPA 2011c).
Scenario. A set of facts, assumptions and inferences about how exposure takes place that aids the
exposure assessor in evaluating, estimating or quantifying exposures (USEPA 1992). Scenarios
might include identification of pollutants, pathways, exposure routes and modes of action, among
others.
Sensitivity analysis. The process of changing one variable while leaving the others constant to
determine its effect on the output This procedure fixes each uncertain quantity at its credible lower
and upper bounds (holding all others at their nominal values, such as medians) and computes the
results of each combination of values. The results help to identify the variables that have the
greatest effect on exposure estimates and help focus further information-gather ing efforts (USEPA
2011b).
Tiered approach. Refers to various hierarchical tiers (levels) of complexity and refinement for
different types of modeling approaches that can be used in risk assessment. A deterministic risk
assessment with conservative assumptions is an example of a lower level type of analysis (Tier 0)
that can be used to determine whether exposures and risks are below levels of concern. Examples
of progressively higher levels include the use of deterministic risk assessment coupled with
sensitivity analysis (Tier 1), the use of probabilistic techniques to characterize either variability or
uncertainty only (Tier 2), and the use of two-dimensional probabilistic techniques to distinguish
U.S. EPA Internal Review Draft 24 Do Not Cite - Quote - or Distribute
-------
between but simultaneously characterize both variability and uncertainty (Tier 3) (USEPA 2004a
and WHO 2008).
Two-dimensional Monte Carlo analysis (2-D MCA). An advanced numerical modeling technique
that uses two stages of random sampling, also called nested loops, to distinguish between
variability and uncertainty in exposure and toxicity variables. The first stage, often called the inner
loop, involves a complete 1-D MCA simulation of variability in risk. In the second stage, often called
the outer loop, parameters of the probability distributions are redefined to reflect uncertainty.
These loops are repeated many times resulting in multiple risk distributions, from which
confidence intervals are calculated to represent uncertainty in the population distribution of risk.
(WHO 2008).
Uncertainty. Uncertainty occurs because of a lack of knowledge. It is not the same as variability.
For example, a risk assessor may be very certain that different people drink different amounts of
water but may be uncertain about how much variability there is in water intakes within the
population. Uncertainty often can be reduced by collecting more and better data, whereas
variability is an inherent property of the population being evaluated. Variability can be better
characterized with more data but it cannot be reduced or eliminated. Efforts to clearly distinguish
between variability and uncertainty are important for both risk assessment and risk
characterization, although they both may be incorporated into an assessment (USEPA 201 Ic).
Uncertainty analysis. A detailed examination of the systematic and random errors of a
measurement or estimate; an analytical process to provide information regarding uncertainty
(USEPA 2006b).
Value of information. An analysis that involves estimating the value that new information can have
to a risk manager before the information is actually obtained. It is a measure of the importance of
uncertainty in terms of the expected improvement in a risk management decision that might come
from better information (USEPA 2001).
Variability. Refers to true heterogeneity or diversity, as exemplified in natural variation. For
example, among a population that drinks water from the same source and with the same
contaminant concentration, the risks from consuming the water may vary. This may result from
differences in exposure (e.g., different people drinking different amounts of water and having
different body weights, exposure frequencies and exposure durations), as well as differences in
response (e.g., genetic differences in resistance to a chemical dose). Those inherent differences are
referred to as variability. Differences among individuals in a population are referred to as inter-
individual variability, and differences for one individual over time are referred to as intra-
individual variability (USEPA 20lie).
U.S. EPA Internal Review Draft 25 Do Not Cite - Quote - or Distribute
-------
REFERENCES4
Howard, R.A. (1966). Decision Analysis: Applied Decision Theory, Proceedings of the Fourth
International Conference on Operational Research, pp. 55-71. Wiley-Interscience.
NRC (National Research Council). 1983. Risk Assessment in the Federal Government: Managing the
Process. Washington, D.C.: The National Academies Press.
NRC. 2006. Health Risks from Dioxin and Related Compounds: Evaluation of the EPA Reassessment.
Washington, D.C.: The National Academies Press.
NRC. 2009. Science and Decisions: Advancing Risk Assessment. Washington, D.C.: The National
Academies Press.
OMB (Office of Management and Budget). 2003. Circular A-4. Washington, D.C.: Office of
Management and Budget, http://www.whitehouse.gov/omb/circulars a004 a-4/.
OMB. 2007. Memorandum for the Heads of Executive Departments and Agencies on Updated
Principles for Risk Analysis. September 19, 2007. M-07-24.
http://www.whitehouse.gov/sites/default/files/omb/memoranda/fy2007/m07-24.pdf.
USEPA (U.S. Environmental Protection Agency). 1992. Framework for Ecological Risk Assessment.
EPA/630/R-92/001. Washington, D.C.: USEPA. http://www.epa.gov/raf/publications/framework-
eco-risk-assessmenthtm.
USEPA. 1997a. Exposure Factors Handbook. EPA/600/P-95/002F a-c. Washington, D.C.: National
Center for Environmental Assessment, Office of Research and Development, USEPA.
http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm7de id=12464.
USEPA. 1997b. Guiding Principles for Monte Carlo Analysis. EPA/630/R-97/001. Washington, D.C.:
Risk Assessment Forum, USEPA. http://www.epa.gov/ncea/pdfs/montcarl.pdf.
USEPA. 1997c. Policy for Use of Probabilistic Analysis in Risk Assessment at the U.S. Environmental
Protection Agency. Memorandum from Fred Hansen, Deputy Administrator. Washington, D.C.:
Science Policy Council, USEPA. http://www.epa.gov/spc/pdfs/probpol.pdf.
USEPA. 1998. Guidelines for Ecological Risk Assessment. EPA/630/R-95/002F. Washington, D.C.:
Risk Assessment Forum, Office of the Science Advisor, USEPA.
http://www.epa.gov/raf/publications/pdfs/ECOTXTBX.PDF.
USEPA. 2000. Science Policy Council Handbook: Risk Characterization. EPA/100/B-00/002.
Washington, D.C.: Office of Science Policy, Office of Research and Development, USEPA.
http://www.epa.gov/spc/pdfs/rchandbk.pdf.
USEPA. 2001. Risk Assessment Guidance for Superfund: Volume HI—Part A, Process for Conducting
Probabilistic Risk Assessment. EPA/540/R-02/002. Washington, D.C.: Office of Emergency and
Remedial Response, USEPA. http://www.epa.gov/oswer/riskassessment/rags3adt/.
USEPA. 2003. Framework for Cumulative Risk Assessment. EPA/630/P-02/001F. Washington, D.C.:
Risk Assessment Forum, Office of the Science Advisor, USEPA.
www.epa.gov/raf/publications/pdfs/frmwrk cum risk assmntpdf
4 All Internet links were checked at the time of publication of this document.
U.S. EPA Internal Review Draft 26 Do Not Cite - Quote - or Distribute
-------
USEPA. 2004a. Air Toxics Risk Assessment Reference Library. Volume 1: Technical Resource Manual.
EPA/453/K-04/001A. Research Triangle Park, NC: Office of Air Quality Planning and Standards,
USEPA. http://www.epa.gov/ttn/fera/data/risk/vol I/title page volume l.pdf.
USEPA. 2QQ4b.An Examination of EPA Risk Assessment Principles and Practices. EPA/100/B-04/001.
Washington, D.C.: Office of the Science Advisor, Office of Research and Development, USEPA.
http://www.epa.gov/osa/pdfs/ratf-final.pdf.
USEPA. 2006a. A Framework for Assessing Health Risks of Environmental Exposures to Children.
EPA/600/R-05/093F. Washington, D.C.: USEPA.
http://ofmpub.epa.gov/eims/eimscomm.getfile7p download id=459047.
USEPA. 2006b. Air Toxics Risk Assessment Reference Library, Volume 3: Community-Scale Assessment.
EPA/452/K-06/001C. Washington, D.C.: USEPA. http://www.epa.gov/ttn/fera/risk atra vol3.html.
USEPA. 2007a. Concepts, Methods and Data Sources for Cumulative Health Risk Assessment of
Multiple Chemicals, Exposures and Effects: A Resource Document. EPA/600/R-06/013F. Washington,
D.C.: USEPA. http://cfpub.epa.gov/ncea/risk/recordisplay.cfm?deid=190187.
USEPA. 2007b. Letter From M. G. Morgan and R. T. Parkin, Science Advisory Board, to S. Johnson,
USEPA. February 28, 2007. EPA/SAB-07/003. Washington, D.C.: Science Advisory Board, Office of
the Administrator, USEPA.
http://yosemite.epa.gov/sab/sabproduct. nsf/55ElB2C78C6085EB8525729C00573A3E/$File/sab-
07-003.pdf.
USEPA. 2010a. Environmental Monitoring and Assessment Program Master Glossary. Last modified
November 8. http://www.epa.gov/emap2/html/pubs/docs/resdocs/mglossary.htmltfee.
USEPA. 2010b. Human Health Risk Assessment Onondaga Lake, Lake Bottom Subsite: Sediment
Consolidation Area, Camillus, NY. Washington, D.C.: USEPA.
http://www.epa.gov/region2/superfund/npl/onondagalake/healthriskassess.pdf.
USEPA. 2011a. Expert Elicitation Task Force White Paper. Washington, D.C.: USEPA.
http://www.epa.gov/stpc/pdfs/ee-white-paper-final.pdf.
USEPA. 2011b. Exposure Factors Handbook: 2011 Edition. EPA/600/R-09/052F. Washington, D.C.:
National Center for Environmental Assessment, Office of Research and Development, USEPA.
http://cfpub.epa. gov/ncea/risk/recordisplay.cfm?deid=23 625 2.
USEPA. 2011c. Integrated Risk Information System Glossary. Last modified August 21.
http://ofmpub.epa.gov/sor internet/registry/termreg/searchandretrieve/glossariesandkeywordli
sts/search.do?details=&glossaryName=IRIS%20Glossary.
USEPA. 2012. Waste and Cleanup Risk Assessment Glossary. Office of Solid Waste and Emergency
Response, USEPA. Last modified December 24.
http://www.epa.gov/oswer/riskassessment/glossary.htm.
USEPA. 2014a. Risk Assessment Forum White Paper: Probabilistic Risk Assessment Methods and Case
Studies. EPA/100/R-09/001A. Washington, D.C.: Risk Assessment Forum, Office of the Science
Advisor, USEPA.
USEPA. 2014b. Framework for Human Health Risk Assessment to Inform Decision Making.
EPA/100/R-14/001. Washington, D.C.: Risk Assessment Forum, Office of the Science Advisor,
USEPA. http://www.epa.gov/raf/frameworkhhra.htm.
U.S. EPA Internal Review Draft 27 Do Not Cite - Quote - or Distribute
-------
WHO (World Health Organization). 2004. International Programme on Chemical Safety Risk
Assessment Terminology. Geneva: WHO.
http://www.who.int/ipcs/methods/harmonization/areas/ipcsterminologypartsland2.pdf.
WHO. 2008. Uncertainty and Data Quality in Exposure Assessment. Geneva: The International
Programme on Chemical Safety, WHO.
http://www.who.int/ipcs/publications/methods/harmonization/exposure assessmentpdf.
U.S. EPA Internal Review Draft 28 Do Not Cite - Quote - or Distribute
------- |