EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Using Probabilistic Methods to Enhance the Role of
Risk Analysis in Decision-Making
With
Case Study Examples
Prepared by Risk Assessment Forum
PRA Technical Panel Working Groups
EPA/100/R-09/001
"THIS INFORMATION IS DISTRIBUTED SOLELY FOR THE PURPOSE OF PRE-DISSEMINATION
PEER REVIEW UNDER APPLICABLE INFORMATION QUALITY GUIDELINES. IT HAS NOT BEEN
FORMALLY DISSEMINATED BY EPA. IT DOES NOT REPRESENT AND SHOULD NOT BE
CONSTRUED TO REPRESENT ANY AGENCY DETERMINATION OR POLICY."
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Disclaimer
This document is a peer review draft. It has not been formally released by the U.S. Environmental
Protection Agency and should not at this stage be construed to represent Agency policy. It is being
circulated for comments on its technical merit and policy implications. Mention of trade names or
commercial products does not constitute endorsement or recommendation for use.
This document was produced by a Technical Panel of the EPA Risk Assessment Forum. The
authors drew on their experience in doing probabilistic assessments and interpreting them to
improve risk management of environmental and health hazards. Interviews, presentations, and
dialogues with risk managers conducted by the Technical Panel have contributed to the insights
and recommendations in this summary and the associated White Papers.
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Foreword
Throughout most of the Environmental Protection Agency's program offices and regions, various
forms of probabilistic methods have been used to answer questions about exposure and risk, to
humans and other organisms, and the environment. EPA risk assessors, risk managers and
others, particularly within the scientific and research divisions have recognized that sophisticated
statistical and mathematic approaches could be still more fully utilized to enhance the quality and
accuracy of Agency risk assessment and risk management. Various stakeholders, inside and
outside the Agency, have called for a more comprehensive characterization of risks, including
uncertainties, in protecting more sensitive or vulnerable populations and life stages.
Therefore, the Office of the Science Advisor of the EPA, together with the Science Policy
Council and members of the Risk Assessment Forum (RAF), identified a need to examine the
use of probabilistic approaches in Agency risk assessment and risk management. An RAF
Technical Panel developed papers (this paper and a managers' summary) which provide a general
overview of the value of probabilistic analyses and similar or related methods, and some
examples of current applications across the Agency.
The goal of these papers is not only to describe potential and actual uses of these tools in the risk
decision process, but also to encourage their further implementation in human, ecological and
environmental risk analysis and related decision making. The enhanced use of probabilistic
analyses to characterize uncertainty in assessments would not only reflect external scientific
advice on how to further advance EPA risk assessment science, but will also help to address
specific challenges faced by managers and improve confidence in Agency decisions
Kevin Teichman
Acting EPA Science Advisor
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Acknowledgments
We would like to acknowledge the scientists and risk assessors who assisted in the preparation of
this paper.
Contributors and Reviewers of this Document
Chris Frey, John Paul, and Pasky Pascual (Leads)
Gary Bangs, Mike Clipper, Kathryn Gallagher, Bob Hetes, Michael Messner, Keeve Nachman,
Haluk Ozkaynak, Zachary Pekar, and Woodrow Setzer
The PRA Technical Panel
The EPA Risk Assessment Forum
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Table of Contents
USING PROBABILISTIC METHODS TO ENHANCE THE ROLE OF RISK ANALYSIS IN
DECISION-MAKING 4
1. INTRODUCTION: RELEVANCE OF UNCERTAINTY TO DECISION MAKING: HOW
PROBABILISTIC APPROACHES CAN HELP 5
1.0. What Is Probabilistic Risk Analysis, and How Does It Address Variability and Uncertainty? 5
1.1. Goals and Intended Audience 5
1.2. Overview of This Document 5
1.3. What Are Common Challenges Facing EPA Risk Decision makers? 6
1.4. What Are Key Questions Often Asked by Decision makers? 7
1.5. Why Is the Implementation of Probabilistic Risk Analysis Important? 7
1.6. How Does EPA Typically Address Scientific Uncertainty and Communicate Variability? 8
1.7. What Are the Limitations of Relying on Default-Based Deterministic Approaches? 9
1.8. What Is EPA's Experience with the Use of Probabilistic Risk Analysis? 10
2. PROBABILISTIC RISK ANALYSIS 12
2.1. What Are Variability and Uncertainty, and How Are They Relevant to Decision-making? 12
2.1.1. Variability 12
2.1.2. Uncertainty 12
2.2. When Is Probabilistic Risk Analysis Applicable or Useful? 13
2.3. How Can Probabilistic Risk Analysis Be Incorporated into Assessments? 13
2.4. What Are the Scientific Community's Views on Probabilistic Risk Analysis, and What Is the
Institutional Support for Its Use? 14
2.5. How Can Probabilistic Risk Assessment Provide More Comprehensive, Rigorous Scientific
Information in Support of Regulatory Decisions? 14
2.6. Are There Additional Advantages of Using Probabilistic Risk Analysis? 15
2.7. What Are the Challenges to Implementation of Probabilistic Analyses? 15
2.8. How Can Probabilistic Risk Analysis Support Specific Regulatory Decision making? 16
2.9. Does Probabilistic Risk Analysis Require More Resources Than Default-Based Deterministic
Approaches? 16
2.10. Doesn't Probabilistic Risk Analysis Require More Data Than Conventional Approaches? 17
2.11. Can Probabilistic Risk Analysis Be Used To Screen Risks or Only in Complex or Refined
Assessments? 17
2.12. Does Probabilistic Risk Analysis Present Unique Challenges to Model Evaluation? 17
2.13. How Do You Communicate Results of Probabilistic Risk Analysis? 18
2.14. Are the Results of Probabilistic Risk Analysis Difficult To Communicate to Decision-makers and
Stakeholders? 19
3. FINDINGS AND RECOMMENDATIONS 21
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
3.1 Findings: How Probabilistic Risk Analysis and Related Analyses Can Improve the Decision-making
Process at EPA 21
3.2. Recommendations for Enhanced Utilization of PRA in EPA 21
3.3 Guidance and Policy 22
3.4 Challenges 23
APPENDIX A: AN OVERVIEW OF SOME OF THE TECHNIQUES USED IN PROBABILISTIC
RISK ANALYSIS 24
A.1. What Is the General Conceptual Approach in Probabilistic Risk Analysis? 24
A.2. What Are the Multiple Types of Probabilistic Risk Analyses, and How Are They Used? 25
A.3. What Are Some Specific Aspects of and Issues Related to Methodology for Probabilistic Risk
Analysis? 27
A.3.1. Developing a Probabilistic Risk Analysis Model 27
A.3.2. Conducting the Probabilistic Analysis 27
APPENDIX B: GLOSSARY 31
APPENDIX C: REFERENCES 36
BIBLIOGRAPHY 38
Probabilistic Risk Analysis Methodology—General 38
Probabilistic Risk Analysis and Decision Making 38
Probabilistic Risk Analysis Methodology—Specific Aspects 38
Sensitivity Analysis 39
Case Study Examples of Probabilistic Risk Analysis—EPA (see also the PRA Case Studies White
Paper) 39
Case Study Examples of Probabilistic Risk Analysis—Other 40
APPENDIX D: CASE STUDY EXAMPLES OF THE APPLICATION OF PROBABILISTIC RISK
ANALYSIS IN U.S. ENVIRONMENTAL PROTECTION AGENCY REGULATORY DECISION-
MAKING 42
1. Introduction 46
2. Overall Approach to Probabilistic Risk Analysis at the U.S. Environmental Protection Agency 47
2.1. U.S. Environmental Protection Agency Guidance and Policies on Probabilistic Risk Analysis 47
2.2. Categorizing Case Studies 47
2.2.1. Group 1 Case Studies 48
2.2.2. Group 2 Case Studies 49
2.2.3. Groups Case Studies 49
3. Case Study Summaries 53
Group 1 Case Studies 53
Case Study 1: Sensitivity Analysis of Key Variables in Probabilistic Assessment of Children's Exposure
to Arsenic in Chromated Copper Arsenate (CCA) Pressure-Treated Wood 53
Case Study 2: Assessment of Relative Contribution of Atmospheric Deposition to Watershed
Contamination 55
Group 2 Case Studies 57
Case Study 3: Probabilistic Assessment of Angling Duration Used in Assessment of Exposure to
Hudson River Sediments via Consumption of Contaminated Fish 57
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 4: Probabilistic Analysis of Dietary Exposure to Pesticides for use in Setting Tolerance
Levels 58
Case Study 5: One-Dimensional Probabilistic Risk Analysis of Exposure to Polychlorinated Biphenyls
(PCBs) via Consumption of Fish from a Contaminated Sediment Site 60
Case Study 6: Probabilistic Sensitivity Analysis of Expert Elicitation of Concentration-Response
Relationship Between Particulate Matter (PM2.5) Exposure and Mortality 64
Case Study 7: Environmental Monitoring and Assessment Program (EMAP): Using Probabilistic
Sampling to Evaluate the Condition of the Nation's Aquatic Resources 66
Group 3 Case Studies 68
Case Study 8: Two-Dimensional Probabilistic Risk Analysis of Cryptosporidium in Public Water
Supplies, with Bayesian Approaches to Uncertainty Analysis 68
Case Study 9: Two-Dimensional Probabilistic Model of Children's Exposure to Arsenic in Chromated
Copper Arsenate (CCA) Pressure-Treated Wood 70
Case Study 10: Two-Dimensional Probabilistic Exposure Assessment of Ozone 72
Case Study 11: Analysis of Microenvironmental Exposures to Particulate Matter (PM2.5) for a
Population Living in Philadelphia, PA 75
Case Study 12: Probabilistic Analysis in Cumulative Risk Assessment of Organophosphorus
Pesticides 77
Case Study 13: Probabilistic Ecological Effects Risk Assessment Models for Evaluating Pesticide Uses
79
Case Study 14: Expert Elicitation of Concentration-Response Relationship Between Particulate Matter
(PM2s) Exposure and Mortality 81
Case Study 15: Expert Elicitation of Sea-Level Rise Resulting from Global Climate Change 83
Case Study 16: Expert Elicitation for Bayesian Belief Network Model of Stream Ecology 85
4. References to Case Studies 87
List of Acronyms and Abbreviations 88
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Using Probabilistic Methods to Enhance the Role of Risk Analysis in
Decision-Making
Executive Summary
Probabilistic risk assessment (PRA) is a group of techniques that provide estimates of the range
and likelihood of hazard, exposure or risk, rather than a single point estimate. Stakeholders,
inside and outside the Agency, have recommended a fuller characterization of risks, including
uncertainties, in protecting more sensitive or vulnerable populations and life stages.
The goal of this white paper is to explain how EPA can achieve broader use of probabilistic
methods and address uncertainty and variability by capitalizing on the wide array of tools and
methods that comprise PRA. The information contained in this document is intended for both
risk analysts and managers faced with determining when and how to apply these tools in their
decisions. This paper begins with a decision-maker's perspective, proceeds to a more technical
discussion, and finally gives a number of illustrative examples of actual EPA applications of
probabilistic analyses.
The white paper describes challenges faced by EPA decision makers, defines and explains the
basic principles of probabilistic analysis, briefly highlights instances where these techniques
have been used in EPA decisions, and describes criteria that may be useful in determining
whether application of probabilistic methods may be useful and/or applicable to a specific
decision. The white paper also describes commonly employed methods to address variability
and uncertainty, including those used in the consideration of uncertainty in scenarios, uncertainty
in models, and variability and uncertainty in the inputs and outputs of models. A general
description is provided of the range of methods from simple to complex, rapid to more time-
consuming, and least to most resource-intensive, and opportunities for utilization. More detailed
examples of applications of these methods are provided, in Appendix D titled "Case Study
Examples of the Application of Probabilistic Risk Analysis in U.S. Environmental Protection
Agency Decision Making."
This document does not prescribe a specific approach but, rather, describes the various stages
and aspects of an assessment or decision process in which probabilistic assessment tools may add
value. This white paper provides answers to common questions regarding PRA, including key
concepts such as scientific and institutional motivations for use of PRA, and challenges in the
application of probabilistic techniques. The white paper describes how PRA can both enhance
the Agency's credibility and improve decision making.
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
1. Introduction: Relevance of Uncertainty to Decision making: How
Probabilistic Approaches Can Help
1.0. What Is Probabilistic Risk Analysis, and How Does It Address Variability and
Uncertainty?
Probabilistic analyses include techniques that can be applied formally to address both variability
and uncertainty. Probability is used in sciences, business, economics, and other fields to examine
existing data and estimate the chance of an event, from health effects to rain to metal fatigue.
One can use probability (chance) to quantify the frequency of occurrence or the degree of belief
in information. For variability, probability distributions are interpreted as representing the
relative frequency of a given state of the system (i.e., that the data are distributed a certain way),
whereas, for uncertainty, they represent the degree of belief or confidence that a given state of
the system exists (i.e., that we have the appropriate data) (Cullen and Frey, 1999). PRA often is
defined narrowly to mean a statistical or thought process used to analyze and evaluate the
variability of available data or to look at uncertainty across data sets.
For the purposes of this document, PRA is a term used to describe a process that uses probability
to incorporate the variability in data sets, and/or the uncertainty in information such as data or
models, into analyses that support environmental risk-based decision making. PRA is used here
broadly to include both quantitative and qualitative methods for dealing with scenario, model,
and input uncertainty. Probabilistic techniques can be used with other types of analysis, such as
benefit-cost analysis, regulatory impact analysis, and engineering performance standards and,
thus, is used for a variety of applications and by experts in many disciplines.
1.1. Goals and Intended Audience
The goals of this white paper are to introduce probabilistic analysis (PRA) and how it can be
used to better inform and improve the decision-making process, and to provide case studies
where it has been used in human health and ecological analyses at EPA (Appendix D). A
secondary goal of this paper is to bridge communication gaps regarding PRA among analysts of
various disciplines, between these analysts and Agency decision makers, and affected
stakeholders. The white paper is also intended to serve as a communication tool to help introduce
key concepts and background information on approaches to risk analysis that incorporate
uncertainty and provide a more comprehensive treatment of variability. Risk analysts, risk
managers, and affected stakeholders can benefit from understanding the potential uses of PRA.
PRA and related approaches can be used to identify further research that can decrease
uncertainty and more thoroughly characterize variability in a risk assessment. This white paper
will explain how PRA is well suited to enhancing the decision-making processes in EPA by
addressing inherent uncertainties faced by managers involved in that process.
1.2. Overview of This Document
This document reviews EPA's interest in and experience with addressing uncertainty and
variability using probabilistic methods; identifies key questions asked or faced by Agency
decision makers; shows how conventional deterministic approaches to risk analysis may not
answer these questions fully; provides examples of applications; and shows how and why
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
"probabilistic risk analysis" (broadly defined) provides added value with regard to regulatory
decision making by more fully characterizing risk estimates. For the purposes of this white
paper, PRA and related tools for both human health and ecological assessments include a range
of approaches from statistical tools, such as sensitivity analysis, to multi-dimensional Monte
Carlo models, geospatial approaches, and expert elicitation. Key points addressed by this
document include definitions and key concepts pertaining to PRA, the need for PRA, benefits
and challenges of PRA, a general conceptual framework for PRA, conclusions regarding
products and insights obtained from PRA, and examples where EPA has used PRA in human
health and ecological analyses. A glossary and a bibliography are provided.
1.3. What Are Common Challenges Facing EPA Risk Decision makers?
EPA decision makers face scientifically complex problems that are compounded by varying
levels of uncertainty and variability. In reality, uncertainty in risk decisions is unavoidable, since
we cannot perfectly model or predict real world situations, but uncertainty can be reduced or
better characterized through knowledge. Variability is inherent in natural systems, and therefore
cannot be reduced, but can also be examined and described. Decision makers often want to know
who is at risk and by how much, the tradeoffs between alternative actions or decisions, and the
likely or possible consequences of decisions. To this end, it is particularly useful to decision
makers to understand the distribution of risk across potentially impacted populations and
ecological systems. It can be important to know the number of individuals experiencing different
magnitudes of risk, the differences in risk magnitude experienced by individuals in different life
stages or populations, or the probability of an event which may lead to unacceptable levels of
risk. Given the limitations of data, traditional methods of risk analyses are not well suited to
produce such estimates. Probabilistic analytical methods are capable of addressing these
shortcomings and can contribute to a more thorough recognition of the impact of data gaps on
the projected risk estimates.
A defensible decision process explicitly takes into account uncertainties and variability and the
rationale or factors influencing how a decision maker addresses these. Factors such as
economics, equity, feasibility, stakeholder input, and other considerations may also be part of the
decision-making process. Decision making typically contend with several key factors, including
multiple, conflicting objectives, uncertainty, and alternative regulatory options available to a
decision maker. In addition, decision analysis provides a theoretical foundation for estimating the
value of collecting more information to allow for more informed decisions. In the face of
uncertainty, decision making is determined not only by science but also by Agency policy.
Where not prohibited by statute, the relative costs and benefits of regulatory alternatives may be
considered in making decisions.
If uncertainty and variability have not been well characterized or acknowledged, potential
complications arise in the process of decision-making that seeks to achieve a balance between
over- and under-regulation. Increased uncertainty can make it more difficult to determine with
reasonable confidence the balance point between costs of regulation and the implications for
avoiding damages and producing benefits. Characterization of these factors, facilitated by
probabilistic analyses, can provide insight in weighing the relative costs and benefits of varying
levels of regulation and also assist in risk communication activities.
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
1.4. What Are Key Questions Often Asked by Decision makers?
Determining decision-maker concerns is a critical first step toward developing a useful and
responsive risk assessment. For example, the appropriate focus and level of detail of the analysis
should be commensurate with decision maker and stakeholder needs as well as appropriate use of
science. Often, analyses are conducted at a level of detail dictated by the issue being addressed,
the breadth and quality of the available information upon which to base an analysis, and the
significance surrounding this decision. The analytical process tends to be an iterative one, and,
even though a guiding set of questions may frame the initial analyses, additional questions can
arise that further direct or even reframe the analyses.
Based on a series of discussions with Agency decision makers and risk assessors, some questions
that are typically posed about risk analyses include the following:
• How representative or conservative is the estimate, (e.g., what is the variability around an
estimate)?
• What are the major gaps in knowledge, and what are the major assumptions used in the
assessment? How reasonable are the assumptions?
• Would my decision be different if the data were different? Would additional data collection
and research likely lead to a different decision? How long will it take to collect the
information, how much would it cost, and would the resulting decision be significantly
different?
• Will the use of additional resources, such as a probabilistic approach, impact the decision
making in a timely manner (i.e., better characterize uncertainties, better identify variability,
impact timelines, etc.)?
• What are the liabilities/consequences of making a decision under the current level of
knowledge and uncertainty?
• What is the percentile of the population to be protected?
• How do the different alternative decision choices and the interpretation of uncertainty and
variability impact the target population?
The questions that arise concerning analyses, including PRA, change depending on the stage and
nature of the decision making and analysis, from planning and scoping through risk
management. The utility of various levels of analysis and levels of sophistication in answering
these questions are illustrated in the case studies section described in Section 1.8 and presented
in Appendix D.
1.5. Why Is the Implementation of Probabilistic Risk Analysis Important?
The principal reason for the inclusion of PRA as an option in the risk assessor's toolbox is
PRA's ability to refine and improve the information leading to decision making by incorporating
known uncertainties. Beginning as early as the 1980s (NRC, 1983), expert scientific advisory
groups have been recommending that risk analyses include a clear discussion of the uncertainties
in risk estimation. The National Research Council (NRC, 1994) stated the need to describe
uncertainty and to capture variability in risk estimates. The Presidential/Congressional
Commission on Risk Assessment and Risk Management (PCRARM, 1997) recommended
against a requirement or need for a "bright line," or single number, level of risk (see Section 2.5
for more on the scientific community's opinion on use of PRA). Regulatory science often
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
requires selection of a limit for a contaminant, yet that limit always contains uncertainty as to
how protective it is. PRA and related tools quantitatively describe the very real variations in
natural systems and living things and how they respond to stressors and the uncertainty in
estimating those responses. Risk characterization became EPA policy in 1995, and the principles
of transparency, clarity, consistency, and reasonableness are explicated in the 2000 Risk
Characterization Handbook (EPA, 2000). Transparency, clarity, consistency, and reasonableness
criteria require decision makers to describe and explain the uncertainties, variability, and known
data gaps in the risk analysis and how they affected resulting decision-making processes
(USEPA, 2000, 1992, 1995).
The use of probabilistic methods also has received support from some decision makers within the
Agency, and these methods have been incorporated into a number of Agency decisions to date.
Program offices such as the Offices of Pesticides, Solid Waste and Emergency Response, Air
and Radiation, and Water, as well as the Office of Research and Development have utilized
probabilistic approaches in different ways and to varying extents, for both human exposure and
ecological risk analyses. In addition, the Office of Solid Waste and Emergency Response has
provided explicit guidance on the use of probabilistic approaches for exposure analysis (EPA,
2001). Some program offices have held training sessions on Monte Carlo simulation (MCS)
software frequently used in probabilistic analyses.
Where it is useful to refine risk estimates, the use of PRA can help in the characterization and
communication of uncertainty, variability, and the impact of data gaps in risk analyses, for
assessors, decision makers, and stakeholders including the target population.
1.6. How Does EPA Typically Address Scientific Uncertainty and Communicate
Variability?
Environmental assessments can be complex, such as exposures to multiple chemicals in multiple
media for a wide ranging population. The Agency often has developed simplified approaches to
characterizing risks through the use of point estimates for model variables or parameters. Such
an approach typically produces point estimates of risks (e.g., 10"5 lifetime probability of cancer
risk for an individual). These are often called "deterministic" assessments. As a result of the use
of point estimates for variables in model algorithms, deterministic risk results usually are
reported as what are assumed to be either average or worst-case estimates and do not contain any
quantitative estimate of the uncertainty in that estimate, nor report to what percentile of the
exposed population the estimate applies. However, the methods typically used in EPA risk
assessments rely on a combination of point values with potentially varying levels of
conservatism and certainty, yielding a point estimate of exposure that may be at some unknown
point in the range of possible risks.
Because uncertainty is inherent in all risk assessments, it is important that the risk assessment
process handle uncertainties in a logical way that is transparent and scientifically defensible,
consistent with the Agency's statutory mission, and responsive to the needs of decision makers
(NRC, 1994). Thus, when data are missing, EPA often uses several options to provide some
bounds on uncertainty and variability, in an attempt to avoid risk underestimation; attempting to
give a single quantification of how much confidence there is in the risk estimate may not be
informative or feasible. For example, in exposure assessment, the practice at EPA is to collect
new data where needed and where time and resources allow. Alternatives include narrowing the
scope of the assessment, using screening level default assumptions that include upper-end values
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
and/or central tendency values that are generally combined to generate risk estimates that fall
within the higher end of the population risk range (USEPA, 2004), using models to estimate
missing values, using surrogate data (e.g., data on a parameter that come from a different region
of the country than the region being assessed), or using professional judgment. The use of
individual assumptions can range from qualitative (e.g., assuming one is tied to the residence
location and does not move through time or space) to more quantitative (e.g., using the 95th
percentile of a sample distribution for an ingestion rate). This approach also can be applied to the
practice of hazard identification and dose-response assessment when data are missing.
Identifying the sensitivity of exposure or risk estimates to key inputs can help in focusing efforts
to reduce uncertainty by collecting more data.
1.7. What Are the Limitations of Relying on Default-Based Deterministic Approaches?
Deterministic risk analysis often is considered a traditional approach to risk analysis because of
the existence of established guidance and procedures regarding its use, the ease with which it can
be performed, and its limited data and resource needs. The use of defaults supporting
deterministic risk assessment provides a procedural consistency that allows for risk assessments
to be feasible and tractable. Risk managers and members of the public tend to be relatively
familiar with deterministic risk assessment, and use of such an approach addresses assessment-
related uncertainties primarily through the incorporation of predetermined default values and
conservative assumptions. It addresses variability by combining input parameters intended to be
representative of typical or higher end exposure (considered to be conservative assumptions).
The intention is often to provide a margin of safety or to construct a screening level estimate of
high-end exposure and risk (i.e., an estimate representative of more highly exposed and
susceptible individuals).
Deterministic risk assessment provides an estimation of exposures and resulting risks that
addresses uncertainties and variabilities in a qualitative manner. The methods typically used in
EPA deterministic risk assessments rely on a combination of point values—some conservative
and some typical—yielding a point estimate of exposure that is at some unknown point in the
range of possible risks. Such an approach is believed to more likely overestimate than
underestimate risks. Although this conservative bias (more likely to over- than underestimate
risks) aligns with the public health mission of EPA (USEPA, 2004), the degree of conservatism
in these risk estimates (and in any concomitant decision) cannot be estimated well or
communicated (Hattis and Burmaster, 1994). This results in unquantified uncertainty in risk
statements.
Estimates generated using these methods are unaccompanied by quantitative information
regarding their precision or potential systematic error and do not account for the distribution of
exposures, effects, and resulting risks across different members of an exposed population.
Although deterministic risk assessments may present qualitative information regarding the
robustness of the estimates, the impact of data and model limitations on the quality of the results
cannot be quantified. Reliance on deterministically derived estimations of risk can result in
decision making based solely on point estimates with an unknown degree of conservatism, which
can complicate comparison of risks or management options. The use of conservative defaults has
long been the target of criticism and has led to the presumption by critics that EPA assessments
are overly conservative and unrealistic. This criticism may reduce the overall perceived
credibility of an Agency decision.
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Deterministic risk assessment is not as well suited as PRA for more complex assessments,
including those of aggregate and cumulative exposures and time-dependent individual exposure,
dose, and effects analyses. Identification and prioritization of contributory sources of uncertainty
can be difficult and time consuming when using deterministic methods, leading to difficulties in
model evaluation and the subsequent appraisal of risk estimates (Cullen and Frey, 1999). These
comprehensive quantitative analyses of model sensitivities are essential for the prioritization of
key uncertainties—a critical step in identifying steps for data collection or research to improve
exposure or risk estimates.
1.8. What Is EPA's Experience with the Use of Probabilistic Risk Analysis?
To assist with the growing number of probabilistic analyses of exposure data, EPA issued
Guiding Principles for Monte Carlo Analysis (EPA, 1997). Probabilistic analysis techniques
such as Monte Carlo analysis, given adequate supporting data and credible assumptions, can be
viable statistical tools for analyzing variability and uncertainty in risk assessments. The EPA
policy for use of probabilistic analysis in risk assessment, released in 1997, is inclusive of human
exposure and ecological risk assessments, but does not rule out probabilistic health effects
analyses. Subsequently, EPA's Science Advisory Board (SAB) and Scientific Advisory Panel
have reviewed PRA approaches to risks by EPA Offices such as Air and Radiation, Pesticides,
and others. Several programs have developed specific guidance on use of PRA, including
Pesticides and Solid Waste and Emergency Response (EPA, 1998, 2001).
To illustrate the practical application of PRA to problems relevant to the Agency, several
example case studies are briefly described here. Appendix D, Case Study Examples of the
Application of Probabilistic Risk Analysis in U.S. Environmental Protection Agency Regulatory
Decision Making, discusses these and other case studies in greater detail, including the
procedures and outcome. The examples are intended to illustrate how some of EPA's programs
and offices currently utilize PRA. They demonstrate how information from probabilistic
analyses, including sensitivity analysis, MCS, and other techniques, were used in decision
making. Some of the approaches that are profiled can be used easily in the planning and scoping
of risk assessments and risk management. Other more complex approaches are used to answer
more specific questions and provide a richer description of the risks. Most show that PRA can
improve or expand information generated by deterministic methods. The case studies illustrate
that the Agency already has applied the science of PRA to ecological risk and human exposure
estimation and has begun using PRA to describe health effects. Some of the applications have
used existing "off the shelf software, whereas others have required significant effort and
resources. Once developed, however, some of the more complex models have been used many
times for different assessments. All have stood the test of internal and external peer review. A list
of the case study examples presented in Appendix D are provided in Table 1 including
categorizations based on type of assessment (i.e., human health or ecological risk assessment);
PRA tools used in the assessment; and program office or region responsible for the assessment.
In several cases, the examples presented represent components of the overall risk assessment that
demonstrate use of multiple PRA techniques.
A few examples that illustrate the variety of PRA uses in EPA are:
10
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Hudson River PCB-Contaminated Sediment Site: Region 2 evaluated the variability
in risks to anglers who consume recreationally caught fish contaminated with PCBs
from sediment contamination in the Hudson River. (Case Study 5)
EMAP program: The Office of Research and Development (ORD) developed and
Office of Water (OW) adopted an applied probabilistic sampling techniques to
evaluate nation's aquatic resources under CWA Section 305(b) (Case Study 7)
Chromated Copper Arsenate Risk Assessment: ORD and the Office of Pesticide
Programs (OPP) conducted a probabilistic exposure assessment of children's
exposure (addressing both variability and uncertainty) to arsenic and chromium from
contact with CCA-treated wood playsets and decks. (Case Study 9)
Evaluating Ecological Effects of Pesticide Uses: OPP developed a probabilistic model
which evaluates acute mortality levels in generic and specific ecological species for
user-defined pesticide uses and exposures. (Case Study 13)
PM2 5 Health Impacts: The Office of Air and Radiation (OAR) used expert elicitation
to more completely characterize, both qualitatively and quantitatively, the
uncertainties associated with the relationship between reduction in PM2.5 and benefits
of reduced PM2.5-related mortality. (Case Study 14)
11
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
2. Probabilistic Risk Analysis
2.1. What Are Variability and Uncertainty, and How Are They Relevant to Decision-
making?
The concepts of variability and uncertainty are introduced here, and the relevance of these
concepts to decision making is discussed.
2.1.1. Variability
Variability refers to real differences over time, space, or members of a population and is a
property of the system being studied (e.g., drinking water rates for each of the many individual
adult residents living in a specific location) (Cullen and Frey, 1999). Variability can arise from
inherently random processes, such as variations in wind speed over time at a given location or
from true variation across members of a population that, in principle, could be explained but that,
in practice, may not be explainable using currently available models or data (e.g., the range of
blood lead levels in 6-year-old children following a specific degree of lead exposure). Of
particular interest in human health risk assessment is inter-individual variability, which typically
refers to differences between members of the same population in either behavior related to
exposure (e.g., dietary consumption rates for specific food items) or biokinetics related to
chemical uptake or toxic response (e.g., gastrointestinal uptake rates for lead following intake).
2.1.2. Uncertainty
Uncertainty is the lack of knowledge of the true value of a quantity or relationships among
quantities. For example, there may be a lack of information regarding the true distribution
variability between individuals for consumption of certain food items. There are a number of
types of uncertainties for both risk analyses and risk management. The following description of
types of uncertainty (drawn from Cullen and Frey, 1999) addresses uncertainties that arise during
risk analyses. These uncertainties can be separated broadly into three categories: (1) scenario
uncertainty, (2) model uncertainty, and (3) input (or data) uncertainty. Each of these is explained
below^
Scenario uncertainty refers to errors, typically of omission, resulting from incorrect or
incomplete specification of the risk scenario to be evaluated. The risk scenario refers to a set of
assumptions regarding the situation to be evaluated, such as (a) the specific sources of chemical
emissions or exposure to be evaluated (one industrial facility or a cluster of varied facilities
impacting the same study area), (b) the specific receptor populations and associated exposure
pathways to be modeled (e.g., indoor inhalation exposure, track-in dust, or consumption of
home-produced dietary items), and (c) the times or activities to be considered (e.g., exposure
only at home, or consideration of workplace or commuting exposure). Misspecification of the
risk scenario can result in underestimation, overestimation, or other mischaracterization of risks.
For instance, underestimation may occur because of exclusion of relevant situations or inclusion
of irrelevant situations with respect to a particular analysis. Overestimation may occur because
of the inclusion of unrealistic or irrelevant situations (e.g., assuming continuous exposure to an
intermittent airborne contaminant source rather than accounting for mobility throughout the day.)
12
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Model uncertainty refers to limitations in the mathematical models or techniques that are
developed to represent the system of interest and often stems from: (a) simplifying assumptions,
(b) exclusion of relevant processes, (c) misspecification of model boundary conditions (i.e.,
range of input parameters), or (d) misapplication of a model developed for other purposes. Model
uncertainty typically arises when the risk model relies on missing or improperly formulated
processes, structures, or equations. Refer to the glossary for additional information.
Input or Parameter uncertainty typically refers to errors in characterizing empirical values used
as inputs to the model (e.g., engineering, physical, chemical, biological, or behavioral variables).
Input uncertainty can stem from random or systematic errors involved in measuring a specific
phenomenon (e.g., biomarker measurements such as the concentration of mercury in human
hair); from statistical sampling errors associated with small sample sizes (if the data are based on
samples selected with a random, representative sampling design); from the use of surrogate data
instead of directly relevant data, or the absence of an empirical basis for characterizing an input
(e.g., absence of measurements for fugitive emissions from an industrial facility); or from the use
of summary measures of central tendency rather than individual observations. Nonlinear random
processes can exhibit a behavior that, for small changes in input values, produces large variation
in results.
2.2. When Is Probabilistic Risk Analysis Applicable or Useful?
PRA is useful in the following types of situations (Cullen and Frey, 1999).
• When a screening level deterministic risk assessment indicates that risks are possibly higher
than a level of concern, and, therefore, a more refined assessment is needed
• When the consequences of using potentially biased point estimates of risk are unacceptably
high
• To estimate the value of collecting additional information to reduce uncertainty
• When significant equity issues are raised by inter-individual variability
• To identify promising critical control points and critical levels when evaluating risk
management alternatives
• To rank exposure pathways, sites, contaminants, and so on for purposes of prioritizing model
development or further research
PRA typically is not necessary in the following types of situations (Cullen and Frey, 1999; EPA
1997).
• When a screening-level deterministic risk assessment indicates that risks are negligible,
presuming that the assessment is known to be biased to produce overestimates of risk
• When the cost of averting the exposure and risk is smaller than the cost of probabilistic
analysis
• When there is little uncertainty or variability in the analysis (This is a rare situation.)
2.3. How Can Probabilistic Risk Analysis Be Incorporated into Assessments?
As illustrated in the accompanying case studies (Appendix D), probabilistic approaches can be
incorporated into any stage of a risk assessment, from problem formulation or planning and
scoping to analysis of alternative risk management decisions. In some situations, PRA can be
used selectively for components of an assessment. It is common in assessments that some model
inputs are known with high confidence (i.e., based on site-specific measurements), whereas
13
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
values for other inputs are less certain (i.e., based on surrogate data collected for a different
purpose). For example, an exposure modeler may determine that there is relevant air quality
monitoring data but a lack of detailed information of human activity patterns in different
microenvironments. Thus, an assessment of variability in exposure to airborne pollutants might
be based on direct use of the monitoring data, whereas assessment of uncertainty and variability
in the inhalation exposure component might be based on statistical analysis of surrogate data or
use of expert judgment. The uncertainties are likely larger for the latter than the former
component of the assessment; thus, efforts to characterize uncertainties associated with pollutant
exposures would focus on the latter.
2.4. What Are the Scientific Community's Views on Probabilistic Risk Analysis, and What
Is the Institutional Support for Its Use?
The NRC recently emphasized its long-standing advocacy for probabilistic risk assessment
(NRC, 2007a,b). Dating from its 1983 Risk Assessment in the Federal Government (NRC,
1983)—which first formalized the risk assessment paradigm—through various reports released
in the late 1980s, all during the 1990s, and through the early 2000s, various NRC panels have
consistently maintained that—because risk analysis involves substantial uncertainties—these
uncertainties should be evaluated within a risk assessment. These panels noted,
(1) in 1989, that when evaluating total population risk, EPA should consider the distribution of
exposure and sensitivity of response in the population (NRC, 1989);
(2) in 1991, that when assessing human exposure to air pollutants, EPA should present model
results along with estimated uncertainties (NRC, 1991);
(3) in 1993, that when conducting ecological risk assessments, EPA should discuss thoroughly
uncertainty and variability within the assessment (NRC, 1993); and,
(4) in 1994, an NRC report, Science and Judgment in Risk Assessment, stated that "uncertainty
analysis is the only way to combat the 'false sense of certainty,' which is causedby a refusal
to acknowledge and [attempt to] quantify the uncertainty in risk predictions" (emphasis in the
original) (NRC, 1994). In 2002, another report suggested that EPA's estimation of health
benefits were not wholly credible because EPA failed to deal formally with uncertainties in
its analyses (NRC, 2002).
Asked to recommend improvements to the Agency's human health risk assessment practices,
EPA's Science Advisory Board echoed the NRC's sentiments and urged the Agency to
characterize variability and uncertainty more fully and more systematically and to replace single-
point uncertainty factors with a set of distributions using probabilistic methods (Parkin and
Morgan, 2007). The key principles of risk assessment cited by the Office of Science and
Technology Policy (OSTP) and the Office of Management and Budget (OMB) include "explicit"
characterization of the uncertainties in risk judgments; they go on to cite the National Academy
of Science's 2007 recommendation to address "variability of effects across potentially affected
populations" (OSTP/OMB, 2007).
2.5. How Can Probabilistic Risk Assessment Provide More Comprehensive, Rigorous
Scientific Information in Support of Regulatory Decisions?
External stakeholders in the past have used the Administrative Procedure Act and the Data
Quality Act to challenge the Agency for a lack of transparency and consistency or for not fully
analyzing and characterizing the uncertainties in risk assessments or risk management decisions
14
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
(see Fisher et al., 2006). The more complete implementation of PRA and related approaches to
deal with uncertainties in decision-making lends support to the overall Agency risk-based
decision-making process.
The results of any assessment, including PRA, are dependent on the underlying methods and
assumptions. Accompanied by the appropriate documentation, PRA may communicate a more
robust representation of risks and corresponding uncertainties. This characterization may take
shape in the form of a range of possible estimates as opposed to the more traditionally presented
single-point values. Depending on the needs of the assessment, ranges can be derived for
variability and uncertainty (or a combination of the two) in both model inputs and resulting
estimations of risk.
2.6. Are There Additional Advantages of Using Probabilistic Risk Analysis?
PRA quantifies how exposures, effects, and risks differ among individuals and provides an
estimation of the degree of confidence with which these estimates may be made, given the
current uncertainty in scientific knowledge and available data. A 2007 NRC panel stated that the
objective of PRAs is not to decide "how much evidence is sufficient" to adopt an alternative but,
rather, to describe the scientific bases of proposed alternatives so that scientific and policy
considerations may be more fully evaluated (NRC, 2007a). EPA's SAB similarly noted that
PRAs provided more "value of information" through quantitative assessment of uncertainty, and
clarify the science underlying Agency decisions (EPA, 2007).
The SAB proposed a number advantages Agency decision-makers could reap from utilization of
probabilistic methods (Parkin and Morgan, 2007).
• A probabilistic reference dose could help reduce the potentially inaccurate implication of zero
risk below the RfD.
• By understanding and explicitly accounting for uncertainties underlying a decision, EPA can
estimate formally the value of gathering more information. By doing so, two benefits follow:
(1) EPA can better prioritize its research needs by investing in areas that yield the greatest
information value; and (2) when making decisions, the Agency can eschew the less-than-
helpful rationale of "too much scientific uncertainly." By candidly acknowledging
uncertainty's ubiquity, EPA can base a decision on more intellectually robust concepts of
comparing present risks against the costs of gathering more information.
• By adopting PRA, EPA sends the appropriate signal to the intellectual marketplace, thereby
encouraging analysts to gather data and to develop methodologies necessary for assessing
uncertainties.
2.7. What Are the Challenges to Implementation of Probabilistic Analyses?
Currently, EPA is using PRA in a variety of programs to support decisions, but challenges
remain regarding the expanded use of these tools within the Agency; these include those that
follow.
• A lack of understanding of the value of PRA to decision making;
• The perception, if not reality, that PRA requires additional resources;
• Limited resources (staff, time, training, or methods) to conduct PRA;
• A lack of clear directive or requirement to utilize PRA in many cases;
15
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
• Lack of understanding of how to incorporate the results of probabilistic analyses into decision
making and how to establish action levels based on the scope of the assessment.
• The complexity of communicating probabilistic analysis results.
• Fuller characterization of the uncertainties in a risk estimate through use of PRA can lead to
more difficult decision making or more complicated risk communication
• The ability to simulate and reanalyze numerous scenarios using various models and input data
could lead to prolonged analyses and delay decisions
These challenges are discussed in more detail below.
2.8. How Can Probabilistic Risk Analysis Support Specific Regulatory Decision making?
Decision makers sometimes perceive that the binary nature of regulatory decisions (e.g. Does an
exposure exceed a reference dose or not? Do emissions comply with Agency standards or not?)
precludes the use of a range of uncertainty, compared with the use of point estimates. Generally,
it is legally necessary to explain the rationale underlying a particular decision. PRA's primary
purpose is to provide information so that decisions are based on the best available science; it is
not to necessarily displace legally mandated decisions with a range of alternatives. By doing a
sensitivity analysis of the influence of the uncertainty on the decision-making process, it can be
determined how or if PRA can help improve the process.
2.9. Does Probabilistic Risk Analysis Require More Resources Than Default-Based
Deterministic Approaches?
PRA can generally be expected to require more resources than standard Agency default-based
deterministic approaches. There is extensive experience within EPA in conducting and reviewing
deterministic risk assessments. These assessments tend to follow standardized methods that
minimize the effort required to conduct them and to communicate the results. Probabilistic
assessments often entail a more detailed analysis, and, as a result, there exists a common
perception that these assessments require substantially more resources than do deterministic
approaches.
Appropriately trained staff and the availability of adequate tools, methods, and guidance are
essential for application of PRA. Proper application of probabilistic methods requires not only
software and data but also guidance and training for both analysts using the tools and for
managers and decision makers tasked with interpreting and communicating the results. In most
circumstances, probabilistic assessments may take more time and effort to conduct than
conventional approaches, primarily because of the comprehensive inclusion of available
information on model inputs.
An upfront increase in resources needed to conduct a probabilistic assessment can be expected,
but development of standardized approaches and/or methods can lead to the routine
incorporation of PRA in Agency approaches (e.g., the Office of Pesticide Programs' use of
DEEM, a probabilistic dietary exposure model). The initial and, in some cases, ongoing resource
cost (e.g., that for development of site-specific models for site assessments) may be offset by a
more informed decision than a comparable deterministic analysis. Probabilistic methods are
useful for identifying effective risk management options and in prioritizing additional data
collection or research aimed at improving risk estimation, ultimately resulting in management
16
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
options that enable improved environmental protection while, simultaneously, conserving greater
resources.
2.10. Doesn't Probabilistic Risk Analysis Require More Data Than Conventional
Approaches?
There are differences in opinion within the technical community as to whether PRA requires
more data than other types of analyses. Although some emphatically believe that PRA requires
more data, others argue that probabilistic assessments make better use of all of the available data
and information. Stahl et al. (2005) discuss when and how much data are necessary for a
decision. PRA can benefit from more data than might be used in a deterministic risk assessment.
For example, where deterministic risk assessments may employ selected point estimates (such as
mean or 95th percentile values) from available datasets for use in model inputs, PRA facilitates
the use of frequency-weighted data distributions, allowing for a more comprehensive
consideration of the available data. In many cases, the data that were used to develop the
presumptive 95th percentile can be employed in the development of probabilistic distributions.
Restriction of PRA to data-rich situations may prevent its application where it is most useful.
Because PRA incorporates information on data quality, variability, and uncertainty into risk
models, the influence of these factors on the characterization of risk can become a greater focus
of discussion and debate.
A key benefit of using PRA is its ability to reveal limitations, as well as strengths, of data that
often are masked by a deterministic approach. In doing so, PRA can help inform research
agendas, as well as support regulatory decision making, based on the state of the best available
science. In summary, PRA typically requires more time for developing input assumptions than a
corresponding deterministic risk assessment, but when incorporated in the relevant steps of the
risk assessment process, PRA can demonstrate real added benefits. In some cases PRA can
provide additional interpretations that compensate for the additional efforts.
2.11. Can Probabilistic Risk Analysis Be Used To Screen Risks or Only in Complex or
Refined Assessments?
Probabilistic methods typically are not necessary where traditional default-based deterministic
methods are adequate for screening risks. Such methods are relatively low cost, are intended to
produce conservatively biased estimates, and are useful for identifying situations in which risks
are so low that no further action is needed. The application of probabilistic methods can be
targeted to situations in which a screening approach indicates that a risk may be of concern or
when the cost of managing the risk is high, creating a need for information to help inform risk
management decisions. PRA fits directly into a graduated hierarchical approach to risk analysis.
PRA also could be used to more fully examine the existing default-based methods based on the
current state of information and knowledge to determine if such methods are truly conservative
and adequate for screening.
2.12. Does Probabilistic Risk Analysis Present Unique Challenges to Model Evaluation?
The concept of "validation" of models used for regulatory decision making has been a topic of
heated discussion. In a recent report on the use of models in environmental regulatory decision
making, NRC recommended the use of the notion of model "evaluation" rather than "validation,"
suggesting use of a process that encompasses the entire life cycle of the model and recognizes
17
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
the spectrum of interested parties in the application of the model, which often extends beyond the
model builder and decision maker. Such a process can be designed to ensure that judgment of the
model application is based not only on its predictive value determined from comparison with
historical data but also on its comprehensiveness, rigor in development, transparency, and
interpretability (NRC, 2007b).
Model evaluation is important in all risk assessments. In the case of PRA, there is an additional
question as to the validity of the assumptions made regarding probability and frequency
distributions for model inputs and their dependencies. Probabilistic information can be accounted
for during evaluation analyses by considering the range of uncertainty in the model prediction
and whether such a range overlaps with the "true" value based on independent data. Thus,
probabilistic information can aid in characterizing the precision of the model predictions and
whether a prediction is significantly different from a benchmark of interest. For example,
comparisons of probabilistic model results and monitoring data were done for multiple models in
developing the cumulative pesticide exposure model. There are also published concurrent PRA
model evaluations using a Bayesian analysis (Clyde, 2000).
2.13. How Do You Communicate Results of Probabilistic Risk Analysis?
The approaches for reporting results from PRA vary depending on the assessment objective and
the intended audience. Beyond the basic 1997 principles and the policy from the same year, the
Risk Assessment Guidance for Superfund: Volume III also provides some guidance on the
quality and criteria for acceptance as well as communication basics (EPA, 2001). There have
been limited studies of how information from PRA regarding variability and uncertainty can or
should be communicated to key audiences such as decision makers and stakeholders (e.g.,
Morgan and Henrion, 1990; Bloom et a/., 1993; Krupnick et a/., 2006). Among the analyst
community, there is often an interest in visualization of the structure of a scenario and model
using influence diagrams and depiction of the variability and uncertainty in model inputs and
outputs using probability distributions in the form of cumulative density functions or probability
18
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Figure 1. Graphical Description of the Likelihood (Probability of Risk) of Toxicity
(Fitted data distribution and confidence intervals)
Source: Frey, H.C. (2004)
distribution functions (Figure 1). Sensitivity of the model output to variability and uncertainty in
model inputs can be depicted using graphical tools.
In some cases, these graphical methods can be useful for those less familiar with PRA, but in
many cases there is a need to translate the quantitative results into a message that extracts the key
insights without burdening the decision maker with obscure technical details. In this regard, the
use of ranges of values for a particular metric of decision-making relevance (e.g., range of
uncertainty associated with a particular estimate of risk) may be adequate. The presentation of
PRA results to a decision maker may be conducted best as an interactive discussion, in which a
principal message is conveyed, followed by exploration of issues such as the source, quality, and
degree of confidence associated with the information. There is a need for development of
recommendations and a communication plan regarding how to communicate the results of PRA
to decision makers and stakeholders, building on the experience of various programs and regions
in this area.
2.14. Are the Results of Probabilistic Risk Analysis Difficult To Communicate to Decision-
makers and Stakeholders?
Research has shown that the ability of decision makers to deal with concepts of probability and
uncertainty is variable. Bloom et al. (1993) surveyed a group of senior managers at EPA and
found that many could interpret information about uncertainty if it was communicated in an
appropriate manner that was responsive to decision-maker interests, capabilities, and needs. In a
more recent survey of ex-EPA officials, Krupnick et al. (2006) concluded that most had
19
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
difficulty understanding information on uncertainty, and that certain formats used to present
uncertainty information were more effective than others. The findings of these studies highlight
the need for practical strategies for communication of results of PRA and uncertainty
information between risk analysts and decision makers, as well as between decision makers and
other stakeholders. The Office of Emergency and Remedial Response has compiled guidance to
assist analysts and managers in understanding and communicating the results of PRA (EPA,
2001).
20
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
3. Findings and Recommendations
3.1 Findings: How Probabilistic Risk Analysis and Related Analyses Can Improve the
Decision-making Process at EPA
PRA is an analytical methodology capable of incorporating information regarding uncertainty
and/or variability in analyses to provide insight regarding the degree of certainty of a risk
estimate and how the risk estimate varies among different members of an exposed population.
Traditional approaches often report risks as "central tendency" or "high end (e.g., 90th percentile
or above)," or "maximum anticipated exposure" and PRA can be used to more fully describe
uncertainty surrounding such estimates and identify the key contributors to variability or
uncertainty in predicted exposures or risk estimates. This information then can be used by
decision makers to achieve a science-based level of safety, to weigh alternative risk management
options, or to invest in researching areas which have the greatest uncertainty and impact on the
risk estimates.
Using PRA, one can obtain insight regarding whether one risk management strategy is more
likely to reduce risks compared to another, and by how much. The methodology facilitates the
investigation of potential changes in decisions that may result from the collection of additional
information that could better characterize variability and potentially reduce uncertainty and helps
determine how expenses incurred by activities to reduce uncertainty are offset by improved
decision-making capabilities gained from the acquisition of that knowledge. PRA can facilitate
the construction and simultaneous consideration of multiple model alternatives. Probabilistic
methods offer a number of tools designed to promote robust management and increased
confidence in decision making through the incorporation of input variability and uncertainty
characterization and prioritization in risk analyses. For example, sensitivity analyses can be used
to identify influential knowledge gaps involved in the estimation of risk, allowing for improved
transparency and the ability to more clearly communicate or articulate the most relevant
information to decision makers and stakeholders. Ultimately, PRA can enhance the Agency's
credibility in its approach to science-based decision making.
3.2. Recommendations for Enhanced Utilization of PRA in EPA
The various tools and methods discussed in this paper can be used at all stages of risk analysis
and also aid the decision-making process by characterizing inter-individual variability and
uncertainties. Probabilistic analyses and related methods are in use in varying degrees across the
Agency:
• The use of Monte Carlo or other probability based techniques to derive a range of
possible outputs from uncertain inputs is a fairly well-developed approach within EPA.
• Although basic guidance exists at EPA on the use and acceptability of PRA for risk
estimation, implementation varies greatly within programs, offices and regions.
• Although highly sophisticated human exposure assessment and ecological risk
applications have been developed, use of PRA models to assess human health effects and
dose-response has been somewhat limited.
21
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Enhanced use of PRA and consistent applications in support of EPA decision-making requires
improved internal capacity for conducting these assessments, as well as interpreting and
communicating such information in the context of decisions. Such improvements of internal
capacity could be accomplished through sharing of experiences, knowledge, and training,
improved policy and guidance, and increased availability of tools and methods.
Some steps to improve implementation include:
• Inform risk managers about the advantages and disadvantages in using PRA techniques
in their decision-making process through lectures, webinars, and communication
regarding the techniques and their use in EPA.
• Train risk assessors so that they can learn about the various tools available, their
applications, software and review considerations, and resources for additional
information (e.g., experts and support services within the Agency).
• Meetings and discussions of PRA techniques and their application with both managers
and assessors will aid in providing greater consistency and transparency to EPA's risk
assessment process and in developing EPA's internal capacity.
• Demonstrate through informational opportunities and resource libraries the various tools
and methods that can be used at all stages of risk analysis and also aid the decision-
making process by characterizing inter-individual variability and uncertainties.
• Promote the sharing of experience, knowledge, models, and best practices via meetings
of risk assessors and risk managers; electronic exchanges, such as the EPA Portal
Environmental Science Connector; and more detailed discussions regarding the case
studies.
• Provide easily available, flexible, modular training for all levels of experience to
familiarize employees to the menu of tools and their capacities.
• Provide introductory as well as advanced training open to all offices.
• Provide live and recorded seminars and Webinars for introductory and supplemental
education, as well as periodic, centralized hands-on training sessions on how to utilize
software programs.
Risk assessors, risk managers, and decision makers need to be provided the information and
training necessary so that they can better utilize these tools. Education and experience will
generate familiarity with these tools that will then lead analysts and decision makers to better
understand the techniques and consider more fully utilizing these techniques.
3.3 Guidance and Policy
Additional guidance can be developed to help analysts and decision makers decide which
statistical tools to use and when to use them, and how probabilistic information can help to
inform the basis of those decisions. Both deterministic and probabilistic approaches and other
statistical methods may be useful at any stage of the risk analysis and decision-making process,
from planning and scoping to characterizing and communicating uncertainty. Such bodies as the
EPA's Science Policy Council can play a role in directing guidance development to help
implement probabilistic and related tools. Examples of guidance needed include:
• Probabilistic approaches to evaluating health effects data
• Probabilistic approaches to ecological risk assessment
22
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
• Integrating probabilistic exposure and risk estimates and communicating uncertainty and
variability.
3.4 Challenges
In general terms, while PRA techniques are currently available which would help inform EPA
decision-making process, research, as well as guidance, is needed to further improve these
methods for more complete implementation of PRA in human health and ecological risk
assessment. Some examples include:
• Although highly sophisticated human exposure assessment and ecological risk
applications have been developed, use of PRA models to evaluate toxicity data has been
very limited. Scientific, technical, and science policy discussions are needed in this area.
• There is no consensus on any one well-accepted general methodology for dealing with
model uncertainty, although there are various examples of efforts to do so. Thus,
additional research on formal methods for treating model uncertainties will be valuable.
• As noted in Appendix A.3, there are significant challenges to properly account for
variability and uncertainty when multiple models are coupled together to represent the
source-to-outcome continuum (e.g., the OPP-Environmental Fate and Effects Division's
aquatic and terrestrial models). Moreover, the coupling of multiple models (e.g.,
emissions, air quality, exposure, dose, effect) may need to involve inputs and
corresponding uncertainties that are incorporated into more than one model, potentially
resulting in complex dependencies (e.g., ambient temperature affects emission rates, air
quality, and human activity that influence total emissions and exposures).
• There may be mismatches in the temporal and spatial resolution of each model, which
confound the ability to propagate variability and uncertainty from one model to another.
For some models, the key uncertainties may be associated with inputs, whereas, for other
models, the key uncertainties may be associated with structure or parameterization
alternatives
23
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Appendix A: An Overview of Some of the Techniques Used in
Probabilistic Risk Analysis
A.1. What Is the General Conceptual Approach in Probabilistic Risk Analysis?
PRA includes several major steps, which parallel the accepted environmental health risk
assessment process. These include (1) problem and/or decision criteria identification, (2) getting
information, (3) interpreting the information, (4) selecting and applying models and methods for
quantifying variability and/or uncertainty, (5) quantifying inter-individual or population
variability and uncertainty in metrics relevant to decision-making, (6) sensitivity analysis to
identify key sources of variability and uncertainty, and (7) reporting of results.
Problem identification deals with identifying the assessment end points, or issues, that are
relevant to the decision-making process, as well as to other stakeholders, and that can be
addressed in a scientific assessment process. Following problem identification, information is
needed from stakeholders and experts regarding the scenarios to evaluate. Based on the scenarios
and assessment endpoints, the analysts select or develop models, which in turn leads to
identification of model input data requirements and acquisition of data or other information (e.g.,
expert judgment encoded as the result of a formal elicitation process) from which to quantify
inputs to the models. The data or other information for model inputs is interpreted in the process
of developing probability distributions to represent variability, uncertainty, or both for a
particular input. Thus, the steps (1) through (4) listed above are highly interactive and iterative in
that the data input requirements and how information is to be interpreted depend on the model
formulation, which depends on the scenario, which in turn depends on the assessment objective.
The assessment objective may have to be refined depending on the availability of information.
Once a scenario, model, and inputs are specified, the model output is estimated. A common
approach is to use Monte Carlo or other probabilistic methods for generating samples from the
probability distributions of each model input, run the model based on one random value from
each probabilistic input, and produce one corresponding estimate of the model outputs. This
process is repeated typically hundreds or thousands
of times to create a synthetic statistical sample of
model outputs. These output data are interpreted as
a probability distribution of the output of interest.
Sensitivity analysis can be performed to determine
which model input distributions are most highly
associated with the range of variation in the model
outputs. The results may be reported in a wide
variety of forms depending on the intended
audience, ranging from qualitative summaries to
tables, graphs, and diagrams.
Levels of Analyses
Sensitivity analysis
Monte Carlo analysis of variability in
- Exposure data
- Human health or ecological effect
data
Monte Carlo analysis of uncertainty
"Cumulative" PRA—multi-pathway or
multi-chemical
Two-dimensional PRA of uncertainty
and variability
Decision uncertainty analysis
Geospatial analysis
Expert elicitation
24
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
A.2. What Are the Multiple Types of Probabilistic Risk Analyses, and How Are They
Used?
There are multiple levels for conducting risk assessments. Graduated approaches to analysis are
widely recognized (e.g., EPA, 1997; EPA, 2001; WHO, 2007). The idea of a graduated approach
is to choose a level of detail and refinement for an analysis that is appropriate to the assessment
objective, data quality, information available, and importance of the decision (e.g., resource
implications).
Detailed introductions to PRA methodology are available elsewhere, such as Ang and Tang
(1984), Cullen and Frey (1999), EPA (2001), Morgan and Henri on (1990) and EPA, 2001. A few
key aspects of PRA methodology are briefly mentioned here. However, readers who seek more
detail should consult these references and see the bibliography for additional references.
The deterministic risk assessment approaches described in Section 1.6 are examples of lower
levels in a graduated approach to analysis, in which risk at the lower levels of analysis is
assessed by conservative, bounding assumptions. If the risk estimate is found to be very low
despite use of conservative assumptions, then there exists a great deal of certainty that the actual
risks to the population of interest for the given scenario are below levels of concern and, thus,
that no further intervention is required, assuming the model specification is correct. However,
when a conservative deterministic risk assessment indicates that a risk may be high, it is possible
that the risk estimate is biased, and the actual risk may be lower. In such a situation, depending
on the resource implications of risk management, it may be appropriate to proceed with a more
refined, or higher level, analysis. If the cost of intervention is less than the cost of further
analysis, then it may be appropriate to simply proceed to the risk management decision as a
preventive measure that is also expedient. In some deterministic assessments, for instance, for
ecological risks, the assumptions are not well assured of conservatism and the estimated risks
might be biased to appear lower than the unseen actual risk
A more refined analysis could involve applications of deterministic risk assessment methods but
with alternative sets of assumptions intended to characterize central tendency and reasonable
upper bounds of exposure, effects, and risk estimates, such that the estimates could be for an
actual individual in the population of interest (rather than a hypothetical maximally exposed
individual). However, such analyses are not likely to provide quantification regarding the
proportion of the population at or below a particular exposure or risk level of concern,
uncertainties for any given percentile of the exposed population, nor priorities among input
assumptions with respect to their contributions to variability and uncertainty in the estimates.
To more fully answer the questions often asked by decision-makers, the analysis can be further
refined by incorporating quantitative comparisons of alternative modeling strategies (to represent
structural uncertainties associated with scenarios or models), by quantifying ranges of variability
and uncertainty in model outputs, and by providing the corresponding ranges for model outputs
of interest. When performing probabilistic analyses such as these, choices are made regarding
whether to focus on quantification of variability only, uncertainty only, both variability and
uncertainty co-mingled (representing a randomly selected individual), or variability and
uncertainty distinguished (e.g., in a two-dimensional depiction of probability bands for estimates
of inter-individual variability) (see Figure 2). The simultaneous but distinct propagation of
variability and uncertainty in a two dimensional framework enables quantification of uncertainty
in the risk for any percentile of the population. For example, one could estimate the range of
25
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
uncertainty in the risk faced by the median member of the population or the 95th percentile
member of the population. Such information can be used by a decision maker (for example) to
gauge the confidence that should be placed in any particular estimate of risk, as well as to
determine whether additional data collection or information might be useful to reduce the
uncertainty in the estimates. The OPP assessment of chromated copper arsenate treated wood
(see Appendix D) used such an approach.
100
cu
E
^
(A
O
O
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
The notion of iteration can be applied broadly to the risk assessment framework. For example, a
first effort to perform an analysis may lead to insight that the assessment questions might be
impossible to address, or that there are additional assessment questions that may be equally or
more important. Thus, iteration can include reconsideration of the initial assessment questions
and the corresponding implications for definition of scenarios, selection of models, and priorities
for obtaining data for model inputs. Alternatively, in a time-limited decision environment, such
probabilistic and sensitivity analyses may offer insight into the effect of risk management
options on risk estimates.
A.3. What Are Some Specific Aspects of and Issues Related to Methodology for
Probabilistic Risk Analysis?
This section briefly touches on a few key aspects of PRA, model development, and associated
uncertainties. Detailed introductions to PRA methodology are available elsewhere, such as Ang
and Tang (1984), Cullen and Frey (1999), EPA (2001), and Morgan and Henrion (1990).
Readers who seek more detail should consult these references and see the bibliography for
additional references.
A3.7. Developing a Probabilistic Risk Analysis Model
There are a number of key issues that should be considered in developing a PRA model. Some of
these are outlined below.
Structural Uncertainty in Scenarios
A potentially key source of uncertainty in an analysis is the scenario, which includes
specification of pollutant sources, transport pathways, exposure routes, timing and locations,
geographic extent, and related issues. As yet, there appears to be no formalized methodology for
dealing quantitatively with uncertainty and variability in scenarios. Decisions regarding what to
include or exclude from a scenario could be recast as hypotheses regarding which agents,
pathways, microenvironments, and so on contribute significantly to the overall exposure and risk
of interest. In practice, however, the use of qualitative methods tends to be more common, given
the absence of a formal quantitative methodology.
Coupled Models
For source-to-outcome risk assessments, it is often necessary to work with multiple models, each
of which represents a different component of a scenario. For example, there may be separate
models for emissions, air quality, exposure, dose, and effects. Such models may have different
spatial and temporal scales. When conducting an integrated assessment, there may be significant
challenges and barriers to coupling such models into one coherent framework. Sometimes, the
coupling is done dynamically in a software environment. In other cases, the output of one model
might be processed manually to prepare the information for input to the next model.
Furthermore, there may be feedbacks between components of the scenario (e.g., poor air quality
might affect human activity, which, in turn, could affect both emissions and exposures) that are
incompletely captured or not included at all. Thus, the coupling of multiple models can be a
potentially significant source of structural uncertainty.
A.3.2. Conducting the Probabilistic Analysis
27
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Quantifying Variability and Uncertainty in Model Inputs and Parameters
Once the models are selected or developed to simulate a scenario of interest, attention typically
turns to development of input data for the model. There is a substantial amount of literature
regarding the application of statistical methods for quantifying variability and uncertainty in
model inputs and parameters based on empirical data (e.g,. Ang and Tang, 1984; Cullen and
Frey, 1999; Morgan and Henri on, 1990; EPA 2001). For example, a commonly used method for
quantifying variability in a model input is to obtain a sample of data, select a type of parametric
probability distribution model to fit to the data (e.g., normal, lognormal, or other form), estimate
the parameters of the distribution based on the data, critique the goodness-of-fit using graphical
(e.g., probability plot) and statistical methods (e.g., Anderson-Darling, Chi-Square, or
Kolmogorov-Smirnov tests), and choose a preferred fitted distribution. This methodology can be
adjusted to deal with various types of data, such as data that are samples from mixtures of
distributions or that contain nondetected (censored) values. Uncertainties can be estimated based
on confidence intervals for statistics of interest, such as mean values, or the parameters of
frequency distributions for variability. Various texts and guidance documents, both Agency and
programmatic, describe these approaches, including the Guiding Principles for Monte Carlo
Analysis (EPA, 1997) and the internet site learner.org.
The most commonly used method for estimating a probability distribution in the output of a
model, based on probability distributions specified for model inputs, is Monte Carlo Simulation
(MCS) (Cullen and Frey, 1999; Morgan and Henri on, 1990). MCS is popular because it is very
flexible. MCS can be used with a wide variety of different types of probability distributions as
well as different types of models. The main challenge for MCS is that it requires repetitive model
calculations to construct a set of pseudo-random numbers for model inputs and the
corresponding estimates for model outputs of interest. There are alternatives to MCS that are
similar but more computationally efficient, such as Latin Hypercube Sampling (LHS).
Techniques are available for simulating correlations between inputs in both MCS and LHS. For
models with very simple functional forms, it may be possible to use exact or approximate
analytical calculations, but, in practice, such situations are encountered infrequently.
There may be situations in which the data do not conform to a well-defined probability
distribution. For such situations, Markov Chain Monte Carlo is an algorithm that samples the
data iteratively and randomly to estimate a so-called "likelihood function" (i.e., the probability
distribution and parameter estimates that provide the most likely explanation of the data). The
likelihood function is a key component of Bayesian inference and, therefore, serves as the basis
for some of the analytical approaches to variability and uncertainty described below.
The use of empirical data presumes that the data are a representative, random sample. However,
if there are known biases or other data quality problems, or if there is a scarcity or absence of
relevant data, then reliance on available empirical data is likely to lead to misleading inferences
in the analysis. Alternatively, estimates of variability and uncertainty can be encoded, using
formal protocols, based on elicitation of expert judgment (e.g., Morgan and Henrion, 1990).
Elicitation of expert judgment for subjective probability distributions is used in situations where
there are insufficient data to support a statistical analysis of uncertainty but in which there is
sufficient knowledge on the part of experts to make an inference regarding uncertainty. For
example, EPA has recently conducted an expert elicitation study on the concentration-response
relationship between annual average ambient PM2 5 exposures and annual mortality (IEC, 2006;
see also Case Studies 6 and 14). Subjective probability distributions that are based on expert
28
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
judgment can be "updated" with new data as they become available using Bayesian statistical
methods.
Structural Uncertainty in Models
There may be situations in which it proves useful to evaluate not just the uncertainties in inputs
and parameter values, but also uncertainties regarding whether a model adequately captures, in a
hypothesized, mathematical, structured form, the relationship under investigation. A qualitative
approach to evaluating the structural uncertainty in a model is to describe critical assumptions
within a model, the documentation of a model, or model quality. Quantitative approaches to
evaluating structural uncertainty in models are manifold. These include parameterization of a
general model that can be reduced to alternative functional forms (e.g., Morgan and Henrion,
1990), enumeration of alternative models in a probability tree (e.g., Evans et al., 1994),
comparing alternative models by evaluating likelihood functions (e.g., Royall, 1997; Burnham
and Anderson, 2002), pooling results of model alternatives using Bayesian updating (e.g.,
Hoeting et al., 1999), or testing the causal relationships within alternative models using Bayesian
Networks (Pearl 2000).
Sensitivity Analysis: Identifying the Most Important Model Inputs
Sensitivity analysis is complementary to probabilistic methods. There are many types of
sensitivity analysis methods, including, for example, simple techniques that involve changing the
value of one input at a time and assessing the effect on an output and statistical methods that
evaluate which of many simultaneously varying inputs contributes the most to the variance of the
model output. Sensitivity analysis can answer the following key questions.
• What is the impact of changes in input values on model output?
• How can variation in output values be apportioned among model inputs?
• What are the ranges of inputs associated with best or worst outcomes?
• What are the key controllable sources of variability?
• What are the critical limits (e.g., emission reduction target for a risk management strategy)?
• What are the key contributors to the output uncertainty?
Thus, sensitivity analysis can be used to inform decision making regarding research priorities
and risk management.
Probabilistic methods typically focus on the forward propagation of uncertainty or variability in
the input to a model with respect to uncertainty or variability in a model output. However, once a
probabilistic analysis is completed, sensitivity analysis typically takes the perspective of looking
backwards to evaluate how much of the variation in the model output is attributable to individual
model inputs (e.g., Frey and Patil, 2002; Mokhtari et al., 2006; Saltelli et al., 2004).
Iteration
There are two major types of iteration in risk assessment modeling. One is iterative refinement of
the type of analysis, perhaps starting with a relatively simple deterministic risk assessment as a
screening step in an initial level of analysis and proceeding to more refined types of assessments
as needed in subsequent levels of analysis. Examples of more refined levels of assessment
include application of sensitivity analysis to deterministic risk assessment; the use of
probabilistic methods to quantify variability only, uncertainty only, or co-mingled variability and
uncertainty (to represent a randomly selected individual); or the use of two-dimensional
29
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
probabilistic methods for distinguishing and simultaneously characterizing both variability and
uncertainty.
The other type of iteration occurs within a particular level and includes iterative efforts to
formulate a model, obtain data, and evaluate the model to prioritize data needs. For example, a
model may require a large number of input assumptions. To prioritize efforts of specifying
distributions for variability and uncertainty for model inputs, it is useful to determine which
model inputs are most influential with respect to the assessment end point. Therefore, sensitivity
can be used based on preliminary assessments of ranges or distributions for each model input to
determine which inputs are the most important to the assessment. Refined efforts to characterize
distributions then can be prioritized to the most important inputs.
30
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Appendix B: Glossary
Analysis. Examination of anything complex to understand its nature or to determine its essential
features (WHO IPCS Risk Assessment Terminology)
Assessment. The analysis and transformation of data into policy-relevant information that can
assist decision making and action.
Assessment end point. 1. Quantitative or qualitative expression of a specific factor or metric
with which a risk may be associated, as determined through an appropriate risk assessment. 2.
An explicit expression of the environmental value that is to be protected, operationally defined
by an ecological entity and its attributes. For example, salmon are valued ecological entities;
reproduction and age class structure are some of their important attributes. Together, salmon
"reproduction and age class structure" form an assessment end point.
Bayesian probability. An approach to probability, representing a personal degree of belief that
something will occur..
Critical control point. A controllable variable that can be adjusted to reduce exposure and risk.
For example, a critical control point might be the emission rate from a particular emission
source. The concept of critical control point is from the hazard assessment and critical control
point concept for risk management that is used in space and food safety applications, among
others.
Critical limit. A numerical value of a critical control point at or below which risk is considered
to be acceptable.
Ecological risk assessment. An ecological risk assessment evaluates the potential adverse
effects that human activities have on the plants and animals that make up ecosystems. The risk
assessment process provides a way to develop, organize, and present scientific information, so
that it is relevant to environmental decisions. When conducted for a particular place, such as a
watershed, the ecological risk assessment process can be used to identify vulnerable and valued
resources, prioritize data collection activity, and link human activities with their potential effects.
Ecosystem. The interacting system of a biological community (plants and animals) and its
nonliving environment.
Expert Elicitation. Expert elicitation (EE) is a systematic process of formalizing and
quantifying, typically in probabilistic terms, expert judgments about uncertain quantities.
Environment. The sum of all external conditions affecting the life, development, and survival of
an organism.
Frequentist (or frequency) probability. A view of probability that concerns itself with the
frequency of events in a long series of trials, or is based upon a data set.
Inputs. Quantities that are input to a model.
Model. 1. A set of constraints restricting the possible j oint values of several quantities. 2. A
hypothesis or system of belief regarding how a system works or responds to changes in its
31
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
inputs. 3. A mathematical function with parameters that can be adjusted so the function closely
describes a set of empirical data. A mechanistic model usually reflects observed or hypothesized
biological or physical mechanisms and has model parameters with real-world interpretation. In
contrast, statistical or empirical models selected for particular numerical properties are best fits
to data; model parameters may or may not have real-world interpretation. When data quality is
otherwise equivalent, extrapolation from mechanistic models (e.g., biologically based dose-
response models) often carries higher confidence than extrapolation using empirical models (e.g.,
logistic models).
Modeling. 1. Development of a mathematical or physical representation of a system or theory
that accounts for all or some of its known properties. Models often are used to test the effect of
changes of components on the overall performance of the system. 2. Use of mathematical
equations to simulate and predict real events and processes. 3. Development or application of
conceptual or graphical methods to depict the structure and organization among major elements
of the system to be modeled.
Model uncertainty (sources of).
Model structure. Reflects competing sets of conceptual, scientific, or technical assumptions
available to develop a model for a particular phenomenon. An example of model structure
uncertainty is the use of epidemiologically derived concentration-response functions for
modeling cancer risk in humans versus the use of toxicological (animal)-based functions
employing human-to-animal extrapolation factors.
Model detail. Reflects simplifying assumptions used to make modeling tractable. For example,
complex nonlinear behavior of chemical uptake from the gastrointestinal system into the blood
stream may be replaced by a rather simplified linear model, especially for specific exposure
(intake) ranges.
Extrapolation. Use of models outside of the parameter space used in their derivation may result
in erroneous predictions. For example, a threshold for health effects may exist at exposure levels
below those covered by a particular epidemiological study. If that study is used in modeling
health effects at those lower levels (and it is assumed that the level of response seen in the study
holds for lower levels of exposure), then disease incidence may be overestimated.
Resolution. Selection of spatial or temporal resolution (i.e., grid size) typically reflects a balance
between a desired level of precision and resources required to model the system. If the grid size
selected is too small, then key patterns of behavior reflected in the smaller step size may be
missed altogether, or the behavior of the system may be misrepresented. For example, efforts to
capture realistic high-end (near upper-bound) risks to farmers around an incinerator may
necessitate a geographical-information-system-based modeling framework precise enough to
model exposures for individual farms. If a less resolved exposure model is used, then risk to the
most exposed farm may be underpredicted.
Model boundaries. Decisions regarding the time, space, number of chemicals, etc., used in
guiding modeling of the system. Risks can be understated or overstated if the model boundary is
misspecified. For example, if a study area is defined to be too large and includes a significant
number of low-exposure areas, then a population-level risk distribution can be diluted by
including less exposed individuals, which can, in turn, result in a risk-based decision that does
not protect sufficiently the most exposed individuals in the study area.
Parameter. 1. A variable, measurable property whose value is a determinant of the
characteristics of a system (e.g., Temperature, pressure, and density are parameters of the
32
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
atmosphere.). 2. A constant or variable term in a function that determines the specific form of the
function but not its general nature, as "a" in f(x) = ax, where "a" determines only the slope of the
line described by f(x). 3. A variable entering into the mathematical form of any probability
distribution model such that the possible values of the variable correspond to different
distributions.
Probability. 1. Frequentist approach/ The frequency with which samples are obtained within a
specified range or for a specified category (e.g., the probability that an average individual with a
particular mean dose will develop an illness). 2. Bayesian approach. The degree of belief
regarding the different possible values of a quantity.
Probabilistic risk analysis. Application of a computational method, often based on a
randomized sampling of available data or information, to produce a probability distribution to
more fully describe the data than selecting a single point in the distribution, e.g., the mean.
Risk. 1. Risk includes consideration of exposure to the possibility of an adverse outcome, the
frequency with which one or more types of adverse outcomes may occur, and the severity or
consequences of the adverse outcomes if such occur. 2. The potential for realization of
unwanted, adverse consequences to human life, health, property, or the environment. 3. The
probability of adverse effects resulting from exposure to an environmental agent or mixture of
agents. 4. The combined answers to (1) What can go wrong? (2) How likely is it? and (3) What
are the consequences?
Risk analysis. 1. A process for identifying, characterizing, controlling, and communicating risks
in situations where an organism, system, subpopulation, or population could be exposed to a
hazard. Risk analysis is a process that includes risk assessment, risk management, and risk
communication (WHO). 2. A detailed examination, including risk assessment, risk evaluation,
and risk management alternatives, performed to understand the nature of unwanted, negative
consequences to human life, health, property, or the environment; an analytical process to
provide information regarding undesirable events; the process of quantification of the
probabilities and expected consequences for identified risks.
Risk assessment. 1. A process intended to calculate or estimate the risk to a given target
organism, system, subpopulation, or population, including the identification of attendant
uncertainties following exposure to a particular agent, taking into account the inherent
characteristics of the agent of concern, as well as the characteristics of the specific target system
(WHO). 2. The evaluation of scientific information on the hazardous properties of environmental
agents (hazard characterization), the dose-response relationship (dose-response assessment), and
the extent of human exposure to those agents (exposure assessment) (NRC, 1983). The product
of the risk assessment is a statement regarding the probability that populations or individuals so
exposed will be harmed and to what degree (risk characterization) (USEPA, 2000). 3. Qualitative
and quantitative evaluation of the risk posed to human health or the environment by the actual or
potential presence or use of specific pollutants.
Risk-informed decision making. An approach to decision making in which insights from
probabilistic risk analyses are considered with other insights and factors.
Risk management. A decision making process that takes into account environmental laws,
regulations, political, social, economic, engineering, and scientific information, including a risk
assessment, to weigh policy alternatives associated with a hazard.
33
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Scenario. 1. An outline or model of an expected or supposed sequence of events. 2. A set of
facts, assumptions, and inferences about how exposure takes place and regarding how exposures
translate into adverse effects that aides the analyst in evaluating, estimating, or quantifying
exposures and risks. Scenarios might include identification of pollutants, pathways, exposure
routes, and modes of action, among others.
Sensitivity analysis. A study of how the variation in data inputs (including inputs to models)
affect the outputs of a model or choice among potential decision options.
Levels. Refers to various hierarchical levels of complexity and refinement for different types of
modeling approaches that can be used in risk assessment. A deterministic risk assessment with
conservative assumptions is an example of a lower level type of analysis that can be used to
determine whether exposures and risks are below levels of concern. Examples of progressively
higher levels include the use of deterministic risk assessment coupled with sensitivity analysis,
the use of probabilistic techniques to characterize either variability or uncertainty only, and the
use of two-dimensional probabilistic techniques to distinguish between but simultaneously
characterize both variability and uncertainty.
Two-dimensional probabilistic analysis. A modeling approach in which inter-individual
variability in exposure and risk is characterized using frequency distributions, and in which
uncertainty in the estimates of statistics of the frequency distributions (e.g., the mean, median,
standard deviation, percentiles) are characterized using probability distributions.
Uncertainty. Occurs because of a lack of knowledge. It is not the same as variability. For
example, a risk assessor may be very certain that different people drink different amounts of
water but may be uncertain about how much variability there is in water intakes within the
population. Uncertainty often can be reduced by collecting more and better data, whereas
variability is an inherent property of the population being evaluated. Variability can be better
characterized with more data but it cannot be reduced or eliminated. Efforts to clearly distinguish
between variability and uncertainty are important for both risk assessment and risk
characterization, although they both may be incorporated into an assessment. [I agree with
Harvey's comment about this sentence.]
Uncertainty analysis. A detailed examination of the systematic and random errors of a
measurement or estimate; an analytical process to provide information regarding uncertainty.
Value of information. A quantitative measure of the value of knowing the outcome of an
uncertain variable prior to making a decision. Decision theory provides a means for calculating
the value of both perfect and imperfect information. The former value, informally known as the
value of clairvoyance, is an upper bound for the latter. Obtaining meaningful value-of-
information measurements requires an awareness of important restrictions (concerning the nature
of free will) on the validity of this kind of information.
Variability. Refers to true heterogeneity or diversity, as exemplified in natural variation . For
example, among a population that drinks water from the same source and with the same
contaminant concentration, the risks from consuming the water may vary. This may result from
differences in exposure (i.e., different people drinking different amounts of water and having
different body weights, different exposure frequencies, and different exposure durations), as well
as differences in response (e.g., genetic differences in resistance to a chemical dose). Those
inherent differences are referred to as variability. Differences among individuals in a population
34
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
are referred to as inter-individual variability, differences for one individual over time is referred
to as intra-individual variability.
35
-------
DRAFT INTERNAL TO EPA DO NOT CITE OR QUOTE
Appendix C: References
Ang, A. H-S., and W.H. Tang. 1984. Probability Concepts in Engineering Planning and Design, Volume 2:
Decision, Risk, and Reliability. John Wiley & Sons, New York.
Bloom, D.L., D.M. Byrne, and J.M. Andreson. 1993. Communicating Risk to Senior EPA Policy-Makers: A Focus
Group Study. Prepared by Bloom Research and the Office of Air Quality Planning and Standards. U.S.
Environmental Protection Agency, Research Triangle Park, NC.
Burnham, K.P., and D.R. Anderson. 2002. Model Selection and Inference: A Practical Information-Theoretic
Approach (2nd ed.). Springer-Verlag, New York.
Cullen, A.C., and H.C. Frey. 1999. Probabilistic Exposure Assessment: A Handbook for Dealing with Variability
and Uncertainty in Models and Inputs. Plenum Press, New York.
Clyde, M. et al. 2000. Effects of Ambient Fine and Coarse Particles on Mortality in Phoenix, Arizona. Duke
University, Durham, N.C.
ON MORTALITY IN PHOENIX, ARIZONAl
EPA, 1995. 1995. Policy for risk characterization. Science Policy Council, Washington, DC.
(http://www.epa.gov/osp/spc/rcpolicy.htm).
EPA. 1997a. Guiding Principles for Monte Carlo Analysis. EPA/630/R-97/001, Risk Assessment Forum. U.S.
Environmental Protection Agency, Washington.
EPA. 1998. Guidance for Submission of Probabilistic Human Health Exposure Assessments to the Office of
Pesticide Programs. Office of Pesticide Programs. U.S. Environmental Protection Agency, Draft, 11/4/98,
http://www.epa.gov/fedrgstr/EPA-PEST/1998/November/Dav-05/6021.pdf (accessed August 8, 2007).
EPA. 2000. EPA Science Policy Council Handbook, Risk Characterization. EPA 100-B-00-002.
EPA. 2001. Risk Assessment Guidance for Superfund: Volume III - Part A, Process for Conducting Probabilistic
Risk Assessment. EPA 540-R-02-002, Office of Emergency and Remedial Response, U.S. Environmental
Protection Agency, Washington, http://www.epa.gov/oswer/riskassessment/rags3adt/ (accessed August 8,
2007).
EPA. 2007. Letter from M.G. Morgan and R.T. Parkin (SAB) to S. Johnson (EPA), February 28, 2007. EPA-SAB-
07-003.
EPA. 2008. Case Study Examples of the Application of Probabilistic Risk Analysis in U.S. Environmental
Protection Agency Regulatory Decision-Making (in review). Risk Assessment Forum, U.S. Environmental
Protection Agency, Washington.
Evans, J.S., J.D. Graham, G.M. Gray, andR.L. Sielken. 1994. A Distributional Approach to Characterizing Low-
Dose Cancel Risk.. Risk Analysis, 14(l):25-34.
Fisher, L., P. Pascual, and W. Wagner. 2006. Mapping the Role of Models in US and EU Risk Regulation: A Legal
Background Paper.
Frey, H.C. 2004. Presentation at the EPA Colloquium on Probabilistic Risk Assessment, April 2004 (unpublished).
Frey, H.C., and S.R. Patil. 2002. Identification and Review of Sensitivity Analysis Methods. Risk Analysis,
22(3):553-578.
Hattis, D.B., and D.E. Burmaster. 1994. "Assessment of Variability and Uncertainty Distributions for Practical Risk
Analyses," Risk Analysis, 14(5):713-730.
Hoeting, J.A.; D. Madigan, A.E. Raftery, and C.T. Volinsky. 1999. BayesianModel Averaging: A Tutorial.
Statistical Science, 14(4):382-417.
IEC. 2006. Expanded Expert Judgment Assessment of the Concentration-Response Relationship Between PM25
Exposure and Mortality, Final Report. Prepared by Industrial Economics, Incorporated for Office of Air
Quality Planning and Standards, U.S. Environmental Protection Agency, Research Triangle Park, NC,
http://www.epa.gov/ttn/ecas/regdata/Uncertaintv/pm ee report.pdf.
Krupnick, A., R. Morgenstern, M. Batz, P. Nelson, D. Burtraw, J. Shih, and M. McWilliams. 2006. Not a Sure
Thing: Making Regulatory Choices Under Uncertainty. Resources for the Future, Washington.
Mokhtari, A., H.C. Frey, and J. Zheng. 2006. Evaluation and Recommendation of Sensitivity Analysis Methods for
Application to Stochastic Human Exposure and Dose Simulation (SHEDS) Models. Journal of Exposure
Science and Environmental Epidemiology, 16(6):491-506.
Morgan, M.G., and M. Henrion. 1990. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and
Policy Analysis. Cambridge University Press, New York.
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
NRC. 1983. Risk Assessment in the Federal Government: Managing the Process. Committee on the Institutional
Means for Assessment of Risks to Public Health, National Academy Press, Washington.
NRC. 1989. Risk Assessment in the Federal Government: Managing the Process. National Research Council,
National Academy Press, Washington.
NRC. 1991. Rethinking the Ozone Problem in Urban and Regional Air Pollution. National Research Council,
National Academy Press, Washington.
NRC. 1993. Issues in Risk Assessment. National Research Council, National Academy Press, Washington.
NRC. 1994. Science and Judgment in Risk Assessment. National Research Council, National Academy Press,
Washington.
NRC. 2002. Estimating the Public Health Benefits of Proposed Air Pollution Regulations. National Research
Council, National Academy Press, Washington.
NRC. 2007a. Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and
Budget. National Research Council, National Academy Press, Washington.
NRC. 2007b. Models in Environmental Regulatory Decision Making. National Research Council, National
Academy Press, Washington.
OSTP/OMB. 2007. Memorandum (M-07-24) for the Heads of Executive Departments and Agencies; from Susan E.
Dudley, Administrator, Office of Information and Regulatory Affairs, Office of Management and Budget,
to Sharon L. Hays, Associate Director and Deputy Director for Science, Office of Science and Technology
Policy: Updated Principles for Risk Analysis, September 19, 2007.
Parkin, R.T., and M.G. Morgan. 2007. Consultation on Enhancing Risk Assessment Practices and Updating EPA's
Exposure Guidelines. EPA-SAB-07-003, Science Advisory Board, U.S. Environmental Protection Agency,
Washington.
PCRARM. 1997. Framework for Environmental Health Risk Management Final Report, Volume 1.
Presidential/Congressional Commission on Risk Assessment and Risk Management,
http://www.riskworld.com/nreports/1997/risk-rpt/pdf/EPAJAN.PDF.
Pearl, J. 2000. Causality: models, Reasoning, and Inference. Cambridge University Press, New York
Royall, R.M. 1997. Statistical Evidence: A Likelihood Paradigm. Chapman & Hall, London.
Saltelli A, S. Tarantola, F. Campolongo, et al. 2004. Sensitivity Analysis in Practice. John Wiley & Sons, New
York.
Stahl, C.H., and A.J. Cimorelli. 2005. How Much Uncertainty Is Too Much and How Do We Know? A Case
Example of the Assessment of Ozone Monitor Network Options. RiskjAjiajYsis.^M^
WHO. 2008. Guidance on Characterizing and Communicating Uncertainty in Exposure Assessment (draft). The
International Programme on Chemical Safety, World Health Organisation, Geneva.
37
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Bibliography
Probabilistic Risk Analysis Methodology—General
Cullen, A.C., and H.C. Frey. 1999. The Use of Probabilistic Techniques in Exposure Assessment: A Handbook for
Dealing with Variability and Uncertainty in Models and Inputs. Plenum, New York.
EPA. 1997. Guiding Principles for Monte Carlo Analysis. EPA/630/R-97/001, U.S. Environmental Protection
Agency, Washington, March 1997.
Frey, H.C., D. Crawford-Brown, J. Zheng, and D. Loughlin. 2003. Hierarchy of Methods to Characterize
Uncertainty: State of Science of Methods for Describing and Quantifying Uncertainty, Draft. Prepared for
E.H. Pechan and Associates, Inc., Springfield, VA, for submittal to U.S. Environmental Protection Agency,
Research Triangle Park, NC, August 12, 2003.
Frey, C., J. Penman, L. Hanle, S. Monni, and S. Ogle. 2006. Uncertainties, Chapter 3 in Volume 1, General
Guidance and Reporting, 2006 IPCC Guidelines for National Greenhouse Gas Inventories. National
Greenhouse Gas Inventories Programme, Inter-Governmental Panel on Climate Change, Technical Support
Unit, Kanagawa, Japan.
Loughlin, D., H.C. Frey, K. Hanisak, and A. Eyth. 2003. Implementation Requirements forthe Development of a
Sensitivity/Uncertainty Analysis Tool for MIMS, Draft. Prepared by Carolina Environmental Program and
N.C. State University for U.S. Environmental Protection Agency, Research Triangle Park, NC, May 6,
2003.
Morgan, M.G., and M. Henrion. 1990. Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and
Policy Analysis. Cambridge University Press, New York.
NARSTO. 2005. Improving Emission Inventories for Effective Air Quality Management Across North America, A
NARSTO Assessment. NARSTO-05-001.
NRC. 1994. Science and Judgment in Risk Assessment. National Research Council, National Academy Press,
Washington.
Probabilistic Risk Analysis and Decision Making
Bloom, D.L., D.M. Byrne, and J.M. Andreson. 1993. Communicating Risk to Senior EPA Policy-Makers: A Focus
Group Study. Prepared by Bloom Research and the Office of Air Quality Planning and Standards, U.S.
Environmental Protection Agency, Research Triangle Park, NC.
EPA. 1997. Memo from Fred Hansen entitled Use of Probabilistic Techniques (including Monte Carlo Analysis) in
Risk Assessment, dated May 15, 1997. U.S. Environmental Protection Agency, Washington.
Krupnick et al. 2006. Not a Sure Thing: Making Regulatory Choices Under Uncertainty. Resources for the Future,
Washington.
McGarity, T.O., and W.E. Wagner. 2003. Legal Aspects of the Regulatory Use of Environmental Modeling.
Environmental Law Reporter, Volume 33, 10751-10774.
NRC. 1996. Understanding Risk: Informing Decisions in a Democratic Society. National Research Council,
National Academy Press, Washington.
Pascual, P., N. Steiber, andE. Sunderland. 2003. Draft Guidance on the Development, Evaluation, and Application
of Regulatory Environmental Models. Prepared by the Council for Regulatory Environmental Modeling,
Office of Science Policy, U.S. Environmental Protection Agency, Washington, November.
Thompson, K.M., and J.D. Graham. 1996. Going Beyond the Single Number: Using Probabilistic Risk Assessment
to Improve Risk Management. Human and Ecological Risk Assessment, 2(4): 1008-1034.
Probabilistic Risk Analysis Methodology—Specific Aspects
Abdel-Aziz, A., and H.C. Frey. 2003b. Development of Hourly Probabilistic Utility NOX Emission Inventories
Using Time Series Techniques: Part I-Univariate Approach. Atmospheric Environment, 37:5379-5389.
Abdel-Aziz, A., and H.C. Frey. 2003c. Development of Hourly Probabilistic Utility NOX Emission Inventories
Using Time Series Techniques: Part II-Multivariate Approach. Atmospheric Environment, 37:5391-5401.
Abdel-Aziz, A., and H.C. Frey. 2004. Propagation of Uncertainty in Hourly Utility NOX Emissions Through a
Photochemical Grid Air Quality Model: A Case Study for the Charlotte, NC Modeling Domain.
Environmental Science and Technology, 38(7):2153-2160.
38
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Gasman, E.A., M.G. Morgan, and H. Dowlatabadi. 1999. Mixed Levels of Uncertainty in Complex Policy Models.
Risk Analysis, 19(l):33-42.
Crawford-Brown and Brown. 1987. A Framework for Assessing the Rationality of Judgments in Carcinogenicity
Hazard Identification. Risk: Health, Safety and Environment, Fall, 307-337.
Draper, D. 1995. Assessment and Propagation of Model Uncertainty. Journal of Royal Statistical Society 57:45-97.
Evans, J.S., J.D. Graham, G.M. Gray, andR.L. Sielken. 1994. A Distributional Approach to Characterizing Low-
Dose Cancer Risk. Risk Analysis, 14(l):25-34.
Frey, H.C., and D.E. Burmaster. 1999. Methods for Characterizing Variability and Uncertainty: Comparison of
Bootstrap Simulation and Likelihood-Based Approaches. Risk Analysis, 19(1): 109-130 (February).
Frey, H.C., and D.S. Rhodes. 1996. Characterizing, Simulating, and Analyzing Variability and Uncertainty: An
Illustration of Methods Using an Air Toxics Emissions Example. Human and Ecological Risk Assessment:
an International Journal, 2(4):762-797 (December).
Frey, H.C., and Y. Zhao. 2004. Quantification of Variability and Uncertainty for Air Toxic Emission Inventories
With Censored Emission Factor Data. Environmental Science and Technology, 38(22):6094-6100.
Zheng, I, and H.C. Frey. 2004. Quantification of Variability and Uncertainty Using Mixture Distributions:
Evaluation of Sample Size, Mixing Weights and Separation between Components. Risk Analysis,
24(3):553-571 (June).
Zheng, J., and H.C. Frey. 2005. Quantitative Analysis of Variability and Uncertainty with Known Measurement
Error: Methodology and Case Study. Risk Analysis, 25(3):663-676.
Sensitivity Analysis
Agro, K.E., C. A. Bradley, and N. Mittmann. 1997. Sensitivity Analysis in Health Economic and Pharmacoeconomic
Studies - an Appraisal of the Literature. Pharmacoeconomics, ll(l):75-88.
Cukier, R.I., H.B. Levine, andK.E. Shuler. 1978. Nonlinear Sensitivity Analysis of Multi-parameter Model
Systems. Journal of Chemical Physics, 26(1): 1-42.
Frey, H.C., and R. Patil. 2002. Identification and Review of Sensitivity Analysis Methods. Risk Analysis, 22(3):553-
577.
Frey, H.C., A. Mokhtari, and J. Zheng. 2004. Recommended Practice Regarding Selection, Application, and
Interpretation of Sensitivity Analysis Methods Applied to Food Safety Process Risk Models. Prepared by
N.C. State University for U.S. Department of Agriculture, Washington, January 30, 2004. Available at
http://www.ncsu.edu/risk.
Helton, J.C., and F.J. Davis. 2002. Illustration of Sampling-Based Methods for Sensitivity Analysis. Reliability
Engineering and System Safety, 81(l):23-69.
Mokhtari, A., and H.C. Frey. 2005a. Recommended Practice Regarding Selection of Sensitivity Analysis Methods
Applied to Microbial Food Safety Process Risk Models. Human and Ecological Risk Assessment,
11(3):591-605.
Mokhtari, A., H.C. Frey, J. Zheng. 2006. Evaluation and Recommendation of Sensitivity Analysis Methods for
Application to EPA-Stochastic Human Exposure and Dose Simulation (SHEDS) Models. Journal of
Exposure Science and Environmental Epidemiology, 16(6) :491-506.
Saltelli, A., K. Chan, and M. Scott. 2000. Sensitivity Analysis, Probability and Statistics Series. John Wiley & Sons,
New York.
Sobol, I.M. 1993. Sensitivity Estimates for Nonlinear Mathematical Models. Mathematical Modeling and
Computation, 1(4):407-414.
Xue, J., V.G. Zartarian, H. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. 2006. A Probabilistic Arsenic
Exposure Assessment for Children Who Contact Chromated Copper Arsenate (CCA)-Treated Playsets and
Decks, Part 2: Sensitivity and Uncertainty Analyses. Risk Analysis, 26(2):533-541.
Case Study Examples of Probabilistic Risk Analysis—EPA (see also the PRA Case
Studies White Paper)
Babendreier, J.E., and K.L. Castleton. 2005. Investigating Uncertainty and Sensitivity in Integrated Multimedia
Environmental Models: Tools for FRAMES-3MRA. Environmental Modeling and Software, 20:1043-
1055.
39
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Blancato, J.N., F.W. Power, R.N. Brown, and C.C. Dary. 2004. Exposure Related Dose Estimating Model
(ERDEM) for Assessing Human Exposure and Dose, EPA/600/R-04/060, U.S. Environmental Protection
Agency, Research Triangle Park, NC, December 2004.
Burke, J. 2005. EPA Stochastic Human Exposure and Dose Simulation for Paniculate Matter (SHEDS-PM), User's
Guide. U.S. Environmental Protection Agency, Research Triangle Park, NC.
Frey, H. C. 2003. Evaluation of an Approximate Analytical Procedure for Calculating Uncertainty in the Greenhouse
Gas Version of the Multi-Scale Motor Vehicle and Equipment Emissions System. Prepared for Office of
Transportation and Air Quality, U.S. Environmental Protection Agency, Ann Arbor, MI, May 30, 2003.
Frey, H.C., and Y. Zhao. 2003. Development of Probabilistic Emission Inventories of Benzene, Formaldehyde and
Chromium for the Houston Domain. Prepared by N.C. State University for U.S. Environmental Protection
Agency, Research Triangle Park, NC, September 2003.
Frey, H.C., R. Bharvirkar, and J. Zheng. 1999. Quantitative Analysis of Variability and Uncertainty in Emissions
Estimation. Prepared by N.C. State University for U.S. Environmental Protection Agency, Research
Triangle Park, NC, July 1999.
Frey, H.C., J. Zheng, Y. Zhao, S. Li, and Y. Zhu. 2002. Technical Documentation of the AuvTool Software for
Analysis of Variability and Uncertainty. Prepared by N.C. State University for Office of Research and
Development, U.S. Environmental Protection Agency, Research Triangle Park, NC, February 2002.
Zartarian, V.G., J. Xue, H.A. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. 2005. A Probabilistic
Exposure Assessment for Children Who Contact CCA-Treated Playsets and Decks Using the Stochastic
Human Exposure and Dose Simulation Model for the Wood Preservative Scenario (SHEDS-WOOD), Final
Report. U.S. Environmental Protection Agency, Washington, EPA/600/X-05/009. Available at
http://www.epa.gov/heasd/sheds/cca_treated.htm.
Zartarian, V.G., J. Xue, H. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. 2006. A Probabilistic Arsenic
Exposure Assessment for Children Who Contact Chromated Copper Arsenate (CCA)-Treated Playsets and
Decks, Part 1: Model Methodology, Variability Results, and Model Evaluation. Risk Analysis, 26(2):515-
532.
Zheng, J., and H.C. Frey. 2002. AuvTool User's Guide. Prepared by N.C. State University for Office of Research
and Development, U.S. Environmental Protection Agency, Research Triangle Park, NC, February 2002.
Case Study Examples of Probabilistic Risk Analysis—Other
Bloyd, Gary, et al. 1996. Tracking and Analysis Framework (TAP) Model Documentation and User's Guide, 1996.
Argonne National Laboratory, ANL/DIS/TM-36 (December).
Frey, H.C., and S. Li. 2003. Quantification of Variability and Uncertainty in AP-42 Emission Factors: Case Studies
for Natural Gas-Fueled Engines. Journal of the Air & Waste Management Association, 53(12): 1436-1447
(December).
Hanna, S.R., Z. Lu, H.C. Frey, N. Wheeler, J. Vukovich, S. Arunachalam, M. Fernau, andD.A. Hansen. 2001.
Uncertainties in Predicted Ozone Concentrations due to Input Uncertainties for the UAM-V Photochemical
Grid Model Applied to the July 1995 OTAG Domain. Atmospheric Environment, 35(5):891-903.
Munns, W.R., Jr., H.A. Walker, and J.F. Paul. 1989. An Ecological Risk Assessment Framework for Examining the
Impacts of Oceanic Disposal Status and Trends Program. In: Oceans '89, Volume 2: Ocean Pollution,
Marine Technology Society and Oceanic Society of the Institute of Electrical and Electronics Engineers,
IEEE Publication No. 89CH2780-5, pp. 664-669.
Munns, W.R., Jr., H.A. Walker, J.F. Paul, and J.H. Gentile. 1996. A Prospective Assessment of Ecological Risks to
Upper Water Column Populations from Ocean Disposal at the 106-Mile Dumpsite. Journal of Marine
Environmental Engineering, 3:279-297'.
Nocito, J.A., H.A. Walker, J.F. Paul, and C. Menzie. 1988. Application of a Risk Assessment for Marine Disposal of
Sewage Sludge at Midshelf and Offshelf Sites. In: Proceedings of the Seventh International Ocean Disposal
Symposium, Environment Canada, pp. 644-663.
Nocito, J.A., H.A. Walker, J.F. Paul, and C.A. Menzie. 1989. Application of a Risk Assessment Framework for
Marine Disposal of Sewage Sludge at Midshelf and Offshelf Sites. In: Aquatic Toxicology and
Environmental Fate: Eleventh Volume, ASTM STP 1007, G.W. Suter II and M.A. Lewis (eds.), American
Society for Testing and Materials, Philadelphia, pp. 101-120.
Paul, J.F.,H.A. Walker, and V.J. Bierman, Jr. 1983. Probabilistic Approach for the Determination of the Potential
Area of Influence for Waste Disposed at the 106-Mile Ocean Disposal Site. In: 106-Mile Waste Disposal
40
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Site Characterization Update Report, J.B. Pearce, D.C. Miller, and C. Herman (eds.), NOAA Technical
Memorandum NMFS-F/NEC-26, National Marine Fisheries Service, Woods Hole, MA, pp. A-l to A-8.
Paul, J.F., V.J. Bierman, Jr., W.R. Davis, G.L. Hoffman, W.R. Munns, C.E. Pesch, P.P. Rogerson, and S.C.
Schimmel. 1988. The Application of a Hazard Assessment Research Strategy to the Ocean Disposal of a
Dredged Material: Exposure Assessment Component. In: Oceanic Processes in Marine Pollution, Vol. 5,
D.A. Wolfe and T.P. O'Connor (eds.), Kreiger Publishing Co., Melbourne, FL, pp. 123-135.
Prager, J.C., V.J. Bierman, Jr., J.F. Paul, and J.S. Bonner. 1986. Sampling the Oceans for Pollution: A Risk
Assessment Approach to Evaluating Low-Level Radioactive Waste Disposal at Sea. Dangerous Properties
of Industrial Materials Report, Vol. 6, No. 3, pp. 2-26.
Walker, H. A., J.F. Paul, J.A. Nocito, and J.H. Gentile. 1988. Ecological and Human Health Risks for Sewage Sludge
Disposal at the 106-Mile Site. In: Proceedings of the Seventh International Ocean Disposal Symposium,
Environment Canada, pp. 625-643.
41
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Appendix D: Case Study Examples of the Application of
Probabilistic Risk Analysis in U.S. Environmental Protection
Agency Regulatory Decision-Making
Prepared by Risk Assessment Forum
PRA Working Group 2
Allison Hess, David Hrdy, John Langstaff, Elizabeth Margosches, Michael
Messner, and Marian Olsen
Disclaimer
This document is a preliminary draft. It has not been formally released by the U.S. Environmental
Protection Agency and should not at this stage be construed to represent Agency policy. It is being
circulated for comments on its technical merit and policy implications. Mention of trade names or
commercial products does not constitute endorsement or recommendation for use.
42
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Foreword
The U.S. Environmental Protection Agency's (EPA's) Risk Assessment Forum was directed by
the Science Policy Council in the Office of the Science Advisor to consider how to better and
more fully implement probabilistic risk analysis (PRA) and related tools in the EPA decision-
making process. A technical panel of senior scientists gathered and identified several ways of
moving the agenda to fully implement PRA. This Appendix focuses on examples of how PRA
approaches have been used at EPA to inform regulatory decisions.
This White Paper was prepared by representatives from various EPA program offices and
regions currently involved in the development and application of PRA techniques. The
workgroup selected the case study examples based on the workgroup's knowledge of the specific
PRA procedures, the types of techniques demonstrated, availability to the reader through the
internet, peer-reviewed, and illustrative of a spectrum of PRA used in EPA. The case studies are
not designed to provide an exhaustive discussion of the wide variety of applications of PRA used
within the Agency but to highlight specific examples reflecting the range of approaches currently
applied within EPA.
43
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Acknowledgments
We would like to acknowledge the scientists and risk assessors who performed the
original analyses on which the summaries of these case studies are based. The names of the
points of contact are included for each study, and many more contributors were involved and
acknowledged in the original work.
Contributors and Reviewers to this Document
Gary Bangs, Jonathan Chen, Lisa Conner, Kathryn Gallagher, and Valerie Zartarian
The PRA Technical Panel
The EPA Risk Assessment Forum
44
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Executive Summary
This Appendix is designed to serve as a resource for risk managers faced with decisions
regarding when to apply PRA techniques to inform environmental decisions and for exposure
and risk assessors who may not be familiar with the wide range of available PRA approaches.
The document outlines categories of PRAs classified by the complexity of analysis to aid the
decision-making process. This approach identifies various PRA tools that include techniques
ranging from a simple sensitivity analysis (e.g., identification of key exposure parameters or data
visualization) requiring limited time, resources, and expertise to develop (Group 1); to
probabilistic approaches, including Monte Carlo analysis, that provide tools for evaluating
variability and uncertainty separately and require more resources and specialized expertise
(Group 2); and to sophisticated techniques of expert elicitation that generally require significant
investment of employee time, additional expertise, and external peer-review (Group 3).
This document describes case studies wherein PRA techniques have been used within this ranked
framework to provide additional information for risk managers. PRA is a scientific tool to help
describe the data or the risk and is one of many inputs considered by the risk manager in the
decision-making process. The case study summaries are provided in a format designed to
highlight how the results of the PRAs were considered in decision-making. These summaries
include specific information on the conduct of the analyses as an aid to determining what tools
might be appropriate for developing specific exposure or risk assessments for other assessments.
The case studies range from examples of less resource-intensive analyses that might assist in
identifying key exposure parameters or the need for more data to more detailed and resource-
intensive approaches. Tools include Monte Carlo modeling, sensitivity analyses, and application
of expert elicitation. Examples of applications in human health and ecological risk assessment
include the exposure of children to chromated copper arsenate treated wood, the relation between
particulates in air and health, dietary exposures to pesticides, modeling sea level change,
sampling watersheds, and modeling bird and animal exposures.
45
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
1. Introduction
Historically, the U.S. Environmental Protection Agency (EPA) has used deterministic risk
assessments, or point estimates of risk, to evaluate cancer risks and noncancer health hazards to
high-end exposed individuals (90th percentile or above) and the average exposed individual
(50th percentile) and, where appropriate, risks and hazards to populations, as required by specific
environmental laws (EPA, 1992a). The use of default values for exposure parameters in risk
assessment provides a procedural consistency that allows risk assessments to be feasible and
tractable (EPA, 2004). The methods typically used in EPA deterministic risk assessments rely on
a combination of point values—some conservative and some typical—yielding a point estimate
of exposure that is at some unknown point in the range of possible risks (EPA, 2004).
The development of sophisticated computational tools over the past 10 years has prompted an
increased interest in analyses that evaluate the variability and uncertainty in the risk assessments;
these include the use of tools such as probabilistic risk analysis, or PRA (EPA, 2001, 2004).
These analyses provide the results of the risk assessment as a probability or likelihood of
different risk levels in a population (describing variability) or to characterize uncertainty in risk
estimates.
This Appendix presents case studies of PRA conducted by EPA over the past 10 to 15 years.
Table 1 summarizes the case studies by title, technique demonstrated, classification based on
Human Health and Ecological Risk Assessment, and the program office responsible for
developing the case studies. This document, by illustration, provides a "snapshot" of utilization
of PRA across various programs in EPA.
46
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
2. Overall Approach to Probabilistic Risk Analysis at the U.S.
Environmental Protection Agency
2.1. U.S. Environmental Protection Agency Guidance and Policies on Probabilistic Risk
Analysis
The case studies presented here build on the principles of PRA outlined in EPA's 1996 Policy
(EPA, 1996) and Guiding Principles for Monte Carlo Analysis (EPA, 1997b) and subsequent
guidance documents on developing and using PRA. Guidance has been developed for the
Agency as well as for individual programs that refers to the use of PRA, including the Risk
Assessment Guidance for Superfund Part III (EPA, 2001); Risk Assessment Forum Framework
for Ecological Risk Assessment (EPA, 1992b); Guidelines for Ecological Risk Assessment
(EPA, 1998); Guidance for Risk Characterization (EPA, 1995a); Policy for Risk Characterization
(EPA, 1995a); Policy on Evaluating Health Risks to Children (EPA, 1995b); Policy for Use of
Probabilistic Analysis in Risk Assessment (EPA, 1997a); Guidance on Cumulative Risk
Assessment. Part 1. Planning and Scoping (EPA, 1997c ; and Risk Characterization Handbook
(EPA, 2000).
As shown in the individual case studies, the range and scope of the PRA will depend on the
overall objectives of the decision that the analysis will inform. The Guiding Principles for Monte
Carlo Analysis lay out the general approach that should be taken in all cases, beginning with
defining the problem and scope of the assessment, so that the best tools and approach may be
selected. The Guiding Principles also describe the process of estimating and characterizing
variability and uncertainty around the risk estimates. Stahl and Cimorelli (2005) and the Risk
Assessment Guidance for Superfund Volume III (EPA, 2001) highlight the importance of
communication between risk assessor and manager. Stahl and Cimorelli (2005) and Jamieson
(1996) indicate it is important to determine whether a particular level of uncertainty is acceptable
or not. The authors also suggest this decision is a matter of context, values, and regulatory
policy. The Risk Assessment Guidance for Superfund Part III (Chapter 2 and Appendix F in
EPA, 2001) describes a process for determining the appropriate level of PRA using a ranked
approach from the less resource- and time-intensive approaches to more sophisticated analyses
(Chapter 2 in EPA, 2001). Further, the Risk Assessment Guidance for Superfund Part III
outlines a process for developing a PRA work plan and a checklist for PRA reviewers (Chapter 2
and Appendix F in EPA, 2001). This guidance also provides information regarding how to
communicate PRA results to risk managers and stakeholders (Chapter 6 in EPA, 2001).
The guidance and policies on uncertainty and variability, and application of the principles of
PRA, all highlight the ongoing need for communication between the risk assessor and risk
manager. The ongoing communication is important in determining the appropriate levels of
analysis for the specific decision.
2.2. Categorizing Case Studies
47
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
The ranked approach used for categorization is a process for a systematic, informed progression
to increasingly more complex risk assessment methods of PRA that is outlined in the Risk
Assessment Guidance for Superfund (EPA, 2001). The use of categories provides a framework
for evaluating the various techniques of PRA. Higher categories reflect increasing complexity
and, in many cases, will require more time and resources. Higher categories also reflect
increasing characterization of variability and uncertainty in the risk estimate, which may be
important for making specific risk management decisions. Central to the approach is a
systematic, informed progression using an iterative process of evaluation, deliberation, data
collection, planning and scoping, development and updates to the work plan, and
communication. All of these steps focus on deciding
(1) whether or not the risk assessment, in its current state (i.e., deterministic risk analysis), is
sufficient to support risk management decisions (a clear path to exiting the process is
available at each step); and
(2) if the assessment is determined to be insufficient, whether or not progression to a higher
group of complexity (or refinement of the current analyses) would provide a sufficient
benefit to warrant the additional effort of performing a PRA.
This paper groups case studies according to level of effort and complexity of the analysis, and
the increasing sophistication of the methods used (Table 1). Although each group generally
represents increasing effort and cost, this may not always be true. The groups are intended to
also reflect the progression from simple to complex analysis that is determined by the interactive
planning and scoping efforts of the risk assessors and managers. The use of particular terms to
describe the groups, including tiers, was avoided due to specific programmatic and regulatory
connotations.
2.2.1. Group 1 Case Studies
Assessments within this group typically involve a sensitivity analysis and serve as an initial
screening step in the risk assessment. Sensitivity analyses identify important parameters in the
assessment where additional investigation may be helpful (Kurowicka and Cooke, 2006).
Sensitivity analysis can be simple or involve more complex mathematical and statistical
techniques such as correlation and regression analysis to determine which factors in a risk model
contribute most to the variance in the risk estimate.
Within the sensitivity analyses, a range of techniques is available: simple, "back of the envelope"
calculation, where the risk parameters are evaluated using a range of exposure parameters to
determine the parameter that contributes most significantly to the risk (Case Study 1); analyses to
rank relative contributions of variables to the overall risk (Case Study 2); and data visualization
using graphical techniques to array the data or Monte Carlo simulations (e.g., scatter plots).
More sophisticated analyses may include sensitivity ratios (i.e., elasticity); sensitivity scores (i.e.,
weighted sensitivity ratios); correlation coefficient or coefficient of determination, r2 (e.g.,
Pearson product moment, Spearman rank); normalized multiple regression coefficient; and
goodness-of-fit tests for subsets of the risk distribution (EPA, 2001).
48
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
The sensitivity analyses typically require limited resources and time to conduct. Results of the
sensitivity analyses are useful in identifying key parameters where additional Group 2 or 3
analyses may be appropriate. Sensitivity analyses are also helpful in identifying key parameters
where additional research will have the most impact on the risk assessment.
2.2.2. Group 2 Case Studies
Case studies within this group include more sophisticated application of probabilistic tools,
including PRA of specific exposure parameters (Case Studies 3 and 4), one-dimensional analyses
(Case Study 5), and probabilistic sensitivity analysis (Case Studies 6 and 7).
The Group 2 case studies require larger time commitments for development, specialized
expertise, and additional analysis of exposure parameter data sources. Depending on the nature
of the analysis, peer involvement or peer review may be appropriate to the evaluation of the
products of the analysis.
2.2.3. Group 3 Case Studies
Assessments within this group are the most resource- and time-intensive analyses of the three
categories. Risk analyses include two-dimensional Monte Carlo analysis that evaluate model
variability and uncertainty (Case Studies 8 through 10); Microexposure Event Analysis, in which
long-term exposure of an individual is simulated as the sum of separate short-term exposure
events (Case Study 11); and Probabilistic Analysis (Case Studies 12 and 13).
Other types of analyses within this group include the expert elicitation method that is a
systematic process of formalizing and quantifying, in terms of probabilities, experts' judgments
about uncertain quantities (Case Studies 14 and 15); Bayesian statistics that is a specialized
branch of statistics that views the probability of an event occurring as the degree of belief or
confidence in that occurrence (Case Study 16); and geostatistical analysis, which is another
specialized branch of statistics that explicitly takes into account the geo-referenced context of
data and the information (i.e., attributes) attached to the data.
The Group 3 analyses require additional time and expertise in the planning and analysis of the
assessment. Within this group, the level of expertise and resource commitments may vary with
techniques such as expert elicitation requiring significantly longer time for planning,
identification of experts, and meetings, when compared with the other techniques.
49
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Table 1. Case Study Titles, Description, Type of Assessment (Human Health or Ecological
Risk Assessment) and Program Office that Developed Assessment.
Study
No.
Title and Case Study Description
HH/
Eco1
Program
Office
Group 1 — Point Estimate - Sensitivity Analysis
1
2
Sensitivity Analysis of Key Variables in Probabilistic Assessment of
Children's Exposure to Arsenic in Chromated Copper Arsenate (CCA)
Pressure-Treated Wood. This case study demonstrates use of a point
estimate sensitivity analysis to identify exposure variables critical to the
analysis summarized in Case Study 9. The sensitivity analysis identified
critical areas for future research and data collection and better characterized
the amount of dislodgeable residue that exists on the wood surface.
Assessment of Relative Contribution of Atmospheric Deposition to
Watershed Contamination. An example of a workbook that demonstrates
how "back-of-the-envelope" analysis of potential exposure rates can be
used to target resources to identify other inputs before further analysis of
air inputs in watershed contamination. Identification of key variables aided
in identifying uncertainties and data gaps to target resource expenditures
for further analysis. A case study example of the application of this
technique is also identified.
HH
Eco
OPP/ORD2
ORD
Group 2 — Probabilistic Risk Analysis, One-Dimensional Monte Carlo Analysis, and
Probabilistic Sensitivity Analysis
Probabilistic Risk Analysis
3
4
Probabilistic Assessment of Angling Duration Used in Assessment of
Exposure to Hudson River Sediments via Consumption of
Contaminated Fish. A probabilistic analysis of one parameter in an
exposure assessment — the time an individual fishes in a large river
system. Development of site-specific information regarding exposure,
with an existing data set for this geographic area, was needed to represent
this exposed population. This information was used in the one-
dimensional PRA described in Case Study 5.
Probabilistic Analysis of Dietary Exposure to Pesticides for Use in
Setting Tolerance Levels. The probabilistic Dietary Exposure
Evaluation Model (DEEM) provides more accurate information on the
range and probability of possible exposures.
HH
HH
Region 21
Superfund
OPP
1 HH = Human Health Risk Assessment; Eco = Ecological Risk Assessment.
2 OPP, Office of Pesticides Programs; ORD, Office of Research and Development; OAR, Office of Air and
Radiation; OW, Office of Water.
50
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Study
No.
Title and Case Study Description
HH/
Eco
Program
Office
Group 2 (cont'd.)
One-Dimensional Monte Carlo Analysis
5
One-Dimensional Probabilistic Risk Analysis of Exposures to
Polychlorinated Biphenyls (PCBs) via Consumption of Fish from a
Contaminated Sediment Site. An example of a one-dimensional PRA (1-
D Monte Carlo analysis of the variability of exposure as a function of the
variability of individual exposure factors.) to evaluate the risks to anglers
who consume recreationally caught fish from a PCB-contaminated river.
HH
Region 21
Superfund
Probabilistic Sensitivity Analysis.
6
7
Probabilistic Sensitivity Analysis of Expert Elicitation of
Concentration-Response Relationship Between PM2.s Exposure and
Mortality. An example of how the probabilistic analysis tools can be used
to conduct a probabilistic sensitivity analysis following an expert
elicitation (Group 3) presented in Case Study 14.
Environmental Monitoring and Assessment Program (EMAP): Using
Probabilistic Sampling Techniques To Evaluate the Nation's
Ecological Resources . A probability-based sampling program designed to
provide unbiased estimates of the condition of an aquatic resource over a
large geographic area based on a small number of samples.
HH
Eco
OAR
ORD
Group 3 — Advanced Probabilistic Risk Analysis — Two-Dimensional Monte Carlo Analysis Including
Microexposure Modeling, Bayesian Statistics, Geostatistics, and Expert Elicitation
Two- Dimensional Probabilistic Risk Analysis
8
9
Two-Dimensional Probabilistic Risk Analysis of Cryptosporidium in
Public Water Supplies, with Bayesian Approaches to Uncertainty
Analysis. An analysis of variability in the occurrence of Cryptosporidium
in raw water supplies and in the treatment efficiency, as well as the
uncertainty in these inputs. This case study includes an analysis of the
dose-response relationship for Cryptosporidium infection.
Two-Dimensional Probabilistic Model of Children's Exposure to
Arsenic in Chromated Copper Arsenate (CCA) Pressure-Treated
Wood. A two-dimensional model that addresses both variability and
uncertainty in the exposures of children to CCA pressure-treated wood.
The analysis was built on the sensitivity analysis described in Case Study
2.
HH
HH
OW
OPP/ORD
51
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Study
No.
Title and Case Study Description
HH/
Eco
Program
Office
Group 3 (cont'd.)
10
Two-Dimensional Probabilistic Exposure Assessment of Ozone
A probabilistic exposure assessment that addresses short-term exposures to
ozone. Population exposure to ambient ozone levels was evaluated using
EPA's Air Pollutants Exposure (APEX) model, also referred to as the
Total Risk Integrated Methodology /Exposure (TRIM.Expo) model.
HH
OAR
Group 3 — Microexposure Event Modeling
Microexposure Event Analysis
11
Analysis of Microenivironmental Exposures to Particulate Matter
(PM2.s) for a Population Living in Philadelphia, PA. A microexposure
event analysis to simulate individual exposures to PM2 5 in specific
microenvironments including the outdoors, indoor residences, offices,
schools, stores, and a vehicle.
HH
Region 3
and ORD
Probabilistic Analysis
12
13
Probabilistic Analysis in Cumulative Risk Assessment of
Organophosphorus Pesticides. A probabilistic computer software
program used to integrate various pathways, while simultaneously
incorporating the time dimensions of the input data to calculate margins of
exposure.
Probabilistic Ecological Effects Risk Assessment Models for
Evaluating Pesticide Uses. A multimedia exposure/effects model that
evaluates acute mortality levels in generic or specific avian species over a
user-defined exposure window.
HH
Eco
OPP
OPP
Group 3 — Expert Elicitation and Bayesian Belief Network
Expert Elicitation
14
15
16
Expert Elicitation of Concentration-Response Relationship Between
Particulate Matter (PM2.5) Exposure and Mortality. An expert
elicitation used to derive probabilistic estimates of the uncertainty in one
element of a cost-benefit analysis used to support the PM2 5 regulations.
Expert Elicitation of Sea-Level Change Resulting from Global Climate
Change. An example of a PRA that describes the probability of sea level
rise and parameters that predict sea level change.
Expert Elicitation for Bayesian Belief Network Model of Stream
Ecology. An example of a Bayesian belief network model of the effect of
increased fine-sediment load in a stream on macroinvertebrate populations.
HH
Eco
Eco
ORD/
OAR
ORD
52
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
3. Case Study Summaries
Group 1 Case Studies
Case Study 1: Sensitivity Analysis of Key Variables in Probabilistic Assessment of
Children's Exposure to Arsenic in Chromated Copper Arsenate (CCA) Pressure-
Treated Wood
This case study provides an example of the application of sensitivity analysis to identify
important variables for population exposure variability for a Group 2 assessment (Case Study 9)
and to indicate areas for further research. Specifically, EPA's Office of Research and
Development (ORD), in collaboration with the Office of Pesticide Programs (OPP) used
sensitivity analyses to identify the key variables in children's exposure to CCA treated wood.
Approach. The sensitivity analyses used two approaches. The first approach estimated baseline
exposure by running the exposure model with each input variable set to its median (50th
percentile) value. Next, alternative exposure estimates were made by setting each input to its
25th or 75th percentile value while holding all other inputs at their median values. The ratio of the
exposure estimate calculated when an input was estimated at its 25th or 75th percentile to the
exposure estimate calculated when the input was at its median value provided a measure of that
input's importance to the overall exposure assessment. The second approach applied multiple
stepwise regression analysis to the data points generated from the first approach. The correlation
between the input variables and the exposure estimates provided an alternative measure of the
input variable's relative importance in the exposure assessment. These two approaches were
used in tandem to identify the critical inputs to the exposure assessment model.
Results of Analysis^ The two sensitivity analyses together identified six critical input variables
that most influenced the exposure assessment. The critical input variables were: wood surface
residue-to-skin transfer efficiency, wood surface residue levels, fraction of hand surface area
mouthed per mouthing event, average fraction of nonresidential outdoor time spent playing on a
CCA-treated playset, frequency of hand-washing, and frequency of hand-to-mouth activity.
Management Considerations: The results of the sensitivity analyses were used to identify the
most important input parameters in the treated wood risk assessments. The process also
identified critical areas for future research. In particular, the assessment pointed to a need to
collect data on the amount of dislodgeable residue that is transferred from the wood surface to a
child's hand upon contact, and to better characterize the amount of dislodgeable residue that
exists on the wood surface.
53
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Document Availability.
Title: The final report on the probabilistic exposure assessment of CCA-treated wood.
Zartarian, V.G., J. Xue, H. A. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. A
Probabilistic Exposure Assessment for Children Who Contact CCA-treated Playsets and Decks
Using the Stochastic Human Exposure and Dose Simulation Model for the Wood Preservative
Scenario (SHEDS-WOOD), Final Report. U.S. Environmental Protection Agency, Washington,
DC, EPA/600/X-05/009. http://www.epa.gov/heasd/sheds/ccajreated.htm
See also: Xue, J., Zartarian, V.G., Ozkaynak, H., Dang, W., Glen, G., Smith, L., and Stallings, C.
A probabilistic arsenic exposure assessment for children who contact chromated copper arsenate
(CAA)-treated playsets and decks, Part 2: Sensitivity and uncertainty analyses. Risk Analysis
26:533,2006.
Contact PersonJDr. Jianping Xue, EPA's Office of Research and Development, 919-541-7962,
xue.jianping@epa.gov
54
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 2: Assessment of Relative Contribution of Atmospheric Deposition to
Watershed Contamination
Watershed contamination can result from several different sources, including direct release into a
water body, input from upstream water bodies, and deposition from airborne sources. Efforts to
control water body contamination begin with an analysis of the environmental sources in order to
identify those parameters providing the greatest contribution and to determine where mitigation
and/or analysis resources should be directed.
Approach. This case study provides an example of a back-of-the envelope analysis of the
contribution of air deposition to overall watershed contamination to identify uncertainties and/or
data gaps as well as to target resource expenditures. (Group 1: Deterministic Analysis). Nitrogen
inputs have been studied in several east and Gulf Coast estuaries due to concerns about
eutrophication. Nitrogen from atmospheric deposition is estimated to be as high as 10 to 40% of
the total input of nitrogen to many of these estuaries and perhaps higher in a few cases. For a
watershed that has not already been studied, a back-of-the envelope calculation could be
prepared based on information based on nitrogen deposition rates measured in a similar area. To
estimate the deposition load directly to the waterbody, one would multiply the nitrogen
deposition rate by the area of the waterbody. The analyst could then estimate the nitrogen load
from other sources, (e.g., point source discharges and runoff) to estimate a total nitrogen load for
the waterbody. The estimate of loading due to atmospheric deposition could then be divided by
the total nitrogen load for the waterbody to estimate the percent contribution directly to the
waterbody from atmospheric deposition.
The May 2003 report by the Casco Bay Air Deposition Study Team titled "Estimating Pollutant
Loading from Atmospheric Deposition Using Casco Bay, Maine" is an analysis using the
methodology described above. The Casco Bay Estuary, located in the southwestern Maine, is
used as a case study. The paper also includes the results of a field air deposition monitoring
program conducted in Casco Bay (1998 - 2000) and favorably compares the estimates developed
for rate of deposition of nitrogen, mercury and PAHs to the field monitoring results. The
estimation approach is a useful starting point for understanding the sources of pollutants entering
water bodies that cannot be accounted for through run-off or point source discharges.
Results of Analysis^ The approach outlined above was applied to the Casco Bay Estuary in
Maine. Resources, tools and strategies for pollution abatement can be effectively targeted at
priority sources if estuaries are to be protected. Understanding the sources and annual loading of
contaminants to an estuary guides good water quality management by defining the range of
controls of both air and water pollution needed to achieve a desired result. The cost of
conducting monitoring to determine atmospheric loading to a water body can be prohibitively
high. Also, collection of monitoring data is a long-term undertaking, since a minimum of three
years of data is advisable in order to "smooth out" inter-annual variability. The estimation
techniques described in this paper can serve as a useful and inexpensive "first-cut" at
understanding the importance of the atmospheric as a pollution source, and can help to pinpoint
those areas where field measurements are needed to guide future management decisions.
55
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Management Consideration: If a review of information on air deposition available for the
analysis indicates a wide range of potential deposition rates, then further study of this input
would lead to better characterization of the air contribution to overall contamination. If the back-
of-the envelope analysis suggests that air deposition is very small relative to other inputs, then
resources should be targeted at studying or reducing other inputs before proceeding with further
analysis of the air inputs.
Document Availability. The back-of-the envelope calculation is outlined in Frequently Asked
Questions about Atmospheric Deposition: A Handbook for Watershed Managers (available at
http://www.epa.gov/air/oaqps/gr8water/handbook/airdep sept.pdf).
Further analysis is available in Deposition of Air Pollutants to the Great Waters - Third Report to
Congress (available at http://www.epa.gov/air/oaqps/gr8water/3rdrpt/index.html)
The Casco Bay Estuary examples is available at: www.epa.gov/owow/airdeposition/index.html.
Contact Person. Gail Lacy at (919) 541-5261 at lacy.gail@epa.gov.
Contact for Casco Site is: Diane Gould at gould.diane@epa.gov.
56
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Group 2 Case Studies
Case Study 3: Probabilistic Assessment of Angling Duration Used in Assessment of
Exposure to Hudson River Sediments via Consumption of Contaminated Fish
In assessing the health impact of contaminated Superfund sites, exposure duration typically is
assumed to be the same as the length of time an individual lives in a specific area (i.e., residence
duration). In conducting the human health risk assessment for the Hudson River PCB Superfund
Site, however, there was concern that exposure duration based on residence duration may
underestimate the time spent fishing (i.e., angling duration).
Risk Analysis. An individual may move from one residence to another and continue to fish in
the same location, or an individual may choose to stop fishing irrespective of the location of his
or her home. EPA Region 2 developed a site-specific distribution of angling duration using the
fishing patterns reported in a New York State-wide angling survey (Connelly et al., 1991) and
migration data for the five counties surrounding the 40-plus miles of the Upper Hudson River
collected as part of the U.S. Census.
Results of Analysis. The 50th and 95th percentile values from the distribution of angling
durations were higher than the default values based on residence duration using standard default
exposure assumptions for residential scenarios and were used as bases for the central tendency
and reasonable maximum exposure point estimates, respectively, in the deterministic assessment.
Management Considerations. The information provided in this analysis was used in the point
estimate analysis. The full distribution was used in conducting a Group 2 PRA for the fish
consumption pathway, which is presented as Case Study 6.
Document Availability. The final risk assessment was released in November 2000 (available at
http ://www. epa. gov/hudson/reports. htm).
Contacts. Remedial Project Manager, Alison Hess, 212-637-3959; Risk Assessor, Marian Olsen,
212-637-4313
57
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 4: Probabilistic Analysis of Dietary Exposure to Pesticides for use in
Setting Tolerance Levels
Under the Federal Food, Drug, and Cosmetic Act (FFDCA), EPA may authorize a tolerance or
exemption from the requirement of a tolerance, to allow a pesticide residue in food, only if the
Agency determines that such residues would be "safe". This determination is made by
estimating exposure to the pesticide and comparing the estimated exposure to a toxicological
benchmark dose (i.e., a dose where there is reasonable certainty of no harm). Until 1998, Office
of Pesticide Programs (OPP) used a software program called the Dietary Risk Evaluation System
(ORES) to conduct its acute dietary risk assessments for pesticide residues in foods. Acute
assessments conducted with ORES assumed that 100% of a given crop with registered uses of a
pesticide was treated with that pesticide and that all such treated crop items contained pesticide
residues at the maximum legal (tolerance) level matching this to a reasonably high consumption
value (around 95th percentile). The resulting ORES acute risk estimates were considered "high-
end" or "bounding" estimates. However, it was not possible to know where the pesticide
exposure estimates from the ORES software fit in the overall distribution of exposures due to the
limits of the tools being used.
Approach: To address these deficiencies, OPP has developed an acute probabilistic dietary
exposure guidance in order to use a model to estimate exposure to pesticides in the food supply.
Rather than the crude "high-end," single point estimates provided by deterministic assessments,
the probabilistic Dietary Exposure Evaluation Model (DEEM) provides specific information on
the range and probability of possible exposures and depending upon the characterization of the
input, 95th percentile regulation generally for lower tiers that do not include percent crop treated,
to the 99.9th percentile for the more refined assessments which would include percent of crop
treated information.
Probabilistic Analysis. This case study provides an example of a one-dimensional probabilistic
risk assessment of dietary exposure to pesticides (Group 2). The DEEM generates acute,
probabilistic dietary exposure assessments using data on (1) the distribution of daily
consumption of specific commodities (e.g., wheat, corn, apples, etc.) by specific individuals, and
(2) the distribution of concentrations of a specific pesticide in those food commodities. Data on
commodity consumption are collected by USDA in its Continuing Survey of Food Intake by
Individuals (CSFII). Pesticide residue concentrations on food commodities are generally
obtained from crop field trials, USDA's Pesticide Data Program (PDF) data, Food and Drug
Administration (FDA) monitoring data, or market basket surveys conducted by the registrants.
Using these data, DEEM is able to calculate an estimate of the risk to the general U.S. population
in addition to 26 population subgroups, including five subgroups for infants and children (infants
less than 1, children 1-2, children 3-5, youth 6-12 and teen 13-19).
Results of Analysis. DEEM has been used in risk assessments to support tolerance levels for
several pesticides (e.g., phosalone) and as part of cumulative risk assessments for
organophosphorus compounds (see Case Study 11) and other pesticides.
Management Considerations: Using the ORES, risk management decisions were being made
without a full picture of the distribution of risk among the population, and also without full
58
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
knowledge of where in the distribution of risk the ORES risk estimate lay. This was of concern
not only for regulators interested in public health protection, but also for the pesticide registrants
who could argue that the Agency was being arbitrary in selecting the level at which to regulate.
For most cases reviewed by OPP to date, estimated exposure at the 99.9th percentile calculated
by DEEM probabilistic techniques is significantly lower than exposure calculated using DRES-
type deterministic assumptions at the unknown percentile.
Document Availability.
Link to DEEM model available at http://www.epa.gov/oppsrrdl/cumulative/methods tools.htm
Contact Person.
David Hrdy at (703) 305-6990 or hrdy.david@epa.gov.
59
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 5: One-Dimensional Probabilistic Risk Analysis of Exposure to
Polychlorinated Biphenyls (PCBs) via Consumption of Fish from a Contaminated
Sediment Site
EPA Region 2 conducted a preliminary deterministic human health risk assessment at the
Hudson River PCBs Superfund site. The deterministic risk analysis showed that consumption of
recreationally caught fish provided the highest exposure among relevant exposure pathways and
resulted in cancer risks and noncancer health hazards that exceeded regulatory benchmarks.
Probabilistic Analysis. Because of the size, complexity, and high level of public interest in this
site, EPA Region 2 implemented a Group 2 probabilistic assessment to characterize the
variability in risks associated with the fish consumption exposure pathway. The analysis was a 1-
Dimensional Monte Carlo analysis of the variability of exposure as a function of the variability of
individual exposure factors. Uncertainty was assessed using sensitivity analysis of the input
variables. Data to characterize distributions of exposure parameters were drawn from the
published literature (e.g., fish consumption rate) or from existing databases such as the U.S.
Census data (e.g., angling duration, see Case Study 3). Mathematical models of the
environmental fate, transport, and bioaccumulation of PCBs in the Hudson River previously
developed were used to forecast changes in PCB concentration over time.
Results of Analysis. The results of the PRA were in line with the deterministic results. For the
Central Tendency individual, point estimates were near the median (50th percentile). For the
Reasonable Maximum Exposure individual, point estimate values were at or above the 95th
percentile of the probabilistic analysis. The deterministic and probabilistic risk analyses were the
subject of a formal peer review by a panel of independent experts.
The Monte Carlo base case scenario is the one from which point estimate exposure factors for
fish ingestion were drawn, thus the point estimate RMEs and the Monte Carlo base case
estimates can be compared. Similarly, the point estimate central tendency (average) and the
Monte Carlo base case midpoint (50th percentile) are comparable. For cancer risk, the point
estimate RME for fish ingestion (1 x 10"3) falls approximately at the 95th percentile from the
Monte Carlo base case analysis. The point estimate central tendency value (3 xlO"5) and the
Monte Carlo base case 50th percentile value (6 xlO"5) are similar. For non-cancer health hazards,
the point estimate RME for fish ingestion (104 for young child) falls between the 95th and 99th
percentiles of the Monte Carlo base case. The point estimate central tendency hazard index (HI)
(12 for young child) is approximately equal to the 50th percentile of the Monte Carlo base case
HI of 11. Figures 1 and 2 provide a comparison of results from the probabilistic analysis with
that of the deterministic risk analysis for cancer risks and non-cancer health hazards.
Management Considerations. Early and continued involvement of the community improved
public acceptance of the results. In addition, careful consideration of the methods used to present
the probabilistic results to the public lead to greater understanding of the findings.
Document Availability. The final risk assessment was released in November 2000 (available at
http://www.epa.gov/hudson/reports.htm).
60
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Contacts. Remedial Project Manager, Alison Hess, 212-637-3959; Risk Assessor, Marian Olsen,
212-637-4313
A comparison of results from the probabilistic analysis with that of the deterministic risk analysis
for cancer risks and non-cancer health hazards. Figures 1 and 2 plot percentiles for 72
combinations of exposure variables (e.g., distributions from creel angler surveys; residence
duration; fishing locations; cooking losses, etc.) of the non-cancer Hazard Index values and the
cancer risks, respectively. In each of these figures, the variability of cancer risk or non-cancer
His for anglers within the exposed population is plotted on the y axis for particular percentiles
within the population. This variability is a function of the variations in fish consumption rates,
fishing duration, differences in fish species ingested, etc. The uncertainty in the estimates is
indicated by the range of either cancer risk or non-cancer HI values plotted on the x-axis. This
uncertianty is a function of the 72 combinations of the exposure factor inputs examined in the
sensitivity analysis. This analysis provides a semi-quantitative confidence interval for the cancer
risks and HI values at any particulate percentile. As these figures show, the intervals span
somewhat less than two orders of magnitude (e.g., < 100 fold). The vertical lines indicate the
deterministic endpoints.
61
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Figure 1. Monte Carlo Cancer Summary Based on a One-Dimensional Probabilistic Risk
Analysis of Exposure to Polychlorinated Biphenyls (PCBs) via Consumption of Fish from
a Contaminated Sediment Site. From Phase 2 Report: Further Site Characterization and
Analysis. Volume 2F - Revised Human Health Risk Assessment, Hudson River PCBs
Reassessment RI/FS. U.S. EPA, November 2000.
Range of Cancer Risk Estimates
Fraction
of
Anglers
with
Risk<
than
Indicate
d Value
100%
75% --
50%
25% --
0%
Central Tendency
Value
RME
Value
o.i i 10 100 1000 10000 100000
Estimated Cancers in Population of 1 Million
62
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Figure 2. Monte Carlo Non-Cancer Hazard Index Summary Based on a One-Dimensional
Probabilistic Risk Analysis of Exposure to Polychlorinated Biphenyls (PCBs) via
Consumption of Fish from a Contaminated Sediment Site. From Phase 2 Reprot: Further
Site Characterization and Analysis. Volume 2F - Revised Human Health Risk Assesment,
Hudson River PCBs Reassessment RI/FS. U.S. EPA, November 2000.
Range of Non-Cancer Hazard Index (HI) Estimates for Fish Ingestion
Fraction of
Anglers
with
Hl<
Indicated
Value
100%
75%
50%
25%
0%
Central
Tendency
«
RME Value
0.1 1 10 100 1000 10000
Incremental Individual Hazard Index
63
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 6: Probabilistic Sensitivity Analysis of Expert Elicitation of
Concentration-Response Relationship Between Particulate Matter (PM2.5) Exposure
and Mortality
In 2002, the National Research Council (NRC) recommended that EPA improve its
characterization of uncertainty in the benefits assessment for proposed regulations of air
pollutants. NRC recommended that probability distributions for key sources of uncertainty be
developed using available empirical data or through formal elicitation of expert judgments. In
response to this recommendation, EPA conducted an expert elicitation evaluation of the
concentration-response relationship between PM2 5 exposure and mortality, a key component of
the benefits assessment of the PM2.5 regulation. Further information on the expert elicitation
procedure and results is provided in Case Study 12. To evaluate the degree to which the results
of the assessment depended on individual experts' judgments or on the methods of expert
elicitation, a probabilistic sensitivity analysis was performed of the results
Probabilistic Risk Analysis. The expert elicitation procedure used carefully constructed
interviews to elicit from each of 12 experts an estimate of the probabilistic distribution for the
average expected decrease in U.S. annual, adult, all-cause mortality associated with a 1 ug/m3
decrease in annual average PM2.5 levels. This case study provides an example of the use of
probabilistic sensitivity analysis (Group 2) as one element of the overall assessment. For the
sensitivity analysis, a simplified benefits analysis was conducted to assess the sensitivity of the
results to the responses of individual experts and to three factors in the study design: (1) the use
of parametric or nonparametric approaches by experts to characterize their uncertainty in the
PM2.5 -mortality coefficient, (2) participation in the Pre-elicitation Workshop, and (3) allowing
experts to change their judgments after the Post-elicitation Workshop. The individual
quantitative expert judgments were used to estimate a distribution of benefits, in the form of
number of deaths avoided, associated with a reduction in ambient, annual average PM2.5
concentrations from 12 to 11 ug/m3. The 12 individual distributions of estimated avoided deaths
were then pooled using equal weights to create a single overall distribution reflecting input from
each expert. This distribution served as the baseline for the sensitivity analysis, which compared
the means and standard deviations of the baseline distribution with several variants.
Results of Analysis. The first analysis examined sensitivity of the mean and standard deviation
of the overall mortality distribution to the removal of individual experts' distributions. In
general, the results suggested a fairly equal split between those experts whose removal shifted
the distribution mean up and those who shifted it down and relatively modest impacts of
individual experts. The standard deviation of the combined distribution also was not affected
strongly by removal of individual experts. The second analysis evaluated whether the use of
parametric or nonparametric approaches affected the overall results. The results suggested that
the use of parametric distributions led to distributions with similar or slightly increased
uncertainty compared with distributions provided by experts who offered percentiles of a
nonparametric distribution. The last analyses evaluated whether participation in the Pre- or Post-
elicitation Workshops impacted the results. Participation in either workshop did not appear to
have a significant effect on experts' judgments, based on measures of change in the baseline
distribution. Overall, the sensitivity analyses demonstrated that the assessment was robust, with
64
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
little dependence on individual experts' judgments or on the specific elicitation methods
evaluated.
Management Considerations. The sensitivity analysis demonstrated the robustness of the PM
2.5 expert elicitation-based assessment by showing that the panel of experts was generally well
balanced and that alternative elicitation methods would not have markedly altered the overall
results.
Document Availability. The details of this analysis are provided in the IEC document titled:
"Expanded Expert Judgment Assessment of the Concentration-Response Relationship Between
PM2.5 Exposure and Mortality" Final Report, September 21, 2006
(www.epa.gov/ttn/ecas/regdata/uncertainty/pm ee report.pdf).
The expert elicitation assessment, along with the Regulatory Impact Analysis (RIA) of the PM2 5
standard, is available at http://www.epa.gov/ttn/ecas/ria.html.
Contact. Lisa Conner, 919-541-5060, conner.lisa@epa.gov
65
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 7: Environmental Monitoring and Assessment Program (EMAP): Using
Probabilistic Sampling to Evaluate the Condition of the Nation's Aquatic Resources
Monitoring is a key tool used to identify where the environment is in healthy biological
condition and requires protection, and where environmental problems are occurring and need
remediation. However, most monitoring is not currently done in a way that allows for
statistically-valid assessments of water quality conditions in unmonitored waters (GAO 2000).
States thus cannot adequately measure the status and trends in water quality in their waters as
required by Clean Water Act Section 305(b).
EMAP's focus has been to develop unbiased statistical survey design frameworks, and sensitive
indicators that allow the condition of aquatic ecosystems to be assessed at state, regional, and
national scales. A cornerstone of EMAP has been the use of probabilistic sampling to allow
representative, unbiased, cost-effective condition assessments for aquatic resources over large
areas. EMAP's statistical survey methods are very efficient, requiring relatively few sample
locations to make valid scientific statements about the condition of aquatic resources over large
areas (e.g., the condition of all the wadeable streams in the Western US).
Probabilistic Analysis. This research program had a number of case studies using probabilistic
sampling designs for different aquatic resources (estuaries, streams, and rivers). An EMAP
probability-based sampling program provides an unbiased estimate of the condition of an aquatic
resource over a large geographic area from a small number of samples. The principal
characteristics of a probabilistic sampling design are: the population being sampled is
unambiguously described; every element in the population has the opportunity to be sampled
with a known probability; and sample selection is carried out by a random process. This
approach allows statistical confidence levels to be placed on the estimates and provides the
potential to detect statistically significant changes and trends in condition with repeated
sampling. In addition, this approach permits the aggregation of data collected from smaller areas
to predict the condition of a large geographic area.
The EMAP design framework allows the selection of unbiased, representative sampling sites and
specifies the information to be collected at these sites. The validity of the overall inference rests
on the design and subsequent analysis to produce regionally representative information. The
EMAP uses Generalized Random Tessellation Stratified (GRTS) Spatially-Balanced Survey
Designs for Aquatic Resources. The spatially-balanced aspect spreads out the sampling
locations geographically, but still ensures that each element has an equal chance of being
selected.
Results of the Analysis. Data collected using the EMAP approach has allowed the Agency to
make scientifically defensible assessments of the ecological condition of large geographic areas
for reporting to Congress under CWA 305(b). The EMAP approach has been used to provide the
first reports on the condition of the nation's estuaries, streams, rivers and lakes, and it is
scheduled to used for wetlands. EMAP findings have been included in EPA's Report on the
Environment, and the Heinz Center's The State of the Nation's Ecosystems. Data collected
through an EMAP approach improve the ability to assess ecological progress in environmental
protection and restoration, and provide valuable information for decision-makers and the public.
66
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
The use of probabilistic analysis methods allows meaningful assessment and regional
comparisons of aquatic ecosystem conditions across the United States. Finally, the probabilistic
approach provides scientific credibility for the monitoring network and aids in identifying data
gaps.
Management Considerations. Use of an EMAP approach addresses criticisms from the
General Accounting Office, the National Academies of Sciences (NAS), the Heinz Center (a
nonprofit environmental policy institution), and others who noted the nation lacked the data to
make scientifically valid characterizations of water quality regionally and across the United
States. The program provides cost-effective, scientifically defensible, and representative data, to
permit the evaluation of quantifiable trends in ecosystem condition, to support performance-
based management, and to facilitate better public decisions regarding ecosystem management.
EMAP's approach has now been adopted by the EPA's Office of Water (OW) to collect data on
the condition of all the nation's aquatic resources. OW, Office of Air and Radiation and Office
of Prevention, Pesticides, and Toxic Substances now have environmental accountability
endpoints using EMAP approaches in their Agency performance goals.
Document Availability. Available at http://www.epa.gov/emap/index.html.
U. S. EPA. 2002. Research Strategy. Environmental Monitoring and Assessment Program. U.S.
EPA, Office of Research and Development, National Health and Environmental Effects
Research Laboratory. U.S. EPA, Research Triangle Park, NC. Available at
www.epa.gov/emap/html/pubs/docs/resdocs/emap research strategy.pdf.
Information on EMAP monitoring designs is available at
http://www.epa.gov/nheerl/arm/designpages/monitdesign/monitoring_design_info.htm
Contacts. Michael McDonald, 919-541-7973, mcdonald.michael@epa.gov: Tony Olsen, 541-
754-4790, olsen.tony@epa.gov
67
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Group 3 Case Studies
Case Study 8: Two-Dimensional Probabilistic Risk Analysis of Cryptosporidium in
Public Water Supplies, with Bayesian Approaches to Uncertainty Analysis
Probabilistic assessment of the occurrence and health effects associated with Cryptosporidium
bacteria in public drinking water supplies was used to support the economic analysis of the final
Long-Term 2 Enhanced Surface Water Treatment Rule (LT2). EPA's Office of Ground Water
and Drinking Water (OGWDW) conducted this analysis and established a baseline disease
burden attributable to Cryptosporidium in Public Water supplies that use surface water sources.
Next, it models the source water monitoring and finished water improvements that will be
realized as a result of the Rule. Post-Rule risk is estimated and the Rule's health benefit is the
result of subtracting this from the baseline disease burden.
Probabilistic Risk Analysis. Probabilistic assessment was used for this analysis as a means of
addressing the variability in the occurrence of Cryptosporidium in raw water supplies, the
variability in the treatment efficiency, as well as the uncertainty in these inputs and in the dose-
response relationship for Cryptosporidium infection. This case study provides an example of a
PRA that evaluates both variability and uncertainty at the same time and is referred to as a two-
dimensional probabilistic risk assessment. The analysis also included probabilistic treatments
of uncertain dose-response and occurrence parameters. Markov Chain Monte Carlo samples of
parameter sets filled this function. This Bayesian approach (treating the unknown parameters as
random variables) differs from classical treatments, which would regard the parameters as
unknown, but fixed (Group 3: Advanced PRA). The risk analysis used existing datasets (e.g.,
occurrence of Cryptosporidium and treatment efficacy) to inform the variability of these inputs.
Uncertainty distributions were characterized based on professional judgment or by analyzing
data using Bayesian statistical techniques.
Results of Analysis. The risk analysis identified the Cryptosporidium dose-response relationship
as the most critical model parameters in the assessment, followed by the occurrence of the
pathogen and treatment efficiency. By simulating implementation of the Rule using imprecise,
biased measurement methods, the assessment provided estimates of the number of public water
supply systems that would require corrective action and the nature of the actions likely to be
implemented. This information afforded a realistic measure of the benefits (in reduced disease
burden) expected with the LT2 rule. In response to SAB comments, additional Cryptosporidium
dose-response models were added to more fully reflect uncertainty in this element of the
assessment.
Management Considerations. The rule underwent external peer review, review by EPA's
Science Advisory Board (SAB) and intense review by the Office of Management and Budget
(OMB). Occurrence and dose-response components of the risk analysis model were
communicated to stakeholders during the Rule's Federal Advisory Committee Act (FACA)
process. Due to the rigor of the analysis and the signed FACA "Agreement in Principle", the
OMB review was straight-forward.
68
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Document Availability. The final assessment of occurrence and exposure to Cryptosporidium
was released in December 2005 (available at
http://www.epa.gov/safewater/disinfection/lt2/regulations.html).
Contact. Michael Messner, 202-564-5268, messner.michael@epa.gov
69
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 9: Two-Dimensional Probabilistic Model of Children's Exposure to
Arsenic in Chromated Copper Arsenate (CCA) Pressure-Treated Wood
Probabilistic models were developed in response to EPA's October 2001 Federal Insecticide,
Fungicide, and Rodenticide Act (FIFRA) Scientific Advisory Panel (SAP) recommendations to
use probabilistic modeling to estimate children's exposures to arsenic from chromated copper
arsenate (CCA) treated playsets and home decks.
Probabilistic Risk Analysis^ EPA's Office of Research and Development (ORD), in
collaboration with the Office of Pesticide Programs (OPP) developed and applied a probabilistic
exposure assessment of children's exposure to arsenic and chromium from contact with CCA-
treated wood playsets and decks. This case study provides an example of the use of two-
dimensional (i.e., addressing both variability and uncertainly) probabilistic exposure assessment
(Group 3: Advanced PRA). The two-dimensional assessment employed a modification of the
ORD's SHEDS (Stochastic Human Exposure and Dose Simulation) model to simulate children's
exposure to arsenic and chromium from CCA-treated wood playsets and decks, and surrounding
soil. Staff from both ORD and OPP collaborated in the development of the SHEDS-Wood
model.
Results of Analysis^ A draft of the probabilistic exposure assessment received SAP review in
December, 2003; the final report was released in 2005. The results of the probabilistic exposure
assessment were consistent with or in the range of the results of deterministic exposure
assessments conducted by several other organizations. The model results were used to compare
exposures under a variety of scenarios, including cold vs. warm weather activity patterns, use of
wood sealants to reduce the availability of contaminants on the surface of the wood, and different
hand-washing frequencies. The modeling of alternative mitigation scenarios indicated that the
use of sealants could result in the greatest exposure reduction, while increased frequency of
hand-washing could also reduce exposure.
OPP used the SHEDS-Wood exposure results in their probabilistic children's risk assessment for
CCA (EPA, 2008). This included recommendations for risk reduction (use of sealants and
careful attention to children's hand-washing) to homeowners with existing CCA wood structures.
In addition, the exposure assessment was used to identify areas for further research, including:
the efficacy of wood sealants in reducing dislodgeable contaminant residues, the frequency with
which children play on or around CCA wood, and transfer efficiency and residue concentrations.
In order to better characterize the efficacy of sealants in reducing exposure, EPA and the
Consumer Product Safety Commission conducted a 2-year study of how dislodgeable
contaminant residue levels changed with the use of various types of commercially-available
wood sealants.
Management Considerations. The SHEDS-wood model was one of Agency's first
probabilistic modeling assessments for regulatory purposes. The OPP used SHEDS results
directly in their final risk assessment for children playing on CCA treated playground equipment
and decks. The model enhanced risk assessment and management decisions and prioritized data
needs related to wood preservatives. The modeling product was pivotal in the risk management
and re-registration eligibility decisions for CCA, and in advising the public how to minimize
70
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
health risks from existing treated wood structures. Industry is also using SHEDS to estimate
exposures to CCA and other wood preservatives. Some states are using the risk assessment as
guidance in setting their regulations for CCA related playground equipment.
Document Availability.
The model results were included in the final report on the probabilistic exposure assessment of
CCA-treated wood surfaces:
Zartarian, V.G., J. Xue, H. A. Ozkaynak, W. Dang, G. Glen, L. Smith, and C. Stallings. A
Probabilistic Exposure Assessment for Children Who Contact CCA-treated Playsets and Decks
Using the Stochastic Human Exposure and Dose Simulation Model for the Wood Preservative
Scenario (SHEDS-WOOD), Final Report. U.S. Environmental Protection Agency, Washington,
DC, EPA/600/X-05/009. http://www.epa.gov/heasd/sheds/cca treated.htm
The final probabilistic risk assessment based on the SHEDS-Wood exposure assessment can be
found at: http://www.epa.gov/oppad001/reregistration/cca/final_cca_factsheet.htm
Results of the sealant studies were released in January, 2007 (available at
http://www.epa.gov/oppadQ01/reregistration/cca/index.htmtfreviews).
The results of the analysis were published as:
Zartarian, V.G., Xue, J., Ozkaynak, H., Dang, W., Glen, G., Smith, L., and Stallings, C. A
probabilistic arsenic exposure assessment for children who contact chromated copper arsenate
(CAA)-treated playsets and decks, Part 1: Model methodology, variability results, and model
evaluation. Risk Analysis 26:515, 2006.
Contact. Valerie G. Zartarian, Ph.D., EPA's Office of Research and Development, 617-918-
1541, zartarian.valerie@epa.gov
71
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 10: Two-Dimensional Probabilistic Exposure Assessment of Ozone
As part of EPA's recent review of the ozone National Ambient Air Quality Standards (NAAQS),
the Office of Air Quality Planning and Standards (OAQPS) conducted detailed probabilistic
exposure and risk assessments in evaluating potential alternative standards for ozone. At
different stages of this review, the Clean Air Scientific Advisory Committee (CAS AC) Ozone
Panel (an independent scientific review committee of EPA's SAB) and the public reviewed and
provided comments on analyses and documents prepared by EPA. A scope and methods plan for
the exposure and risk assessments was developed in 2005 (EPA, 2005). This plan was intended
to facilitate consultation with CASAC, as well as public review, and to obtain advice on the
overall scope, approaches, and key issues in advance of the completion of the analyses. This case
study describes the probabilistic exposure assessment, addressing short-term exposures to ozone.
The exposure estimates were used as an input to the health risk assessment for lung function
decrements in all children and asthmatic school-aged children based on exposure-response
relationships derived from controlled human exposure studies.
Probabilistic Exposure Analysis. Population exposure to ambient ozone levels was evaluated
using EPA's APEX model, also referred to as the Total Risk Integrated Methodology/Exposure
(TREVI.Expo) model. Exposure estimates were developed for recent ozone levels, based on 2002
to 2004 air quality data, and for ozone levels simulated to just meet the existing 0.08 ppm, 8-h
ozone NAAQS and several alternative ozone standards, based on adjusting 2002 to 2004 air
quality data. Exposure estimates were modeled for 12 urban areas located throughout the United
States for the general population, all school-age children, and asthmatic school-age children.
This exposure assessment is described in a technical report (EPA, 2007b). The exposure model,
APEX, is documented in a user's guide and technical document (EPA, 2006a,b). A Monte Carlo
approach was used to produce quantitative estimates of the uncertainty in the APEX model
results, based on estimates of the uncertainties for the most important model inputs. The
quantification of model input uncertainties, a discussion of model structure uncertainties, and the
results of the uncertainty analysis are documented in Langstaff (2007).
Results of Analysis. Uncertainty in the APEX model predictions results from uncertainties in the
spatial interpolation of measured concentrations, the microenvironment models and parameters,
people's activity patterns, and, to a lesser extent, model structure. The predominant sources of
uncertainty appear to be the human activity pattern information and the spatial interpolation of
ambient concentrations from monitoring sites to other locations. The primary policy-relevant
finding was that the uncertainty in the exposure assessment is small enough to lend confidence to
the use of the model results for the purpose of informing the Administrator's decision on the
ozone standard.
The following figure illustrates the uncertainty distribution for one model result, the percent of
children with exposures above 0.08 ppm-8hr while at moderate exertion. The "point estimate" of
20 percent is the result from the APEX simulation using the best estimates of the model inputs.
The corresponding result from the Monte Carlo simulations ranges from 17 to 26 percent, with a
95 percent uncertainty interval (UI) of 19 to 24 percent. Note that the uncertainty intervals are
not symmetric since the distributions are skewed.
72
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Uncertainty distribution for the estimated percent of children with any 8-hour exposures
above 0.08 ppm-8hr at moderate exertion (point estimate is 20%)
450
17% 18% 19% 20% 21% 22%
Percent of children
23%
24%
25%
26%
Management Considerations._The exposure analysis also provided information on the
frequency with which population exposures exceeded several potential health effect benchmark
levels that were identified based on evaluation of health effects in clinical studies.
The exposure and risk assessments are summarized in Chapters 4 and 5, respectively, of the
Ozone Staff Paper (EPA, 2007a). The exposure estimates over these potential health effect
benchmarks were part of the basis for the Administrator's March 27, 2008, decision to revise the
ozone NAAQS from 0.08 to 0.075 ppm, 8-h average (see 73 FR 16436).
Document Availability.
_Langstaff, J. E. (2007). Analysis of Uncertainty in Ozone Population Exposure Modeling.
OAQPS Staff Memorandum to Ozone NAAQS Review Docket (OAR-2005-0172). Available at
http://www.epa.gov/ttn/naaqs/standards/ozone/s_ozone_cr_td.html.
EPA (2007a). Review of National Ambient Air Quality Standards for Ozone: Assessment of
Scientific and Technical Information - OAQPS Staff Paper. OAQPS, U.S. EPA, RTF, NC.
Available at http://www.epa.gov/ttn/naaqs/standards/ozone/s_ozone_cr_sp.html.
EPA (2007b). Ozone Population Exposure Analysis for Selected Urban Areas. OAQPS, U.S.
EPA, RTF, NC. Available at http://www.epa.gOv/ttn/naaqs/standards/ozone/s ozone crtd.html.
EPA (2006a,b). Total Risk Integrated Methodology (TRIM) - Air Pollutants Exposure Model
Documentation (TRIM.Expo / APEX, Version 4) Volume I: User's Guide; Volume II: Technical
73
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Support Document. OAQPS, U.S. EPA, RTF, NC. June 2006. Available at
http://www.epa.gov/ttn/fera/human apex.html.
EPA (2005). Ozone Health Assessment Plan: Scope and Methods for Exposure Analysis and
Risk Assessment. OAQPS, U.S. EPA, RTF, NC. Available at
http://www.epa.gOv/ttn/naaqs/standards/ozone/s o3 cr_pd.html.
Contact.John E. Langstaff, EPA's Office of Air and Radiation, 919-541-1449,
Langstaff. John@epa. gov
74
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 11: Analysis of Microenvironmental Exposures to Particulate Matter
(PM2.5) for a Population Living in Philadelphia, PA
This case study used the Stochastic Human Exposure and Dose Simulation model for particulate
matter (SHEDS-PM) developed by EPA's National Exposure Research Laboratory (NERL) to
prepare a probabilistic assessment of population exposure to particulate matter (PM) in
Philadelphia, PA. This case study simulation was prepared to accomplish three goals: 1) to
estimate the contribution of PM of ambient (outdoor) origin to total PM exposure, 2) to
determine the major factors that influence personal exposure to PM, and 3) to identify factors
that contribute the greatest uncertainty to model predictions.
Probabilistic Risk Analysis. The two-dimensional probabilistic assessment used a
microexposure event technique to simulate individual exposures to PM in specific
microenvironments (outdoors, indoor residence, office school, store, restaurant or bar, and in a
vehicle). The assessment used spatially and temporally interpolated ambient PM2.5
measurements from 1992-93 and 1990 U.S. Census data for each census tract in Philadelphia.
This case study provides an example of both two-dimensional (variability and uncertainty)
probabilistic assessment and microexposure event assessment (Group 3: Advanced PRA).
Results of Analysis. Results of the analysis showed that that human activity patterns did not
have as strong an influence on ambient PM2.5 exposures as was observed for exposure to indoor
PM2.5 sources. Exposure to PM2.5 of ambient origin contributed a significant percent of the daily
total PM2.5 exposures, especially for the segment of the population without exposure to
environmental tobacco smoke in the residence. Development of the SHEDS-PM model using the
Philadelphia PM2 5 case study also provided useful insights into data needs for improving inputs
to the SHEDS-PM model, reducing uncertainty and further refinement of the model structure.
Some of the important data needs identified from the application of the model include: daily PM
measurements over multiple seasons and across multiple sites within an urban area, improved
capability of dispersion models to predict ambient PM concentrations at greater spatial resolution
and over a year time period, measurement studies to better characterize the physical factors
governing infiltration of ambient PM2 5 into residential microenvironments, further information
on particle-generating sources within the residence, and data for the other indoor
microenvironments not specified in the model.
Management Considerations: The application of the SHEDS-PM model to the Philadelphia
population gave insights into data needs and areas for model refinement. The continued
development and evaluation of the SHEDS-PM population exposure model are being conducted
as part of EPA/ORD's effort to develop a source-to-dose modeling system for PM and air toxics.
This type of exposure and dose modeling system is considered to be important for scientific and
policy evaluation of the critical pathways, as well as exposure factors and source types
influencing human exposures to a variety of environmental pollutants, including particulate
matter.
75
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Document Availability.
The assessment is available at http://www.epa.gov/heasd/pm/pdf/exposure-model-for-pm.pdf.
The results of the analysis were published as:
Burke, J., Zufall, M., and Ozkaynak, H. A population exposure model for particulate matter:
Case study results for PM2.5 in Philadelphia, PA. Journal of Exposure Analysis and
Environmental Epidemiology 11: 470, 2001.
Georgepoulos, P.G., Wang, S.-W., Vyas, V.M., Sun, Q., Burke, J., Vedantham, R., McCurdy,
T., and Ozkayanak, H. A source-to-dose assessment of poulation exposure to fine PM and ozone
in Philadelphia, PA, during a summer 1999 episode. J. Expos. Analysis and Environm. Epi. 1-
19, 2005.
Contact:. Janet M. Burke, EPA's Office of Research and Development, 919-541-0820,
burke.janet@epa.gov
76
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 12: Probabilistic Analysis in Cumulative Risk Assessment of
Organophosphorus Pesticides
In 1996, Congress enacted the Food Quality Protection Act (FQPA), which requires EPA to
consider "available evidence concerning the cumulative effects on infants and children of such
residues and other substances that have a common mechanism of toxicity" when setting pesticide
tolerances (i.e., the maximum amount of pesticide residue legally allowed to remain on food
products). FQPA also mandated that EPA completely reassess the safety of all existing pesticide
tolerances (those in effect since August 1996) to ensure that they are supported by up-to-date
scientific data and meet current safety standards. Because organophosphorus pesticides (OPs)
were assigned priority for tolerance reassessment, these pesticides were the first "common
mechanism" group identified by EPA's OPP. The ultimate risk management goal associated with
this cumulative risk assessment (CRA) was to establish safe tolerance levels for this group of
pesticides, while meeting the FQPA standard for protecting infants and children.
Probabilistic Risk Analysis. This case study provides an example of an advanced probabilistic
assessment (Group 3). In 2006, EPA analyzed exposures to 30 OPs through food consumption,
consumption of drinking water, and exposure via pesticide application. EPA used Calendex, a
probabilistic computer software program (available at
http://www.exponent.com/calendex software), to integrate various pathways, while
simultaneously incorporating the time dimensions of the input data. Based on the results of the
exposure assessment, EPA calculated margins of exposure (MOEs) for the total cumulative risk
from all pathways.
The food component of the OPs CRA was highly refined, as it was based on residue monitoring
data from the USDA's PDF and supplemented with information from the FDA's Surveillance
Monitoring Programs and Total Diet Study. The residue data were combined with actual
consumption data from USDA's Continuing Survey of Food Intakes by Individuals using
probabilistic techniques. The CRA evaluated drinking water exposures on a regional basis. The
assessment focused on areas where combined OP exposure is likely to be highest within each
region. Primarily, the analysis used probabilistic modeling to estimate the co-occurrence of OP
residues in water. Monitoring data were not available with enough consistency to be the sole
basis for the assessment; however, they were used to corroborate the modeling results. Data
sources for the water component of the assessment included USD A Agricultural Usage Reports
for Field Crops, Fruits, and Vegetables; USDA Typical Planting and Harvesting Dates for Field
Crops and Fresh Market and Processing Vegetables; local sources for refinements; and
monitoring studies from the U.S. Geological Survey and other sources. Finally, exposure via the
oral, dermal, and inhalation routes resulting from applications of OPs in and around homes,
schools, offices, and other public areas were assessed probabilistically for each of the seven
regions. The data sources for this part of the assessment included information from surveys and
task forces, special studies and reports from published scientific literature, EPA's Exposure
Factors Handbook (USEPA, 1997), and other sources.
Results of Analysis. The OPs CRA presented potential risk from single-day (acute) exposures
across a year and from a series of 21-day rolling averages across the year. MOEs at the 99.9th
percentile were approximately 100 or greater for all populations for the 21-day average results.
77
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
The only exception is a brief period (roughly 2 weeks) in which drinking water exposures
resulting from phorate use on sugarcane result in MOEs near 80 for children aged 1 to 2 years.
Generally, exposures through the food pathway dominated total MOEs, and exposures through
drinking water were substantially lower throughout most of the year. Residential exposures were
substantially smaller than exposures through both food and drinking water.
The OPs CRA was very resource intensive. Work began in 1997 with the preparation of
guidance documents and the development of a CRA methodology. Over 2 to 3 years, more than
25 people spent 50 to 100% of their time working on the assessment, with up to 50 people
working on the CRA at critical periods. EPA has spent approximately $1 million on this
assessment (e.g., for computers, models, and contractor support).
Management Considerations. The OP CRA was a landmark demonstration of the feasibility of
a regulatory level assessment of the risk from multiple chemicals. On its completion, EPA
presented the CRA at numerous public technical briefings and FIFRA SAP meetings, and made
all of the data inputs available to the public. OPP's substantial effort to communicate
methodologies, approaches, and results to the stakeholders aided in the success of the OPs CRA.
The stakeholders expressed appreciation for the transparent nature of the OPs CRA and
recognized the innovation and hard work that went into developing the assessments.
Document Availability. The 2006 assessment and related documents are available at
http://www.epa.gov/pesticides/cumulative/common mech groups.htm#op.
Contact. David Miller, 703-305-5352; miller.davidj@epa.gov .
78
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 13: Probabilistic Ecological Effects Risk Assessment Models for
Evaluating Pesticide Uses
As part of the process of developing and implementing a probabilistic approach for ecological
risk assessment, an illustrative case was completed in 1996. The illustrative case involved both
deterministic and probabilistic risk analysis for effects of a hypothetical chemical X on birds and
aquatic species. Following the recommendations of the SAP and in response to issues raised by
OPP risk managers, the Agency began an initiative to refine the ecological risk assessment
process for evaluating the effects of pesticides to terrestrial and aquatic species within the
context of FIFRA, the main statutory authority for regulating pesticides at the Federal level.
Among the key goals and objectives of EPA's initiative were to:
• incorporate probabilistic tools and methods to provide an estimate on the magnitude and
probability of effects;
• build on existing data requirements for registration;
• utilize, wherever possible, existing databases and create new ones from existing data
sources to minimize the need to generate additional data; and
• focus additional data requirements on reducing uncertainty in key areas.
After proposing a four-level risk assessment scheme, with higher levels reflecting more refined
risk assessment techniques, the Agency developed pilot models for both terrestrial and aquatic
species. Refined risk assessment models (Level II) were then developed and used in a generic
chemical case study that was presented to the SAP in 2001.
Probabilistic Analysis. This case study describes an advanced probabilistic model for ecological
effects of pesticides (Group 3). The terrestrial Level II model (version 2.0) is a multimedia
exposure/effects model that evaluates acute mortality levels in generic or specific avian species
over a user-defined exposure window. The spatial scale is at the field level, which includes the
field and surrounding area. Both areas are assumed to meet the habitat requirements for each
species, and contamination of edge or adjacent habitat from drift is assumed to be zero. For each
individual bird considered in a run of the Level II model, a random selection of values is made
for the major exposure input parameters to estimate an external oral dose equivalent for that
individual. The estimated dose equivalent is compared to a randomly assigned tolerance for the
individual preselected from the dose/response distribution. If the dose is greater than the
tolerance, the individual is scored "dead," if not, the individual is scored "not dead." After
multiple iterations of individuals, a probability density function of percent mortality is generated.
May 29-31, 1996, the Agency presented two ecological risk assessment case studies to SAP for
review and comment. Although recognizing and generally reaffirming the utility of EPA's
current deterministic assessment process, SAP offered a number of suggestions for improvement.
Foremost among their suggestions was a recommendation to move beyond the existing
deterministic assessment approach by developing the tools and methodologies necessary to
conduct a probabilistic assessment of effects. Such an assessment would estimate the magnitude
and probability of the expected impact and define the level of certainty and variation involved in
the estimate, information that risk managers within EPA's OPP also had requested in the past.
79
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
The aquatic Level II model is a two-dimensional Monte Carlo risk model consisting of three
main components: (1) exposure, (2) effects, and (3) risk. The exposure scenarios used at Level II
are intended to provide estimates for vulnerable aquatic habitats across a wide range of
geographical conditions under which a pesticide product is used. The Level II risk evaluation
process yields estimates of likelihood and magnitude of effects for acute exposures. For the
estimate of acute risks, a distribution of estimated exposure and a distribution of lethal effects are
combined through a two-dimensional Monte Carlo analysis to obtain a distribution of joint
probability functions. For the estimate of chronic risks, a distribution of exposure concentrations
is compared to a chronic measurement endpoint. The risk analysis for chronic effects provides
information only on the probability that the chronic end point assessed will be exceeded, not on
the magnitude of the chronic effect expected.
Results of Analysis. As part of the process of developing and implementing a probabilistic
approach for ecological risk assessment, a case study was completed. The case study involved
both deterministic and probabilistic risk analyses for effects of ChemX on birds and aquatic
species. The deterministic screen conducted on ChemX concluded qualitatively that it could pose
a high risk to both freshwater fish and invertebrates and showed that PRA was warranted. Based
on the probabilistic analysis, it was concluded that the use of ChemX was expected to
infrequently result in significant freshwater fish mortalities but routinely result in reduced growth
and other chronic effects in exposed fish. Substantial mortalities and chronic effects to sensitive
aquatic invertebrates were predicted to routinely occur after peak exposures.
Management Considerations. In its review of the case study, the FIFRA SAP congratulated
the Agency on the effort made to conduct PRA of pesticide effects in ecosystems. The panel
commented that the approach had progressed greatly from earlier efforts, and that the intricacy of
the models was surprisingly good, given the time interval in which the Agency had to complete
the task. Following the case study, the Agency refined the models based on the SAP comments.
In addition, the terrestrial Level II model was refined to include dermal and inhalation exposure.
Document Availability. An overview of the models is available at
http://www.epa.gov/oppefedl/ecorisk/fifrasap/rra_exec_sum.htm#Terrestrial.
Contact: Donna Randall 703-605-1298, randall.donna@epa.gov
80
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 14: Expert Elicitation of Concentration-Response Relationship Between
Particulate Matter (PM2.s) Exposure and Mortality
In 2002, the NRC recommended that EPA improve its characterization of uncertainty in the
benefits assessment for proposed regulations of air pollutants. NRC recommended that
probability distributions for key sources of uncertainty be developed using available empirical
data or through formal elicitation of expert judgments. A key component of EPA's approach for
assessing potential health benefits associated with air quality regulations targeting emissions of
PM2.5 and its precursors is the effect of changes in ambient PM2.5 levels on mortality. Avoided
premature deaths constitute, on a dollars basis, between 85 and 95% of the monetized benefits
reported in EPA's retrospective and prospective Section 812A benefit-cost analyses of the Clean
Air Act (EPA, 1997 and 1999) and in Regulatory Impact Analysis (RIAs) for rules such as the
Heavy Duty Diesel Engine/Fuel Rule (EPA, 2000) and the Non-road Diesel Engine Rule (EPA,
2004). In response to the National Research Council (NRC) recommendation, EPA conducted an
expert elicitation evaluation of the concentration-response relationship between PM2 5 exposure
and mortality.
Probabilistic Risk Analysis. This case study provides an example of the use of expert elicitation
(Group 3) to derive probabilistic estimates of the uncertainty in one element of a cost-benefit
analysis. Expert elicitation uses carefully structured interviews to elicit from each expert a best
estimate of the true value for an outcome or variable of interest, as well as their uncertainty about
the true value. This uncertainty is expressed as a subjective probabilistic distribution of values
and reflects each expert's interpretation of theory and empirical evidence from relevant
disciplines, as well as their beliefs about what is known and not known about the subject of the
study. For the PM2.5 expert elicitation, the process consisted of development of an elicitation
protocol, selection of experts, development of a briefing book, conducting elicitation interviews,
the use of expert workshops prior to and following individual elicitation of judgments, and the
expert judgments themselves. The elicitation involved personal interviews with 12 health experts
who have conducted research on the relationship between PM2.5 exposures and mortality.
The main quantitative question asked each expert to provide a probabilistic distribution for the
average expected decrease in U.S. annual, adult, and all-cause mortality associated with a 1-
ug/m3 decrease in annual average PM2.5 levels. When answering the main quantitative question,
each expert was instructed to consider that the total mortality effect of a 1-ug/m3 decrease in
ambient annual average PM2 5 may reflect reductions in both short-term peak and long-term
average exposures to PM2 5. Each expert was asked to aggregate the effects of both types of
changes in their answers.
The experts were given the option to integrate their judgments about the likelihood of a causal
relationship or threshold in the concentration-response function into their own distributions or to
provide a distribution "conditional on" one or both of these factors.
Results of Analysis. The project team developed the interview protocol between October 2004
and January 2006. Development of the protocol was informed by an April 2005 symposium held
by the project team, where numerous health scientists and analysts provided feedback; by
81
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
detailed pretesting with independent EPA scientists in November 2005; and by discussion with
the participating experts at a pre-elicitation workshop in January 2006. The elicitation interviews
were conducted between January and April 2006. Following the interviews, the experts
reconvened for a post-elicitation workshop in June 2006, in which the project team anonymously
shared the results of all experts with the group.
The median estimates for the PM2.5 mortality relationship were generally similar to estimates
derived from the two epidemiological studies most often used in benefits assessment. However,
in almost all cases, the spread of the uncertainty distributions elicited from the experts exceeded
the statistical uncertainty bounds reported by the most influential epidemiologic studies,
suggesting that the expert elicitation process was successful in developing more comprehensive
estimates of uncertainty for the PM2.5 mortality relationship. The uncertainty distributions for
PM2.5 concentration-response resulting from the expert elicitation process were used in the RIA
for the revised NAAQS standard for PM2 5 (promulgated on September 21, 2006). Because the
NAAQS are exclusively health based standards, this RIA played no part in EPA's determination
to revise the Pm2.5 NAAQS. Benefits estimates in the RIA were presented as ranges and
included additional information on the quantified uncertainty distributions surrounding the points
on the ranges, derived from both epidemiological studies and the expert elicitation results.
OMB's review of the RIA was completed in March 2007.
Management Considerations. The NAAQS are exclusively health-based standards, so these
analyses were not used in any manner by EPA in determining whether to revise the NAAQS for
PM2.5. Moreover, the request to engage in the expert elicitation did not come from the Clean
Air Scientific Advisory Committee, or CASAC, the official peer review body for the NAAQS,
so that a decision to conduct the analyses does not reflect CASAC advice that such analyses
inform NAAQS determinations. The analyses addressed comments from the National Research
Council that recommended that probability distributions for key sources of uncertainty be
addressed. The analyses were used in EPA's retrospective and prospective Section 812A benefit-
cost analyses of the Clean Air Act (EPA, 1997 and 1999) and in RIAs for rules such as the
Heavy Duty Diesel Engine/Fuel Rule (EPA, 2000) and the Non-road Diesel Engine Rule (EPA,
2004). In response to the NRC recommendation, EPA conducted an expert elicitation evaluation
of the concentration-response relationship between PM2.5 exposure and mortality.
Document Availability. The assessment is available at http://www.epa.gov/ttn/ecas/ria.html.
Contact:_Lisa Conner, 919-541-5060, conner.lisa@epa.gov
82
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 15: Expert Elicitation of Sea-Level Rise Resulting from Global Climate
Change
The United Nations Framework Convention on Climate Change requires nations to implement
measures for adapting to rising sea level and other effects of changing climate. To decide on an
appropriate response, coastal planners and engineers weigh the cost of these measures against the
likely cost of failing to prepare, which depends on the probability of the sea rising a particular
amount. The U.S. National Academy of Engineering recommended that assessments of sea level
rise should provide probability estimates. Coastal engineers regularly employ probability
information when designing structures for floods, and courts use probabilities to determine the
value of land expropriated by regulations. This case study describes the development of a
probability distribution for sea level rise, using models employed by previous assessments, as
well as the expert opinions of 20 climate and glaciology reviewers about the probability
distributions for particular model coefficients.
Probabilistic Analysis. This case study provides an example both of an analysis describing the
probability of sea level rise, as well as an expert elicitation of the likelihood of particular models
and probability distributions of the coefficients used by those models to predict future sea level
rise (Group 3). The assessment of the probability of sea level rise used existing models
describing the change in five components of sea level rise associated with greenhouse-gas-
related climate change (thermal expansion, small glaciers, polar precipitation, melting and ice
discharge from Greenland, ice discharge from Antarctica). To provide a starting point for the
expert elicitation, initial probability distributions were assigned to each model coefficient based
on the published literature.
Once the initial probabilistic assessment was completed, the draft report was circulated to expert
reviewers considered most qualified to render judgments on particular processes for revised
estimates of the likelihood of particular models and the model coefficients' probability
distributions. Experts representing both extremes of climate change science (those who predicted
trivial consequences and those who predicted catastrophic effects; those whose thinking had been
excluded from previous assessments) were included. The experts were asked to provide
subjective assessments of the probabilities of various models and model coefficients. These
subjective probability estimates were used in place of the initial probabilities in the final model
simulations. Different reviewer opinions were not combined to produce a single probability
distribution for each parameter, but, rather, each reviewer's opinions were used in independent
iterations of the simulation. The group of simulations resulted in the probability distribution of
sea level rise.
Results of Analysis. The analysis, completed with a budget of $100,000, provided a
probabilistic estimate of sea level rise for use by coastal engineers and regulators. The results
suggested that there is a 65% chance that sea level will rise 1 mm/year more rapidly in the next
30 years than it has been rising in the last century. Under the assumption that nonclimatic factors
do not change, the projections suggested that there is a 50% chance that global sea level will rise
45 cm, and a 1% chance of a 112-cm rise by the year 2100. The median prediction of sea level
rise was similar to the midpoint estimate of 48 cm published by the Intergovernmental Panel on
83
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Climate Change (TPCC, 2006) shortly thereafter.. The report also found a 1% chance of a 4-5
meter rise over the next two centuries.
Management Considerations: Both reports (EPA 1995; Titus and Narayanan 1996) discuss
several uses of the results of this study. By providing a probabilistic representation of sea level
rise, the assessment allows coastal residents to make decisions with recognition of the
uncertainty associated with predicted change. Rolling easements that vest when the sea rises to
a particular level can be properly valued in both arms-length transaction sales or when
calculating the allowable deduction for a charitable contribution of the easement to a
conservancy. Cost-benefit assessments of alternative infrastructure designs—which already
consider flood probabilities—can also consider sea level rise uncertainty. Assessments of the
benefits of preventing an acceleration of sea level rise can more readily include low-probability
outcomes, which can provide a better assessment of the true risk when the damage function is
nonlinear, which is often the case.
Document Availability.
EPA 1995. The Probability of Sea Level Rise. Washington, D.C.: Climate Change Division.
http://epa.gov/climatechange/effects/coastal/slrmaps_probability.html
IPCC (1996). Climate Change 1995: The Science of Climate Change. Contribution of Working
Group I to the Second Assessment of the Intergovernmental Panel on Climate Change.
Cambridge University Press. Cambridge CB2 2RU ENGLAND.
Titus, J. G. and V Narayanan. 1996. The Risk of Sea Level Rise: A Delphic Monte Carlo
Analysis in which Twenty Researchers Specify Subjective Probability Distributions for Model
Coefficients within their Respective Areas of Expertise - Climatic Change, 33: 151-212
(1996). http://epa.gov/climatechange/effects/coastal/Risk_of_nse.html
Contact: James G. Titus, 202-343-9307; titus.jim@epa.gov
84
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
Case Study 16: Expert Elicitation for Bayesian Belief Network Model of Stream
Ecology
The identification of the causal pathways leading to stream impairment is a central challenge to
understanding ecological relationships. Bayesian belief networks (BBNs) are a promising tool
for modeling presumed causal relationships, providing a modeling structure within which
different factors describing the ecosystem can be causally linked, and uncertainties expressed for
each linkage.
BBNs can be used to model complex systems that involve several interdependent or interrelated
variables. In general, a BBN is a model that evaluates situations where some information is
already known, and incoming data are uncertain or partially unavailable. The information is
depicted with influence diagrams that present a simple visual representation of a decision
problem, for which quantitative estimates of effect probabilities are assigned. As such, BBNs
have the potential for representing ecological knowledge and uncertainty in a format that is
useful for predicting outcomes from management actions or for diagnosing the causes of
observed conditions. Generally, specification of a BBN can be performed using available
experimental data, literature review information (secondary data), and expert elicitation.
Attempts to specify a BBN for the linkage between fine sediment load and macroinvertebrate
populations using data from literature reviews have failed because of the absence of consistent
conceptual models and lack of quantitative data or summary statistics needed for the model. In
light of these deficiencies, an effort was made to use expert elicitation to specify a BBN for the
relationship between fine sediment load resulting from human activity and populations of
macroinvertebrates. The goals of this effort were to examine whether BBNs might be useful for
modeling stream impairment and to assess whether expert opinion could be elicited to make the
BBN approach useful as a management tool.
Probabilistic Risk Analysis. This case study provides an example of expert elicitation in the
development of a BBN model of the effect of increased fine sediment load in a stream on
macroinvertebrate populations (Group 3). For the purpose of this study, a stream setting (a
Midwestern, low-gradient stream) and a scenario of impairment (introduction of excess fine
sediment) were used. Five stream ecologists with experience in the specified geographic setting
were invited to participate in an elicitation workshop. An initial model was depicted using
influence diagrams, with the goals of structuring and specifying the model using expert
elicitation. The ecologists were guided through a knowledge elicitation session in which they
defined factors that described relevant chemical, physical, and biological aspects of the
ecosystem. The ecologists then described how these factors were connected and were asked to
provide subjective, quantitative estimates of how different attributes of the macroinvertebrate
assemblage would change in response to increased levels of fine sediment. Elicited input was
used to restructure the model diagram and to develop probabilistic estimates of the relationships
among the variables.
Results of Analysis. The elicited input was compiled and presented in terms of the model as
structured by the stream ecologists and their model specifications. The results were presented
both as revised influence diagrams and with Bayesian probabilistic terms representing the
elicited input. The study yielded several important lessons. Among these were the elicitation
85
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
process takes time (including an initial session by teleconference as well as a 3-day workshop),
defining a scenario with an appropriate degree of detail is critical, and elicitation of complex
ecological relationships is feasible.
Management Considerations. The study was considered successful for several reasons. First,
the feasibility of the elicitation approach to building stream ecosystem models was demonstrated.
The study also resulted in the development of new graphical techniques to perform the
elicitation. The elicited input was interpreted in terms of predictive distributions to support fitting
a complete Bayesian model. Further, the study was successful in bringing together a group of
experts in a particular subject area for the purpose of sharing information and learning about
expert elicitation in support of model building. The exercise provided insights into how best to
adapt knowledge elicitation methods to ecological questions and informed the assembled stream
ecologists on the elicitation process and on the potential benefits of this modeling approach. The
explicit quantification of uncertainty in the model not only enhances the utility of the model
predictions but also can help guide future research.
Document Availability. Black, P and Stockton, T. 2005. Using Knowledge Elicitation to Inform
a Bayesian Belief Network Model of a Stream Ecosystem. Neptune and Company, Inc. July.
Yuan, L. TI: A Bayesian Approach for Combining Data Sets to Improve Estimates of Taxon
Optima, AGU, 86(18), Jt. Assem. Suppl, Abstract # NB41E-04, 2005.
Contact: Lester Yuan, 703-347-8534, yuan.lester@epa.gov
86
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
4. References to Case Studies
Jamieson, D. (1996). Scientific uncertainty and the political process. Ann. Am. Acad. Pol. Soc. Sci., 545:35-43.
Kurowicka, D. and Cooke, R. (2006). "Uncertainty Analysis With High Dimensional Dependent Modeling." Wiley
Series in Probability and Statistics, May, 2006.
Stahl, C.H., Cimorelli, AJ. (2005). How much uncertainty is too much and how do we know? A case example of the
assessment of ozone monitor network options. Risk Anal, 25:1109-1120.
U. S. Environmental Protection Agency (EPA). (1992a). Guidelines for exposure assessment. EPA 600Z-92/001.
Risk Assessment Forum, Washington, DC, 170 pp.
(http://cfpub.epa.gov/ncea/raf/recordisplay.cfm? deid=559070).
U. S. Environmental Protection Agency (EPA). (1992b). Framework for ecological risk assessment. EPA/630/R-
92/00 I.Washington, DC.
U. S. Environmental Protection Agency (EPA). (1995a). Policy for risk characterization. Science Policy Council,
Washington, DC (http://www.epa.gov/osp/spc/rcpolicv.htm).
U. S. Environmental Protection Agency (EPA). (1995b). Policy on evaluating health risks to children. Science
Policy Council, Washington, DC (http://www.epa.gov/osp/spc/memohlth.htm).
U. S. Environmental Protection Agency (EPA). (1997a). Policy for use of probabilistic analysis in risk assessment at
the U.S. Environmental Protection Agency. Fred Hansen, Deputy Administrator. Science Policy Council,
Washington, DC (http://www.epa.gov/osp/spc/probpol.htm).
U. S. Environmental Protection Agency (EPA). (1997b). Guiding principles for Monte Carlo analysis. EPA/630/R-
97/001. Risk Assessment Forum, Office of Research and Development, Washington, DC.
U. S. Environmental Protection Agency (EPA). (1997c). Guidance on cumulative risk assessment. Part 1: Planning
and scoping. Science Policy Council, Washington, DC (http://www.epa.gpv/osp/spc/cumrisk2.htm).
U. S. Environmental Protection Agency (EPA). (1998). Guidelines for ecological risk assessment. EPA/630/R-
95/002F. Risk Assessment Forum, Washington, DC, 171 pp.
(http ://cfpub. epa. gov/ncea/cfm/recordisplav .cfm?deid= 12460).
U. S. Environmental Protection Agency (EPA). (2000). Science Policy Council Handbook: Risk Characterization
Handbook. EPA 100-BOO-002. Science Policy Council, Washington, DC, December
(http://www.epa.gov/osp/spc/rchandbk.pdf).
U. S. Environmental Protection Agency (EPA). (2001). Risk assessment guidance for Superfund: Volume III—Part
A, Process for conducting probabilistic risk assessment. EPA 540-R-02-002. Office of Emergency and
Remedial Response, Washington, DC, December
(http://www.epa.gov/superfund/programs/risk/rags3a/index.htm).
U. S. Environmental Protection Agency (EPA). (2004). An Examination of EPA Risk Assessment Principles and
Practices. U.S. EPA, Office of the Science Advisor, Office of Research and Development, Washington,
DC. EPA/100/B-04/001 March 2004 (http://www.epa.gov/OSA/ratf.htm).
87
-------
EXTERNAL REVIEW DRAFT - DO NOT CITE OR QUOTE
List of Acronyms and Abbreviations
APEX Air Pollutants Exposure Model
BBN Bayesian belief network
CASAC Clean Air Scientific Advisory Committee
CCA chromated copper arsenate
CRA cumulative risk assessment
CSFII Continuing Survey of Food Intake by Individuals
DEEM Dietary Exposure Evaluation Model
ORES Dietary Risk Evaluation System
Eco ecological risk assessment
EMAP Environmental Monitoring and Assessment Program
EPA U.S. Environmental Protection Agency
FACA Federal Advisory Committee Act
FDA Food and Drug Administration
FIFRA Federal Insecticide, Fungicide, and Rodenticide Act
FQPA Food Quality Protection Act
HH human health
LT long-term
MOEs margins of exposure
NAAQS National Ambient Air Quality Standards
NRC National Research Council
OAQPS Office of Air Quality Planning and Standards
OAR Office of Air and Radiation
OGWDW Office of Groundwater and Drinking Water
OMB Office of Management and Budget
OP organophosphorous pesticide
ORD Office of Risk Analysis
OW Office of Water
PCB polychlorinated biphenyl
PDF Pesticide Data Program
PM particulate matter
PRA probabilistic risk analysis
RIA Regulatory Impact Analysis
SAB Science Advisory Board
SAP Scientific Advisory Panel
SHEDS Stochastic Human Exposure and Dose Simulation
TRIM Total Risk Integrated Methodology
TRIM.Expo Total Risk Integrated Methology/Exposure Model
UI Uncertainty Interval
USDA U.S. Department of Agriculture
88
------- |