Summary Report of the Meeting to Peer Review
EPA's Draft Report: "Using Probabilistic
Methods to Enhance the Role of Risk Analysis in
Decision-Making with Case Study Examples"
Arlington, VA
May 6, 2010
Submitted to:
Risk Assessment Forum
Office of the Science Advisor
U.S. Environmental Protection Agency
Washington, DC
Submitted by:
Eastern Research Group, Inc.
110 Hartwell Avenue
Lexington, MA 02421
FINAL REPORT
July 12, 2010
Notice
This report was prepared by Eastern Research Group, Inc. (ERG), a U.S. Environmental Protection
Agency (EPA) contractor, as a general record of discussion during the Peer Review Meeting on:
EPA's Draft Report, Using Probabilistic Methods to Enhance the Role of Risk Analysis in
Decision-Making with Case Study Examples, held May 6, 2010, in Arlington, Virginia. This report
captures the main points and highlights of the meeting. It is not a complete record of all details
-------
discussed, nor does it embellish, interpret, or enlarge upon matters that were incomplete or unclear.
Statements represent the individual views of meeting participants.
Contents
1 INTRODUCTION 1
2. OPENING REMARKS
3
2.1 EPA Remarks 3
2.2 Public Comments 6
2.3 Reviewer Discussions 6
2 REVIEWER DISCUSSION: GENERAL COMMENTS 7
3 REVIEWER DISCUSSION: QUESTION 1 9
4 REVIEWER DISCUSSION: QUESTION 2 14
5 REVIEWER DISCUSSION: QUESTION 3 15
6 REVIEWER DISCUSSION: QUESTION 4 18
7 REVIEWER DISCUSSION: FINAL COMMENTS AND RECOMMENDATIONS
21 Appendix A: Peer Reviewers Appendix B: Charge to Peer Reviewers Appendix C:
Reviewer Pre-Meeting Comments Appendix D: Observers Appendix E: Meeting Agenda
Appendix F: Slides of EPA Opening Presentation
1. Introduction
The U.S. Environmental Protection Agency (EPA) developed the guidance, Using Probabilistic
Methods to Enhance the Role of Risk Analysis in Decision-Making with Case Study Examples and
Using Probabilistic Methods to Enhance the Role of Risk Analysis in Decision-Making: Managers
Summary (hereafter referred to as "PRA Papers") in order to catalyze the use of probabilistic methods
within the Agency. Divided into the main white paper, case studies provided as Appendix D to the
white paper, and the managers' summary, these documents provide an introduction to the use of
probabilistic methods in the context of Agency decision-making.
Although probabilistic risk assessment (PRA) is widely accepted by the scientific community, it has
not been widely applied within the Agency, despite earlier guidance published in 1998, Risk
Assessment Guidance for Superfund. Discussing benefits, costs, and technical considerations of
performing PRA, the PRA Papers are one tool EPA is developing to better communicate the value of
probabilistic methods to managers and risk assessors.
-------
In early 2010, Eastern Research Group, Inc. (ERG), an EPA contractor, organized an independent
peer review of the External Review Draft of the PRA Papers to assess their scientific quality and
utility. Five nationally recognized experts (Appendix A) conducted this review:
Scott Ferson (Chair), Applied Biomathematics
Annette Guiseppi-Elie, DuPont Engineering
Dale Hattis, Clark University
Igor Linkov, U.S. Army Engineer Research and Development Center
John Toll, Windward Environmental LLC
ERG provided the reviewers with the two review documents and a charge (Appendix B), which
asked for their comments on various aspects of the PRA Papers. Each reviewer also received copies
of all written comments submitted during the public comment period, which they were asked to
consider.
In the first stage of the review, the experts worked individually to prepare written pre-meeting
comments (Appendix C), which were provided to all reviewers and EPA prior to the meeting. In the
second stage, ERG convened a one-day peer review meeting on May 6, 2010, in Arlington, Virginia.
The meeting was attended by 20 observers (Appendix D), 2 of whom attended by teleconference.
Observers included EPA staff and members of the public. Appendix E provides the meeting agenda.
This report summarizes the meeting proceedings. Section 2 presents the opening remarks, and
Sections 3 through 8 summarize the reviewers' discussions organized by charge question. The
appendices provide the following materials: list of reviewers (Appendix A), charge to peer
reviewers (Appendix B), reviewer pre-meeting comments (Appendix C), list of observers
(Appendix D), meeting agenda (Appendix E), and slides of EPA's opening presentation at the peer
review workshop (Appendix F).
2. Opening Remarks
Jan Connery (ERG), the meeting facilitator, opened the meeting by welcoming the reviewers and
observers, who included EPA document authors, other EPA staff, and interested members of the
public. Connery asked the reviewers and EPA document authors to introduce themselves. Peer
reviewers provided brief background information about their relevant areas of expertise and affirmed
that they had no conflict of interest in reviewing the PRA Papers.
Connery reviewed the meeting agenda (Appendix E). She noted that the pre-meeting comments
(Appendix C) were developed by reviewers working individually prior to the meeting and were
considered preliminary. Connery clarified that all meeting discussions would be conducted by the
peer reviewers. Reviewers could request, and observers could offer, clarifications where necessary
and relevant; however, observers should not join the discussions.
Connery then introduced Mary Greene, Deputy Director at EPA's Office of the Science Advisor
(OSA).
2.1 EPA Remarks
-------
Greene welcomed the reviewers, and thanked them for participating as reviewers. She reflected on
the need for EPA to more comprehensively characterize risk, in part by better incorporating
probabilistic methods into Agency activities.
The four co-chairs of the Risk Assessment Forum's (RAF's) Technical Panel on Probabilistic Risk
Assessment—Kathryn Gallagher, Executive Director of the RAF; Haluk Ozkaynak, National
Exposure Research Laboratory; Marian Olsen, Region 2; and Bob Hetes, National Health and
Environmental Effects Research Laboratory—presented, respectively, a brief history of the
documents' development and overviews of the main white paper, case studies, and managers'
summary. Slides of that presentation are provided in Appendix F.
The RAF formed the Probabilistic Risk Assessment Technical Panel, comprised of members from
across the Agency. The primary goal of the PRA Papers is to improve communication of PRA
approaches in order to enhance their application across the Agency. The papers will identify the role
of PRA in decision-making, outline the strengths and limitations of the PRA approaches, and identify
needs and barriers to implementation. The PRA Papers have already undergone internal review by the
PRA Technical Panel, by other RAF members not on the Panel, and by EPA's Science Policy Council
Steering Committee. The development of these PRA Papers was catalyzed by uneven implementation
of PRA across the Agency and the need to inform risk assessors and managers about the uses of PRA.
The main white paper, intended primarily for risk assessors, provides a basic understanding of PRA
methods and what they can contribute to EPA decision-making. The paper is not an exhaustive guide
to PRA; rather, it gives an overview of basic information and directs readers to other, more detailed
resources. It was written at level intended to be accessible to risk assessors to communicate
strengths, weaknesses, and opportunities of PRA. The underlying principle of the document is a
tiered approach with methods ranging from simple to complex. However, in light of the diverse
nature of EPA's work, there is no unified strategy on how to implement PRA.
There are three key reasons to apply probabilistic methods: 1) they explicitly quantify variability and
uncertainty at each step of the risk assessment (RA) process; 2) they can be used to guide data
collection to reduce uncertainty; and 3) they enhance the decision-making process by quantifying
uncertainties among alternatives. A force field analysis (see slide 16 in Appendix F) illustrates tools
and actions that EPA can implement to overcome challenges limiting current use of PRA and the role
of the white paper within that context.
The case study examples provide a picture of how PRA activities have developed since the original
policy was issued in 1995. Examples are drawn from EPA Regional and Program Office activities
with both ecological and human health issues in mind. These case studies are important for risk
managers and assessors to see how the Agency has applied PRA in the past. Each case study
describes the study, type of risk analysis, results, and management considerations. Case studies were
selected from EPA analyses with publicly available literature and an available contact (so that risk
managers have the opportunity to ask questions). Case studies were divided into three groups based
on level of complexity, type of analysis, and resource demands.
The managers' summary was written to give EPA managers and decision-makers basic information
on PRA and how it can contribute to decision-making. The summary is written from a manager's
perspective and limits technical discussion of probabilistic methods. All EPA decisions are made in
the presence of uncertainty: PRA can help by characterizing uncertainties of alternative decisions,
-------
identifying critical parameters and tipping points, and overall improving confidence in decisions.
PRA also facilitates risk management decisions across a variable population where protecting
sensitive subgroups is important. The managers' summary concludes by discussing practical
considerations associated with pursuing PRA, including resource needs (time, funding, and expertise),
data availability, and communicating results.
EPA will revise the draft documents considering the peer review and public comments, and plans to
publish a final version sometime in the fall of 2010. Training and briefing will accompany document
distribution to EPA managers and staff.
At the conclusion of EPA's presentation, the peer reviewers asked several questions:
One reviewer asked whether risk managers who were not familiar or comfortable with
PRA had been interviewed during the document development process. EPA replied that limited
interviews were conducted both with those who have and have not performed PRA, but interviews
were not exhaustive or Agency-wide. The reviewer followed up by asking if risk assessors had also
been interviewed and whether they had shown resistance to using the technique. EPA noted that
preceding the PRA Technical Panel's effort, some PRA training had been provided to risk assessors
by various EPA Program Offices and Regions, and it did not seem that assessors were actively
resistant to using PRA; rather, the assessors give managers what they want under practical constraints
(time, money). EPA added that it is important to establish a dialogue between risk managers and risk
assessors and to consider
when PRA methods are appropriate for a given situation (e.g., PRA is too slow for
emergency response situations).
One reviewer asked EPA for what types of decisions they foresaw risk managers using PRA.
EPA responded that it could be highly variable, ranging from Regional risk managers at Superfund
sites to managers at Program Offices dealing with regulatory decisions on a nation-wide basis. EPA
noted that problem formulation could be better addressed.
One reviewer asked who identified the interviewees. EPA replied that representatives of the
Regions and Program Offices on the PRA Technical Panel organized participants from their location.
Interviewees represent a mix of interest levels and experience with conducting PRA despite being a
small sample.
One reviewer complimented the Risk Assessment Guidance for Superfund (RAGS) and
questioned how these PRA Papers differ. EPA responded that RAGS was targeted for Superfund, but
the PRA Papers are written for more diverse situations and capture a broader perspective on PRA. In
addition, the PRA Papers include dialogue about decision-making in general, update the science in
RAGS, and take a fresh look at the issue.
One reviewer questioned why PRA has not gained traction in the Agency in general despite
some early adopters. EPA suggested that risk managers and assessors (and their audience) are
comfortable using deterministic models and interpreting their associated results. They can also
perform deterministic assessments in a timely manner. The risk manager does see added-value to
performing PRA.
One reviewer asked how the PRA Papers would be advertised and distributed within the
Agency. EPA anticipates having seminars addressing "PRA 101" and an online clearinghouse of case
studies and training materials. The PRA Papers serve as overarching guidance that directs readers to
more detailed sources. The PRA Papers may also be distributed at scientific meetings, and a separate
article may be published in the peer-reviewed literature.
One reviewer asked EPA how they identified the total number of case studies. EPA responded
-------
that it was an informal process. PRA Technical Panel members spoke with Regional and Program
Offices in combination with limited Internet searching and identified 16 case studies for the
documents under review.
One reviewer asked whether EPA had tried to assess the success and utilization of RAGS.
EPA thought this raised an important point: having good documents does not mean they will be
utilized. Managers may not have the time to read long, detailed documents; EPA needs to disseminate
information in digestible, appealing formats that are understandable and lead to utilization.
One reviewer asked whether the main white paper and managers' summary are two separate
documents or one document. EPA clarified that they are two different documents that will be
circulated together.
One reviewer asked about the case studies. The reviewer was interested in learning
whether information was solicited in a structured manner, whether EPA had asked respondents how
PRA affected their decision-making, and whether case studies from outside the Agency were
considered. EPA responded that they gave respondents a general outline of the type of
information they were seeking and fleshed out respondents' answers based on the literature. It
was difficult to capture the extent to which PRA impacted decision-making. In the Hudson River
example, PRA helped show that different ingestion rates still created unacceptably high risks. Using
examples from outside the Agency was not considered because it was important to the PRA Technical
Panel to have readily available contacts for each case study to answer questions. Another reviewer
followed up by asking about the public comments that criticized the case studies as contradicting the
purpose of the PRA Papers. EPA replied that an official response to comments for the Hudson River
case has been issued, and that otherwise they would read and consider public comments carefully.
One reviewer asked whether EPA had contractual support in writing the document. EPA
responded that there was no contractual support, and that one PRA Technical Panel member currently
working at North Carolina State University was a federal employee at the time of writing.
2.2 Public Comments
Connery opened the meeting to observer comments. No comments were provided.
2.3 Reviewer Discussions
Connery then turned the meeting over to Scott Ferson, chair for reviewer discussions, to begin the
discussions. The reviewers opened with general comments, then discussed the charge questions,
and closed by reviewing their recommendations. Sections 3 through 8 of this report summarize
those discussions.
3. Reviewer Discussion: General Comments
During this portion of the meeting, each reviewer gave general comments about the documents and
highlighted key elements of their pre-meeting comments.
One reviewer expressed a general appreciation for the challenge of writing such a document, which
tackles a complex issue affecting all EPA operations, and hoped that over the course of the day
reviewers would find the best way to encourage the use of probabilistic methods in EPA. Several
other reviewers echoed these sentiments, citing a real need for such a document.
-------
Another reviewer expressed concern about how the document framed probabilistic methods as new
and different compared to current methods. Probabilistic analysis is well-established, well-validated,
and cost-effective. In fact, EPA was one of the first agencies to adopt the use of probabilistic methods
in the 1990s with the development of RAGS, despite the fact that PRA has not been widely applied
within the Agency since then. The reviewer noted that the case studies have the crucial role of
convincing risk managers that these are well-established and useful tools, especially when linked to
decision-making. The reviewer felt that the remaining sections of the documents reiterate information
covered by RAGS. In order to persuade risk managers, the PRA Papers should be as practical as
possible. This reviewer noted that the current case studies section (limited to 16 studies) does not
reflect the true abundance of case studies. Lester et al. (2007) documents an increasing number of
publications describing applications of PRA since the 1990s. This review paper located
approximately 2,000 abstracts related to PRA in the worldwide literature published since 2000, of
which 500 were specifically relevant to Superfund and 200 were extensively reviewed by the authors.
Having a comprehensive review of the case study literature is important in convincing risk managers
that PRA is a validated tool with an international following.
Another reviewer emphasized the need to catalyze change at the highest levels of the Agency,
beginning with restructuring fundamental legislative mandates to incorporate PRA. The reviewer
suggested that risk management standards need to be reinterpreted in light of modern understanding
of uncertainty and variability. The reviewer also suggested that the PRA Technical Panel needs to
convey to senior leadership that the Agency cannot do its job without a full understanding of the
concepts of variability and uncertainty, and how they relate to both equity and economic efficiency
aspects of risk management goals. The reviewer recommended that the Office of the General Counsel
should be involved at an early stage since they have significant influence over risk how risk
management is conducted.
This reviewer outlined four risk management criteria that would be informed by PRA techniques:
"fair process"; "fair distribution"; "do the very best you can"; and "first do no harm." Fair process
means that information is communicated fairly and fully from the technical side to the general
audience about the nature of the situation and possible tradeoffs. Fair distribution requires assessing
the distribution of risks incurred compared to the benefits of assuming those risks, and seeing how
likely it is that the state of the world will be improved. "Do the best you
Lester, R.R., L.C. Green, and I. Linkov. 2007. Site specific applications of probabilistic health risk assessment:
review of the literature since 2000. Risk Analysis 27:635-658.
can" is embedded in the context of limited resources and priority setting. Managers are forced to
allocate limited resources, both Agency and societal, and uncertainty and variability affect this
resource distribution. An important goal should be to make the best overall change in the world per
unit of allocated Agency resources. Finally, "first do no harm" means that regulatory decisions should
come with a reasonable assurance that the choices improve the world. These four risk management
criteria are value propositions that can be easily understood by risk managers and are informed by
PRA. These value propositions can convince risk managers that PRA is not just in the details, but can
actually be vital to their jobs. This reviewer concluded with three warnings about the PRA Papers:
The incongruity between devoting significant attention to exposure variability and uncertainty
without devoting equal attention to the variability and uncertainty in toxicity, especially in the
development of reference concentrations (RfCs) and reference doses (RfDs), leaves the Agency open
to ridicule. The National Research Council (NRC) recently recommended restructuring RfDs in terms
-------
of risk-specific doses (a specific incidence of harm or less, with a given degree of confidence).
Because this is a choice that needs to be made at the highest level of the Agency, it is imperative to at
least inform risk managers of this impending issue.
The PRA Papers tend to present probabilistic methods as fully developed tools ready for
"prime-time application." However, it is only fair to warn users that there are still problems in
existing techniques. For example, standard statistical calculations tend to overestimate variability and
underestimate uncertainty. Accounting for these problems is not trivial.
The tiered approach presented by the documents tends to support the pervasive idea that
probabilistic methods occur only at the very highest tier of analysis. What usually happens is that 85
percent of resources are used on deterministic approaches to a risk problem and probabilistic analyses
are tacked on the end. However, probabilistic methods can often be more important at the beginning
of a study than at the end (e.g., screening, sensitivity analysis). It is important to inform risk managers
that probabilistic methods can be used at any tier. Another reviewer seconded the idea that
probabilistic methods should be used early in the process, particularly to facilitate communication. By
presenting a range of outcomes at the beginning of the process, managers can avoid appearing to
reverse decisions later on.
Another reviewer noted that reluctance to use PRA stems from risk managers, not assessors, and so
communication between the assessors and managers is critical. In the document, linking examples
(especially decisions) to the text will help to demonstrate how PRA can work.
The final reviewer remarked on the widespread belief that PRA has never changed or improved a
decision. Clear examples where PRA significantly improved outcomes are lacking. While this
reassures users of deterministic methods, it is important to be upfront that PRA might not change
decisions. Rather, it helps make better justified and defensible decisions. With additional justification
come additional costs. This reviewer suggested that part of PRA's unpopularity stems from the
appearance of "smoke and mirrors" and from fear on the part of risk managers. It is important for the
PRA Papers to be candid about PRA, including disclosing the limits of PRA, so that managers are not
blindsided during the process. In addition, the text needs to be more compelling, possibly supported
with illustration.
4. Reviewer Discussion: Question 1
1. EPA's application of Probabilistic Risk Assessment (PRA) in decision-making covers a range
of activities including: regulatory decisions (i.e., ranging from emergency actions to national
regulations), site-specific decisions (e.g., the Superfundprogram); and various media. The
PRA papers under review attempted to provide a balanced presentation of the advantages and
disadvantages of using PRA to support this range of decisions. Based on your review of the
document, please comment on the following:
la. Have the papers provided a balanced view of the usefulness of PRA in
decision-making, and if not, describe how it is unbalanced?
Reviewers suggested several ways the document could be restructured to provide a more balanced
view of PRA. One reviewer was concerned that the document isolated PRA from the broader context
of decision-making, which includes knowledge of the relevant hard science, decision science, and
political economics. In terms of content, one reviewer noted that the documents did a reasonable job
covering the issues, but were somewhat lacking with respect to toxicology, especially upcoming
changes to EPA's Integrated Risk Information System (IRIS) process to reflect that toxicity criteria
-------
should not be treated as single point estimates. PRA can accommodate toxicity as well as exposure
assumptions, but the latter are not well reflected in the PRA Papers. Some of the ecological and
communication issues could also be better highlighted. Organizing the document by topic areas could
be very helpful. Another reviewer noted that references should be updated to reflect literature
published since 2008.
One reviewer listed key ideas that have not received enough attention in the documents:
PRA is not just another "turn-the-crank" process which uses different numbers. Rather, it
reframes the problem for risk managers.
Probabilities are conditional—that is, they are snapshots of what you know at one particular
point in time—and so the results of PRA reflect changing knowledge. The paper should emphasize
the need for long-term monitoring and iterative refinement. Risk assessors should remain involved
throughout the entire process.
The costs of ignoring uncertainty are as important at the costs of performing PRA. Although
the costs of ignoring uncertainty are hard to quantify, they will lead to inefficient distribution of
resources.
Not all uncertainties are equally important. Decision criteria may eliminate the need to
quantify uncertainty for some variables.
Eliminating uncertainty and fully characterizing variability will not eliminate problems with
risk assessment and risk management. PRA does not solve fundamental disagreements over costs and
decision criteria.
Although PRA provides conclusions about uncertainty, the value of PRA is in having
confidence in decisions. Stating uncertainty as confidence forces a top-down perspective which helps
managers understand their options.
Other reviewers felt that the documents, especially the managers' summary and case studies,
could be written to be more relevant and accessible for practitioners. Individual reviewer points
included:
The documents place too much attention on the principles of PRA, while the description of
how PRA can affect decision-making is extremely weak. The documents should focus on linking
PRA to decision-making and illustrating the point with specific examples.
The managers' summary should be focused on specific decisions; for example, on alternative
courses of action that managers might consider. The document could have two pages of very specific
decision frames with information on how PRA can help and examples of what has already been done.
The extensive overlap with the main white paper needs to be eliminated.
The documents should be better integrated so that readers are not forced to make the links
between the documents and the case studies. Case studies that exemplify the issues at hand should be
referenced in the text and given brief explanation. Hypothetical examples can serve the same purpose
if no relevant case study is available.
The variety of case studies is treated in a shallow way. The utility of PRA for a wide variety
of applications within the Agency's mission should be made explicit.
The use of diagrams would frame the documents for the intended audience within the Agency
and likely improve communication of ideas.
lb. Are there additional advantages or disadvantages of using PRA that were not identified and
should be included? Please provide a description of any additional advantages or
disadvantages you have identified.
Reviewers primarily identified additional disadvantages of PRA that should be included in the
-------
PRA Papers. One reviewer noted that disadvantages provide insight into PRA.
Without tight project controls that keep analysts on track, PRA can lead to wasted
expenditures. However, sensitivity analysis can provide the appropriate focus to analyst activities, and
wasted resources indicate sloppy practice rather than a true disadvantage of PRA.
There is no guarantee that the conclusions of a PRA will be used by regulatory
decision-makers. Even after a PRA is complete, decision-makers might still rely on the original
screening-level assessment values via applicable or relevant and appropriate requirements (ARARs).
This disadvantage is an institutional issue, and comes down to how ARARs should be used when a
site-specific assessment has been performed.
PRA is incompatible with one-off decision processes (where a final decision is
rendered after a discrete period of sampling and analysis). Important uncertainties are revealed during
the
PRA process, which necessitates greater flexibility and suggests the benefit of long-term
monitoring and adaptive management strategies. Another reviewer added that with high-end
strategies such as adaptive management it is crucial to have a clear decision context and decision
model.
PRA can be used to obfuscate risk management decisions in adversarial settings and can be
perceived as being easily manipulated to achieve a specific agenda.
PRA requires the additional burden of gathering dependence information (i.e., pairwise and
higher order interactions of the distributions). Dependence information is not self-evident, and it
represents a large burden to analysts performing PRA that is not addressed in deterministic models.
PRA must avoid the ensemble problem. Probability distributions always refer to one
particular population, and due to the variability that exists across populations it is not acceptable to
substitute one ensemble's characterization with another's.
Three additional advantages of PRA were noted:
PRA allows you to think clearly about the issues and tradeoffs that decision-makers face.
Probabilistic tools are suited for communication, and communication should be identified as
an objective early in the process.
PRA can facilitate evaluation of multiple stressors as part of a relevant complex evaluation.
lc. Are there criteria or considerations that were not adequately addressedfor determining
whether or not probabilistic methods may be useful to characterize uncertainty and
variability and support EPA decision-making? If so, please describe such considerations.
Reviewers offered the following considerations with respect to determining the usefulness of
probabilistic methods to support EPA decision-making:
PRA is not appropriate for every situation (e.g., one-off decisions). However, uncertainty
analysis should be mandatory at the start of every long-term project in order to assess the need for
PRA.
Before PRA begins, there should be a process in place to bring in external reviewers.
PRA should only be used where designed to assess confidence in decisions (versus assessing
uncertainty in underlying processes).
PRA should not be pursued in a vacuum. Considerations for using probabilistic methods
should be mapped to risk managers' objectives. Risk managers should not invest in PRA unless it has
a reasonable chance of being considered by decision-makers. Value of Information (Vol) analysis
may be necessary. Where the decision is already robust, there is no need for PRA.
-------
The Office of General Counsel and the legal arms of the Program Offices should be involved
in determining how PRA can be used in addressing specific legislative mandates. Risk managers are
focused on specific legislative mandates, and the document should attempt to address the differences
in mandates across the Agency with respect to the utility of PRA.
PRA may have different values at different levels of decision-making (e.g., priority-setting
decisions versus rulemaking).
Upfront interaction between the risk assessor and risk manager and problem formulation are
important to achieving the desired results in the end-product.
One reviewer was concerned that the document misrepresented why there is confusion about how
much data PRA requires. The reviewer noted that the amount of data needed is not a "controversy"
but rather a function of the probabilistic approach. Different approaches (e.g., 2dimensional Monte
Carlo, probability bounding) require different amounts of data. Those differences should be spelled
out for risk managers. This reviewer also felt that the document neglected to mention the controversy
that managers might face when using subjectivist methods. People may believe that the personal
beliefs of the analyst should not have any role in formal analysis conducted by EPA. The reviewer
noted that it is the responsibility of the PRA Technical Panel to at least frame the debate for managers
and also to mention the costs associated with subjectivist techniques, particularly expert elicitation.
Id. Considering risk managers are a major audience for these documents, especially for the
"Managers' Summary " document, what additional information should be included in the
document to aid them in decisions regarding the application of PRA ?
One reviewer suggested completely rewriting the managers' summary as a vision statement on how to
perform risk assessment moving forward. Managers need to understand why there is little evidence to
show for the last 20 years of usage. The managers' summary should set a direction for EPA so that
the wealth of expertise on PRA is better incorporated into the Agency's decision processes.
Describing the tools yet again is not the way to get there. The reviewer recommended that, in
particular, the vision statement should jump straight to page 7 of the current summary, which lists
seven key questions for decision-makers. These key questions would be the new headings of the
managers' summary and would be answered from the perspective of PRA (and uncertainty analysis in
general). The reviewer felt that answering those questions, not just posing them, would show EPA
how to use PRA more effectively in the next 10 years than it has been used in the past 20 years. The
new summary would be cross-referenced with the case studies and the technical paper.
One reviewer objected to the seemingly random order of questions by which the managers' summary
was organized. At minimum, these questions should be posed alongside stated objectives and
organized by topical areas. The reviewer recommended that a section on decision analysis is needed
to provide context, although it should stick to strictly practical elements. Alternatively, the summary
could be structured as a series of decision-specific summaries (e.g., Superfund, pesticides) followed
by some general analysis. The reviewer noted that the language is straightforward and easy to
understand, but it is not the information that managers need in order to implement PRA.
Another reviewer disagreed with the need for a larger context of decision analysis. Risk managers are
already aware of the nature of their job. The real purpose of the document is to answer for managers,
"What is PRA going to do for me?" The random question structure is apropos for this situation, and it
would be even better if the questions were available in any order in a web-based structure (like an
FAQ).
-------
A reviewer noted that collecting new data for PRA not only contributes to that particular study but
adds to the wealth of knowledge that can be applied to related problems. In general, problems can at
least be characterized by thinking creatively about analogies with existing data. For example, without
doing any toxicological testing on an uncharacterized chemical, analysts can look at results for a
similar ensemble of chemicals and at least get a sense of the toxicity of the unknown chemical. This
incentive for managers to invest in new data should be described in the managers' summary.
Another reviewer advocated for the use of graphics and hypertext links, rather than just text, as a
pragmatic way to reach out to managers. In particular, a roadmap figure would provide context, and a
figure illustrating the tiered approach would highlight the iterative or looped nature of the process.
Having this as a web-based document would increase upfront accessibility, and increasing continuity
between the text and examples would help to illustrate points. The reviewer also noted that the term
"tiered approach" does not appear in the glossary despite being a main concept.
Finally, one reviewer suggested presenting in the managers' summary some of the common types of
PRA results that managers will see, such as a cumulative density function (CDF). There are a limited
number of types of results that PRA will yield, and managers need to understand upfront the type of
answers that PRA provides and how it provides them. Another reviewer disagreed with adding this
type of content, contending that probability distributions should be opaque to risk managers. For risk
managers, PRA should boil down to statements about confidence in alternative choices. Overall,
reviewers agreed that the summary should include enough information about results so that
managers have confidence in their decisions and can effectively perform their job. However,
reviewers had diverse opinions about what that level of information is, whether it is the bottom line
in the form of confidence statements, basic knowledge of the statistical functions (e.g., CDFs), or
enough understanding of variability and uncertainty to be able to detect instances where more
information is needed and subsequently be able to ask the right questions.
5. Reviewer Discussion: Question 2
2. The PRA papers were designed to address the use of these techniques in both human
health and ecological risk assessments.
Have the current documents adequately addressed the unique issues, if any, associated
with application of PRA to both human health effects and ecological risk? If not, what
additional information/case studies should be presented specific to human or ecological
risk assessment?
One reviewer felt that the case studies were disproportionately focused on human health, but was not
surprised since ecological applications of PRA tend to originate from outside EPA. Despite those
limitations, Case Study 13 did identify the big three uncertainty issues in ecological risk assessment:
1) deriving risk-based effect thresholds, 2) linking organism-level measurement endpoints to
population-and community-level endpoints, and 3) accounting for the effects of landscape on
exposure and risk. Although these issues are identified, they cannot be more thoroughly addressed in
the case studies. The reviewer said that there are more case studies that could be cited, but not all of
them are in the published literature and many more are outside the Agency. Another reviewer noted
that ecological case studies with multiple stressors should be included. A third reviewer agreed that
the case studies had a greater focus of human health than ecological risk and that the balance could be
improved. However, another reviewer disagreed, proposing that there was no real difference between
human health and ecological risk assessment.
-------
A reviewer reiterated the need to bring in a broader perspective on the use of PRA as a widely
used, internationally respected tool with hundreds of applications. The type of information
presented in Lester et al. (2007) should be updated and included with the case studies.
One reviewer re-emphasized the importance of probabilistic assessment for non-cancer health
effects and of adding the probabilistic piece to quantifying uncertainties in carcinogenic
assessments. It creates a disconnect if exposures are characterized probabilistically and
health-effect numbers are not. Another reviewer noted that focus on the toxicological side is
lacking, and proposed that sources from outside the Agency (e.g., the American Chemistry Council
submission, via the public comment process, on dioxins) could be used if someone outside the
Agency was willing to serve as a contact.
6. Reviewer Discussion: Question 3
3. In the Manager's Paper, EPA recognized the variety of PRA tools that may be used in
decision-making. The document states: "This document does not prescribe a specific approach
but, rather, describes the various stages and aspects of an assessment or decision process in
which probabilistic assessment tools may add value. The concept of iterative or tiered analyses
to address this continuum is a widely accepted goal in risk assessment, and the same principle
applies to PRA as well. There is a wide range of methods and approaches to PRA, of varying
complexity and rigor, which can be appliedfor different purposes ranging from basic sensitivity
analysis to integrated analysis of variability and uncertainty. The goal is to choose a level of
detail and refinement for an analysis appropriate to the overall objectives of the decision and
the types of available data and analyses needed to support decisions. " Based on your review of
the document:
3 a. Please comment on the completeness of the discussion of the tiered approach for
decision-making regarding the application of PRA. What, if any, additional refinements
to the tiered approach outlined in the document can you recommend to further support
management decisions regarding use of PRA ?
Overall, reviewers support the tiered approach, but noted that probabilistic methods are not limited
to higher tiers, nor must users begin at the lowest tier and proceed linearly to the highest tier. A
graphic depicting the flexible and iterative nature of the process would be beneficial for readers.
One reviewer noted that uncertainty analysis should be conducted earlier in the process. It can be
useful for screening-level analysis to quickly get to a no-further-action decision, where appropriate.
Discussion should include information about the appropriateness of performing uncertainty analysis
(probabilistic or not) in the earlier stages of risk assessment. The document should also recognize that
PRA can be very valuable in identifying which uncertainties will affect decision-making; this may
trigger further analysis and thus lead to iterations within tiers. Another reviewer agreed, emphasizing
that reserving PRA until the end stages of analysis significantly reduces its utility. "Quick and dirty"
uncertainty analyses using analogous data at the beginning of a process will provide an empirical
basis for decisions that have to be made later in the process. This type of mechanism increases the
efficiency of the process, and again is an incentive to collect raw data and build up databases.
One reviewer noted that the document presents a tiered approach for risk assessment but not for
decision-making. Again, the decision component is lacking. There is no information on specific
-------
decisions EPA has made. The document should embed PRA in the decision framework and link it to
specific management decisions. Another reviewer commented that PRA should be used to inform
management decisions, not to make decisions. The "techie" performing PRA is not the
decision-maker—the manager is.
3b. Please comment on the use of a tiered approach to when to use PRA considering
management considerations of cost, time, and resources.
Overall, reviewers felt that using the right techniques at the right time will lead to a
cost-effective approach. Considerations individual reviewers noted included:
Using probabilistic methods or uncertainty analysis early in the process can ultimately bring
faster closure and save money. Uncertainty analysis helps form hypotheses and provides the basis for
the right experimental design to answer the right question.
To be cost-effective in situations where there are many contaminants in the environment (e.g.,
Superfund site, urban setting), PRA should be focused on risk drivers—those chemicals and receptors
most likely to make a difference to the decision-maker's choice. The risks of other chemicals are
subsumed under risk drivers, and measures protective of risk drivers are then checked to see if they
are protective of the other chemicals. Although the public generally dislikes treating certain variables
preferentially, it is important to counsel risk managers that this is a valid, cost-effective measure.
PRA allows the multiple parties involved in decision-making to agree to disagree on many
issues. The values of many variables are irrelevant (i.e., a wide range of values has no effect on the
outcome), and so parties can focus attention and resources on coming to consensus on those
uncertainties that are driving parties to conflicting positions. Another reviewer noted that probability
bounding is a great technique to use to hone in on those variables for which parties can agree to
disagree. Probability bounds get progressively narrower, but the problem never has to be reframed.
Risk managers themselves are in fact experts on project dynamics and have institutional
experience dealing with the tradeoffs of cost, time, and resources. Risk managers should work closely
with analysts in this regard to ensure that the right tools are applied to the right problem in a
cost-effective manner.
One reviewer expressed this as a value-of-information question: Does sponsoring PRA make sense
for the decision at hand? The reviewer recommended that the decision to use PRA should be based on
a clear decision model which visualizes cost, time, and resources. Without such a decision model, risk
managers will only be guessing at the value of PRA, and the tiered process of risk analysis will not
help managers consider tradeoffs with cost and time. The current discussion in the document on this
point is incomplete. Another reviewer added that the document should then have some discussion of
value-of-information analysis, because managers will not automatically understand that this requires
mathematical modeling of the decision process and that they must first buy into the decision model.
A different reviewer disagreed with the intensive discussion on decision analysis, stressing that
performing PRA is a hypothesis testing problem, not a decision problem. The reviewer suggested that
it is possible to discuss PRA without framing it as a decision problem; arguably, managers themselves
do this. From the authors' perspectives, a focus on decision-making may be out of context. Another
reviewer rejected taking PRA out of the decision-making frame in the context of this EPA document.
This reviewer said that the document implicitly assumes that managers are making decisions;
otherwise there is no need or purpose for the document. One reviewer asked EPA if PRA is assumed
to be performed in a decision context. EPA responded that risk assessment is usually intended for a
specific decision-making purpose, although it may be a "regulate/do not regulate" decision rather than
a decision between alternatives.
-------
7. Reviewer Discussion: Question 4
4. The papers are intended to communicate information to EPA risk assessors and managers.
4a. Is each document written at the appropriate level of detail for the intended audience and
length of document? Specifically, is too much or too little detail presented for the
respective audiences? Please provide suggestions on how to address any such
shortcomings.
Reviewers were more concerned about the content of the documents than the level of detail.
Several reviewers made suggestions about the content of the managers' summary:
The managers' summary should be rewritten as a vision statement organized by the seven key
questions considered by decision-makers.
The managers' summary should be more distinct from the main white paper, and address the
unique needs of managers. The managers' summary should explain how risk assessors can help risk
managers throughout the PRA process.
The managers' summary should have a checklist, "How do I tell when I'm being scammed?"
for the set of techniques and distributions common to PRA. This will help demystify PRA for risk
managers. For example, managers should know that when the bounds are important, analysts should
not implicitly assume that no values outside the 95 percentile will occur. This artificially limits the
range of parameters.
Parts of the managers' summary read like a textbook and were not compelling or relevant.
Definitions of variability and uncertainty sound like they were written by risk assessors and not by
risk managers.
The managers' summary would benefit from the context figure on slide 16 of the EPA
presentation.
One reviewer had a question of clarification for EPA. The reviewer asked whether the PRA
Technical Panel had spoken to risk managers, and if so, was the content of the document what the
risk managers asked for. Did the PRA Technical Panel have the expectation that this document
would meet the needs of risk managers? EPA responded that this document captured the technical
needs of on-the-ground decision-makers. However, it is important to consider that in many
situations reasons other than science will be driving the decision.
Another reviewer commented that the core of problem is the separation of risk management from risk
assessment. PRA is only one component of the decision, and it should be clear from the beginning
how much influence PRA will have. If political decisions have already been made, the reviewer
continued, then it might not be efficient to devote resources to PRA. The real value of PRA is when
the technical aspect is important, and the Agency needs to re-integrate science with decision-making.
EPA noted that the Science Policy Council and senior-level management have been and will continue
to be included in the process. Another reviewer noted that the heritage of the separation of risk
assessment and risk management is that risk managers and risk assessors are not as familiar with each
other's disciplines as would benefit the process.
Reviewers also had concerns about the main white paper:
-------
The document should focus on lessons learned and challenges moving forward. The document
should eliminate redundancy with past sources.
The document should have more pointers about where to go for more information.
References should be updated to reflect current literature.
Section 2.5, which addresses a key question (How can PRA provide more comprehensive,
rigorous scientific information in support of regulatory decisions?) should be developed further.
The internal redundancy embedded in the document is better suited to a web-based,
hyperlinked format then the linear format of a paper document.
4b. Is the level of detail of information in the case studies adequate, or is more information
needed? Please provide specific suggestions on how to improve the case studies, if you have
any.
One reviewer noted that in order to derive maximum benefit from the case studies, EPA needs to
force interviewees to answer the question, "How did PRA affect decision-making?" All the case
studies should answer a structured set of question so that EPA ends up with a library of case studies
with consistent content. If PRA did not influence the decision, then the interviewee should address
whether performing the analysis was still beneficial. The case studies need to systematically get at the
relevance of the PRA, not just the technical details of how it was performed.
Another reviewer noted that the subsection on "Management Considerations" might point to that type
of information. A reviewer responded that the subsection neglects to discuss the relevance of the
analysis and proposed an alternative two-part structure for the case studies section. The first section
would broadly discuss international application of PRA and justify its use, and the second section
would focus on specific case studies with management implications.
A reviewer added that case studies should not be limited to success stories, but also those cases where
PRA was ignored or was a dead-end. Another reviewer noted that explicit coverage of risk
management criteria that were informed by PRA would be beneficial (e.g., "If the confidence had
been higher, I would have chosen this alternative instead."). A third reviewer added that the main
documents should have explicit call outs to these case studies in the text.
4c. Please discuss whether, and how, the White Paper could be improved to help the reader to
better understand how PRA can help address and communicate variability and uncertainty.
One reviewer advocated for graphical representation of variability and uncertainty, noting that
Morgan and Henrion, and Finkel are two good sources of ideas. Two other reviewers seconded the
addition of graphics. A fourth reviewer reiterated that "uncertainty" should be framed as
"confidence" for decision-makers.
A reviewer noted that the more sophisticated analyses associated with PRA often lead to greater
statements of uncertainty because they account for more factors. Decision-makers should not judge
the quality of an analysis by the size of the uncertainty.
4d. Are the citations and references sufficient, or are there critical references which need to be
added? If so, please provide those citations and their relevance to the papers.
The reviewers felt that Frey was over-represented in the citations. References should be expanded to
-------
include a more representative spread. One reviewer noted that references addressing subjective
probability and dependence issues need to be added. Specific references are provided in the
reviewers' pre-meeting comments.
8. Reviewer Discussion: Final Comments and
Recommendations
The chair opened discussion to any additional comments from reviewers. One reviewer noted that
the focus on PRA tends to be too limiting. Many probabilistic methods and models that enhance
risk analysis have not traditionally been included in the risk assessment process, but should be
included. Another reviewer thought that these types of models and methods were outside the scope
of this document, which is particularly focused on PRA, not probabilistic methods in general.
Reviewers did not agree whether the document's title matched its scope.
The chair asked reviewers to consider whether they had arrived at any recommendations.
Reviewers agreed on the following recommendations:
Greater use of graphics would enhance the documents. In particular, three key figures should
illustrate 1) the context of the document (roadmap figure), 2) the problem formulation framework,
and 3) the tiered approach (non-linear).
The document should be web-based and hyper-linked.
More emphasis and coverage should be provided on the toxicity side of human health,
including both non-cancer and cancer endpoints. The documents should address the NRC comments
on treating RfDs and RfCs probabilistically.
Case studies and other examples should be enhanced and better connected to the text. Case
studies should be standardized and have greater emphasis on how PRA affects management
decisions.
The document should support the use of probabilistic methods to justify preferential treatment
of risk drivers in cases of multiple stressors.
The documents should contain some discussion of dependence issues and the ensemble
problem. However, one reviewer cautioned that this type of information should not be discussed in
detail due to the potential for overwhelming risk managers.
More discussion should be devoted to ecological risk and the tone of the document
should be more inclusive. The documents should have an explicit discussion of the similarities and
differences of ecological and human health risk assessment.
Reviewers noted their different views on the following items:
Reviewers did not agree whether the documents should focus more heavily on
decision-making. Three reviewers were in favor of emphasizing decision-making, but one reviewer
thought that performing PRA in the absence of a decision-making context was still a legitimate
process with a place in the document.
Reviewers disagreed on the benefit of using PRA as a communication tool. One reviewer
thought that PRA is always a better framework for communication, particularly to the public,
because of its greater flexibility. For example, as mentioned earlier, by presenting a range of
outcomes at the beginning of the process, managers can avoid appearing to reverse decisions later
on (i.e., avoid some of the pitfalls of explaining why a less conservative evaluation is appropriate
because the deterministic default evaluation shows a potential issue). However, another reviewer
thought that the probabilistic approach should not be preferred over a deterministic approach,
-------
unless Vol analysis demonstrates that the probabilistic approach has substantial added benefits.
This is because the public can more easily understand a threshold, bright-line value.
Connery thanked the reviewers for their participation. Gallagher thanked the reviewers on behalf of
the EPA for contributing their time and expertise, and expressed that their comments would help
improve the documents.
Peer Review Workshop of EPA's Draft Report,
Using Probabilistic Methods to Enhance the
Role of Risk Analysis in DecisionMaking with
Case Study Examples
Sheraton Crystal City Hotel
Arlington, VA
May 6, 2010
Reviewers
Scott Ferson, Ph.D. (Chair)
Senior Scientist Applied Biomathematics 6317514350 Email: scott@ramas.com
Annette GuiseppiElie, Ph.D.
Environmental Engineer/Scientist
E. I. DuPont De Nemours and Company 1007 Market Street Wilmington, DE 19898 8642245058
Email: annette .guiseppielie@usa. dupont. co m
Dale Hattis, Ph.D.
Research Professor George Perkins Marsh Institute Clark University Worcester, MA 6172832521
Email: dhattis@clarku.edu
Igor Linkov, Ph.D.
APPENDIX A
PEER REVIEWERS
United States
Environmental Protection Agency
Office of the Science Advisor, Risk Assessment Forum
United States Environmental Protection Agency Office of the Science Advisor, Risk Assessment
Forum
-------
Team Leader Risk and Decision Science Focus Area Lead
U.S. Army Engineer Research & Developmental Center Concord, MA 6172339869 Email:
igor. linko v@usace. army. mil
John Toll, Ph.D.
Partner Windward Environmental LLC 200 West Mercer Street Suite 401 Seattle, WA 98119 206812-
5433 Email: johnt@windwardenv.com
APPENDIX B
CHARGE TO PEER REVIEWERS
Technical Charge to External Peer Reviewers
Contract No. EP-C-07-024
Task Order No. 74
April 1,2010
External Review for EPA's draft "Using Probabilistic Methods to Enhance the Role of
Risk Analysis in Decision-Making, with Case Study Examples" and "Using Probabilistic
Methods to Enhance the Role of Risk Analysis in Decision-Making: Managers' Summary"
2.1 BACKGROUND INFORMATION
Probabilistic risk assessment (PRA) is a group of techniques that provide estimates of the range and
likelihood of hazard, exposure or risk, rather than a single point estimate. Various stakeholders, inside
and outside the Agency, have called for a more comprehensive characterization of risks, including
uncertainties, in particular those decisions protecting more sensitive or vulnerable populations and life
stages. Therefore, the Office of the Science Advisor of the EPA, together with EPA's Science Policy
Council and members of EPA's Risk Assessment Forum (RAF), identified a need to examine the use
of probabilistic approaches in Agency risk assessment and risk management. An RAF Technical
Panel developed two papers, one entitled "Using Probabilistic Methods to Enhance the Role of Risk
Analysis in Decision-Making With Case Study Examples" and the other a "Managers' Summary" of
the first document, to provide a general overview of the value of probabilistic analyses and similar or
related methods, and some examples of current applications across the Agency. The purpose of these
papers is not only to describe potential and actual uses of these tools in the risk decision process, but
also to encourage their further implementation in human, ecological and environmental risk analysis
and related decision making. The enhanced use of probabilistic analyses to characterize variability
and uncertainty in assessments would not only reflect external scientific advice on how to further
advance EPA risk assessment science, but will also help to address specific challenges faced by
managers and improve confidence in Agency decisions. The draft document was prepared by the
-------
Probabilistic Risk Analysis Technical Panel of EPA's Risk Assessment Forum and has undergone
internal peer review.
2.2 CHARGE QUESTIONS
1. EPA's application of Probabilistic Risk Assessment (PRA) in decision making covers a range of
activities including: regulatory decisions (i.e., ranging from emergency actions to national
regulations), site-specific decisions (e.g., the Superfund program); and various media. The PRA
papers under review attempted to provide a balanced presentation of the advantages and
disadvantages of using PRA to support this range of decisions. Based on your review of the
document, please comment on the following:
la. Have the papers provided a balanced view of the usefulness of PRA in decision making, and
if not, describe how it is unbalanced?
lb. Are there additional advantages or disadvantages of using PRA that were not identified and
should be included? Please provide a description of any additional advantages or
disadvantages you have identified,
lc. Are there criteria or considerations that were not adequately addressed for determining
whether or not probabilistic methods may be useful to characterize uncertainty and variability
and support EPA decision-making? If so, please describe such considerations.
Id. Considering risk managers are a major audience for these documents, especially for the
"Managers' Summary" document, what additional information should be included in the
document to aid them in decisions regarding the application of PRA?
2. The PRA papers were designed to address the use of these techniques in both human health
and ecological risk assessments.
Have the current documents adequately addressed the unique issues, if any, associated with
application of PRA to both human health effects and ecological risk? If not, what additional
information/case studies should be presented specific to human or ecological risk assessment?
2 In the Manager's Paper, EPA recognized the variety of PRA tools that may be used in decision
making. The document states: "This document does not prescribe a specific approach but, rather,
describes the various stages and aspects of an assessment or decision process in which probabilistic
assessment tools may add value. The concept of iterative or tiered analyses to address this continuum
is a widely accepted goal in risk assessment, and the same principle applies to PRA as well. There is a
wide range of methods and approaches to PRA, of varying complexity and rigor, which can be
applied for different purposes ranging from basic sensitivity analysis to integrated analysis of
variability and uncertainty. The goal is to choose a level of detail and refinement for an analysis
appropriate to the overall objectives of the decision and the types of available data and analyses
needed to support decisions."
Based on your review of the document:
3a. Please comment on the completeness of the discussion of the tiered approach for decision
making regarding the application of PRA. What, if any, additional refinements to the tiered
approach outlined in the document can you recommend to further support management
decisions regarding use of PRA?
3b. Please comment on the use of a tiered approach to when to use PRA considering
management considerations of cost, time and resources.
-------
4. The papers are intended to communicate information to EPA risk assessors and managers.
4a. Is each document written at the appropriate level of detail for the intended audience and
length of document? Specifically, is too much or too little detail presented for the respective
audiences? Please provide suggestions on how to address any such shortcomings.
4b. Is the level of detail of information in the case studies adequate, or is more information needed?
Please provide specific suggestions on how to improve the case studies, if you have any.
4c. Please discuss whether, and how, the White Paper could be improved to help the reader to better
understand how PRA can help address and communicate variability and uncertainty.
4d. Are the citations and references sufficient, or are there critical references which need to be added?
If so, please provide those citations and their relevance to the papers.
APPENDIX C
REVIEWER PRE-MEETING COMMENTS
Peer Review Workshop of EPA's Draft
Report, Using Probabilistic Methods to
Enhance the Role of Risk Analysis in
Decision-Making
with Case Study Examples
Pre-Meeting Comments
-------
May 3, 2010
Notice
Pre-meeting comments were prepared by each consultant individually prior to the meeting. They
are preliminary comments only, and are used to help consultants become familiar with the
document and charge questions, develop the agenda, and identify key issues for discussion.
During the meeting, consultants may expand on or change opinions expressed in their
pre-meeting remarks and may introduce additional issues. For these reasons, pre-meeting
comments should be regarded as preliminary and do not reflect the final conclusions and
recommendations of individual consultants. After the meeting, reviewers will prepare
post-meeting comments that will reflect their final views. Post-meeting comments will be
provided in the workshop report.
Contents
Biographies
C-7
Scott
Ferson
C7
Annette Guiseppi-Elie
C-7
Dale Hattis
C-8
Igor
Linkov
C-8
John
Toll
C-9
RESPONSES TO CHARGE QUESTIONS
C-ll
1. EPA's application of Probabilistic Risk Assessment (PRA) in decision making
covers a range of activities including: regulatory decisions (i.e., ranging from
emergency actions to national regulations), site-specific decisions (e.g., the
Superfund program); and various media. The PRA papers under review attempted to
provide a balanced presentation of the advantages and disadvantages of using PRA
to support this range of decisions. Based on your review of the document, lease
comment on the following: C-l 1
la. Have the papers provided a balanced view of the usefulness of PRA in decision
making, and if not, describe how it is
unbalanced? C-ll
-------
lb. Are there additional advantages or disadvantages of using PRA that were not
identified and should be included? Please provide a description of any additional
advantages or disadvantages you have
identified C-15
lc. Are there criteria or considerations that were not adequately addressed for
determining whether or not probabilistic methods may be useful to characterize
uncertainty and variability and support EPA decision-making? If so, please describe
such considerations C-16
Id. Considering risk managers are a major audience for these documents, especially for
the "Managers' Summary" document, what additional information should be
included in the document to aid them in decisions regarding the application of
PRA? C-19
1 The PRA papers were designed to address the use of these techniques in both
human health and ecological risk assessments.
C-21 Have the current documents
adequately addressed the unique issues, if any, associated with application of PRA to both
human health effects and ecological risk? If not, what additional information/case studies
should be presented specific to human or ecological risk assessment?
C-21
2 In the Manager's Paper, EPA recognized the variety of PRA tools that may be
used in decision making. The document states: "This document does not prescribe a
specific approach but, rather, describes the various stages and aspects of an assessment or
decision process in which probabilistic assessment tools may add value. The concept of
iterative or tiered analyses to address this continuum is a widely accepted goal in risk
assessment, and the same principle applies to PRA as well. There is a wide range of
methods and approaches to PRA, of varying complexity and rigor, which can be applied
for different purposes ranging from basic sensitivity analysis to integrated analysis of
variability and uncertainty. The goal is to choose a level of detail and refinement for an
analysis appropriate to the overall objectives of the decision and the types of available
data and analyses needed to support decisions." C-23
3a. Please comment on the completeness of the discussion of the tiered approach for
decision making regarding the application of PRA. What, if any, additional
refinements to the tiered approach outlined in the document can you recommend to
further support management decisions regarding use of PRA?
C-23
3b. Please comment on the use of a tiered approach to when to use PRA considering
management
considerations of cost, time and resources C-25
4. The papers are intended to communicate information to EPA risk assessors and managers.
C-26
4a. Is each document written at the appropriate level of detail for the intended audience and
length
of document? Specifically, is too much or too little detail presented for the respective
audiences? Please provide suggestions on how to address any such
-------
shortcoming s C -26
4b. Is the level of detail of information in the case studies adequate, or is more information
needed? Please provide specific suggestions on how to improve the case studies, if you have
any C-27
4c. Please discuss whether, and how, the White Paper could be improved to help the reader to
better understand how PRA can help address and communicate variability and
uncertainty.C-28
4d. Are the citations and references sufficient, or are there critical references which need to be
added? If so, please provide those citations and their relevance to the
papers C-29
ADDITIONAL REVIEWER COMMENTS
C-31
Reviewer Biographies
Scott Ferson, Ph.D.
Senior Scientist
Applied Biomathematics
Scott Ferson is a senior scientist at Applied Biomathematics and an adjunct professor at the
School of Marine and Atmospheric Sciences at Stony Brook University. He holds a Ph.D. from
SUNY and an A.B. from Wabash College. He has over 100 publications on environmental risk
analysis and uncertainty propagation which have appeared in Reliability Engineering and System
Safety, International Journal of Approximate Reasoning, Computer Methods in Applied
Mechanics and Engineering, Accountability in Research: Policies and Quality Assurance, etc. He
serves on the editorial board of Human and Ecological Risk Assessment. He is author of five
books, including RAMAS Risk Calc Software 4.0: Risk Assessment with Uncertain Numbers
(Lewis Publishers), and several software packages. His research, funded by the National Institutes
of Health, NASA, Sandia National Laboratories, Electric Power Research Institute, NSF, etc.,
focuses on developing reliable mathematical and statistical tools for risk assessments and on
methods for uncertainty analysis when empirical information is very sparse, including methods
for quality assurance for Monte Carlo assessments, exact methods for detecting clusters in very
small data sets, backcalculation methods for use in remediation planning, and distribution-free
methods of risk analysis.
Annette Guiseppi-Elie
Principle Consultant, Risk Assessment
DuPont Engineering, Corporate Remediation Group
-------
Dr. Guiseppi-Elie is a Principal Consultant on Exposure and Risk Assessment issues for the
DuPont Company. Also, she is an Adjunct Associate Professor in Environmental Engineering and
Earth Sciences at Clemson University. She has served on a number of scientific entities in her role
as technical expert and advocate for the use of sound scientific principles and data in conducting
environmental health risk assessments. These organizations include the USEPA Science Advisory
Board, the Mickey Leland Center National Urban Air Toxics Research Center, the International
Programme on Chemical Safety's Planning Committee on Harmonization of Exposure
Assessment, and the American Chemistry Council's Human Health Exposure Assessment
Technical Implementation Panel. She is past Chair of the American Industrial Heath Council's
Environmental Health Risk Assessment Committee. Her expertise is in the areas of site and risk
assessment, specifically, exposure assessments and includes environmental fate and transport
processes. Dr. Guiseppi-Elie has conducted environmental site assessments and risk assessments
both in the US and internationally. Her doctoral research focused on the fate and transport of
dioxins in the environment. Her current research interests are in the areas of integrated/cumulative
exposure and risk assessment and the relationship between indoor, outdoor and personal air
exposures (e.g., the World Trade Center Indoor Air Assessment). Her work experience includes
service on the faculty at Drexel University, service to Mobil Oil and the Exxon Corporation. Dr.
Guiseppi-Elie received the B.Sc. degree in Chemistry and Zoology and the M.Sc. in Crop
Production Entomology from the University of the West Indies (UWI), Trinidad in 1977 and
1979, respectively. She received a M.Sc. degree in Pollution and Environmental Control from the
University of Manchester Institute of Science and Technology (UMIST), England in 1980 and her
Ph.D. in Civil Engineering from the University of Maryland College Park (UMCP), USA in 1987.
Dale Hattis, Ph.D.
Research Professor
Center for Technology, Environment, and Development
George Perkins Marsh Institute
Clark University
Dale Hattis, Ph.D., is a professor in the Center for Technology, Environment, and Development at
Clark University. His research focuses on methodologies for quantitative health risk assessment
for cancer and non-cancer health effects, human variability in susceptibility to toxic effects, and
pharmacokinetic and Monte Carlo simulation modeling. Previously he worked as a research
associate at the Massachusetts Institute of Technology. Dr. Hattis has served on numerous
committees, including the U.S. EPA Science Advisory Board's Environmental Health Committee,
the National Research Council's Committee on Neurotoxicology and Models for Assessing Risk,
and the National Toxicology Program's Center for the Evaluation of Risks to Human
Reproduction. His work includes numerous publications in professional journals and books such
as Toxicokinetics in Risk Assessment, Risk Policy Report, and Environmental Health
Perspectives.
-------
Igor Linkov, Ph.D.
Team Leader
Risk and Decision Science Focus Area Lead
U.S. Army Engineer Research and Development Center
Dr. Igor Linkov is the Risk and Decision Science Focus Area Lead with the U.S. Army Engineer
Research and Development Center, and Adjunct Professor of Engineering and Public Policy at
Carnegie Mellon University. Dr. Linkov has a B.S. and M.Sc. in Physics and Mathematics
(Polytechnic Institute, Russia) and a Ph.D. in Environmental, Occupational and Radiation Health
(University of Pittsburgh). He completed his postdoctoral training in Biostatistics and Toxicology
and Risk Assessment at Harvard University. Dr. Linkov has managed multiple ecological and
human health risk assessments and risk management projects that include alternative assessments
and beneficial sediment use. Many of his projects have included application of the
state-of-the-science modeling and software tools (e.g., probabilistic and Bayesian Monte-Carlo,
spatially-explicit modeling) to highly complex sites (e.g., Hudson River, Dow Midland, Natick
Soldier Systems Command, Elizabeth Mine) and projects (e.g., restoration and remediation
planning, insuring emerging risks, risk-based prioritization of countermeasures). He has published
widely on environmental policy, environmental modeling, and risk analysis, including eleven
books and over 100 peer-reviewed papers and book chapters. Dr. Linkov has organized more than
dozen national and international conferences and continuing education workshops on risk
assessment, environmental security, decision analysis, climate change, risk communication,
nanotechnology and modeling and participated in organizing many others. Dr. Linkov has served
on many review and advisory panels for DHS, EPA, NSF, EU and other US and international
agencies. The Governor of Massachusetts has appointed Dr. Linkov to serve as a Scientific
Advisor to the Toxic Use Reduction Institute. He is the recipient of the 2005 SRA Chauncey Stan-
Award for exceptional contribution to Risk Analysis.
John Toll, Ph.D.
Partner
Windward Environmental LLC
Ph.D., Engineering & Public Policy, Carnegie Mellon
University, 1989 B.S., Chemical Engineering, University of
Iowa, 1983
Dr. Toll has 20 years' experience doing environmental risk and uncertainty analysis in academia
and consulting. He is known for conducting innovative analyses and for seeking ways to improve
the manner in which science and technology inform environmental policy, planning, and
management decisions. He has authored or co-authored numerous scientific papers and book
chapters on ecological modeling and risk assessment and environmental decision analysis. His
work has appeared in Bioscience, Environmental Toxicology & Chemistry, Human & Ecological
Risk Assessment, Journal of Environmental Management, Marine Pollution Bulletin, Risk
Analysis and elsewhere. He currently serves on the editorial board of Integrated Environmental
-------
Assessment & Management.
Responses to Charge Questions
1. EPA's application of Probabilistic Risk Assessment (PRA) in decision making
covers a range of activities including: regulatory decisions (i.e., ranging from
emergency actions to national regulations), site-specific decisions (e.g., the
Superfund program); and various media. The PRA papers under review attempted
to provide a balanced presentation of the advantages and disadvantages of using
PRA to support this range of decisions. Based on your review of the document, lease
comment on the following:
la. Have the papers provided a balanced view of the usefulness of PRA in decision
making, and if not, describe how it is unbalanced?
Reviewer
Comments
Ferson
The discussion about the usefulness of PRA is generally well balanced, and serviceably
reviews the compelling advantages of PRA. The coverage of the range of EPA activities
and how they might be supported by PRA is also reasonably broad, if rather shallow. It
might be helpful to explicitly consider the role of PRA in each kind of activity if that
coverage is important, though I am not sure that it is. The abundance of citations to Frey,
however, represents an embarrassing lack of balance. Frey is an author or coauthor on
four papers cited in Appendix C. This seems excessive given the nature of the other
references listed. He is on twenty-two (!) papers in the bibliography, which just seems
completely indulgent. This should be moderated. The writing style is, if not imbalanced,
at least inconsistent. The text is redundant in many spots, and should be strongly edited
to improve its prose style, list seriality, and consistency in formatting and punctuation.
Because the documents are awfully dryly written, their stated goal of encouraging the use
of PRA may be a bit muted.
-------
The papers provide a reasonably balanced view of the utility of PRA. The key message
being that the probabilistic techniques have a significant role in risk-based decisions and,
hence, these types of analyses should be encouraged. The other key, of course, is that the
techniques vary and their use should be tailored to suit the needs of the assessment. This
point is highlighted in the documents but a more focused presentation of the tiered
approach is needed. Specifically, in terms of balance, it seems that the emphasis of the
documents is on human health and exposure issues. While ecological risk assessments
are acknowledged and some case studies are provided, it would be appropriate to include
a section in the text that addresses ecological assessments explicitly. The same is true and
even more so for toxicity parameters. While the documents address the issue, it is clear
that there is little practice in this area and, hence, the topic is not well addressed.
Probabilistic tools can be used to incorporate variability and uncertainty in dose-response
assessments and more knowledge of their use will hopefully lead to more acceptance. It
should be noted that both of these areas are identified in 3.3 as needing Guidance. The
final topic identified but not adequately addressed is on the utility of PRA to
communicate results, whether the audience is a risk manager or the general public (as
broad categories). The topic is mentioned but on balance it not
addressed as it deserves, since it is arguably a powerful use of
probabilistic techniques.
There are several problems with the current documents that need
to be fixed in the next iteration. Briefly:
Hattis
Sections 2.4 and 2.6 seem to end the review with a 2007 NRC
report and a 2007 EPA Science Advisory Board recommendation
for redefinition of the RfD in probabilistic terms. This must be
updated to include the latest recommendations (NRC 2009) for
redefinition of the RfD as a risk specific dose, and facilitate
inclusion of traditional threshold-type toxicity effects in benefits
estimates for regulatory impact analyses. This is a major issue that
will require the attention of agency risk managers for
resolution—not only whether to do this, but what exactly should
be the target levels of residual risk (incidence of harm at the RfD,
reflecting variability in individual thresholds for response for
traditional toxic modes of action) and confidence level that the
risk at the RfD will be at the chosen incidence level or below. The
reference section and bibliography (pp. 36-41) do not appear to
include any citations more recent than 2008, also does not cite
some of my work that I would suggest is relevant:
Hattis, D., Lynch MK. 2010. Alternatives to Pollutant-by-Pollutant
-------
Dose-Response Estimation for Air Toxics, report/conference paper by
Abt Associates, Inc. to the U.S. Environmental Protection Agency
under EP-W-05-022 WA 3-80, Final, March 2010.
(This is the latest update to the "Straw Man" series of papers illustrating
the implementation of probabilistic methods in defining reference doses
and assessing benefits for reducing exposures to toxicants that act in part
by traditional individual threshold processes. This version, among other
things, makes provision for interactions with background pathological
processes, as recommended by NRC (2009), and shows how the system
can inform assessments for "data poor" toxicants.)
Hattis, D. Failure to Communicate—Comment: Regulator/Risk
perspective on Burzala and Mazzuchi's Paper, In Cooke, R.M. (ed)
(2009) Uncertainty Modeling in Dose Response: Bench Testing
Environmental Toxicity, John Wiley and Sons, Inc., Hoboken, N.J.,
ISBN: 978-0-470-44750-5, pp. 153-159.
Hattis, D. and Lynch, M. K. "Empirically Observed Distributions of
Pharmacokinetic and Pharmacodynamic Variability in
Humans—Implications for the Derivation of Single Point Component
Uncertainty Factors Providing Equivalent Protection as Existing RfDs."
In Toxicokinetics in Risk Assessment J. C. Lipscomb and E. V. Ohanian,
eds., Informa Healthcare USA, Inc., 2007, pp. 69-93.
Hattis, D., and Anderson, E. "What Should Be The Implications Of
Uncertainty, Variability, And Inherent 'Biases'/'Conservatism' For
Risk Management Decision Making?" Risk Analysis. Vol. 19, pp.
95-107 (1999).
(This older paper is important in that it explicitly addresses risk
management issues related to uncertainty and variability. One
conclusion is that managers should care differently about upper
percentiles on these two probabilistic scales. It suggests that
managers will want to have reasonably high confidence (e.g. 95% on
the confidence/uncertainty dimension) that they are protecting a very
high percentage of the real people (e.g. all but one in 10,000,
100,000, or even 1,000,000 of the real people in our diverse human
population).
Second, there is a balance issue in the implicit representation of
PRA techniques as entirely off-the-shelf tools that are entirely
ready for prime time. In fact there are substantial needs to address
important issues in the practical quantification of both variability
and uncertainty using now-standard techniques. The fact is,
conventional calculations of variability based on observations in a
-------
data set, such as a standard deviation, tend to overstate the real
variation that affects differential risks because any set of
observations will include some measurement error. The presence
of the measurement error means that the observations are more
spread out among people than the underlying things being
measured. On the other hand, conventional measures of
uncertainty, such as standard errors in mean values of data, or
regression coefficients generally understate real uncertainty
because they are only based on fluctuations among the observable
data within the set of data values—ignoring often important but
not directly measurable sources of systematic error. Sources of
systematic error include miscalibration of instruments and
imperfections in the representative ness of samples for the
populations or (for site measurements) places where
contamination is likely to exist. It is possible to make estimates of
this kind of distortion,'however at present it is practically never done,
and should be. The current documents do allude to some different
sources of model errors and "scenario errors", which is well and good,
but full frankness in selling the PRA ideas should include this issue, I
think, and not imply that all the problems in quantifying uncertainties
are well understood, with fully off-the-shelf recipes available. I
recognize that the authors may not have wanted to raise complications
that would imply that PRA is "not ready for prime time". There are
many tools in the PRA framework that are fully ready to be used, but
these tools still need further development/maintenance/ "sharpening" to
become better and truer to the facts of environmental risks in the future.
1
Shlyakhter AI 1994. An improved framework for uncertainty analysis:
Accounting for unsuspected errors. Risk Analysis 14:441-447.
The stated goal of the document is to "present general concepts and principles of
PRA, to describe how PRA can improve basis for Agency decisions, and
provide illustration on how PRA can be used in risk estimation..." The main
body of the documents is primarily focused on presentation of general PRA
principles (and repeat to a large extent previous EPA documents, specifically
2001 PRA RAGS). Description of how PRA can be integrated in the Agency
decisions is very weak. Illustration on PRA use is done in Appendix 4 only with
very limited discussion in the main body of the document. The imbalance is very
clear, more emphasis should be given to linking PRA with specific decisions and
illustrating that PRA is useful and has been extensively used by this agency and
others.
Linkov
-------
Specifically in the Managers' summary, too much emphasis is placed on
regulatory decisions, typically with two implied alternatives (regulate/ do not
regulate), while real decisions involve the evaluation of alternative courses of
action. This means that PRA is used only to analyze risk for the status quo rather
than under each alternative. If specific types of Agency decisions had been more
emphasized in the Managers' summary, the problem would have been avoided.
For example, a qualitative discussion of site remediation alternatives, the factors
that typically influence them, and how these factors may be related to the final
risk, could be helpful in discussing PRA use for contaminated sites.
On p3 it is mentioned that the Managers' summary will explain how PRA can
"allow for more detailed comparison of alternative risk management options
in terms of estimated impacts on both protection and costs". This has not
been discussed in the document.
It's been 20 years now since the first significant publications on dealing with
uncertainty in quantitative environmental risk analysis (Morgan and Henrion
1990; Finkel 1990). Those publications made essentially the same case for the
usefulness of PRA in decision-making as the papers we're peer reviewing, and
yet little progress has been made over the ensuing decades.
Toll
If the papers provided a balanced view of the usefulness of PRA in
decision-making then they'd shed light on the lack of progress since 1990. They
don't. My conclusion, then, is that they don't provide a balanced view of the
usefulness of PRA in decision-making. Here's what' lacking:
Focus more on how to use uncertainty analysis to characterize how confident decision
makers should be in their choices. As Wilson (2000) puts it, "... uncertainty is the bane of any
decision maker's existence. Thus anyone who wants to inform decisions using scientific
information needs to assure that their analyses transform uncertainty into confidence in
conclusions." Environmental risk assessments are complicated and it's easy to get lost in the
details. The translation of uncertainty into confidence statements forces a "top down" perspective
that promotes accounting for whether and how uncertainties affect choices (Toll et al. 1997).
Highlight that probabilities are conditional. That leads to value of information analysis
(Dakins et al. 1996), and VOI bumps up against institutional barriers to effective long-term
monitoring, adaptive management and interim remedies. Until we overcome those barriers PRA
will remain on the margins of environmental decision making.
Place greater emphasis on the costs of ignoring uncertainty (McConnell 1997; Toll 1999).
Better help readers recognize that not all uncertainties should be analyzed. For example,
exposure variability in a human population doesn't necessarily belong in a PRA if the decision's
already been made to base risk management choices on risk to a relatively highly exposed
subpopulation. Reintroducing the concept of a decision variable (Morgan and Henrion 1990;
Finkel 1990) could be useful in this regard.
-------
lb. Are there additional advantages or disadvantages of using PRA that were not
identified and should be included? Please provide a description of any additional
advantages or disadvantages you have identified.
Reviewer
Comments
Ferson
For the most part, the discussion of advantages and disadvantages of PRA is reasonably
complete, given the intended audiences of the documents and their intent, with one major
exception.. A significant disadvantage of PRA that is not mentioned is that it requires
assumptions about the stochastic dependence among all of the distributional inputs. This
means that the parameterization of a probabilistic model requires not just estimates of the
distributions of the various inputs, but also estimates of all the dependencies among
them. A deterministic model, in contrast, needs no such specifications. Typically,
independence is assumed throughout, but this is not always reasonable and can
sometimes be ridiculous (as between body size and ingestion rate for instance). In
general, pairwise (or higher) correlations or more complex dependence functions must be
specified. Although sometimes neglected in discussions, this represents an often
significant burden on the analyst. In the middle paragraph on page 2 of the Managers"
Summary, the advantage of PRA that it informs decision makers about specific segments
of the population at risk. At this important point in the text, I had expected it to say what
it tells about those segments, perhaps mentioning that there are different possible values
because of variability or uncertainty or both, with different likelihoods.
Guiseppi-Elie
The documents appear to be adequate in identifying the advantages and disadvantages of
the PRA. One advantage not explicitly identified is the potential to be used for
comparing across multiple stressors. The use of relative risk models has been used for
some time. While, they were initially used for ecological evaluations, they are also useful
to incorporate disparate endpoints, including human health. Further, current applications
can include socio-economic considerations. See for example:
As noted above, a key advantage is
the use of the "probabilistic" tools for communication. However, it is important that if
communication (particularly to the public) is an objective then this needs to be identified
early in a project and an appropriate plan developed to ensure success.
Hattis
The main advantage that I see that has not been fully articulated is just that without
understanding variability and uncertainty, it is not possible to think clearly about the risk
issues that are central to risk managers" jobs. Risk managers also need understanding in
this area to have a chance to make sensible choices between delaying action for more
research and near term decision-making.
Linkov
[no comment provided]
-------
There are several disadvantages of using PRA that weren't
identified. The disadvantages are important to identify because by
thinking about them, decision makers can learn how to harness the
advantages of PRA. Here are some of the disadvantages:
Toll
Without tight project controls to keep analysts focused on uncertainties that matter
to decisions, PRA can lead to wasteful expenditures. This is of course the flip side of
some of the advantages that the papers identified (e.g., sensitivity analysis can focus
analyses on important uncertainties).
There's no guarantee that the PRA will be considered by regulatory decision
makers. For example, ARARs are based on generic assumptions that, in the context of
site-specific decisions, can be shown through PRA to be overly conservative. In that case
there's a great deal of uncertainty about whether the PRA will be persuasive (and
therefore worth the effort). Analysts and decision makers have to be willing and able to
"play forward" the decision process to establish trust that, and understanding about how,
the PRA will be used.
PRA is incompatible with one-off decision processes. So, for example, if parties
are seeking a final remedy rather than an interim remedy, and want to minimize long-term
monitoring expenses, then a PRA might not be appropriate.
PRA can be used to obfuscate risk management decisions. Particularly in
adversarial settings it might not be possible to rule out the possibility that PRA will be
used that way. That drives up the expected transaction costs and therefore drives down
the expected value of the PRA.
lc. Are there criteria or considerations that were not adequately addressed for
determining whether or not probabilistic methods may be useful to
characterize uncertainty and variability and support EPA decision-making? If
so, please describe such considerations.
Reviewer Comments
The text mentions there is controversy about whether PRA needs a lot of
data or can be conducted no matter what the data availability. In fact,
there is not really controversy, so much as different methods/approaches
with different capabilities. The traditional one-or two-dimensional
assessment based on a frequentist interpretation of probability does
require a lot of empirical information. When a subjectivist interpretation
of probability is allowed, or when a probability-bounding approach is
employed, then one can always do an assessment with whatever data
happens to be convenient. To my mind, this is a critical consideration that
was not really given the attention in the documents that it needs.
-------
Ferson
The documents also neglect to describe the controversy (this time real)
associated with using the subjectivist interpretation of probability. Some
analysts and stakeholders think it inappropriate to use personal belief in
formal analyses conducted by the federal government. While it is clearly
beyond the scope of the documents to review the entire
frequentist/subjectivist debate, it would seem germane and essential to
explain in at least these practical terms what the debate has been about for
the last two hundred years.
Finally, the cost associated with expert elicitations, which are often important in
subjectivist assessments, was not mentioned. This too is a significant consideration in
deciding whether and how to do PRA. Elicitations are not cheap, and are sometimes
quite expensive. People who might want to use expert elicitations should know about this
issue, as well as a few of the other associated limitations (controversy, complexity, etc.).
There is a recent EPA document on expert elicitation that should be cited.
Guiseppi-Elie
The documents appear to be adequate in identifying the technical merits and utility of
PRA to address uncertainty and variability for decision-making. However, much of these
arguments have been known and articulated by the Agency and others for well over 10
years. Indeed, guidance has been in place since 1997. Some obvious impediments to the
use have been discussed in the documents, not the least of these are the perceptions
around the complex nature of the assessments and whether they were needed when staff
were not empowered to make decisions outside the typical conservative norms. However,
probabilistic techniques can be extremely informative and appropriately applied can give
context around uncertainty and variability that can make for more informed decisions.
These techniques encourage the use of all available data and this is a point that the
documents may emphasize better. The development of these documents and the
acknowledgement of the need to encourage the appropriate use of PRA techniques are
laudable. The Agency is to be encouraged to continue along these lines particularly
providing training opportunities and emphasizing successes.
-------
Yes, I think there is not adequate presentation of how uncertainty and variability ideas
factor into risk management choices under different types of enabling statutes with
different mandates for balancing public and environmental health protection goals,
economic efficiency considerations, and the equity/fairness issues among various
involved parties. The most helpful introduction to these issues is a pair of papers I
published in the 1990's: Hattis, D., and Anderson, E. "What Should Be The Implications
Of Uncertainty, Variability, And Inherent 'Biases'/'Conservatism' For Risk Management
Decision Making?" Risk Analysis, Vol. 19, pp. 95-107 (1999). Hattis, D. "Human
Interindividual Variability in Susceptibility to Toxic Effects—From Annoying Detail to a
Central Determinant of Risk" Toxicology Vol. Ill, pp. 5-14, (1996). For example, in the
first of these papers I discuss four different risk management standards, three of which
can be informed by probabilistic information. I will include here an outline of these four
criteria and some discussion of them from the concluding section of the paper: • "Fair
process, open disclosure and, to the extent practicable, voluntary acceptance of risk; •
Equity (fairness) in the distribution of risks in relation to the benefits derived from
accepting those risks;
The greatest possible effectiveness of governmental agencies in using
limited resources to achieve health and safety goals ("Do the very best you can");
and
The principle of "first, do no harm" from medical ethics.
The first of these is primarily qualitative, although as seen above there are
process benefits for improved understanding and communication of both
variability and uncertainty. The latter three criteria can be informed by different
kinds of quantitative variability and uncertainty analyses.
For example, evaluations for the "fairness" criterion are best informed by
analyses that make explicit the distributional implications of alternative hazard
control measures for different affected groups. Variants of older "significant risk"
criteria need to be adapted for this purpose. Minimally, if we are to retain
numerical criteria for "significant risk" as guideposts for risk management, then
these numerical criteria can be more fully defined in terms of these three
dimensions. To be clear in formulating a social objective relating to the equity
(fairness) of a risk distribution, the risk manager needs to decide what fraction of
the population (Y) must be kept below one or more X levels of individual risk with
-------
what level of confidence (Z). One of us has tentatively suggested in earlier work
that if one wishes to focus on a single point in this three dimensional space for
equity purposes, risk managers should choose to focus on a relatively high
percentile in the variability distribution (which could be adapted to cover the size
of the exposed population) but only a moderately high (e.g. 90th-95th fractile)
value in the confidence dimension.
Evaluations for the "do the best you can" (efficiency) criterion, on the other hand,
will more often utilize distributional information to calculate societal aggregate net
benefits from alternative possible uses of specific kinds of limiting resources for
prevention. The final criterion, "first do no harm" adapted from medical ethics0
implies that there should be reasonable confidence that efforts undertaken in the name
of risk control will not do more harm than good. Standards of due care in modern medical
practice require some reasonable degree of evidence that remedies marketed for a
specific medical condition are likely to have a favorable ratio of therapeutic benefits to
risks for the patient. Advocates of regulatory control measures have an obligation to
examine the potential consequences of their social prescriptions with enough breadth
and care to be confident that they are not offering the policy equivalent of "snake oil".'"'.
These suggested criteria should not be seen as a kind of formula to be programmed into
a computer in place of human decision making. Rather, we hope they will contribute to
an evolving language that can (1) accurately represent our advancing technical
understanding of the facts and (2) frankly and compassionately convey our maturing
understanding of the relevant value questions."
A. Johnson "Do No Harm: Axiom of Medical Ethics," in: Philosophical Medical Ethics: Its Nature
and Significance, ed. S. F. Spicker and H. T. Englehart, Jr., (D. Reidel Publishing Co.,
Dordrecht-Holland) 27 (1977).
C. Whipple "Redistributing Risk," Regulation May/June 37 (1985).
The considerations introduced are reasonably complete, but they should be
presented in a more convincing way rather than a list. For more clarity, these
considerations (Managers' Summary, pi 1) should be mapped to the Risk
Manager's objectives. On the negative side of PRA, there might be internal
constraints which would render PRA impossible or
Linkov
decrease significantly the probability of the process going forward (funding constraints,
human resources). Also, the use of resources (time and money) and what they might
imply in terms of objectives (e.g. more time before a remediation decision might lead to
increased health risk during that time) should be considered. On the positive side, using
PRA might help making a better informed decision which would improve the expected
outcomes. This means an implicit decision analysis of the costs of PRA vs. its benefits
has to be made to decide if PRA is worth pursuing.
-------
The previous comments address this charge question. Paraphrasing: • Parties should
consider whether the decision process is one-off (i.e., building step-wise toward a final
decision) or iterative and adaptive. If it's one-off then PRA should be used with caution.
If it's the latter then PRA should be mandatory. • Decision makers should be prepared to
bring in external peer reviewers if and when it becomes apparent that PRA is being used
to obfuscate decisions. • Responsible parties should be wary about investing in PRA until
they receive reasonable assurances that the PRA will be appropriately considered by
regulatory decision makers. • No one should engage in PRA unless and until the PRA is
designed to assess confidence in decisions (as opposed to uncertainty in underlying
processes and data).
Id. Considering risk managers are a major audience for these documents,
especially for the "Managers' Summary" document, what additional
information should be included in the document to aid them in decisions
regarding the application of PRA?
Reviewer
Comments
Ferson
It would be very valuable to managers new to PRA to see concrete examples of the major
graphical and numerical ways that the results of a PRA will be presented. They have to
be able to understand what comes out of the analysis and seeing examples is critical. For
instance, I think it would be helpful to have an additional figure after Figure 3 in the
Managers' Summary depicting a cumulative distribution function, a complementary
cumulative distribution, and a density function together for comparison. There should
likewise be some graphical illustrations to accompany many if not all of the case studies.
Such figures would help to bring the cases to life, and they would go a long way toward
inducting the reader into the discipline. I realize that the authors made a pedagogical
decision not to include such figures, but I think they should reconsider this issue. The
Managers' Summary should also include highly abbreviated explanations of the
controversy of subjectivism and the cost of elicitations. It is fair to let them know the
trouble they take on as well as the benefits they enjoy from adopting PRA.
Guiseppi-Elie
The information and discussion in the "Managers' Summary" is better articulated and
presented than in the main paper. The use of the risk equation and the use of graphics to
illustrate the concepts are more than helpful but needed to help convey what are arguably
-------
complex concepts. It is puzzling why the main document lacks graphics even to the same
level. I would suggest a "roadmap" figure (conceptual model or risk equation or
ecological framework diagram) that can provide context for where and how these
techniques fit into the risk assessment process. Also, I would suggest more continuity
with the text and examples, perhaps with the use of a roadmap/conceptual model figure.
Even though the examples are in an Appendix, the illustrations are useful for in the main
document to provide context. An explicit Figure/discussion on the use of a tiered
approach (in support of Section 2.11) would be helpful in the text (as opposed to the
Appendix). Note that "Tiered" approaches are not defined in either document. There is a
definition of "levels" which suggest the concept of tiered evaluations. I would suggest
specifically adding the term to the glossaries.
Hattis
As I indicated in my response to question lc above, there needs to be a full discussion of
different implications of variability vs uncertainty for risk management choices,
including relationships to different enabling statues, with their rather vague and diverse
mandates for health and environmental protection.
Linkov
An outline and more structure would help. Current documents include a list of questions
in almost arbitrary order. These questions and answers should be structured in topical
section. For example, stated objectives ("present general concepts and principles of PRA,
to describe how PRA can improve basis for Agency decisions, and provide illustration on
how PRA can be used in risk estimation...") naturally call for there sections (1. General
Concepts; 2. PRA use for EPA Decisions, and 3. Applications). This structure would
provide a way for managers to search for specific answers. A brief section on Decision
Analysis and how it is interrelated with PRA should be added. P6 of the White paper is a
good start, although too condensed for a reader with no background training to
understand (decision analysis and value of information are mentioned in one paragraph
without much explanation). One of the common problems mentioned in the White Paper
(pi 6) is the "lack of understanding of how to incorporate the results of probabilistic
analyses into decision making and how to establish action levels based on the scope
of the assessment." This is of primary concern to the risk managers vs. risk assessors
and has not been addressed adequately.
-------
I was dissatisfied with the managers' summary because it reads like past documents that
have been ineffectual. Managers would be better advised to go back to Finkel (1990)
because it covered the same ground, plus presenting some really nice information about
communicating uncertainty. EPA should consider rewriting the managers' summary as a
vision statement. It could start by articulating why PRA should be a good idea (as it does
now), but it should also acknowledge that after 20 years PRA hasn't lived up to its
promise, describe barriers to widespread adoption, and identify institutional changes that
could lead to fulfilling the promise of PRA (in say the next ten years).
2. The PRA papers were designed to address the use of these techniques in both
human health and ecological risk assessments.
Have the current documents adequately addressed the unique issues, if any, associated
with application of PRA to both human health effects and ecological risk? If not, what
additional information/case studies should be presented specific to human or ecological
risk assessment?
Reviewer
Comments
Ferson
I do not believe that there are issues unique to one side. The texts focus mostly on human
health examples, but have attentively included a reasonable representation of ecological
examples as well. The same is true for the case studies.
Guiseppi-Elie
Both human health effects and ecological risk assessments are covered in the document
but consistent with current practice in these areas, the discussion and examples are
limited. For human health effects, the trend is towards ranges (at a minimum) rather than
single point estimates of "toxicity" e.g., EPA proposal for toxicity criteria for
trichloroethylene (EPA 2001; EPA/600/P-01/002A). However, given a choice of a range,
assessors tend to default to the conservative end of the range. A specific discussion on
this aspect is warranted. A case study to illustrate would likewise be useful. On the
discussion on ecological risk, a specific discussion question that addresses "stressors"
and the role of PRA in variability and uncertainty for these parameters (which are often
more so than for human health) is warranted. The following may be a useful reference
for case studies: Landis, W. G. 2005. Regional Scale Ecological Risk Assessment Using
the Relative Risk Model. CRC Press Boca Raton pp 286. Edited Book.
-------
Hattis
Most of the current discussion is oriented to human health, and within human health to
the exposure portions of the pathway to harm. There needs to be much more discussion
of the issues involved and implications of the NRC (2009) proposed redefinition of the
RfD, and the capability to estimate risks of adverse effects under alternative poly choices
for population exposures both above and below current and redefined RfDs. I tihink the
current document is remiss in not mentioning these major issues except in the context of
a 2007 recommendation by the SAB,
Linkov
Appendix 4 is actually one of the best and most important portions of the document.
Applications and wide use of PRA should be featured in the main body of the paper. To
strengthen the case, I would recommend adding references to applications of PRA
reported in open literature. Review papers (e.g., Lester et al, 2006) may be especially
valuable to illustrate a wide use of PRA in the field
Toll
The document (specifically Case Study 13) identifies what I would consider the big three
uncertainty issues in ecological risk assessment, namely, 1) deriving risk-based effect
thresholds, 2) linking organism-level measurement endpoints and population-level
assessment endpoints and 3) accounting for the effects of landscape on exposure and
risk, but it's not possible to adequately address those issues in the format of the
current documents.
On the human health side, I think it's important, as I mentioned in my response to
charge question 1(a), to help readers recognize that not all uncertainties should
be analyzed in a PRA. The example I used is that exposure variability in a human
population doesn't necessarily belong in a PRA if the decision's already been
made to base risk management choices on risk to a relatively highly exposed
subpopulation. In that case, ingestion rate might better be treated as a decision
variable than as a random variable, for example.
3. In the Manager's Paper, EPA recognized the variety of PRA tools that may be
used in decision making. The document states: "This document does not
prescribe a specific approach but, rather, describes the various stages and
aspects of an assessment or decision process in which probabilistic assessment
tools may add value. The concept of iterative or tiered analyses to address this
continuum is a widely accepted goal in risk assessment, and the same principle
applies to PRA as well. There is a wide range of methods and approaches to
PRA, of varying complexity and rigor, which can be applied for different
purposes ranging from basic sensitivity analysis to integrated analysis of
variability and uncertainty. The goal is to choose a level of detail and
refinement for an analysis appropriate to the overall objectives of the decision
and the types of available data and analyses needed to support decisions."
Based on your review of the document:
-------
3a. Please comment on the completeness of the discussion of the tiered approach for
decision making regarding the application of PRA. What, if any, additional
refinements to the tiered approach outlined in the document can you recommend to
further support management decisions regarding use of PRA?
Reviewer
Comments
Ferson
The discussion about the tiered approach is standard but perhaps a bit shallow. The use
of probabilistic methods mostly in higher tiers is explained. In fact, there is not strict
limitation of PRA to higher tiers, and the counterexamples might be interesting too.
Guiseppi-Elie
I agree with the positions taken on using a tiered approach. In the Manager's Summary,
the concept is appropriately articulated. The discussion could be enhanced with the use
of a graphic to illustrate the concept. I would suggest a discussion on using all available
data. Also, while starting as simple as possible to make the decision is appropriate, this
should not preclude the use of a "higher" tier as a starting point if the data are available
and the "stake" suggests such. In effect, it may appropriate, for example, to start with a
1-D Monte Carlo simulation, if the data is available, the decision worthy of the effort
and the ability to communicate from range rather than a "worst-case" single point more
relevant.
Hattis
In the Manager's Paper, EPA recognized the variety of PRA tools that may be used in
decision making. The document states: "This document does not prescribe a specific
approach but, rather, describes the various stages and aspects of an assessment or
decision process in which probabilistic assessment tools may add value. The concept of
iterative or tiered analyses to address this continuum is a widely accepted goal in risk
assessment, and the same principle applies to PRA as well. There is a wide range of
methods and approaches to PRA, of varying complexity and rigor, which can be applied
for different purposes ranging from basic sensitivity analysis to integrated analysis of
variability and uncertainty. The goal is to choose a level of detail and refinement for an
analysis appropriate to the overall objectives of the decision and the types of available
data and analyses needed to support decisions."
-------
First a quibble. In the first line of the question above, I would prefer to substitute "used
to inform decision making" in place of "used in decision making." More generally, I
think that current tiering approaches have led to (1) neglect of even very basic analysis
of uncertainty and variability when it would be efficient to do so in early stages of
screening-type analyses; often it seems that 80% of the work on a risk issue has been
done before probabilistic analysis is even started, and (2) neglect of the potential of
structured probabilistic studies to help calibrate the screening procedures used at early
stages of relatively data-poor choices. At the moment it seems that these screening
procedures are mostly designed by seat-of-the-pants, what-seems-reasonable
techniques. This can lead to implicit unexamined policy choices that are never
effectively analyzed for management review. 3a. Please comment on the completeness
of the discussion of the tiered approach for decision making regarding the application
of PRA. What, if any, additional refinements to the tiered approach outlined in the
document can you recommend to further support management decisions regarding use
of PRA? Usual conception of tiers is wrong; in fact PRA is more not less valueable for
more data poor situations
Linkov
As is, the document does not "describe various stages and aspects of an assessment or
decision process in which probabilistic assessment tools may add value." It repeats
description of the Tiered process introduced in RAGS PRA document. A decision
process needs to be introduced. The decisions should relate to actual EPA decisions
(e.g., remedy selection) vs. the risk assessment process (e.g., do I move from Tier 1 to
Tier 2 assessment). The risk assessment process should be related explicitly to manager
needs and not be considered in isolation, as it is done now. In addition, I recommend
adding references to examples illustrating the use of these possible RA approaches in
the Case studies of the White Paper in specific decision contexts.
Toll
The concept of tiering is important and the discussion that's presented is appropriate. It
should be expanded though to make it easier for risk managers to set aside
COC-receptor combinations that aren't driving risks. We run into situations in
(ecological) risk assessments where PRA is indicated for the risk driver(s), but we're
carrying along other COC-receptor combinations that really should be set aside until
after we've decided what to do about the risk driver(s), at which point the effectiveness
of the remedy for the non-risk drivers would be assessed. In our experience risk
managers can be uncomfortable with this approach and we end up spending
disproportionate effort on parallel treatment of non-risk drivers.
3b. Please comment on the use of a tiered approach to when to use PRA considering
management considerations of cost, time and resources.
-------
Reviewer
Comments
Ferson
Although I don't disagree with the points made in the documents on the subject, it is
possible to employ fully probabilistic methods in the context of screening. This can be
done conveniently with probability bounding. The idea is to bound probability values or
probability distributions. When the bounds are far apart, the analysis expresses
uncertainty about what the true risks are in an appropriately conservative way. As the
bounds converge, the assessment is equivalent to a traditional probabilistic analysis. The
tiering consists of revisions of the bounds with additional or more site-specific data, but
does not require setting up a new assessment framework. Such a probability bounding
approach has been used for both human health and ecological risk assessments at two
Superfund sites (Housatonic in Massachusetts and Calcasieu in Louisiana). There are
commercial and freeware software tools supporting this approach, which is compatible
with both frequentist and Bayesian interpretations of probability.
Guiseppi-Elie
I agree with the position on the use of cost, time and resources. However, I would add
that the relevance of the decision is probably a key consideration. Recall the use of
double loop Monte Carlo simulations for the Hazardous Waste Identification Rule, i.e.,
for major regulatory decisions, the use of the appropriate techniques should take
precedence (while balancing cost, time and resources) in getting the appropriate
decision.
Hattis
As indicated above, I think the usual consignment of PRA only to rare and highly data
rich situations is far from ideal. Because it can add information from other, parallel
cases, PRA can actually be more helpful for data-poor situations. For extensive
discussion, see section 3 of my recent conference paper: Air Toxics, report/conference
paper by Abt Associates, Inc. to the U.S. Environmental Protection Agency under
EP-W-05-022 WA 3-80, Final, March 2010.
Linkov
See response above. The decision context needs to be introduced. It should be noted that
even though the decision to use PRA could be based on the considerations listed above,
a value of information approach needs to be implemented to address this issue (see
White Paper p7, section 1.4., paragraph 2)
Toll
The tiering concept is important. It's not realistic to set hard and fast criteria for when to
use PRA because it depends to great extent on project team dynamics, project process
and schedule, and what's at stake. The tiered approach should be refined to make it
easier to opt into a PRA for risk drivers while opting out for other COC-receptor pairs
(passing them forward for what could be a cursory remedy effectiveness evaluation
once a remedy's been selected for the risk drivers).
4. The papers are intended to communicate information to EPA risk
assessors and managers.
-------
4a. Is each document written at the appropriate level of detail for the intended audience
and length of document? Specifically, is too much or too little detail presented for the
respective audiences? Please provide suggestions on how to address any such
shortcomings.
Reviewer
Comments
Ferson
There is appropriate detail in the Managers' Summary document. The figures are quite
nice, although I have suggestions for improving them. There is also appropriate detail in
the first part of the main document, although it seems not that much deeper than the
Managers' Summary. Some more detail, and maybe a figure or two, about how PRAs
actually work would be helpful in Appendix A. It is a bit of a disappointment to read
17+24 pages, finally be told to go look in Ang&Tang, Cullen&Frey, or
Morgan&Henrion for any hint about what PRA is doing. The reader should have enough
of an idea of how probability distributions are slammed together to make sense of the
case studies. Section 2.5 is remarkably terse given the centrality of the question in the
section header (especially considering the first paragraph more properly belongs in the
previous section). This would be the section that deserves beefing up if any does. Would
it be possible to list examples of how exactly a PRA communicates a more 'robust
representation of risks', perhaps in examples across the spectrum of EPA's activities? I
wouldn't want to increase the redundance in the document, so nevermind this comment if
the response to it is to replicate text or ideas from elsewhere in the document to this
section. I suppose the case studies are supposed to serve this purpose, but they're out of
place if the section title is to be justified by its content.
Guiseppi-Elie
The Manager's Summary is clear and has the appropriate level of detail. I think that the
use of graphics to highlight points is helpful. The use of a roadmap type figure to further
illustrate where the PRA fits in the process would be additive. This type of figure could
also help with context and continuity in the documents. The main paper would benefit
from review by a single technical editor. The style and structure of the discussion is not
consistent. While the intent is clear, some of the discussion is not easily understood. (For
example, the first paragraph of the Executive Summary ends with a caveat that seems out
of place). I would highly recommend inclusion of graphics (along the lines of
suggestions for the Manager's Summary) to aid in readability. While the question format
works for the Manager's Summary, the style in the main paper is distracting. A more
typical format (which could include some questions) with some call-out boxes to
highlight topics is favored. Also, as suggested earlier, some continuity between the text
and the examples would be useful. Assuming that the document is likely to be available
on line or at least in a link-active pdf format, the ability to move to the relevant examples
would be helpful.
-------
Hattis
I think the management paper in particular needs much more detail on how uncertainty
and variability issues relate to the risk management criteria implemented by different
EPA offices under different enabling statutes. Additionally, I think the technical
definitions of uncertainty and variability need to be covered very early in the
management summary. Withouth better developing these ideas, the whole effort to
demystify PRA and inform the managers that this is important for their work is likely for
naught.
Linkov
The current drafts repeat information already presented in previous EPA documents (e.g.,
PRA RAGS). As stated in general comments, we need to refocus these documents and
present PRA within an EPA decision framework, and show that this is an established and
validated tool. Not enough detail on how to incorporate PRA in the decisions (in both
documents), i.e. the link between risk assessment and risk management. This is the main
criticism for the two documents.
Toll
My issues with these documents have to do with scope, not length or level of detail. The
papers focused on the mechanics of PRA from the perspectives of analysts and decision
makers but the mechanics of PRA are already well documented elsewhere. After 20
years PRA hasn't lived up to its promise because it's inherently suited to iterative,
adaptive management, whereas policies and regulations tend to artificially
compartmentalize environmental problems and drive us toward one-off solutions that fit
within the mandate of the governing rules. A second concern that we encounter in
site-specific risk assessments is high uncertainty about whether and how a PRA will be
used by decision makers, and those trying to influence decisions. These are the problems
that these documents need to focus on if the objective is to promote the appropriate use
of PRA.
4b. Is the level of detail of information in the case studies adequate, or is more
information needed? Please provide specific suggestions on how to improve the case
studies, if you have any.
Reviewer
Comments
Ferson
I don't think the level of detail is adequate in the case studies. Some, especially 2, were
hard to follow. They would be improved by graphical depictions of the difference that
PRA made in the assessment, or the results it produced, as shown in case studies #5 and
#10. It would be useful to have slightly fuller outlines of the models employed. For
instance, in case study #1, how many inputs were there, out of which six were critical?
Guiseppi-Elie
While not the same in each instance, the level of detail is appropriate. Links could be
provided to the main references/documents. It would be helpful to link text and
examples.
-------
Hattis
[no comment provided]
The case studies are the best portions of the document; the level
of detail in the case studies is just right to understand how the end
probability distributions were obtained. However, what lacks is the
"So what?" component. A probability distribution has been
obtained, so what? In some cases, it has helped solve a specific
decision problem. In some other cases, the information obtained
can be used for helping make a range of decisions. What are these
decision problems? What are the alternative courses of action?
How has this sophisticated PRA helped in making better
decisions? Or how can it help in the future? Of all the case studies,
only case studies 9 and 15 address this issue.
Linkov
I found the case study summaries to be useful and I'd like to see more of
them, particularly for ecological PRAs. Enough information is presented
to get the reader started if (s)he wants to learn more about a particular
use of PRA. Completing the case studies would require far more detail
than could reasonably be expected in these documents.
Toll
4c. Please discuss whether, and how, the White Paper could be improved to help the
reader to better understand how PRA can help address and communicate variability
and uncertainty.
Reviewer
Comments
Ferson
Again, I think pictures of outputs would serve that purpose.
-------
Guiseppi-Elie
Graphical representations (particularly in the main document) are needed.
Hattis
I think some basic succinct phrases could help communicate. For example, "variability
involves real differences among people or things in the real world that affect risks;
uncertainty is mostly in your head, involving imperfection of our information about the
values of parameters related to risks. We are, of course generally uncertain about the
amount of variability in parameters affecting risks."
Linkov
See General Comments (in the Additional Comments section)
Toll
I would recommend against adding a primer on communicating variability and
uncertainty; this is a topic that's much better left to practitioners faced with real risk
assessment problems. Case studies on successfully communicating variability and
uncertainty would be a good way to improve the papers.
4d. Are the citations and references sufficient, or are there critical references which
need to be added? If so, please provide those citations and their relevance to the
papers.
Reviewer
Comments
Ferson
The citations are relatively few and, as mentioned above, unbalanced. I don't think any
reference is 'critical', but many might be helpful. I thought, for instance, Vick (2002),
and Bernardini and Tonon (2010) would be useful. Some reference about
dependence/correlation issues would also be appropriate. Our old report (Ferson et al.
2004) might work, but there are others too, such as perhaps one of Roger Cooke's
papers. References Vick, S.G. 2002. Degrees of Belief: Subjective Probability and
Engineering Judgment. ASCE Press. Bernardini, A., and F. Tonon. 2010. Bounding
Uncertainty in Civil Engineering: Theoretical Background. Springer. Ferson, S., R.
Nelsen, J. Hajagos, D. Berleant, J. Zhang, W.T. Tucker, L. Ginzburg and W.L.
Oberkampf. 2004. Dependence in Probabilistic Modeling, Dempster-Shafer Theory, and
Probability Bounds Analysis. Sandia National Laboratories, SAND20043072,
Albuquerque, NM. www.ramas.com/depend.pdf
Guiseppi-Elie
While the text is short on citations, the reference lists seem adequate. A few other
references are included in the comments on relative risk models.
-------
Hattis
As mentioned earlier, the references badly need updating to include nearly 2 years of
additional material (more recent than 2008). Above in my response to charge question #1
I have cited some papers of mine that could be helpful.
Linkov
The reference list is not adequate. Dr. Frey co-authored more than half of the cited
papers. Even though he is one of the undisputed leaders in the field, a substantial body of
work has been done by others and should be represented. Significant gaps include
publications on application case studies which are crucial in convincing managers that
PRA has been validated by now.
Toll
I've cited seven documents, six of which (i.e., all except Morgan & Henrion (1990))
weren't cited in the reports. The comments where they're cited provide the relevance to
the papers. Literature cited Dakins ME, Toll JE, Small MJ, Brand KP. 1996. Risk-based
environmental remediation: Bayesian Monte Carlo analysis and the expected value of
sample information. Risk Anal 16(l):67-79. Finkel AM. 1990. Confronting uncertainty
in risk management: a guide for decision-makers. Center for Risk Management,
Resources for the future, Washington, DC.
McConnell KE. 1997. Using cost-benefit analysis in the management of
contaminated sediments (Appendix E). In: Contaminated sediments in
ports and waterways: cleanup strategies and technologies. National
Academies Press, Washington, DC, pp. 239-256.
Morgan MG, Henrion M. 1990. Uncertainty: a guide to dealing with uncertainty in
quantitative risk and policy analysis. Cambridge University Press,
Cambridge, UK.
Toll J, Pavlou S, Lee D, Zaragosa L, Shelley P. 1997. Using decision analysis
in the management of contaminated sediments (Appendix E). In:
Contaminated sediments in ports and waterways: cleanup strategies and
technologies. National Academies Press, Washington, DC, pp. 257-284.
Toll JE. 1999. Elements of environmental problem-solving. Hum Ecol Risk
Assess 5(2):275-280.
Wilson JD. 2000. Risk management cannot abide uncertainty. In: Exner JH,
Trehy ML, eds, Symposia papers presented before the Division of
Environmental Chemistry, August 20-24, 2000. pp. 268-269 in Preprints of
Extended Abstracts, vol. 40 no. 2. American Chemical Society, Washington,
DC, [Cited 4/24/10.] Available from:
-------
http://www.envirofacs.org/Pre-printsA/ol%2040%20No%202/Risk/p15.PDF.
Additional Reviewer Comments
ADDITIONAL COMMENTS - SCOTT FERSON
I also have many comments on minor matters including typographical errors, formatting
inconsistencies, and pedagogical suggestions for the texts, such as misuse of the word
'percentile', the gibberish phrase 'cumulative density function' and the following two
points of vocabulary. Firstly, in Figure 1 on page 19, it is absurd to call the confidence
bounds 'lower' and 'upper' if the upper is below the lower; call them 'left' and 'right'.
Secondly, it is perverse to adopt the word 'evaluation' as a code for what the rest of the
scientific world universally calls 'validation'. Doing so would unnecessarily isolate EPA
and engender confusion.
ADDITIONAL COMMENTS - DALE HATTIS
Below I make some more detailed comments on specific aspects of the two documents:
Manager's summary
Points under why should I care emphasize external technical recommendation. A
better approach might emphasize relevance to choices that managers need to frame
to improve implementation of mandates in legislation. A particular major decision
that may need to be faced in the future is the redefinition of the RfD in terms of a
"risk specific dose" as recently recommended by an NRC report (2009)—e.g.
greater than 95% confidence that the incidence of mild adverse effects will be less
than 1/100,000, as offered in my "Straw Man" proposal for a probabilistic system
for assessing non-cancer healthy effects thought to occur by individual
threshold-type mechanisms.
"The enhanced use of PRA and characterization of uncertainty would respond to outside
recommendations and potentially enhance the overall transparency and quality of EPA
assessments. These approaches would provide additional tools to address specific
challenges faced by managers and improve confidence in Agency decisions. Specifically,
PRA can inform decision makers about specific segments of the population at risk, not
just the mean (average) or extreme values. A PRA can also confirm or support the
conclusions of a deterministic risk estimate. Having this information can be important to
risk managers if a different decision might be made when the upper or lower ends of the
range of estimated exposures, doses, or risks are used."
What is PRA? How does it compare with current practice?
"A basic characteristic of PRA is that it does not generate a single point estimate but
rather produces a range and likelihood that a particular exposure, dose, or effect will
occur. Probabilistic risk assessment, in its simplest form, is a group of statistical
-------
techniques that allow analysis of variability and uncertainty to be incorporated into
exposure and/or risk assessments. As mentioned above, this kind of information can be
critical in risk management decisions especially if one were interested in a specific
portion of the population and the likelihood of exposure or risk is known."
Here the concepts of variability and uncertainty are mentioned without clear definitions,
and most importantly introduction about how managers should care differently about
them. Mention critical idea that we should care about protecting all or nearly all the real
people (variability) with reasonably high confidence (uncertainty).
An important defect in current use of PRA is to regard it as only an ultimate tier part
of an analysis. In practice this means that probabilistic ideas are not addressed when
the analyst initially process available data, but are an afterthought after 80% of the
effort on a project has already been expended. Propagation of uncertainties by
defining central vs "high end" point estimates has basic problems compared to an ab
initio set of distributional analyses. Another problem is that the 4-step presentation
does not clearly separate variability from uncertainty considerations. This is an
obstacle to clear thinking.
Variability and uncertainty only start to be defined on page 4 of the manager's
summary. This should come much earlier and be framed not just in technical terms but
in terms of direct relevance to management goals.
Figure 1—double headed arrows are confusing; why not just single headed arrows
beginning at the source of emissions and ending with health outcome? One could also
note that each box represents places where measurements can be made and each arrow
represents places where interventions can be done. Possibly use the following figure
from my old biomarkers.
Responsiveness
Escape
Environmental
Complex
Control
i i ai iopui i
Behavior
—*
/
Dose Response
Relationships
Escape Environmental Complex
Response Physical Relationships
Transport
Dose
Control
Behavior
Economic Discharge Presence AU ^Activity Effects
53 Absorbed 3 Dose
-------
7^
. A . . Toxicology
Engineering Air Dispersion
Dispersion through Molecular biology
groundwater, surface
water
The following points are in some cases confusing in that they obscure the variability
vs uncertainty dichotomy that has just been explained:
How representative or conservative is the estimate, (e.g., what is
the variability around an estimate)?
This uses "variability" when uncertainty is the relevant concept.
What are the major gaps in knowledge, and what are the major
assumptions used in the assessment? How reasonable are the assumptions?
This is a rather qualitative inventory of uncertainties. It fails to focus attention on
quantification opportunities by asking questions like—under reasonably
plausible alternative assumptions, how different would the risk/benefit
assessments be?
Would my decision be different if the data were different? Would
additional data collection and research likely lead to a different
decision? How long will it take to collect the information, how much
would it cost, and would the resulting decision be significantly
different?
This appropriately introduces the value of information idea, but it would be good to
further
develop the concept of value of information analysis.
Will the use of additional resources, such as a probabilistic approach, impact the
decision making in a timely manner (i.e., better characterize uncertainties, better identify
variability, impact timelines, etc.)?
What are the liabilities/consequences of making a decision under the current level
of knowledge and uncertainty?
Now the issue is switched to variability questions
What is the percentile of the population to be protected?
How do the different alternative decision choices and the interpretation of
-------
uncertainty and variability impact the risks and benefits to the target population under
alternative regulatory choices.
Characterize uncertainty in estimates (what is the degree of confidence in the
estimate?). That is, could the prediction be off by a factor of 2, a factor of 10, or a factor
of 1,000?
The phrasing here implies, incorrectly that it is possible to specify ranges in
absolute terms without making reference to specific numerical confidence limits
• Identify the critical parameters and assumptions which most impact or influence a
decision and the risk assessment;
My opintion is that "impact" as a verb is never preferable to the plain language
"affect".
Identify the "tipping points" where the decision option chosen would be different
if the risk
estimates were different, or a different assumption were valid closer to the truth
Estimate the likelihood that critical data values exist or the validity of
assumptions;
it is not clear to me what is meant by "critical data values exist"? Are you talking
about
"thresholds"? If not what?
• Estimate the degree of confidence in a particular decision and/or the likelihood of
specific
decision errors
This assumes that specific decisions can be classified as "erroneous" but this
implies a
defined numerical criterion for a specific decision, which is almost never the case
Estimate, (in conjunction with other techniques, such as sensitivity analysis and
value of information) the possibility of alternative outcomes with additional information,
or estimate trade-offs related to different risks or risk management decisions;
Identify impact of additional information on decision making considering the cost
and time to obtain the information and resulting change in decision (that is, value of
information).?
This is a laudable goal which is rarely achieved in practical analyses
Explicitly define the exposures for various sectors of the population (whom are we
-------
trying to protect?) That is, will the regulatory action keep 50% of the population, 90%
of the population 99.9% or some other fraction of the population below a specified
exposure, dose, or risk target?
Good; but with what confidence/uncertainty are these predictions made?
Provide information including the variability in the exposures among the population, and
information on the percentile of the population that is being evaluated in the risk
assessment (i.e., people who consume a glass of water/ day or people who consume a
gallon of water/day). This information is helpful in addressing comments:
o from the regulatory community on conservatism of EPA's risk assessments;
o from the community regarding concerns whether their particular exposures were
assessed in the risk assessment,
o about whom or what is being protected by a risk management action, and
o whether and what additional research may be needed to reduce uncertainty.
OK
Figure 3—generally good, but it should not imply that the median and the average
are the same thing. The arithmetic mean should be explicitly identified as the
measure most needed for economic juxtapositions of costs and benefits, serving
economic efficiency goals for management actions. High percentile estimates of
exposure and risk in the variability dimension should be identified as helpful for
addressing equity/fairness issues. Is an unreasonable burden being imposed on
some small fraction of the population under one policy choice or another?
I would recommend creating a similar figure to illustrate the uses of
quantitative characterization of uncertainty.
Does PRA require more data than conventional approaches?
In general, PRA requires more data than conventional approaches because distributions of
values rather than single values are used. How much more data is required is often the
topic of debate in the technical community. Minimum data needs vary depending on the
analytical approach used; empirical-based (observational or frequentist) methods have
significant data requirements compared to so called subjective methods. However, some
of the data that would be applied in a frequentist approach may already be available as
part of the underlying data set used in standard deterministic analyses. As a result, PRA
can be applied in most cases, as long as methods used are appropriate for the available
body of evidence and data.
It is important to communicate that probabilistic techniques can inform more data
poor situations if the effort is made to build databases of putatively analogous
cases. For example imagine there is no toxicity information available for a
particular chemical but one has a database of potencies of a large number of other
chemicals, or other chemicals in the same category of chemical structure or
putative mode of action for prouding the same effect. In that case the untested
-------
chemical can be viewed as a random draw from the previously studied cases.
Definitions section at the end:
If you are going to define frequentist probability you should also include the
Bayesian, particularly as this is often a more useful framework for decision
making.
Larger More Technical Document on Using Probabilistic Methods
Executive Summary
Uses variability and uncertainty terms without first defining/explaining them Executive
summary defines goal as helping managers use allegedly preexisting tools; proably
should emphasize that these ideas can help managers reframe risk management goals of
the agency in ways that reflect clearer thinking about the "facts" related to our
information about environmental and ecological hazards, and foster further development
of technical approaches and relevant data bases as needed to achieve better faster
understanding needed to set priorities and achieve better risk management results for both
data rich and data poor hazards. By emphasizing the availability of the tools it
inadvertently represents the probabilistic tools as currently available off-the-shelf for
immediate application. While there has been substantial development of off the shelf
tools, there are also needs for appreciable further development, and I think it is only fair
to disclose this to the managers. This can first be done, I think in connection with the
basic descriptions of variability and uncertainty in sections 2.1.1 and 2.1.2 respectively,
although it would be good to allude to it earlier. The fact is, conventional calculations of
variability based on observations in a data set, such as a standard deviation, tend to
overstate the real variantion that affects differential risks because any set of observations
will include some measurement error. The presence of the measurement error means that
the observations are more spread out among people than the underlying things being
measured. On the other hand, conventional measures of uncertainty, such as standard
errors in mean values of data generally understate real uncertainty because they are only
based on fluctuations observable within the set of data values—ignoring often important
but not directly measurable sources of systematic error. Sources of systematic error
include miscalibration of instruments and imperfections in the representative ness of
samples for the populations or (for site measurements) places where contamination is
likely to exist.
2. 1—what are variability and uncertainty and how are they relevant to decision-making
2.1.1—end of paragraph—transfer "or toxic response" after the parentheses and add,
(e.g., differences among people in the internal dose needed to produce a specific amount
of neurological impairment). Sections 2.4 and 2.6 seem to end the review with a 2007
NRC report and a 2007 EPA Science Advisory Board recommendation for redefinition of
the RfD in probabilistic terms. This must be updated to include the latest
-------
recommendations (NRC 2009) for redefinition of the RfD as a risk specific dose, and
facilitate inclusion of traditional threshold-type toxicity effects in benefits estimates for
regulatory impact analyses. The reference section and bibliography (pp 36-41) do not
appear to include any citations more recent than 2008, also does not cite some of my
work that I would suggest is relevant:
In 2.9 and 2.10 there should be some emphasis on the idea that building databases of
putatively analogous cases can help substitute for case-specific data where these are
unavailable. See my recent conferenece paper (attached). Another issue arises in
2.11—probabilistic analyses can inform decisionmakers about how to test and
recalibrate, if necessary, the default procedures used for common "screening" analyses
for routine and data-poor situations.
ADDITIONAL COMMENTS - IGOR LINKOV
GENERAL COMMENTS
The US EPA has been on forefront in funding research and developing guidance
document for the use of Probabilistic Risk Assessment in environmental and stakeholder
settings. Even though a significant attention to PRA took place in late 1990s and
culminated in the development and adopting the US EPA RAGS guidance for PRA in
2001, PRA use has been still limited. Given the continuing pressure from EPA SAB,
NRC and peer reviewers as well as dramatic decrease in cost of computational resources
required for Monte-Carlo simulations, the fact of its limited use is very surprising and
requires immediate intervention. The US EPA White paper and Managers summary on
"Using Probabilistic Methods to Enhance the Role of Risk Analysis in Decision Making"
is an important step in this direction: the lack of PRA applications is a likely result of
misconceptions about PRA and potential lack of trust by managers in its usefulness. This
is why the focus of this document is extremely important - it should address risk
managers concerns and provide a solid basis for PRA acceptance by the stakeholder
community at large.
Given the availability of very thorough EPA RAGS PRA guidelines and multiple books
and papers, it is important to focus the White Paper and Managers Summary on specific
concerns of user community. I think the lack of PRA applications results from: (i)
manager's requirement to see PRA as validated and easily applied methodology; and (ii)
needs to clearly integrate PRA in specific decision contexts. Unfortunately both the
white paper and Managers Summary in its current form just summarize previously
published documents and do not address these potential concerns. Nevertheless, the
relevant information is presented in the document and I think it will take just a little to
refocus it.
We need to stop presenting PRA as a "new approach" and compare it with "current
practice" (e.g., p. 2 of Manager's summary). PRA is not new, it should be presented
-------
as a robust and established tool with multiple applications done already and with
many new coming in the pipeline. The very last appendix (Appendix D) should be a
Key Focus of presentation.
Even though relevance of PRA to decision making is highlighted in multiple places in the
report, including the title, the document does very little in linking PRA with decision.
Even though multiple "buzzwords" are used (such as value of information, risk informed
decision making, decision analysis), the documents do nothing new as compared to
previously published papers to make this link more transparent and useful. Many
scientists are currently working on linking risk assessment with different
decision-analytical approaches including multi-criteria decision analysis, Bayesian belief
networks, value of information analysis, etc. A clear link to these techniques is essential
to communicate in new EPA documents.
MISCELLANEOUS COMMENTS
Both documents contain Glossaries that are very similar. It does not make sense to repeat
them.
Glossary entries are very heterogeneous and have different styles. Some of entries are
very brief, others are lengthy. Some are just definitions, others read like portion of
textbook chapters.
Both documents include long lists of unstructured and often overlapping questions and
answers. I do not see a point in repeating these. Q&A should be different in two
documents or at least should be focused on different issues.
Q&A are often in different style. Some of answers are open-ended (e.g., "Below we
describe.." and this "below" never happens), other seem to be lifted from textbook-type
of material and are not to the point. These all need to be harmonized.
p. 2 "Why should I care..this section is written with a clear human health (HH)
perspective, ERA should be added.
p.2 "What is PRA..PRA should not be presented as something new and
contrasted with "current practice." It is current practice.
p. 2, last sentence - very confusing and focused on HH
p.3 , top section - it is good to introduce decision analysis here
p3. Middle - "Below we describe... " Where it is described? What does
"below..mean in Q&A settings?
p.3 last question. The question is not answered, it has a vague wording that it will
be done in the following sections...
p. 4, last question. The focus is on variability, why is uncertainty not discussed in
detail? Figure 2 - Why do we need to have a frame including all the information that is
-------
not important?
p. 6, first question. The paragraph is lifted from somewhere without revision. By
contrast with what? Model uncertainty has not been discussed before.
p.6, second question. Recent research shows that people make better decisions under
uncertainty when uncertainty is represented in the right way.
p. 8 "PRA helps..." PRA does not help in identifying alternatives available to decision
maker. PRA can help in evaluation of these alternatives. DA should be introduced as a
way to identify alternatives.
P8 "What questions can PRA address". The diagram is misleading, as the graph and its
explanations show more how PRA results can be interpreted and what they mean. It
would be more adequate in the "Communication of PRA Results to the Manager and
Community. Does presentation of results matter?" section.
p. 10, first line. Hudson RA includes treatment of both uncertainty and variability.
p. 10. Add date to PRA RAGS reference.
p. 11, The bulleted list on the top is very confusing.
Peer Review Workshop of EPA's Draft
Report, Using Probabilistic Methods to
Enhance the Role of Risk Analysis in
DecisionMaking with Case Study
Examples
Sheraton Crystal City Hotel
APPENDIX D
OBSERVERS
United States
Environmental Protection Agency
Office of the Science Advisor, Risk Assessment Forum
United States Environmental Protection Agency Office of the Science Advisor,
Risk Assessment Forum
-------
Arlington, VA
May 6, 2010
Final List of Observers
Ted Berner
Statistician
U.S. Environmental Protection Agency 1200 Pennsylvania Avenue, NW 8601P
Washington, DC 20460 7033478583 Email: berner.ted@epa.gov
Katie Boronow (writer)
ERG 2300 Wilson Boulevard Suite 350 Arlington, VA 22201 7038418389 Email:
katie. boronow@erg. co m
Jan Connery (ERG Facilitator)
ERG 110 Hartwell Avenue Lexington, MA 02421 7816747322 Email:
jan.conneryi@erg.com
Julie Fitzpatrick
Risk Assessment Forum Office of the Science Advisor
U.S. Environmental Protection Agency 1200 Pennsylvania Avenue, NW Washington, DC
20460 2025644212 Email: fitzpatrick.julie@epa.gov
Kathryn Gallagher
Executive Director, RAF
U.S. Environmental Protection Agency 1200 Pennsylvania Avenue, NW 8105R
Washington, DC 22202 2025641398 Email: gallagher.kathryn@epa.gov
Mary Greene
Deputy Director Office of the Science Advisor
U.S. Environmental Protection Agency 1200 Pennsylvania Avenue, NW
Washington, DC 20460 2025647966
Maria Hegstad
Associate Editor Inside EPA's Risk Policy Report 1919 South Eads Street Arlington, VA
22202 Email: mhegstad@iwpnews.com
Bob Hetes
Acting Director, Research Planning and Coordination National Health and Environmental
Effects Research Laboratory Office of Research & Development
U.S. Environmental Protection Agency 109 T.W. Alexander Drive B30502
Research Triangle Park, NC 27711 9195411589 Email: hetes.bob@epa.gov
Michael Messner
Standards and Risk Management Division
U.S. Environmental Protection Agency 1200 Pennsylvania Avenue, NW 4607M
Washington, DC 20460 2025645268
Anand Mudambi
Chemist Office of Water
U.S. Environmental Protection Agency 1200 Pennsylvania Avenue, NW (4608TO
-------
Washington, DC 20460 2025642817 Email: mudambi.anand@epa.gov
Marian Olsen
Senior Human Health Risk Assessor Program Support Branch Emergency and
Remedial Response
U.S. Environmental Protection Agency 290 Broadway New York, NY 10007 2126374313
Email: olsen.marian@epa.gov
Haluk Ozkaynak
Senior Scientist
U.S. Environmental Protection Agency 109 T.W. Alexander Drive E20501 Research
Triangle Park, NC 27711 9195415172 Email: ozkaynak.haluk@epa.gov
John Punzi
Scientist Pesticide Data Program Science and Technology
U.S. Department of Agriculture 8609 Sudley Road Manassas, VA 20110 7033302300
xll5 Email: john.punzi@usda.gov
Resha Putzrath (via teleconference)
Navy and Marine Corps Public Health Center 3223 N Street, NW Washington, DC 20007
2022901140 Email: resha.putzrath@med.navy.mil
Linda Wilson (via teleconference)
Environmental Scientist New York State Office of the Attorney General 120 Broadway New
York, NY 10271 2124168948 Email: linda.wilson@ag.ny.gov
Kimberly Wise
Regulatory and Scientific Affairs American Petroleum Institute 1220 L Street, NW
Washington, DC 20005 2026828473 Email: wisek@api.org
APPENDIX E
MEETING AGENDA
United States
Environmental Protection Agency
Office of the Science Advisor, Risk Assessment Forum
United States Environmental Protection Agency Office of the Science Advisor,
Risk Assessment Forum
Peer Review Workshop of EPA's Draft
Report: Using Probabilistic Methods to
Enhance the Role of Risk Analysis in
•3-EPA
-------
DecisionMaking with Case Study
Examples
Sheraton Crystal City Hotel Arlington, VA May 6,
2010
Agenda
8:00 a.m. Registration/Check in
8:30 a.m. Welcome, Introductions, Meeting Purpose & Agenda Jan Connery, ERG
(contractor)
8:40 a.m. EPA Welcome Remarks Mary Greene,
Deputy Director, EPA Office of the Science Advisor
8:45 a.m. EPA Background Presentation Kathryn Gallagher,
Haluk Ozkaynak, Marian Olsen, & Bob Hetes, EPA
9:30 a.m. Public Comment Jan Connery
10:00 a.m. BREAK
10:15 a.m. Use of the PRA Document Scott
Ferson (Chair) & Panel
IA. Have the papers provided a balanced view of the usefulness of PRA in
decision making, and if not, describe how it is unbalanced?
IB. Are there additional advantages or disadvantages of using PRA that
were not identified and should be included? Please provide a description of
any additional advantages or disadvantages you have identified.
IC. Are there criteria or considerations that were not adequately addressed
for determining whether or not probabilistic methods may be useful to
characterize uncertainty and variability and support EPA decisionmaking?
If so, please describe such considerations.
ID. Considering risk managers are a major audience for these documents,
especially for the "Managers' Summary" document, what additional
information should be included in the document to aid them in decisions
regarding the application of PRA?
-------
11:30 a.m.
Application to Human Health and Ecological Risk
Assessments Scott Ferson & Panel
2. Have the current documents adequately addressed the unique issues, if any,
associated with application of PRA to both human health effects and ecological risk?
If not, what additional information/case studies should be presented specific to
human or ecological risk assessment?
12:15 p.m. LUNCH
1:30 p.m. Tiered Approach Scott Ferson & Panel
3A. Please comment on the completeness of the discussion of the tiered approach for
decision making regarding the application of PRA. What, if any, additional
refinements to the tiered approach outlined in the document can you recommend to
further support management decisions regarding use of PRA?
3B. Please comment on the use of a tiered approach to when to use PRA considering
management considerations of cost, time and resources.
2:00 p.m. Communication to EPA Risk Assessors & Managers Scott Ferson & Panel
4A. Is each document written at the appropriate level of detail for the intended
audience and length of document? Specifically, is too much or too little detail
presented for the respective audiences? Please provide suggestions on how to
address any such shortcomings.
4B. Is the level of detail of information in the case studies adequate, or is more
information needed? Please provide specific suggestions on how to improve the case
studies, if you have any.
4C. Please discuss whether, and how, the White Paper could be improved to help the
reader to better understand how PRA can help address and communicate variability
and uncertainty.
4D. Are the citations and references sufficient, or are there critical references which
need to be added? If so, please provide those citations and their relevance to the
papers.
2:30 p.m. BREAK
2:45 p.m. Communication to EPA Risk Assessors & Managers (cont.) . Scott Ferson & Panel
4C. Please discuss whether, and how, the White Paper could be improved to help the
reader to better understand how PRA can help address and communicate variability
and uncertainty.
4D. Are the citations and references sufficient, or are there critical references which
need to be added? If so, please provide those citations and their relevance to the
papers.
3:15 p.m. Additional Discussion Issues Scott Ferson & Panel
3:30 p.m. Reviewer Final Comments Scott Ferson & Panel
-------
„ -n „ , Jan Connerv &
3:50 p.m. Closing Remarks EPA/'OSA
4:00 p.m. ADJOURN
APPENDIX F
SLIDES OF EPA OPENING PRESENTATION
eha uverview
probabilistic Methods to tnnance
the note ot hisk
m
I
e
Decision-Making "
External Peer Review Meeting May 6, 2010
Kathryn Gallagher, Haluk Ozkaynak, Marian Olsen, and Bob Hetes Risk Assessment Forum
Technical Panel Co-Chairs
uverview or tsriermg ana
Hurpose/History or uocuments
Kathryn Gallagher, Ph.D.
Executive Director
Risk Assessment Forum
Office of the Science Advisor
-------
Overview o
1) Purpose and History of Documents
2) Main Document Overview
3) Case Study Overview
4) Managers' Summary Overview
3
United States Environmental Protection Agency
PRA Technical
* Sponsor: EPA Risk Assessment Forum
* Volunteer effort
• Regional, Program Office, arid ORD members
• Includes practitioners, senior scientists and science
advisors
• Support of Science Policy Council and Science
Advisor
4
United States Environmental Protection Agency
rtiA i ecnmcai ranei iviemoers
Gary Bangs, OSA/RAF Lead, retired . John Langstaff 0AR/0AQPS
-------
Kathryn Gallagher, OSA, Co-chair .Hawey Richmond,
Bob Hetes, ORD/NHEERL, Co-chair OAR/OAQPS, retired
Half k Ozkaynak, ORD/NERL , Co-» Zachary Pekar, OAR/OAQPS
• Mike Messner, OW
Marian Olsen, Region 2, Case Studies Lead
a • Ed Odenkirchen, OPP/EFED
Donna Randall, OPP/EFED
Brad Schultz, ORD
Khoan Dinh, OPPT
Pasky Pascual, ORD/NCER
Elizabeth Margosches, OPPT
John Paul, ORD
Jeff Swartout, ORD/NCEA
Chris Frey, ORD /NERL
Audrey Galizia, ORD/NCEA
Mike Dellarco, ORD/NCEA
David Miller, OPP/HED
Cynthia Stahl, Region 3
Nancy Rios-Jafolla, Region 3
Hurpose or we hha wnite napers
Principal Goal -Enhance the recognition and
application of PRA methods across the Agency
Anticipated Outcomes
Identify the role of PRA to support EPA decisions
Describe particular needs of the risk managers with respect to use of
probabilistic methods
Outline benefits and limitations of PRA methods
Identify specific needs, barriers, issues of concern, and what is needed to
promote the acceptance of these methods
wnite paper _ internal tHA tieview history
Draft prepared by and reviewed by RAF PRA Technical
Panel
Draft reviewed by RAF members across disciplines
Revised based on comments by RAF reviewers
Science Policy Council Steering Committee review and
approval
-------
I
tp/i reer neview uomments
PRA is unevenly implemented, for various reasons:
Lack of knowledge, funds, other resources
Needed:
Provide technical support to risk assessors on PRA
Inform risk managers and risk assessors of uses of PRA and related tools
«
-------
Three Conceptual Parts of White Papers
1. Main White paper
• Intended primarily for risk assessors
2. Case studies
• Examples ot how used in EPA
3. Managers' Summary
• Focused more on risk managers / decision
makers
9
United States Environmental Protection Agency
A
<0
La
*
"JA ^
*****
—|
mk
3
3
5
3
nr
u M
fiUU
m
Main White Paper
Haluk Ozkaynak, Ph.D.
Senior Scientist
Office of Research and Development
National Exposure Research Laboratory
10
United States Environmental Protection Agency
Mam wnne raper
Purpose - to provide a basic understanding of PRA methods
-------
and what they can provide to EPA decision making
Resource - basic information and guide where to go for more information
Provide foundation for future training, seminars and guidance
11
iviam wnite naper on using hma ivietnoas
White paper primarily aimed more towards risk assessors more than for
risk managers/decision-makers
General background on EPA's needs and experiences with PRA
Utility of PRA for exposure and risk analysis and informed risk
management decision-making
Descriptions of methods (including sensitivity and uncertainty analysis),
ranging from:
simple to complex
rapid to more time consuming
least to most resource intensive
Opportunities for further utilization
12
mam wnite raper on using hha ivietnoas
White paper primarily aimed more towards risk assessors more than for
risk managers/decision-makers
General background on EPA's needs and experiences with PRA
Utility of PRA for exposure and risk analysis and informed risk
management decision-making
Descriptions of methods (including sensitivity and uncertainty analysis),
ranging from:
simple to complex
rapid to more time consuming
least to most resource intensive
Opportunities for further utilization
12
wain wnne fa per: wny use HroDaomsuc wietnoasy
Facilitates explicit consideration and quantification of variabilities and
uncertainties at each step in the risk assessment process
Results from model sensitivity and uncertainty analyses can be used to
guide future data collection activities to improve limitations in knowledge by
reducing uncertainty
Provides a valuable tool for risk management or policy evaluations by
identifying and providing the degree of scientific uncertainty in alternatives
considered during the decision-making process 14
-------
Probabilistic Risk
Case Studies
Marian Olsen, Dr.P.H., M.P.H.
Emergency and Remedial Response
Division - Region 2
17
United States Environmental Protection Agency
irSJ
fffl
Case Study • Goals
* Provide a "Snapshot" of PRA activities across
various programs in EPA over the past 10 or more
years
* Human Health
* Ecological
* Program and Regional Offices
* Demonstrate various analysis approaches
* Inform risk assessors and managers on use of
PRA techniques
18
United States Environmental Protection Agency
-------
Case Study Format
* Study description/summary
* Type of risk analysis
* Results of analysis
* Management considerations
* Document availability
* Contacts
19
United States Environmental Protection Agency
k Jia.
i f'a
li. ml ¦¦
Case Study Selection Criteria
• Analysis conducted by EPA
• Publically available
• Representative of varying levels of
complexity
• Contact available
20
United States Environmental Protection Agency
uroup 7
Sensitivity Analysis
Least resource intensive
-------
• Basis for
• more complex mathematical and statistical techniques
• Useful in determining
risk model factors contributing to the variance in the risk estimate
need for additional data collection
uroup z
• Compared to Group 1 involves increased
additional time commitments for development
specialized expertise
• More sophisticated application of PRA
Specific exposure parameters/data sources
One-dimensional analyses
• Probabilistic sensitivity analysis
Expert elicitation results
Sampling protocol
Varied levels of peer involvement or peer review
uroup 3
• Compared to Groups 1 and 2
• Increased need for data, resources and time
• Includes
Two-dimensional Monte Carlo analysis
Microexposure event analysis
Expert elicitation
Bayesian statistics
Geostatistical analysis
Managers summary
Bob Hetes, MSPH
Acting Director, Research Planning and Coordination
Office of Research and Development
National Environmental Effects Health Research Laboratory
Managers summary
-------
Purpose - to provide a EPA managers/decision makers with
basic information on the methods and what they can provide to EPA
decision making
General concepts and principles of PRA
Describe how PRA can improve the bases of Agency decisions
Provide illustrations of how PRA has been used
Compare PRA with current practice
Considerations in pursuing PRA
Geared more toward managers than practitioners
Addressing uommon unanenges ractng uecision-
maKers
EPA must always make decisions in the presence of uncertainty.
PRA can help by:
Providing more information about possible impacts of alternative decisions
Providing clarity on whom we are protecting and the confidence In estimates of
protection
Allowing for more detailed comparison of alternatives estimated impacts on
protection and costs
Improving overall confidence in specific decisions
uenerai uoncepts or hha Tor Managers
Uncertainty and its impact on decisions
Variability and its impact on decisions
• How PRA can inform decisions
Characterize uncertainty in estimate (degree of confidence)
Identify critical parameters and assumptions
Identify "tipping points" where decision option would change
Estimate likelihood critical data values exist or validiity of assumptions
Estimate degree of confidence in decision / likelihood of specific decision errors
Types of risk management questions that PRA can address in the context of characterizing
variability
Question B: What is the risk for an individual experiencing high-end risk impacts in the population (e.g., 95%ile)
-------
A/hat is the riskfor
dian or average)
the population?
Question C: Whs
population (or num
experiencing at leas
of risk (e.g., 1 in
Question A: What is the risk for
isrcerit of
ipulafon
Question C: What percentage o
population (or number of individuals) experiencing at least a specific degree of risk (e.g., 1 in
atypical (median or average) individual in the population? ~ "
million cancer)
Risk distribution
percent of
population
low
high
Risk level
F-16
hactors to uonsiaer in using hha
Resources needed to develop the PRA arid review the document,
Expertise of EPA staff to develop a PRA or review a PRA submitted by a
contractor or member of the regulated community;
Data availability and format (e.g., electronic or paper copy) to develop
distributions to include in the PRA,
Time needed for the development and review of the analyses,
Funding, either intramural or extramural that may be necessary for development
and review of the document,
Peer-review including either internal and/or external review which has time and
cost implications, and
Communicating results to the scientific community, Agency executives,
stakeholders and the general public.
Next steps
-------
Revised the draft documents considering peer review and
public comments
Further internal review
Publish as final documents
Provide trainings/briefings to EPA managers and staff on the
documents
------- |