EPA/SAB/79/001
Review of
"A Method of Assessing the Health Risks Associated
With Alternative Air Quality Standards for Ozone"
(Draft dated July 1978J
A Report of the Subcommittee on Health Risk Assessment
September 1979
Science Advisory Board
U.S. Environmental Protection Agency
Washington, D.C, 20460
-------
EPA NOTICE
This report has been written as part of the activities of
the Science Advisory Board, a public advisory group providing
extramural scientific information to the Administrator and other
officials of the Environmental Protection Agency. .The Board is
structured to provide a balanced expert assessment of the
scientific matters related to problems facing the Agency, This
report has not been reviewed for approval by the Agency, hence
its contents do not necessarily represent the views and policies
of the Environmental Protection Agency, nor does mention of
trade names or commercial products constitute endorsement or
recommendation for use.
-------
SCIENCE ADVISORY BOARD
SUBCOMMITTEE ON HEALTH RISK ASSESSMENT
Chairpersoni
Dr. Anita S. Curran, Commissioner, Department of Health, Westchester
County, County Office Building 2, 150 Grand Street, White
Plains, New York 10601
Members;
Dr. Bernard Altshuler, Institute of Environmental Medicine, New
York University Medical Center, 550 First Avenue, New York,
New York 10016
Dr. Judy A. Bean, College of Medicine, Department of Preventive
Medicine and Environmental Health, University of Iowa, Iowa
City, Iowa 52242
Dr. Jennifer L. Kelsey, Associate Professor of Epidemiology,
Yale University School of Medicine, Department of
Epidemiology and Public Health, 60 College Street, New
Haven, Connecticut 06510
Dr, John Logsdon, Director of Graduate Program in Science,
Technology, and Public Policy/ George Washington University,
Washington, D.C. 20052
Dr. Granger Morgan, Department of Engineering and Public Policy,
Carnegie-Mellon University, 5000 Forbes Avenuet Pittsburgh,
Pennsylvania 15213
Dr. D, Warner North, Decision Focus Inc., 1801 Page Mill Road,
Palo Alto, California 94304
Dr. Geoffrey Watson, Chairman, Department of Statistics, Fine
Mail, Princeton University, Princeton, New Jersey 08540
SAB Staff;
Mr. Lloyd Taylor, 'Staff Officer (Until July 6, 1979), Science
Advisory Board, A-101, U.S. Environmental Protection Agency,
Washington, D.C. 20460
Ms, Sarah Mills, Staff Assistant, Science Advisory Board, A-101,
U.S* Environmental Protection Agency, Washington, D.C.
20460 (Until July 26, 1979)
Mr. Ernst Linde, Staff Officer (After July 6, 1979), Science
Advisory Board, A-101, U.S. Environmental Protection Agency,
Washington, D.C. 20460 Phone: (202) 472-9444
-------
INTRODUCTION
The Subcommittee on Health Risk Assessment of the Science
Advisory Board was established to advise the Agency on
methodologies for assessing the health risks of various
environmental pollutants. The Subcommittee was requested to
review the methodology proposed by EPA's Office of Air Quality
Planning and Standards .(OAQPS)/ "A Method for Assessing the
Health Risks Associated With Alternative Air Quality Standards
For Oaone," draft dated July 1978. The Subcommittee reviewed
the methodology at a public meeting on April 19 and 20, 1979,
and provides the following report.
CONCLUSIONS
I. The Office of AirQuality Planning and Standards is
commended for recognizing the need for and undertaking the
development of quantitative subjectiye_d_ecision..._ai<3s foy
in_ their assessment of the .health risks of air pollutants.
s
II. The methodology developed by OAQPS is not ready
for applieation_i_n the process o_f_ establishing air quality
standards.
-------
CONCLOSION I
The Office of Air Quality Planningand Standardsis
commendedforrecognizing the need for and undertaking
the development of quantitative subjective decision aids
for use in their assessment of the health risks of air
pollutants.
This approach recognizes that when objective information
is scarce, subjective judgments by scientific experts are often
essential in making quantitative estimates of risks associated
with environmental pollution. The approach organizes informa-
tion on subjective judgments, identifies issues about which
subjective opinions ;are made and makes the analytical process
of assessing risks irore explicit and open for public review.
Properly done, probabilistic methodology will facilitate
communication between scientific experts and decision makers.
RECOMMENDATIONS
1. The Subcommittee urges the Agency to expand its
efforts to develop and evaluate the•use of judgmental
probabilities in the process oŁ establishing national
air quality standards.
While EPA should strengthen its own in-house capability for
assessing risks by developing a variety of methodologies, it
should also seek proposals from a number of outside experts to
allow formulation and comparison of alternative approaches.
This should assure that the greatest number of potentially
useful proposals are considered and provide an opportunity to
compare the results of EPA's proposed methodology with other
-------
approaches. However,the Subcommittee emphasizes that the
application of such decision .aids in the standard setting
process must be avoided_until thesetools have been carefully
refined and evaluated* As there is no practical way to verify
conclusions which flow from the use of such methodology/ its
value to regulatory decision-making depends upon the extent to
which the results are believed. Since no single formulation
can be demonstrated to.be "best," it is important that several
be tried and compared and that an opportunity exist for the
development of understanding and consensus. A clear under-
standing of the concept of a methodology and the procedures
for applying it, the results of alternative analytical
methodologies, and the opinions of experts about the validity
and reliability of a given methodology is needed to establish
confidence in the results. Although the developmental work may,
focus on the upcoming standard for carbon monoxide, no newly
proposed methodology should be used in making the regulatory
decision until credibility has been established.
2. Under no circumstances should use of such techniques
become a replacement for the research needed to
establish an objective data base for assessing health
risks,
Because they are quantitative in nature and tend to produce
quantitative graphical results, decision aids that use expert
probabilistic judgments tend to "look scientific," but the
techniques are not a substitute for scientific fact.
-------
3, EPA should undertake or fund a major review of the
experimental work addressing techniques and problems
in eliciting quantitative expert opinions.
There is a body of literature which identifies important
limitations and biases in people1s abilities to make quantita-
tive probabilistic judgments. This literature may not only
assist in improving the elicitation of expert judgments but may
also form a basis of legal objections to the use of quantitative
decision aids. EPA's decisions on the use of methodologies
involving judgmental probabilities should be made with a full
appreciation of this literature.
CONCLUSION II .. '
The methodology developed by OAQPS is not ready for
application in the process of^establishing air quality
s t and a r els.
A number of difficulties have been identified in the
formulation:
s
1. The methodology is complex and proposes a new
analytical procedure that is not adequately defined.
2, The mathematics and computer processing tend to
obscure the fundamental modeling assumptions, making
the analysis impenetrable to all but mathematical
specialists. This is exactly what decision analysis
techniques should not do.
-------
3. The conceptual basis has not been clearly described,
and standardized procedures for applying the
methodology have not been developed.
4. Consideration of alternative approaches has not been
documented and the results have not been compared with
those of alternative approaches.
RECOMMENDATIONS
1. QAQPS should formulate a plan outlining how the Agency
will (a) develop the proposed methodology, including
standards, and protocols for application, (b) consider
alternative approaches, and (c) select and establish
the credibility of the best methodology,
2. OAQPS should publish, a short, easily understood
description of the proposed methodology.
The paper should be written for a general audience and
should not rely heavily on abstract statistical concepts. In
addition, the authors should submit scientific papers on the
methodology to criticallyedited peer review journals.
»»'
3. The encoding process' and the method of dealing with
differences in expert opinion need to be improved.
Meetings and interviews with experts are a critical part
of the encoding process due to the risk of misunderstandings,
errors and loss of credibility. The study on ozone is the only
available example of how EPA would apply the proposed
-------
methodology. The interviews in this example do not meet
accepted standards. The authors, should familiarize themselves
with the available literature on the encoding procedure. Key
concepts used in the interviews need to be explicitly defined.
Differences in judgments should be reviewed with the experts
to see if there is a true difference or a basis for consensus.
4, The expert committee's final deliberations in
arriving at a probability distribution should
be a matter of public record*
5. OAQPS described two approaches to developing
a "dose-response" relationship. These formulations
struck a number of members of the Subcommittee as
awkward.
One approach used in the July 1978 report elicited
opinions on the level of air pollution that would cause health
effects in 1%, 5%, and 10% of. the individuals in the most
sensitive part of the population. The other approach fixes
the level of air pollution and asks what fraction of the most
sensitive population would suffer health effects at a given
level of air pollution. EPA should work on developing the
latter approach, as it provides the most useful information
for the decision-maker.
-------
6, Because of the considerable uncertainties on the air
quality side of the model, this- portion of the model
should be reformulated so that expert subjective
judgments can be used there as well.
The Subcommittee expressed concern about the
appropriateness of a model which attempts to combine a variety
of different expert judgments into a single summary finding.
If this is to be done, careful' discussion and justification of
the procedures applied "and the weightings used are .required.
Some members would prefer a methodology which pairs health
effects and air quality judgments and allows the Administrator
and his staff to examine and explore the full range of resulting
outcomes through appropriate summary displays. No quantitative
decision aid can replace the need for a deep technical
understanding of the nature and sources of the uncertainties
involved..
ADDITIONAL COMMENTS
The Subcommittee was asked to comment on several questions
relevant to the use of the OAQPS methodology. Most of these
have already been addressed. The following are comments on some
of the questions not previously covered:
1, Are there Alternative___Methodologies or Modifications?
It is apparent from the material presented at the meeting
that there are a number of separate groups in EPA doing risk
assessment. It is also apparent that these groups have little
or no direct communication with each other. The Subcommittee
-------
8
urges that 'EPA develop operational linkages between these
groups. It is also suggested that these activities be reviewed
and selectively presented to this Subcommittee for our input and
advice as to which methodologies seem to hold the most promise,
which might be replicated in other sections of SPA, and when to
use which proposed method and under what conditions.
2. Under What Conditions Should the Methodology be Qsed
or Not Used?
Encoding judgmental probabilities should be used only where
it enhances comaunications between the scientific experts and
the decision-makers. At one extreme, the uncertainty in the
scientific data base may be so small that encoding expert
opinion is not necessary. At the other extreme are cases in
which the uncertainty in the data is so great that formal
decision analysis approaches are not appropriate. Appropriate
use lies between these extremes and will have to be judged on
a case-by-case basis-
3. How Should Experts be Selected?
A principal criterion is that the user (EPA) and the public
have confidence in .the experts. There.are no rigid rules, but
the following guidelines may be helpful.
a. The experts should encompass relevant disciplines
and points of view within the scientific
community. The goal should be to involve experts
of recognized credibility.*
One member of the Subcommittee feels strongly that, insofar as
practical, the experts should be free of personal interests
or affiliations which might cast a reasonable suspicion about
the neutrality of their judgments.
-------
b. The process of identifying and selecting experts
should be explicit and open.
c. Each expert should be' willing publicly to
acknowledge responsibility for the information
and judgments he or she provides.
cU Review and debate on the information and judgments
should be encouraged both among the experts them-
selves and among their peers in the scientific
community.
4. How Many Experts^ Should be Interviewed?
The Subcommittee suggests no specific number. The number
will.depend upon the complexity of the issues, the variety of
scientific opinion, and the different kinds of expertise that
are applicable. The experimental literature on eliciting
quantitative expert opinion should be consulted, and EPA should
consider undertaking sensitivity tests to determine how the
results might-vary with the number of experts.
------- |