&EPA
United States
Environmental Protection
Agency
Evidence Integration in Integrated Science Assessments (ISAs):
A Case Study from the Draft Particulate Matter ISA
Michael J. Stewart, Ellen Kirrane, Thomas J. Luben, Jason Sacks, Barbara Buckley, Jennifer Nichols
National Center for Environmental Assessment | Environmental Media Assessment Group
Michael J. Stewart I Stewart.michael@epa.gov I 919-541-7524
Background:
The National Center for Environmental Assessment (NCEA) develops Integrated Science
Assessments (ISAs) as a key part of the Clean Air Act mandated reviews of the National
Ambient Air Quality Standards (NAAQS), which are set for six criteria pollutants: particulate
matter (PM), ozone, oxides of nitrogen, sulfur oxides, lead, and carbon monoxide. EPA
establishes primary NAAQS to protect public health, including sensitive lifestages or
populations, such as children or people with pre-existing disease. Secondary standards are
established to protect against adverse ecological and other welfare effects. The ISAs identify,
evaluate, integrate, and synthesize the comprehensive body of scientific evidence. This
generally includes hundreds to thousands of studies spanning epidemiology, controlled human
exposure, animal toxicology, dosimetry, exposure science, atmospheric science, welfare effects,
and ecology. NCEA employs a weight of evidence framework in developing ISAs, integrating
findings from the various lines of evidence and drawing conclusions on causality. More
specifically, ISAs use a five-level hierarchical causal framework, incorporating aspects of the
Hill criteria to assess causality (e.g., consistency, coherence, biological plausibility, temporality,
etc.) and classify whether evidence is sufficient to conclude a "causal relationship", "likely to be
a causal relationship", "suggestive of, but not sufficient to infer, a causal relationship",
"inadequate to infer a causal relationship", or "not likely to be a causal relationship." Each level
of the hierarchy represents the extent to which we can rule out chance, confounding or other
biases. In ISAs, these causality determinations are presented both in a narrative form and in
summary tables delineating the rationales and key evidence supporting the conclusion, reflecting
the application of the framework and characterization of the evidence. In this case poster, an
example from the draft PM ISA is presented, demonstrating the evaluation and integration of
multiple lines of evidence underlying the conclusion that there is a "causal relationship" between
short-term PM2 5 exposure and cardiovascular effects.
Aspects of Causality1
ISA Development
Literature Search and
Study Selection
(See Figure III)
Description
An inference of causality is strengthened when a pattern of elevated risks is observed
across several independent studies. The reproducibility of findings constitutes one of the
strongest arguments for causality. Statistical significance is not the sole criterion by which
the presence or absence of an effect is determined. If there are discordant results among
investigations, possible reasons such as differences in exposure, confounding factors, and
the power of the study are considered.
An inference of causality from one line of evidence (e.g., epidemiologic, controlled human
exposure, animal, or ecological studies) may be strengthened by other lines of evidence
that support a cause-and-effect interpretation of the association. There may be coherence
in demonstrating effects from evidence across various fields and/or across multiple study
designs or related health endpoints within one scientific line of evidence. For example,
evidence on welfare effects may be drawn from a variety of experimental approaches
(e.g., greenhouse, laboratory, and field) and subdisciplines of ecology (e.g.. community
ecology, biogeochemistry, and paleontological/historical reconstructions).
Biological plausibility
Biological gradient
(exposure-response
relationship)
An inference of causality is strengthened by results from experimental studies or other
sources demonstrating biologically plausible mechanisms. A proposed mechanism, which
is based on experimental evidence and which links exposure to an agent to a given effect,
is an important source of support for causality.
A well-characterized exposure-response relationship (e.g., increasing effects associated
with greater exposure) strongly suggests cause and effect, especially when such
relationships are also observed for duration of exposure (e.g.. increasing effects observed
following longer exposure times).
The finding of large, precise risks increases confidence that the association is not likely
due to chance, bias, or other factors. However, it is noted that a small magnitude in an
effect estimate may or may not represent a substantial effect in a population.
Experimental evidence Strong evidence for causality can be provided through "natural experiments" when a
change in exposure is found to result in a change in occurrence or frequency of health or
welfare effects.
Evidence of a temporal sequence between the introduction of an agent and appearance of
the effect constitutes another argument in favor of causality.
Evidence linking a specific outcome to an exposure can provide a strong argument for
causation. However, it must be recognized that rarely, if ever, does exposure to a pollutant
invariably predict the occurrence of an outcome, and that a given outcome may have
multiple causes.
Structure activity relationships and information on the agent's structural analogs can
provide insight into whether an association is causal. Similarly, information on mode of
action for a chemical, as one of many structural analogs, can inform decisions regarding
likely causality.
Sample Causality Text: Short-term Exposure to PM2 5
and Cardiovascular Effects2
A large body of recent evidence confirms and extends the evidence from the previous ISA
indicating that there is a "causal relationship" between short term PM2 5 exposure and
cardiovascular effects. In the current review, evidence supporting the causality determination
includes generally positive associations reported from epidemiologic studies of hospital
admissions and emergency department (ED) visits for cardiovascular related effects, and in
particular, for ischemic heart disease and heart failure. Results from these observational
studies are in agreement with experimental evidence from controlled human exposure and
animal toxicological studies of endothelial dysfunction, as well as with endpoints indicating
impaired cardiac function, increased risk of arrhythmia, changes in heart rate variability
(HRV), increases in blood pressure (BP), and increases in indicators of systemic
inflammation, oxidative stress, and coagulation. Results from observational panel studies,
though not entirely consistent, also provide some evidence of increased risk of arrhythmia,
decreases in HRV, increases in BP, and changes in cardiac electrophysiology. Thus, the
combination of evidence from experimental and epidemiologic panel studies provides
coherence and biological plausibility for the results from observational epidemiologic studies.
Finally, epidemiologic studies of cardiovascular-related mortality provide additional evidence
and contributes to the continuum of effects from biomarkers of inflammation and coagulation,
subclinical endpoints (HRV, BP, endothelial dysfunction), ED visits and hospital admissions
for outcomes such as ischemic heart disease (IHD) and congestive heart failure (CHF), and
eventually death. The current body of evidence also reduces uncertainties from the previous
review related to the potential for copollutant confounding and biological plausibility for
cardiovascular effects following short term PM2 5 exposure.
Sample Causality Table: Short-term Exposure to PM2 5
and Cardiovascular Effects2
Rationale for Causal Determination
Key Evidence
ISAs Causality Framework1
Consistent epidemiologicevidence from Increases in ED visits and hospital admissions for IHD and CHF in
multiple, high quality studies at relevant multicity studies conducted in the U.S., Canada, Europe, and Asia
PN/bs concentrations	Increases in cardiovascularmortalityin multicity studies conducted in the
U.S., Canada, Europe, and Asia.
Evaluation of Individual Study Quality
After study selection, the quality of individual studies is evaluated by U.S. EPA or outside experts in the fields of
atmosphenc science, exposure assessment, dosimetry, animal toxicology, controlled human exposure,
epidemiology, biogeochemistry, terrestrial and aquatic ecology, and other welfare effects, considenna the design,
methods, conduct, and documentation of each study. Strengths and limitations of individual studies tnat may affect
the interpretation of the study are considered.
Develop Initial Sections
Review and summarize conclusions from
previous assessments and new study results
and findings by discipline and category of
outcome/effect (e.g., toxicological studies of lung
function or biogeochemical studies of forests)
G
Peer Input Consultation
Review of initial draft materials by scientists
from both outside and within the LI S. EPA in
public meeting or public teleconference.
Evaluation, Synthesis, and Integration of Evidence
Integrate evidence from scientific disciplines. Evaluate evidence for related groups of endpoints or outcomes to
draw conclusions for specific health or welfare effect categories, integrating nearth or welfare effects evidence with
information on mode of action and exposure assessment.
*t
Development of Scientific Conclusions and Causal Determinations
Characterize weight of evidence and develop judgments regarding causality for health or welfare effect categories.
Develop conclusions regarding concentration- or dose-response relationships, potentially at-risk populations,
lifestages, or ecosystems.
*t
Draft Integrated Science Assessment
Evaluation and integration of newly published studies
Clean Air Scientific Advisory Committee
Independent review of draft documents for
scientific quality and sound implementation of
causal framework during public meetings.
Final Integrated Science Assessment
Public Comments
Comments on draft ISA solicited by the U.S. EPA
U.S. Environmental Protection Agency
Office of Research and Development
This poster contains results from the external review Draft PM ISA. This information has been distributed solely for the purpose of
predissemination peer review under applicable information quality guidelines. It has not been formally disseminated by EPA. It does not
represent and should not be construed to represent any Agency determination or policy.
WEIGHT OF EVIDENCE FOR CAUSAL DETERMINATION
Health Effects
Ecological and Other Welfare Effects
Causal relationship
Evidence is sufficient to conclude that there is a causal
relationship with relevant pollutant exposures (e.g., doses
or exposures generally within one to two orders of
magnitude of recent concentrations). That is, the pollutant
has been shown to result in health effects in studies in
which chance, confounding, and other biases could be
ruled out with reasonable confidence. For example:
(1) controlled human exposure studies that demonstrate
consistent effects, or (2) observational studies that cannot
be explained by plausible alternatives or that are
supported by other lines of evidence (e.g., animal studies
or mode of action information). Generally, the
determination is based on multiple high-quality studies
conducted by multiple research groups.
Evidence is sufficient to conclude that there is a causal
relationship with relevant pollutant exposures. That is, the
pollutant has been shown to result in effects in studies in which
chance, confounding, and other biases could be ruled out with
reasonable confidence. Controlled exposure studies (laboratory
or small- to medium-scale field studies) provide the strongest
evidence for causality, but the scope of inference may be limited.
Generally, the determination is based on multiple studies
conducted by multiple research groups, and evidence that is
considered sufficient to infer a causal relationship is usually
obtained from the joint consideration of many lines of evidence
that reinforce each other.
Likely to be a causal
relationship
Evidence is sufficient to conclude that a causal
relationship is likely to exist with relevant pollutant
exposures. That is, the pollutant has been shown to result
in health effects in studies where results are not explained
by chance, confounding, and other biases, but
uncertainties remain in the evidence overall. For example:
(1)	observational studies show an association, but
copollutant exposures are difficult to address and/or other
lines of evidence (controlled human exposure, animal, or
mode of action information) are limited or inconsistent, or
(2)	animal toxicological evidence from multiple studies
from different laboratories demonstrate effects, but limited
or no human data are available. Generally, the
determination is based on multiple high-quality studies.
Evidence is sufficient to conclude that there is a likely causal
association with relevant pollutant exposures. That is, an
association has been observed between the pollutant and the
outcome in studies in which chance, confounding, and other
biases are minimized but uncertainties remain. For example, field
studies show a relationship, but suspected interacting factors
cannot be controlled, and other lines of evidence are limited or
inconsistent. Generally, the determination is based on multiple
studies by multiple research groups.
Suggestive of, but not
sufficient to infer, a causal
relationship
Evidence is suggestive of a causal relationship with
relevant pollutant exposures but is limited, and chance,
confounding, and other biases cannot be ruled out. For
example: (1) when the body of evidence is relatively small,
at least one high-quality epidemiologic study shows an
association with a given health outcome and/or at least
one high-quality toxicological study shows effects relevant
to humans in animal species, or (2) when the body of
evidence is relatively large, evidence from studies of
varying quality is generally supportive but not entirely
consistent, and there may be coherence across lines of
evidence (e.g., animal studies or mode of action
information) to support the determination.
Evidence is suggestive of a causal relationship with relevant
pollutant exposures, but chance, confounding, and other biases
cannot be ruled out. For example, at least one high-quality study
shows an effect, but the results of other studies are inconsistent.
Inadequate to infer a Evidence is inadequate to determine that a causal
causal relationship	relationship exists with relevant pollutant exposures. The
available studies are of insufficient quantity, quality,
consistency, or statistical power to permit a conclusion
regarding the presence or absence of an effect.
Evidence is inadequate to determine that a causal relationship
exists with relevant pollutant exposures. The available studies an
of insufficient quality, consistency, or statistical power to permit a
conclusion regarding the presence or absence of an effect.
Not likely to be a causal
relationship
Evidence indicates there is no causal relationship with
relevant pollutant exposures. Several adequate studies,
covering the full range of levels of exposure that human
beings are known to encounter and considering at-risk
populations and lifestages, are mutually consistent in not
showing an effect at any level of exposure.
Evidence indicates there is no causal relationship with relevant
pollutant exposures. Several adequate studies examining
relationships with relevant exposures are consistent in failing to
show an effect at any level of exposure.
Consistent evidence from controlled
human exposure studies atrelevant
PM25 concentrations
Consistent changes in measures ofendothelial dysfunction
Generally consistent evidence for small increases in measures of blood
pressure following CAPs exposure
Additional evidence of conduction abnormalities, heart rate variability,
impaired heart function, systemic inflammation/oxidative stress
Consistent evidence from animal
toxicological studies atrelevant PM2.5
concentrations
Consistent changes in indicatorsof endothelial dysfunction.
Additional evidence of changes in impaired heart function, conduction
abnormalities/arrhythmia, heart rate variability, blood pressure, systemic
inflammation/oxidative stress
Epidemiologicevidence from copollutant The magnitude of PM25 associations remain positive, butin somecases
models provides somesupportforan
independent PM2.5 association
are reduced with larger confidence intervals in copollutantmodels with
gaseous pollutants. Further support from copollutant analyses indicating
positive associations for cardiovascular mortality. Recent studies that
examined potential copollutant confounding are limited to studies
conducted in Europe and Asia.
When reported, correlations with gaseous copollutants were primarily in
the low to moderate range (r< 0.7).
Consistentpositive epidemiologic	Positive associations consistentlyobserved across studies that used
evidence for associations between PM25	ground-based (i.e., monitors), model (e.g., CMAQ, dispersion models)
exposure and CVD ED visits and hospital and remote sensing (e.g., AOD measurements from satellites) methods,
admissions across exposure	including hybrid methods that combine two or more of these methods,
measurement metrics
Epidemiologicevidence supports a
log-linear, no-threshold
concentration-response (C-R)
relationship
Generally cons istent evidence for
biological plausibility of cardiovascular
effects
Strong evidence for coherence of effects across scientific disciplines and
biological plausibilityfor a range ofcardiovasculareffects in responseto
short-term PM2 5 exposure. Includes evidence for reduced myocardial
blood flow, altered vascular reactivity, andST segmentdepression.
Uncertainty regarding geographic
heterogeneity in PM25 associations
Multicity U.S. studies demonstrate city-to-city and regional heterogeneity
in PM25-CVD ED visit and hospital admission associations. Evidence
supports thata combination offectors including composition and
exposure factors may contribute to the observed heterogeneity.
* CMAQ= Community Multiscale Air Quality Modeling System; A0D= Aerosol Optical Depth;
CAPs = Concentrated Ambient Particles
References:
1.	Preamble to the ISA: https://efpub.epa.siov/ncea/isa/recordisplav.cfm?deid=310244
2.	ISA for PM (External Review Draft): http://cflnt.rtpnc.epa.gov/ncea/prod/recordispla'
cfm?deid=341593
W
Printed on 100% recycled/recyclable p<
with a minimum 50% post-consumer
fiber using vegetable-based ink.

-------