#• \
\ d?
PRO*^
Integrated Review Plan for the National
Ambient Air Quality Standards for Ozone
and Related Photochemical Oxidants
Volume 2: Planning for the Review and the
Integrated Science Assessment
-------
-------
EPA-452/R-24-001 b
December 2024
Integrated Review Plan for the
National Ambient Air Quality Standards for
Ozone and Related Photochemical Oxidants
Volume 2: Planning for the Review and the Integrated
Science Assessment
U.S. Environmental Protection Agency
Office of Air Quality Planning and Standards
Office of Air and Radiation
and
Center for Public Health and Environmental Assessment
Office of Research and Development
Research Triangle Park, NC
-------
DISCLAIMER
This plan serves as a public information document and as a management tool for
the U.S. Environmental Protection Agency's (EPA's) Center for Public Health and
Environmental Assessment and Office of Air Quality Planning and Standards in
conducting the review of the air quality criteria and the national ambient air quality
standards for ozone and related photochemical oxidants. This document is being
circulated to facilitate discussion with the Clean Air Scientific Advisory Committee and
for public comment to inform the EPA's current review of the air quality criteria and the
national ambient air quality standards for ozone and related photochemical oxidants. It
does not represent and should not be construed to represent an Agency determination
or policy. Mention of trade names or commercial products does not constitute
endorsement or recommendation for use.
-------
TABLE OF CONTENTS
1 INTRODUCTION 1-1
2 POLICY-RELEVANT ISSUES IN THE CURRENT REVIEW 2-1
2.1 Review of the Primary O3 Standard 2-1
2.2 Review of the Secondary O3 Standard 2-4
3 DEVELOPMENT OF THE INTEGRATED SCIENCE ASSESSMENT 3-1
3.1 Organization of the ISA in the Current Review 3-1
3.2 Scope of the ISA in the Current Review 3-3
3.2.1 Atmospheric Science 3-4
3.2.2 Exposure Science 3-8
3.2.3 Health Effects 3-9
3.2.4 Welfare Effects 3-12
3.3 PROCESS FOR DEVELOPING THE ISA 3-14
3.3.1 Literature Search 3-16
3.3.2 Identifying Potentially Relevant Studies 3-17
3.3.3 Evaluation of Individual Study Quality 3-22
3.3.4 Integration of Evidence and Determination of Causality 3-23
3.3.5 Quality Management 3-26
3.3.6 CASAC Peer Review 3-27
3.4 SCIENTIFIC QUESTIONS TO GUIDE EVALUATION OF THE EVIDENCE 3-27
3.4.1 Source to Concentration - Air Quality, Atmospheric Science, Fate, and Transport
3-28
3.4.2 Human Exposure 3-29
3.4.3 Dosimetry 3-31
3.4.4 Biological Plausibility 3-32
3.4.5 Health Outcomes 3-33
3.4.6 At-Risk Lifestages and Populations 3-39
3.4.7 Welfare Effects 3-41
4 REFERENCES 4-1
APPENDIX: ISA DEVELOPMENT PROCESS A-1
-------
The planning phase of the U.S. Environmental Protection Agency's (EPA's) reviews
of the air quality criteria and the national ambient air quality standards (NAAQS)
includes development of an integrated review plan (IRP), which is composed of three
volumes. Volume 1 provides background information and serves as a reference for the
public and the Clean Air Scientific Advisory Committee (CASAC). Volume 2 (this
document) addresses the general approach for the review, identifies policy-relevant
issues in the review and describes key considerations in the EPA's development of the
ISA. This document is the subject of CASAC consultation and public comment. Volume 3
describes key considerations in the EPA's planning with regard to any quantitative risk
and exposure analyses to be considered for the review. In order that consideration of
the availability of new scientific evidence in the review inform these plans, the
development and public release of Volume 3 will generally coincide with the availability
of the draft ISA. At that time, Volume 3 is the subject of CASAC consultation and public
-------
1 INTRODUCTION
The U.S. Environmental Protection Agency (EPA) is conducting a review of the air
quality criteria and the national ambient air quality standards (NAAQS) for ozone (O3)
and related photochemical oxidants.1 This document is the second of three volumes
that will comprise the integrated review plan (IRP) for the O3 NAAQS review. Volume 2
of the IRP contains the current plans for the general approach for the review, as well as
key planning considerations for the development of the integrated science assessment
(ISA). The NAAQS review process provides an integrative assessment of relevant
scientific information and will focus on key aspects of the NAAQS, including the basic
elements of the standards: the indicator,2 averaging time, form,3 and level. These
elements, which together serve to define each ambient air quality standard, are
considered collectively in evaluating the protection to public health and welfare
afforded by the standards.
Volume 1 of the IRP includes background information on prior reviews of the
health and welfare-based air quality criteria and standards for O3 and related
photochemical oxidants, a summary of key aspects of the basis for the existing O3
NAAQS, and a summary of the status and anticipated milestones for the current review.
Volume 1 also includes an appendix that provides an overview of the key aspects of
existing ambient air monitoring requirements for O3. Volume 2 of the IRP (this
document) presents the general approach for this review, the policy-re levant questions
guiding the review, and the plans for the development of the ISA. Specifically, Chapter 2
1 As noted in Volume 1 of the IRP, this review also incorporates the EPA's ongoing reconsideration of the
2020 decision on the O3 NAAQS.
2 The "indicator" of a standard defines the chemical species or mixture that is to be measured in
determining whether an area attains the standard. For example, the indicator of the current NAAQS for
photochemical oxidants is ozone (O3).
3 The "form" of a standard defines the air quality statistic that is to be compared to the level of the
standard in determining whether an area attains the standard. For example, the form of the annual
PM2.5 NAAQS is the 3-year average of the weighted annual mean PM2.5 concentration, while the form of
the current 3-month Pb NAAQS is a 3-month average concentration not to be exceeded during a 3-
year period.
December 2024
1-1
-------
of Volume 2 outlines the general approach of the NAAQS review and details a set of
policy-relevant questions intended to focus this review on the critical scientific and
policy issues. Chapter 3 of Volume 2 presents plans for the ISA, including the document
organization, scope, and specific questions for consideration given the overarching
policy-relevant questions for the review. Volume 3 of the IRP is the planning document
for quantitative risk and exposure analyses (REAs) for the review to be considered in the
policy assessment (PA). Volume 3 will be developed with consideration of the availability
of new evidence as identified in the draft ISA. Accordingly, public release of Volume 3
will generally coincide with that of the draft ISA, and it will be the subject of a
consultation with the CASAC at that time.
December 2024
1-2
-------
2 POLICY-RELEVANT ISSUES IN THE CURRENT REVIEW
The approach planned for considering the information available in this review of
the current primary and secondary O3 standards is framed by a series of questions, the
answers to which are intended to inform the Administrator's judgments as to whether
the current standards provide the requisite protection of public health and welfare and
the Administrator's decisions as to whether to retain or revise these standards. The ISA,
REA and PA developed in this review of the O3 NAAQS will provide the basis for
addressing these questions. The approach for considering these questions is
summarized for the primary standard in section 2.1 and for the secondary standard in
section 2.2.
2.1 REVIEW OF THE PRIMARY Os STANDARD
The approach planned for this review of the primary O3 standard is fundamentally
based on using the Agency's assessment of the current scientific evidence, quantitative
assessments of exposures and/or risks, and other associated analyses (e.g., air quality
analyses) to inform the Administrator's judgments regarding a primary standard that is
requisite to protect public health with an adequate margin of safety. This approach
involves translating scientific and technical information into the basis for addressing a
series of key policy-re levant questions using both evidence- and exposure-/risk-based
considerations. This series of key questions related to the primary standard is presented
below, in the context of the general approach for the review.
The approach planned for this review of the primary O3 standard will build on the
substantial body of work developed during prior reviews and the associated
conclusions, taking into account the more recent scientific information and air quality
data now available to inform our understanding of the key policy-relevant issues in this
review. Key aspects of the basis for the decision establishing the current primary O3
standard in 2015 and retaining that standard without revision in 2020 are summarized in
Volume 1 of the IRP. The ISA, REA (as warranted), and PA developed in this review will
provide the basis for addressing the key policy-relevant questions in the review, and
2-1
-------
these assessments will inform the Administrator's judgments and decision as to whether
to retain or revise the primary O3 standard.
The final decision on the primary standard is largely a public health policy
judgment by the Administrator.4 Final decisions on the NAAQS must draw upon
scientific information and analyses about health effects and risks, as well as judgments
regarding the array of uncertainties that are inherent in the scientific evidence and
analyses. Consistent with the Agency's approach across all NAAQS reviews, the
approach of the PA to informing these judgments is based on a recognition that the
available health effects evidence generally reflects a continuum that includes exposures
for which scientists generally agree health effects are likely to occur through lower levels
at which the likelihood and magnitude of response become increasingly uncertain. This
approach is consistent with the requirements of the NAAQS provisions of the Clean Air
Act (CAA) and with how the EPA and the courts have historically interpreted the CAA.
These provisions require the Administrator to establish primary standards that are
requisite to protect public health with an adequate margin of safety. In so doing, the
Administrator seeks to establish primary standards that are neither more nor less
stringent than necessary for this purpose. The provisions do not require that a standard
be set at a zero-risk level, but rather at a level that protects the public health, including
the health of sensitive groups, with an adequate margin of safety.5
With regard to the available body of scientific evidence assessed in the ISA and
used as a basis for developing and/or interpreting the exposure/risk analyses, the PA
summarizes the policy-re levant aspects, including information available in this review
that may inform public health judgments regarding the significance or adversity of key
effects. The PA addresses questions such the following:
4 Key aspects of the decisions made in the last review, including the Agency's consideration of important
policyjudgments concerning the scientific and exposure/risk information and associated uncertainties
and limitations, as well as the Administrator's public health policyjudgments regarding an adequate
margin of safety, are summarized in section 1.2 of Volume 1 of this IRP.
5 More than one population group may be identified as sensitive or at risk in a NAAQS review. The
decision in the review of the primary standards will reflect consideration of the degree to which
protection is provided for these sensitive population groups. To the extent that any particular
population group is not among the identified sensitive groups, a decision that provides protection for
the sensitive groups would be expected to also provide protection for other population groups.
2-2
-------
• Is there newly available evidence that indicates the importance of photochemical
oxidants other than O3 with regard to abundance in ambient air and the potential for
human exposures and associated health effects?
• Does the currently available scientific evidence, including that newly available in this
review, alter prior conclusions regarding the nature of health effects attributable to
human exposure to O3 in ambient air? Is there evidence of health effects beyond those
identified in the last review?
• Does the current evidence alter our understanding of populations 6 that are
particularly at risk from O3 exposures?
• Does the current evidence alter past conclusions regarding the exposure duration and
concentrations associated with health effects? To what extent does the scientific
evidence indicate health effects attributable to exposures to O3 concentrations lower
than previously reported and what are important uncertainties in that evidence?
• To what extent have previously identified uncertainties in the health effects evidence
been reduced, or do important uncertainties remain? Have new ones been identified?
Similarly, the PA summarizes the currently available exposure and risk
information, whether newly developed in this review or predominantly developed in the
past and interpreted in light of current information. In so doing, the PA addresses
questions such as the following:
• What are the nature and magnitude of exposures and associated health risks for air
quality conditions meeting the existing standard and potential alternative standards,
as appropriate?
• What are the important uncertainties associated with these risk and exposure
estimates?
In evaluating the policy implications of the currently available information,
including that newly available in this review, the PA considers various aspects of the
scientific evidence and exposure/risk information, related limitations and associated
uncertainties with regard to the adequacy of the existing standard and potential
6 As used here and similarly throughout this document, the term population (in the context of health and
the primary standard) refers to persons having a quality or characteristic in common, such as a specific
pre-existing illness or a specific age or lifestage. Some populations may be at increased risk of health
effects occurring with exposure to O3 as a result of any of a variety of factors, including genetic or
developmental aspects, disease or smoking status, and factors related to socioeconomic status,
reduced access to health care, or increased exposure.
2-3
-------
alternatives, as appropriate. These evidence- and exposure/risk-based considerations
are organized around a set of questions, such as the following:
• Does the currently available information call into question the use of O3 as the
indicator for the primary standard for photochemical oxidants? Is support provided for
considering a different indicator?
• Does information newly available in this review indicate the potential for health effects
of concern to occur in at-risk populations at lower exposures than previously
understood? What are important uncertainties associated with this information?
• To what extent are the estimates of exposures and risks to at-risk populations
associated with air quality conditions just meeting the current standard, or potential
alternative standards, reasonably judged important from a public health perspective?
What are important uncertainties associated with these exposure/risk estimates?
Based on these evaluations, the draft PA develops preliminary conclusions
regarding the support provided by the currently available information for an array of
policy options that it may be appropriate to consider. The PA discusses the strengths,
limitations and uncertainties in the information and what this indicates with regard to
the public health protection expected to be afforded by the current or potential
alternative standards, considering all elements of the standard- indicator, averaging
time, form and level - collectively. This discussion also recognizes any policyjudgments
that might be involved in decisions by the Administrator for the various options. The
CASAC, in fulfilling its responsibility under section 109 of the CAA, reviews the
preliminary conclusions of the draft PA and provides its advice on the adequacy of the
current standards and any revisions that may be appropriate under the Act. The final PA
then summarizes the staff conclusions, along with the evidence-and exposure/risk-
based considerations, and the CASAC advice for consideration by the Administrator in
deciding whether to retain or revise the primary O3 standard.
2.2 REVIEW OF THE SECONDARY Os STANDARD
The approach planned for this review of the secondary O3 standard is
fundamentally based on using the Agency's assessment of the current scientific
evidence, quantitative assessments of exposures and/or risks, and other associated
analyses (e.g., air quality analyses) to inform the Administrator's judgments regarding a
secondary standard that is requisite to protect the public welfare from known or
2-4
-------
anticipated adverse effects associated with the presence of O3 and related
photochemical oxidants in the ambient air. This approach involves translating scientific
and technical information into the basis for addressing a series of key policy-relevant
questions using both evidence- and exposure-/risk-based considerations. This series of
key questions related to the secondary standard is presented below, in the context of
the general approach for the review.
The approach planned for this review of the secondary O3 standard will build on
the substantial body of work developed during prior reviews, and the associated
conclusions, taking into account more recent scientific information and air quality data
now available to inform our understanding of the key policy-relevant issues in this
review. Key aspects of the basis for the decision establishing the current secondary O3
standard in 2015 and retaining it without revision in 2020 are summarized in Volume 1
of the IRP. The ISA, REA (as warranted), and the PA developed in this new review will
provide the basis for addressing the key policy-relevant questions and these
assessments will inform the Administrator's judgments and decision as to whether to
retain or revise the secondary O3 standard.
The final decision on the secondary standard is largely a public welfare policy
judgment by the Administrator. Final decisions on the NAAQS must draw upon scientific
information and analyses about welfare effects and risks, as well as judgments about
how to weigh the array of uncertainties that are inherent in the scientific evidence and
analyses. Consistent with the Agency's approach across all NAAQS reviews, the
approach of the PA to inform these judgments is based on a recognition that the
available welfare effects evidence generally reflects a continuum that include exposures
for which scientists generally agree that effects are likely to occur through lower levels
at which the likelihood and magnitude of effects become increasingly uncertain. This
approach is consistent with the requirements of the NAAQS provisions of the CAA and
with how the EPA and the courts have historically interpreted the CAA. These provisions
require the Administrator to establish secondary standards that are requisite to protect
public welfare from any known or anticipated adverse effects associated with the
presence of the pollutant in ambient air. In so doing, the Administrator seeks to
establish secondary standards that are neither more nor less stringent than necessary
2-5
-------
for this purpose. The provisions do not require that secondary standards be set at a
zero-risk or background level or to eliminate all welfare effects, but rather to protect the
public welfare from those effects that are judged to be adverse.
With regard to the available body of scientific evidence assessed in the ISA and
used as a basis for developing and/or interpreting the exposure/risk analyses, the PA
summarizes the policy-re levant aspects, including information available in this review
that may inform policy judgments regarding the significance or adversity of key effects
on the public welfare. The PA addresses questions such the following:
• Is there newly available evidence that indicates the importance of photochemical
oxidants other than O3 with regard to abundance in ambient air, and the potential for
welfare effects?
• Does the currently available scientific evidence, including that newly available in this
review, alter prior conclusions regarding the nature of welfare effects attributable to O3
in ambient air? Is there evidence of welfare effects beyond those identified in the last
review?
• What does the evidence indicate regarding metrics supported for use in assessment of
the potential for vegetation-related effects of O3 in ambient air?
• Is there new evidence on factors that influence relationships between O3
concentrations and vegetation-related or other welfare effects?
• To what extent does the available evidence provide information on quantitative
exposure-response relationships for 03-related welfare effects that can inform
judgments on the likelihood of occurrence of welfare effects in areas with air quality
that meets the current standard, or potential alternatives, as appropriate?
• What does the available information indicate for considering potential public welfare
protection from 03-related climate effects?
• What are important uncertainties in the evidence? To what extent have important
uncertainties in the evidence identified in the last review been reduced and/or have
new uncertainties been recognized?
• What does the available information provide to inform consideration of the public
welfare implications of 03-related welfare effects? Is there newly available information
in this review?
Similarly, the PA also considers the currently available exposure and risk
information, whether newly developed in this review or predominantly developed in the
2-6
-------
past and interpreted in light of current information. In so doing, the PA addresses
questions, such as the following:
• What are the nature and magnitude of vegetation exposures associated with
conditions meeting the existing standard and potential alternative standards, as
appropriate, at sites across the U.S., particularly in specially protected areas, such as
Class I areas, and what do they indicate regarding the potential for 03-related
vegetation impacts?
• What are the important associated uncertainties?
In evaluating of the policy implications of the currently available information
including that newly available in this review, the PA considers various aspects of the
scientific evidence and exposure/risk information, related limitations and associated
uncertainties with regard to the adequacy of the existing standard and potential
alternatives, as appropriate. These evidence- and exposure/risk-based considerations
are organized around a set of questions, such as the following:
• Does the currently available information call into question the use of O3 as the
indicator for the secondary standard for photochemical oxidants? Is support provided
for considering a different indicator?
• Does information newly available in this review indicate the potential for welfare
effects of concern to occur at lower exposures than previously understood? What are
important uncertainties associated with this information?
• To what extent are the estimates of vegetation exposures, associated with conditions
meeting the current standard or potential alternative standards at sites across the U.S.,
particularly in specially protected areas, such as Class I areas, and what they indicate
regarding the potential for 03-related impacts to the public welfare, reasonably judged
important from a public welfare perspective?
• What are the important uncertainties associated with the evidence and exposure/risk
information, and the associated characterization of potential for public welfare effects?
Based on these evaluations, the draft PA develops preliminary conclusions
regarding the support provided by the currently available information for an array of
policy options that may be appropriate to consider. The PA discusses the strengths,
limitations and uncertainties in the information and what this indicates with regard to
the public welfare protection expected to be afforded by the current or potential
alternative standards, considering all elements of the standard - indicator, averaging
time, form and level - collectively. This discussion also recognizes any policy judgments
2-7
-------
that might be involved in decisions by the Administrator for the various options. The
CASAC, in fulfilling its responsibility under section 109 of the CAA, reviews the
preliminary conclusions of the draft PA and provides its advice on the adequacy of the
current standard, and any revisions that may be appropriate under the Act. The final PA
then summarizes the staff conclusions, along with the evidence-and exposure/risk-
based considerations, and the CASAC advice for consideration by the Administrator in
deciding whether to retain or revise the secondary O3 standard.
2-8
-------
3 DEVELOPMENT OF THE INTEGRATED SCIENCE
ASSESSMENT
The ISAs are intended to review, synthesize, and evaluate the scientific evidence
related to public health or welfare effects of air pollutant exposures, consistent with the
air quality criteria described in Section 108 of the CAA and with the EPA's obligation to
periodically review, and revise as appropriate, those air quality criteria under Section
109. The content of the ISA, in conjunction with additional technical and policy
assessments and advice from the CASAC and input from the public, provides the
scientific basis for the EPA's decisions on the NAAQS. This section provides information
relevant to the development of the ISA for Ozone and Related Photochemical Oxidants
as part of the current review of the primary and secondary O3 NAAQS. Sections 3.1 and
3.2 provide overviews of the anticipated organization and scope of the ISA, respectively.
Section 3.3 summarizes the planned approach to developing the ISA, including literature
search and screening efforts. Section 3.4 presents scientific questions to guide the
development of the ISA in the current review.
3.1 ORGANIZATION OF THE ISA IN THE CURRENT REVIEW
The organization of the ISA for Ozone and Related Photochemical Oxidants will
be consistent with that used in the recent assessments for other criteria pollutants (e.g.,
U.S EPA, 2020a; U.S EPA, 2020b; U.S. EPA, 2024). It will be organized around a series of
detailed, topic-specific chapters7 and an Integrated Synthesis drawn from the
information in those chapters. Chapters will provide thorough assessments of the
scientific evidence pertaining to specific topic areas, including atmospheric science,
exposure and dosimetry, various human health outcomes, and public welfare/ecological
effects. Each chapter will contain an evaluation of results from recent studies building
7 Recent ISAs used the term "appendices" to denote individual sections of the assessment. In this ISA for
Ozone and Related Photochemical Oxidants, the term "chapter(s)" will be used. Discipline-specific
chapters contribute to the Integrated Synthesis, which is a concise synopsis of ISA conclusions and a
synthesis of the findings considered in characterizing pollutant exposures and relationships with health
or welfare effects.
3-1
-------
upon key conclusions and evidence presented in previous assessments. Chapters for
each health outcome or welfare effect category (e.g., respiratory effects, cardiovascular
effects, ecological effects, climate effects) will reflect full assessments of the causal
nature of relationships between pollutant exposures and health or welfare effects that
result in key science judgments (i.e., causality determinations, see section 3.3.4). These
"causality determinations" will be based on consideration of various aspects of the
evidence, including consistency within a scientific discipline, coherence across
disciplines, biological plausibility, and other factors as discussed in section 3.3.4 and the
appendix. For human health outcomes, causality determinations will also consider the
evidence that certain populations and lifestages may experience greater risks of
pollutant-related effects due to intrinsic factors (e.g., age, genetics), acquired factors
(e.g., pre-existing disease), extrinsic factors (e.g., nutritional status, socioeconomic status,
and/or factors affecting dose or exposure (e.g., sex, age, outdoor activity) (section 3.4.6).
Chapters will additionally present targeted evaluations of the evidence on other
pollutant-specific, policy-relevant issues to support the summary discussion of those
issues in the Integrated Synthesis. These other issues vary by discipline, and often
include conclusions on concentration-, exposure-, and/or dose-response relationships;
strengths and limitations of various exposure estimates and study designs; the impact of
potential confounding factors on observed effects; and the timing of effects (i.e., lag
structure of associations and/or averaging times of exposures) associated with
exposures to O3 and related photochemical oxidants.
The Integrated Synthesis section of the ISA provides a concise synopsis of ISA
conclusions and a synthesis of the key findings considered in characterizing pollutant
exposures and relationships with health or welfare effects. The Integrated Synthesis
typically includes summaries of causality determinations for the effect categories
assessed and other relevant information such as information on pollutant-related
sources, emissions, atmospheric science, exposure, and dosimetry. The Integrated
Synthesis also summarizes the evidence with regard to factors that modify pollutant
exposure and susceptibility and populations and/or lifestages that may be at increased
risk of pollutant-related effects.
3-2
-------
In addition to the topic-specific chapters and the Integrated Synthesis, the ISA for
Ozone and Related Photochemical Oxidants will include a Preface that summarizes
major legal and historical aspects of prior NAAQS reviews, an Executive Summary
written to be accessible to a wide range of audiences, and a process chapter. The
process chapter will describe the approach taken to develop the ISA, including the
methods for literature search and review, documentation, evaluation of individual study
quality, public engagement, and quality assurance. It will also provide a general
description of the CASAC review of the draft ISA, and information on any supplementary
materials, such as information accessible through the Health and Environmental
Research Online (HERO) database for the ISA.8 The process chapter will draw from the
general approach described in the appendix of this IRP and from comments on the
appendix received from members of the CASAC O3 Review Panel as part of the
consultation with CASAC. The approach described in the appendix builds on the 2015
Preamble to the ISAs (U.S. EPA, 2015), with updates reflecting advances implemented in
recent ISAs (U.S EPA, 2020a; U.S EPA, 2020b; U.S. EPA, 2024), the EPA's consideration of
recommendations on the ISA causality framework from an ad hoc committee of the
National Academies of Sciences, Engineering, and Medicine (NASEM) (NASEM, 2022),
and comments on a previous version of the appendix by the CASAC NOx Review Panel
(Sheppard, 2024).
3.2 SCOPE OF THE ISA IN THE CURRENT REVIEW
The primary and secondary O3 NAAQS are intended to protect public health and
welfare from exposures to O3 and related photochemical oxidants (see section 3.2.1 for
a description of photochemical oxidants). Thus, the ISA developed in this review will
evaluate the available atmospheric science, human exposure and dosimetry, human
health effects, and welfare effects evidence (e.g., ecology and climate) for O3 and related
photochemical oxidants. Of this group of pollutants, O3 is the most routinely measured
and has traditionally been emphasized in the associated health and ecological effects
literature. Thus, as in previous reviews, it is anticipated that the ISA for this review will
8 HERO is a database of scientific studies and other references used to develop EPA assessments and is
available at is available at: https://heronet.epa.gov.
3-3
-------
focus mainly on the assessment of health and welfare effects resulting from exposure to
surface-level concentrations of tropospheric O3, with evidence for other photochemical
oxidants evaluated when available.
The ISA for Ozone and Related Photochemical Oxidants will evaluate relevant
studies that have become available since the cutoff-date for the 2020 ISA (i.e., March 30,
2018) in the context of studies evaluated in previous assessments (i.e., U.S. EPA, 2013;
U.S. EPA, 2009; U.S. EPA, 2006; U.S. EPA, 1996; U.S EPA, 1986; U.S. EPA, 1978). For topic
areas in which research efforts have subsided and older studies remain the definitive
works available in the literature, those older studies from previous assessments will be
the primary focus of the ISA's evaluation. The ISA will also consider studies identified in
response to the FRN call for information and studies recommended as part of the peer
input consultation or CASAC consultation/review. The sections below define the scoping
criteria to be used to screen the available scientific literature and evaluate studies for
their potential to inform the evidence assessment in the ISA. To meet ISA scoping
criteria, studies must present new information or analyses and must have undergone
scientific peer review.
3.2.1 Atmospheric Science
The ISA will present and evaluate relevant data, and summarize the current
scientific understanding based on evidence available from previous reviews and new
evidence that has emerged since the 2020 ISA concerning the sources and ambient air
concentrations of O3 in the U.S. lower troposphere and surface boundary layer. Ozone
present in the lower troposphere is predominantly formed through photochemical
reaction involving precursor gases such as reactive volatile organic compounds and NOx
(i.e., nitric oxide and nitrogen dioxide).
The ISA will use discipline-specific scoping statements to identify potentially
relevant atmospheric science studies (see Table 3-1 and the appendix). Importantly,
application of scoping statements that consider pollutant sources, transport and
transformation, exposure/extent, and measurement and modeling (i.e., STEM
statements) is consistent with current best practices for reporting or evaluating health
science data as recommended by the NASEM. The STEM statement defines the
3-4
-------
objectives of the atmospheric science evidence assessment and establishes criteria that
should be met for a study to be considered for inclusion in the ISA. A study meeting any
of the four aspects of the STEM statement will be considered for inclusion in the ISA.9
The atmospheric science STEM statement for the ISA shown in Table 3-1 has been
informed by the body of evidence from previous ISAs and air quality criteria documents
(AQCDs),10 and by expert knowledge of the relevant scientific literature.
With the focus provided by the STEM statement, the atmospheric science chapter
will present and evaluate the latest data related to emissions of precursors of O3 and
related photochemical oxidants, chemistry and transport of O3 and related
photochemical oxidants, measurement and modeling of O3 and background O3, and
spatial and temporal patterns of O3 in ambient air.
9 This contrasts with the PECOS statements used for health effects studies that require all of the listed
criteria to be met (A.3.2). The atmospheric science and exposure science fields cover diverse topics
along the source-to-exposure continuum (NRC, 2004), with each criterion corresponding to a specific
aspect of this continuum.
10 The last AQCD was published by the EPA in 2006. After 2006, Integrated Science Assessments replaced
AQCDs as the science assessments supporting the NAAQS reviews.
3-5
-------
Table 3-1. STEM statement to define the criteria and framework for identifying
relevant atmospheric studies for O3 and related photochemical
oxidants.
Statement
Description
Source (S)
Studies reporting on quantitative emissions of the precursor sources of O3 and related
photochemical oxidant precursors as well as observations of physical and chemical
characteristics that add to our understanding of the sources and emissions of O3 and
related photochemical oxidant precursors, including background O3.
Transport and
Transformation
(T)
Studies investigating atmospheric formation, transformation, transport, fate, or deposition
processes involving O3 and related photochemical oxidants, including transport of air
pollutants at various scales (i.e., national/global, regional, urban, neighborhood),
chemical transformations in the atmosphere, and estimates of atmospheric deposition
that add to our understanding of atmospheric processes relevant to U.S. conditions.
Exposure/Extent
(E)
Studies reporting observations and estimates of ambient air concentrations and their
trends for O3 and related photochemical oxidants relevant to U.S. conditions, including
spatial variability on various scales (i.e., national/global, regional, urban, neighborhood);
temporal trends such as diurnal, weekday/weekend, seasonal, and long-term trends; or
characteristics, such as composition or relationship with atmospheric properties that
provide up to date concentration estimates or add to our understanding of spatiotemporal
concentration trends.
Measurement and
Modeling (M)
Studies describing methods of measurement of O3 and related photochemical oxidants
by federal reference and equivalency methods, satellite remote sensing estimates, low-
or medium-cost sensor estimates, or research methods; and modeling techniques (e.g.,
chemical transport modeling) for characterizing O3 and related photochemical oxidant
concentrations in ambient air, including the evaluation of measurement principles and
modeling assumptions, examination of potential bias and uncertainties, and method
intercomparisons that are relevant to the NAAQS or to studies in this ISA.
The term "photochemical oxidants" is used in each of the four scoping
statements of Table 3-1 because O3 is only one of a large number of species in a
complex pollution mixture that forms from a series of chemical reactions involving
volatile organic compounds and nitrogen oxides that are initiated photochemically by
sunlight. The other components of this photochemical pollution mixture include related
photochemical oxidants, other criteria pollutants (NO2 and PM2.5), and a large variety of
other photochemically formed organic and inorganic species. When the NAAQS for
photochemical oxidants was established in 1971, the total amount of oxidants in the
atmosphere was originally measured without specifically identifying which oxidants were
present. Instead of an individual species, it was the oxidizing capacity of the
photochemical pollution mixture itself that was designated as the indicator.
3-6
-------
The 1970 Photochemical Oxidants AQCD (U.S. DHEW, 1970) precisely defined
total (photochemical) oxidants as those compounds that will oxidize a reference material
(potassium iodide) that is not capable of being oxidized by atmospheric oxygen. At the
time, the only photochemical oxidants that had been observed in the atmosphere were
NO2 and peroxyacetyl nitrate (PAN). In 1979, O3 replaced photochemical oxidants as the
NAAQS indicator as evidence emerged that concentrations of other oxidants were much
lower than O3 concentrations. Now hydrogen peroxide, other peroxyacyl nitrates
besides PAN, and peroxy compounds such as peroxybenzoyl nitrate, and
alkylhydroperoxides have also been observed as coexisting photochemical oxidants
alongside O3 in ambient air or laboratory experiments. In Table 3-1, the term
photochemical oxidants is interpreted to apply to any of these substances, but not to
oxidants that have either not been observed in the atmosphere or are too weak to
oxidize the potassium iodide reference material designated in the 1970 Photochemical
Oxidants AQCD (U.S. DHEW, 1970).
Other components of the photochemical pollution mixture like nitric acid,
formaldehyde, acrolein, formic acid and other aldehydes, and carboxylic acids are at
times referred to as oxidants in the context of oxidation-reduction reactions. Although
typically present along with O3, these species are not photochemical oxidants according
to the 1970 AQCD definition (U.S. DHEW, 1970). Studies focusing on their sources,
chemistry, concentration trends, and measurement will be considered beyond the scope
of the ISA. However, as described in the Extent statement in Table 3-1, studies on the
relationship between O3 and atmospheric composition are considered relevant, and they
also have the potential to be informative for interpreting health effects data. Therefore,
studies that focus on the relationship between O3 and other criteria pollutants or
between O3 and the overall composition or characteristics of multi-pollutant
photochemical pollution mixtures are considered within the scope of the atmospheric
science chapter of this ISA.
Some of the other components of the photochemical pollution mixture are
outside the scope of this ISA because they are included in other ISAs and their public
health and welfare impacts are considered in reviewing other NAAQS. For example, the
health effects of N02 exposure are addressed by the ISA for Oxides of Nitrogen - Health
3-7
-------
Criteria (U.S. EPA, 2016; U.S. EPA, 2024). Particulate nitrate and secondary organic
aerosol were included as part of the 2019 PM ISA (U.S. EPA, 2019; U.S. EPA, 2022).
Welfare effects of NO, NO2, PAN, and nitric acid were discussed in the 2020 ISA for
Oxides of Nitrogen, Oxides of Sulfur and Particulate Matter - Ecological Effects (U.S.
EPA, 2020a).
3.2.2 Exposure Science
Similar to the atmospheric science chapter, the scope of the exposure science
evidence assessment will be defined by a discipline-specific STEM statement (see Table
3-2 and the appendix). The ISA will present and evaluate relevant evidence related to
exposure continuums for O3 and related photochemical oxidants, characterization of
photochemical oxidant exposures, and exposures to factors that may confound
associations in epidemiologic studies (e.g., copollutants). The ISA will consider key
uncertainties from the last review and the extent to which new scientific evidence may
inform our ability to characterize and/or reduce those uncertainties during the current
review. The ISA will also evaluate the literature relating to dosimetry of inhaled O3 and
related photochemical oxidants.
3-8
-------
Table 3-2. STEM statement to define the criteria and framework for identifying
relevant exposure studies for O3 and related photochemical oxidants.
Statement
Description
Source (S)
Ambient O3 and related photochemical oxidants and indoor sources generating O3 and
contributing to O3 indoor chemistry.
Transport and
Transformation (T)
Atmospheric and environmental processes of tropospheric O3 and related photochemical
oxidants, including the transport of air pollutants at various scales (i.e., national/global,
regional, urban, neighborhood, and microenvironments), chemical transformations (e.g.,
photochemical reactions), and dynamics within microenvironments (e.g., infiltration and
indoor chemistry).
Exposure/Extent (E)
Exposure levels of tropospheric O3 and related photochemical oxidants relevant to
ambient air in the U.S., characterized by various surrogates (e.g., ambient air
concentrations, microenvironmental concentrations, personal exposure) and exposure
determinants (i.e., factors which may lead to differential exposures, such as activity
patterns and socioeconomic status), including characterizing concentrations and
spatiotemporal temporal trends of various exposure surrogates and examining
populations experiencing elevated exposures or the exposure patterns (e.g., exposure
level, duration, and frequency) experienced by populations identified in health studies as
being at increased risk of effects.
Measurement and
Modeling (M)
Measurement methods (e.g., federal reference and equivalent methods, passive
samplers, sensors, and remote sensing) and modeling techniques (e.g., land use
regression chemical transport, and microenvironmental models) characterizing ambient
air, indoor/microenvironmental air, and personal exposures, including the evaluation of
measurement principles and modeling assumptions, examination of potential bias and
uncertainties, and comparison of different techniques.
3.2.3 Health Effects
The ISA will use discipline-specific population, exposure, comparison, outcome,
study design (PECOS) statements to define the set of relevant health effects studies (see
Table 3-3 through Table 3-5 and section A.3.2 in the appendix). The PECOS statements
help to identify the objectives of the assessment and establish criteria that should be
met to consider a study for inclusion in the ISA, thereby facilitating identification of the
potentially relevant literature and informing the integration and synthesis of study
results. The PECOS statements are informed by the body of evidence from previous ISAs
and AQCDs, expert knowledge of the relevant scientific literature, and by recent ambient
air quality information (i.e., as described for exposure study criteria in Tables 3-3, 3-4
and 3-5). Studies meeting all five aspects of the PECOS statement will be considered for
inclusion in the ISA to inform human health effects.
The health chapters of the ISA will evaluate the scientific literature related to a
range of health outcomes associated with exposures to O3 including, but not limited to,
3-9
-------
respiratory effects, cardiovascular effects, metabolic effects, reproductive and
developmental effects, nervous system effects, cancer, and mortality. Building upon the
2020 ISA, the EPA will review the available epidemiologic, controlled human exposure,
and animal toxicological evidence related to these and other health outcome categories
to the extent data are available. The results of recent studies will be integrated with the
findings from the 2020 ISA along with any new interpretations of previous findings that
the recent studies may support. The ISA will also integrate previous information on the
populations and lifestages at increased risk with new evidence for existing and any
newly identified risk factors.
Table 3-3. PECOS statement to define the criteria and framework for identifying
relevant epidemiologic studies for O3 and related photochemical
oxidants.
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Population (P): Any human population, including populations or lifestages that might be at increased risk.
Exposure (E): Short-term (exposure averaging time<30 days) or long-term (exposure averaging time > 30 days)
exposure to O3 and related photochemical oxidants relevant to ambient air in the U.S. study.
Comparison (C): Per unit increase (i.e., ppb) for humans exposed to lower concentrations of O3 and related
photochemical oxidants compared to higher concentrations (e.g., categorical comparisons between different
exposure metric quantiles).
Outcome (0): Change or difference in risk (incidence/prevalence) of health outcome (e.g., respiratory effects,
cardiovascular effects, total mortality, nervous system effects, metabolic effects, reproductive and developmental
effects, or cancer) per change or difference in exposure.
Study Design (S): Epidemiologic studies, such as panel, case-crossover, time-series, case-control, cohort, cross-
sectional, and quasi-experimental studies, with appropriate timing of exposure for the health outcome of interest.
3-10
-------
Table 3-4. PECOS statement to define the criteria and framework for identifying
relevant controlled human exposure studies for O3 and related
photochemical oxidants.
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Population (P): Human volunteers enrolled in controlled exposure studies, including volunteers representing
populations or lifestages that might be at increased risk of pollutant-related health effects.
Exposure (E): Controlled inhalation exposure to O3 or other related photochemical oxidant species- in a single
session or across multiple exposure sessions. Pollutant exposures must be controlled by the experimenters and
not simply a measure of ambient or occupational exposure.
Comparison (C): An appropriate control exposure to filtered air or room air for each study participant or an
appropriately matched comparison group exposed to filtered air or room air.
Outcome (0): Outcomes of interest are those that relate to human health, including effects on the respiratory
system, cardiovascular system, immune system, nervous system, diabetes, cancer, or reproduction and
development. Effects of interest include changes in indicators or measures of physiological function, health-
relevant biomarkers, and organ structure. Effects can be directly measured in exposed study participants or in
cells, tissues, or fluids isolated from study participants.
Study Design (S): Studies that perform controlled human exposures meeting the above criteria or that analyze
data from previously conducted controlled human exposures (e.g., reanalysis, meta-analysis).
Table 3-5. PECOS statement to define the criteria and framework for identifying
relevant animal toxicological studies for O3 and related photochemical
oxidants.
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Population (P): Laboratory nonhuman mammalian animal species (e.g., nonhuman primate, mouse, rat, guinea
pig, minipig, rabbit, cat, dog) of any lifestage including models of increased susceptibility.
Exposure (E): Short-term (exposure duration <30 days) or long-term (exposure duration >30 days) exposure to
O3 and related photochemical oxidants relevant to ambient air in the U.S. (generally, <2 ppm).11
Comparison (C): An appropriate control group exposed to clean air (e.g., room air, filtered air) control.
Outcome (0): Outcomes of interest are those that relate to human health, including effects on the respiratory
system, cardiovascular system, immune system, nervous system, diabetes, cancer, reproduction and
development, or other human health effects. Effects of interest include changes in indicators or measures of
physiological function, health-related biomarkers, and organ structure. Effects can be directly measured in
exposed animals or in cells, tissues, or fluids isolated from animals.
Study Design (S): Controlled exposure studies of animals in vivo meeting the above criteria.
11 Deposition of O3 resulting from a 2-hour exposure to 2 ppm O3 in a resting rat is roughly equivalent to
deposition of O3 resulting from a 2-hour exposure to 0.4 ppm O3 in an exercising human (Hatch et al,
1994; Hatch et al, 2013). This concentration cutoff is also consistent with that used in the 2020 ISA for
Ozone and Related Photochemical Pollutants (U.S. EPA, 2020b). Experimental studies investigating the
effects of concentrations greater than 2 ppm may be considered for inclusion in the ISA if they provide
insight into biological plausibility.
3-11
-------
3.2.4 Welfare Effects
With respect to ecological effects, this ISA will build on information available
during the last review describing the effect of O3 exposure on vegetation and
ecosystems. To further refine criteria for identification of potentially relevant ecology
studies, the ISA uses Level of Biological Organization, Exposure, Comparison, Endpoint,
and Study Design (LECES). The LECES statement was developed by the EPA to provide a
Question Formulation System best suited to the nature of ecology studies, comparable
to PECOS in health studies and STEM in atmospheric studies. LECES works on the same
principle as PECOS and is a system of five binary gates that a reference must go through
in order to be accepted. A reference that fails any of the five tests (L/E/C/E/S) is rejected.
Each of the five tests may comprise several sub-tests. A reference that fails any of the
sub-tests is rejected. For research evaluating ecological effects, emphasis will be placed
on recent studies that: (1) evaluate effects of exposures resulting from O3 concentrations
comparable to those occurring in North American airsheds in order to filter out studies
for consideration in the ISA conducted in areas where O3 concentrations are much
higher than the U.S. due to higher emissions of O3 precursors like nitrogen oxides and
volatile organic compounds, different climate and geographical variables influencing
local and regional atmospheric chemistry, and where there may be fewer controls on
anthropogenic emissions sources compared to North America, and (2) investigate
effects on any individual, population (in the sense of a group of individuals of the same
species), community, or ecosystem in North America (Table 3-6). In instances when a
"causal relationship" was concluded in the 2020 ISA (i.e., visible foliar injury, vegetation
growth, reduced plant reproduction, reduced yield/quality of agricultural crops, reduced
productivity, alteration of belowground biogeochemical cycles, alteration of terrestrial
community composition), the current review will only evaluate studies conducted in
North America. For all other ecological endpoints in Table 3-6 (i.e., terrestrial water
cycling, carbon sequestration, plant phenology or mortality, herbivore growth and
reproduction, insects, other wildlife, plant-animal signaling) there are no geographic
constraints, and all available evidence will be considered for inclusion in the ISA.
3-12
-------
Table 3-6. LECES statement to define the criteria and framework for identifying
relevant ecological studies for O3 and related photochemical oxidants.
Level of Biological Organization, Exposure, Comparison,
Ecological Endpoint Endpoint, and Study Design (LECES) Tool
Visible foliar injury; vegetation growth and
reproduction; yield/quality of agricultural
crops; productivity; belowground
biogeochemical cycling; terrestrial
community composition
concentration in the test series is in the range described above.
Comparison: Relevant control sites, treatments, or parameters
Endpoint: Visible foliar injury, alteration of vegetative growth and
reproduction, yield/quality of agricultural crops, productivity,
belowground biogeochemical cycles, terrestrial community
composition
Study Design: Laboratory, greenhouse, OTC, FACE, field, gradient,
or modeling studies
Level of Biological Organization: For any species, an individual,
population, community, or ecosystem in North America
Exposure: Concentrations occurring in the environment,
experimental O3 concentrations within an order of magnitude of
recent concentrations (163 ppb was the max observed as described
in the 2020 ISA, Appendix 1, Table 1-1), or at least one
Terrestrial water cycling; carbon Level of Biological Organization: For any species, an individual,
sequestration; plant phenology, or mortality; population, community, or ecosystem in any continent3
insects, other wildlife; plant-animal signaling Exposure: Concentrations occurring in the environment,
experimental O3 concentrations within an order of magnitude of
recent concentrations (163 ppb was the max observed as described
in the 2020 ISA, Appendix 1, Table 1-1), or at least one
concentration in the test series is in the range described above.
Comparison: Relevant control sites, treatments, or parameters
Endpoint: Alteration of terrestrial water cycling; carbon
sequestration; plant phenology, mortality; growth, reproduction, and
survival of insects and other wildlife; plant—animal signaling
Study Design: Laboratory, greenhouse, OTC, FACE, field, gradient,
or modeling studies
Level of Biological Organization: Unit of study.
Exposure: Environmental variable to which population is exposed.
Comparison: Change in endpoint observed by unit increase in concentration of O3 in the same or in a control
population.
Endpoint: Measurable endpoint resulting from exposure.
Study Design: Laboratory, field, gradient, open-top chamber (OTC), free-air carbon dioxide enrichment (FACE),
greenhouse, and modeling studies.
Notes: Ecological populations are defined as a group of individuals of the same species.
a In cases where a comprehensive list of affected species was available, nonagricultural North American species were separated
out from the larger data sets and evidence on the North American species was evaluated (e.g., foliar injury, biomass).
Consistent with the inclusion of "climate" in the list of effects on welfare in
Section 302(h) of the CAA, the ISA will consider the effects of the O3 and its precursors
on climate. The ISA will focus on such effects related to tropospheric O3. The ISA will not
focus on downstream ecosystem effects, human health effects, or future air quality
3-13
-------
projections resulting from changes in climate. Studies that inform the isolated effects of
O3 in climate forcing, as well as the effects on regional climate within the U.S., are within
the scope of the literature to be considered in the review (Table 3-7). The same
principles described above for LECES apply to the PECOS statement presented in Table
3-7. Although there are inputs to the background tropospheric O3 from stratospheric
intrusion and other stratospheric processes, the focus of the ISA is on tropospheric O3 as
noted above. However, the studies included in the ISA on O3 concentration trends may
necessarily include those of the photochemical oxidants and methane that are related to
O3 formation.
Table 3-7. PECOS tool to define the parameters and provide a framework for
identifying relevant studies on the effects of tropospheric O3 on
climate.
Effect on Climate
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Changes in radiative forcing (RF)
Population/Geographical scope: evaluations of radiative forcing at the
regional, continental, and/or global scale
Exposure: tropospheric O3 concentration distributions in 3D
(observed/modeled)
Comparison: pre-industrial or other relevant baseline
scenarios/conditions
Outcome: changes in RF resulting from changes in tropospheric O3
Study Design: observations or modeling studies
Changes in climate (e.g., surface
temperature, hydrological cycle)
Population/Geographical scope: evaluations of climate effects at the
regional, continental, and/or global scale
Exposure: tropospheric O3 and precursor concentration distributions in
3D (observed/modeled)
Comparison: pre-industrial or other relevant baseline
scenarios/conditions
Outcome: subsequent climate effects (via radiative forcing) (e.g., global
surface temperature) resulting from changes in tropospheric O3
Study Design: observational or modelling studies
PoDulation/GeograDhical scope: group of organisms/spatial extent of studv
Exposure: environmental variable (concentrations of tropospheric Os and precursors)
Comparator: radiative forcing or climate effects observed from unit change in tropospheric Ch concentration.
Outcome: relevant radiative forcing or climate outcomes resulting from change in tropospheric Ch.
Studv design: observations (ground, ozonesonde, satellite), modelling.
3.3 PROCESS FOR DEVELOPING THE ISA
This section summarizes the anticipated approach to developing the ISA for O3
and related photochemical oxidants in the current review. This approach draws from the
more detailed discussion of ISA development in the appendix to this IRP. As noted
3-14
-------
previously, this approach builds upon the 2015 Preamble to the ISAs (U.S. EPA, 2015),
with updates reflecting advancements implemented in recent ISAs (U.S EPA, 2020a; U.S
EPA, 2020b; U.S. EPA, 2024), the EPA's consideration of recommendations on the ISA
causality framework from a NASEM ad hoc committee (NASEM, 2022), and comments
on a previous version of the appendix by the CASAC NOx Review Panel (Sheppard,
2024). Comments on this IRP and appendix the CASAC O3 Review Panel will additionally
be considered in developing the draft ISA for Ozone and Related Photochemical
Oxidants in this review. The sections below provide a high-level overview of the updated
ISA development process as it is being applied in the current review and an overview of
initial results of the literature search and screening efforts.
The process for developing the ISA for Ozone and Related Photochemical
Oxidants began when the Call for Information was published in the Federal Register in
2023 (88 FR 58264, August 25, 2023). At that time, the public was invited to contribute
to the review by commenting on policy-relevant issues and by submitting potentially
relevant research studies.12 Additionally, a 4-day virtual workshop was held on May 13-
16, 2024 (89 FR 31200, April 24, 2024), to inform this review of the NAAQS for O3 and
related photochemical oxidants. The workshop was open to the public, and experts from
various disciplines, including atmospheric science, exposure assessment, toxicology,
epidemiology, ecology, climate, and risk analysis were invited to discuss new research or
innovative analyses that address a variety of policy-relevant issues related to reviewing
the O3 standards. Experts' input and public comments have been considered in
developing this IRP, and research studies identified by public commenters have been
included in initial literature screening efforts. The sections below summarize key steps
involved in developing the ISA for Ozone and Related Photochemical Oxidants in this
review, including searching the scientific literature and identifying potentially relevant
studies, evaluating individual study quality, integrating evidence, developing causality
12 Public comments were submitted to the docket for the Integrated Science Assessment as a part of the
review of the primary and secondary NAAQS for O3 and related photochemical oxidants (Docket ID
Number: EPA-HQ-ORD-2023-0435). This docket can be accessed at: https://www.regulations.gov/docket
/EPA-HQ-QRD-2023-0435.
3-15
-------
determinations, quality management, and obtaining the CASAC's advice. Each of these
steps is discussed in greater detail in the appendix of this IRP.
3.3.1 Literature Search
The EPA works to identify potentially relevant new studies for inclusion in the ISA
by conducting multipronged systematic searches that include extensive mining of
literature databases related to the ISA disciplines. Potentially relevant studies will also be
identified from those submitted by members of the public in response to the Call for
Information (88 FR 58264, August 25, 2023) and those recommended by the CASAC.
For the literature searches, the EPA will apply systematic review methodologies to
identify peer-reviewed scientific studies that are potentially relevant to this ISA. The
literature search is a combination of two types of literature identification techniques,
forward citation searching and broad keyword searches. Both of these techniques will be
used to search bibliographic databases (i.e., PubMed and Web of Science) and both will
be limited to the timeframe of 2018-present. Forward citation searching uses a set of
curated relevant published references (i.e., references that were cited in previous ISAs) as
a seed set to identify potentially relevant references. This seed set comprises peer-
reviewed references cited in previous ISAs and AQCDs. The previously mentioned
databases are then searched for any references that cite at least one member of the
seed set.
The second phase of literature searching will consist of a broad keyword search
targeting references relevant to the pollutant. We plan to employ topic classification
using a curated seed set specific to each discipline to identify potentially relevant papers
in the keyword search result set. Some disciplines may not have a seed set that provides
the topic classifier tool with an accurate picture of what relevant papers look like. Those
disciplines will employ a more targeted literature search, combining discipline-specific
keywords along with the pollutant keywords. Results from the keyword search and
forward citation search will be combined and deduplicated to form the results set that
will be screened as described below.
3-16
-------
3.3.2 Identifying Potentially Relevant Studies
New studies identified during the literature search, the call for information, and
from CASAC will be evaluated for potential relevance using a multi-tiered literature
screening approach designed to maximize efficiency while simultaneously ensuring
relevant studies are identified. Studies are initially evaluated for potential relevance by
comparing their titles and abstracts to the discipline-specific scoping criteria defined by
STEM, PECOS, or LECES statements (A.3). Because the number of criteria pollutant-
related studies identified in initial literature searches can be very large (e.g., hundreds of
thousands of studies across disciplines and outcomes), machine-assisted ranking
literature review software tools (e.g., DistillerSR (Hamel et al., 2020); SWIFT-Active
Screener (Howard et al., 2020); Living Literature Review (U.S. EPA, 2023)) are employed
to maximize screening efficiency. Final study inclusion and exclusion decisions reached
while using these tools are documented in the Health and Environmental Research
Online (HERO) database.13 Studies that appear to meet the ISA scoping criteria based on
the title and abstract screen, together with studies that cannot be definitively identified
as out of scope, are retained for further evaluation of the full text.
To gain some early perspective on the evolution of the scientific literature since
the 2020 ISA, topic maps have been developed from initial literature search results for
each of the health disciplines, atmospheric science, and exposure disciplines. The topic
maps were created using VOSviewer visualization software (Van Eck, 2023). Each topic
map shows a network of keywords pulled from the results of the Web of Science
forward citation search for each discipline. Due to the number of keywords generated by
the search, only keywords with 20 or more occurrences in the reference set are shown in
the topic maps. The keywords were vetted by scientists for each of the disciplines to
generate the final topic maps.
Topic maps as shown below give a top-down look at what research areas we
might expect to see while screening literature search results. While this is not a
comprehensive list of topics that will be discussed in the ISA, it helps to anticipate the
breadth of topics and what topics might have a larger set of literature. It also shows how
13 Available at https://hero.epa.qov
3-17
-------
the disciplines have changed over the years being covered in the ISA; the color scheme
of the visualizations represents the years that the literature search spans, 2018-present.
The size of the word node represents the relative number of papers that use that word
as a keyword; spacing between nodes represents frequency of co-occurrence. Figures
3-1 to 3-5 below show the topic maps and an explanation of the data being
represented.
( januag-ternn trends J ( emission Inventories ]
f chemical characterization 1
[ air pr.ll, itinn ^ ^ I 1
[ ^oaivanabiiity ) ( boundary layer )
r~ TT~i ij *>imt tmninn' 1 ("^T3n5pOft~j ( 5 _] ( . ) f nonmethane hydrocarbons 1
I data assimilation p0"em'"lons J -7rir,. 1— ,—
f uncertainty j
aiiry~T
[ dry deposition ] f > ( halogen chemls^]
•- ' I relative humidity J
f 1 ( variability ] [ modeling prof'le> J [ ozone formation ) L parogen perf
( time series ) 1 nitrogen dioxide ] 1 ^ f permyacetyi nitrate ) (>OtOOXid^
( sermnei-5 J ( ) (nitrous and ) ( radical chemistry j
® C^emlsslonsInventories j [ atmospheric chemistly
1 J , ^ Tchemical mechanisms I ( toioen®
——^ [ satellite ] [ emissions |ePQ5|ti°h~l
( tropomi ) [ max-dods measurements ] f^T) ^
r inve^on 1 ( [_mtrlc oxide j [oxidation
[ temperature^ votIjH,q nro3nir rnmnn„nHc 1
(v»lid«aDn ) , ^ , I,» L^^^^fvdatile organic compounds ]
^ J [ remote sensing ) L b°ttom-VP ) [ emission factors ] [ detomposioS^"] q
( chemical mechanism ) ( organic paroy radicals")
—["amrnsi vanar cn^ ) ———— ' [ biogenic volatile organic comp ]
[ retrieval ]
V • - — / . . 1 cnemicai mecnamsm I
[ blomass burning J
f^iT) (¦ larHi-USe J [ formaldehyde [ quantification ] ( hydmperojyl )_
[ algor ithm j f trace gases ] , s ^
( spectroscopy J
, f fire emissions J f methane oxidation 1
f wildfire J ( , , ">—' " " —'
C^T) K ' ( oil and gas )
B ( . , [ )
^ Biwcnbttf tr» ]
Figure 3-1. Topic map for atmospheric science (137 keywords total).
3-18
-------
( available sensors'")
( field calibration""^
f neural networks )
f deep learning^
f air quality monitoring )
( random forest ) f 5en-,or5 ^ low-cost sensors^)
( interpolation )
tropomi )
SD
f data fusion )
[ machine learning )
measurement error
Q_
[ cmaq )
("rf d"m)
satellite j
( models )
( spatial variability )
CED
[ vyildfires j
[tropospheric ozone) [ air quality [ OZOne J
II \ r J- r health impact assessment I
[ urban ]
( sensitivity ] [ "°* 1 [ climate ]
peurope J '
( epidemiology ) ( use-crossover )
[ lur ) ( exposure assessment ] ( time-series ]
( no2 ] r^~l (IT)
air pollution
f environmental justice )
O
[ atmospheric chemistry )
/ ^-~,rEEai emissions i , .
[ transport ]—^ ^ [ climate change J
f boundary layer J
^ haze j
( road ) 1 short-terre
| pi~pi2 5 | [ long-term exposure ]
- f emission factors 1 / ¦>
( trapcph.ric degradation ) ( aE^nfc jW" ~tong KonS J
( ..rem.i.y )
, J S°° )
f (—susceptibility ]
[ maternal exposure 1 '
( vegetation )
[ traffic ) [ ambient-temperature ] ( cohort study )
[ jack carbon")
I critical windows
[ birth weight ]
( mitigation 1 co2 V- : y— ¦, [ greenspace J
[ oxidation ) ^k { dispersion X du5t J , behavior 1 ( physical activity )
^ / v f prenatal exposure 1
[ r1»ti». totally J | size distribution ] [ popul.tion cposu.e ) I ¦
fvcics") ( ambient air ]
^ deposition )
( formaldehyde )
G
D
[ ^PS ] ^ ^ [ averting behaviofdiesel exhaust")
[ personal exposures )
( energy consumption ) ( health risk assessment )
[ indoor )
G*D [ indoor air quality ]
[ buildings )
[ smoking )
f ventilation 1 r—-——
^ > ( schools J
f homesJ
f infiltration )
^ penetration )
Figure 3-2. Topic map for exposure sciences (130 keywords total).
3-19
-------
( global health )
f burden of disease J
(wdT)
( heart failure!
( Stroke ) [ JCTiylwrMm cMIM.nl Hdtsnrcl )"
[ cardiopulmonary mortality j
—£ coronary heart disease j
[ critical windows 1
f congenital anomalies 1
arterial stiffness |
I blood pressure
f metabolic syndrome")
[ heart ratevariability |
[ non-accidental mortality) cardiovascular disease
( life-style j
diabetes
r longitudinal study 1 [ prenatal ^
f gestational age I
^ ^ birth weight jjirt^ r" 'tvnmac
lung cancer
( superoxide-dismutase j
f fertility j
mortality (ancer I
( vitamm e |
pregnancy outcomes
[ fetal growth )
v .....carriage )
( progesterone ] Z .
' ( cervical length J
:iirth
f hospitalization'
f polymorphisms
i
I
adiposity j
j pulmonary func
[ long term 1
( biomarkers )
I ">Pd )
psychological effects
[ smokingj
( interstitial lung disease j
( population health J , —
¦ J I pulmonary fibrosis 1
f respiratory disease 1
fiii^D k ¦
cytokines
^ endoplasmic-reticulum s
( cell-death
^ nutrition J
J [ infection ]
[ fetal/infanc mofcalicy |
( adhd )
^ autism j
( Kh'-°Phrgn'a ) ^ neurodeve I oproenta I aisordeiT")
[ fetal membranes]
^ central nervous system j
(respiratory infection ] [ lung function ] f chj|dren
[ emergency department visits j
multiple sclerose
parkinson s disease j
f alzheimers disease 1
( pediatric j
( phanotype ( B«clw-dfflWw>c«s ) ( innate immunity )
f microbiota ^
asthma
^ pulmonary exacerbations j
c
bronchoalueolar I
f dendritic cells j
( cognitive effects |
( memory j
( neural networts J f
[ allergic response ]
f atopic di
[ allergies )
[ whee:ing~)
2018
2020
2022
2024
Figure 3-3. Topic map for epidemiology (139 keywords total).
3-20
-------
( heart rate variability J
[ blood pressure )
( hypertension ]
( short-term exposure )
[ cardiovascular disease ]
[ adlpose-tissue~]
( diabetes-mellltus ) ( insulin resistance ) [ inmiena ]
( atherosclerosis j
(ot-lij.) ( lung function 1
( long-term eaposure ) » rSStaTl
[ covid-19 j '
1 ) (T5S5SST)
(tmmiM
( metabolism )
[ cognitive function ] ( pulmonmy jurfactanr )
(T^-l (suspense) ® [asthma]
, , . f inflammation ] , .
I neuroinflammation J I. J I allergy J
( microglia ) , , , >
( mitochondrial dysfunction )| OXldatlVe StreSS I I ""mune response ]
iy responsiveness |
[ Parkinson's disease~) f.aBWtfc.'Mgcte) I receptor esporisivenes; |
(. brjjn ) (jgggjgj^) Q^Tj Lvitamin eJ
L lung inflammation ]
. [ alzheimer's disease") |gg||| (nrf2 ) ( lung injury J UffiffifiBlBy
(. t-cells ]
f neurodegenerative diseases J
[ neurotoxicity ] [ dna-damage )
( macrophages |
( autophagy ')
2022
2024
2018
2020
Figure 3-4. Topic map for toxicology (76 keywords total).
For welfare effects, key topic areas (derived from previous ISAs) that will be used
to organize the forthcoming ISA have been identified and separate literature searches
were already conducted for each topic area. Figure 3-5 illustrates the number of studies
identified by these forward citation searches in Web of Science for each topic.
3-21
-------
# studies
Ecological Science
bJD
C
"to
c
"D
C
<~5
<0
+¦»
to
o
0)
OJ
i-
in
0
£
IE
L_
4->
£
0)
ro
X
CL
1133
cud
c
~D
O
-a
c
CD
$
O
i-
c
0
Q.
t/i
a»
a:
1
0)
j_
3
C
o
U
3
"D
O
a
CUD
_o
O
c
aj
1
«j +¦»
> .2 <5
.t £ =
+? -c
u
3
"D
O
£* W5
3 .£
£ o
o
aj
tXD
O
-Si
_D
c
o
E
o
CQ
"D
C
(0
331
418
596
CUD
£
u
>
U
i-
-------
the evaluation of individual study quality will be captured in flow diagrams that
document the number of references identified from each database searched (i.e.,
PubMed and Web of Science), the number of references evaluated in each screening
step, the number of studies included and excluded at each step, and general reasons for
reference exclusion.
3.3.4 Integration of Evidence and Determination of Causality
The ISA for Ozone and Related Photochemical Oxidants will evaluate and
integrate the recent scientific evidence on the health and welfare effects of O3 exposures
with evidence from previous assessments. Based on this integration, the ISA will reach
conclusions on the weight-of-evidence supporting cause-effect relationships between
exposures and various health outcomes and welfare effects. These "causality
determinations" reflect overall confidence in such cause-effect relationships based on
integrating the full body of evidence within and across disciplines. The ISA framework
for reaching these causality determinations recognizes that, compared to any single
study, the availability of multiple studies evaluating a particular topic, each with different
strengths and limitations, provides a more robust foundation for evaluating the overall
strength of the evidence. To aid in forming weight-of-evidence judgments, the ISAs
consider various aspects of the scientific evidence including consistency and coherence
across studies, support for biological plausibility and underlying mechanisms, support
for exposure- or dose-response relationships, and several others (Appendix, section
A.7.2.1). Limitations in the evidence base can result from the presence of similar
uncertainties within a particular subset of studies (e.g., studies similarly affected by
confounding, exposure error, species extrapolation) or uncertainties that exist across the
broader body of evidence (e.g., inconsistent evidence across disciplines). When the
evidence base includes a group of studies with the same or similar uncertainties of a
particular type, caution is used when developing causality determinations so as not to
misrepresent and perpetuate errors (Savitz and Forastiere, 2021, Savitz et al., 2019).
The ISA causality framework for this review builds on the established framework
described in the 2015 Preamble to the ISAs (U.S. EPA, 2015), with updates reflecting
advancements implemented in recent ISAs (U.S EPA, 2020a; U.S EPA, 2020b; U.S. EPA,
3-23
-------
2024), the EPA's consideration of recommendations on the ISA causality framework from
a NASEM ad hoc committee (NASEM, 2022), as well as the input from the CASAC NOx
Review Panel (Sheppard, 2024). It includes a five-level hierarchy to classify the weight-
of-evidence for causation as either causal; likely to be causal; suggestive of, but not
sufficient to infer, a causal relationship; inadequate to infer a causal relationship; and not
likely to be a causal relationship (Table 3-8). This causality determination framework is
applied to both human health effects and public welfare effects, consistent with NASEM
advice (NASEM, 2022). The ISA causality framework is described in detail in the appendix
(section A.7.2.1) to this IRP.
3-24
-------
Table 3-8. Causality determinations for health outcomes and welfare effects.
Descriptor
Health Evidence Characteristics
Ecological and Other Welfare
Evidence Characteristics
Causal relationship
Evidence is sufficient to conclude that
there is a causal relationship with relevant
pollutant exposures. That is, the pollutant
exposures have been shown to result in
health effects across studies in which
chance, confounding, and other biases
can be ruled out with reasonable
confidence. A "causal" relationship is
generally based on multiple high-quality
studies conducted by different research
groups. Evidence supporting this
determination can include controlled
human exposure studies that consistently
demonstrate effects and/or observational
studies reporting consistent health effect
associations that, when considered in light
of study quality and coherence with other
lines of evidence (i.e., controlled human
exposure studies, animal toxicological
studies, and mode of action information),
cannot be explained by plausible
alternatives.
Evidence is sufficient to conclude that
there is a causal relationship with
relevant pollutant exposures. That is,
the pollutant has been shown to result in
effects across studies in which chance,
confounding, and other biases could be
ruled out with reasonable confidence.
Controlled exposure studies (laboratory
or small- to medium-scale field studies)
provide the strongest evidence for
causality, but their scope of inference
may be limited. Generally, the
determination is based on multiple
studies conducted by multiple research
groups, and evidence that is considered
sufficient to infer a causal relationship is
usually obtained from the joint
consideration of many lines of evidence
that reinforce each other.
Likely to be a
causal relationship
Evidence is sufficient to conclude that a
causal relationship is likely to exist with
relevant pollutant exposures. That is, the
pollutant exposures have been shown to
result in health effects in studies where
chance, confounding, and other biases are
minimized but uncertainties remain in the
evidence overall. A "likely to be causal"
relationship is generally based on multiple
high-quality studies conducted by different
research groups. Evidence supporting this
determination can include 1) multiple high-
quality observational studies consistently
reporting health effect associations, but
with uncertainty remaining related to
potential confounding and/or limited
coherence with other lines of evidence
(i.e., controlled human exposure studies,
animal toxicological studies, mode of
action information) or 2) consistent
evidence in animal models and/or in vitro
models (e.g., for cancer-related effects)
that can be reasonably extrapolated to
human health, but limited availability of
human data.
Evidence is sufficient to conclude that
there is a likely causal association with
relevant pollutant exposures. That is, an
association has been observed between
the pollutant and the outcome in studies
in which chance, confounding, and other
biases are minimized but uncertainties
remain. For example, field studies show
a relationship, but suspected interacting
factors cannot be controlled, and other
lines of evidence are limited or
inconsistent. Generally, the
determination is based on multiple
studies by multiple research groups.
3-25
-------
Suggestive of, but
not sufficient to
infer, a causal
relationship
Evidence is suggestive of, but not
sufficient to infer, a causal relationship with
relevant pollutant exposures. That is, the
pollutant exposures have been shown to
result in health effects, but chance,
confounding, and bias cannot be ruled out
with confidence. Evidence supporting a
"suggestive" relationship can be comprised
of studies of varying quality that may be
generally supportive of pollutant-related
effects, but not entirely consistent and with
limited coherence across lines of
evidence. A suggestive determination can
be reached with relatively small bodies of
evidence or, in rare cases, one high quality
study.
Evidence is suggestive of a causal
relationship with relevant pollutant
exposures, but chance, confounding,
and bias cannot be ruled out with
confidence. For example, at least one
high-quality study shows an effect, but
the results of other studies are
inconsistent.
Inadequate to infer
a causal
relationship
Evidence is inadequate to determine that a
causal relationship exists with relevant
pollutant exposures. That is, the evidence
supporting an "inadequate" relationship is
limited and available studies are of
insufficient quantity, quality, consistency,
and/or statistical power to permit a
conclusion regarding the presence or
absence of an effect.
Evidence is inadequate to determine that
a causal relationship exists with relevant
pollutant exposures. The available
studies are of insufficient quality,
consistency, or statistical power to
permit a conclusion regarding the
presence or absence of an effect.
Not likely to be a
causal relationship
Evidence indicates there is no causal
relationship with relevant pollutant
exposures. Several adequate studies,
covering the full range of exposures that
human beings are known to encounter
and considering at-risk populations and
lifestages, are consistent in not showing
an effect at any level of exposure
concentration.
Evidence indicates there is no causal
relationship with relevant pollutant
exposures. Several adequate studies
examining relationships with relevant
exposures are consistent in failing to
show an effect at any level of exposure.
3.3.5 Quality Management
The EPA has an agency-wide quality assurance (QA) policy outlined in the EPA
Environmental Information Quality Policy (see CIO 2105.4) supported by the EPA
Environmental Information Quality Procedures (CIO 2105-P-01.4). As required by CIO
2105.1, the EPA Office of Research and Development (ORD) maintains a Quality
Management Program, which is documented in an internal Quality Management Plan
(QMP). The ISAs are designated as Highly Influential Scientific Assessments (HISA) and
are classified as ORD QA Category A. As such, the O3 ISA is subject to the EPA's Quality
Management Program requirements for a Quality Management Plan and adheres to the
3-26
-------
Programmatic Quality Assurance Project Plan (PQAPP) for the Integrated Science
Assessment Program, (QAPP ID: L-HEEAD-0030253-QP-l-7). The ISA will be subjected to
management and QA clearance review, and during this step, the CPHEA QA Manager
verifies that the EPA QA requirements are met (see the appendix for more detail).
3.3.6 CASAC Peer Review
Section 109(d) of the CAA establishes the requirement for an independent
scientific committee to review the Air Quality Criteria (contained in the ISA in the current
process) and the NAAQS, and to recommend to the EPA Administrator any new NAAQS
and any revisions to existing criteria and NAAQS that may be appropriate (see 42 U.S.C.
7409(d)(2)). The Clean Air Scientific Advisory Committee (CASAC) was established to
fulfill these requirements, and a draft of the ISA will be sent to the CASAC for review.
Coincident with the CASAC review, the draft ISA will also be made available to the
public, with a Federal Register notice announcing public availability and providing
instructions for submitting comments on the draft ISA to the public docket.
As described in the appendix (section A.8), the CASAC will be supplemented by
an expert panel that includes broad scientific expertise on O3 and related photochemical
oxidants and on the science-policy issues important for this review of the primary and
secondary O3 NAAQS. The panel will develop a draft advisory report with
recommendations for the EPA Administrator on the draft ISA. The report will be
transmitted to the CASAC for discussion and deliberation. If the CASAC determines the
contents of the report are appropriate, the committee will adopt the report and transmit
it to the EPA Administrator to reflect its statutorily mandated advice to the Agency.
The EPA will carefully consider advice received from the CASAC and comments
from the public in revising and updating the draft ISA. After appropriate revisions are
made, the final ISA will be made available on the EPA website. A notice announcing the
availability of the final ISA will be published in the Federal Register.
3.4 SCIENTIFIC QUESTIONS TO GUIDE EVALUATION OF THE EVIDENCE
As noted above, the ISA for Ozone and Related Photochemical Oxidants being
developed in this review will build upon the evidence assessed and the conclusions
reached in the 2020 ISA and prior assessments. Studies that have become available since
3-27
-------
the 2020 ISA will be integrated with older studies evaluated in previous assessments.
Based on the recent evidence and considering advice from CASAC, conclusions from the
2020 ISA will be re-evaluated. A series of scientific questions will guide evaluation of the
recent literature, with a focus on 1) whether new scientific evidence reinforces or calls
into question the conclusions reached in the 2020 ISA; 2) whether uncertainties from the
last review have been reduced and/or whether new uncertainties have emerged; and 3)
the degree to which new lines of evidence have become available to support areas of
evaluation not considered in previous assessments. The specific scientific questions that
will guide evaluation of the literature for each discipline are described in the sections
below.
3.4.1 Source to Concentration - Air Quality, Atmospheric Science, Fate, and
Transport
The ISA will present and evaluate data related to ambient air concentrations of O3
and related photochemical oxidants; precursor sources leading to the presence of O3
and related photochemical oxidants in the atmosphere; and physical and chemical
processes that determine the formation, degradation, and lifetime of O3 and related
photochemical oxidants in the atmosphere. The following questions will guide
evaluation of the scientific literature for air quality, atmospheric science, and fate and
transport:
• What new information is available to inform our understanding of the atmospheric
chemistry of O3? What new information is available with respect to formation,
transport, and transformation of O3 that may be important in assessing health and
welfare effects from multipollutant exposures? How do precursor sources (e.g.,
methane, volatile organic compounds) influence chemistry and spatiotemporal
variability of O3?
• What are the relevant spatial and temporal scales for considering emissions of
precursors to O3 in ambient air? What new information is available regarding existing
and emerging energy, industrial, transportation, and agricultural sources of
precursors and their impacts on O3 formation? To what extent do wildland and
prescribed fires impact urban, suburban, and rural ambient O3 concentrations?
• To what extent have new methods been developed to improve measurements of O3
in ambient air? How have these new methods reduced interference problems in
measuring O3? What advances have taken place in the development of low-cost
3-28
-------
community sensor technologies? What advances have taken place in the
development of satellite-based remote sensing technologies? What limitations
remain?
• What new modeling methods and refinements have been developed that improve
our understanding and predictive capabilities of spatial and temporal patterns of O3?
• Based on recent air quality and emissions data, what is known about recent
emissions and resulting ambient air concentrations of O3? How have precursor
emissions sources and concentrations of O3 changed since the 2020 ISA? To what
extent can new data sources (e.g., satellites, community sensors) or air quality
analyses be used to improve the characterization of ambient air concentrations of
O3? To what extent can new tools from satellites and hybrid or fusion models be
used to characterize ambient air O3 concentrations?
• What spatial and temporal patterns can be observed in ambient air concentrations of
O3? What do ambient air quality characterizations (including the influence of
meteorological parameters) indicate about spatial patterns on neighborhood, urban,
regional, and national scales?
• Based on air quality and precursor emissions data for O3 and atmospheric chemistry
models, what improvements have been made in estimating background
concentrations of O3, and what are likely background concentrations in the absence
of anthropogenic emissions?
• What information is available on interactions between O3 and other photochemical
oxidants in the atmosphere that may alter the spatial distributions of O3?
• To what extent have uncertainties in data, modeling, and measurements been
reduced from the previous reviews?
• What effects have pandemic related lockdowns, increasing temperatures due to
climate change, and increasing wildland fire activity had or have on precursor
emissions and ambient air concentrations of O3?
3.4.2 Human Exposure
The ISA will evaluate factors that influence human exposure to O3 and related
photochemical oxidants of ambient air origin and factors that influence measurement
error and other uncertainties associated with extrapolation of ambient air
concentrations to personal exposures, particularly in the context of interpreting results
from epidemiologic studies. The following questions will guide the evaluation of the
scientific literature for exposures to O3 and related photochemical oxidants:
3-29
-------
How have personal or microenvironmental exposure measurement techniques for O3
and related photochemical oxidants, such as sensors and passive samplers, been
advanced in recent years? How have microenvironment-based exposure models
been advanced to estimate personal exposures in various microenvironments? What
measurement errors are associated with these emerging techniques?
How have modeling or hybrid modeling techniques, such as sub-grid scale modeling
within chemical transport models, air quality dispersion models, and land use
regression models, advanced in recent years? What new information is available
regarding modeled estimates of spatially resolved (e.g., at the micro-, middle-, and
neighborhood-scales) ambient air O3 and related photochemical oxidant
concentrations used for exposure assessment?
To what extent have data fusion approaches that combine ambient air concentration
measurements with air quality models recently been developed to improve the
spatial and temporal resolution of exposure estimates to O3 or related
photochemical oxidants within a community? What advancements have been made
regarding validation of these techniques?
What have newer methods of acquiring time-activity data (e.g., cell phone data,
social media location data) contributed toward new understanding of total exposure
to O3 and related photochemical oxidants?
What new information is available regarding the fate and transport of O3 and related
photochemical oxidants indoors in relation to human exposure to O3 and related
photochemical oxidants of ambient origin, including reactions with surfaces (e.g.,
human skin) and airborne organic compounds?
How have trends related to climate change and changing seasonality of O3 (e.g.,
opening of windows, increased use or air conditioning, frequency of wildfires)
impacted personal exposure to O3 and related photochemical oxidants?
What new information exists regarding characterization of exposure measurement
error in assessment of short-term and long-term exposures to O3 and related
photochemical oxidants and how does that error influence personal-ambient air
exposure relationships? What implications does exposure measurement error have
on inference about epidemiologic associations observed between photochemical
oxidants and health effects? Do the implications vary according to factors such as
exposure duration, study design, and exposure assessment method?
What is the relationship between O3 measured at stationary monitoring sites and
personal short-term and long-term exposure? What evidence is available regarding
these relationships in environments near sources of pollutants that interact with or
participate in the formation of O3 (e.g., NOx, VOCs)?
3-30
-------
• What new information exists regarding exposure to O3 and related photochemical
oxidants in a multipollutant context with other gaseous pollutants (e.g., NOx, VOCs),
and particle phase pollutants (e.g., ultrafine particles and carbonaceous species)?
- How does information about pollutant co-exposures aid in evaluation of
potential confounders in epidemiologic associations between O3 and related
photochemical oxidants and health effects?
- What new information exists about the relationship between O3, NOx, and VOC
concentrations and indicators of near-source pollution including distance to
sources of copollutants (e.g., major roadways) that interact with or participate
in the formation of O3 and these source activity levels (e.g., traffic counts)?
• What new information is available regarding differences in exposure patterns for O3
and related photochemical oxidants and personal-ambient air exposure relationships
among various lifestages and specific groups within populations?
- What new information is available on spatial and temporal trends in exposures
to O3 and related photochemical oxidants in ambient air, particularly for
groups and lifestages that may be at increased risk of health effects?
- What implications do potential differences in exposure measurement error
have on inferences about relationships with health effects observed in general
population studies versus those conducted in specific lifestages and groups
within the population (e.g., people with underlying health conditions)?
3.4.3 Dosimetry
The ISA will evaluate literature focusing on dosimetry for O3 and related
photochemical oxidants. In general, dosimetry refers to quantification of the amount of
O3 retained in the lung (i.e., not exhaled) or the dose of O3 or its active metabolites (e.g.,
aldehydes or peroxides) delivered to target cells or tissues. A dosimetry section was not
included in the 2020 ISA. Instead, the 2020 ISA referenced the 2006 AQCD and 2013 ISA
as the sources providing an "extensive background on dosimetry and potential
pathways underlying health effects for these responses." For the ISA in this review, the
following questions will guide the evaluation of the scientific literature for dosimetry:
• What new information is available on the effects of host factors such as lifestage, sex,
pre-existing disease, genetic background, and physical activity on the uptake,
metabolism, and elimination of O3 and/or its reaction products?
• What new information is available to discern the relative contributions of O3 and its
reaction products in the respiratory tract?
3-31
-------
• What new information is available that improves the ability to quantify with a
reasonable degree of certainty the amounts of O3 or its reaction products that reach
epithelial cells, penetrate into tissues, or enter systemic circulation?
• Dosimetric models for O3 have provided cross-species extrapolations. What new
studies are available for O3 that improve the ability to extrapolate among species
with consideration to known differences such as antioxidant levels among species?
• Is new information available for inhalation dosimetry of other 03-related
photochemical oxidants?
3.4.4 Biological Plausibility
The ISA will evaluate literature focusing on biological mechanisms that may
underlie the health outcomes associated with exposure to O3 and other related
photochemical oxidants. These topic areas will be evaluated using both human and
animal data. The following questions will guide the evaluation of the scientific literature
related to biological plausibility:
• What new information is available to inform our understanding of the potential
biological mechanisms underlying responses to O3 exposures, or exposures to other
related photochemical oxidants, at concentrations defined in the ISA to be policy
relevant (see Tables 3-3 to 3-5), with a focus on response pathway(s) and exposure-
dose-response relationships?
• What information is available to characterize intra- and inter-individual variability in
biological responses following exposure to O3 and/or related photochemical
oxidants?
• What are the effects of host factors such as lifestage, sex, pre-existing disease, and
genetic background on cellular and tissue responses, as well as biological
mechanisms, that may underlie health effects associated with exposure to O3 and/or
related photochemical oxidants?
• What biological processes, from the molecular to whole organ level, can be
qualitatively or quantitatively compared across species (i.e., human vs. animal)?
• Do other inhaled air pollutants interact with or influence the mechanisms underlying
the health effects of O3 and/or other related photochemical oxidants? If so, how
might this information provide understanding of the potential for a copollutant to
act as an effect measure modifier of health effects related to O3 and/or related
photochemical oxidants?
3-32
-------
3.4.5 Health Outcomes
The ISA will evaluate health effects that occur following both short- and long-
term exposures to O3 and related photochemical oxidants as examined in
epidemiologic, controlled human exposure, and animal toxicological studies. The health
effects evidence will be integrated with available information on exposure, dosimetry,
and biological plausibility to inform the key ISA conclusions, including causality
determinations. The evidence integration will focus on (1) whether recent studies
support or call into question the causality determinations and other conclusions made in
the 2020 ISA; (2) whether recent evidence supports causality determinations or other
conclusions for health outcomes not included in the 2020 ISA; and (3) whether new
evidence reduces uncertainties identified in the last review and whether additional
uncertainties have been identified.
Causality determinations from the 2020 ISA are summarized in Table 3-10 below.
The strongest evidence supported a "causal" relationship between short-term O3
exposures and respiratory effects. In particular, controlled human exposure studies
provided evidence of lung function decrements, increased respiratory symptoms, and
increased markers of inflammation in healthy volunteers exposed to 6.6 hours of
average O3 concentrations of 60 or 70 ppb with quasi-continuous exercise. Dose-
dependent increases in airway responsiveness were also noted after exposures to
approximately 80, 100, and 120 ppb O3. This evidence was supported by epidemiologic
studies that consistently showed positive associations between increased ambient air O3
concentrations and respiratory effects in healthy populations, but also provided
evidence of O3 associations with asthma exacerbation, COPD exacerbation, and hospital
admissions and ED visits for combined respiratory disease. Additionally, there was
consistent evidence of an association between short-term increases in ambient air O3
concentrations and increases in respiratory mortality. Animal toxicological studies
provided additional support, demonstrating 03-induced changes in ventilatory
parameters, increased airway responsiveness, and lung injury and inflammatory
responses. Animal studies also provided mechanistic information that informed
plausible pathways by which O3 exposure could result in respiratory effects.
3-33
-------
Evidence for long-term exposures to O3 and respiratory effects supported a
"likely to be causal" relationship. This was based largely on consistent findings of
positive associations between long-term O3 concentrations and new onset asthma in
children in longitudinal epidemiologic studies. Infant monkeys exposed to O3 cyclically
over several months showed lung structural and functional changes consistent with
asthma development. Enhancement of allergic responses was also seen in rodent
models of repeated O3 exposure.
The 2020 ISA for Ozone and Related Photochemical Oxidants was the first to
have a causality determination specifically for metabolic effects. The available evidence
supported a "likely to be causal" determination. This was based on several animal
studies showing dysregulation of insulin and glucose homeostasis following O3
exposure. Animal and human studies also showed 03-mediated changes in fatty acid
and triglyceride levels and increases in stress hormone responses that could underlie
changes to metabolic function. The epidemiology literature base was limited in number
but did provide supporting evidence of positive associations between O3 concentrations
and metabolic endpoints such as changes in blood glucose, insulin resistance,
hospitalization for diabetic ketoacidosis, and diabetic coma.
3-34
-------
Table 3-9. Summary of causality determinations from the 2020 ISA organized by
health outcome.
Health Outcome
Causality Determination
Respiratory Effects
Short-Term Exposure
Causal
Long-Term Exposure
Likely to be causal
Metabolic
Short-Term Exposure
Likely to be causal
Long-Term Exposure
Suggestive
Cardiovascular Effects
Short-Term Exposure
Suggestive
Long-Term Exposure
Suggestive
Nervous System
Short-Term Exposure
Suggestive
Long-Term Exposure
Suggestive
Total Mortality
Total Mortality and Short-Term Exposure
Suggestive
Total Mortality and Long-Term Exposure
Suggestive
Reproductive and Developmental Effects
Male/Female Fertility, Reproduction Long-Term Exposure
Suggestive
Pregnancy and Birth Outcomes, Long-Term Exposure
Suggestive
Cancer
Cancer, Long-Term Exposure
Inadequate
In the current review, causality determinations from the 2020 ISA will be revisited
in light of recent evidence including evidence for any additional outcomes. The
following questions will guide evaluation of the health effects literature for short-term
and long-term exposure to O3 and related photochemical oxidants:
• Since the last review, what new or emerging 03-related health effect endpoints have
been evaluated in epidemiologic, controlled human exposure, or animal toxicological
studies? Is there new evidence that further informs endpoints included in the 2020
ISA? Specifically, is there new evidence that (1) is consistent with causality
determinations in the 2020 ISA; (2) contradicts causality determinations in the 2020
ISA; or (3) further improves the understanding of biological plausibility, health
outcomes for at-risk populations, and health outcomes at lower O3 concentrations?
3-35
-------
What is the relationship between O3 exposure and cardiovascular effects, given the
inconsistent evidence from controlled human exposure studies? Have recent
controlled human exposure or animal toxicological studies examined potential
mechanisms of action by which short-term and/or long-term O3 exposures may
result in health effects, particularly cardiovascular effects and other effects not
determined to be "causal" in previous ISAs? Do recent experimental studies provide
insights into the biological plausibility of these 03-induced effects?
Is there new evidence that can enhance our understanding of 03-related health
effects under various exposure conditions, such as low concentrations, different
exposure durations, or varying patterns of exposure (e.g., with or without exercise)?
To what extent can controlled human exposure studies accurately describe the
responses of populations likely to be most at risk of 03-related respiratory effects,
such as children with asthma?
How does O3 exposure affect metabolic-related health outcomes such as metabolic
syndrome and diabetes across different populations and lifestages? What are the
mechanisms by which O3 exposure perturbs glucose and insulin homeostasis in
humans and animals?
How do different patterns of O3 exposure (e.g., short-term high-concentration vs.
long-term low-concentration) influence health outcomes? What is the impact of
exposure history on the health outcomes of study populations in epidemiologic
studies? How do exposure history and co-exposure to other pollutants influence the
relationship between O3 exposure and respiratory effects?
What is the severity and prevalence of health responses to short-term O3 exposures
at and below 60 - 70 ppb in children and individuals with asthma?
What new evidence adds to the understanding of which lifestages and populations
are at increased risk of 03-related health effects? What new evidence informs
conclusions regarding inter-individual variability in response to O3 exposures?
What new or emerging methodologies or study designs are available in
epidemiology to (1) address potential effect modifiers (e.g., genetic traits and
socioeconomic status) and confounders (both chemical and nonchemical stressors),
(2) disentangle the effects of long-term and short-term exposure to O3, (3) better
understand potential heterogeneity in O3 effects assessed in U.S. multicity studies,
and (4) understand the role of O3 as a mediator?
How do the results of recent studies inform the shape of the concentration-response
relationship for O3 and various health outcomes (e.g., mortality, hospital admissions),
especially for exposures near the lower end of the distribution of ambient O3
concentrations?
3-36
-------
• To what extent do other factors serve as potential confounders in epidemiologic
studies (e.g., age, socioeconomic status (SES))? In such studies, to what extent can
health impacts due to O3 and related photochemical oxidants be separated from the
health impacts of these other factors?
• What evidence is available regarding the nature of health effects from exposures to
ambient air pollutant mixtures that include O3? To what extent does the evidence
support attribution of these health effects to exposures to O3 and related
photochemical oxidants? What information about the independent health effects of
exposure to O3 and related photochemical oxidants can be identified from the
various lines of available evidence, including epidemiologic, controlled human
exposure, and animal toxicological studies?
• What new information is available to assess the influence of exposure measurement
error on uncertainty in epidemiologic study results?
• How can the influence of exposure measurement error be assessed through the
examination of various study designs, study populations, exposure assessment
methods, spatial and/or temporal variability in ambient air concentrations, spatial
alignment of study population and ambient measurements, and analytical models?
• Is new information available regarding health effects resulting from exposure to O3-
related photochemical oxidants?
Specific Questions Related to Short-Term Exposures
• Are there any newly identified health outcomes associated with short-term O3
exposure?
• Does new evidence clarify the impact of copollutant confounding on the relationship
between O3 exposure and health outcomes?
• What recent evidence is available to inform policy-relevant considerations of the O3
NAAQS for short-term O3 exposures and respiratory effects? Do recent controlled
human exposure and animal toxicological studies continue to provide support for or
call into question prior conclusions on relationships between short-term O3
exposures and respiratory health effects? Do recent studies report 03-attributable
effects at lower O3 exposure concentrations or for different durations or patterns of
exposure than indicated by studies available in the last review? Does recent evidence
inform the understanding of potential for respiratory health effects to short-term O3
exposures in potentially susceptible populations such as children or those with
underlying lung disease?
• How do results of recent studies expand understanding of the relationship between
short-term exposure to O3 and cardiovascular effects that underlie cardiac mortality,
such as ischemic heart disease, heart failure, or vascular effects? Does recent
3-37
-------
evidence improve coherence across disciplines for heart rate variability, blood
pressure, and outcomes such as cardiovascular hospital admissions or emergency
department visits? Where data from multiple types of evidence are not coherent, can
discrepancies be reasonably explained?
• To what extent is short-term exposure to O3 related to or associated with the
progression of diabetes, other metabolic diseases, and/or other endocrine system
effects? To what extent does the newly available evidence identify health outcomes
related to or associated with O3 exposures that were not previously identified?
• Across the evaluated health effects, what new evidence is available regarding
exposures of different durations?
• Does the new evidence reduce uncertainty related to development of health effects
at exposure concentrations at or near the current standard?
Specific Questions Related to Long-Term Exposures
• Are there any newly identified health outcomes associated with long-term O3
exposure? What new information is available to differentiate between the effects of
chronic exposure to lower O3 concentrations versus acute, repeated exposures to
higher concentrations across various health outcomes?
• What new evidence has emerged regarding the impact of long-term O3 exposure on
respiratory health? Do recent epidemiologic and toxicological studies reinforce the
biological plausibility of relationships between long-term O3 exposure and
respiratory health effects? Have new studies identified 03-attributable effects at
lower concentrations compared to those reported in the previous review? How do
recent studies enhance our understanding of the link between long-term O3
exposure and asthma development, impairment of lung development, pulmonary
inflammation and oxidative stress, and allergic responses?
• How do recent studies expand our understanding of the relationship between long-
term O3 exposure and cardiovascular effects and mortality?
• What new evidence exists linking long-term O3 exposure to diabetes, other
metabolic diseases, and effects on different organ systems?
• How do recent studies improve our understanding of the relationship between O3
exposure and adverse birth outcomes, fertility (e.g., infertility, sperm quality),
pregnancy outcomes (e.g., preeclampsia, gestational hypertension), and
developmental outcomes (e.g., neurocognitive effects)? Is there new evidence linking
O3 exposure during critical developmental windows to increased health risks later in
life?
3-38
-------
• To what extent does new literature support or challenge the existence of a
biologically plausible relationship between long-term O3 exposure and nervous
system effects (e.g., cognitive decline and autism)?
• How do recent studies expand our understanding of the relationship between long-
term O3 exposure and mortality, including respiratory-related mortality, other cause-
specific mortality, and various exposure windows and metrics? Do recent studies
provide information on health effects related to long-term exposure windows other
than annual or lifetime averages (e.g., preconception, pregnancy average, pregnancy
trimester average)? What data are available comparing associations of health effects
among various long-term O3 exposure metrics (e.g., annual, seasonal, pregnancy
average)?
3.4.6 At-Risk Lifestages and Populations
The EPA has developed a framework to provide a consistent and transparent
basis for informing the level of confidence for conclusions that specific lifestages or
populations may be at increased risk of pollutant-related health effects according to one
of four levels: adequate evidence, suggestive evidence, inadequate evidence, and
evidence of no effect (see the appendix, section A.7.2.2). Conclusions from the 2020 ISA
for Ozone and Related Photochemical Oxidants on populations potentially at increased
risk are summarized in Table 3-10 below.14
14 Table 3-10 was modified from the 2020 ISA for Ozone and Related Photochemical Oxidants (U.S. EPA,
2020).
3-39
-------
Table 3-10. Summary of evidence for potential increased exposures to O3 and
related photochemical oxidants and increased risk of 03-related health
effects.
Evidence Classification
Factor Evaluated
Adequate evidence
Pre-existing Asthma
Outdoor workers
Genetic factors (genetic variants, including
HMOX-1, ARG, and GSTP1)
Diet (reduced intake of vitamins E and C)
Lifestage: Children
Older adults
Suggestive evidence
Pre-existing obesity
SES: Low SES
Sex: Females
Inadequate evidence
Pre-existing COPD
Pre-existing cardiovascular disease
Pre-existing diabetes
Smoking
Race/ethnicity
The ISA in this review will evaluate an array of factors that characterize potential
"at-risk" populations and lifestages: intrinsic factors (biological factors such as age or
genetic variants), acquired factors (e.g., pre-existing disease), extrinsic factors
(nonbiological factors such as nutritional status, SES), and/or factors affecting dose or
exposure (e.g., sex, age, outdoor activity or work, SES, physical activity). These factors
may influence risk by increasing exposure, dose, or biological effect at a given dose, and
some factors (e.g., SES) may contribute to risk in multiple ways. In the current review, the
ISA will evaluate whether new information supports or calls into question our previous
understanding of the human populations and lifestages that may be at increased risk for
experiencing health effects associated with exposures to O3 and related photochemical
oxidants. The following questions will guide the evaluation of the human health
evidence for potential at-risk populations and lifestages.
3-40
-------
• What new evidence is available to further support or call into question the at-risk
determinations made for lifestages or populations in the 2020 ISA for Ozone and
Related Photochemical Oxidants?
• What recent evidence from epidemiologic, controlled human exposure, and animal
toxicological studies can be used to inform conclusions related to at-risk
populations, such as genetic traits that may underlie susceptibility or additional
lifestages or populations (e.g., preexisting diseases such as diabetes) potentially at
increased risk of an 03-related health effect?
• Is there new information that identifies a combination (i.e., co-occurring) of risk
factors that can lead to one or more lifestages or populations being at greater risk
compared to another? What new experimental and observational studies are
available to improve our understanding of critical exposure windows?
• To what extent does working outdoors, attending school or daycare, or otherwise
spending a large amount of time outdoors contribute to greater overall exposures to
O3 and related photochemical oxidants and increase the risk of related health
effects? What new information is available to inform disparities in O3 exposure across
different sociodemographic groups?
3.4.7 Welfare Effects
In the 2020 ISA, the welfare effects evidence for O3 focused on effects on
vegetation and ecosystems, and the role of tropospheric O3 in climate change (Table 3-
11 and Table 3-12). The EPA will build on the 2020 ISA by evaluating newly available
literature related to O3 and related photochemical oxidants, and these welfare effects.
3-41
-------
Table 3-11. Summary of causality determinations from the 2020 ISA organized by
ecological effect endpoints.
Endpoirit
Causality Determination
Visible foliar injury
Causal relationship
Reduced vegetation growth
Causal relationship
Reduced plant reproduction
Causal relationship
Increased tree mortality
Likely to be causal relationship
Reduced yield and quality of agricultural crops
Causal relationship
Alteration of herbivore growth and reproduction
Likely to be causal relationship
Alteration of plant-insect signaling
Likely to be causal relationship
Reduced productivity in terrestrial ecosystems
Causal relationship
Reduced carbon sequestration in terrestrial ecosystems
Likely to be causal relationship
Alteration of belowground biogeochemical cycles
Causal relationship
Alteration of terrestrial community composition
Causal relationship
Alteration of ecosystem water cycling
Likely to be causal relationship
Table 3-12. Summary of causality determinations from the 2020 ISA organized by
tropospheric O3 effects on climate.
Effect on climate
Causality Determination
Radiative forcing
Causal relationship
Temperature, precipitation, and related climate variables
Likely to be causal relationship
3.4.7.1 Ecological Effects
The ISA will evaluate the literature related to O3 exposures at levels of biological
organization from the organism to the ecosystem, including effects on biodiversity.
Evidence from experimental (e.g., laboratory, greenhouse, OTC, FACE), field, gradient, or
modeling studies that address effects of O3 on ecological endpoints will be considered
to identify concentrations at which effects are observed (Table 3-6). The focus will be on
information necessary for interpretation of effects and on newly available information
since the last ISA.
3-42
-------
3.4.7.1.1 Individual and Population-level Effects
Ambient O3 concentrations are known to cause foliar injury and decreased
growth and biomass accumulation in annual, perennial and woody plants, including
agronomic crops, annuals, shrubs grasses, and trees. Evidence in the 2020 ISA was
sufficient to infer a causal relationship between O3 exposure and effects on vegetation,
including visible foliar injury, reduced growth, reduced plant reproduction, tree
mortality, and reduced yield and quality from individual plants that are agricultural crop
species. Evidence for foliar injury included data from field, lab, and chamber studies
dating back to the 1960s. Decreased growth at the plant scale has been well established
for several decades and may translate to effects at the stand and then ecosystem scales.
Information on metrics of plant reproduction (e.g., observed flower number, fruit
number, fruit weight, seed number, rate of seed germination) and evidence for direct
negative effects of O3 on reproductive tissues has become increasingly available. The
2020 ISA determined there was a likely causal relationship between O3 and altered
herbivore growth and reproduction. Experimental studies covering a range of species at
varying levels of O3 exposure frequently showed statistically significant effects on
herbivores; however, effects on growth and reproduction were highly context- and
species-specific.
In the current review, specific policy-relevant questions related to O3 effects on
individual and population-level effects include the following:
• Is there new information on foliar injury, biomass loss, crop species, tree mortality
or plant reproduction attributable to O3 in ambient air for U.S. species?
• Is there new information on factors influencing the relationship between O3 and
visible foliar injury?
• Is there new information on non-tree species, non-commodity crops, and species
that are threatened and endangered or culturally significant?
• What new information is available on O3 effects on insect herbivores and other
wildlife?
3.4.7.1.2 Community-level Effects
O3 exposure can lead to loss of sensitive species and alter community
composition of plants and microorganisms in some ecosystems. Evidence in the 2020
ISA was sufficient to infer a causal relationship between O3 and alteration of terrestrial
3-43
-------
community composition. Studies of the impact of O3 on species competition and
community composition showed declines in community composition of above-ground
and below-ground communities. The 2020 ISA determined there was a likely causal
relationship between O3 exposure and alteration of plant-insect signaling. Evidence from
multiple studies show altered or degraded emissions of chemical signals from plants
and reduced detection of volatile plant signaling compounds by insects, including
pollinators, in the presence of O3.
In the current review, specific policy-relevant questions related to O3 community-
level effects include the following:
• What new information is available about O3 effects at the community level, such
as changes in biodiversity and community composition, and altered species
interactions?
• What new information is available on O3 effects on volatile plant signaling
compounds and plant-insect signaling?
3.4.7.1.3 Ecosystem-level Effects
Effects at the individual plant level can result in ecosystem effects such as
changes in productivity, below-ground processes, carbon sequestration, water cycling,
and nutrient cycling. The 2020 ISA determined there was a causal relationship between
O3 exposure and reduced productivity. Results of long-term experiments provided
evidence of the association of O3 exposure and reduced productivity at the ecosystem
level of organization, which were supported by decreased plant growth and modeling
studies. The 2020 ISA also determined there was a causal relationship between O3
exposure and alteration of below-ground biogeochemical cycles, including altered
carbon allocation to below-ground tissues and altered rates of leaf and root production,
turnover, and decomposition. These shifts can affect overall carbon loss and nitrogen
loss from the ecosystem. Studies from the leaf and plant level provided biologically
plausible mechanisms, and results from experimental studies consistently showed
responses of below-ground processes to O3 exposure. The 2020 ISA determined there
was a likely causal relationship between O3 exposure and reduced carbon sequestration.
Evidence for that conclusion was primarily from global and regional modeling
simulations. The 2020 ISA determined there was a likely causal relationship between O3
3-44
-------
and alteration of terrestrial water cycling. Alteration of stomatal functioning may affect
water use in leaves, whole plants, and at the watershed level based on field and
modeling studies.
In the current review, specific policy-relevant questions related to O3 effects on
ecosystem processes include the following:
• What new information is available, including for 03-related effects on ecosystem
services, alteration of below-ground biogeochemical cycles, decreased
productivity, reduced carbon sequestration, and alteration of terrestrial
ecosystem water cycling?
• Are there newly identified ecological endpoints or processes affected by O3?
3.4.7.1.4 Air Quality Indices and Exposure-Response Relationships
Exposure indices are metrics that quantify exposure as it relates to measured
plant response (e.g., reduced growth). In the 2020 ISA, exposure indices (e.g., W126) that
cumulated and differentially weighted the higher hourly average concentrations and
included the mid-level values offered the most reliable approach for use in developing
response functions and comparing studies, as well as for defining future indices for
vegetation protection. Exposure-response relationships were available for several tree
and crop species from a variety of experiments.
In the current review, specific policy-relevant questions related to air quality
indices and exposure-response include the following:
• Are there new U.S. studies that use various O3 metrics to further characterize O3
effects on plant foliar injury and/or growth?
• Are there new studies that improve the characterization of O3 exposure-response
at the local, regional, and/or national scale for the effects determined to be
causal or likely causal? Which are the relevant exposure indices for such
relationships?
3.4.7.2 Effects on Climate
The ISA will present information on how changes in tropospheric O3 might affect
radiative forcing, including subsequent effects on climate endpoints such as surface air
temperature. The focus will be on information necessary for interpretation of effects and
on newly available information since the last ISA. Specific questions include:
3-45
-------
• What new information is available to reduce the uncertainties in the radiative
forcing and climate response attributed to tropospheric O3?
• To what extent has our understanding improved on the effects of tropospheric O3
on climate, isolated from the broader context of other climate forcers, including
precursor species and co-pollutants such as aerosols?
• What feedbacks affect the climate response to radiative perturbations due to
changes in tropospheric O3 concentrations (e.g., via changes in the terrestrial
carbon sink)? What new information is available on including these feedbacks in
global and regional climate modeling studies?
• What recent advances have been made in understanding tropospheric O3 effects
on regional climate in the U.S.?
3-46
-------
4 REFERENCES
Hamel C, Kelly SE, Thavorn K, Rice DB, Wells GA, Hutton B. (2020). An evaluation of
DistillerSR's machine learning-based prioritization tool for title/abstract screening
- impact on reviewer-re levant outcomes. BMC Med Res Methodol. 15;20(1):256.
Hatch, G.E., Slade, R., Harris, L.P., McDonnell, W.F., Devlin, R.B., Koren, H.S., Costa, D.L.
and McKee, J. (1994). Ozone dose and effect in humans and rats. A comparison
using oxygen-18 labeling and bronchoalveolar lavage. Am J Resp Critical Care
Med, 150(3), 676-683.
Hatch, G.E., McKee, J., Brown, J., McDonnell, W., Seal, E., Soukup, J., Slade, R., Crissman, K.
and Devlin, R. (2013). Biomarkers of dose and effect of inhaled ozone in resting
versus exercising human subjects: Comparison with resting rats. Biomark Insights,
19;8:53-56.
Howard BE, Phillips J, Tandon A, Maharana A, Elmore R, Mav D, Sedykh A, Thayer K,
Merrick BA, Walker V, Rooney A, Shah RR. (2020). SWIFT-active screener:
accelerated document screening through active learning and integrated recall
estimation. Environ Int. 2020;138: 105623.
National Academies of Sciences, Engineering, and Medicine (NASEM). (2022). Advancing
the Framework for Assessing Causality of Health and Welfare Effects to Inform
National Ambient Air Quality Standard Reviews. Washington, DC: The National
Academies Press. Available at: https://dQi.Org/l0.1722 1022).
Savitz, DA, Forastiere, F. (2021). Do pooled estimates from meta-analyses of
observational epidemiology studies contribute to causal inference? Occupational
and Environmental Medicine. 78(9), 621-622.
Savitz, DA; Wellenius, GA, Trikalinos, TA. (2019). The Problem with Mechanistic Risk of
Bias Assessments in Evidence Synthesis of Observational Studies and a Practical
Alternative: Assessing the Impact of Specific Sources of Potential Bias. American
Journal of Epidemiology188(9): 1581-1585.
Sheppard, EA (2022). Letter from Elizabeth A. Sheppard Chair, Clean Air Scientific
Advisory Committee, to Administrator Michael S. Regan. Re: CASAC Review of the
EPA's Integrated Science Assessment (ISA) for Ozone and Related Photochemical
Oxidants (Final Report - April 2020). November 22, 2022. EPA-CASAC-23-001.
Available at: https://casac.epa.gov/ords/sab/f?p = 105:18:8476900499267:::
RP.18:P18 ID2614.
4-1
-------
Sheppard, EA (2023). Letter from Elizabeth A. Sheppard Chair, Clean Air Scientific
Advisory Committee, to Administrator Michael S. Regan. Re: CASAC Review of the
EPA's Policy Assessment (PA) for the Reconsideration of the Ozone National
Ambient Air Quality Standards (External Review Draft Version 2). June 9, 2023.
EPA-CASAC-23-002. Available at: https://casac.epa.goV/ords/sab/r/sab apex/
casac/Q?report id = 1114&reauest=APPLICATION PRQCESS%3DREPORT DOC&se
ssion=278565345624.
Sheppard, EA (2024). Consultation on the EPA's Integrated Review Plan for the Primary
National Ambient Air Quality Standards for Oxides of Nitrogen. Volume 2:
Planning for the Review and the Integrated Science Assessment (March 2024)
EPA-CASAC-24-001.
U.S. DHEW (1970). Air Quality Criteria for Photochemical Oxidants. National Air Pollution
Control Administration Washington, DC. U.S. DHEW. publication no. AP-63. NTIS,
Springfield, VA; PB-190262/BA. Available at: https://nepis.epa.gov/Exe/ZvPDF.cqi/
KF.PDF?Dock(\ J« I ! ! mf,
U. S. EPA (2019). Integrated Science Assessment (ISA) for particulate matter (Final
Report). Office of Research and Development, National Center for Environmental
Assessment. Research Triangle Park, NC. U.S. EPA. EPA/600/R-19/188.
U.S. EPA (2020a). Integrated Science Assessment for Oxides of Nitrogen, Oxides of
Sulfur, and Particulate Matter—Ecological Criteria (Final Report). Office of
Research and Development, National Center for Environmental Assessment.
Research Triangle Park, NC. U.S. EPA. EPA/600/R-20/278
U.S EPA (2020b). Integrated Science Assessment (ISA) for Ozone and Related
Photochemical Oxidants (Final Report) Office of Research and Development,
National Center for Environmental Assessment. Research Triangle Park, NC. U.S.
EPA. EPA/600/R-20/012.
U.S. EPA (2021) Appendix D. HAWC User's Guide and Frequently Asked Questions. U.S.
Environmental Protection Agency, Washington, DC, EPA/600/R-20/345F
U.S. EPA (2022). Supplement to the 2019 Integrated Science Assessment for Particulate
Matter (Final Report). U.S. Environmental Protection Agency, Washington, DC,
E PA/63 5/R-22/028.
U.S. EPA (2023). Living Literature Review [Computer Software], RTP, NC.
https://hero.epa.gov/hero/inde v reference/details/reference id ! L; ' ^ 11.
4-2
-------
U.S. EPA (2024) Integrated Science Assessment (ISA) for Lead (Final report). Office of
Research and Development, National Center for Environmental Assessment.
Research Triangle Park, NC. U.S. EPA. EPA/600/R-23/375
U.S. EPA (2024). Integrated Review Plan for the Primary National Ambient Air Quality
Standards for Oxides of Nitrogen. Volume 2: Planning for the Review and the
Integrated Science Assessment. Office of Research and Development, National
Center for Environmental Assessment. Research Triangle Park, NC. U.S. EPA. EPA-
452/R-24-010b.
Van Eck, NJ, Waltman, L (2023) VOSviewer Manual. Univeristeit Leiden. Available at:
https://www.vosviewer.com/documentation/Manual VOSviewer 1.620.pdf.
4-3
-------
APPENDIX: ISA DEVELOPMENT PROCESS
TABLE OF CONTENTS
A.1. INTRODUCTION A-3
A.2. OVERVIEW OF ISA ORGANIZATION AND DEVELOPMENT A-4
A.2.1. ISA Organization A-4
A.2.2. ISA Development A-6
A.3. DEFINING THE SCOPE OF THE ISA A-9
A.3.1. Atmospheric and Human Exposure Sciences Studies A-9
A.3.2. Health Effects Studies A-12
A.3.3. Ecology Studies A-15
A.3.4. Other Welfare Studies A-17
A.4. APPROACH TO LITERATURE SEARCH AND LITERATURE SCREENING A-18
A.4.1. Literature Search A-18
A.4.2. Literature Screening A-19
A.5. EVALUATING INDIVIDUAL STUDY QUALITY A-22
A.5.1. Atmospheric and Human Exposure Science A-24
A.5.2. Epidemiology A-28
A.5.3. Controlled Human Exposure A-42
A.5.4. Experimental Animal Studies and Emerging Approaches in Toxicology A-46
A.5.5. Ecological and Other Welfare Effects A-50
A.6. EXTRACTING DATA FROM RELEVANT STUDIES A-52
A.7. DRAFTING ISA SECTIONS A-54
A.7.1. Draft Chapters and Obtain Peer Input A-54
A.7.2. Develop Policy-Relevant Scientific Conclusions A-55
A.7.3. Develop the Integrated Synthesis A-74
A.8. PEER REVIEW AND PUBLIC COMMENT A-74
A.9. QUALITY MANAGEMENT A-76
A. 10. NASEM CONCLUSIONS, RECOMMENDATIONS, AND CONSIDERATION OF
FUTURE ASSESSMENT DEVELOPMENTS A-78
A-1
-------
A.10.1. Heterogeneity in Exposure Responses A-78
A.10.2. Study Quality Evaluation A-78
A.10.3. Transparency A-79
A.10.4. Expertise A-79
A.11. REFERENCES A-81
TABLE OF FIGURES
Figure A-1. The general process for Integrated Science Assessment development A-8
Figure A-2. Literature flow diagram A-21
TABLE OF TABLES
Table A-1. Generic STEM statement to define the criteria and framework for identifying
relevant atmospheric science studies A-11
Table A-2. Generic STEM statement to define the criteria and framework for identifying
relevant human exposure science studies A-12
Table A-3. Generic PECOS statement to define the criteria and framework for
identifying relevant epidemiologic studies A-14
Table A-4. Generic PECOS statement to define the criteria and framework for
identifying relevant controlled human exposure studies A-14
Table A-5. Generic PECOS statement to define the criteria and framework for
identifying relevant animal toxicological studies A-15
Table A-6. Generic LECES statement to define the criteria and framework for
identifying relevant ecological studies A-17
Table A-7. Example Data Extraction Table - Epidemiologic Studies A-53
Table A-8. Example Data Extraction Table - Animal Toxicological Studies A-53
Table A-9. Example Data Extraction Table - Controlled Human Exposure Studies.... A-53
Table A-10. Example Data Extracted Table - Exposure Science Studies A-54
Table A-11. Aspects of the evidence important to judging causality A-58
Table A-12. Causality determinations for health and welfare effects A-64
Table A-13. Characterization of evidence for factors potentially increasing the risk of
pollutant-related health effects A-69
A-2
-------
A.1. INTRODUCTION
The Integrated Science Assessments (ISAs) review, synthesize, and evaluate
policy-relevant scientific information1 and reach key science judgments intended to
inform the EPA's periodic reviews of the National Ambient Air Quality Standards
(NAAQS) for criteria air pollutants as required under the federal Clean Air Act (CAA).
Section 108 of the CAA requires that"[a]ir quality criteria for an air pollutant shall
accurately reflect the latest scientific knowledge useful in indicating the kind and extent
of all identifiable effects on public health or welfare ISAs document air quality
criteria and form the scientific foundation for establishing and reviewing NAAQS. The
ISAs, in conjunction with additional technical and policy assessments, are used to inform
the rulemaking process for the six criteria air pollutants: ozone (O3) and related
photochemical oxidants, oxides of nitrogen, sulfur oxides, carbon monoxide (CO), lead
(Pb), and particulate matter (PM). Primary NAAQS are set for the protection of public
health, while secondary NAAQS are set to protect public welfare. Under CAA Section
302(h) [42 U.S.C. 7602(h)], "effects on welfare" include, but are not limited to, "effects on
soils, water, crops, vegetation, manmade materials, animals, wildlife, weather, visibility,
and climate, damage to and deterioration of property, and hazards to transportation, as
well as effects on economic values and on personal comfort and well-being." The CAA
requires periodic review of the standards and the science on which the standards are
based to ensure protection of public health and welfare.
This appendix provides an overview of ISA development, building on the process
described in the 2015 Preamble to the ISAs (U.S. EPA, 2015). It will be the subject of a
consultation with the CASAC O3 Panel, and Panel comments will inform development of
upcoming ISAs. This appendix, together with CASAC advice on drafts of upcoming ISAs,
will ultimately serve as the foundation for updating the 2015 Preamble (U.S. EPA, 2015).
1 Pol icy-re levant scientific information includes the results of scientific studies that meet defined scoping
criteria and study quality criteria, and that inform causality determinations and other ISA conclusions
such as those related to populations that may be at increased risk of pollutant-related effects;
concentration-, exposure-, or dose-response relationships; and the strengths and limitations of various
study designs and approaches.
A-3
-------
The approach to ISA development described in this appendix reflects updates
and advancements presented in recently completed ISAs (U.S. EPA, 2019a; U.S. EPA,
2020a; U.S EPA, 2020b, U.S. EPA, 2024) and recommendations made by an ad hoc
committee of the National Academies of Science, Engineering, and Medicine (NASEM)
charged with reviewing the current ISA framework for reaching causality determinations
(U.S. EPA, 2015). Those recommendations are presented in the NASEM report titled
Advancing the Framework for Assessing Causaiity of Heaith and Welfare Effects to Inform
National Ambient Air Quality Standard Reviews (NASEM, 2022). This appendix
additionally reflects consideration of recent input from members of the Clean Air
Scientific Advisory Committee (CASAC) and the CASAC Oxides of Nitrogen Panel (EPA-
CASAC-24-001) on the Integrated Review Plan for the Primary National Ambient Air
Quality Standards for Oxides of Nitrogen. Volume 2: Planning for the Review and the
Integrated Science Assessment (March 2024) (U.S. EPA, 2024).
The subsequent sections present an overview of ISA organization and
development (A.2), a detailed description of the updated ISA development process (A.3
through A.8), a summary of the quality management process that governs ISA
development (A.9), and a brief overview of how this appendix aligns with the
conclusions and recommendations of the NASEM report (NASEM, 2022) (A.10).
A.2. OVERVIEW OF ISA ORGANIZATION AND DEVELOPMENT
A.2.1. ISA Organization
The ISAs are organized around a series of detailed, topic-specific chapters2 and
an Integrated Synthesis that draws from those chapters. Chapters provide thorough
assessments of the scientific evidence pertaining to specific topic areas, including
atmospheric science, exposure and dosimetry, human health outcomes, and welfare
effects. Each chapter contains an evaluation of results from recent studies that build
upon key conclusions and evidence presented in previous assessments. The ISAs use a
causality framework to evaluate and integrate multiple lines of scientific evidence and
2 In the past, some ISAs have utilized chapters (U.S. EPA, 2013b; U.S. EPA, 2013a), while other more recent
ISAs (U.S EPA, 2020a; U.S EPA, 2020b; U.S. EPA, 2024) have referred to topic-specific sections as
appendices.
A-4
-------
draw conclusions about health and welfare effects of exposures to criteria air pollutants.
Chapters for each health outcome or welfare effect category (e.g., respiratory effects,
cardiovascular effects, ecological effects, climate effects) reflect full assessments of the
causal nature of relationships between pollutant exposures and health or welfare effects
that result in key science judgments (i.e., causality determinations, see A.7.2.1). These
causality determinations are based on the consideration of various aspects of the
evidence, including consistency and coherence across studies, biological plausibility or
underlying mechanisms, and other aspects as discussed below (A.7.2.1.1). For human
health outcomes, causality determinations also consider the evidence that certain
populations or lifestages may be at higher risk of pollutant-related effects because they
experience higher exposures and/or because they experience more serious effects
following exposures (A.7.2.2). Chapters additionally present targeted evaluations of the
evidence on other pollutant-specific policy-relevant issues to support the summary
discussion of those issues in the Integrated Synthesis. These other issues vary by
pollutant and discipline, and often include conclusions on concentration-, exposure-,
and/or dose-response relationships; strengths and limitations of various exposure
estimates and study designs; the impact of potential confounding factors on effects; and
the timing of effects (i.e., lag structure of associations and/or duration of exposure
associated with an effect).
The Integrated Synthesis is a concise synopsis of ISA conclusions and a synthesis
of the findings considered in characterizing pollutant exposures and relationships with
health or welfare effects. The Integrated Synthesis typically includes summaries of
causality determinations and salient information for each topic area, including
information on pollutant-related sources, emissions, atmospheric science, exposures,
and dosimetry. The Integrated Synthesis also discusses factors that modify pollutant
exposure and susceptibility (e.g., socioeconomic status), as well as populations and/or
lifestages that may be at increased risk of pollutant-related effects.
In addition to the Integrated Synthesis and supporting chapters, the ISAs also
generally include a Preface that summarizes major legal and historical aspects of prior
NAAQS reviews, an Executive Summary written to be accessible to a wide range of
audiences, and a process chapter. The process chapter describes and documents the
A-5
-------
approach taken to develop the ISA, typically including the methods for literature search
and review, individual study quality evaluation, public engagement, and quality
assurance considerations.
A.2.2. ISA Development
ISAs are developed principally by scientists within the EPA's Center for Public
Health and Environmental Assessment (CPHEA) with extensive knowledge and expertise
in their respective fields, including atmospheric chemistry, climate science, exposure
science, dosimetry, toxicology, epidemiology, biogeochemistry, plant ecophysiology,
wildlife biology, ecotoxicology, terrestrial and aquatic ecology, and others. When
additional subject matter expertise is required, the EPA solicits extramural scientists to
supplement internal expertise, thereby ensuring that each ISA provides an accurate
reflection of the most up-to-date scientific knowledge.
The process for developing an ISA begins with a Call for Information published in
the Federal Register that announces the start of a NAAQS review and invites the public
to contribute to the review by submitting potentially relevant research studies in
identified subject areas. At this stage, the public is also given the opportunity to
comment on policy-re levant issues to be addressed in the review. Information and
comments received from the public inform the planning phase of the review, including
the development of an Integrated Review Plan (IRP). The IRP presents background
information on the NAAQS program in general, the NAAQS for the criteria pollutant
under review, key policy-relevant science issues for the review, the anticipated process
and plans for developing the ISA and other assessments, and the anticipated schedule
for the review.
The EPA consults with the CASAC and solicits public comment on the assessment
plans presented in the IRP. As described further in Section A.8, the CASAC is an
independent committee composed of scientific experts charged with providing advice
to the EPA Administrator on the NAAQS and on the underlying scientific foundation for
the standards. Early in a NAAQS review, the EPA typically supplements the seven-
member CASAC with a pollutant-specific review panel to inform the CASAC's advice.
Given the breadth of scientific and technical information evaluated during NAAQS
A-6
-------
reviews, CASAC panels reflect a wide range of expertise. The specific expertise varies
across pollutant-specific panels, but generally requires members with expert knowledge
of atmospheric chemistry, climate science, exposure science, dosimetry, toxicology,
epidemiology, medicine, public health, biostatistics, ecological sciences, and risk
assessment. Consistent with NASEM recommendations (NASEM, 2022, p.8), critical
disciplines are often represented by multiple panel members to facilitate advice from a
range of perspectives.
Following the CASAC consultation on the IRP, the EPA develops the ISA to
provide the scientific foundation for the review of the NAAQS. After public release of a
draft ISA, the CASAC reviews the document, recommending revisions as necessary
before a final ISA is issued. The ISA development process is summarized in Figure A-1
and described in detail in Sections A.3 through A.9.3 These sections describe the
approaches used for defining the scope of the ISA (A.3), conducting the literature search
and identifying potentially relevant studies (A.4), evaluating individual study quality
(A.5), extracting data from relevant studies (A.6), drafting the ISA (A.7), obtaining
scientific and public review of the draft ISA (A.8), and conducting quality management
(A.9).
3 Recognizing that the EPA continually strives to innovate and improve both the process for developing
ISAs and the ISAs themselves, Agency staff routinely monitor advancements in scientific fields related to
evidence integration and weight-of-evidence evaluations, as well as other relevant fields that could be
incorporated into the ISAs. As a result, the general process outlined in Figure A-1 and discussed in
Sections A.3 through A.9 may evolve over time, as has been the case with the framework for ISA
development described in the 2015 Preamble (U.S. EPA, 2015). This strategy is consistent with the
NASEM recommendation to monitor the scientific literature to determine if emerging approaches to
research synthesis and evidence integration "might be adapted to improve Integrated Science
Assessment causal determinations" (NASEM, 2022, p. 132).
A-7
-------
Source: Modified from Figure II of the Preamble to the Integrated Science Assessments U.S. EPA, 2015.
Figure A-1.The general process for Integrated Science Assessment development.
A-8
-------
A.3. DEFINING THE SCOPE OF THE ISA
As noted above, the ISAs include a series of topic-specific chapters that provide
detailed assessments of the policy-relevant scientific evidence. These chapters focus on
integrating policy-relevant studies that have become available since the previous ISA
with older studies evaluated in earlier assessments. Emphasis is placed on new and
emerging information and studies that address scientific uncertainties and limitations
identified in prior reviews. Important older studies (e.g., those included in previous ISAs)
may be discussed in detail to reinforce key concepts and conclusions and may be
reinterpreted in consideration of more recent data. Older studies may be the primary
focus of the ISA in some subject areas or scientific disciplines where research efforts
have subsided or where these older studies remain the definitive works available in the
literature. The ISAs emphasize studies that are most relevant for ambient air-related
exposures, including studies examining pollutant concentrations that reflect the range of
those exposures across microenvironments. Studies examining higher exposure
concentrations may be included if they provide evidence of the potential biological
mechanism(s) for an observed effect. Each ISA identifies specific literature scoping
criteria to focus the assessment on the most relevant studies. General criteria that guide
the ISA scoping decisions for studies of atmospheric science, climate science, exposure,
dosimetry, human health effects, and welfare effects are discussed below.
A.3.1. Atmospheric and Human Exposure Sciences Studies
To be included in the ISA, relevant atmospheric and human exposure science
studies must have undergone scientific peer review4 and should have been published or
accepted for publication within the predefined literature search cutoff dates.5 Consistent
with the health and welfare evidence (discussed in sections A.3.2 to A.3.4), the ISA uses
discipline-specific scoping statements to identify potentially relevant atmospheric and
exposure science studies. These scoping statements include consideration of pollutant
4 In the atmospheric science chapter, results of published studies are often supplemented by targeted air
quality data and analyses conducted by the EPA.
5 Though studies recommended during the CASAC review process may be considered for inclusion in the
ISA even if those studies fall outside the stated literature cutoff dates.
A-9
-------
sources, transport and transformation, exposure/extent, and measurement and
modeling (STEM). The STEM statements define the objectives of the atmospheric science
and exposure evidence assessments and establish criteria that should be met to
consider a study for inclusion in the ISA. A study meeting any of the four aspects of the
STEM statement is considered for inclusion in the atmospheric science or exposure
assessment.6
For atmospheric and exposure sciences, flexibility is built into STEM statement
application by maintaining a broad scope for subject areas with few studies identified
and dynamically narrowing the scope (e.g., in areas with more studies). This is
accomplished by iteratively revisiting relevance criteria to account for novel scientific
findings, geographic similarity to the United States, representativeness or diversity of
environmental conditions, quality of measurement or modeling method used, or other
refinements.
For most subject areas, study selection is carried out by application of the STEM
statement to individual studies. However, if a very large number of relevant studies are
identified for a subject area, alternative search and screening strategies can be applied
to identify the most relevant and influential studies. This general approach to focusing
on the most relevant and influential atmospheric and exposure science studies is
consistent with the EPA's approach in other assessments, including in the Integrated
Risk Information System (IRIS) and Provisional Peer Reviewed Toxicity Value (PPRTV)
programs (Thayer et al., 2022).
The STEM statements for a specific ISA are informed by the body of evidence
from the previous ISAs and Air Quality Criteria Documents (AQCDs) and by expert
knowledge of the relevant scientific literature. These scoping statements can serve to
highlight well-established areas of research, as well as gaps in the literature and
uncertainties from previous assessments (U.S EPA, 2020a; U.S. EPA, 2016). Importantly,
application of STEM statements is consistent with current best practices for reporting or
5 This contrasts with the PECOS statements used for health effects studies that require all of the listed
criteria to be met (A.3.2). The atmospheric science and exposure science fields cover diverse topics
along the source-to-exposure continuum (NRC, 2004), with each criterion corresponding to a specific
aspect of this continuum.
A-10
-------
evaluating health science data as recommended by the NASEM (NASEM, 2022). Generic
STEM statements for atmospheric science and human exposure studies are provided
below (Table A-1 and Table A-2, respectively). These generic statements provide a
scoping framework that can be modified as appropriate for specific ISAs.
Table A-1. Generic STEM statement to define the criteria and framework for
identifying relevant atmospheric science studies.
Statement
Description
Source (S)
Studies reporting quantitative emissions estimates, as well as observations of physical
and chemical characteristics that add to our understanding of pollutant or precursor
primary emissions.
Transport and
Transformation (T)
Studies investigating atmospheric fate and transport, transformation, and deposition
processes, including transport of air pollutants at various scales (i.e., national/global,
regional, urban, neighborhood), atmospheric chemical transformations, and estimates
of atmospheric concentrations and deposition that add to our understanding of
atmospheric processes.
Exposure/Extent (E)
Studies reporting observations and estimates of ambient air concentrations and their
trends, including spatial variability at various scales (i.e., national/global, regional,
urban, neighborhood); temporal trends such as diurnal, weekday/weekend, seasonal,
and long-term trends; or characteristics, such as composition or relationship with
atmospheric properties that provide up-to-date concentrations and estimates or add to
our understanding of spatiotemporal concentration trends.
Measurement and
Modeling (M)
Studies describing methods of measurement by federal reference and equivalency
methods, satellite estimates, low-cost sensor estimates, or research methods, and
modeling techniques (e.g., chemical transport modeling) for characterizing pollutant
concentrations in ambient air, including the evaluation of measurement principles and
modeling assumptions, examination of potential bias and uncertainties, and method
intercomparisons that are relevant to the NAAQS or to studies in the ISA.
A-11
-------
Table A-2. Generic STEM statement to define the criteria and framework for
identifying relevant human exposure science studies.
Statement
Description
Source (S)
Emissions from outdoor or indoor sources (e.g., traffic or gas ranges emissions),
including both anthropogenic sources (e.g., industrial emissions) and natural sources
(e.g., wildfires) and pollutants formed through chemical reactions in the atmosphere
(e.g., 03).
Transport and
Transformation (T)
Atmospheric and environmental processes, including the transport of air pollutants at
various scales (i.e., national/global, regional, urban, neighborhood, middle, micro
scales, and microenvironments), chemical transformations (e.g., photochemical
reactions), deposition, and dynamics within microenvironments (e.g., indoor
chemistry).
Exposure/Extent (E)
Exposure levels characterized by various surrogates (e.g., ambient air measurement,
near-source measurement, microenvironmental measurement, personal exposure
measurement and modeling, biomarkers of exposure) and exposure determinants
(i.e., factors leading to differential exposures, such as proximity to sources, activity
patterns, and socioeconomic status). This includes spatiotemporal trends of various
exposure surrogates and populations experiencing elevated exposures or exposure
patterns (e.g., exposure level, duration, and frequency) identified in health studies as
being linked to increased health risks.
Measurement and
Modeling (M)
Measurement (e.g., federal reference and equivalent methods, passive samplers,
remote sensing, and biomonitoring approaches) and modeling techniques (e.g., land
use regression, chemical transport, and microenvironmental models) characterizing
ambient air, indoor/microenvironmental air, and personal exposures, including the
evaluation of measurement principles and modeling assumptions, examination of
potential bias and uncertainties, and comparison of different techniques.
A.3.2. Health Effects Studies
Human health evidence assessed in the ISA provides the scientific foundation for
EPA decisions on the primary NAAQS. To be considered for inclusion in the ISA, relevant
human health studies must have undergone scientific peer review and should have been
published or accepted for publication within the predefined literature search cutoff
dates7. Studies can be considered for inclusion if they present original research or new
analyses of existing data. To further refine criteria for identification of potentially
relevant health studies, the ISAs use discipline-specific population, exposure,
comparison, outcome, study design (PECOS) statements. PECOS statements help define
the objectives of the assessment and establish relevance criteria that should be met to
7 Studies recommended during the CASAC review process may be considered for inclusion in the ISA even
if those studies fall outside the stated literature cutoff dates.
A-12
-------
consider a study for inclusion in the ISA, thereby facilitating identification of potentially
relevant literature. To focus on the most informative human exposures, an upper limit is
sometimes used for exposure concentrations tested in epidemiologic and animal
toxicological studies. All studies that meet the PECOS criteria during title and abstract
and full text review undergo study quality evaluation (A.5).
Meta-analyses may be considered for inclusion in an ISA to the extent that (1) the
time period from which the underlying literature is drawn is relevant to the ISA time
period (e.g., so that the pooled effect estimates are reflective of current understanding
of the literature); (2) the underlying literature would meet the PECOS criteria; and (3) the
underlying literature would be policy-relevant. Studies with pooled effect estimates (e.g.,
meta-analyses and pooled analyses) are considered with caution when evaluating
causality, so as not to misrepresent and perpetuate errors from approximating estimates
(Savitz and Forastiere, 2021, Savitz et al., 2019). Review articles that are out of scope or
that are limited to summarizing and interpreting existing studies, without presenting
new information or analyses, are not considered for inclusion in ISAs.
PECOS statements are informed by the body of evidence from previous ISAs and
AQCDs, as well as by expert knowledge of the policy-relevant scientific issues. Generic
examples of PECOS statements are provided below for epidemiologic (Table A-3),
controlled human exposure (Table A-4), and animal toxicological (Table A-5) studies. In
each ISA, these generic PECOS statements may be modified as appropriate for specific
pollutants and health outcome categories.
A-13
-------
Table A-3. Generic PECOS statement to define the criteria and framework for
identifying relevant epidemiologic studies.
Exposure Duration
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Short-term or Long-
term exposure
Population (P): Any human population, including populations or lifestages that might
be at increased risk.
Exposure (E): Exposure to the pollutant(s) under evaluation at concentrations relevant
to ambient air in the U.S.
Comparison (C): Per unit increase in pollutant exposure (e.g., in ppb or pg/m3) or
populations exposed to lower concentrations of pollutant compared to higher
concentrations (e.g., categorical comparisons between different exposure metric
quan tiles).
Outcome (0): Change or difference in risk (incidence/prevalence) of health outcome
(e.g., respiratory effects, cardiovascular effects, total mortality, nervous system effects,
metabolic effects, reproductive and developmental effects, or cancer) per change or
difference in exposure.
Study Design (S): Epidemiologic studies, including but not limited to, panel, case-
crossover, time-series, case-control, cohort, cross-sectional, and quasi-experimental
studies with appropriate timing of exposure for the health outcome of interest.
Table A-4. Generic PECOS statement to define the criteria and framework for
identifying relevant controlled human exposure studies.
Exposure Duration
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Single or repeated
short-term exposures
Population (P): Human volunteers enrolled in controlled exposure studies, including
volunteers representing populations or lifestages that might be at increased risk of
pollutant-related health effects.
Exposure (E): Controlled pollutant exposure, where pollutant exposures must be
controlled by the experimenters and not simply a measure of ambient air or
occupational exposure.
Comparison (C): An appropriate control exposure (e.g., filtered air or room air) for
each study participant or an appropriately matched comparison group exposed to
filtered air or room air.
Outcome (0): Outcomes of interest are those that relate to human health including, but
not limited to, effects on the respiratory system, cardiovascular system, immune
system, nervous system, diabetes, cancer, reproduction and development, or other
human health effects. Effects of interest include changes in indicators or measures of
physiological function, health-relevant biomarkers, and organ structure. Effects can be
directly measured in exposed study participants or in cells, tissues, or fluids isolated
from study participants.
Study Design (S): Studies that perform controlled human exposures meeting the
above criteria or that analyze data from previously conducted controlled human
exposures (e.g., reanalysis, meta-analysis).
A-14
-------
Table A-5. Generic PECOS statement to define the criteria and framework for
identifying relevant animal toxicological studies.
Exposure Duration
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Short-term or long-
term exposure
Population (P): Laboratory nonhuman mammalian animal species (e.g., nonhuman
primate, mouse, rat, guinea pig, minipig, rabbit, cat, dog) of any lifestage including
models of increased susceptibility.
Exposure (E): Controlled exposure to relevant pollutant concentrations.8
Comparison (C): An appropriate control exposure (e.g. filtered air or room air).
Outcome (0): Outcomes of interest are those that relate to human health including, but
not limited to, effects on the respiratory system, cardiovascular system, immune
system, nervous system, diabetes, cancer, reproduction and development, or other
human health effects. Effects of interest include changes in indicators or measures of
physiological function, health-related biomarkers, and organ structure. Effects can be
directly measured in exposed animals or in cells, tissues, or fluids isolated from
animals.
Study Design (S): Controlled exposure studies of animals in vivo meeting the above
criteria.
A.3.3. Ecology Studies
The scientific evidence for the secondary NAAQS addresses effects on welfare.
Under CAA Section 302(h) [42 U.S.C. 7602(h)], effects on welfare are defined as
including, among others, effects on soils, water, crops, vegetation, animals, and wildlife,
which are all components of ecosystems. ISAs define ecosystem as "a functional unit
consisting of living organisms, their nonliving environment, and the interactions within
and between them (U.S. EPA, 2024)." This matches the definition used by the
Intergovernmental Panel on Climate Change (Allwood et al., 2014). Ecosystems can be
natural, cultivated, or urban (U.S. EPA, 1986) and may be defined on a functional or
structural basis. Ecosystem function refers to the suite of processes and interactions
among the ecosystem components that involve energy or matter. Examples include
water dynamics and the flux of trace gases such as rates of photosynthesis,
decomposition, and nutrient cycling. Ecosystem structure includes species abundance,
richness, distribution, diversity, evenness, and composition measured at the population
or community scales, which may be further defined by spatial boundaries such as those
relating to an ecosystem, region, or global scale.
8 The definition of "relevant" concentration may differ across ISAs.
A-15
-------
For consideration of ecological effects of criteria air pollutants, the atmosphere
and terrestrial, freshwater, and coastal systems are interconnected. Exposure may be
direct, via gaseous or aerosol forms, or indirect, through deposition to soil, water,
vegetation, or fauna. Some air pollutants, such as Pb, oxides of nitrogen, and sulfur
oxides may undergo subsequent transport and transformation through environmental
media (air, soil, water, sediment, biota); O3 is directly taken up into vegetation from the
atmosphere via dry deposition. Cytological or biochemical changes associated with
initial exposure to criteria air pollutants in individual organisms may lead to effects at
higher levels of biological organization (i.e., from the subcellular and cellular level
through the individual organism and up to ecosystem-level processes). Uptake of
atmospherically deposited criteria air pollutants from soils, water, sediment, and biota
(via diet or direct absorption), subsequent bioaccumulation, and toxicity vary greatly
between biological species and across taxa. The type of experimental approach (e.g.,
controlled laboratory exposure, growth chamber, open-top chamber, mesocosm,
gradient, field study, etc.) is also taken into consideration when assessing the effects of
criteria air pollutants.
To be included in the ISA, relevant ecological studies (i.e., studies of effects on
terrestrial and aquatic ecosystems) must have undergone scientific peer review and
been published or accepted for publication within the predefined literature search cutoff
dates. Emphasis is placed on studies that evaluate effects at or near ambient
concentrations, unless lower or higher than ambient concentrations are part of a set of
exposures that includes ambient or near ambient exposures, in an experiment designed
to study exposure-response. Studies may be evaluated for relevance to U.S. air quality
considerations and inclusion in the ISA regardless of the country in which they were
conducted, provided that they contribute significantly to the general understanding of
air pollutant effects. Studies can be considered for inclusion if they present original
research or new analyses of existing data. To further refine criteria for identification of
potentially relevant ecology studies, ISAs use Level of Biological Organization, Exposure,
Comparison, Endpoint, and Study Design (LECES). The LECES statement was developed
by the EPA to provide a Question Formulation System best suited to the nature of
ecology studies, comparable to PECOS in health studies and STEM in atmospheric
A-16
-------
studies. LECES works on the same principle as PECOS. LECES and PECOS are systems of
binary gates (five of them in both systems) that a reference must go through in order to
be accepted. A reference that fails any of the five tests (L/E/C/E/S or P/E/C/O/S) is
rejected. Each of the five tests may comprise several sub-tests. A reference that fails any
of the sub-tests is rejected. The LECES statements aid in identifying relevant evidence in
the literature for ecological effects of the pollutant being assessed; specific details and
criteria may vary by pollutant and assessment. Other topics within scope, in addition to
effects described in the LECES criteria, may also vary by pollutant and assessment.
Table A-6. Generic LECES statement to define the criteria and framework for
identifying relevant ecological studies.
Statement
Description
Level of Biological
Organization
For any species, individual organism, population, biological community, or ecosystem
present in environments in North America or similar to those in North America.
Exposure
Concentrations occurring in the environment, experimental concentrations within a
given range of recent concentrations, or at least one concentration in the test series in
the range described above.
Comparison
Relevant control sites, treatments, or parameters.
Endpoint
Effects relevant to the specific pollutant(s) being assessed (e.g., growth, survival,
reproduction, community composition, productivity, ecosystem processes, or others).
Study Design
Laboratory, greenhouse, chamber, mesocosm, fumigation, field, gradient, mechanistic
modeling, or other studies.
A.3.4. Other Welfare Studies
Under section 302(h) of the CAA, effects on welfare are defined to also include
"effects on manmade materials,... weather, visibility, and climate, damage to and
deterioration of property, and hazards to transportation, as well as effects on economic
values and on personal comfort and well-being." Other evidence evaluated in an ISA for
the secondary NAAQS may include these non-ecological welfare effects. Some welfare
effects are often associated with specific criteria air pollutants. For example, materials
damage and soiling and effects on visibility may be considered in the particulate matter
A-17
-------
(PM) ISA and effects on climate may be evaluated in both the PM and O3 ISA. During ISA
development and scoping, targeted literature searches may be conducted to evaluate
whether any new information on known non-ecological welfare effects or additional
welfare effects not considered in prior ISAs has become available.
A.4. APPROACH TO LITERATURE SEARCH AND LITERATURE
SCREENING
A.4.1. Literature Search
The EPA uses a multipronged approach to identify scientific studies that may be
relevant for inclusion in a particular ISA. As described below, this approach employs
both targeted and broad literature search strategies. It also includes consideration of
studies identified by public commenters, studies recommended during the peer input
workshop or the CASAC review/consultation process (A.7 and A.8), and studies identified
by EPA scientists based on professional expertise.
One targeted approach to literature identification that may be employed is
forward citation searching. For this approach, a set of relevant published references is
identified as a seed set and databases (e.g., PubMed and Web of Science) are then
queried to identify recently published literature that has cited any of the references in
the seed set. The seed set comprises data-containing peer-reviewed references cited in
the previous ISAs or AQCDs. Seed sets are specific to topic area or discipline (e.g.,
atmospheric science, epidemiology). Because each seed set is highly relevant to the
topic of interest, this targeted approach to reference identification can be more precise
than keyword searches, and it further allows for relevance ranking based on the number
of references in the results returned by a query that match references in the seed set.
A broader approach to literature identification is the use of keywords to query
literature databases. When this approach is applied, a set of broad keywords is curated
for each pollutant and/or topic and then used to search relevant databases (e.g.,
PubMed and Web of Science). The results of a broad keyword search may then be
further separated by scientific discipline (e.g., epidemiology, toxicology, ecology) using
automatic topic classification. In the classification step, a set of highly relevant
references for each discipline is used to train machine learning algorithms. The
A-18
-------
classification algorithm then categorizes the keyword search results into relevant and
non-relevant sets based on information from the seed set. This method varies in
effectiveness across disciplines due to the broad range of topics and variability in term
usage in some sets of keyword search results. Discipline-specific keyword searches (e.g.,
searches specifically related to pollutant sources, atmospheric chemistry, climate
science, exposure assessment, dosimetry/toxicokinetics, epidemiology, controlled
human exposure, animal toxicology, or ecology) may also be used to capture literature
pertinent to the pollutant of interest in citation databases (e.g., PubMed and Web of
Science). Keyword search and forward citation search results can be combined and
deduplicated to form a complete set of scientific studies to be screened, as described
below.
A.4.2. Literature Screening
Studies identified during the literature search described above are evaluated for
potential relevance using a multi-tiered literature screening approach designed to
maximize efficiency while simultaneously ensuring relevant studies are identified.
Studies are initially evaluated for potential relevance by comparing their titles and
abstracts to the discipline-specific scoping criteria defined by STEM, PECOS, or LECES
statements (A.3). Because the number of criteria pollutant-related studies identified in
initial literature searches can be very large (e.g., hundreds of thousands of studies across
disciplines and outcomes), machine-assisted ranking literature review software tools
(e.g., DistillerSR [Ottawa, Ontario, Canada] Hamel et al., 2020; SWIFT-Active Screener
[Sciome, RTP, NC, USA] Howard et al., 2020; Living Literature Review [U.S. EPA, RTP, NC,
USA] (U.S. EPA, 2023))9 are employed to maximize screening efficiency. Final study
inclusion and exclusion decisions reached while using these tools are documented in the
Health and Environmental Research Online (HERO) database.10 Studies that appear to
meet the ISA scoping criteria based on the title and abstract screen, together with
studies that cannot be definitively identified as out of scope, are retained for further
evaluation of the full text.
9 Available at https://hero.epa.gov/hero/index.cfm/reference/details/reference id/11327944
'"Available at https://hero.epa.gov
A-19
-------
Following title and abstract screening, ISA relevance is further evaluated by
comparing the full text of documents to the predefined STEM, PECOS, and LECES
scoping criteria described above. Studies not meeting the scoping criteria are excluded
from further consideration and are tagged in HERO as not STEM-, PECOS-, or LECES-
relevant. Retained studies are tagged in HERO as "considered for inclusion" in the ISA.
Studies identified as STEM-, PECOS-, or LECES-relevant are further evaluated for
individual study quality as described below (A.5). The approach described below can be
adapted as appropriate for an individual ISA, and the specific study quality criteria for a
given assessment are informed by expert knowledge of the literature from previous
reviews and by recommendations from a peer input workshop, the CASAC, and
members of the public during initial phases of planning and ISA development. Studies
that are determined to meet scoping criteria and judged of sufficient quality are tagged
in HERO for inclusion in the ISA. Studies that do not meet study quality criteria are
tagged in HERO for exclusion.
Results of the literature search and screening efforts and the evaluation of
individual study quality can be captured in flow diagrams that document the number of
references identified from each database searched (i.e., PubMed and Web of Science),
the number of references evaluated in each screening step, and the general reasons for
reference exclusion (see Figure A-2 for an example of anticipated formatting).
Visualizations of the literature identification and screening steps may be produced to
aid in understanding current trends in the literature. These visualizations may include
topic maps based on unsupervised cluster analyses of terms found in titles/abstracts or
evidence maps based upon categorization that occurs at the title and abstract screening
step.
A-20
-------
c
-------
types used to support biological plausibility, supplementary scientific information may
be derived from studies deemed out of the scope of PECOS statements (e.g., exposure
concentrations higher than PECOS-defined cut-offs, alternative model systems that do
not meet PECOS criteria).
A.5. EVALUATING INDIVIDUAL STUDY QUALITY
Studies that are determined to be STEM-, PECOS-, or LECES-relevant are
evaluated for study quality. Individual study quality is evaluated by considering the
design, methods, and documentation of each study, but not study results. While results
are not considered as part of a study quality evaluation, additional examination of study
design and statistical methods may be appropriate to understand results that are
inconsistent with overall trends observed in a particular body of evidence. The ISA's
general approach to study quality evaluation considers the strengths and limitations of
individual studies, including the possible role of chance, confounding, and other biases
that may affect study interpretation and the strength of inference that can be drawn from
study results. Consistent with NASEM advice (NASEM, 2022, p. 5), the sections below
identify foundational study design attributes and analysis approaches that are
considered when evaluating studies and describe how these attributes and approaches
can influence the inference drawn from individual studies.
For studies of health effects in particular, the specific attributes considered in
evaluating study quality include (1) study design, (2) study population or test model, (3)
pollutant, (4) exposure assessment or assignment, (5) outcome assignment evaluation,
(6) potential confounding (i.e., for epidemiology), and (7) statistical methodology. These
attributes, described further below, are informed by existing EPA guidelines related to
cancer, neurotoxicity, reproductive toxicity, developmental toxicity, and exposure
assessment (U.S. EPA, 2005; U.S EPA, 1998; U.S EPA, 1996; U.S EPA, 1991; U.S EPA, 2019b)
and are consistent with current best practices for reporting or evaluating health science
data as recommended by the NASEM (NASEM, 2022).
Effective evaluation of study quality relies most fundamentally on transparency,
including clearly reporting key data, assumptions, methods, formulas, input parameters,
QA/QC procedures, statistical models/coding, reasoning process, and limitations. Such
A-22
-------
transparency can increase confidence in individual study results (NASEM, 2022, p. 6-7).
The sufficiency of study documentation and reporting can be evaluated based on (1)
whether a person with a general knowledge of the research area can understand the
described approach, (2) whether the study can, in principle, be independently verified or
replicated based on the reported methodology, should original data be available, and
(3) whether the limitations can be characterized based on the reported assumptions and
uncertainties (WHO, 2008). The ISA assessment of the scientific quality of individual
studies is framed by the following general questions from the 2015 Preamble to the
ISAs:
• Were the study design, study groups, methods, data, and results adequately
justified and clearly presented in relation to the study objectives to allow for
study evaluation, including evaluation of underlying assumptions and study
limitations?
• Are the air quality, exposure, and/or dose metrics sufficiently representative of or
pertinent to ambient air and are they adequately documented?
• Do the analytical methods used in the study provide adequate sensitivity and
precision?
• Are the statistical analyses appropriate, properly performed, and appropriately
interpreted?
• Were the study populations, participants, organism model systems, ecosystems,
or study site(s) appropriately selected and sufficiently well-defined to allow for
meaningful comparisons between study or exposure groups?
• Are the health, ecological, or other welfare effect measurements meaningful and
valid?
• Were likely confounders controlled for and modifying factors examined in the
study design and/or statistical analysis?
• Was the study conducted with appropriate oversight by ethics boards or
committees (e.g., documenting approval from a U.S. or foreign-equivalent
Institutional Review Board (IRB) for human studies or from an Institutional Animal
Care and Use Committee (IACUC) for animal studies)?
Answers to these questions, and the presence or absence of particular study
attributes, are not used as a checklist in evaluating study quality, and the identification
of uncertainties that may influence study interpretation does not necessarily lead to a
A-23
-------
conclusion that the study should be excluded from the ISA but may provide context for
the broader process of evidence integration and inform the development of ISA
conclusions such as causality determinations. Final study inclusion and exclusion
decisions reached as a result of study quality evaluation are documented in the HERO
database or the Health Assessment Workspace Collaborative (HAWC).11 The sections
below provide a general discussion of the study attributes considered in the ISA's
evaluation of individual study quality for atmospheric science and exposure science
(A.5.1), epidemiology (A.5.2), controlled human exposure studies (A.5.3), animal
toxicological studies (A.5.4), and ecological and other welfare studies (A.5.5). These
attributes can be adapted for specific ISAs, as appropriate.
A.5.1. Atmospheric and Human Exposure Science
Atmospheric sciences provide information on air pollutant sources, atmospheric
transport and transformation, ambient air concentrations, and techniques for measuring
or modeling air pollutants and their precursors. Human inhalation exposure to air
pollutants involves contact with the pollutant(s) at the interface of the breathing zone
over a specified length of time (U.S EPA, 2019b). Together, atmospheric and exposure
science studies can provide context for interpreting the health and welfare effects
evidence when those studies include information on emissions, ambient air
concentrations, properties, processes, and constituents of criteria air pollutants (e.g., in
the case of photochemical oxidants and oxides of nitrogen); spatiotemporal variations;
various exposure surrogates (e.g., ambient air concentration, indoor concentration, and
total personal exposure); and variability and uncertainty associated with exposure
estimates. Of particular interest is understanding the strengths and limitations of
exposure surrogates commonly used in epidemiologic health effects studies, including
information on errors in measuring or modeling the exposure surrogate and errors
resulting from using the surrogate to approximate true exposure.
The ISA study quality evaluation for atmospheric and exposure science studies
considers the applicability and utility of the study, the soundness of the study approach,
the clarity and completeness of the study, and its treatment of uncertainty and
"Available at https://hero.epa.qov and https://hawc.epa.gov/
A-24
-------
variability. These study attributes are adopted from the EPA exposure assessment
guidelines (U.S. EPA, 2019b)12 and can be used collectively to inform conclusions on
study quality.
A.5.1.1. Applicability and Utility
Applicability and utility refer to the extent to which information presented in a
study is relevant for its intended use (U.S EPA, 2019b). To be considered for inclusion in
an ISA, atmospheric and exposure science studies should evaluate the pollutant in
ambient air (and in other media, as relevant) and/or along exposure pathways (e.g.,
emissions, transformation, transport) pertinent to U.S. populations or ecosystems, with a
focus on the policy-relevant issues that help frame assessment of the health and welfare
effects evidence. These issues can vary across the ISAs and generally include
characterizing emissions sources, transport and transformation processes, measurement
and modeling methods, temporal and spatial patterns of ambient air pollutant
concentrations and exposures, meteorological impacts on pollutant concentrations,
relationships and correlations between pollutants within complex mixtures, infiltration to
indoor environments, transfers to other media, and the strengths and limitations of
various exposure estimation approaches. For studies examining exposure surrogates, the
ISA evaluates various aspects of exposure measurement error, some of which are
discussed below in the context of the health evidence (e.g., errors in measurements,
inherent assumptions in exposure modeling, uncertainties in model input parameters,
spatiotemporal variations in pollutant concentrations, activity patterns, and building
ventilation). The ISA assessment of applicability and utility is aided by clear
documentation of study design, data collection/generation techniques, and any
underlying assumptions.
A.5.1.2. Soundness
Soundness refers to the extent to which the scientific and technical procedures,
measures, methods, or models employed to generate information are reasonable for,
12 Analogous study quality evaluation guidelines have not previously been developed for atmospheric
science. Thus, the ISA study quality evaluation of atmospheric science studies is based on adaptation of
the study attributes described in the EPA exposure assessment guidelines (U.S EPA, 2019b).
A-25
-------
and consistent with, the intended application (U.S EPA, 2019b). In evaluating soundness,
the ISAs focus on atmospheric or environmental science and exposure studies that use
quality-assured measurement and/or modeling techniques. Studies should include or
reference clear and comprehensive descriptions of the measurement or modeling
techniques used, evaluation procedures and performance metrics, strengths and
limitations of techniques, and quality-control procedures. The ISAs place more weight
on studies that use methodologies appropriate for meeting study goals and clearly
justify those methodological decisions. For example, considerations when evaluating the
soundness of measurement- and model-based atmospheric science or exposure studies
are described below.
• Measurement studies: The ISAs place more weight on studies that document or
reference high quality measurement data, including uncertainty, bias, sensitivity,
specificity, stability, repeatability, and data management practices consistent with
best practices among similar measurement studies. Also included are studies
employing innovative, newly developed methods with less developed data
quality documentation that show promise as potential alternative methods that
could have advantages over current methods, and methods that contribute
unique insight into atmospheric processes and concentration trends that
complement established, high quality measurement methods.
• Modeling studies: The ISAs place more weight on studies that include or
reference clear descriptions of acceptable rationales for model selection and
model assumptions, model input parameters (e.g., the data source for input
parameters, the rationale for input data selection, quality of input data in terms of
accuracy, precision, representativeness, completeness, and consistency), model
calibration/validation procedures (e.g., goodness-of-fit criteria for acceptance of
the parameter value, procedures to handle outliers, procedures to validate
results, model parameter sensitivity analysis, impact of parameter uncertainty on
results), and model evaluation results (e.g., for chemical transport models), as well
as clear statements of data transfer, transformation and storage procedures.
A.5.1.3. Clarity and Completeness
Clarity and completeness refer to study transparency, which is a crosscutting
attribute (WHO, 2008) essential for evaluating all other attributes of data and study
quality. For example, sufficient documentation of key assumptions, methods, formulas,
input parameters, reasoning processes, and limitations increases the transparency of a
study. Key aspects of a study evaluated for transparency are summarized below.
A-26
-------
• Study Rationale: Study transparency is greater for studies that present clear
rationales for study design, location, population selection (in the case of exposure
studies), method selection, and conclusions.
• Assumptions: The most informative studies clearly articulate assumptions related
to study design and methods used (e.g., dispersion model assumptions,
assumptions associated with using central monitoring sites for exposure
assignment), as well as assumptions associated with chemistry, transport,
dispersion, and/or exposure modeling techniques, and the appropriateness of the
assumptions.
• Data and procedures: Studies that clearly describe QA and QC procedures, key
data, data sources and methods, and statistical methods (e.g., methods for
missing data and imputation; quantitative relationships between and within
pollutant measurements, such as regression slopes, intercepts, and fit statistic)
are considered most informative.
A.5.1.4. Uncertainty and Variability
Uncertainty in environmental or human exposure assessment studies comes from
incomplete or incorrect information about an environmental measurement or true
exposure; variability describes the natural heterogeneity of measurements or estimates.
Uncertainty and variability are associated with each element of atmospheric or
environmental analyses and exposure assessments. Evaluation of uncertainty and
variability can increase our understanding of the reliability of analyses and the data
needed to improve them. In considering studies for inclusion in the ISA, preference is
given to studies that characterize uncertainties in measurement and/or modeling
approaches and the factors contributing to uncertainties in study analyses.
Human exposure studies can be of particular use in the ISAs if they characterize
exposure errors relevant to interpreting epidemiologic study designs or if they examine
the potential for differential exposures across various populations. For example, classical
error in the exposure surrogate of an epidemiologic study, defined as error scattered
about the true exposure and independent of the true exposure, is generally expected to
reduce precision and to negatively bias associations between air pollution and health
effects. In contrast, Berkson error, in which the true value varies randomly around the
measured value and the measurement error is independent of the measured value,
A-27
-------
reduces precision but is not expected to bias the health effect estimate (Goldman et al.,
2011).
Air pollutant concentrations and depositions exhibit significant spatial and
temporal variability, which in turn contributes to variations in personal exposure to air
pollutants, underscoring the importance of assessing this variability across diverse
populations. Studies of differential exposures can identify populations with elevated
exposures, characterize exposures for populations at higher risk of pollutant-related
effects (e.g., due to pre-existing disease), and evaluate the factors contributing to
variability in exposures (e.g., activity patterns, proximity to sources).
A.5.2. Epidemiology
Epidemiologic studies report associations between pollutant exposures and a
spectrum of health effects (e.g., changes in heart or lung function) and outcomes (e.g.,
hospital admissions, mortality) in the general population and in groups potentially at
increased risk of pollutant-related effects. Epidemiologic studies examine the effects of
real-world exposures on diverse populations, including individuals with severe asthma
who cannot ethically or safely participate in controlled human exposure studies. When
integrated with other lines of scientific evidence (i.e., controlled human exposure and
animal toxicology), epidemiologic studies can provide information that supports
elucidation of the exposure to effect continuum, including the exposure duration of
effect, the lag structure of associations (i.e., the lag time between exposure and effect)
and the concentration-response relationships across the range of ambient air pollutant
concentrations experienced by populations. When evaluating epidemiologic studies,
inference is stronger for studies that clearly describe the primary and any secondary
aims of the study, or specific hypotheses being tested. In evaluating the quality of
epidemiologic studies, the ISAs additionally consider the appropriateness of the (1)
study design, (2) study population, (3) pollutant(s) evaluated, (4) exposure assessment or
assignment, (5) outcome assignment evaluation, (6) control for potential confounding,
and (7) statistical methodology. Each of these study attributes is discussed below.
A-28
-------
A.5.2.1. Study Design
Study designs that focus on environmental exposures, such as air pollution, vary
depending on the data available and the exposure duration examined, but broadly can
encompass cohort, cross-sectional, time-series, case-crossover, panel, and ecological
designs. In addition, quasi-experimental designs have been used to mimic randomized
experiments and reduce potential bias. Such studies can be informative in the
assessment of causality and can include, but are not limited to, intervention studies,
studies of natural experiments, and accountability studies. Across various designs, ISAs
emphasize studies with broadly representative study populations that reduce selection
bias and improve generalizability. Other things being equal, studies with larger sample
sizes increase statistical power and precision and therefore offer notable advantages
over those with smaller sample sizes. Because national, regional, and multicity studies
examine associations of health effects across geographical locations and typically in
larger and broader populations, the ISAs generally emphasize results from these studies
when available.
Air pollution epidemiologic studies examining short-term exposures (i.e., up to 30
days) employ time-series, case-crossover, or panel designs to evaluate relationships
between short-term (e.g., day-to-day) changes in air pollution exposures and specific
health outcomes at the population level (e.g., mortality, hospital admissions, emergency
department visits, symptoms, clinical outcomes, or biomarkers). Time-series and case-
control studies relate temporal variation in the exposure and outcome, and allow
individuals or populations to serve as their own controls, making them less prone than
cross-sectional studies to confounding by factors that differ between individuals (e.g.,
socioeconomic status, age, smoking status). Therefore, among epidemiologic studies of
short-term exposures, time-series and case-crossover designs are emphasized over
cross-sectional studies. Inference from time-series and case-crossover designs is
stronger when they include appropriate controls for factors that vary temporally and
that are correlated with the exposure of interest and the health outcome (i.e., time-
varying confounders). Time-series and case-crossover designs share equivalent
estimating functions, though time-series studies control for these time-varying
confounders (e.g., daily temperature, day of the week, seasonal illness rates,) with
A-29
-------
regression terms, whereas case-crossover studies use a combination of referent
matching (e.g., by day of the week, season) and modeling (e.g., meteorological
variables).
Panel studies are longitudinal studies that typically examine smaller study
populations over a short period of time. Panel studies can assess outcomes that are not
typically examined in larger time-series or case-crossover studies, such as health
biomarkers, clinical assessments, and symptoms. Additionally, due to smaller sample
sizes, panel studies may incorporate personal measures of exposure. Panel studies can
be particularly informative if they employ clearly defined scripted activity patterns (e.g.,
study participants walk the same routes at the same times of day), measure personal
exposures to ambient air pollutants, and measure outcomes at consistent, well-defined
lags after exposures.
Air pollution epidemiologic studies examining long-term exposures (i.e., longer
than 30 days) and specific health outcomes at the population level (e.g., disease
incidence or progression, mortality) often employ cross-sectional or ecological study
designs, which measure exposures and outcome(s) at a single point in time, or cohort
designs, which assess exposure before the outcome(s) occur. Because cohort studies can
better inform the temporality of the relationship between long-term pollutant exposures
and outcomes, inference is generally stronger for cohort studies, including case-control
studies nested within a cohort (e.g., for rare diseases), than for cross-sectional studies,
ecological studies, or case-control studies not nested within a prospective cohort. Long-
term exposure studies that do not employ a cohort design can have uncertainties
related to potential reverse causality (i.e., cross-sectional studies), the appropriateness of
the control group (i.e., case-control), and the validity of inference about individuals from
aggregated or group-level data (i.e., ecological studies). Because studies of long-term
exposures evaluate associations based on spatial and/or temporal variation, inference
from cohort studies is stronger when they address the potential for confounding by
factors that vary spatially across populations (e.g., smoking rates, socioeconomic status
or SES) and by factors that vary temporally (e.g., trends over time).
Additionally, some epidemiologic studies employ study designs and/or statistical
approaches that, compared to traditional regression models, are intended to better
A-30
-------
account for confounders and to mimic randomized experiments and reduce potential
bias. In the peer-reviewed literature, these epidemiologic studies are often referred to as
causal inference studies, studies that use causal modeling methods, or quasi-
experimental studies. Herein, to prevent confusion with the main scientific conclusions
presented within an ISA (i.e., the causality determinations), this document refers to
causal inference studies as alternative methods for confounder control. Examples of
statistical approaches used in such studies include, but are not limited to, generalized
propensity scores, inverse probability weighting models, negative exposure controls,
and instrument variables. When evaluating studies that employ these alternative
methods for confounder control, the ISA uses an approach consistent with that used for
evaluating traditional regression models. As described further below (Section A.5.2.8),
this approach considers the clarity, plausibility, internal consistency, and validity of study
assumptions; the degree to which confounding has been appropriately considered and
addressed; and the degree to which statistical uncertainties are appropriately
characterized and quantified.
A.5.2.2. Study Population
The population evaluated in an epidemiologic study can impact the strength of
inference drawn from that study. The bullets below describe the attributes of
epidemiologic study populations considered as part of the ISA study quality evaluations.
• Representativeness: There is greater confidence in results for study populations
that are recruited from, and representative of, the target population. Selection
bias can influence results in either direction and may not affect the internal
validity of results but rather reduce the generalizability (i.e., external validity) of
findings to the target population.
• Inclusion and exclusion criteria for participants: Clearly specified criteria for
including and excluding subjects and the reporting of baseline information on
participants that are lost to follow-up can aid in evaluating potential selection
bias.
• Participation rates: Studies with high participation and low loss to follow-up over
time that is independent of exposure or health status are considered to have low
potential for selection bias.
• Health conditions: For studies that evaluate populations with underlying health
conditions, independent clinical assessment of the health condition is considered
A-31
-------
the most reliable approach to identifying the study population, though self-
report of physician diagnosis could also be considered a reliable approach for
some conditions (e.g., respiratory and cardiovascular diseases). Comparison
groups with and without an underlying health condition are more informative if
groups are from the same source population.
A.5.2.3. Pollutant
The primary NAAQS are intended to provide requisite protection of public health
against the effects of exposures to criteria air pollutants in ambient air. To inform
decisions on the NAAQS, each ISA focuses on studies that evaluate the effects of
exposures to the particular criteria pollutant(s) under evaluation. Thus, emphasis is
placed on epidemiologic studies that evaluate associations with individual criteria
pollutants, or with components of a criteria pollutant (e.g., fine particulate matter (PM2.5)
in the case of the criteria pollutant particulate matter) that are particularly relevant for
reviewing the NAAQS. Studies only reporting associations with undefined mixtures (e.g.,
diesel exhaust) or their surrogates (e.g., distance to the roadway) are generally not used
to inform ISA conclusions.
Studies of mixtures can be informative in an ISA if health effect associations with
the particular criteria pollutant under evaluation are also presented. Such studies reflect
real-world exposures and can provide insight into combinations of effects or potential
modification of the criteria pollutant's effect by the broader pollutant mixture. Inference
in the context of the ISA is generally strongest from mixture studies that present (1)
independent effect estimates for the pollutant of interest that adequately control for or
otherwise address the potential for confounding by co-occurring pollutants (Section
A.5.2.6, below) and (2) formal analyses examining how co-occurring pollutants and/or
the broader pollution mixture may modify or confound the independent effects of the
pollutant of interest. In contrast, studies only presenting associations with mixtures (i.e.,
with no independent effect estimates for the criteria pollutant under evaluation) are
typically not policy-relevant in the context of the ISA.
A.5.2.4. Exposure Assessment or Assignment
The primary NAAQS are intended to provide requisite protection of the public
health against the effects of exposures to criteria air pollutants in ambient air. However,
A-32
-------
information about ambient air exposures is rarely available for individual study
participants. Often, epidemiologic studies use surrogates for personal exposures.
Exposure surrogates commonly used in epidemiologic studies include concentrations
measured by ambient air monitors, modeled ambient or outdoor air concentrations,
indoor pollutant concentrations, exposures measured with personal samplers or sensors,
and biomarker concentrations. The ISAs emphasize epidemiologic studies with clear
justifications for their exposure surrogates (e.g., in terms of capturing spatiotemporal
variation in exposures), as well as studies that compare results across multiple valid
exposure assessment methods. The bullets below describe attributes of epidemiologic
study exposure estimates that are considered as part of the ISA study quality
evaluations.
• Spatiotemporal variability: Ambient air concentrations of criteria pollutants vary
spatially and temporally. Exposure estimates typically have smaller biases and
smaller reductions in precision for pollutants with less spatial variation, compared
with spatially heterogeneous pollutants (Sheppard et al., 2005). For pollutants
with lower spatial variability, exposure measurement error typically causes health
effect estimates to be underestimated (Zeger et al., 2000). In these situations,
biases and decreases in the precision of the association (i.e., wider 95% CIs) tend
to be relatively small (Rothman and Greenland, 1998; Zeger et al., 2000). In the
ISAs, studies with validated exposure estimation methods that capture
spatiotemporal variability appropriate for the study design and location carry
greater weight, especially for studies involving pollutants with relatively large
spatiotemporal variabilities (e.g., traffic-related pollutants, such as N02). An
emphasis is placed on exposure estimation models (e.g., land use regression
(LUR), ensemble machine learning methods) for which the model validation
and/or cross-validation values are available. Additionally, exposure estimation
models should include the spatial and temporal resolutions of the model.
Emphasis is placed on studies that provide rationale for the exposure assessment
approach to capture the spatial variability, model specification or
misspecification, and smoothness of the concentration surfaces {Richmond-
Bryant, 2020, 6713663}. Spatiotemporal variability can be addressed using
location or year as fixed effects, autocorrelation to account for clustering by
geography or time, robust errors for unmeasured sources of spatial or temporal
variability, hybrid modeling approaches of exposure estimation to capture
changes over a smaller scale, hierarchical Bayesian spatiotemporal models, or
time-activity data (Talbot et al., 2023; Chen et al., 2018; Zou et al., 2004; Zou et al.,
2013; Yoo et al., 2021).
A-33
-------
• Exposure duration: Regardless of the approach to estimating pollutant
exposures, inference is stronger when the exposure duration corresponds with
the time course for physiological changes in the outcome (e.g., short-term
exposures can result in asthma emergency department visit while multi-year
exposures may be appropriate for cancer and other diseases elicited by long-
term exposures).
• Lag Time: When examining outcomes associated with short-term exposures,
study evaluation may focus on specific lags between exposure and outcome
based on evidence for the time course of physiologic changes related to the
health effect or outcome being analyzed. When the existence of a lag between
exposure and outcome is supported, the following hierarchy is used in the
process of selecting results from individual studies of short-term exposures:
(1) Distributed lag models;
(2) Average of multiple days (e.g., 0-2);
(3) If a priori lag days were used by the study authors, these are the effect
estimates presented; or
(4) If a study focuses on only a series of individual lag days, expert judgment is
applied to select the appropriate result(s) to emphasize considering the time
course for physiologic changes for the health effect or outcome being
evaluated.
• Source of exposure estimates: Epidemiologic studies often estimate exposures
to ambient air pollutants using fixed-site ambient air monitors, remote sensing
approaches, modeling, or a combination of inputs from multiple sources. For
some pollutants, studies use biomarker concentrations as estimates of exposure
(e.g., blood lead concentrations). Each of these is discussed below.
- Fixed-site monitors: Concentrations reported from fixed-site monitors
are most informative if they are correlated with personal exposures; if
monitors are located close to study subjects; if monitored concentrations
within a location are correlated; or if monitored concentrations are
combined with time-activity information. Some studies use (low-cost)
sensors to estimate pollutant exposures. Sensors can provide relatively
high spatial and temporal resolution, though sensor validation and
calibration can be a challenge, because sensor readings may be affected or
interfered with by coexisting pollutants, meteorological conditions, and
baseline drifting. These issues should be considered when interpreting
studies that use them.
A-34
-------
- Personal monitoring: Personal monitoring characterizes exposures at the
individual level and provides exposure data attributable to both ambient
air and non-ambient-air sources. Measurement errors (e.g., instrument
error and representativeness of exposures to ambient air) associated with
personal monitoring should be documented.
- Remote sensing: Remote sensing-based approaches can use data from
satellites to estimate ground-level concentrations of some pollutants (e.g.,
PM2.5, NO2, and O3). Remote sensing approaches should be calibrated
using ground-level monitoring networks and their accuracy, precision, and
any interferences should be clearly documented.
- Models: Some epidemiologic studies use atmospheric models to estimate
exposures, either in place of or to supplement measurements from
ambient air monitors. These models may include LUR models, chemical
transport models (e.g., EPA's Community Multiscale Air Quality (CMAQ)
modeling system), dispersion models, geostatistical models (e.g., kriging),
population stochastic models, ensemble machine learning models, and
hybrid models that use inputs from multiple sources of information (e.g.,
including models, satellites, and monitors). 13 Uncertainty in exposure
predictions from these models is largely influenced by model formulations
and the quality of model input data pertaining to precursor emissions or
meteorology, which tends to vary on a study-by-study basis. Emphasis is
generally placed on exposure predictions from models that incorporate
exposure measurements. Studies that use models to estimate exposures
should clearly document the procedures for model development and
validation, as well as model performance under the conditions of the
study.
- Biomarkers: For some pollutants (e.g., Pb), epidemiologic studies use
biomarkers to estimate exposures (e.g., blood lead levels). As noted above
for personal monitoring, biomarkers provide exposure estimates at the
individual level and that are attributable to both ambient air and non-
ambient-air sources. Depending on the biomarker used, exposure
biomarkers may be quite limited with regard to the specific timing and
duration of the exposure represented, as well as how that biomarker may
relate to ambient concentrations. When used, biomarkers should be clearly
13 Population stochastic microenvironmental models are not typically used for exposure assessment in
epidemiologic modeling because the stochastic component of the model can add measurement error
to the health effect estimate. However, microenvironmental exposure models can appropriately inform
risk and exposure assessments conducted as part of a NAAQS review.
A-35
-------
justified and measured using valid, reliable methods with appropriate
characterization of variability.
A.5.2.5. Outcome Assessment and Evaluation
Confidence in study results is greater when outcomes are based on independent
clinical assessments of the health condition, without knowledge of exposure status. This
can include data from clinical examinations conducted as part of the study, as well as
data obtained from hospitals, insurance providers or other health care organizations.
Outcomes based on interviews, self-reports, or analysis of biological indicators are
generally viewed with relatively high confidence when they are defined by consistent
criteria and when outcome data are collected by validated, reliable methods without
knowledge of exposure status. For example, independent, clinical assessment is valuable
for outcomes like lung function or incidence of disease, but self-report of physician
diagnosis may be adequate for outcomes for which validation studies have
demonstrated good reliability of self-reported health outcomes (Murgia et al., 2014;
Weakley et al., 2013; Barr et al., 2002; Muhajarine et al., 1997; Toren et al., 1993). For
biological samples, the stability of the biomarker(s) of interest and the sensitivity and
precision of the analytical method are considered. In general, if errors in the outcome
assessment are not correlated with exposure status, those errors tend to bias results
toward the null.
A.5.2.6. Potential Confounding
Confounding is "...a confusion of effects. Specifically, the apparent effect of the
exposure of interest is distorted because the effect of an extraneous factor is mistaken
for or mixed with the actual exposure effect (which may be null)" (Rothman and
Greenland, 1998). A confounder is associated with both the exposure and the outcome.
Factors are considered to be potential confounders if they are associated with, but not
caused by, the exposure and are risk factors for the health outcome. Not accounting for
confounders can introduce bias and may produce artifactual associations, thus
emphasizing the importance of evaluating the approaches used in individual studies to
account for potential confounding variables and how the strengths of those approaches
might influence weight-of-evidence considerations in causality determinations (NASEM,
2022, pp. 5-6). Inference is stronger for epidemiologic studies that explicitly identify
A-36
-------
confounders and their potential impacts on results and that take steps to minimize
those impacts, often through statistical adjustment and/or study design (e.g.,
minimizing/eliminating confounding through randomization, also see Section A.5.3.6). In
assessing studies that consider confounders, ISAs give preference to those studies that
clearly document the appropriateness of models selected, the validity of methods
employed, and the assumptions underlying those methods. Studies that only provide
correlations or unadjusted (e.g., crude, univariate) effect estimates are not considered.
Potential confounders vary by study population, exposure, and outcome. To
control for confounders, a variety of statistical methods can be employed. Potential
confounders included in statistical analyses should be based on a thoughtful review of
published literature and the evidence of relationships between variables. In considering
the issue of confounding on study conclusions, studies in the ISA may utilize various
approaches for identifying potential confounders, controlling for the role of such
confounders, and accounting for unknown confounders, as described below.
• Identifying potential confounders: Strategies for identifying potential
confounders can include a priori biological considerations, published literature,
causal diagrams (e.g., directed acyclic graphs), and/or statistical analyses.
• Controlling for confounders: Scientific judgment is needed to identify the likely
sources and extent of confounding and determine how effectively selected study
designs and analyses control for confounders. Multivariate regression models are
often used to detect and control for potential confounders by adjusting for
factors that might confound results. These models attempt to control for
characteristics that may differ systematically between exposed and unexposed
study participants. Other approaches that have been applied include, but are not
limited to, matching and weighting methods (Stuart, 2010); g-computation and
substitution estimators (Keil et al., 2020); doubly robust estimators (Diaz, 2019),
bounding approaches (Richardson et al., 2014); quantitative bias analysis
approaches (Lash et al., 2014, Weuve et al., 2018); and sensitivity analyses. The
approach selected should avoid post-treatment confounding caused by
inappropriate adjustment for post-treatment variables (e.g., collider bias).
Additionally, alternative methods for confounder control (i.e., causal inference
frameworks) have been used in epidemiologic studies to mimic key aspects of
randomized trials.
• Accounting for unknown confounders: Approaches to handling unknown
confounders include, but are not limited to, instrumental variables (Greenland,
A-37
-------
2000); bounding approaches (Richardson et al., 2014); quantitative bias analysis
approaches (Lash et al., 2014, Weuve et al., 2018); and sensitivity analyses (e.g.,
VanderWeele et al. 2017). Additionally, study designs in which individuals or
populations serve as their own controls (see Section A.5.2.2) inherently reduce
potential confounding by some unknown or unmeasured confounders. In
prioritizing studies for inclusion in the ISA, preference is given to studies with
clear discussions of the assumptions underlying selected approaches, the
robustness of those assumptions, and any sensitivity analyses conducted.
Confidence that unmeasured confounders are not responsible for study findings
is increased when multiple studies are conducted in various settings using
different subjects or exposures, each of which might eliminate different sources
of confounding. Multicity studies can provide insight into the potential impact of
unknown confounders on study results through the use of a consistent method
to analyze data across locations with different concentrations of copollutants and
other covariates.
For short-term exposure studies, concern is greatest for potential confounders
that vary temporally on time scales similar to the variation in exposures and health
outcomes. Confounders commonly considered in studies of short-term exposures
include, but are not limited to, the following:
• Meteorology,
• Copollutants, particularly those arising from the same source(s) as the pollutant of
interest,
• Day of week and season,
• Stress and noise,
• Allergen exposure, and
• Long-term temporal trends.
For studies of long-term exposures, concern is greatest for potential confounders
that vary spatially. Confounders commonly considered in studies of long-term
exposures include, but are not limited to, the following:
• SES, race/ethnicity, and age,
• Smoking rates,
• Stress and noise,
• Residential housing age,
• Occupational exposures, and
A-38
-------
• Short-term exposures.
Additionally, studies of long-term exposure that do not also appropriately
address relevant time-varying confounders (e.g., societal patterns and trends in smoking
rates, medication use) can lead to erroneous conclusions regarding support for causal
relationships. A number of methods have been employed to handle time-varying
confounders in studies of long-term exposure, including, but not limited to, marginal
structural models (Robins et al., 2000); regression adjustment using the longitudinal g-
computation formula (Bang and Robins, 2005; Hernan and Robins, 2016); longitudinal
inverse probability weighting (Ertefaie and Stephens, 2010); doubly robust sequential
regression estimators (Diaz et al., 2023; Stitelman et al., 2012); and difference-in-
differences designs (Wing et al., 2018; Greenstone and Gayer, 2009).
For epidemiologic studies that use biomarkers (e.g., blood lead) to estimate
exposures, there is additional concern regarding the specific timing, duration, and/or
frequency of exposures. Depending on the biomarker used, there may be limitations in
the conclusions that can be drawn about the relationship between the exposure and
outcome.
Across exposure durations, the ISAs additionally evaluate the degree to which co-
occurring ambient air pollutants may confound health effect associations with the
criteria pollutant(s) under evaluation. An emphasis on epidemiologic studies that
attempt to minimize such potential "copollutant" confounding may increase confidence
in conclusions regarding outcomes associated with the pollutant under evaluation.
Copollutant modeling (i.e., two-pollutant models) can reduce concern for this
confounding, particularly when correlations between copollutants are relatively low (e.g.,
r < 0.4). However, when correlations are high (e.g., r > 0.7), collinearity between
pollutants makes copollutant modeling less informative, leading to greater uncertainty
in the degree to which reported associations reflect health effects of exposure to the
specific criteria pollutant(s) under evaluation.
A.5.2.7. Effect Measure Modification
Effect measure modification occurs when the effect of a pollutant exposure on a
health outcome of interest differs between subgroups or strata of risk factors (Rothman
and Greenland, 1998). When a risk factor is an effect modifier, it changes the magnitude
A-39
-------
of the association between the pollutant exposure and the health outcome in stratified
analyses. For example, the presence of a pre-existing disease or indicator of low SES
(e.g., educational attainment, household income) may act as an effect modifier if it is
associated with increased risk of effects from air pollution exposure. It is often possible
to stratify the association between health outcome and exposure by one or more of
these potential effect modifiers. For risk factors that modify the association, effect
estimates for each stratum will differ from one another and from the overall estimate,
indicating there to be different quantitative relationships between the exposure metric
and outcome for populations represented by these variables.
Inference can be particularly strong from studies that consider the potential
impacts of effect measure modification, especially when the modifying factors are
coherent with information from other lines of evidence regarding the biological
pathways connecting pollutant exposures with particular health effects (e.g., larger
pollutant-related effects in human populations with pre-existing cardiovascular disease
would be coherent with animal evidence of cardiovascular effects). Such studies can also
be important for quantitatively assessing risks to populations and lifestages at increased
risk of health effects for the criteria pollutant evaluated (Section A.7.2.2). Uncertainty in
inference is lower from studies that articulate and justify assumptions of treatment
effect heterogeneity and that include appropriate diagnostics (e.g., multiple
comparisons) to account for potentially spurious findings.
Evaluation of effect measure modification in the evidence base informs causality
determinations in several ways. First, the presence of effect measure modification can
help identify potentially at-risk populations. Consistent evidence that at least one
population subgroup is at risk of pollutant-related health effects can provide strong
support for a causality determination for that health effect category, though a lack of
evidence for effect measure modification where there is evidence of a pollutant-related
health effect in the general population does not weaken the support for causality.
Evidence for effect measure modification can also explain heterogeneity in results across
studies, which could reduce uncertainties regarding inconsistent evidence. Finally, effect
measure modification can provide supporting information on mechanisms (e.g., genetic
polymorphisms or pre-existing disease) contributing to pollutant-related health effects.
A-40
-------
A.5.2.8. Statistical Methodology
Statistical assumptions should be articulated in the context of a study design
description. Statistical methods that are appropriate for the power of the study carry
greater weight. For example, categorical analyses with small sample sizes can bias
results toward or away from the null. Statistical tests such as correlations, t-tests, and
chi-squared tests are not considered sensitive enough for adequate inferences
regarding health effect associations. Appropriateness and limitations in statistical
approaches should be clearly articulated. Sensitivity analyses with alternative statistical
models or specifications can inform the stability of findings and aid in judgments of the
strength of inference that can be drawn from results. In the case of multiple
comparisons, consistency in the pattern of association can increase confidence that
associations are not due to chance alone. For all methods, the pattern of effect
estimates and precision of the estimates (e.g., width of 95% CI) across studies are
important considerations for assessing the strength and/or patterns of associations,
rather than relying only on statistical significance, which is highly dependent on study
design (Gelman and Greenland, 2019; Greenland et al., 2016; Wasserstein and Lazar,
2016; Wasserstein et al., 2019).
Some epidemiologic studies employ alternative methods for confounder control
that are intended to better account for confounders and to mimic randomized
experiments and reduce potential bias. As discussed above (A.5.2.1), these studies are
often referred to as causal inference studies or studies that used causal modeling
methods in the peer-reviewed literature. In its review of the ISA weight-of-evidence
framework, the NASEM committee noted that "the utility of a causal inference
framework in an individual study depends on its appropriate use" and "[t]he ability to
show mathematical equivalence between statistical and causal parameters does not
make the assumptions required for such equivalence true" (NASEM, 2022). Furthermore,
the NASEM noted that well-designed studies 1) articulate the scientific question in terms
of potential and counterfactual outcomes, 2) specify available data and a causal model,
3) articulate a set of assumptions on the causal model that allow the identification of the
causal parameter of interest as an observable statistical quantity, 4) analyze the data to
estimate the identified statistical quantity, and 5) interpret the statistical results as causal
A-41
-------
relations according to the validity of the assumptions made in steps 3 and 4, and
quantify the statistical uncertainties (NASEM, 2022, pp. 163-164). Thus, consistent with
the approach to evaluating epidemiologic studies that employ traditional regression
models, the ISA considers whether study assumptions are clear and plausible, whether
confounding has been carefully considered and addressed with appropriate approaches,
whether study assumptions are internally consistent and valid, and whether statistical
uncertainties are appropriately quantified when evaluating studies that use such
alternative methods for confounder control (Pearce et al., 2019).
A.5.3. Controlled Human Exposure
Controlled human exposure studies (also known as human clinical studies)
evaluate the health effects of experimental exposures in human volunteers under highly
controlled and carefully regulated environmental conditions and activity levels. These
studies provide direct evidence of physiological and/or biomolecular effects following
air pollution exposures and help to identify the biological pathways linking exposures to
health effects in humans. They can provide strong support for the biological plausibility
of relationships between air pollutant exposures and health effects that are reported by
epidemiologic studies, and precise information on exposure- or dose-response
relationships in populations matched for individual level characteristics, often at
exposure concentrations at or near those common in ambient air. Thus, data from
controlled human exposure studies can provide direct evidence of causal relationships
in humans and can help compensate for some of the limitations in epidemiologic
studies (e.g., potential confounding by co-occurring pollutants or other factors, like
exposure error) (NASEM, 2017). For ethical and practical reasons, controlled human
exposure studies are generally limited to examining relatively healthy people or those
with mild or moderate diseases; short exposure durations; and exposure concentrations
expected to elicit no more than mild, transient effects. As a result, these studies
generally do not include individuals who may be at the highest risk of pollutant-related
health effects (e.g., children, older adults, people with more severe disease or
comorbidities).
A-42
-------
Controlled human exposure studies should clearly describe the primary and any
secondary aims of the study, or specific hypotheses being tested. In evaluating the
quality of these studies, the ISAs additionally consider the appropriateness of the study
design, study population, pollutants examined, approach to assigning exposures,
outcome assessment, control for potential confounders and statistical analysis. Each of
these study attributes is discussed below.
A.5.3.1. Study Design
In prioritizing studies for inclusion in the ISA, preference is given to balanced
crossover (repeated measures) or parallel study designs that include control exposures
to clean air. In a crossover design, each study participant is exposed to both the air
pollutant under evaluation and to clean air, under the same conditions (e.g., ventilation
rate), as a control. In this design, each study participant serves as their own control,
minimizing inter-individual confounders. In crossover studies, a sufficient and specified
time between exposure days should be provided to avoid carry-over effects from prior
exposure days. In studies using a parallel design, all arms should be matched for
individual characteristics, such as age, sex, race/ethnicity, anthropometric properties,
and health status. In studies evaluating effects of disease, appropriately matched healthy
controls are preferred to aid interpretation of study results. The evaluation of study
design generally includes consideration of factors that minimize bias in results, such as
randomization; blinding; allocation concealment of study subjects, investigators, and
research staff; and withdrawal/exclusion of subjects. Studies must include appropriate
control groups to allow for accurate interpretation of results relative to criteria pollutant
exposure.
A.5.3.2. Study Population
Depending on the study design, subjects recruited into study groups should be
matched for age, sex, race/ethnicity, anthropometric properties, and health status. In
studies evaluating effects of specific subject characteristics (e.g., disease, genetic
polymorphism), appropriately matched healthy controls are preferred. Relevant
characteristics and health status should be reported for each experimental group. For
the examination of populations with an underlying health condition (e.g., asthma),
A-43
-------
independent, clinical assessment of the health condition is ideal, but self-report of
physician diagnosis generally is considered reliable for respiratory and cardiovascular
disease outcomes. Criteria for including and excluding subjects should be indicated
clearly, and the loss or withdrawal of recruited subjects during a study should be
reported.
A.5.3.3. Pollutant
As described above for epidemiologic studies, each ISA focuses on studies that
evaluate the effects of exposures to the criteria pollutant(s) under evaluation. Thus,
emphasis is placed on controlled human exposure studies that evaluate individual
criteria pollutants or components of a criteria pollutant (e.g., PM2.5 in the case of
particulate matter) that are particularly relevant for reviewing the NAAQS. Studies of
pollutant mixtures (e.g., O3 as part of an oxidant mixture) can be informative if health
effects of exposure to the criteria pollutant under evaluation, presumably a component
of the mixture, are also examined separately. Such studies can provide insight into
potential modification of the effect of the criteria pollutant by other individual pollutants
or by a broader pollutant mixture. Ideally, studies should report the source, purity, and
form of the pollutant(s) examined.
A.5.3.4. Exposures
Controlled human exposure studies that approximate expected human exposures
in terms of concentration, duration, exercise level, ventilation rate, and method of
exposure are of particular interest for the ISAs. In prioritizing studies for inclusion in the
ISA, preference is given to studies that evaluate pollutant exposure concentrations close
to existing ambient air concentrations, though studies that use higher exposure
concentrations may still provide information relevant to consideration of the biological
plausibility of effects associated with lower exposures and/or dosimetry. Studies should
have measures in place to adequately monitor and control exposure conditions,
including pollutant concentrations, temperature, and relative humidity. The method of
exposure (e.g., chamber, facemask) should be specified, and activity level of subjects
during exposures should be well-characterized. Preference is also given to studies that
include control exposures (e.g., to filtered air or room air). Study subjects should be
A-44
-------
randomly exposed without knowledge of the exposure condition, and exposure metrics
should be well characterized, including external exposure, intake dose, dosing regimen,
and exposure route.
A.5.3.5. Outcome Assessment and Evaluation
For each experiment and each experimental group, including controls, precise
details should be provided describing the endpoints examined, how they are measured,
and when and where they are evaluated. Endpoints should be assessed in the same
manner for control and exposure groups using valid, reliable methods. This includes
using the same procedures in terms of time after exposure, methods, endpoint
evaluator, and similar attributes. Blinding of endpoint evaluators is ideal, especially for
qualitative endpoints such as histopathology. Time of the endpoint evaluations is a key
consideration that will vary depending on the endpoint evaluated. Endpoints should be
assessed at time points that are appropriate for the research questions, and those time
points should be clearly justified.
A.5.3.6. Potential Confounding
To limit potential confounding, studies included in the ISA use either a crossover
repeated measures design, in which each study participant serves as their own control,
or a parallel exposure design, in which experimental and control groups are matched for
individual level characteristics (e.g., race/ethnicity, sex, body weight, smoking history,
age) and time-varying factors (e.g., seasonal and diurnal patterns), similar to the
epidemiologic studies (Section A.5.2.6). Exposures should be well-characterized to
evaluate independent effects of the pollutant(s) under study.
A.5.3.7. Statistical Methodology
Statistical methods should be described clearly and be appropriate for the study
design and research question, including correction for multiple comparisons when
appropriate. Statistical significance is generally used to evaluate the findings of
controlled human exposure studies. Detection of statistical significance is influenced by
a variety of factors, including, but not limited to, the size of the study, exposure and
outcome measurement error, and statistical model specifications. Sample size is not a
criterion for exclusion, though the sample size should provide adequate power to detect
A-45
-------
hypothesized effects. Because statistical tests have limitations, consideration is also
given to trends in data and reproducibility of results. Consistent trends across studies
can be informative, even if results of individual studies are not statistically significant.
A.5.4. Experimental Animal Studies and Emerging Approaches in
Toxicology
Animal toxicological studies evaluate the health effects of air pollutants using
animal models (e.g., non-human primates, mice, rats, and guinea pigs) under well-
controlled pollutant exposure environments. Investigators expose non-human
mammalian animal species to known concentrations of air pollutants under carefully
regulated laboratory conditions. Experimental animal studies provide critical information
on potential human health effects, exposure- and dose-response relationships, and
underlying toxicological pathways and mechanisms of action. One major uncertainty
associated with animal studies is the representativeness of responses in animals to
humans, given potential differences in metabolism, hormonal regulation, breathing
patterns, lung structure, physiology, and anatomy. When these differences are
appropriately characterized, experimental animal studies can inform and improve our
understanding of biological mechanisms, providing support for the biological
plausibility of relationships between air pollutant exposures and health effects that may
be indicated by epidemiologic studies, and address uncertainties in other lines of
evidence (e.g., confounding in epidemiologic studies). In addition, depending on the
animal model used and the study design employed, experimental animal studies may
help inform an understanding of the potential for biological responses and health
effects in certain populations that may be at increased risk of a criteria pollutant-related
health effect.
Emerging approaches in toxicology include new approach methodologies
(NAMs), a term which refers to any technology, methodology, approach, or combination
of these tools that can inform chemical hazard and risk assessment while avoiding the
use of animal testing, including in siiico, in chemico, in vitro, and ex vivo approaches
A-46
-------
(ECHA, 2016; U.S EPA, 2018; U.S EPA, 2021b).14 While ISAs have historically relied on in
vitro and/or in vivo studies to provide mechanistic insight, data collected using well-
validated NAMs may also provide insight into the occurrence of health effects when
guidelines are established for the use of these methods in air pollution studies.
Toxicological studies should clearly describe the primary and any secondary aims
of the study, or specific hypotheses being tested. In evaluating the quality of these
studies, the ISAs additionally consider the appropriateness of the study design, test
model, pollutant(s) examined, exposure assignment and approach, outcome assessment,
variable control, and statistical analysis. Each of these study attributes is discussed
below.
A.5.4.1. Study Design
Toxicological studies should include appropriately time-matched control
exposures, should randomize assignment to exposure groups and, where possible,
should blind research personnel from endpoint evaluation and analysis. Blinding of
research personnel to study groups may not be possible due to animal welfare and
experimental considerations; however, differences in the monitoring or handling of
animals across groups by research personnel should be minimized. Studies should use
methods to limit differences in baseline characteristics of control and exposure groups,
and groups should be subjected to identical experimental procedures and conditions to
the extent possible (e.g., in terms of animal care and housing). The evaluation of study
design generally includes consideration of factors that minimize bias in results, such as
randomization, blinding, and unexplained loss of animals. Where applicable, approval of
14 Experimentation performed using a computer is referred to as an in-silico approach (Cronin, 2009).
Chemical reactivity tests that are performed in the absence of biological materials are referred to as in
chemico approaches (Cronin, 2009). In vitro tests are performed outside of a living organism using
established cell lines, while ex vivo approaches are those that involve the collection of
cells/tissues/organs from living organisms for use in experimentation (European Commission et al.,
2020).
A-47
-------
study protocols by appropriate institutional animal care and use committees must be
obtained (European Commission et al., 2020; Cronin et al., 2009).15
A.5.4.2. Test Model
Toxicological studies should provide a clear justification for their chosen model
system. Unless data indicate otherwise, laboratory nonhuman mammalian animal
species of any lifestage, stock, and strain are considered appropriate for evaluating the
effects of pollutant exposure. Ideally, studies should report species, strain, substrain,
genetic background, age, sex, and weight. It is preferred that the authors test for effects
in both sexes and multiple lifestages and report results for each group separately. All
animals used in a study should be accounted for, and the rationale for any exclusion of
animals or data should be provided.
A.5.4.3. Pollutant
As described above for other disciplines, each ISA focuses on studies that
examine the effects of exposures to the particular criteria pollutant(s) under evaluation.
Thus, emphasis is placed on toxicological studies that examine individual criteria
pollutants or components of a criteria pollutant (e.g., PM2.5 in the case of particulate
matter) that are particularly relevant for reviewing the NAAQS. Studies of pollutant
mixtures can be informative if the health effects of exposure to the criteria pollutant
under evaluation, presumably a component of the mixture, are also examined
separately. Such studies can provide insight into potential modification of the effect of
the criteria pollutant by other individual pollutants or by a broader pollutant mixture.
Ideally, studies should report the source, purity, and form of the pollutant(s) examined.
A.5.4.4. Exposures
Animal toxicological studies should have measures in place to adequately control
exposure conditions, including the identity of the target pollutant, exposure
concentrations, temperature, and relative humidity. Studies that approximate expected
human exposures in terms of concentration, duration, timing of exposure, and method
15 For historical studies that pre-date institutional animal care and use committee requirements detailed in
the Animal Welfare Act, a modified approach to study quality evaluation is applied where use of an
animal care and use committee is not an absolute requirement.
A-48
-------
of pollutant administration are of particular interest, though for some pollutants
exposures within two orders of magnitude of recent ambient air concentrations may be
considered relevant (e.g., if they assess a previously unreported effect or mode of action
for an observed effect or examine multiple concentrations to elucidate exposure-
response relationships). This range in relevant exposures is meant to account for
differences in dosimetry, toxicokinetics, and biological sensitivity between humans,
including groups at increased risk, and the various animal species and strains used in
toxicological studies. Studies using exposure concentrations or doses at the higher end
of this range may be considered to the extent that they provide information relevant to
understanding mode of action or mechanisms, inter-species variation, or factors that
may confer increased risk in human populations.
The focus for ISAs is generally on inhalation exposure, but oral and intravenous
exposures may also be informative for pollutants for which other exposure routes are
relevant and/or if a biomarker (e.g., blood lead) is being examined. Non-inhalation
exposure experiments that target the respiratory tract (e.g., intratracheal instillation [IT])
may be informative if they provide information relevant to biological plausibility and
dosimetry. As discussed above, in vitro studies in validated model systems may be
considered for inclusion in an ISA, though the relevance of in vitro exposure
concentrations to human inhalation exposures are often an additional source of
uncertainty. All studies should include exposure control groups (e.g., filtered air or room
air).
A.5.4.5. Outcome Assessment and Evaluation
For each treatment group, including controls, precise details should be provided
describing the endpoints examined, how they were measured, and when and where they
were evaluated. Endpoints should be assessed in the same manner for control and
exposure groups using valid, reliable methods. This includes using the same procedures
in terms of time after exposure, methods, endpoint evaluator, and similar attributes.
Limits of detection should be provided for quantitative assays, when available. Blinding
of endpoint evaluators is ideal, especially for qualitative endpoints such as
histopathology. Timing of the endpoint evaluations is a key consideration that will vary
A-49
-------
depending on the endpoint under investigation. Endpoints should be assessed at time
points that are appropriate for the research questions.
A.5.4.6. Variable Control
To limit potential impact of other variables on study results, studies included in
the ISA should match experimental and control groups for individual-level
characteristics and time-varying factors. Individual characteristics include strain, sex,
body weight, litter size, feed, and water consumption. Exposures should be well
characterized to evaluate independent effects of the pollutant(s) under study.
A.5.4.7. Statistical Methodology
Statistical methods should be described clearly and should be appropriate for the
study design and research question, including correction for multiple comparisons when
appropriate. Results should be reported with sufficient detail to allow for independent
interpretation (e.g., quantitatively with a measure of variance). Statistical significance is
generally used to evaluate study findings. Detection of statistical significance is
influenced by a variety of factors, including, but not limited to, the sample size used in
the study, exposure and outcome measurement error, and statistical model
specifications. Sample size is not a criterion for exclusion, though the sample size should
provide adequate power to detect hypothesized effects. Because statistical tests have
limitations, the ISAs also consider trends in results across toxicology studies and
reproducibility of study results. Consistent trends across studies can be informative for
weight-of-evidence evaluations, even if results of individual studies are not statistically
significant.
A.5.5. Ecological and Other Welfare Effects
Given the diversity of organisms, levels of ecological organization, endpoints,
study designs, analysis methods, modeling approaches, and reporting conventions that
must be integrated to comprehensively assess the effects of criteria pollutants on
ecosystems, ISAs focus on the general questions formulated in the introduction to this
section (A.5) to guide study evaluation. Generally, ecological and other welfare-focused
studies are considered for further evaluation if they are published in a peer-reviewed
journal, though further scrutiny is applied during full-text screening steps to identify
A-50
-------
whether the methods are clearly described; the selected exposure assessment methods
are appropriate for the research question evaluated; the assumptions, uncertainties, and
limitations of the methods are clearly stated; and QA testing has been performed.
Typical reasons that a study may be excluded from further consideration in the ISAs are:
exposure concentrations exceed concentration guidelines, as specified in the LECES
statement (Table A-6); concentrations are not reported; it is not possible to establish if
the effect was due to a specific pollutant (e.g., in a mixtures study); a lack of statistical
testing for endpoints of interest; inadequate or missing description of methods; or
inadequate study design. For ecological effects, studies at higher concentrations are
used to evaluate ecological effects only when they are part of a range of concentrations
that also includes more ambient-relevant concentrations, or when they inform
understanding of modes of action and illustrate the range of sensitivity to air pollutants
across taxa or across biomes and ecoregions. In evaluating quantitative exposure-
response relationships, emphasis is placed on findings from studies conducted in the
U.S. and Canada as these represent the ecological and climatic conditions most relevant
for review of the NAAQS.
In evaluating studies on climate and visibility, emphasis is placed on studies that
use well-established measurement and modeling techniques, especially those that
report uncertainty or compare results from a variety of techniques. In climate modeling,
novel methods, particularly those using long-term satellite observations, may also be
informative in addressing knowledge gaps not well characterized by existing techniques.
Relevant climate studies include those evaluating direct and indirect climate impacts of
criteria air pollutants at a global scale; for visibility, studies conducted in the U.S. and
Canada provide information more applicable for reviewing the NAAQS. In both cases,
studies that evaluate effects by source sector or region (e.g., regional climate modeling
studies) are particularly informative. Studies that report impacts of multiple PM
components (in the case of visibility) and multiple criteria pollutants (in the case of
climate) are useful in evaluating interactions and the relative contributions of
atmospheric constituents. For example, in evaluating the climate forcing effects of O3, it
is useful to understand the atmospheric chemistry involving CO and NOx (the sum of
nitric oxide and nitrogen dioxide) that affects atmospheric concentrations of O3.
A-51
-------
Visibility impairment and valuation studies that explicitly separate rules for visibility16
from concerns about health risks of air pollution are particularly relevant in considering
a welfare-based secondary NAAQS for pollutants that affect visibility.
A.6. EXTRACTING DATA FROM RELEVANT STUDIES
For health studies determined to be appropriate for inclusion in an ISA, relevant
study data are extracted into evidence tables to facilitate evidence evaluation,
information integration, and evidence inventory creation.17 Data are extracted using
literature review software tools (e.g., DistillerSR) or directly into spreadsheets by the EPA
scientists or other personnel trained to perform this task. In all cases, the integrity of
extracted data is confirmed according to the quality assurance plan described below
(A.9). Extracted study summary data are compiled in evidence tables that provide a
high-level overview of the evaluated studies and are used to assist with data analysis,
evidence synthesis, and formulation of conclusions. Examples of data extraction tables
are provided below (Tables A-7 through A-10).
Given the diversity of organisms, endpoints, study types and reporting
conventions in welfare studies, tables are only developed for small subsets of studies
that are similar enough for integration to be feasible and informative.
15 Available at httpsi//www.epa.go¥/¥isibilit¥/¥isibilit¥-~reQulator¥-~actions
17 If a potentially informative study is missing information relevant to extraction endpoints, the EPA may
contact the corresponding author to request that information. The request and author's response will
be included in the docket for that ISA. If the missing information is not available, the study may be
deemed inappropriate for inclusion in the ISA.
A-52
-------
Table A-7. Example Data Extraction Table - Epidemiologic Studies.
Concentration
Effect
Estimates
95% CI
Study
Study
Study
Exposure
and
Outcome
Statistical
Reference
Design
Population
Assessment
Copoiiutant
Assessment
Methods
Examination
Study
Population
Exposure
Mean/median
Description
Model type
ORs,
design
details
model,
monitor data
value
(reported by
of outcome
and list of
confounders
RRs, |B
HERO ID,
author(s),
Location
study and/or
Years
Annual,
standardized)
Min, Max
Pearson or
Spearman
rho value(s)
for each
copoiiutant
year
(recruitment)
(follow-up)
N
monthly
Model
development
and
validation
Max, maximum; min, minimum; N, sample size; ORs, odds ratio; RRs, risk or rate ratios.
Table A-8. Example Data Extraction Table - Animal Toxicological Studies.
Study
Reference
Study
Population
Exposure Details
Endpoints Examined
HERO ID,
author(s), year
Species/strain,
age, & sample
size
Exposure route, concentration, particle
size distribution (if applicable), duration,
timing of exposure, control group
composition, exposure biomarker (if
relevant)
Outcome(s) measured,
timing of outcome
measurement, results
summary
Table A-9. Example Data Extraction Table - Controlled Human Exposure Studies.
Study
Reference
Study
Population
Exposure Details
Endpoints Examined
HERO ID,
author(s), year
Population details
(i.e., Sample size,
age, sex, disease
state, etc.)
Target pollutant concentration or range,
subject activity, duration of exposure,
frequency of exposure
Outcome(s) measured, timing
of outcome measurement,
results summary
A-53
-------
Table A-10.Example Data Extracted Table - Exposure Science Studies.
Study
Reference
Study Rationale
Sampling and
Modeling
Data Analysis
Exposure
Factors
Results
HERO ID,
Hypothesis; Study
Exposure
Statistical
Exposure
Exposure
author(s), year
location selection;
indicators;
analyses
factor data and
levels for
Inclusion and
sampling
methods;
the basis for
various
exclusion criteria for
frequency and
summary
choosing
populations;
study participants
duration;
statistics;
certain values
sources and
(age, gender, ethnic
Sampling
uncertainty and
of exposure
extent of
group, health status,
methods; model
variability in the
factors
exposure
SES, etc.); Sample
input parameters
findings; model
errors;
size determination;
performance
correlations
Data quality
with
objectives
copollutants
exposures
A.7. DRAFTING ISA SECTIONS
A.7.1. Draft Chapters and Obtain Peer Input
The ISA authors draft topic-specific chapters in two stages. During the first stage,
initial drafts are developed and used to inform discussions during a peer input
workshop as described below. Initial draft chapters typically include an introduction, a
summary of the previous ISA conclusions, an overview of the scope of the assessment in
the current ISA, and a review of the available science. The integrated summary and
policy-relevant conclusions (e.g., causality determinations, conclusions on at-risk
populations) are not developed in this initial drafting stage.
The peer input workshop brings together the EPA and external subject matter
experts from a variety of disciplines to review the draft chapters. The purpose of the
workshop is to obtain early feedback from experts in relevant fields to ensure that the
draft ISA reflects the most up-to-date, policy-relevant science. Discussions at the peer
input workshop can also provide a foundation for initial integration of evidence within
and across disciplines. During the peer input workshop, expert panelists are asked to
address several overarching questions, often including, but not limited to, the following:
• To what extent do the initial draft materials capture the key studies published
since the cutoff date of the prior ISA?
• What are panelists' views on the specific issues that should be considered or
highlighted and that will be important for integrating evidence across disciplines?
A-54
-------
• To what degree are study results accurately reported and appropriately
interpreted?
EPA staff may ask workshop panelists additional questions related to the
organization and content of specific draft chapters. The public is invited to listen to the
discussion during the workshop, but no draft ISA materials are shared publicly, and
comments are not solicited from the public at this time. With consideration of the
feedback received at the workshop, the ISA authors update the initial draft chapters,
integrate evidence across disciplines, and develop policy-relevant scientific conclusions,
including causality determinations and conclusions on populations that may be at
increased risk. The approaches to integrating evidence and developing these
conclusions are discussed in detail in the following sections.
A.7.2. Develop Policy-Relevant Scientific Conclusions
Drawing upon the results of studies determined to be relevant and of adequate
quality, the ISAs reach multiple policy-relevant conclusions intended to inform the
NAAQS review. Most prominently, the ISAs use structured frameworks to reach
conclusions on the strength of the scientific evidence supporting causal relationships
between pollutant exposures and adverse public or welfare effects (i.e., causality
determinations) and on the populations that may be at increased risk of such adverse
health effects (i.e., at-risk populations).18 While discussed separately, these frameworks
are carefully integrated within the ISA, with the evidence for increased risk in certain
populations often providing substantial support for causality determinations. The ISA
frameworks for reaching conclusions on causality and at-risk populations are described
further below.
A.7.2.1. Causality Determinations
The 1964 Surgeon General's report on tobacco smoking defined "cause" as a
"significant, effectual relationship between an agent and an associated disorder or
18 The ISAs also typically present findings on other policy-relevant issues, which vary by pollutant and
discipline and can include concentration-, exposure-, and/or dose-response relationships; the exposure
concentrations below which evidence for effects is limited; the potential adversity of particular effects;
the effects of particular pollutant components (e.g., for PM2.5) and other topics. These are described
more fully in individual ISAs.
A-55
-------
disease in the host" (Hew, 1964). More generally, a cause is an agent that brings about
an effect or a result. Unlike an association, a causal claim supports the creation of
counterfactual claims; that is, a claim about what the world would have been like under
different or changed circumstances (IOM, 2008).
The ISAs evaluate and integrate scientific evidence on health or welfare effects of
criteria pollutant exposures and, based on this integration and the weight of evidence in
support of causation, present a determination regarding the existence of a causal
relationship between pollutant exposure and various health or welfare effects.
Evaluating the potential for cause-effect relationships between criteria pollutant
exposures and health and welfare effects is made challenging by the fact that many of
the effects or outcomes evaluated in the ISAs have complex etiologies. Plant growth and
ecosystem health can be affected by a number of factors that include, but are not
limited to, air pollution exposures. Diseases such as asthma, coronary heart disease, and
cancer are typically initiated by multiple environmental and biological factors. Outcomes
in a given individual can depend on factors such as age, genetic background, nutritional
status, immune competence, and social factors (IOM, 2008; Gee and Payne-Sturges,
2004). Further, exposure to a combination of agents could cause synergistic or
antagonistic effects. Thus, the observed risk may represent the net effect of many
actions and counteractions.
The ISA framework for reaching causality determinations recognizes that,
compared to any single study, the availability of multiple studies evaluating a particular
topic, each with different strengths and limitations, provides a more robust foundation
for evaluating the overall strength of the evidence. The existing framework is described
in the 2015 Preamble to the ISAs (U.S. EPA, 2015). That framework was informed by the
weight-of-evidence approaches formulated by other regulatory and science agencies,
including the National Academy of Sciences (NAS) Institute of Medicine (IOM, 2008), the
International Agency for Research on Cancer (IARC, 2006), the EPA Guidelines for
Carcinogen Risk Assessment (U.S EPA, 2005), the Centers for Disease Control and
Prevention (CDC, 2004), the EPA's Integrated Risk Information System (U.S EPA, 2022),
and the World Health Organization (WHO, 2021). The frameworks used by each of these
organizations are similar in nature but adapted to different purposes and have proven
A-56
-------
effective in providing uniform structure and language for causality determinations. The
causality framework described below builds on the 2015 Preamble to the ISAs (U.S. EPA,
2015), with updates reflecting CASAC feedback over multiple ISAs since 2015 and recent
NASEM recommendations (NASEM, 2022).
A.7.2.1.1. Aspects of the evidence important to
judging causality
In making judgments regarding the potential for causal relationships, the ISAs
consider various aspects of causality drawn from previous historical efforts, which
focused largely on epidemiologic studies. The 1964 Surgeon General's report on
tobacco smoking discussed criteria for the evaluation of epidemiologic studies, focusing
on consistency, strength, specificity, temporal relationship, and coherence (Hew, 1964).
Sir Austin Bradford Hill (Hill, 1965) articulated similar aspects of causality in
epidemiology and public health that have been widely adopted (IOM, 2008; IARC, 2006;
U.S EPA, 2005; CDC, 2004). The EPA has adapted this list of characteristics for use in the
ISA causality determinations specific to health and welfare effects of criteria pollutant
exposures (U.S. EPA, 2015).
Table A-11 and the accompanying text describe key aspects of the evidence base
that the ISA considers in judging causality. Although these aspects provide a framework
for assessing the evidence, they do not lend themselves to being considered in terms of
simple formulas or fixed rules leading to conclusions about causality (Hill, 1965). For
example, one cannot simply count the number of studies reporting statistically
significant results or statistically nonsignificant results and reach credible conclusions
about the relative weight of evidence and the likelihood of causality. The aspects cannot
be used as a strict checklist, and failure to meet one or more of the principles does not
automatically preclude a determination of causality [see discussion in CDC (2004)].
Rather, these aspects provide a framework for systematic appraisal of the body of
evidence, informed by peer input and public comment, which includes weighing
alternative views on controversial issues. Additional context for interpreting the aspects
in Table A-11 is provided in subsequent sections.
A-57
-------
Table A-11.Aspects of the evidence important to judging causality.
Aspect
Description
Consistency
An inference of causation is strengthened when a pattern of elevated risks is
observed across multiple independent studies. The reproducibility of findings in
different groups and using different study designs constitutes one of the strongest
arguments for causation. Statistical significance is not the sole criterion by which
the presence or absence of an effect is determined. If there are discordant results
among investigations, possible reasons such as differences in exposure,
confounding factors, and the power of the study are considered.
Coherence
An inference of causation is strengthened when multiple lines of evidence
independently support the occurrence of related effects following pollutant
exposure. Such coherence can be demonstrated by evidence across various
disciplines and/or study designs.
Biological plausibility
An inference of causation is strengthened by results from experimental studies or
other sources demonstrating biologically plausible mechanisms by which pollutant
exposures could lead to adverse outcomes.
Biological gradient
(i.e., exposure-
response relationship)
A well-characterized exposure- or dose-response relationship (e.g., larger, more
serious effects associated with higher exposure/dose) can strongly support
causation, especially when such relationships are observed across multiple
disciplines and durations of exposure.
Strength of the
observed association
The finding of large, precise risks increases confidence that an association is likely
not due to chance, bias, or other factors. However, an effect estimate that is small
in magnitude does not necessarily indicate a lack of causation.
Experimental
evidence
Strong evidence supporting causation can be provided by experimental studies
(i.e., controlled human exposure and toxicological) and by studies of "natural
experiments" when a change in exposure is found to result in a change in
occurrence or frequency of health or welfare effects.
Temporality of the
observed association
Evidence that pollutant exposure precedes the appearance of the effect, and that
the interval between exposure and effect are reasonable based on available
science, supports causation.
Specificity of the
observed association
Evidence linking exposure to a specific outcome can provide strong support for
causation. However, lack of specificity does not necessarily indicate a lack of
causation, since it is rarely expected that a pollutant exposure will invariably
predict the occurrence of an outcome, and since a given outcome may have
multiple causes.
A.7.2.1.1.1. Consistency
In assessing the consistency of findings across studies for evaluating the weight
of evidence, the ISAs emphasize the pattern of results in a body of epidemiologic or
experimental studies examining related outcomes. Consistency of findings is informed
by the repeated observation of effects or associations across multiple independent
studies. Results that are reproducible across variations in study designs or analytic
choices "can be viewed as more robust and as stronger evidence for a causal
A-58
-------
relationship" (NASEM, 2022, p. 7). Thus, the strength of the evidence is increased when
similar findings are reported in different populations under different circumstances. For
epidemiologic studies, the ISA's evaluation of consistency includes consideration of the
direction and magnitude of associations across independent studies, with greater
emphasis on the pattern of results across studies than on the statistical significance of
results in individual studies. Statistical significance is influenced by a variety of factors
including, but not limited to, the size of the study population, exposure and outcome
measurement error, and statistical model specifications. Statistical significance may be
informative; however, it is just one means of evaluating confidence in the observed
relationship and assessing the probability of chance as an explanation. As statistical
inferences may result in both false positives and false negatives, the ISAs emphasize the
pattern of associations across epidemiologic studies. Statistical significance of results is
traditionally given greater emphasis in the evaluation of consistency across controlled
human exposure and animal toxicological studies, though the pattern of results across
such experimental studies using similar designs and examining related effects can also
be informative. Discordant results among independent studies may be explained by
differences in study methods, random errors, exposure errors, confounding factors, or
study power, and the ISAs explore such potential explanations for studies with results
that are not consistent.
A.7.2.1.1.2. Coherence
In evaluating coherence of the evidence base, the ISAs examine the degree to
which studies from different disciplines, or studies from the same discipline with
fundamentally different designs, support the occurrence of effects as a result of
pollutant exposures. For example, the evidence base may include epidemiologic studies
reporting positive associations between pollutant exposures and cardiovascular events
that are coherent with controlled human exposure and/or experimental animal studies
demonstrating changes in cardiac or vascular function following exposures. Evidence
that is coherent across disciplines and/or study designs provides stronger support for a
causality determination than any individual line of evidence alone.
For bodies of evidence characterized by limited coherence, the implications for
causality determinations should be carefully considered. In particular, it is important to
A-59
-------
account for how study design limitations can impact results and potentially explain
inconsistencies in the broader evidence base. For example, caution may be appropriate
when reaching conclusions on the absence of effects, or on the exposure concentrations
below which effects no longer occur, in controlled human exposure studies that examine
relatively healthy volunteers with short exposure durations and short outcome
evaluation periods. Such study characteristics could bias results towards not detecting
effects that may occur in less healthy populations and/or with longer-term exposures or
evaluation periods.
A.7.2.1.1.3. Experimental Evidence and Biological
Plausibility
In making judgments regarding causality, the ISAs specifically consider the extent
to which experimental studies provide evidence of effects resulting from air pollutant
exposures. Experimental studies provide valuable information on the relationships
between exposures and observed effects under well-defined conditions, and they can
provide insight into the biological plausibility of associations reported in epidemiologic
studies. Biological plausibility for a causal relationship between pollutant exposure and a
particular effect can be supported by experimental studies that provide an
understanding of the mode of action through which pollutant exposures lead to health
effects. This understanding may span multiple levels of biological organization
including, but not limited to, molecular and cellular events in the pathways leading to
disease. A complete understanding of the mode of action is rarely available and is not
necessary for it to be biologically plausible that an exposure-effect relationship reflects a
causal relationship. Rather, experimental studies demonstrating that a pollutant elicits
key physiological events in the pollutant exposure-to-response pathway, and the
relationships between those events, can provide strong support for causal inferences of
health effect associations reported in epidemiologic studies. Due to the complexities
and uncertainties related to extrapolation from non-human experimental model
systems, the ISA may draw upon toxicological studies that use exposure concentrations
higher than those considered relevant for typical ambient air exposures in human
populations to inform consideration of biological plausibility of relationships between
pollutant exposures and various health effects.
A-60
-------
A.7.2.1.1.4. Biological Gradient
The presence of concentration-, exposure-, and/or dose-response relationships in
a study can increase confidence in a finding that exposure may be causative, particularly
when such relationships are demonstrated across scientific disciplines and/or multiple
independent studies in which confounders have been addressed. The shapes of
concentration-, exposure-, or dose-response curves, and whether those curves are linear
across the range of ambient air exposures, can also be an important consideration in
characterizing the public health impacts associated with pollutant exposures. Sources of
variability and uncertainty in interpreting concentration-, exposure-, and dose-response
relationships include limitations in the data available toward the lower and upper ends
of the concentration range, the possible influence of exposure measurement error over
the range of concentrations, and variability in response among individuals with respect
to air pollution health effects. These sources of variability and uncertainty tend to
smooth and "linearize" concentration-, exposure-, and dose-response functions. As a
result, they can obscure the existence of nonlinear relationships and thresholds and may
explain why exposure-response data from epidemiologic studies of ambient air
concentrations for some environmental pollutants (e.g., PM, O3, Pb, environmental
tobacco smoke, radiation) do not exhibit population-level thresholds for cancer or
noncancer health effects, even though likely mechanisms include nonlinear processes
for some key events.
A.7.2.1.1.5. Strength and Specificity of
Associations
In evaluating the strength of observed associations in epidemiologic studies, the
ISAs consider the magnitude, statistical precision (i.e., informed by width of confidence
intervals), and the specificity of health outcomes across multiple studies. In large studies
that adequately account for potential confounding factors, strong associations can serve
to increase confidence that findings are not due to a weak unmeasured confounder,
chance, or other biases. However, health effects evaluated in the ISAs tend to have
multiple contributing factors (e.g., genetics, disease, lifestyle, environmental), and the
magnitude of the contribution from air pollution exposures will depend on the
prevalence of other risk factors in the study population. Thus, in studies that
A-61
-------
appropriately account for potential confounding factors and other sources of bias, a
small effect size does not rule out a causal relationship with the air pollutant, and such
an effect can be important from a public health perspective if it impacts large segments
of the population. A small effect can represent a shift in the distribution of responses in
the study population and may increase the proportion of individuals with clinically
important outcomes.
A.7.2.1.1.6. Temporality of the Observed
Association
Temporality of an observed association refers to the temporal sequence of
exposure and observed effects. For a causal relationship, pollutant exposure must
happen before effects. Experimental animal and controlled human exposure studies
demonstrating exposure-induced health effects provide strong support for appropriate
temporal relationships between exposures and effects reported in observational
epidemiologic studies. Not all observational studies provide evidence of temporality. For
example, cohort studies, by design, are generally better suited to address the
consideration of the temporal sequence of exposure and effect than cross-sectional
studies.
A.7.2.1.2. ISA framework for making causality
determinations
Using the aspects of the evidence described above to make judgments related to
causality, the ISAs assess the relevant scientific literature to draw conclusions on the
causal nature of the relationships between relevant pollutant exposures and health and
welfare effects. These "causality determinations" reflect overall confidence in causal
relationships based on the strengths and limitations of the full body of evidence,
integrated within and across disciplines. In its review of the causality framework in the
2015 Preamble to the ISAs (U.S. EPA, 2015), the NASEM supported this approach, noting
that "[a] weight of evidence approach, which combines assessment of the scientific
literature with expert judgment to weigh that complex literature, is a scientifically
defensible approach for the ISA causal determination framework" (NASEM, 2022, p.
126). The ISAs evaluate evidence for major health and welfare effects categories or
groups of related endpoints (e.g., respiratory effects), characterizing the strengths and
A-62
-------
limitations of evidence for individual endpoints within the broader category. Limitations
in the evidence base can result from the consistent presence of uncertainties within a
group of studies (e.g., studies similarly affected by confounding, exposure error, and/or
species extrapolation) or uncertainties that exist across the broader body of evidence
(e.g., inconsistent evidence across disciplines, lack of coherence). The ISAs generally rely
on qualitative uncertainty evaluations, though quantitative analysis approaches such as
meta-regression are used in some situations. Rigorous external peer-review by the
CASAC and pollutant-specific expert panels is critical to informing ISA conclusions on
uncertainty and/or bias in the body of evidence supporting causality determinations.
For ecological effects, evaluations of causality generally include the quantitative
characterization and prediction of commensurate changes in effects with changes in
exposure. Exposure-response relationships are often determined for a specific ecological
system and scale (e.g., Northwestern coniferous forests) and may not be generalizable
to larger regions. With increasing scale, two difficulties in generalizing exposure-
response models emerge: first, more variables come into play whose impact on the
relationship is increasingly less likely to have been quantified, and second, the species,
populations or ecosystems to which they apply do not extend to larger scales.
The ISA causality determinations are articulated using a framework with a five-
level hierarchy based on the weight of evidence for causation (Table A-12). The NASEM
endorsed this approach, noting that the five categories for classifying causality
determinations "are scientifically defensible given the precautionary nature of the CAA"
(NASEM, 2022, p. 3). The standardized language used in the framework to describe
specific determinations is adapted from sources across the federal government and the
wider scientific community, especially the EPA Guidelines for Carcinogen Risk Assessment
(U.S EPA, 2005), U.S. Surgeon General's report, The Heaith Consequences of Smoking
(CDC, 2004), and NAS IOM document, Improving the Presumptive Disability Decision-
Making Process for Veterans (IOM, 2008). Table A-12 presents the human health and
welfare descriptors for each of the determinations in the ISA causality framework.
A-63
-------
Table A-12.Causality determinations for health and welfare effects.
Descriptor
Health Evidence Characteristics
Ecological and Other Welfare
Evidence Characteristics
Causal
relationship
Evidence is sufficient to conclude that there
is a causal relationship with relevant
pollutant exposures. That is, the pollutant
exposures have been shown to result in
health effects across studies in which
chance, confounding, and other biases can
be ruled out with reasonable confidence. A
"causal" relationship is generally based on
multiple high-quality studies conducted by
different research groups. Evidence
supporting this determination can include
controlled human exposure studies that
consistently demonstrate effects and/or
observational studies reporting consistent
health effect associations that, when
considered in light of study quality and
coherence with other lines of evidence (i.e.,
controlled human exposure studies, animal
toxicological studies, and mode of action
information), cannot be explained by
plausible alternatives.
Evidence is sufficient to conclude that
there is a causal relationship with
relevant pollutant exposures. That is,
the pollutant has been shown to result in
effects across studies in which chance,
confounding, and other biases could be
ruled out with reasonable confidence.
Controlled exposure studies (laboratory
or small- to medium-scale field studies)
provide the strongest evidence for
causality, but their scope of inference
may be limited. Generally, the
determination is based on multiple
studies conducted by multiple research
groups, and evidence that is considered
sufficient to infer a causal relationship is
usually obtained from the joint
consideration of many lines of evidence
that reinforce each other.
Likely to be a
causal
relationship
Evidence is sufficient to conclude that a
causal relationship is likely to exist with
relevant pollutant exposures. That is, the
pollutant exposures have been shown to
result in health effects in studies where
chance, confounding, and other biases are
minimized, but uncertainties remain in the
evidence overall. A "likely to be causal"
relationship is generally based on multiple
high-quality studies conducted by different
research groups. Evidence supporting this
determination can include 1) multiple high-
quality observational studies consistently
reporting health effect associations, but with
uncertainty remaining related to potential
confounding and/or limited coherence with
other lines of evidence (i.e., controlled
human exposure studies, animal
toxicological studies, mode of action
information) or 2) consistent evidence in
animal models and/or in vitro models (e.g.,
for cancer-related effects) that can be
reasonably extrapolated to human health,
but limited availability of human data.
Evidence is sufficient to conclude that
there is a likely causal association with
relevant pollutant exposures. That is, an
association has been observed between
the pollutant and the outcome in studies
in which chance, confounding, and other
biases are minimized but uncertainties
remain. For example, field studies show
a relationship, but suspected interacting
factors cannot be controlled, and other
lines of evidence are limited or
inconsistent. Generally, the
determination is based on multiple
studies by multiple research groups.
A-64
-------
Suggestive of,
but not sufficient
to infer, a causal
relationship
Evidence is suggestive of, but not sufficient
to infer, a causal relationship with relevant
pollutant exposures. That is, the pollutant
exposures have been shown to result in
health effects, but chance, confounding, and
bias cannot be ruled out with confidence.
Evidence supporting a "suggestive"
relationship can include studies of varying
quality that may be generally supportive of
pollutant-related effects, but not entirely
consistent and with limited coherence across
lines of evidence. A suggestive
determination can be reached with relatively
small bodies of evidence or, in rare cases,
one high quality study.
Evidence is suggestive of a causal
relationship with relevant pollutant
exposures, but chance, confounding,
and bias cannot be ruled out with
confidence. For example, at least one
high-quality study shows an effect, but
the results of other studies are
inconsistent.
Inadequate to
infer a causal
relationship
Evidence is inadequate to determine that a
causal relationship exists with relevant
pollutant exposures. That is, the evidence
supporting an "inadequate" relationship is limited
and available studies are of insufficient
quantity, quality, consistency, and/or
statistical power to permit a conclusion
regarding the presence or absence of an
effect.
Evidence is inadequate to determine that
a causal relationship exists with relevant
pollutant exposures. The available
studies are of insufficient quality,
consistency, or statistical power to
permit a conclusion regarding the
presence or absence of an effect.
Not likely to be a
causal
relationship
Evidence indicates there is no causal
relationship with relevant pollutant
exposures. Several adequate studies,
covering the full range of exposures that
human beings are known to encounter and
considering at-risk populations and
lifestages, are consistent in not showing an
effect at any level of exposure.
Evidence indicates there is no causal
relationship with relevant pollutant
exposures. Several adequate studies
examining relationships with relevant
exposures are consistent in failing to
show an effect at any level of exposure.
Each level of the causality hierarchy is delineated by the degree to which chance,
confounding, and other biases can be ruled out as explanations of study results with
reasonable confidence. The conclusion on which level within the hierarchy best fits a
particular body of evidence is informed by considering the aspects of the evidence
described above (i.e., consistency, coherence, biological plausibility, biological gradient,
strength of associations, experimental evidence, temporality, and specificity). ISAs use
these aspects to guide evidence integration both across populations and within groups
that share characteristics with the potential to modify exposure-response relationships,
particularly characteristics that may place a group at higher risk. As noted by the NASEM,
"[heightened human response can be due to age, comorbidities, or other
A-65
-------
environmental, socio-economic, behavioral, epigenetic or genetic factors" NASEM, 2022,
p. 4). Given this, the NASEM cautioned that "considering, or highlighting, only overall
average population or broad ecosystem effects can obscure causal relationships that
exist for more sensitive subgroups, subspecies, communities, or ecosystems" (NASEM,
2022, p. 128). Consistent with this advice, the ISAs take into consideration the
heterogeneity in exposure-response relationships often demonstrated between
individuals and populations. When studies informing causality determinations consistently
demonstrate elevated pollutant-related risks in populations with particular characteristics, the
resulting determinations reflect the strength of the evidence for effects in those populations.
A.7.2.2. At-risk populations
As noted above, some populations and lifestages may be at greater risk of criteria
pollutant-related health effects than the general population. Higher risks could be due
to intrinsic, extrinsic, and/or exposure-related factors. Intrinsic factors such as genetics,
lifestage, or disease status can contribute to larger or more serious responses to a
particular pollutant exposure, and physiological differences between groups (e.g.,
differences in breathing patterns between children and adults) can contribute to higher
internal pollutant doses in some populations. Extrinsic factors (e.g., nutritional status)
and exposure-related factors (e.g., working outdoors, living near roadways or other
pollution sources) can also contribute to increased risk, and many of the characteristics
commonly used to classify populations potentially at increased risk (e.g., race, SES,
educational attainment) are surrogates for the combined impact of several intrinsic,
extrinsic, and exposure-related factors. The co-occurrence of risk factors in some
populations, including those with environmental justice concerns or minority groups
who may have legacy impacts for some risk factors, presents a complex public health
challenge. A critical part of characterizing the public health impacts of criteria pollutant
exposures under the NAAQS is identifying the specific populations that are at greater
risk of pollutant-related health effects and understanding, where possible, the
combinations of intrinsic, extrinsic, and exposure-related factors that confer the greatest
risk.
The scientific community has used a variety of terms to classify the populations
that may be at increased risk of pollutant-related health effects. These terms, which have
A-66
-------
been defined inconsistently across the scientific literature, include susceptible,
vulnerable, and sensitive populations (Vinikoor-lmler et al., 2014; Sacks et al., 2011; U.S.
EPA, 2009; U.S EPA, 2010). The lack of consensus in terminology across the scientific
community led previous reviews and early ISAs to adopt the term "susceptible
populations" to encompass the various factors that could confer increased risk
(Vinikoor-lmler et al., 2014; Sacks et al., 2011; U.S. EPA, 2009; U.S EPA, 2010). However,
this terminology proved problematic because the broader scientific community often
describes susceptible populations as those at increased risk specifically due to biological
or intrinsic factors such as pre-existing disease or lifestage. Therefore, starting with the
2013 ISA for Ozone and Related Photochemical Oxidants (U.S. EPA, 2013b), the term "at-
risk" was adopted to encompass the broad range of intrinsic, extrinsic, and exposure-
related factors that may confer increased risk of criteria pollutant-related health effects
in particular populations and lifestages.
In assessing the overall public health impact of criteria pollutant exposures, the
ISAs identify, evaluate, and characterize risk factors to inform conclusions on the
populations and lifestages that may be at increased risk. As described further below, the
ISAs use a structured framework to characterize potential risk factors and guide
evaluation of the evidence across scientific disciplines to assess the overall confidence
that a specific factor may result in a population or lifestage being at increased risk of an
air pollutant-related health effect. In doing so, the ISAs draw from the evidence
integration underlying causality determinations, with a focus on the epidemiologic,
controlled human exposure, and animal toxicological studies that provide information
on pollutant-related effects in particular populations or lifestages, as well as available
information on differential exposures and dosimetry.
Regarding epidemiologic studies, the ISAs focus particularly on studies that
include stratified analyses, or analyses of effect measure modification, and on studies
that examine effects that are overwhelmingly or exclusively present in specific
populations. Stratified analyses and analyses of effect measure modification can
compare various populations or lifestages exposed to similar air pollutant
concentrations within the same study design. Studies that evaluate effects in specific
populations or lifestages can provide evidence of increased risk in populations that are
A-67
-------
uniquely affected (e.g., lung function development in children; heart failure, heart
attacks, or strokes in people with pre-existing cardiovascular disease). When evaluating
results across epidemiologic studies, consistent with the approach to informing causality
determinations, emphasis is placed on patterns or trends in results in the various
populations evaluated.
Some controlled human exposure and animal toxicological studies evaluate
potential risk factors, such as health status (e.g., pre-existing asthma) or genetic
background, though study participants with serious health conditions are usually
excluded from controlled human exposure studies and limitations in animal models of
human disease often result in important uncertainties. However, when available, these
experimental studies are important for establishing coherence across disciplines. They
can provide information about the independent effects of the air pollutant under
evaluation as well as the biological plausibility of effects observed in epidemiologic
studies examining specific populations. Additionally, dosimetry studies can further
inform the plausibility of a population being at increased risk by demonstrating whether
the deposition and distribution of an air pollutant within the body varies across
populations or lifestages.
An important consideration in evaluating the health evidence for potential at-risk
populations or lifestages is variability across studies in how those groups are defined.
For example, risk in populations with well-controlled pre-existing disease (e.g., asthma,
hypertension) could be substantially different from the risk in populations with
uncontrolled disease. Variability across studies in how potential at-risk populations are
defined can similarly exist for other factors (e.g., body mass index vs. other indicators of
body composition, various indicators of SES, and various age ranges used to define
lifestages). A related consideration is variability within populations or lifestages, such as
behavioral differences, biological differences, and adherence to medical treatments. The
ISAs consider such sources of variation where relevant because they may affect the
extent to which studies can reliably identify a population or lifestage that may be at
increased risk of pollutant-related effects.
In addition to the health evidence, the ISAs consider evidence for differential
exposures when evaluating support for the identification of populations and lifestages
A-68
-------
that may be at increased risk. When available, data from studies examining pollutant
exposures in specific populations can be integrated with data for health effects in those
populations. Such combinations of exposure and health data can inform the ISA's
evaluation of the intrinsic, extrinsic, and exposure-related factors that may confer the
greatest risk to criteria pollutant-related health effects.
The ISA's characterization of risk factors consists of evaluating the evidence
across scientific disciplines and assessing overall confidence that a specific factor may
result in a population or lifestage being at increased risk of an air pollutant-related
health effect. The ISAs use a structured framework with four categories— "adequate
evidence," "suggestive evidence," "inadequate evidence," and "evidence of no effect"—
to characterize the evidence for at-risk populations. These categories are described
below in Table A-13.
Table A-13.Characterization of evidence for factors potentially increasing the risk
of pollutant-related health effects.
Classification
Health Effects
Adequate evidence
There is substantial, consistent evidence within a discipline to conclude that a factor
results in a population or lifestage being at increased risk of air pollutant-related
health effect(s) relative to some reference population or lifestage. Where applicable,
this evidence includes coherence across disciplines. Evidence includes multiple high-
quality studies.
Suggestive evidence
The collective evidence suggests that a factor results in a population or lifestage
being at increased risk of air pollutant-related health effect(s) relative to some
reference population or lifestage, but the evidence is limited due to inconsistency
within a discipline or, where applicable, a lack of coherence across disciplines.
Inadequate evidence
The collective evidence is inadequate to determine whether a factor results in a
population or lifestage being at increased risk of air pollutant-related health effect(s)
relative to some reference population or lifestage. The available studies are of
insufficient quantity, quality, consistency, and/or statistical power to permit a
conclusion to be drawn.
Evidence of no effect
There is substantial, consistent evidence within a discipline to conclude that a factor
does not result in a population or lifestage being at increased risk of air pollutant-
related health effect(s) relative to some reference population or lifestage. Where
applicable, the evidence includes coherence across disciplines. Evidence includes
multiple high-quality studies.
A-69
-------
A.7.2.3. Public Welfare Impact
Once a determination is made regarding the causality of relationships between
the pollutant and endpoint category, important questions regarding the public welfare
impact include:
• What endpoints or services appear to be differentially affected (i.e., at greater or less
risk of experiencing effects)? What elements of the ecosystem (e.g., types, regions,
taxonomic groups, populations, and functions) appear to be affected, or are more
sensitive to effects?
• What is concluded from the evidence regarding other types of welfare effects?
• Under what exposure conditions (e.g., amount deposited or concentration, duration,
and pattern) are effects seen?
• What is the shape of the concentration-, exposure-, or dose-response relationship?
• Are there geographical differences in welfare effects responses?
To address these questions, the quantitative evidence is evaluated to characterize
pollutant concentrations and exposure durations at which effects were observed.
Evidence is considered from multiple and diverse types of studies, and a study or set of
studies that best approximates the concentration-, exposure-, or dose-response
relationships between welfare endpoints and the pollutant may be identified. Controlled
experimental studies provide the most direct and quantifiable concentration-, exposure-
, or dose-response data on the effects of pollutant exposures. To the extent available,
the ISAs also evaluate results from less controlled field studies that characterize the
shape of the relationship between a pollutant and an endpoint. Other types of data may
also inform evaluation of concentration-, exposure-, or dose-response relationships,
particularly relative to modes of action and characteristics of ecosystems that may make
them at greater or less risk of experiencing effects.
A.7.2.3.1. Evaluating Adversity of Ecological and Other
Welfare Effects
The final step in assessing the public welfare impact of an air pollutant is the
evaluation of the level considered to be adverse. A secondary standard, as provided in
Section 109(b)(2) of the CAA, must "specify a level of air quality the attainment and
A-70
-------
maintenance of which, in the judgment of the Administrator, based on such criteria, is
requisite to protect the public welfare from any known or anticipated adverse effects
associated with the presence of such air pollutant in the ambient air." In setting
standards that are "requisite" to protect public welfare, as provided in Section 109(b),
the EPA's task is to establish standards that are neither more nor less stringent than
necessary for these purposes.
The adversity of ecological effects can be characterized at any biological level of
organization, ranging from the cellular level to the individual organism, and to the
population, community, and ecosystem levels. In the context of ecology, a population is
a group of individuals of the same species, and a community is an assemblage of
populations of different species that inhabit an area and interact with one another. An
ecosystem is the interactive system formed from all living organisms and their abiotic
(physical and chemical) environment within a given area (IPCC, 2007). The boundaries of
what could be called an ecosystem are somewhat arbitrary, depending on the focus of
interest or study. Thus, the extent of an ecosystem may range from very small spatial
scales to, ultimately, the entire Earth (IPCC, 2007), and a pollutant may have adverse
effects at any of those scales.
Effects on an individual organism are generally not considered to be adverse to
public welfare, but if effects occur on enough individuals within a population, then
communities and ecosystems may be disrupted. Changes to populations, communities,
and ecosystems can in turn result in an alteration of ecosystem processes. Ecosystem
processes are defined as the metabolic functions of ecosystems, including energy flow,
elemental cycling, and the production, consumption, and decomposition of organic
matter (U.S. EPA, 2002). Growth, reproduction, and mortality are species-level endpoints
that may be clearly linked to community and ecosystem effects and are considered to
be adverse when negatively affected. Other endpoints, such as changes in behavior and
physiological stress, can decrease ecological fitness of an organism but are harder to
link unequivocally to effects at the population, community, and ecosystem level. In
assessing ecological conditions, it has been suggested that adversity be considered
beyond the species level by linking stress-related effects at the species level and effects
at the ecosystem level as discussed in A Framework for Assessing and Reporting on
A-71
-------
Ecobgica[ Condition: an SAB report (U.S. EPA, 2002). Additionally, the National Acid
Precipitation Assessment Program (NAPAP, 1991) uses the following working definition
of "adverse ecological effects" in the preparation of reports to Congress mandated by
the CAA: "any injury (i.e., loss of chemical or physical quality or viability) to any
ecological or ecosystem component, up to and including the regional level, over both
long and short terms."
Beyond species-level impacts, consideration of ecosystem services allows for
evaluation of how pollutant exposure may adversely impact species or processes of
particular economic or cultural importance to humans. Ecosystem services are the
benefits that people obtain from ecosystems (UNEP, 2003). According to the Millennium
Ecosystem Assessment, ecosystem services include "provisioning services such as food
and water; regulating services such as regulation of floods, drought, land degradation,
and disease; supporting services such as soil formation and nutrient cycling; and cultural
services such as recreational, spiritual, religious, and other nonmaterial benefits" (UNEP,
2003). On a broader scale, ecosystem services may provide indicators for ecological
impacts. For example, a more subtle ecological effect of pollution exposure may result in
a clearly adverse impact on ecosystem services if it results in a population decline in a
species that is recreationally or culturally important.
Under the broad definition of ecosystem services from UNEP (2003), the adverse
impacts of climate change and visibility impairment are both closely tied to the complex
role of criteria pollutants in those effects. Criteria air pollutants have both direct and
indirect effects on the radiative forcing of climate, via their direct impact on the
radiation budget and by driving the atmospheric chemical budgets of climate-forcing
pollutants. To illustrate, CO has a relatively small direct radiative forcing, but influences
the concentrations of O3 and methane, both of which are greenhouse gases. PM
constituents, by contrast, have both direct and indirect effects on the climate system.
Black carbon and sulfate have direct impacts on atmospheric warming and surface
cooling, respectively, but also change the properties of clouds and thus have multiple
indirect effects on climate. In addition, interactions among multiple pollutants add to
the complexity of evaluating the climate impact of individual criteria pollutants.
Although the impacts of criteria air pollutants on climate affect terrestrial and aquatic
A-72
-------
environments in diverse ways over multiple time scales, their effect on temperature is
the most evident metric of adverse climate impacts. There are also subsequent longer-
term effects on precipitation and large-scale atmospheric circulations, as well as
relatively rapid feedbacks from the changes in these climate variables that affect the
composition of the troposphere.
Indirect effects on terrestrial ecosystems and any feedbacks of these effects are
more difficult to link back to changes in concentrations of individual pollutants
regulated under the NAAQS. These include the adverse impacts of O3 on vegetation and
its uptake of atmospheric CO2 (i.e., the terrestrial carbon sink) and the resulting
feedback to the climate system. The adverse impacts of U.S. emissions relative to global
emissions and concentrations are informed by regional climate modeling and ensemble
modeling studies, including consideration of uncertainty and spatial and temporal
variability.
Impacts on visibility are among the welfare effects that are potentially relevant for
an air pollutant. The adverse impacts of criteria pollutants on visibility may be expressed
in terms of psychological stress, such as from the impairment of aesthetic quality and
enjoyment of the visual environment, or in monetary terms, such as willingness to pay to
improve visual air quality or impacts on property values. While pollutant concentrations
in ambient air are quantifiable, perceptions of environmental damage and willingness to
remediate them are much more subjective. Understanding the relationship between
pollutant concentrations and perceived visual air quality, including distinguishing
between concerns about health risks and perceived visibility impairment due to air
pollution, is an important consideration for both.
Adversity of materials damage is evaluated by considering the impact to human
and economic well-being. Physical damage and soiling impair the function and aesthetic
qualities of materials. Additionally, damage to property and cultural heritage sites due
to pollutant deposition may be considered as adverse impacts associated with
pollutants in ambient air.
A-73
-------
A.7.3. Develop the Integrated Synthesis
The Integrated Synthesis draws from the detailed assessment of the evidence in
the ISA chapters. It provides a concise synopsis of the ISA conclusions and synthesis of
key information and findings considered in characterizing pollutant exposures and
relationships with health or welfare effects. The Integrated Synthesis includes summaries
of policy-re levant information for each topic area covered in the ISA chapters, including
atmospheric science, sources, and environmental distribution; exposure, biomarkers, and
toxicokinetics; the nature of health or welfare effects associated with pollutant exposure,
including causality determinations; and the human populations and lifestages at
increased risk of the effects of pollutant exposure. The Integrated Synthesis also
summarizes the evidence and conclusions for other policy-re levant issues. These vary
across ISAs and can include the exposure durations, metrics, and concentrations eliciting
health or welfare effects; the shapes and statistical precision of concentration-,
exposure-, or dose-response functions; and the potential adversity and public health or
welfare significance of certain adverse effects.
A.8. PEER REVIEW AND PUBLIC COMMENT
Section 109(d)(2) of the CAA addresses the appointment and advisory functions
of an independent scientific review committee. Among other requirements, section
109(d)(2) directs the Administrator to appoint this committee, which is to be composed
of "seven members including at least one member of the National Academy of Sciences,
one physician, and one person representing state air pollution control agencies." Section
109(d)(2) additionally provides that the independent scientific review committee "shall
complete a review of the criteria...and the national primary and secondary ambient air
quality standards...and shall recommend to the Administrator any new...standards and
revisions of existing criteria and standards as may be appropriate...." Since the early
1980s, this independent review function has been performed by the CASAC of the EPA's
Science Advisory Board.
In reviewing the draft ISA, the CASAC is assisted by an ad hoc panel of
independent subject matter experts. Ad hoc panels for each review consist of members
of the CASAC supplemented by additional independent experts in the subject matter for
A-74
-------
that review. CASAC panels are convened to provide broad expertise related to the
particular criteria pollutant under evaluation and the science-policy issues important for
the review. Given the breadth of scientific and technical information evaluated during
NAAQS reviews, CASAC panels reflect a wide range of expertise. The specific expertise
varies across panels, but typically includes expert knowledge of atmospheric science,
climate science, human exposure, dosimetry, toxicology, epidemiology, medicine, public health,
biostatistics, ecological sciences, and risk assessment. Consistent with NASEM
recommendations (NASEM, 2022, p. 8), critical disciplines are often represented by
multiple panel members to facilitate advice from a range of perspectives. Ad hoc CASAC
panels are typically chaired by an experienced CASAC member familiar with the process.
The CASAC panels convene at public meetings that are announced in the Federal
Register and that provide an opportunity for public comment. Draft advisory reports,
conveying recommendations to the EPA on the draft ISA, are prepared by the ad hoc
panels and are transmitted to the chartered CASAC for discussion and deliberation.
These reports convey advice to the EPA regarding the ISA conclusions (e.g., causality
determinations, at-risk populations, exposure-response relationships) and the ISA's
approaches to evaluating, weighing, and integrating evidence to reach those
conclusions, and they often identify additional studies that the CASAC believes should
be included in the ISA. If the chartered CASAC determines the contents of a report are
appropriate, the CASAC will adopt the report and transmit it to the EPA to reflect its
statutorily mandated advice to the Agency.
The EPA carefully considers advice received from the CASAC and comments from
the public in revising and updating the draft ISA. This may include consideration of
additional studies identified during peer review that meet the ISA's scoping and study
quality criteria. After appropriate revisions are made, the final document is made
available on the EPA website. A notice announcing the availability of the final ISA is
published in the Federal Register. More information on EPA's peer review practices can
be found in EPA's Peer Review Handbook and via the EPA CASAC peer review website
(https://casac.epa.g0v/0rds/sab/r/sab apex/casac/home).
A-75
-------
A.9. QUALITY MANAGEMENT
Quality management helps to ensure the scientific information used by the EPA
to inform policymakers, the public, and other stakeholders is high-quality. Agency-wide,
the EPA Quality Program provides the framework for planning, implementing,
documenting, and assessing work performed by the Agency, and for carrying out
required quality assurance and quality control (QA/QC) activities. Additionally, the
Quality Program covers the implementation of the EPA Information Quality Guidelines
and the EPA Environmental Information Quality Policy and Procedures (U.S. EPA, 2002).
The agency-wide quality assurance (QA) policy outlined in the EPA Environmental
Information Quality Policy (see CIO 2105.4) supported by the EPA Environmental
Information Quality Procedures (CIO 2105-P-01.4). As required by CIO 2105.1, the EPA
Office of Research and Development (ORD) maintains a Quality Management Program,
which is documented in an internal Quality Management Plan (QMP). All environmental
information operations (EIO), including the ISAs, are subject to the EPA's Quality
Management Program requirements for a Quality Management Plan (QMP) and a
Quality Assurance Project Plan (QAPP). For the ISA Program specifically, management of
quality assurance is documented in a Programmatic Quality Assurance Project Plan
(PQAPP) [Integrated Science Assessment Programmatic QAPP, L-HEEAD-0030253-QP-1-
7], which describes the technical approach and QA/QC procedures associated with the
ISA Program.
Adherence to the ORD QMP and the ISA PQAPP ensures that all data generated,
collected, evaluated, or used in an ISA are "of the type and quality needed and expected
for their intended use" and that all information disseminated by the ISAs adheres to a
high standard for quality, including objectivity, utility, and integrity. Furthermore, the
EPA's HERO database has its own QC processes, as documented in HERO's Quality
Assurance Project Plan (QAPP), L-HEEAD-0032852-QP-1-5. The EPA CPHEA QA
managers (QAMs) are responsible for the review and approval of quality-related
documentation. The CPHEA ISA scientists are responsible for the evaluation of all inputs
to the ISAs, including primary (new) and secondary (existing) data from others, to ensure
their quality is appropriate for use in the ISAs. CPHEA adheres to Data Quality
Objectives, which identify the most appropriate inputs to the science assessment, and
A-76
-------
CPHEA provides QA instruction and training to researchers involved with environmental
information operations.
The approaches utilized to search the literature and to select and evaluate studies
were detailed in the preceding subsections. Generally, ISA scientists rely on scientific
information found in peer-reviewed journal articles, books, and government reports. The
ISAs can integrate information extracted from multiple sources to create new figures,
tables, or summations, which are subject to rigorous quality assurance measures to
ensure their accuracy. Documentation of the quality of extracted information includes
the types of QA/QC checks performed and the approach to verifying information
extractions (e.g., verification by a second individual). The QA/QC checks for extracted
information include comparing of entries to information from the original publication,
checking conversions (e.g., ppm to |jg/m3), confirming effect levels, and inserting and
verifying electronic citations that are converted to HERO links. In addition, QA reviews of
key information from all types of health effect studies are performed. Furthermore,
publicly available databases (e.g., HERO) have their own QA processes that are outlined
in the QAPP.
The ISAs are designated as Highly Influential Scientific Assessments (HISA) and
classified as ORD QA Category A, (high profile research). Category A designations
require reporting of all critical QA activities, including audits. During assessment
development, the ISAs undergo periodic quality audits. A Technical Systems Audit (TSA)
of each ISA is conducted by the EPA or an independent contractor to verify that all
QA/QC procedures were adequately performed and documented. The ISAs are
subjected to management and QA clearance review, and during this step, the CPHEA QA
Manager verifies that the EPA QA requirements are met.
The EPA is committed to providing public access to environmental information.
The EPA's Information Quality Guidelines for Ensuring and Maximizing the Quality,
Objectivity, Utility, and Integrity of Information Disseminated by the Environmental
Protection Agency reflects the EPA's commitment to the quality of the information the
Agency disseminates.
A-77
-------
A.10. NASEM CONCLUSIONS, RECOMMENDATIONS, AND
CONSIDERATION OF FUTURE ASSESSMENT DEVELOPMENTS
The NASEM, sponsored by the EPA, convened an ad hoc committee to assess the
ISA causality framework, as described in the 2015 Preamble to the ISAs (U.S. EPA, 2015).
The committee reached seven conclusions, three of which endorsed fundamental
aspects of the current ISA causality framework: 1) A weight-of-evidence approach
appropriately allows the EPA to integrate across study designs and disciplines (NASEM,
2022, p. 111); 2) The five categories in the current framework can appropriately
characterize causality in ISAs (NASEM, 2022, p. 112); and 3) The ISA causality framework
can adequately guide determinations for both health and welfare (NASEM, 2022, p. 112).
The additional four conclusions are accompanied by recommendations for
improving the framework. Recommendations address heterogeneity in exposure
responses, study quality evaluation, transparency, and needed expertise. Below, these
NASESM conclusions and recommendations are summarized, and the sections of this
appendix that address those recommendations are identified.
A.10.1. Heterogeneity in Exposure Responses
The committee concluded that"[t]he current framework separates description of
vulnerable groups and sensitive ecosystems or species from causal determinations,
potentially obscuring understanding of causal relationships for the more exposure-
sensitive groups of subjects when the causal category determinations are presented at a
broader level." The committee further recommended describing how heterogeneity in
exposure responses is considered in the framework to ensure causal determinations fully
account for evidence of effects in sensitive groups. (Ref: pages 3-4 and 113, NASEM
Report). This recommendation is addressed in Sections A.2.1, A.7.2.1.2, and A.7.2.2. Text
in these sections clarifies the practice of considering evidence in populations at higher
risk as an inherent part of causality determinations.
A. 10.2. Study Quality Evaluation
The committee concluded that "The causal determination framework described in
the Preamble provides general guidance for individual study quality evaluation, but
minimal detail regarding determination of individual study relevance, study inclusion or
A-78
-------
exclusion, or influence on weight of evidence causal determinations." The committee
further recommended 1) including a set of foundational study design attributes to be
considered when selecting and evaluating studies, and 2) providing explicit guidance for
assessing the approaches used to account for confounders. (Ref: pages 4-5 and 114-
116, NASEM Report)
Substantial detail on study quality evaluation is included in Section A.5, including
foundational study design attributes that are considered and how those attributes can
affect the strength of inference drawn from particular studies. Sections A.5.2.6, A.5.3.6,
and A.5.4.6 specifically address how potential confounding is considered as part of study
quality evaluation.
A. 10.3. Transparency
The committee concluded that "[T]he Preamble's causal determination framework
does not provide explicit guidance regarding how the potential reproducibility and
replicability of individual studies should affect the influence of those studies on causal
assessments." The committee further recommended developing guidance for assessing
individual study documentation of data, methods, and assumptions and for how the use
of that assessment informs the influence of the individual study in the weight-of-
evidence approach. (NASEM, 2022, p. 5-6 and 112).
Substantial detail on how ISAs consider documentation of data, methods and
assumptions, and replicability/reproducibility is included throughout Section A.5. In
addition, Section A.7.1.1.1 discusses the role of reproducibility (i.e., consistent results
across studies using various approaches and data) specifically in causality
determinations.
A. 10.4. Expertise
The committee concluded that "Given the broad range of topics covered when
determining causality for criteria pollutants, access to a broad range of expertise within
and outside EPA...is needed throughout the causal determination process to ensure
incorporation of the latest scientific knowledge." The committee further recommended
articulating a clear process for identifying and incorporating the necessary expertise for
ISA development (NASEM, 2022, p. 7 and 118). Sections A.2.2 and A.8 document the
A-79
-------
range of expertise typically required in developing an ISA. This includes expertise from
EPA staff, contractors, peer input panels, the CASAC, and pollutant-specific CASAC
panels.
The committee also recommended monitoring research in the scientific literature
on evidence integration and the evolution of other frameworks used to assess causality
(NASEM, 2022, p. 117). Section A.2.2 notes that EPA staff routinely monitor
advancements in scientific fields related to evidence integration and weight-of-evidence
evaluations, as well as other relevant fields that could improve the ISAs.
A-80
-------
A.11. REFERENCES
Allwood, J. M., Bosetti, V., Dubash, N. K., Gomez-Echeverri, L., & von Stechow, C. (2014).
In Climate Change 2014: Mitigation of Climate Change. Contribution of Working
Group III to the Fifth Assessment Report of the Intergovernmental Panel on Climate
Change, 1249-1279.
Bang, H and Robins, JM (2005). Doubly robust estimation in missing data and causal
inference models. Biometrics 61(4): 962-972.
Barr, RG, Herbstman, J, Speizer, FE and Camargo, CA, Jr. (2002). Validation of self-
reported chronic obstructive pulmonary disease in a cohort study of nurses. Am J
Epidemiol 155(10): 965-971.
Chen W, Qian L, Shi J, Franklin M. Comparing performance between log-binomial and
robust Poisson regression models for estimating risk ratios under model
misspecification. BMC Med Res Methodol. 2018 Jun 22;18(1 ):63.
CDC (2004). National Center for Chronic Disease Prevention and Health Promotion (US)
Office on Smoking and Health. The Health Consequences of Smoking—50 Years
of Progress: A Report of the Surgeon General. U.S. Department of Health and
Human Services. Centers for Disease Control and Prevention Atlanta (GA).
Available at: https://pybmed.ncbi.nlm.nih.gov/24455788/.
Cronin, MTD, Bajot, F, Enoch, SJ, Madden, JC, Roberts, DW and Schwobel, J (2009). The In
Chemico-ln Silico Interface: Challenges for Integrating Experimental and
Computational Chemistry to Identify Toxicity. Alternatives to Laboratory Animals
37(5): 513-521.
Diaz, I (2019). Machine learning in the estimation of causal effects: Targeted minimum
loss-based estimation and double/debiased machine learning. Biostatistics 21(2):
353-358.
Diaz, I, Williams, N, Hoffman, KL and Schenck, EJ (2023). Nonparametric causal effects
based on longitudinal modified treatment policies. Journal of the American
Statistical Association 118(542): 846-857.
ECHA (2016). New approach methodologies in regulatory science: Proceedings of a
scientific workshop, Helsinki, 19-20 April 2016. Helsinki, Finland. Available at:
http://dx.doi.org/10.2823/543644.
A-81
-------
Ertefaie, A and Stephens, DA (2010). Comparing approaches to causal inference for
longitudinal data: Inverse probability weighting versus propensity scores. The
International Journal of Biostatistics 6(2): 14.
European Commission, Joint Research Centre, Adcock, I, Novotny, T, Nic, M, Dibusz, K,
Hynes, J, Marshall, L and Gribaldo, L (2020). Advanced non-animal models in
biomedical research - Respiratory tract diseases. Publications Office of the
European Union. Available at: https://data.eyropa.eu/doi/102760/725821.
Gee, GC and Payne-Sturges, DC (2004). Environmental health disparities: A framework
integrating psychosocial and environmental concepts. Environ Health Perspect
112(17): 1645-1653.
Gelman, A and Greenland, S (2019). Are confidence intervals better termed "uncertainty
intervals"? BMJ (Clinical Research Edition) 366:15381.
Goldman, GT, Mulholland, JA, Russell, AG, Strickland, MJ, Klein, M, Waller, LA and
Tolbert, PE (2011). Impact of exposure measurement error in air pollution
epidemiology: Effect of error type in time-series studies. Environ Health 10: 61.
Greenland, S (2000). An introduction to instrumental variables for epidemiologists. Int J
Epidemiol 29: 722-729.
Greenland, S, Senn, SJ, Rothman, KJ, Carlin, JB, Poole, C, Goodman, SN and Altman, DG
(2016). Statistical tests, P values, confidence intervals, and power: a guide to
misinterpretations. Eur J Epidemiol 31(4): 337-350.
Greenstone, M., & Gayer, T. (2009). Quasi-experimental and experimental approaches to
environmental economics. J. Envir Econ Manag, 57(1), 21-44.
Hamel, C, Kelly, SE, Thavorn, K, Rice, DB, Wells, GA and Hutton, B (2020). An evaluation
of DistillerSR's machine learning-based prioritization tool for title/abstract
screening - impact on reviewer-relevant outcomes. BMC Med Res Methodology
20(1): 256.
Hernan, MA and Robins, JM (2016). Using big data to emulate a target trial when a
randomized trial is not available. Am J Epidemiol 183(8): 758-764.
Hew (1964). Smoking and health: Report of the advisory committee to the surgeon
general of the public health service. U.S. Department of Health, Education, and
Welfare. Washington, DC. Available at:
http://profiles.nlm.nih.gov/ps/retrieve/ResourceMetadata/NNBBMQ.
A-82
-------
Hill, AB (1965). The environment and disease: Association or causation? Proceedings of
the Royal Society of Medicine 58(5): 295-300.
Howard, BE, Phillips, J, Tandon, A, Maharana, A, Elmore, R, Mav, D, Sedykh, A, Thayer, K,
Merrick, BA, Walker, V, Rooney, A and Shah, RR (2020). SWIFT-Active Screener:
Accelerated document screening through active learning and integrated recall
estimation. Environ Int 138: 105623.
IARC (2006). IARC monographs on the evaluation of carcinogenic risks to humans:
Preamble. World Health Organization. Lyon, France. Available at:
http://monographs.iarc.fr/ENG/Preamble/.
IOM, Ed. (2008). Improving the presumptive disability decision-making process for
veterans. National Academies Press Washington, DC.
IPCC, 2007: Climate Change 2007: The Physical Science Basis. Contribution of Working
Group I to the Fourth Assessment Report of the Intergovernmental Panel on
Climate Change [Solomon, S., D. Qin, M. Manning, Z. Chen, M. Marquis, K.B.
Averyt, M. Tignor and H.L. Miller (eds.)]. Cambridge University Press, Cambridge,
United Kingdom and New York, NY, USA, 996 pp.
Keil, AP, Buckley, JP, Katie, MO, Ferguson, KK, Zhao, S and White, AJ (2020). A quantile-
based g-computation approach to addressing the effects of exposure mixtures.
Environ Health Perspect 128(4): 47004.
Lash, TL, Fox, MP, MacLehose, RF, Maldonado, G, McCandless, LC and Greenland, S
(2014). Good practices for quantitative bias analysis. Int J Epidemiol 43(6): 1969-
1985.
Muhajarine, N, Mustard, C, Roos, LL, Young, TK and Gelskey, DE (1997). Comparison of
survey and physician claims data for detecting hypertension. J Clin Epidemiol
50(6): 711-718.
Murgia, N, Brisman, J, Claesson, A, Muzi, G, Olin, AC and Toren, K (2014). Validity of a
questionnaire-based diagnosis of chronic obstructive pulmonary disease in a
general population-based study. BMC Pulmonary Medicine 14: 49.
NAPAP (1991) The U.S. National Acid Precipitation Assessment Program: 1990
Integrated Assessment Report. NAPAP Office of the Director. Washington, DC.
NASEM (2017). National Academies of Sciences, Engineering, and Medicine 2017.
Controiied Human Inhaiation-Exposure Studies at EPA. National Academies Press
(US). Washington (DC). Available at: http://dx.doi.org/10.17226/24618.
A-83
-------
NASEM (2022). National Academies of Sciences, Engineering, and Medicine 2022.
Advancing the Framework for Assessing Causality of Health and Welfare Effects
to Inform National Ambient Air Quality Standard Reviews. The National
Academies Press. Washington, DC. Available at: https://doi.o 26612.
Pearce, Neil; Vandenbroucke, Jan P.; Lawlor, Deborah A. Causal Inference in
Environmental Epidemiology: Old and New Approaches. Epidemiology 30(3):p
311-316, May 2019.
Richardson, A, Hudgens, MG, Gilbert, PB and Fine, JP (2014). Nonparametric bounds and
sensitivity analysis of treatment effects. Statistical Science 29(4): 596-618.
Richmond-Bryant and Long. Influence of exposure measurement errors on results from
epidemiologic studies of different designs. Journal of Exposure Science &
Environmental Epidemiology, 2020, 30(3), 420-429.
Robins, JM, Hernan, MA and Brumback, B (2000). Marginal structural models and causal
inference in epidemiology. Epidemiology 11(5): 550-560.
Rothman, KJ and Greenland, S (1998). Modern Epidemiology. Lippincott-Raven.
Philadelphia, PA.
Sacks, JD, Stanek, LW, Luben, TJ, Johns, DO, Buckley, BJ, Brown, JS and Ross, M (2011).
Particulate-matter induced health effects: Who is susceptible? Environ Health
Perspect 119(4): 446-454.
Savitz, DA and Forastiere, F (2021). Do pooled estimates from meta-analyses of
observational epidemiology studies contribute to causal inference? Occup
Environ Med 78(9): 621-622.
Savitz, DA, Wellenius, GA and Trikalinos, TA (2019). The Problem with Mechanistic Risk of
Bias Assessments in Evidence Synthesis of Observational Studies and a Practical
Alternative: Assessing the Impact of Specific Sources of Potential Bias. Am J
Epidemiol 188(9): 1581-1585.
Sheppard, L, Slaughter, JC, Schildcrout, J, Liu, JS and Lumley, T (2005). Exposure and
measurement contributions to estimates of acute air pollution effects. J Expo Anal
Environ Epidemiol 15(4): 366-376.
Stitelman, OM, De Gruttola, V and van der Laan, MJ (2012). A general implementation of
TMLE for longitudinal data applied to causal inference in survival analysis. The
International Journal of Biostatistics 8(1): 26.
A-84
-------
Stuart, EA (2010). Matching methods for causal inference: A review and a look forward.
Statistical Science 25(1): 1-21.
Talbot D, Mesidor M, Chiu Y, Simard M, Sirois C. An Alternative Perspective on the
Robust Poisson Method for Estimating Risk or Prevalence Ratios. Epidemiology.
2023 Jan 1;34(1):1-7.
Thayer, KA, Angrish, M, Arzuaga, X, Carlson, LM, Davis, A, Dishaw, L, Druwe, I, Gibbons, C,
Glenn, B, Jones, R, Kaiser, JP, Keshava, C, Keshava, N, Kraft, A, Lizarraga, L, Persad,
A, Radke, EG, Rice, G, Schulz, B and Vetter, N (2022). Systematic Evidence Map
(SEM) template: Report format and methods used for the US EPA Integrated Risk
Information System (IRIS) program, Provisional Peer-Reviewed Toxicity Value
(PPRTV) program, and other "fit for purpose" literature-based human health
analyses. Environ Int 169(Elsevier): 107468.
Toren, K, Brisman, J and Jarvholm, B (1993). Asthma and asthma-like symptoms in adults
assessed by questionnaires: A literature review. Chest 104(2): 600-608.
UNEP (United Nations Environment Programme). (2003). Ecosystems and human well-
being: A framework for assessment. Washington, DC: Island Press.
U.S. EPA (1986). Air quality criteria for lead: Volume I of IV [EPA Report], Research
Triangle Park, NC. EPA-600/8-83/028aF Available at:
http: 3a.gov/ncea/cfm/recQrdisplay.cfm7deid=32647.
U.S. EPA (1991). Guidelines for Developmental Toxicity Risk Assessment. U.S.
Environmental Protection Agency. Available at:
https://nepis.epa.gov/exe/ZvPU RLcgi?Dockev=3QQQ2L7L.txt.
U.S. EPA (1996). Guidelines for Reproductive Toxicity Risk Assessment. U.S.
Environmental Protection Agency, Risk Assessment Forum. Washington, DC.
EPA/630/R-96/009. Available at:
https://nepis.epa.gov/Exe/ZvPU RL.cgi?Dockev=30004YQB.txt.
U.S. EPA (1998). Guidelines for neurotoxicity risk assessment. U.S. Environmental
Protection Agency, Risk Assessment Forum. Washington, DC. EPA/630/R-95/001F.
Available at: http://www.epa.gov/risk/gyidelines-neurotoxicitv-risk-assessment.
U.S. EPA (2002). A framework for assessing and reporting on ecological condition. EPA
Science Advisory Board. Washington, DC. EPA-SAB-EPEC-02-009A. available at:
https://nepis.epa.aov/Exe/ZvPDF.cai/P100JQSY.PDF7Dockev 1QSY.PDF.
A-85
-------
U.S. EPA (2005). Guidelines for carcinogen risk assessment. Risk Assessment Forum.
Washington, DC. EPA630P03001F. Available at: https://www.epa.gov/
sites/production/files/201309/documents/cancer guidelines final 3-25-05.pdf.
U.S. EPA (2009). Integrated Science Assessment for Particulate Matter (Final Report).
Office of Research and Development, National Center for Environmental
Assessment. Research Triangle Park, NC. U.S. EPA. EPA-600/R-08-139F. December
2009. Available at: https: <> \\ nl_, pa.gov/ncea/risk/recQrdisplay.',
deid=216S46.
U.S. EPA (2010). Integrated Science assessment (ISA) for Carbon Monoxide (Final report).
U.S. Environmental Protection Agency. Washington, DC. EPA/600/R-09/019F.
Available at: http://cfpub.epa.gov/nceM mih t^cordispi \ »'eid = 218686.
U.S. EPA (2013a). Integrated Science Assessment (ISA) for Lead (Final Report). Office of
Research and Development, National Center for Environmental Assessment.
Washington, DC. U.S. EPA. EPA/600/R-10/075F. July 2013. Available at:
https://assessments.epa.gov/isa/document/&d(
U.S. EPA (2013b). Integrated Science Assessment of Ozone and Related Photochemical
Oxidants (Final Report). Office of Research and Development, National Center for
Environmental Assessment. Research Triangle Park, NC. U.S. EPA. EPA-600/R-10-
076F. February 2013. Available at: https://nepis.epa.gov/Exe/ZvPURL.cgi7Dockev
ETF.txt.
U.S. EPA (2015). Preamble to the Integrated Science Assessments (ISA). Office of
Research and Development, National Center for Environmental Assessment.
Washington, DC. U.S. EPA. EPA/600/R-15/067. Available at:
https://ordspyb.epa.gov/ords/eims/eimscomm.getfile7p download id = 526136.
U.S. EPA (2016). Integrated Science Assessment for Oxides of Nitrogen-Health Criteria.
U.S. Environmental Protection Agency, Office of Research and Development,
National Center for Environmental Assessment. Research Triangle Park, NC.
EPA/600/R-15/068. Available at:
http; ¦¦ u ni 3pa.gov/ncea/isa/recordisplavmh> Jeid=310879.
U.S. EPA (2018). Strategic plan to promote the development and implementation of
alternative test methods within the TSCA program. Office of Chemical Safety and
Pollution Prevention. Washington D.C. EPA-740-R1-8004. Available at:
https://www.epa.gov/sites/production/files/2018-
06/documents/epa alt strat plan 6-20-18 clean final.pdf.
A-86
-------
U.S. EPA (2019a). Integrated Science Assessment (ISA) for Particulate Matter (Final
Report). U.S. Environmental Protection Agency, Office of Research and
Development, National Center for Environmental Assessment. Washington, DC.
U.S. EPA. EPA/600/R-19/188. December 2019. Available at:
https://www.epa.gov/naaqs/particulate-matter-pm-standards-integrated-
science-assessments-current-review.
U.S. EPA (2019b). Guidelines for Human Exposure Assessment. Risk Assessment Forum.
Washington, DC. EPA/100/B-19/001. Available at: https://www.epa.gov/sites/
production/files/2020-01/documents/guidelines for hurr posure
assessment final2019.pdf.
U.S. EPA (2020a). Integrated Science Assessment for Oxides of Nitrogen, Oxides of
Sulfur, and Particulate Matter—Ecological Criteria (Final). U.S. Environmental
Protection Agency, Office of Research and Development, Center for Public Health
and Environmental Assessment. Research Triangle Park, NC. EPA/600/R-20/278.
Available at: https u ub.epa.gov/ncea/isa/recordispj \ - 'aid=349473.
U.S. EPA (2020b). Integrated Science Assessment (ISA) for Ozone and Related
Photochemical Oxidants (Final). U.S. Environmental Protection Agency, Office of
Research and Development. Washington, DC. EPA/600/R-20/012. Available at:
https://ordspyb.epa.gov/ords/eims/eimscomm.getfile7p download id = 540022.
U.S. EPA (2021a). Health Assessment Workspace Collaborative (HAWC) Washington, DC.
U.S. EPA (2021b). EPA new approach methods work plan: reducing use of vertebrate
animals in chemical testing. Washington D.C. Available at:
https://www.epa.gov/chemical-research/epa-new-approach-methods-work-plan-
reducing-use-vertebrate-animals-chemical.
U.S. EPA (2022). ORD staff handbook for developing IRIS assessments. U.S.
Environmental Protection Agency, Office of Research and Development, Center
for Public Health and Environmental Assessment. Washington, DC. EPA 600/R-
22/268. Available at: https://cfpub.epa.gov/ncea/iris drafts/recordisplav.cfm?deid
=356370.
U.S. EPA (2023). Living Literature Review [Computer Software], RTP, NC.
https://hero.epa.gov/hero/index.cfm/reference/details/reference ic
U.S. EPA (2024). Integrated Science Assessment (ISA) for Lead (Final Report). Office of
Research and Development, Center for Public Health and Environmental
Assessment. Washington, DC. U.S. EPA. EPA/600/R-23/375. January 2024.
A-87
-------
Available at:
https://ordspyb.epa.gov/ords/eims/eimscomm.getfile | ^ wriload id = 548355.
VanderWeele, T. J., & Ding, P. (2017). Sensitivity analysis in observational research:
introducing the E-value. Ann Intern Med, 167(4), 268-274.
Vinikoor-lmler, L. C., Owens, E. O., Nichols, J. L., Ross, M., Brown, J. S., & Sacks, J. D.
(2014). Evaluating potential response-modifying factors for associations between
ozone and health outcomes: a weight-of-evidence approach. Environ Health
Perspect, 122(11), 1166-1176.
Vinikoor-lmler, LC, Owens, EO, Nichols, JL, Ross, M, Brown, JS and Sacks, JD (2014).
Evaluating potential response-modifying factors for associations between ozone
and health outcomes: A weight-of-evidence approach. Environ Health Perspect,
122(11): 1166-1176.
Wasserstein, RL and Lazar, NA (2016). The ASA's statement on p-values: Context,
process, and purpose. American Statistician 70(2): 129-133.
Wasserstein, RL, Schirm, AL and Lazar, NA (2019). Moving to a world beyond "p<0.05".
American Statistician 73(sup1): 1-19.
Weakley, J, Webber, MP, Ye, F, Zeig-Owens, R, Cohen, HW, Hall, CB, Kelly, K and Prezant,
DJ (2013). Agreement between obstructive airways disease diagnoses from self-
report questionnaires and medical records. Preventive Medicine 57(1): 38-42.
Weuve, J, Sagiv, SK and Fox, MP (2018). Quantitative bias analysis for collaborative
science. Epidemiology 29(5): 627-630.
WHO (2008). Uncertainty and Data Quality in Exposure Assessment. The International
Programme on Chemical Safety. Geneva. WHO. Available at: https://apps.who.int/
iris/handle/10665/44017.
WHO (2021). World Health Organization giobai air quality guidelines: particulate matter
(PM2.5 and PMw), ozone, nitrogen dioxide, sulfur dioxide and carbon monoxide.
World Health Organization. Geneva. Available at:
https://apps.who.int/iris/handle/10665/345329.
Wing, C, Simon, Kand Bello-Gomez, RA (2018). Designing difference in difference
studies: Best practices for public health policy research. Annu Rev Public Health
39(1): 453-469.
A-88
-------
Yoo EH, Pu Q, Eum Y, Jiang X. The Impact of Individual Mobility on Long-Term Exposure
to Ambient PM2.5: Assessing Effect Modification by Travel Patterns and Spatial
Variability of PM2.5. Int J Environ Res Public Health. 2021 Feb 23;18(4):2194.
Zeger, SL, Thomas, D, Dominici, F, Samet, JM, Schwartz, J, Dockery, D and Cohen, A
(2000). Exposure measurement error in time-series studies of air pollution:
Concepts and consequences. Environ Health Perspect 108(5): 419-426.
Zou G. A modified poisson regression approach to prospective studies with binary data.
Am J Epidemiol. 2004 Apr 1;159(7):702-6.
Zou GY, Donner A. Extension of the modified Poisson regression model to prospective
studies with correlated binary data. Stat Methods Med Res. 2013 Dec;22(6):661-
70.
A-89
-------
Development Team for Appendix of Volume 2 the Integrated Review
Plan
Executive Direction
Dr. Steven J. Dutton (Division Director)—Center for Public Health and
Environmental Assessment, Office of Research and Development, U.S.
Environmental Protection Agency, Research Triangle Park, NC
Dr. Scott Jenkins (Branch Chief)—Center for Public Health and Environmental
Assessment, Office of Research and Development, U.S. Environmental Protection
Agency, Research Triangle Park, NC
Dr. Chris Weaver (Division Director)—Center for Public Health and Environmental
Assessment, Office of Research and Development, U.S. Environmental Protection
Agency, Research Triangle Park, NC
Dr. Liz Naess (Branch Chief)—Center for Public Health and Environmental
Assessment, Office of Research and Development, U.S. Environmental Protection
Agency, Research Triangle Park, NC
AUTHORS, CONTRIBUTORS, AND REVIEWERS
Authors
Dr. David M. Lehmann—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Qingyu Meng—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Parker Duffney—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Jeffrey Herrick—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
A-90
-------
Dr. Kristopher Novak—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Meredith Lassiter—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Jean-Jacques Dubois—Center for Public Health and Environmental
Assessment, Office of Research and Development, U.S. Environmental Protection
Agency, Research Triangle Park, NC
Dr. Uma Shankar—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Scott Jenkins—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Dr. Alison Krajewski—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Ms. Christine Alvarez—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Ms. Amanda Haddock—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Anne Barkley—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Mr. Evan Coffman—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Dr. Peter Byrley—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
A-91
-------
Contributors
Dr. James Brown—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Ms. LaShonda Bunch—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Kirstin Hester—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Dr. Stephen McDow—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Ms. Olivia Birkel—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, DC.
Reviewers
Dr. Britta Bierwagen—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency, DC.
NC
Dr. Wayne Cascio—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Dr. Tara Greaver—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Dr. Kate Schofield—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency, DC
Dr. Mary Hutson—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Ms. Leigh Meyer—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, DC
Dr. Samantha Jones—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
A-92
-------
Dr. Jennifer Nichols—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Deirdre Murphy—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Dr. Erika Sasser—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Ms. Karen Wesson—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Dr. Steven Dutton—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Dr. Scott Jenkins (Branch Chief)—Center for Public Health and Environmental
Assessment, Office of Research and Development, U.S. Environmental Protection
Agency, Research Triangle Park, NC
Dr. Chris Weaver (Division Director)—Center for Public Health and Environmental
Assessment, Office of Research and Development, U.S. Environmental Protection
Agency, Research Triangle Park, NC
Dr. Liz Naess (Branch Chief)—Center for Public Health and Environmental
Assessment, Office of Research and Development, U.S. Environmental Protection
Agency, Research Triangle Park, NC
A-93
-------
United States Office of Air Quality Planning and Standards EPA-452/R-24-001 b
Environmental Protection Health and Environmental Impacts Division March 2024
Agency Research Triangle Park, NC
------- |