#• \
\ d?
PRO*^
Integrated Review Plan for the Primary National
Ambient Air Quality Standards for Oxides of
Nitrogen.
Volume 2: Planning for the Review and the
Integrated Science Assessment
-------
-------
EP A-452/R-24-01 Ob
March 2024
Integrated Review Plan for the Primary
National Ambient Air Quality Standards for
Oxides of Nitrogen.
Volume 2: Planning for the Review and the Integrated
Science Assessment
U.S. Environmental Protection Agency
Office of Air Quality Planning and Standards
Health and Environmental Impacts Division
and
Center for Public Health and Environmental Assessment
Office of Research and Development
Research Triangle Park, NC
-------
DISCLAIMER
This document serves as a public information document and as a management tool for the U.S.
Environmental Protection Agency's (EPA's) Center for Public Health and Environmental
Assessment and the Office of Air Quality Planning and Standards in conducting the review of
the health-based air quality criteria and the primary national ambient air quality standards for
oxides of nitrogen. This document is being circulated to facilitate discussion with the Clean Air
Scientific Advisory Committee and for public comment to inform the EPA's current review of
the health-based air quality criteria and the primary national ambient air quality standards for
oxides of nitrogen. It does not represent and should not be construed to represent an Agency
determination or policy. Mention of trade names or commercial products does not constitute
endorsement or recommendation for use.
-------
TABLE OF CONTENTS
1 INTRODUCTION 1-1
2 POLICY-RELEVANT IS SUES IN THE CURRENT REVIEW 2-1
2.1 Review of the Primary NO2 Standards 2-1
3 DEVELOPMENT OF THE INTEGRATED SCIENCE ASSESSMENT 3-1
3.1 Organization of the ISA in the Current Review 3-1
3.2 Scope of the ISA in the Current Review 3-3
3.2.1 Health Effects 3-4
3.2.2 Atmospheric Science 3-7
3.2.3 Exposure Science & Dosimetry 3-9
3.3 Process for Developing the ISA 3-10
3.3.1 Literature Search 3-11
3.3.2 Identifying Potentially Relevant Studies 3-12
3.3.3 Evaluation of Individual Study Quality 3-15
3.3.4 Integration of Evidence and Determination of Causality 3-16
3.3.5 Quality Management 3-18
3.3.6 CASAC Peer Review 3-19
3.4 Scientific Questions to Guide Evaluation of the Evidence 3-19
3.4.1 Source to Concentration - Air Quality, Atmospheric Science, Fate, and Transport..
3-20
3.4.2 Exposure 3-21
3.4.3 Dosimetry 3-22
3.4.4 Biological Plausibility 3-23
3.4.5 Health Outcomes 3-23
3.4.6 At-Risk Lifestages and Populations 3-28
4 REFERENCES 1
APPENDIX A ISA DEVELOPMENT PROCESS
A. 1. INTRODUCTION A-1
A.2. Overview of ISA Organization and Development A-2
A.2.1. ISA Organization A-2
A.2.2. ISA Development A-3
A 3. DEFINING THE SCOPE 01 THE ISA A-7
i
-------
A.3.1. Health Effects Studies A-7
A.3.2. Atmospheric and Exposure Sciences Studies A-l 1
A.4. APPROACH TO LITERATURE SEARCH AND LITERATURE SCREENING ... A-14
A.4.1. Literature Search A-14
A.4.2. Literature Screening A-15
A.5. EVALUATING INDIVIDUAL STUDY QUALITY A-18
A.5.1. Epidemiology A-20
A.5.2. Controlled Human Exposure A-30
A.5.3. Experimental Animal Studies and Emerging Approaches in Toxicology A-33
A.5.4. Atmospheric, Environmental, and Exposure Science A-37
A.6. EXTRACTING DATA FROM RELEVANT STUDIES A-40
A.7. DRAFTING ISA SECTIONS A-41
A. 7.1. Draft Chapters and Obtain Peer Input A-41
A.7.2. Develop Policy-Relevant Scientific Conclusions A-42
A.7.3. Develop the Integrated Synthesis A-54
A.8. PEER REVIEW OF AND PUBLIC COMMENT ON THE DRAFT ISA A-55
A.9. QUALITY MANAGEMENT A-56
A.10. REFERENCES A-58
ii
-------
PREFACE
The planning phase of the U.S. Environmental Protection Agency's (EPA's) reviews of
the air quality criteria and the national ambient air quality standards (NAAQS) includes
development of an integrated review plan (IRP), which is made available for public comment
and provided to the Clean Air Scientific Advisory Committee (CASAC) for consultation. As a
result of recent efforts to improve the efficiency of the planning phase and to facilitate the receipt
of timely input from the CASAC and the public, the IRP for the current review of the primary
NAAQS for oxides of nitrogen is comprised of three volumes. Volume 1 provides background
information on the health-based air quality criteria and the primary NAAQS for oxides of
nitrogen and may serve as a reference for the public and the CASAC in their consideration of the
subsequent two volumes. Volume 2 (this document) addresses the general approach for the
review and planning for the integrated science assessment (ISA) and will be the subject of a
consultation with the CASAC. This volume identifies policy-relevant issues in the review and
describes key considerations in the EPA's development of the ISA. Volume 3 is the planning
document for quantitative analyses to be considered in the policy assessment (PA), including
exposure and risk analyses as warranted. It will describe key considerations in the EPA's
planning with regard to any quantitative exposure/risk analyses to inform the review. To ensure
that the availability of new evidence is taken into account when developing the current review,
the development and public release of Volume 3 will generally coincide with the availability of
the draft ISA and it will be the subject of a consultation with the CASAC at that time.
in
-------
1 INTRODUCTION
The U.S. Environmental Protection Agency (EPA) is conducting a review of the health-
based air quality criteria and the primary (health-based) national ambient air quality standards
(NAAQS) for oxides of nitrogen. Ambient concentrations of oxides of nitrogen are influenced by
both direct nitrogen dioxide (NO2) emissions and by emissions of nitric oxides (NO), with the
subsequent conversation of NO to NO2 primarily through reaction with ozone (O3). A large
number of oxidized nitrogen species in the atmosphere are formed from the oxidation of NO and
NO2. These include nitrate radicals (NO3), nitrous acid (HONO), nitric acid (HNO3), dinitrogen
pentoxide (N2O5), nitryl chloride (CINO2), peroxynitric acid (HNO4), peroxyacetyl nitrate and its
homologues (PANs), other organic nitrates, such as alkyl nitrates (including isoprene nitrates),
and particulate nitrate (pNOs). The sum of these reactive oxidation products and NO plus NO2
comprise the oxides of nitrogen.1' 2 Consistent with the reviews completed in 2010 and 2018, this
review focuses on health effects associated with gaseous oxides of nitrogen3 and the protection
afforded by the primary NO2 standards. The gaseous oxides of nitrogen include NO2 and NO, as
well as their gaseous reaction products. Total oxides of nitrogen include these gaseous species as
well as particulate species (e.g., nitrates). Health effects and non-ecological welfare effects
associated with the particulate species are addressed in the review of the NAAQS for particulate
matter (PM).4 The EPA is separately reviewing the ecological welfare effects associated with
and the secondary standards for oxides of nitrogen, oxides of sulfur, and PM.5
This Volume (2) of the integrated review plan (IRP) contains the current plans for the
general approach for the review, as well as key planning considerations for the development of
the integrated science assessment (ISA). The NAAQS review process provides an integrative
assessment of relevant scientific information and will focus on key aspects of the NAAQS,
1 The focus is on NO2 in this document, as this is in the indicator for the current standards and is most relevant to the
evaluation of health evidence.
2 Section 108(c) of the Clean Air Act specifies that: "Such criteria [for oxides of nitrogen] shall include a discussion
of nitric and nitrous acids, nitrites, nitrates, nitrosamines, and other carcinogenic and potentially carcinogenic
derivatives of oxides of nitrogen." By contrast, within air pollution research and control communities, the terms
"nitrogen oxides" and NOx are often restricted to refer to only to the sum of NO and NO2.
3 These gaseous oxides of nitrogen can also be referred to as "nitrogen oxides" and include a broad category of
gaseous oxides of nitrogen (i.e., oxidized nitrogen compounds), including NO2, NO, and their various reaction
products.
4 Additional information on the PM NAAQS is available at: https://www.epa.gov/naaqs/particulate-matter-pm-air-
quality-standards.
5 Additional information on the currently ongoing and prior reviews of the secondary NAAQS for oxides of
nitrogen, oxides of sulfur, and PM is available at: https://www.epa.gov/naaqs/nitrogen-dioxide-no2-and-sulfur-
dioxide-so2-secondary-air-quality-standards.
1-1
-------
including the basic elements of the standards: the indicator,6 averaging time, form,7 and level.
These elements, which together serve to define each ambient air quality standard, are considered
collectively in evaluating the protection to public health afforded by the standards.
This document is the second of three volumes that will comprise the IRP for the primary
NO2 NAAQS review. Volume 1 includes introductory or background information on the
legislative requirements for reviews of the NAAQS, an overview of the review process,
background information on prior reviews of the health-based air quality criteria and primary
standards for oxides of nitrogen and a summary of key aspects of the basis for the existing
primary NO2 NAAQS, and a summary of the status and anticipated milestones for the current
review. Volume 1 also includes an appendix that includes an overview of the key aspects of
existing ambient air monitoring requirements for NOx. Volume 2 (this document) presents the
general approach for this review, the policy-relevant questions guiding the review, and the plans
for the development of the ISA. Specifically, Chapter 2 of Volume 2 outlines the general
approach of the NAAQS review and details a set of policy-relevant questions intended to focus
this review on the critical scientific and policy issues. Chapter 3 of Volume 2 presents plans for
the ISA, including the document organization, scope, and specific questions for consideration in
light of the overarching policy-relevant questions for the review. Together, Volumes 1 and 2
provide the current information regarding this review of the primary NAAQS for oxides of
nitrogen. Volume 3 of the IRP, the planning document for quantitative analyses to be considered
in the policy assessment (PA), will be developed with consideration of the availability of new
evidence as identified in the development of the ISA. Accordingly, the public release of Volume
3 of the IRP will generally coincide with that of the draft ISA and it will be the subject of a
consultation with the CASAC at that time.
6 The "indicator" of a standard defines the chemical species or mixture that is to be measured in determining
whether an area attains the standard. For example, the indicator of the current NAAQS for photochemical
oxidants is ozone (O3).
7 The "form" of a standard defines the air quality statistic that is to be compared to the level of the standard in
determining whether an area attains the standard. For example, the form of the annual PM2.5 NAAQS is the 3-year
average of the weighted annual mean PM2 5 concentrations, while the form of the current 3-month Pb NAAQS is
a 3-month average concentration not to be exceeded during a 3-year period.
1-2
-------
2 POLICY-RELEVANT ISSUES IN THE CURRENT
REVIEW
The approach to considering the information available in this review of the health-based
air quality criteria and the current primary NO2 standards is framed by a series of questions, the
answers to which are intended to inform the Administrator's judgment as to whether the current
standards provide requisite protection of public health, and his decisions as to whether to retain
or revise these standards. The ISA and PA developed in this new review of the primary NO2
NAAQS will provide the basis for addressing these questions. These assessments focus on
policy-relevant scientific information and analyses intended to address key questions related to
the adequacy of these standards.
The overarching question in each NAAQS review is:
• Do the currently available scientific evidence and exposure/risk-based information
support or call into question the adequacy of the protection afforded by the current
standard(s)?
As appropriate, a NAAQS review also addresses a second overarching question:
• What alternative standards, if any, are supported by the currently available
scientific evidence and exposure/risk-based information and are appropriate for
consideration?
In considering these overarching questions in the PA, a series of key policy-relevant
issues particular to a given review are addressed. The policy-relevant issues thus far identified
for this review of the primary NO2 standards are presented in section 2.1 as a series of questions.
2.1 REVIEW OF THE PRIMARY NO2 STANDARDS
The approach planned for this review of the primary standards is fundamentally based on
using the Agency's assessment of the current scientific evidence, quantitative assessments of
exposures and/or risks, and other associated analyses (e.g., air quality analyses) to inform the
Administrator's judgments regarding primary standards that are requisite to protect public health
with an adequate margin of safety. This approach involves translating scientific and technical
information into the basis for addressing a series of key policy-relevant questions using both
evidence- and exposure-/risk-based considerations. This series of key questions related to the
primary standards is presented below, in the context of the general approach for the review.
The planned approach for this review of the primary NO2 standards will build on the
substantial body of work developed during the course of the prior reviews and the associated
conclusions, taking into account the more recent scientific information and air quality data now
2-1
-------
available to inform our understanding of the key policy-relevant issues in this review. Key
aspects of the basis for the decision establishing the primary annual NO2 standard in 1971 and
the primary 1-hour NO2 standard in 2010, and retaining them without revision in 2018 are
summarized in Volume 1. The ISA, risk and exposure analyses (as warranted), and PA
developed in this review will provide the basis for addressing the key policy-relevant questions
in the review, and these assessments and analyses will help inform the Administrator's decisions
as to whether to retain or revise the primary NO2 standards.
The final decision on the primary standards is largely a public health policy judgment by
the Administrator.8 Final decisions must draw upon scientific information and analyses about
health effects and risks, as well as judgments about how to deal with the range of uncertainties
that are inherent in the scientific evidence and analyses. The approach of the PA to informing
these judgments is based on a recognition that the available health effects evidence generally
reflects continuums that include ambient air exposures for which scientists generally agree health
effects are likely to occur through lower levels at which the likelihood and magnitude of
response become increasingly uncertain. This approach is consistent with the requirements of the
NAAQS provisions of the Clean Air Act (CAA) and with how the EPA and the courts have
historically interpreted the Act. These provisions require the Administrator to establish standards
that are requisite to protect public health with an adequate margin of safety. In so doing, the
Administrator seeks to establish standards that are neither more nor less stringent than necessary
for this purpose. The provisions do not require that standard be set at a zero-risk level, but rather
at a level that avoids unacceptable risks to public health, including the health of sensitive
groups.9
Evaluations in the PA are intended to inform the Administrator's public health policy
judgments and decisions. In so doing, the PA considers the potential implications of various
aspects of the scientific evidence, the exposure/risk-based information, and the associated
uncertainties and limitations. The Agency's consideration of the full set of evidence and
information available in this review will inform the answer to the following initial overarching
question for the review:
8 Key aspects of the decisions made in the last review, including the Agency's consideration of important policy
judgments concerning the scientific and exposure/risk information and associated uncertainties and limitations, as
well as the Administrator's public health policy judgments regarding an adequate margin of safety are
summarized in section 3 of Volume 1 of this IRP.
9 More than one population group may be identified as sensitive or at risk in a NAAQS review. The decision in the
review of the primary standards will reflect consideration of the degree to which protection is provided for these
sensitive population groups. To the extent that any particular population group is not among the identified
sensitive groups, a decision that provides protection for the sensitive groups would be expected to also provide
protection for other population groups.
2-2
-------
• Do the currently available scientific evidence and exposure-/risk-based information
support or call into question the adequacy of the public health protection afforded
by the current primary NO2 standards?
In reflecting on this question, we10 will consider the available body of scientific evidence,
assessed in the ISA and used as a basis for developing and/or interpreting the risk/exposure
analyses, including whether it supports or calls into question the scientific conclusions reached in
the last review regarding health effects related to exposure to oxides of nitrogen in ambient air.
Information available in this review that may be informative to public health judgments
regarding significance or adversity of key effects will also be considered. Additionally, the
currently available exposure and risk information, whether newly developed in this review or
predominantly developed in the past and interpreted in light of current information, will be
considered, including the extent to which it may continue to support judgments made in the last
review. Further, in considering this question with regard to the primary NO2 standards, as in all
NAAQS reviews, we give particular attention to exposures and health risks to at-risk
populations.11 As in past reviews of the primary NO2 NAAQS, this will likely include a focus on
people with pre-existing respiratory disease, children, and older adults.
Evaluation of the available scientific evidence and risk/exposure information with regard
to this consideration of the current primary standards will focus on key policy-relevant issues by
addressing a series of questions such as the following:
• To what extent has new information strengthened or otherwise altered the scientific support
for the occurrence of adverse health effects as a result of short- and/or long-term exposure to
gaseous oxides of nitrogen in ambient air?
- What evidence is available from recent studies to inform our understanding of the
nature of exposures to oxides of nitrogen that are linked to various health outcomes?
10 The PA, like the OAQPS Staff Paper in earlier reviews, is a document that provides a transparent OAQPS staff
analysis and conclusions regarding the adequacy of the current standards and potential alternatives that are
appropriate to consider before the issuance of proposed and final decisions. This evaluation of policy implications
is intended to help "bridge the gap" between (1) the Agency's scientific and technical assessments (as presented
in the ISA and the quantitative exposure and risk analyses) and (2) the judgments required of the EPA
Administrator in determining whether it is appropriate to retain or revise the NAAQS. In this way, the PA
integrates and interprets the information from the ISA and quantitative exposure and risk analyses to frame policy
options for consideration by the Administrator. Consistent with this context for the PA, the term "we" throughout
this chapter refers to staff in the EPA's Office of Air Quality Planning and Standards (OAQPS).
11 As used here and similarly throughout this document, the term population (in the context of health and the
primary standards) refers to persons having a quality or characteristic in common, such as a specific pre-existing
illness or a specific age or lifestage. Identifying at-risk populations involves consideration of susceptibility and
vulnerability. Susceptibility refers to innate (e.g., genetic or developmental aspects) or acquired (e.g., disease or
smoking status) sensitivity that increases the risk of health effects occurring with exposure to oxides of nitrogen.
Vulnerability refers to an increased risk of oxides of nitrogen-related health effects due to factors such as those
related to socioeconomic status, reduced access to health care or exposure.
2-3
-------
- What does the available evidence, including that recently available, indicate about
health effects associated with specific oxides of nitrogen (e.g., NO2, NO)?
- To what extent is key scientific evidence available to improve or alter our
understanding of the health effects associated with various time periods of exposures,
including short-term (e.g., 1-hour) and long-term exposures (e.g., more than one
month to years) to oxides of nitrogen?
o At what pollutant concentrations do these health effects occur?
o To what extent is new information available to improve our understanding of
the range of ambient air concentrations within which oxides of nitrogen
contribute to health effects?
o Is there evidence of effects at oxides of nitrogen exposure concentrations lower
than those at which effects were previously observed or in areas that would
likely have met the current primary NO2 standards?
o To what extent are health effects found to be associated with oxides of nitrogen
in epidemiologic studies being elicited by oxides of nitrogen exposure versus
exposure to one or more co-occurring pollutants (e.g., PM2.5, CO, O3, SO2,
other traffic-related pollutants)?
o To what extent is new information available to improve the characterization of
the severity and/or potential adversity of N02-induced respiratory effects
reported in controlled human exposure studies? To what extent does such
information inform an understanding of effects in at-risk populations?
- Has new information altered our understanding of human lifestages and populations
that are particularly at increased risk for experiencing health effects associated with
exposure to oxides of nitrogen?
o What new information is available to inform our understanding of potential
health effects in at-risk populations and lifestages living, working, playing, or
going to school near ambient air sources of oxides of nitrogen (e.g., near
roads)?
o To what extent is new information available regarding co-occuring risk factors
that may be related to increased risk for experiencing health effects associated
with exposure to oxides of nitrogen (e.g., children with asthma)?
o Is there new information on the nature of the exposure-response relationship in
different at-risk lifestages and/or populations?
• To what extent is new information available to improve our understanding of the NO2
concentration gradients around important sources, such as major roads and combustion
sources, and how those gradients relate to ambient air monitoring concentrations across
larger areas?
• To what extent does risk or exposure information suggest that exposures of concern are likely
to occur with recent ambient air NO2 concentrations in the U.S. or with concentrations that
just meet the current primary NO2 standards?
2-4
-------
- Are the estimated exposures/risks considered in this review of sufficient magnitude
such that the health effects might reasonably be judged to be important from a public
health perspective?
- What new information is available to improve our understanding of exposure
measurement error and the role of exposure in epidemiologic inference, particularly
for interpreting long-term exposure studies?
- What are the important uncertainties associated with any exposure/risk estimates?
• To what extent have important uncertainties identified in the last review been reduced and/or
have new uncertainties emerged?
• To what extent does the newly available information reinforce or call into question any of the
basic elements (i.e., indicator, form, averaging time, and level) of the current primary NO2
standards?
If the information in the current review suggests that revision of the current primary
standards would be appropriate to consider, the PA will evaluate how the standards might be
revised based on the available scientific information, air quality assessments, and exposure/risk
information and will consider what the available information indicates as to the health protection
expected to be afforded by the current or potential alternative standards. Such an evaluation may
consider the effect of revision of one or more elements of a standard (indicator, averaging time,
form, and level), with the effect being evaluated based on the resulting potential standard and all
of its elements collectively. Based on such evaluations, the PA would then identify potential
alternative standards (specified in terms of indicator, averaging time, form, and level) intended to
reflect a range of alternative policy judgments as to the degree of protection that is requisite to
protect public health with an adequate margin of safety, as well as options for standards expected
to achieve it. Evaluation of what revision(s) of the standard(s) might be appropriate to consider
would be framed by specific policy-relevant questions such as the following:
• Does the currently available information call into question the use of NO2 as the indicator for
the primary standards for oxides of nitrogen? Is support provided for considering a different
indicator?
• Does the currently available information call into question the current averaging times? Is
support provided for considering different averaging times for the standards?
• What does the currently available information indicate with regard to the range of levels and
forms of alternative standards that may be supported, and what are the uncertainties and
limitations in that information?
• What do the available analyses indicate with regard to exposures and risks associated with
specific alternative standards? What are the associated important uncertainties? To what
extent might such alternatives be expected to reduce adverse impacts attributable to oxides of
nitrogen in ambient air, and what are the associated uncertainties in the estimated reductions?
2-5
-------
The approach to reaching conclusions on the current primary standards and, as
appropriate, on potential alternative standards is summarized in general terms in Figure 2-1.
2-6
-------
Adequacy of Current Standards
Evidence-based Considerations
> Does currently available evidence and related
uncertainties strengthen or call into question prior
conclusions?
¦ Evidence of health effects not previously
identified or at lower exposures?
¦ Newly identified at-risk populations?
¦Effects at lower concentrations than previously
observed or in areas that would have likely met
current standards?
Exposure/Risk-based Considerations
> Nature, magnitude and importance of
estimated exposures and risks associated
with just meeting the current standards?
> Roles of annual and 1-hour standards?
/'Uncertainties in the exposure and risk
estimates?
Consider
Retaining
Current Standards
Consideration of Potential Alternative Standards
1
r
Elements of Potential Alternate Standards
>Indicator, Averaging Time, Form, Level
1
f
Potential Alternative Standards for Consideration
Figure 2-1. Overview of general approach for review of the primary NO2
standards.
2-7
-------
3 DEVELOPMENT OF THE INTEGRATED SCIENCE
ASSESSMENT
The ISAs are intended to summarize and assess the scientific evidence related to public
health or welfare effects of air pollutant exposures, consistent with the air quality criteria defined
in Section 108 of the CAA and with the EPA's obligation to periodically review, and revise as
appropriate, those air quality criteria under section 109. The content of the ISA, in conjunction
with additional technical and policy assessments and advice from the CASAC and input from the
public, provides the scientific basis for the EPA's decisions on the NAAQS. This section
provides information relevant to the development of the ISA for Oxides of Nitrogen - Health
Criteria as part of the current review of the primary NO2 NAAQS. Sections 3.1 and 3.2 provide
overviews of the anticipated organization and scope of the ISA, respectively. Section 3.3
summarizes the planned approach to developing the ISA, including preliminary results of the
literature search and screening effort. Section 3.4 presents scientific questions to guide the
development of the ISA in the current review.
3.1 ORGANIZATION OF THE ISA IN THE CURRENT REVIEW
The organization of the ISA for Oxides of Nitrogen - Health Criteria will be consistent
with that used in the recent assessments for other criteria pollutants (e.g., U.S EPA, 2020a; U.S
EPA, 2020b). It will be organized around a series of detailed, topic-specific chapters12 and an
Integrated Synthesis drawn from the information in those chapters. Chapters will provide
thorough assessments of the scientific evidence pertaining to specific topic areas including
atmospheric science, exposure and dosimetry, and various human health outcomes. Each chapter
will contain an evaluation of results from recent studies integrated with evidence from previous
assessments. Chapters for each health outcome category (e.g., respiratory effects) will include
detailed conclusions reflecting the overall strength of the evidence supporting cause-effect
relationships between pollutant exposures and particular health effects. These "causality
determinations" will be based on consideration of various aspects of the evidence, including
consistency within a scientific discipline, coherence across disciplines, biological plausibility,
and other factors as discussed in section 3.3.4 and Appendix A. Causality determinations will
additionally consider the populations in which health effects have been demonstrated to occur
and the evidence that certain populations are at increased risk of pollutant-related effects because
12 Recent ISAs used the term "appendices" to denote individual sections of the assessment. In this ISA for Oxides of
Nitrogen, and future ISAs, the term "chapter(s)" will be used to make a clear distinction between the main body
of the document and any attachments to the ISA containing supporting information.
3-1
-------
they are more sensitive to pollutant exposures and/or because they experience higher exposures.
Chapters will additionally present targeted evaluations of the evidence on other scientific issues
that may be particularly relevant for subsequent policy considerations. These other issues may
include the concentration-, exposure-, or dose-response relationships for particular health effects;
the strengths and limitations of various exposure estimation approaches and study designs; the
appropriate time lags between exposure and effect or the appropriate exposure periods for
particular effects; and the public health significance of effects associated with exposures to NO2
and other oxides of nitrogen.
Drawing from supporting chapters, the Integrated Synthesis will provide a concise
synopsis of ISA conclusions and a synthesis of key findings considered in characterizing
pollutant exposures and relationships with health effects. The Integrated Synthesis will include
summaries of information for each topic area, including information on pollutant-related sources,
emissions, and atmospheric science; exposures and biokinetics; and health effects. For the health
effects evidence, the Integrated Synthesis will summarize ISA causality determinations,
conclusions on the populations and/or lifestages that may be at increased risk of pollutant-related
effects, and other chapter conclusions on policy-relevant scientific issues.
In addition to the topic-specific chapters and the Integrated Synthesis, the ISA for Oxides
of Nitrogen - Health Criteria will include a Preface that summarizes major legal and historical
aspects of prior NAAQS reviews, an Executive Summary written to be accessible to a wide
range of audiences, and a Process Chapter. The Process Chapter will describe the approach taken
to develop the ISA, including the methods for literature search and review, documentation,
evaluation of individual study quality, public engagement, and quality assurance. The Process
Chapter will draw from the general approach described in Appendix A of this IRP and from
comments on Appendix A from members of the CAS AC Oxides of Nitrogen Health Panel. The
approach described in Appendix A builds on the 2015 Preamble to the IS As (U.S. EPA, 2015),
with updates reflecting advances implemented in recent IS As (U.S EPA, 2020a; U.S EPA,
2020b) and the EPA's consideration of recommendations on the ISA causality framework from
an ad hoc committee of the National Academies of Sciences, Engineering, and Medicine
(NASEM) (NASEM, 2022). The Process Chapter will also provide a general description of the
CASAC review process and information on any supplementary materials, such as information
accessible through the Health and Environmental Research Online (HERO) database for the ISA,
with updates reflecting advances implemented in recent IS As (U.S EPA, 2020a; U.S EPA,
2020b) and the EPA's consideration of recommendations on the ISA causality framework from
an ad hoc committee of the National Academies of Sciences, Engineering, and Medicine
(NASEM) (NASEM, 2022). The Process Chapter will also provide a general description of the
CASAC review process and information on any supplementary materials, such as information
3-2
-------
accessible through the Health and Environmental Research Online (HERO) database for the ISA.
13
3.2 SCOPE OF THE ISA IN THE CURRENT REVIEW
The primary NO2 NAAQS are intended to protect public health from exposures to NO2
and other gaseous oxides of nitrogen (see section 3.2.2 for a description of oxide of nitrogen
compounds).14 Thus, the ISA developed in this review will evaluate the atmospheric science,
human exposure and dosimetry, and human health effects evidence for the gaseous oxides of
nitrogen. The evidence for human health effects associated with organic and inorganic nitrates
was evaluated in the Integrated Science Assessment for Particulate Matter (U.S. EPA, 2019a),
and considered in the reconsideration of the particulate matter NAAQS (89 FR 16202, March 6,
2024). The evidence for ecological effects of oxides of nitrogen was reviewed in conjunction
with the evidence for ecological effects of sulfur oxides and particulate matter in the ISA for
Oxides of Nitrogen, Oxides of Sulfur, and Particulate Matter - Ecological Effects (U.S EPA,
2020a).
The ISA for Oxides of Nitrogen - Health Criteria will evaluate relevant studies that have
become available since the cutoff-date for the 2016 ISA (i.e., March 2014) in the context of
studies evaluated in previous assessments (i.e., U.S. EPA, 2016; U.S. EPA, 2008; U.S. EPA,
1993; U.S EPA, 1982). For topic areas in which research efforts have subsided and older studies
remain the definitive works available in the literature, those older studies from previous
assessments will be the primary focus of the ISA's evaluation. The sections below define the
scoping criteria used to screen the available scientific literature and evaluate studies for their
potential to inform the evidence assessment in the ISA. To meet ISA scoping criteria, studies
must present new information or analyses and must have undergone scientific peer review.
Review articles that are limited to summarizing and interpreting existing studies, without
presenting new information or analyses, are outside the scope of the ISA. The following sections
present additional discipline-specific literature scoping criteria for studies of human health
effects (3.2.1), atmospheric science (3.2.2), and exposure and dosimetry (3.2.3).
13 HERO is a database of scientific studies and other references used to develop EPA assessments and is available at
is available at: https://heronet.epa.gov.
14 Section 108(c) of the CAA indicates that the air quality criteria relating to NO2 include consideration of "nitric
and nitrous acids, nitrites, nitrates, nitrosamines, and other carcinogenic and potentially carcinogenic derivatives
of oxides of nitrogen."
3-3
-------
3.2.1 Health Effects
The ISA will use discipline-specific population, exposure, comparison, outcome, study
design (PECOS) statements to define the set of relevant health effects studies (see Table 3-1
through Table 3-3 and Appendix A). The PECOS statements help to identify the objectives of the
assessment and establish criteria that should be met to consider a study for inclusion in the ISA,
thereby facilitating identification of the potentially relevant literature and informing the
integration and synthesis of study results. The PECOS statements are informed by the body of
evidence from previous ISAs and air quality criteria documents (AQCDs),15 expert knowledge of
the relevant scientific literature, and by recent ambient air quality information (i.e., as described
for exposure criteria in Tables 3-1 and 3-3). Studies meeting all five aspects of the PECOS
statement will be considered for inclusion in the ISA.
The health chapters of the ISA will evaluate the scientific literature related to a range of
health outcomes associated with exposures to oxides of nitrogen including, but not limited to,
respiratory effects, cardiovascular effects, reproductive and developmental effects, cancer, and
mortality. Building upon the 2016 ISA, the EPA will review the available epidemiologic,
controlled human exposure, and animal toxicological evidence related to these and other health
outcome categories to the extent data are available. The results of recent studies will be
integrated with the findings from the 2016 ISA along with any new interpretations of previous
findings that the recent studies may support. The ISA will also integrate previous information on
the populations and lifestages at increased risk with new evidence for existing and any newly
identified risk factors.
15 The last AQCD was published by the EPA in 2006. Moving forward the science assessments supporting the
NAAQS review were renamed the Integrated Science Assessments.
3-4
-------
Table 3-1. PECOS statement to define the criteria and framework for identifying
relevant oxides of nitrogen epidemiologic studies.
Exposure
Duration
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Short-term
exposure
Population (P): Any human population, including populations or lifestages that might be at
increased risk;
Exposure (E): Short-term exposure (i.e., up to 30 days) to oxides of nitrogen concentrations
relevant to ambient air in the U.S.;16
Comparison (C): Per unit increase (i.e., ppb) or humans exposed to lower concentrations of
oxides of nitrogen compared to higher concentrations (e.g., categorical comparisons between
different exposure metric quantiles);
Outcome (O): Change or difference in risk (incidence/prevalence) of health effects (e.g.,
respiratory, cardiovascular, metabolic syndrome and diabetes, total mortality, reproductive and
developmental);
Study Design (S): Epidemiologic studies, such as panel, case-crossover, time-series, case-
control studies, cohort, cross-sectional studies, and quasi-experimental, with appropriate timing
of exposure for the health outcome of interest.
Long-term
exposure
Population (P): Any human population, including populations or lifestages that might be at
increased risk;
Exposure (E): Long-term exposure (i.e., longer than 30 days) to oxides of nitrogen
concentrations relevant to ambient air in the U.S.;16
Comparison (C): Per unit increase (i.e., ppb) or humans exposed to lower concentrations of
oxides of nitrogen compared to higher concentrations (e.g., categorical comparisons between
different exposure metric quantiles);
Outcome (O): Change or difference in risk (incidence/prevalence) of health effects (e.g.,
respiratory, cardiovascular, metabolic syndrome and diabetes, total mortality, reproductive and
developmental, cancer);
Study Design (S): Epidemiologic studies, such as panel, case-crossover, time-series, case-
control studies, cohort, cross-sectional studies and quasi-experimental, with appropriate timing
of exposure for the health endpoint of interest.
16 The ISA for Oxides of Nitrogen - Health Criteria will focus on the studies most likely to inform scientific and
policy conclusions relevant to ambient air quality in the United States. To efficiently identify such studies during
screening, PECOS exposure criteria for epidemiologic studies include a concentration cutoff. Specifically,
epidemiologic studies of NO2 should report overall average NO2 exposures (e.g., averaged across study
populations/locations and over study periods) at or below 22 ppb. This concentration cutoff reflects the 98th
percentile of annual average NO2 concentrations measured at ambient air monitors in the U.S. during the most
recent 15 years of available data (i.e., 2008-2022) (U.S EPA, 2023b). Consistent with the PECOS exposure
criteria for animal toxicology studies (Table 3-3), epidemiologic studies reporting average NO2 concentrations
above 22 ppb will be considered for inclusion in the ISA if those studies provide valuable and/or unique insights
into policy-relevant issues (e.g., studies that examine unique endpoints, use alternative methods for confounder
control (causal inference methods), examine potentially at-risk populations, examine associations in under-
represented locations, etc.).
3-5
-------
Table 3-2. PECOS statement to define the criteria and framework for identifying
relevant oxides of nitrogen controlled human exposure studies.
Exposure
Duration
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Population (P): Human volunteers enrolled in controlled exposure studies, including
volunteers representing populations or lifestages that might be at increased risk of pollutant-
related health effects;
Exposure (E): Controlled inhalation exposure to NO2 or other oxide of nitrogen - pollutant
exposures must be controlled by the experimenters and not simply a measure of ambient or
occupational exposure;
Single or
repeated
short-term
exposures
Comparison (C): An appropriate control exposure to filtered air or room air for each study
participant or an appropriately matched comparison group exposed to filtered air or room air;
Outcome (0): Outcomes of interest are those that relate to human health, including effects on
the respiratory system, cardiovascular system, immune system, nervous system, diabetes,
cancer, or reproduction and development. Effects of interest include changes in indicators or
measures of physiological function, health-relevant biomarkers, and organ structure. Effects
can be directly measured in exposed study participants or in cells, tissues, or fluids isolated
from study participants;
Study Design (S): Studies that perform controlled human exposures meeting the above
criteria or that analyze data from previously conducted controlled human exposures (e.g.,
reanalysis, meta-analysis).
3-6
-------
Table 3-3. PECOS statement to define the criteria and framework for identifying
relevant oxides of nitrogen animal toxicological studies.
Exposure
Duration
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Short-term
exposure
Population (P): Laboratory nonhuman mammalian animal species (e.g., nonhuman primate,
mouse, rat, guinea pig, minipig, rabbit, cat, dog) of any lifestage including models of increased
susceptibility;
Exposure (E): Short-term (i.e., up to 30 days) inhalation exposure to relevant oxides of
nitrogen concentrations (i.e., 5 ppm or below);17
Comparison (C): An appropriate control group exposed to clean air (e.g., room air, filtered air)
control;
Outcome (0): Outcomes of interest are those that relate to human health, including effects on
the respiratory system, cardiovascular system, immune system, nervous system, diabetes,
cancer, reproduction and development, or other human health effects (e.g., nervous system).
Effects of interest include changes in indicators or measures of physiological function, health-
related biomarkers, and organ structure. Effects can be directly measured in exposed animals
or in cells, tissues, or fluids isolated from animals;
Study Design (S): Controlled exposure studies of animals in vivo meeting the above criteria.
Long-term
exposure
Population (P): Laboratory nonhuman mammalian animal species (e.g., nonhuman primate,
mouse, rat, guinea pig, minipig, rabbit, cat, dog) of any lifestage including models of increased
susceptibility;
Exposure (E): Long-term (i.e., longer than 30 days) inhalation exposure to relevant oxides of
nitrogen concentrations (i.e., 5 ppm or below);17
Comparison (C): An appropriate control group exposed to clean air (e.g., room air, filtered air)
control;
Outcome (O): Outcomes of interest are those that relate to human health, including effects on
the respiratory system, cardiovascular system, immune system, nervous system, diabetes,
cancer, reproduction and development, or other human health effects (e.g., nervous system).
Effects of interest include changes in indicators or measures of physiological function, health-
related biomarkers, and organ structure. Effects can be directly measured in exposed animals
or in cells, tissues, or fluids isolated from animals;
Study Design (S): Controlled exposure studies of animals in vivo meeting the above criteria.
3.2.2 Atmospheric Science
The term "oxides of nitrogen" refers to oxidized nitrogen compounds, including nitric
oxide (NO), NO2, and other oxidized nitrogen-containing compounds formed from NO and NO2.
Nitrogen dioxide can also react with a variety of atmospheric species to produce organic and
inorganic nitrates, which contribute to atmospheric particulate matter (U.S. EPA, 2016; U.S.
17 Five ppm is approximately two orders of magnitude higher than peak NO2 concentrations in the ambient air in the
U.S. As discussed in Appendix A (A.5.3.4) and in the 2015 Preamble to the ISAs (U.S. EPA, 2015), animal
exposures within one to two orders of magnitude of recent ambient air concentrations are considered relevant to
ambient air exposures. This concentration cutoff is also consistent with that used in the 2016 ISA for Oxides of
Nitrogen - Health Criteria (U.S. EPA, 2016). Experimental studies investigating the effects of concentrations
greater than 5 ppm may be considered for inclusion in the ISA if they provide insight into biological plausibility.
3-7
-------
EPA, 2019a). This review of the primary NO2 NAAQS focuses on evaluating the health effects
associated with exposure to the gaseous oxides of nitrogen. The atmospheric chemistry,
exposure, and health effects associated with nitrogen compounds present in particulate matter
(PM) were most recently considered in the U.S. EPA's review of the NAAQS for PM (U.S.
EPA, 2019a). Based on definitions commonly used in the atmospheric science literature, the
abbreviation NOy will be used to refer to all oxides of nitrogen and NOx will be used to refer
specifically to the sum of NO2 and NO concentrations (40 CFR Part 58.1).
The ISA will use discipline-specific scoping statements to identify potentially relevant
atmospheric science studies (see Table 3-4 and Appendix A). Importantly, application of scoping
statements that consider pollutant sources, transport and transformation, exposure/extent, and
measurement and modeling (STEM) is consistent with current best practices for reporting or
evaluating health science data as recommended by the NASEM. The STEM statement defines
the objectives of the atmospheric science assessment and establishes criteria that should be met
to be considered for inclusion in the ISA. A study meeting any of the four aspects of the STEM
statement will be considered for inclusion in the ISA. The STEM statement for the ISA shown in
Table 3-4 has been informed by the body of evidence from the previous ISAs/AQCDs18 and by
expert knowledge of the relevant scientific literature.
With the focus provided by the STEM statement, the Atmospheric Science chapter will
present and evaluate the latest data related to emissions sources of oxides of nitrogen, emissions
chemistry and concentration trends, spatial and temporal patterns for oxides of nitrogen in
ambient air, and the spatial and temporal trends in oxides of nitrogen emissions and
concentrations. The impact of the COVID-19 pandemic on emissions and ambient air NOy
concentrations will be discussed. In addition, the assessment will include information about near-
road NO2 monitoring in the U.S. and advances in measurement and modeling methods, including
new studies of Federal Reference Method and Federal Equivalent Method performance,
improvements in more advanced spectroscopic measurements methods, and recent innovations in
atmospheric modeling of oxides of nitrogen.
18 The last AQCD was published by the EPA in 2006. Moving forward the science assessments supporting the
NAAQS review were renamed the Integrated Science Assessments.
3-8
-------
Table 3-4. STEM statement to define the criteria and framework for identifying
relevant oxides of nitrogen atmospheric studies.
Statement
Description
Source (S)
Studies reporting quantitative emissions estimates of oxides of nitrogen as well as
observations of physical and chemical characteristics that add to our understanding of
sources and emissions of oxides of nitrogen.
Transport and
Transformation (T)
Studies investigating atmospheric fate and transport, transformation, and deposition
processes involving oxides of nitrogen, including transport of air pollutants at various
scales (i.e., national/global, regional, urban, neighborhood), chemical transformations in
the atmosphere, and estimates of atmospheric deposition that add to our understanding
of atmospheric processes.
Exposure/Extent (E)
Studies reporting observations and estimates of ambient air concentrations and their
trends for oxides of nitrogen relevant to U.S. conditions, including spatial variability on
various scales (i.e., national/global, regional, urban, neighborhood); temporal trends such
as diurnal, weekday/weekend, seasonal, and long-term trends; or characteristics, such as
composition or relationship with atmospheric properties that provide up to date
concentrations and estimates or add to our understanding of spatiotemporal
concentration trends.
Measurement and
Modeling (M)
Studies describing methods of measurement of oxides of nitrogen by federal reference
and equivalency methods, satellite remote sensing estimates, low-cost sensor estimates,
or research methods; and modeling techniques (e.g., chemical transport modeling) for
characterizing oxides of nitrogen concentrations in ambient air, including the evaluation of
measurement principles and modeling assumptions, examination of potential bias and
uncertainties, and method intercomparisons that are relevant to the NAAQS or to studies
in this ISA.
3.2.3 Exposure Science & Dosimetry
Similar to the Atmospheric Science chapter, the scope of the Exposure Science chapter
will be defined by a discipline-specific STEM statement (see Table 3-5 and Appendix A). The
ISA will present and evaluate relevant evidence related to exposure continuums for NO2 and
other oxides of nitrogen, characterization of oxides of nitrogen exposures, and exposures to
factors that may confound associations in epidemiologic studies (e.g., copollutants). The ISA
will consider key uncertainties from the last review and the extent to which new scientific
evidence may inform our ability to characterize and/or reduce those uncertainties during the
current review. The ISA will also evaluate the literature relating to dosimetry of inhaled oxides
of nitrogen.
3-9
-------
Table 3-5. STEM statement to define the criteria and framework for identifying
relevant oxides of nitrogen exposure studies.
Statement
Description
Source (S)
Emissions from outdoor (e.g., traffic) or indoor (e.g., cookstove emission) sources of
oxides of nitrogen.
Transport and
Transformation (T)
Atmospheric and environmental processes of oxides of nitrogen, including the transport of
air pollutants at various scales (i.e., national/global, regional, urban, neighborhood,
middle, micro scales, and microenvironments), including near-source (e.g., near traffic)
transport and transformation, and advances in chemical transformations and deposition
from the atmosphere (e.g., photochemical reactions) and microenvironments (e.g., indoor
chemistry).
Exposure/extent (E)
Exposure levels of oxides of nitrogen, relevant to ambient air in the U.S, characterized by
various surrogates (e.g., ambient air concentrations, microenvironmental concentrations,
personal exposure) and exposure determinants (i.e., factors which may lead to differential
exposures, such as proximity to sources, activity patterns, and socioeconomic status),
including characterizing concentrations and spatiotemporal temporal trends of various
exposure surrogates and examining populations experiencing elevated exposures or the
exposure patterns (e.g., exposure level, duration, and frequency) experienced by
populations identified in health studies as being at increased risk of effects.
Measurement and
Modeling (M)
Measurement methods (e.g., federal reference and equivalent methods, passive
samplers, sensors, and remote sensing) and modeling techniques (e.g., land use
regression and dispersion models) characterizing ambient air, indoor/microenvironmental
air, and personal exposures, including the evaluation of measurement principles and
modeling assumptions, examination of potential bias and uncertainties, and comparison
of different techniques.
3.3 PROCESS FOR DEVELOPING THE ISA
Appendix A to this volume of the IRP presents planned updates to the ISA development
approach described in the 2015 Preamble to the ISAs (U.S. EPA, 2015). As noted previously, the
process described in Appendix A builds on the approach described in the 2015 Preamble to the
ISAs, with updates reflecting advances implemented in recent ISAs and the EPA's consideration
of recommendations on the ISA causality framework from an ad hoc committee of the NASEM
(NASEM, 2022). Comments on the process described in Appendix A provided by members of
the CAS AC Oxides of Nitrogen Health Panel will be considered in developing the draft ISA for
Oxides of Nitrogen - Health Criteria in this review. The sections below provide a high-level
overview of the updated ISA development process as it is being applied in the current review and
an overview of initial results of the literature search and screening efforts.
The process for developing the ISA for Oxides of Nitrogen - Health Criteria began when
the Call for Information was published in the Federal Register (87 FR 75625, December 9,
2022). At that time, the public was invited to contribute to the review by commenting on policy-
3-10
-------
relevant issues and by submitting potentially relevant research studies. 19Public comments have
been considered in developing this IRP, and research studies identified by public commenters
have been included in initial literature screening efforts. The sections below summarize key steps
involved in developing the ISA for Oxides of Nitrogen - Health Criteria in this review, including
searching the scientific literature and identifying potentially relevant studies, evaluating
individual study quality, integrating evidence, developing causality determinations, quality
management, and obtaining the CASAC's advice. Each of these steps is discussed in greater
detail in Appendix A of this IRP.
3.3.1 Literature Search
The EPA works to identify potentially relevant new studies for inclusion in the ISA by
conducting multipronged systematic searches that include extensive mining of literature
databases on specific topics in a variety of disciplines. As noted above, the process for
identifying relevant literature began with a Call for Information published in the Federal Register
Notice inviting the public to submit relevant scientific research studies and data that have been
published or accepted for publication (87 FR 75625, December 9, 2022). As part of the public
comments in response to this invitation, 149 peer-reviewed research studies published in
scientific journals were submitted for the EPA's consideration. Research studies submitted by
the public in response to this Call for Information, along with other studies being considered for
the ISA, can be viewed in the project page in EPA's HERO database.20 The EPA reviewed these
studies for relevance following the literature screening process described below.
In addition to studies submitted in response to the Call for Information, the EPA applied
systematic review methodologies to identify peer-reviewed scientific studies relevant to this ISA.
To maximize identification of pertinent published papers for each discipline, literature search
strategies were guided by the discipline-specific scoping statements described above in section
3.2. The literature searching and screening methodology used for this ISA generally followed the
process depicted in Figure A-l of Appendix A. The EPA used a combination of forward citation
searches and keyword searches to find relevant literature in PubMed and Web of Science
published between March 2014 and June 2023. This date range provides overlap with the
literature publication dates for the 2016 ISA, facilitating the identification of studies that may
have become available soon after the literature search was conducted in the last review. For the
19 Public comments were submitted to the docket for the Integrated Science Assessment as a part of the review of
the primary NAAQS for NOx (Docket ID Number: EPA-HQ-ORD-2022-0831). This docket can be accessed at:
https://www.regulations.gov/docket/EPA-HQ-ORD-2022-0831.
20 The HERO database for this review is available at:
https.V/heronet. epa.gov/heronet/index. cfm/project/page/project_id/4 767.
3-11
-------
forward citation searches, relevant published studies cited in previous ISAs or AQCDs were
identified as a seed set and then more recent literature that cited any of the references in the seed
set were identified and considered for inclusion. Keyword searches were developed using strings
of relevant search terms and exclusion terms to capture literature relevant to oxides of nitrogen
for each discipline (i.e., atmospheric science, exposure science, dosimetry, epidemiology,
controlled human exposure, and animal toxicology). For search results focused on human health
effects, automatic topic classification was used to separate studies with relevant keywords related
to ambient air pollution exposure and health outcomes from studies without such keywords. This
process uses machine learning to classify references based on a set of already identified relevant
papers. Finally, a small number of references were also identified for consideration in this ISA
by EPA expert scientists and by review of citations included in previous assessments or in newly
identified literature.
Applying this process, the EPA identified a total of 210,904 unique new studies for title
and abstract screening across disciplines (Figure 3-1). To provide a high-level view of the
volume and types of studies identified at this stage, the number of new studies identified was
further refined by scientific discipline. Studies identified by the literature search are documented
in the project page in the HERO database.
3.3.2 Identifying Potentially Relevant Studies
New studies identified during the literature search have been evaluated for potential
relevance using a multipronged literature screening approach designed to maximize efficiency
and the likelihood that relevant studies are identified. Initially, studies have been evaluated by
comparing their titles and abstracts to the discipline-specific scoping criteria defined by PECOS
or STEM statements. Reflecting the large number of studies identified, machine ranking tools
(e.g., SWIFT-Active Screener (Sciome, RTP, NC, USA) (Howard et al., 2020) and Living
Literature Review (U.S. EPA, Durham, NC, USA) (U.S EPA, 2023a) were used to maximize
efficiency. Title and abstract screening resulted in exclusion of a total of 203,619 studies deemed
out of scope, leaving a total of 7,285 potentially relevant new studies (Figure 3-1).21
21 The number of records excluded at the full text level for scoping and study quality deficiencies will be added to
the figure after completion of screening at the full text level. At that time, the number of studies included in the
ISA will also be added to the figure. The final figure, including all values, will be included in the Process chapter
of the ISA for Oxides of Nitrogen - Health Criteria.
3-12
-------
¦o
ai
¦a
3
o
c
Records identified through other sources:
Public comment (n = 149)
Records identified through
database searches:
(n = 224,333)
Full-text records assessed
for eligibility (n = 7,285)
Duplicate records removed
before screening (n = 13,429)
1
Studies included in review (n = TBD)
Records screened at the
Ti/Ab level (n = 210,904)
Records excluded due to study
scoping deficiencies (n = 203,619)
Records excluded by study scoping criteria (n = TBD)
Records excluded due to study quality deficiencies (n = TBD)
Figure 3-1. Preliminary literature flow diagram for the oxides of nitrogen ISA. Detailed
literature screening results and include/exclude decisions can be found in the HERO
database. (Available at:
https://heromt.epa.gov/heronet/mdex.cjhi/projeci/page/project_id/4767). TBD, to be
determined; Ti/Ab, title and abstract.
A preliminary breakdown of the number of new studies currently under consideration for
inclusion in the ISA, organized by discipline, is shown in Table 3-6. The specific types of studies
being considered for inclusion can be visualized using evidence maps (Figure 3-2 and Figure 3-
3).
3-13
-------
Table 3-6. Preliminary literature search and screening results by scientific discipline.
Discipline
Number of Studies
Identified by Literature
Search
Potentially Relevant
Studies Identified by
Ti/Ab Screening
Number of Studies
Considered for Inclusion
Atmospheric Science
90,126
728
TBD
Exposure Science
58,296
2,316
TBD
Epidemiology
48,026
4,078
TBD
Controlled Human
Exposure
614
12
TBD
Animal toxicology
13,842
151
TBD
Total number unique studies = 210.904 (duplicates removed)
Notes: TBD = to be determined; Ti/Ab = title and abstract.
Health Outcome
# studies
>•
o
08
0ZJ m
3 5
<_> & +_»
1/1 —
> s
.2 %
~0 -D
ru •—
U Q
03
*->
c
- C
« £
3 CL
o -2
cu (I)
cc D
cu
u
c
(U
u
a.
cu
X
c
a)
DC
CUD
_o
o
4-"
ro
s
a;
3
_a>
cu
_o
•3
C
o
Epidemiology
1,583
1,325
836
411
503
22
234
64
20
23
23
1,035
1,404
Controlled Human Exposure
11
5
0
0
0
2
0
0
0
0
0
\
0
Toxicological
45
40
12
5
43
2
11
5
4
2
1
\
16
Figure 3-2. Preliminary evidence map depicting potentially relevant epidemiologic,
controlled human exposure, and toxicological studies identified during title and
abstract literature screening.
3-14
-------
Atmospheric Science
o
Q.
X
o O
3 °
O ?
LO —
< £
E ^
.= ru
h- Q-
Q
E
<
ro
c
o
"O
3
+¦»
to
c
o
+¦»
c
0)
>
0)
+¦»
c
Exposure Science
-------
appropriate consideration of confounders in studies conducted during COVID-related
lockdowns,23 and strength of study design.24 Preliminary evaluation of epidemiologic study
quality will be followed by the full study quality evaluation, across all disciplines, as described
in Appendix A. The full study quality evaluation will be the final step in full-text screening to
identify studies for inclusion in the ISA and to inform the level of confidence to be placed in
inferences that can be drawn from particular studies.
Literature review software such as DistillerSR (Ottawa, Ontario, Canada) (Hamel et al.,
2020) or HAWC (U.S EPA, 2021a) is used for management of individual study quality
evaluations. Studies that are determined to meet scoping criteria and that are judged of sufficient
quality based on the approach described in Appendix A (section A. 5) are tagged in HERO for
inclusion in the ISA. When fully available, results of the literature search and screening efforts
and the evaluation of individual study quality will be captured in flow diagrams that document
the number of references identified from each database searched (i.e., PubMed and Web of
Science), the number of references evaluated in each screening step, and general reasons for
reference exclusion (Figure 3-1).
3.3.4 Integration of Evidence and Determination of Causality
The ISA for Oxides of Nitrogen - Health Criteria will evaluate and integrate the recent
scientific evidence on the health effects of oxides of nitrogen exposures with evidence from
previous assessments. Based on this integration, the ISA will reach conclusions on the weight of
evidence supporting cause-effect relationships between oxides of nitrogen exposures and various
health outcomes. These "causality determinations" reflect overall confidence in such cause-effect
relationships based on integrating the full body of evidence within and across disciplines. The
ISA framework for reaching these causality determinations recognizes that, compared to any
single study, the availability of multiple studies evaluating a particular topic, each with different
under investigation. The lack of such validation was an important uncertainty in some epidemiologic studies
evaluated in the last review of the primary NO2 NAAQS (e.g., 83 FR 17268, April 18, 2018).
23 Studies described as "natural experiments" conducted during COVID lockdown(s) should explicitly consider
potential confounders common during lockdown periods, such as changes in employment status, activity patterns
(e.g., time spent outdoors versus indoors, driving, working, exercising), stress levels, access to health care, and/or
mask wearing.
24 Specifically, epidemiologic studies that examine populations outside North America should be multicity and/or
multi-country and they should address policy-relevant topics (e.g., studies that use alternative methods for
confounder control (causal inference methods, quasi-experimental studies), copollutant confounding, effect
measure modification for potential at-risk factors (race/ethnicity, age, SES indicators, etc.), exposure-
/concentration-response relationships). Studies with these characteristics are most likely to be influential in ISA
causality determinations and other conclusions. Studies that examine populations in North America will not be
excluded from the ISA based on these study characteristics alone as such studies may be useful for evaluating
potential policy options in subsequent steps of the NAAQS review.
3-16
-------
strengths and limitations, provides a more robust foundation for evaluating the overall strength of
the evidence. To aid in forming weight-of-evidence judgments, the ISAs consider various aspects
of the scientific evidence including consistency and coherence across studies, support for
biological plausibility, support for exposure- or dose-response relationships, and several others
(Appendix A, section A.7.2.1). Limitations in the evidence base can result from the presence of
similar uncertainties within a particular subset of studies (e.g., studies similarly affected by
confounding, exposure error, species extrapolation, etc.) or uncertainties that exist across the
broader body of evidence (e.g., inconsistent evidence across disciplines). When the evidence
base includes a group of studies with the same or similar uncertainties of a particular type,
caution is used when developing causality determinations so as not to misrepresent and
perpetuate errors (Savitz and Forastiere, 2021, Savitz et al., 2019).
The ISA causality framework for this review builds on the established framework
described in the 2015 Preamble to the ISAs, with updates reflecting advances implemented in the
recent ISAs and in consideration of recent NASEM recommendations (NASEM, 2022). It
includes a five-level hierarchy to classify the weight-of-evidence for causation as either causal;
likely to be causal; suggestive of, but not sufficient to infer, a causal relationship; inadequate to
infer a causal relationship; and not likely to be a causal relationship (Table 3-7). The updated
draft of the ISA causality framework is described in detail in Appendix A (Section A.7.2.1) to
this IRP.
3-17
-------
Table 3-7. Causality determinations for health outcomes.
Descriptor
Evidence Characteristics
Causal relationship
Evidence is sufficient to conclude that there is a causal relationship with relevant
pollutant exposures. That is, the pollutant exposures have been shown to result in
health effects in studies in which chance, confounding, and other biases can be
ruled out with reasonable confidence. A "causal" relationship is generally based on
multiple high-quality studies conducted by different research groups. Evidence
supporting this determination can include controlled human exposure studies that
consistently demonstrate effects and/or observational studies reporting consistent
health effect associations that, when considered in light of study quality and
coherence with other lines of evidence (i.e., controlled human exposure studies,
animal toxicological studies, and mode of action information), cannot be explained
by plausible alternatives.
Likelytobeacausal
relationship
Evidence is sufficient to conclude that a causal relationship is likely to exist with
relevant pollutant exposures. That is, the pollutant exposures have been shown to
result in health effects in studies where results are not explained by chance,
confounding, and other biases, but uncertainties remain in the evidence overall. A
"likely to be causal" relationship is generally based on multiple high-quality studies
conducted by different research groups. Evidence supporting this determination can
include 1) multiple high-quality observational studies consistently reporting health
effect associations, but with uncertainty remaining related to potential confounding
and/or limited coherence with other lines of evidence (i.e., controlled human
exposure studies, animal toxicological studies, mode of action information) or 2)
consistent evidence in animal models and/or in vitro models (e.g., for cancer-related
effects) that can be reasonably extrapolated to human health, but limited availability
of human data.
Suggestive of, but
not sufficient to infer,
a causal relationship
Evidence is suggestive of, but not sufficient to infer, a causal relationship with
relevant pollutant exposures. That is, the pollutant exposures have been shown to
result in health effects, but chance, confounding, and bias cannot be confidently
ruled out. Evidence supporting a "suggestive" relationship can be comprised of
studies of varying quality that may be generally supportive of pollutant-related
effects, but not entirely consistent and with limited coherence across lines of
evidence. A suggestive determination can be reached with relatively small bodies of
evidence, or, in rare cases, one high quality study.
Inadequate to infer a
causal relationship
Evidence is inadequate to determine that a causal relationship exists with relevant
pollutant exposures. That is, the evidence supporting an "inadequate" relationship is limited
and available studies are of insufficient quantity, quality, consistency, and/or statistical
power to permit a conclusion regarding the presence or absence of an effect.
Not likely to be a
causal relationship
Evidence indicates there is no causal relationship with relevant pollutant exposures.
Several adequate studies, covering the full range of levels of exposure that human
beings are known to encounter and considering at-risk populations and lifestages,
are consistent in not showing an effect at any level of exposure.
3.3.5 Quality Management
The EPA has an agency-wide quality assurance (QA) policy outlined in the EPA Quality
Manual for Environmental Programs (see CIO 2105-P-01.1) and follows the specifications
outlined in EPA Order CIO 2105.1. As required by CIO 2105.1, the EPA Office of Research and
3-18
-------
Development (ORD) maintains a Quality Management Program, which is documented in an
internal Quality Management Plan. The ISAs are designated as Highly Influential Scientific
Assessments (HISA) and are classified as ORD QA Category A. As such, the oxides of nitrogen
ISA is subject to the EPA's Quality Management Program requirements for a Quality
Management Plan and adheres to the Program Quality Assurance Project Plan (PQAPP) for the
Integrated Science Assessment Program, (QAPP ID: L-HEEAD-0030253-QP-1-6). The ISA will
be subjected to management and QA clearance review, and during this step, the CPHEA QA
Manager verifies that the EPA QA requirements are met (see Appendix A for more detail).
3.3.6 CASAC Peer Review
Section 109(d) of the CAA establishes the requirement for an independent scientific
committee to review the Air Quality Criteria (i.e., the ISA in the current process) and the
NAAQS, and to recommend to the EPA Administrator any new NAAQS and any revisions to
existing criteria and NAAQS that may be appropriate (see 42 U.S.C. 7409(d)(2)). The CASAC
was established to fulfill these requirements, and a draft of the ISA will be sent to the CASAC
for review. Coincident with the CASAC review, the draft ISA will also be made available to the
public, with a Federal Register notice announcing public availability and providing instructions
for submitting comments on the draft ISA to the public docket.
As described in Appendix A (Section A. 8), the CASAC will be supplemented by a panel
that includes broad scientific expertise related to oxides of nitrogen and on the science-policy
issues important for this review of the primary NO2 NAAQS. The panel will develop a draft
advisory report with recommendations for the EPA Administrator on the draft ISA. The report
will be transmitted to the CASAC for discussion and deliberation. If the CASAC determines the
contents of the report are appropriate, the committee will adopt the report and transmit it to the
EPA Administrator to reflect its statutorily mandated advice to the Agency.
The EPA will carefully consider advice received from the CASAC and comments from
the public in revising and updating the draft ISA. After appropriate revisions are made, the final
ISA will be made available on the EPA website. A notice announcing the availability of the final
ISA will be published in the Federal Register.
3.4 SCIENTIFIC QUESTIONS TO GUIDE EVALUATION OF THE
EVIDENCE
As noted above, the ISA for Oxides of Nitrogen - Health Criteria being developed in this
review will build upon the evidence assessed and the conclusions reached in the 2016 ISA and
prior assessments. Studies that have become available since the 2016 ISA will be integrated with
the older studies that have been evaluated in previous assessments. Based on the recent evidence,
3-19
-------
conclusions from the 2016 ISA will be re-evaluated. A series of scientific questions will guide
the evaluation of the recent literature, with a focus on 1) whether new scientific evidence
reinforces or calls into question the conclusions reached in the 2016 ISA; 2) whether
uncertainties from the last review have been reduced and/or whether new uncertainties have
emerged; and 3) the degree to which new lines of evidence have become available to support
areas of evaluation not considered in previous assessments. The specific scientific questions that
will guide the evaluation of the literature for each discipline are described in the sections below.
3.4.1 Source to Concentration - Air Quality, Atmospheric Science, Fate, and Transport
The ISA will present and evaluate data related to ambient air concentrations of oxides of
nitrogen; sources leading to the presence of oxides of nitrogen in the atmosphere; and physical
and chemical processes that determine the formation, degradation, and lifetime of oxides of
nitrogen in the atmosphere. The following questions will guide the evaluation of the scientific
literature for air quality, atmospheric science, and fate and transport.
• What new information is available to inform our understanding of the atmospheric chemistry
of oxides of nitrogen? How does new information characterize the role of atmospheric
chemistry in determining relationships among oxides of nitrogen species? What new
information is available with respect to formation, transport, and transformation of oxidized
nitrogen species that may be important in assessing health effects from multipollutant
exposures? How does the near-source environment (e.g., near major highways or large
combustion sources) influence chemistry and spatiotemporal variability of oxides of
nitrogen?
• What new information exists regarding characterization of sources of oxides of nitrogen to
ambient air in both urban and rural environments? What are the relevant spatial and temporal
scales for considering emissions of oxides of nitrogen to ambient air? What new information
is available regarding existing and emerging energy, industrial, transportation, and
agricultural sources and their impacts on emissions of oxides of nitrogen?
• To what extent have new methods been developed to improve measurements of oxides of
nitrogen in ambient air, particularly those that measure NO2 directly? How have these new
methods reduced interference problems in measuring oxides of nitrogen? What advances
have taken place in the development of low-cost community sensor technologies? What
advances have taken place in the development of satellite-based remote sensing
technologies? What limitations still remain?
• What new modeling methods and refinements have been developed that improve our
understanding and predictive capabilities of spatial and temporal patterns of NO2 and, more
broadly, NOy?
• Based on recent air quality and emissions data, what is known about recent emissions and
resulting ambient air concentrations of oxides of nitrogen? How have emissions and
concentrations of NOx and of NO2 changed since the 2016 ISA? To what extent can new
data sources (e.g., satellites, community sensors) or air quality analyses be used to improve
the characterization of ambient air concentrations of oxides of nitrogen?
3-20
-------
• What spatial and temporal patterns can be seen in ambient air concentrations of NO2 and the
broader category of NOy concentrations? In particular, what spatial and temporal patterns
can be seen on a micro-scale near sources including major roadways, industrial operations,
residential fuel combustion, or wildland fires? What do ambient air quality characterizations
(including examinations of the influence of meteorological parameters) indicate about spatial
patterns on neighborhood, urban, regional, and national scales?
• Based on air quality and emissions data for oxides of nitrogen and atmospheric chemistry
models, what improvements have been made in estimating background concentrations of
oxides of nitrogen, and what are likely background concentrations in the absence of
anthropogenic emissions?
• What information is available on interactions between oxides of nitrogen and copollutants in
the atmosphere that may alter the spatial distributions of oxides of nitrogen?
• To what extent have uncertainties in data, modeling, and satellite measurements been
reduced from the previous reviews?
• What effects have pandemic related lockdowns, increasing environmental temperatures, and
increasing wildland fire activity had on NOx emissions and ambient air concentrations of
NO2 and other oxides of nitrogen?
3.4.2 Exposure
The ISA will evaluate the factors that influence exposure to oxides of nitrogen in ambient
air and the measurement error and other uncertainties associated with extrapolation of ambient
air concentrations to personal exposures to oxides of nitrogen of ambient air origin, particularly
in the context of interpreting results from epidemiologic studies. The following questions will
guide the evaluation of the scientific literature for exposures to NO2 and other oxides of nitrogen.
• How have personal or microenvironmental exposure measurement techniques for oxides of
nitrogen, such as sensors and passive samplers, been advanced in recent years? What
measurement errors are associated with these emerging techniques?
• How have modeling or hybrid modeling techniques such as sub-grid scale modeling within
chemical transport models, air quality dispersion models, and land use regression models
been advanced in recent years? What new information is available regarding modeled
estimates of spatially-resolved (at the micro-, middle-, and neighborhood-scales) ambient air
NO2 and other oxides of nitrogen species concentrations used for exposure assessment?
• To what extent have data fusion approaches that combine ambient air concentrations with air
quality models been recently developed to improve the spatial and temporal resolution of
exposure estimates within a community? What advancements have been made regarding
validation of data fusion and their ability to estimate source attribution for exposures to NO2
or other oxides of nitrogen species?
• How do instrumentation errors (e.g., interference in measurements of ambient air NO2
concentrations from other nitrogen compounds) affect assessing health effects of exposures
to oxides of nitrogen in epidemiologic studies?
3-21
-------
• To what extent do recent studies examine the relationship between near-road oxides of
nitrogen, on-road oxides of nitrogen, and in-vehicle exposures to oxides of nitrogen?
• What new information is available regarding the interaction of indoor oxides of nitrogen with
organic compounds emitted indoors to form organic nitrogen compounds?
• What new information exists regarding characterization of exposure measurement error in
assessment of short-term and long-term exposures to oxides of nitrogen and how that error
influences personal-ambient air exposure relationships? What implications does exposure
measurement error have on inference about epidemiologic associations observed between
oxides of nitrogen and health effects? Do the implications vary according to factors such as
exposure duration, study design, and exposure assessment method?
• What are the relationships between oxides of nitrogen measured at stationary monitoring
sites and personal short-term and long-term exposure? What evidence is available regarding
these relationships in environments near roads or other sources?
• What new information exists regarding exposure to oxides of nitrogen in a multipollutant
context with other gaseous pollutants (e.g., carbon monoxide), particle phase pollutants (e.g.,
ultrafine particles, black carbon, organic carbon, transition metals) generated by traffic or
other combustion sources, or of a mixture of traffic-related pollutants?
- How does information about pollutant co-exposures aid in evaluation of potential
confounders in epidemiologic associations between oxides of nitrogen and health
effects?
- What new information exists about the relationship between NO, NO2, NOx, and NOy
concentrations and indicators of near-source pollution including distance to sources
(e.g., major roadways) and source activity levels (e.g., traffic counts)?
• What new information is available regarding differences in exposure patterns for oxides of
nitrogen and personal-ambient air exposure relationships among various lifestages and
specific groups within populations?
- What new information is available on spatial and temporal trends in exposures to
oxides of nitrogen in ambient air, particularly for groups and lifestages that may be at
increased risk of health effects?
- To what extent is information available characterizing how well the current area-wide
and near-road NO2 monitoring sites represent exposures to populations living near
major roads?
- What implications do potential differences in exposure measurement error have on
inferences about relationships with health effects observed in general population
studies versus those conducted in specific lifestages and groups within the population
(e.g., people with underlying health condition)?
3.4.3 Dosimetry
The ISA will evaluate literature focusing on dosimetry that may underlie the health outcomes
associated with exposure to NO2, NO, and other oxides of nitrogen. These topic areas will be
3-22
-------
evaluated using both human and animal data. The following questions will guide the evaluation
of the scientific literature for dosimetry.
• What are the effects of host factors such as lifestage, sex, pre-existing disease, genetic
background, and physical activity on the uptake of NO2 and/or NO and cellular and tissue
responses that may underlie health effects associated with exposure to oxides of nitrogen?
• What information is available to discern the relative contributions to local NO2 and/or NO of:
(1) ambient air exposures to NO2 and/or NO; (2) dietary consumption of nitrite and nitrate
which undergo transformation to NO; and (3) endogenous formation of NO2 and/or NO?
• What NO2 and/or NO reaction products, including oxides of nitrogen metabolites, can be
found in the cells, tissues, or fluids of the respiratory tract and in the systemic circulation that
may serve as markers of NO2 and/or NO exposure and effect?
• To what extent can the inhalation dosimetry of NO2 and/or NO be extrapolated between
species, qualitatively or quantitatively?
• To what extent is information available on dosimetry of oxides of nitrogen other than NO2
and NO?
3.4.4 Biological Plausibility
The ISA will evaluate literature focusing on modes of action that may underlie the health
outcomes associated with exposure to NO2, NO, and other oxides of nitrogen. These topic areas
will be evaluated using both human and animal data. The following questions will guide the
evaluation of the scientific literature related to biological plausibility.
• What new information is available to inform our understanding of the potential biological
mechanisms underlying responses to NO2 and/or NO exposures, or exposures to other oxides
of nitrogen, at concentrations defined in the ISA to be policy relevant (see Tables 3-1 to 3-3),
with a focus on response pathway(s) and exposure-dose-response relationships?
• What information is available to characterize intra- and inter-individual variability in
biological responses following exposure to NO2, NO, and/or other oxides of nitrogen?
• What are the effects of host factors such as lifestage, sex, pre-existing disease, and genetic
background on cellular and tissue responses, as well as biological mechanisms, that may
underlie health effects associated with exposure to oxides of nitrogen?
• What biological processes, from the molecular to whole organ level, can be qualitatively or
quantitatively compared across species (i.e., human vs. animal)?
• Do interactions with other inhaled pollutants influence the mechanisms underlying the health
effects of NO2, NO, and/or other oxides of nitrogen? If so, how might this information
provide understanding of the potential for a copollutant to act as an effect measure modifier
of health effects related to oxides of nitrogen?
3.4.5 Health Outcomes
The ISA will evaluate health effects that occur following both short- and long-term
exposures to oxides of nitrogen (predominantly NO2) as examined in epidemiologic, controlled
3-23
-------
human exposure, and animal toxicological studies. The health effects evidence will be integrated
with available information on exposure, dosimetry, and biological plausibility to inform the key
ISA conclusions, including causality determinations. The evidence integration will focus on (1)
whether recent studies support or call into question the causality determinations made in the
2016 ISA, (2) whether recent evidence supports causality determinations or other conclusions for
health outcomes not included in the 2016 ISA, and (3) whether new evidence reduces
uncertainties identified in the last review and whether additional uncertainties have been
identified.
Causality determinations from the 2016 ISA are summarized in Table 3-8 below. The
strongest evidence was for relationships between NO2 exposures and asthma exacerbation (short-
term exposures) and asthma development (long-term exposures), likely through the formation of
secondary oxidation products in the respiratory tract (U.S. EPA, 2016, Section 4.3.2.1) and the
induction of oxidative stress, inflammation, allergic responses, and altered immune function
(U.S. EPA, 2016, Figures 1-2 and 4-1). Epidemiologic studies reported associations between
short-term increases in ambient air NO2 concentrations and increased incidence of hospital
admissions and emergency department visits for asthma, increases in respiratory symptoms and
airway inflammation in people with asthma, and decreases in lung function in children with
asthma. The biological plausibility of NCh-induced asthma exacerbation was supported by
controlled human exposure studies that showed increased airway reactivity and allergic
inflammation in adults with asthma exposed at rest to NO2 at ambient air-relevant
concentrations. Support for effects of long-term NO2 exposures came from epidemiologic studies
indicating associations with asthma incidence in children and from experimental studies
characterizing a potential mode of action for NO2. Remaining uncertainties included the lack of
an apparent dose-response relationship in controlled human exposure studies examining NO2-
induced airway reactivity and the potential for epidemiologic associations to be confounded by
co-occurring pollutants (e.g., other traffic-related pollutants). Compared to the evidence for
asthma-related effects, the evidence supporting other health outcomes was subject to greater
uncertainty as reflected in "suggestive" or "inadequate" causality determinations (Table 3-8).
3-24
-------
Table 3-8. Summary of causality determinations from the 2016 ISA organized by health
outcome.
Health Outcome
Causality Determination
Respiratory Effects
Respiratory Effects and Short-Term Exposure
Causal
Respiratory Effects and Long-Term Exposure
Likely to be causal
Cardiovascular Effects and Diabetes
Cardiovascular Effects and Short-Term Exposure
Suggestive
Cardiovascular Effects and Diabetes and Long-Term Exposure
Suggestive
Total Mortality
Total Mortality and Short-Term Exposure
Suggestive
Total Mortality and Long-Term Exposure
Suggestive
Reproductive and Developmental Effects
Fertility, Reproduction, & Pregnancy
Inadequate
Birth Outcomes
Suggestive
Postnatal Development
Inadequate
Cancer
Cancer and Long-Term Exposure
Suggestive
In the current review, the causality determinations from the 2016 ISA will be revisited in
light of recent evidence, and evidence for any additional outcomes will be examined. The
following questions will guide the evaluation of the health effects literature for short-term and
long-term exposure to NO2 and other oxides of nitrogen.
• What do studies across scientific disciplines (i.e., epidemiologic, controlled human exposure,
animal toxicological) indicate about the strength of evidence supporting health effects of
short-term and long-term exposures to NO2 and other oxides of nitrogen? To what extent has
the strength of evidence changed for effects examined in previous reviews (i.e., respiratory
effects, cardiovascular effects and diabetes, reproductive and developmental effects, total
mortality, and cancer)? Does recent evidence support additional health effect outcome
categories of exposures to oxides of nitrogen?
• To what extent have recent studies addressed key uncertainties identified in the evidence in
the 2016 ISA, including uncertainty in the epidemiologic evidence due to potential
confounding by copollutants and potential exposure measurement error, and uncertainty in
the controlled human exposure evidence due to the lack of an apparent dose-response
relationship for airway hyperresponsiveness at NO2 exposure concentrations near those
occurring in ambient air?
• To what extent do recent epidemiologic, controlled human exposure, and animal
toxicological studies provide information on health effects related to specific oxides of
nitrogen including, but not limited to, NO2 and NO?
3-25
-------
How does recent evidence for health effects associated with oxides of nitrogen compare
among healthy individuals, those with pre-existing disease states (e.g., people with asthma or
cardiovascular disease), particular lifestages, or groups characterized by other factors that
potentially modify risk (e.g., genetics, nutritional status)?
Do recent studies provide new information on the range of ambient air and exposure
concentrations over which NCh-related health effects, or effects associated with other oxides
of nitrogen, are most well-characterized?
To what extent does the new scientific evidence support the occurrence of health effects of
exposure to oxides of nitrogen at lower ambient air or exposure concentrations than those
previously demonstrated? What are the uncertainties in the evidence for health effects at
relatively low exposure or ambient air concentrations (e.g., uncertainty in occurrence,
adversity, public health importance of effects)?
What recent evidence is available regarding the shape of concentration-response relationships
between exposure to oxides of nitrogen and various health endpoints? Is there evidence to
support the identification of a discernible threshold below which adverse health effects do
not occur?
What do recent studies indicate regarding the health impacts of reductions in concentrations
of oxides of nitrogen in ambient air (e.g., due to policy intervention) or reductions in
exposures (e.g., due to changes to indoor sources)?
To what extent does new evidence indicate that observed health effect associations are
attributable specifically to ambient air oxides of nitrogen versus other pollutants contained in
the complex ambient air pollution mixture? What information about the independent health
effects of exposure to oxides of nitrogen can be gleaned from the various lines of available
evidence, including epidemiologic, controlled human exposure, and animal toxicological
studies?
How does confounding by other traffic-related copollutants (e.g., particulate matter, carbon
monoxide) or meteorological factors influence relationships observed between health effects
and both short- and long-term exposures to oxides of nitrogen? To what extent do other
factors serve as potential confounding factors in epidemiologic studies (e.g., age,
socioeconomic status (SES), and other exposures such as noise)? In such studies, to what
extent can health impacts due to oxides of nitrogen be separated from the health impacts of
these other factors?
What new information is available to assess the influence of exposure measurement error on
uncertainty in epidemiologic study results?
- How can the influence of exposure measurement error be assessed through the
examination of various study designs, study populations, exposure assessment
methods, spatial and/or temporal variability in ambient air concentrations, spatial
alignment of study population and ambient measurements, and analytical models?
- To what extent can monitored ambient air NO2 concentrations used in epidemiologic
studies reflect oxides of nitrogen exposures in study populations under various
environmental conditions, such as a near-source environment?
3-26
-------
- To what extent can recent data from near-road monitors better characterize or reduce
exposure measurement error in epidemiologic studies?
- What information is available regarding the time-activity patterns of study subjects
including time spent outdoors, spatial distribution of study subjects, and ambient air
monitors?
• What evidence is available regarding the nature of health effects from exposures to ambient
air pollutant mixtures that include oxides of nitrogen? To what extent does the evidence
support attributing these health effects to exposures to NO2 or other oxides of nitrogen,
another ambient air pollutant that is correlated with oxides of nitrogen, or to the pollutant
mixtures that oxides of nitrogen may be representing?
• What new information is available on the health effects of oxides of nitrogen exposures in
populations spending time or living near roads or other sources? To what extent do findings
from experimental studies provide biological plausibility for the effects observed in
epidemiologic studies?
• To what extent does recent evidence indicate that particular exposure patterns, such as
repeated short-term NO2 exposures versus persistent long-term exposures, contribute to
disease development?
Specific Questions Related to Short-Term Exposures
• How do results of recent studies or new interpretations of previous findings expand our
understanding of the relationship between short-term exposure to oxides of nitrogen and
airway hyperresponsiveness or other lung function changes, inflammation, host defense
against infectious disease, respiratory symptoms, and asthma exacerbations?
• What new information is available on the effects of short-term exposure to oxides of nitrogen
on acute cardiovascular events in humans such as myocardial infarction, stroke, increases in
blood pressure, and arrhythmias?
• To what extent do recent studies of short-term exposure to oxides of nitrogen indicate
associations with total mortality or with health effects beyond the respiratory and
cardiovascular systems?
• What is the extent of coherence of findings for effects such as hospital admissions,
emergency department visits, and mortality with changes in lung function, airway
hyperresponsiveness, heart rate variability, and vasomotor function? What other biomarkers
of early effect may be used in the assessment of health effects?
• To what extent does recent information across epidemiologic, controlled human exposure,
and animal toxicological studies on the pattern of exposure to oxides of nitrogen (e.g., peak,
repeated peak, average) provide understanding of the time course for changes in health
effects? What new information is available on time-activity patterns of study subjects such as
time spent outdoors or activity levels that can aid in understanding key aspects of exposure to
or dosimetry of ambient air oxides of nitrogen that are associated with health effects?
• To what extent does recent data from epidemiologic, controlled human exposure, and animal
toxicological studies provide information on health effects related to various short-term
exposure durations (e.g., 1-hour, 24-hour, multi-day)?
3-27
-------
Specific Questions Related to Long-Term Exposures
• How do the results of recent studies expand our understanding of the relationships between
long-term exposure to oxides of nitrogen and chronic respiratory effects manifested as a
reduction in lung function, a reduction in lung development, or morphological changes in the
lung?
• To what extent do recent studies indicate that long-term exposure to oxides of nitrogen
promotes exacerbation and/or development of asthma or other chronic lung diseases,
cardiovascular diseases, and other health conditions?
• To what extent do recent studies find that long-term exposure to oxides of nitrogen contribute
to changes in molecular and cellular processes that could result in adverse cognitive,
behavioral, reproductive, developmental, cancer, or other effects?
• What information is available on the effects of exposures to oxides of nitrogen on health
outcomes in populations living near major roads or working on or near major roads? To what
extent do recent studies disentangle the effects of NO2 and other oxides of nitrogen from co-
occurring traffic-related pollutants?
• What information is available regarding the effect of long-term, low-concentration exposure
to oxides of nitrogen on an individual's sensitivity to short-term but higher concentration
exposures?
• Do recent studies provide information on health effects related to long-term exposure
windows other than annual or lifetime average (e.g., preconception, pregnancy average,
pregnancy trimester average)? What data are available comparing associations of health
effects among various long-term oxides of nitrogen exposure metrics (e.g., annual, seasonal,
pregnancy average)?
3.4.6 At-Risk Lifestages and Populations
The EPA has developed a framework to provide a consistent and transparent basis for
informing the level of confidence for conclusions that specific lifestages or populations may be
at increased risk of pollutant-related health effects according to one of four levels: adequate
evidence, suggestive evidence, inadequate evidence, and evidence of no effect (see Appendix A,
Section A.7.2.3). Conclusions from the 2016 ISA on populations potentially at increased risk are
summarized in Table 3-9 below.25
25 Table 3-9 was extracted directly from the (U.S. EPA, 2016).
3-28
-------
Table 3-9. Summary of evidence for potential increased nitrogen dioxide exposure and
increased risk of nitrogen dioxide-related health effects.26
Evidence Classification
Factor Evaluated
Adequate evidence
Asthma
Lifestage: Children
Older adults
Suggestive evidence
SES: Low SES
Sex: Females
Diet: Reduced antioxidant intake
Inadequate evidence
COPD
Cardiovascular disease
Diabetes
Genetic factors
Obesity
Smoking
Physical activity
Race/ethnicity
Residence in urban areas
Proximity to roadways
The ISA in this review will evaluate an array of factors that characterize potential "at-
risk" populations and lifestages: intrinsic factors (biological factors such as age or genetic
variants), acquired factors (e.g., pre-existing disease), extrinsic factors (nonbiological factors
such as nutritional status, SES), and/or factors affecting dose or exposure (e.g., sex, age, outdoor
activity or work, SES, physical activity). The various factors listed above may influence risk by
increasing exposure, dose, or biological effect at a given dose, and some factors (e.g., SES) may
contribute to risk in multiple ways. In the current review, the ISA will evaluate whether new
information supports or calls into question our previous understanding of the human populations
and lifestages that may be at increased risk for experiencing health effects associated with
exposures to oxides of nitrogen. The following questions will guide the evaluation of the human
health evidence for potential at-risk populations and lifestages.
• Does recent information on the health risks of NO2 exposure, or exposure to other oxides of
nitrogen, support the 2016 ISA conclusions for people with asthma or other pre-existing
respiratory disease, children and older adults, and people with low SES? Is there new
26 Table modified from the 2016 oxides of nitrogen ISA (U.S. EPA, 2016).
3-29
-------
evidence supporting increased risk from exposure to other oxides of nitrogen or new
evidence for additional potential at-risk populations or lifestages?
What new information is available on the health effects in populations spending time near
important sources of NOx emissions (e.g., roads)? To what extent does living, working,
attending school or daycare, or otherwise spending time on or near major roads contribute to
greater overall exposures to oxides of nitrogen and increase the risk of related health effects?
Given the concentration gradients observed for oxides of nitrogen in ambient air with
distance to roads, what information is available regarding the sizes and sociodemographic
characteristics of populations living near major roads? What does recent evidence indicate
regarding the public health importance of NOx emissions from sources other than roadways?
What information is available that provides insight as to whether a potential at-risk
population or lifestage experiences higher exposures or a higher dose of oxides of nitrogen,
has a greater biological response to a given exposure, and/or experiences health effects at
lower exposure concentrations?
What information is available to quantify the magnitude of greater biological response or risk
of health effects associated with exposure to oxides of nitrogen in a particular at-risk lifestage
or population?
Is recent evidence supporting potential at-risk lifestages or populations coherent across
disciplines?
What does new evidence on effect measure modification indicate regarding populations at
increased risk of health effects from exposure to oxides of nitrogen (e.g., young age,
residence near major roads, lower SES, and asthma; older age and pre-existing
cardiovascular disease; preexisting respiratory disease or prior respiratory infection)?
3-30
-------
4 REFERENCES
Hamel, C, Kelly, SE, Thavorn, K, Rice, DB, Wells, GA and Hutton, B (2020). An evaluation of
DistillerSR's machine learning-based prioritization tool for title/abstract screening -
impact on reviewer-relevant outcomes. BMC Medical Research Methodology 20(1): 256.
Howard, BE, Phillips, J, Tandon, A, Maharana, A, Elmore, R, Mav, D, Sedykh, A, Thayer, K,
Merrick, BA, Walker, V, Rooney, A and Shah, RR (2020). SWIFT-Active Screener:
Accelerated document screening through active learning and integrated recall estimation.
Environ Int 138: 105623.
NASEM (2022). National Academies of Sciences, Engineering, and Medicine 2022. Advancing
the Framework for Assessing Causality of Health and Welfare Effects to Inform National
Ambient Air Quality Standard Reviews. The National Academies Press. Washington,
DC. Available at: https://doi.org/10.17226/26612.
Savitz, DA and Forastiere, F (2021). Do pooled estimates from meta-analyses of observational
epidemiology studies contribute to causal inference? Occup Environ Med 78(9): 621-622.
Savitz, DA, Wellenius, GA and Trikalinos, TA (2019). The Problem With Mechanistic Risk of
Bias Assessments in Evidence Synthesis of Observational Studies and a Practical
Alternative: Assessing the Impact of Specific Sources of Potential Bias. Am J Epidemiol
188(9): 1581-1585.
U.S. EPA (1993). Air Quality Criteria for Oxides of Nitrogen. Office of Health and
Environmental Assessment. Environmental Criteria and Assessment Office. Washington,
DC U.S. EPA. EPA-600/8-91-049aF-cF. August 1993. Available at:
https://cfpub. epa.gov/ncea/risk/recordisplay. cfm ?deid=40179.
U.S. EPA (1982). Air quality criteria for oxides of nitrogen. U.S. Environmental Protection
Agency, Environmental Criteria and Assessment Office. Research Triangle Park, NC.
EPA/600/8-82-026.
U.S. EPA (2015). Preamble to the Integrated Science Assessments (ISA). Office of Research and
Development, National Center for Environmental Assessment. Washington, DC. U.S.
EPA. EPA/600/R-15/067. Available at:
https://ordspub.epa.gov/ords/eims/eimscomm.getfile7p downloadid=526136.
U.S. EPA (2016). Integrated Science Assessment for Oxides of Nitrogen-Health Criteria. U.S.
Environmental Protection Agency, Office of Research and Development, National Center
for Environmental Assessment. Research Triangle Park, NC. EPA/600/R-15/068.
Available at: https://cfpub. epa.gov/ncea/isa/recordisplay. cfm ?deid=310879.
U.S. EPA (2019). Integrated Science Assessment (ISA) for Particulate Matter (Final Report).
U.S. Environmental Protection Agency, Office of Research and Development, National
Center for Environmental Assessment. Washington, DC. U.S. EPA. EPA/600/R-19/188.
4-1
-------
December 2019. Available at: https://www.epa.gov/naaqs/particulate-matter-pm-
slandards-i nlegraled-science-assessmenls-current-re view.
U.S. EPA (2020a). Integrated Science Assessment (ISA) for Ozone and Related Photochemical
Oxidants (Final). U.S. Environmental Protection Agency, Office of Research and
Development. Washington, DC. EPA/600/R-20/012. Available at:
https://ordspub.epa.gov/ords/eims/eimscomm.getfile7p download id=540022.
U.S. EPA (2020b). Integrated Science Assessment for Oxides of Nitrogen, Oxides of Sulfur, and
Particulate Matter—Ecological Criteria (Final). U.S. Environmental Protection Agency,
Office of Research and Development, Center for Public Health and Environmental
Assessment. Research Triangle Park, NC. EPA/600/R-20/278. Available at:
https://cfpub. epa.gov/ncea/isa/recordisplay. cfm ?deid=349473.
U.S. EPA (2021). Health Assessment Workspace Collaborative (HAWC) Washington, DC.
U.S. EPA (2023a). Overview of nitrogen dioxide (N02) air quality in the United States,
Updated: June 29, 2023. Washington, DC. Available at:
https://www.epa.gov/system/files/documents/2023-06/N02 2022.pdf.
U.S. EPA (2023b). LLR Online (pre-release Version 007EC800) Washington, DC.
U.S. EPA (2008). Integrated Science Assessment for Oxides of Nitrogen - Health Criteria.
Office of Research and Development, National Center for Environmental Assessment.
Research Triangle Park, NC. U.S. EPA. EPA/600/R-08/071. July 2008. Available at:
http://cfpub. epa.gov/ncea/cfm/recordisplay. cfm?deid=194645.
4-2
-------
APPENDIX A ISA DEVELOPMENT PROCESS
A.l. INTRODUCTION
The Integrated Science Assessments (ISAs) review, synthesize, and evaluate policy-
relevant scientific information27 and reach key science judgments intended to inform the EPA's
reviews of the National Ambient Air Quality Standards (NAAQS). This appendix provides an
overview of the ISA development process, with a focus on human health and exposure.28 It
builds on the process described in the 2015 Preamble to the ISAs (U.S. EPA, 2015) and on
updates and advancements presented in recently completed ISAs (U.S EPA, 2020a; U.S EPA,
2020b). The process presented in this appendix additionally reflects the EPA's consideration of
recommendations made by an ad hoc committee of the National Academies of Science,
Engineering, and Medicine (NASEM) charged with reviewing the current ISA framework for
reaching causality determinations (U.S. EPA, 2015). Those recommendations are presented in
the NASEM report titled Advancing the Framework for Assessing Causality of Health and
Welfare Effects to Inform National Ambient Air Quality Standard Reviews (NASEM, 2022). It
builds on the process described in the 2015 Preamble to the ISAs (U.S. EPA, 2015) and on
updates and advancements presented in recently completed ISAs (U.S EPA, 2020a; U.S EPA,
2020b). The process presented in this appendix additionally reflects the EPA's consideration of
recommendations made by an ad hoc committee of the National Academies of Science,
Engineering, and Medicine (NASEM) charged with reviewing the current ISA framework for
reaching causality determinations (U.S. EPA, 2015). Those recommendations are presented in
the NASEM report titled Advancing the Framework for Assessing Causality of Health and
Welfare Effects to Inform National Ambient Air Quality Standard Reviews (NASEM, 2022). It
builds on the process described in the 2015 Preamble to the ISAs (U.S. EPA, 2015) and on
updates and advancements presented in recently completed ISAs (U.S EPA, 2020a; U.S EPA,
2020b). The process presented in this appendix additionally reflects the EPA's consideration of
recommendations made by an ad hoc committee of the National Academies of Science,
Engineering, and Medicine (NASEM) charged with reviewing the current ISA framework for
27 Policy-relevant scientific information includes the results of scientific studies that inform ISA conclusions such as
causality determinations and conclusions on the populations that may be at increased risk of pollutant-related
health effects, as well as conclusions on other policy-relevant issues. These other issues vary by pollutant and
discipline and often include characterization of concentration-response relationships, strengths and limitations of
various exposure estimates and study designs, impact of potential confounders on health effect associations,
timing of effects, etc.
28 The ISA development process for the welfare effects evidence is discussed in the 2015 Preamble to the ISAs (U.S.
EPA, 2015) and in recent ISAs (U.S EPA, 2020a; U.S EPA, 2020b).
A-l
-------
reaching causality determinations (U.S. EPA, 2015). Those recommendations are presented in
the NASEM report titled Advancing the Framework for Assessing Causality of Health and
Welfare Effects to Inform National Ambient Air Quality Standard Reviews (NASEM, 2022). It
builds on the process described in the 2015 Preamble to the IS As (U.S. EPA, 2015) and on
updates and advancements presented in recently completed IS As (U.S EPA, 2020a; U.S EPA,
2020b). The process presented in this appendix additionally reflects the EPA's consideration of
recommendations made by an ad hoc committee of the National Academies of Science,
Engineering, and Medicine (NASEM) charged with reviewing the current ISA framework for
reaching causality determinations (U.S. EPA, 2015). Those recommendations are presented in
the NASEM report titled Advancing the Framework for Assessing Causality of Health and
Welfare Effects to Inform National Ambient Air Quality Standard Reviews (NASEM, 2022).
As part of the Integrated Review Plan (IRP) for the review of the primary NO2 NAAQS,
this appendix describes the process being applied to develop the ISA for Oxides of Nitrogen -
Health Criteria. It will be subject to a consultation with the Clean Air Scientific Advisory
Committee (CASAC) Oxides of Nitrogen Primary NAAQS Review Panel. Following CASAC
consultation, an updated version of this appendix will be included in the draft ISA for Oxides of
Nitrogen - Health Criteria, where it will be subject to CASAC review and public comment.
Ultimately, the process used in this review to develop the ISA for Oxides of Nitrogen - Health
Criteria will be expanded to include welfare effects and implemented in developing future ISAs.
The sections below present an overview of ISA organization and development (A.2), a detailed
description of the updated ISA development process (A. 3 through A. 8), and a summary of the
quality management process that governs ISA development (A.9).
A.2. OVERVIEW OF ISA ORGANIZATION AND DEVELOPMENT
A.2.1. ISA Organization
The ISAs are organized around a series of detailed, topic-specific chapters29 and an
Integrated Synthesis that draws from those chapters. Chapters provide thorough assessments of
the scientific evidence pertaining to specific topic areas including atmospheric science, exposure
and dosimetry, and human health outcomes. Each chapter contains an evaluation of results from
recent studies building upon key conclusions and evidence presented in previous assessments.
Chapters for each health outcome category (e.g., respiratory effects, cardiovascular effects)
reflect full assessments of the causal nature of relationships between pollutant exposures and
29 In the past, some ISAs have utilized chapters (U.S. EPA, 2013b; U.S. EPA, 2013a), while other ISAs more
recently (U.S EPA, 2020a; U.S. EPA, 2024) referred to topic-specific sections as appendices. In the past, some
ISAs have utilized chapters (U.S. EPA, 2013b; U.S. EPA, 2013a), while other ISAs more recently (U.S EPA,
2020a; U.S. EPA, 2024; U.S EPA, 2020b) referred to topic-specific sections as appendices.
A-2
-------
health effects that result in key science judgments (i.e., causality determinations, see A.7.2.1).
These causality determinations are based on the consideration of various aspects of the evidence,
including consistency and coherence across studies, biological plausibility, and other aspects as
discussed below (A.7.2.2). For human health outcomes, causality determinations also consider
the evidence that certain populations and lifestages may experience larger risks of pollutant-
related effects because they are "at-risk" to those effects and/or because they experience higher
exposures (A.7.2.3). Chapters additionally present targeted evaluations of the evidence on other
pollutant-specific policy-relevant issues to support the summary discussion of those issues in the
Integrated Synthesis. These other issues vary by pollutant and discipline, and often include
conclusions on concentration-, exposure-, and/or dose-response relationships; strengths and
limitations of various exposure estimates and study designs; the impact of potential confounding
factors on health effect associations; and the timing of effects (i.e., lag structure of associations
and/or averaging times of exposures).
Drawing from the detailed assessment of the scientific evidence provided in the chapters,
the Integrated Synthesis provides a concise synopsis of the ISA conclusions and synthesizes the
key findings considered in characterizing pollutant exposures and relationships with health
effects. The Integrated Synthesis typically includes summaries of key information for each topic
area, including information on pollutant-related sources, emissions, and atmospheric science;
exposures and dosimetry; and health effects. The Integrated Synthesis summarizes the ISA
causality determinations, conclusions on the populations and/or lifestages that may be at
increased risk of pollutant-related effects, and conclusions on other key policy-relevant issues,
including but not limited to pre-existing disease(s), genetic factors, lifestage(s), socioeconomic
status (SES), race/ethnicity, sex, urbanicity, proximity to roadways, stress, behavioral factors
(diet, smoking, physical activity); copollutant confounding; and/or exposure/concentration
response.
In addition to the Integrated Synthesis and supporting chapters, the ISAs also generally
include a Preface that summarizes major legal and historical aspects of prior NAAQS reviews,
an Executive Summary written to be accessible to a wide range of audiences, and a Process
Appendix. The Process Appendix describes the approach taken to develop the ISA, typically
including the methods for literature search and review, individual study quality evaluation,
public engagement, and quality assurance considerations as well as documentation for these
activities.
A.2.2. ISA Development
ISAs are developed principally by scientists within the EPA's CPHEA with extensive
knowledge in their respective fields including atmospheric science, exposure science, dosimetry,
A-3
-------
human exposure, animal toxicology, and epidemiology.30 When additional subject matter
expertise is required, the EPA solicits extramural scientists to supplement internal expertise,
thereby ensuring that each ISA provides an accurate reflection of the most up-to-date scientific
knowledge.
The process for developing an ISA begins with a Call for Information published in the
Federal Register that announces the start of a NAAQS review and invites the public to contribute
to the review by submitting potentially relevant research studies in identified subject areas. At
this stage, the public is also given the opportunity to comment on policy-relevant issues to be
addressed in the review. Information and comments received from the public inform the planning
phase of the review, including the development of an IRP. The IRP presents background
information on the NAAQS program in general and on the NAAQS for the pollutant under
review, key policy-relevant science issues for the review, the anticipated process and plans for
developing the ISA and other assessments, and the anticipated schedule for the review.
The EPA consults with the CASAC and solicits public comment on the assessment plans
presented in the IRP. As described further in Section A.8, the CASAC is an independent
committee composed of scientific experts charged with providing advice to the EPA
Administrator on the NAAQS and on the underlying scientific foundation for the standards.
Early in a NAAQS review, the EPA typically supplements the seven-member CASAC with a
pollutant-specific review panel to inform the CASAC's advice. Given the breadth of scientific
and technical information evaluated during NAAQS reviews, CASAC panels reflect a wide
range of expertise. The specific expertise varies across panels, but typically requires members
with expert knowledge of atmospheric science, human exposure, dosimetry, toxicology,
epidemiology, medicine, public health, biostatistics, and risk assessment. Consistent with
NASEM recommendations (NASEM, 2022, p.8), critical disciplines are often represented by
multiple panel members in order to facilitate advice from a range of perspectives.
Following the CASAC consultation on the IRP, the EPA develops the ISA to provide the
scientific foundation for the review. After public release of the draft ISA, the CASAC reviews
the document, recommending revisions as necessary before the final ISA is published. The
remainder of this appendix describes the process involved in developing an ISA. The process is
summarized in Figure A-l and described in detail in Sections A.2 through A.8.31 These sections
30 For reviews that include secondary standards expertise in ecological and other welfare effects is also included.
31 Recognizing that the EPA continually strives to innovate and improve both the process for developing ISAs and
the ISAs themselves, Agency staff routinely monitor advancements in scientific fields related to evidence
integration and weight-of-evidence evaluations, as well as other relevant fields that could improve the ISAs. As a
result, the general process outlined in Figure A-l and discussed in Sections A.2 through A.8 may evolve over
time, as has been the case with the framework for ISA development described in the 2015 Preamble (U.S. EPA,
A-4
-------
describe the approaches used for defining the scope of the ISA (A. 3), conducting the literature
search and identifying potentially relevant studies (A.4), evaluating individual study quality
(A. 5), extracting data from relevant studies (A.6), drafting ISA (A.7), obtaining scientific and
public review of the draft ISA (A.8), revising and finalizing the ISA (A.9), and quality
management (A.9).
2015). This strategy is consistent with the NASEM recommendation to monitor the scientific literature to
determine if emerging approaches to research synthesis and evidence integration "might be adapted to improve
Integrated Science Assessment causal determinations" (NASEM, 2022, p. 132).
A-5
-------
Source: Modified from Figure II ofthe Preamble to the Integrated Science Assessments U.S. EPA, 2015.
Figure A-l. The general process for Integrated Science Assessment development.
A-6
-------
A.3. DEFINING THE SCOPE OF THE ISA
As noted above, the ISAs include a series of topic-specific chapters that provide detailed
assessments of the policy-relevant scientific evidence. These chapters focus on integrating
policy-relevant studies that have become available since the previous ISA with older studies
evaluated in earlier assessments. Emphasis is placed on new and emerging information and
studies that address scientific uncertainties and limitations identified in prior reviews. Important
older studies (i.e., those included in the previous ISAs) may be discussed in detail to reinforce
key concepts and conclusions and are open to reinterpretation considering more recent data.
Older studies may be the primary focus of the ISA in some subject areas or scientific disciplines
where research efforts have subsided and where these older studies remain the definitive works
available in the literature. The ISAs emphasize studies that are most relevant for ambient air-
related exposures, including those examining pollutant concentrations that reflect the range of
ambient air-related exposures across microenvironments. Studies examining higher exposure
concentrations (i.e., one to two orders of magnitude greater than ambient air concentrations) may
be included if they provide evidence of the potential biological mechanism(s) for an observed
effect. Each ISA identifies specific literature scoping criteria to focus the assessment on the most
relevant studies. General criteria that guide the ISA scoping decisions for studies of human
health effects, atmospheric science, exposure, and dosimetry are discussed below.32
A.3.1. Health Effects Studies
To be considered for inclusion in the ISA, relevant health studies must have undergone
scientific peer review and have been published or accepted for publication within the predefined
literature search cutoff dates. Studies can be considered for inclusion if they present original
research or new analyses of existing data. To further refine criteria for identification of
potentially relevant health studies, the ISAs use discipline-specific population, exposure,
comparison, outcome, study design (PECOS) statements. PECOS statements help to define the
objectives of the assessment and establish relevance criteria that should be met to consider a
study for inclusion in the ISA, thereby facilitating identification of the potentially relevant
literature. To focus on exposure concentrations most relevant to humans, an upper limit is
sometimes used for exposure concentrations tested in epidemiologic and animal toxicological
studies. All studies that meet the PECOS criteria during title and abstract and full text review
undergo study quality evaluation (A.4 and A.5).
32 For ISAs that include welfare effects, analogous scoping criteria are developed for scoping decisions on welfare
effects studies (e.g., U.S EPA, 2020a).
A-7
-------
Meta-analyses may be considered for inclusion in an ISA to the extent that (1) the time
period from which the underlying literature is drawn is relevant to the ISA time period (e.g., so
that the pooled effect estimates are reflective of current understanding of the literature); (2) the
underlying literature would meet the PECOS criteria; and (3) the underlying literature would be
policy-relevant. Studies with pooled effect estimates (e.g., meta-analyses and pooled analyses)
are considered with caution when evaluating causality so as not to misrepresent and perpetuate
errors from approximating estimates (Savitz and Forastiere, 2021, Savitz et al., 2019). Review
articles that are out of scope or that are limited to summarizing and interpreting existing studies,
without presenting new information or analyses, are not considered for inclusion in the ISA.
PECOS statements are informed by the body of evidence from the previous ISAs and Air
Quality Criteria Documents (AQCDs) and by expert knowledge of the policy-relevant scientific
issues. Generic examples of PECOS statements are provided below for epidemiologic (see Table
A-l), controlled human exposure (see Table A-2), and animal toxicological (see Table A-3)
studies. In each ISA, these generic PECOS statements may be modified as appropriate for
specific pollutants and health outcome categories.
A-8
-------
Table A-l. Generic PECOS statement to define the criteria and framework for
identifying relevant epidemiologic studies.
Exposure Duration
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Short-term exposure
Population (P): Any human population, including populations or lifestages that might
be at increased risk;
Exposure (E): Short-term exposure to the pollutant(s) under evaluation (e.g., up to 30
days) at concentrations relevant to ambient air in the U.S.;
Comparison (C): Per unit increase in pollutant exposure (e.g., in ppb or pg/m3) or
populations exposed to lower concentrations of pollutant compared to higher
concentrations (e.g., categorical comparisons between different exposure metric
quantiles);
Outcome (0): Change or difference in risk (incidence/prevalence) of health outcome
per increase in exposure;
Study Design (S): Epidemiologic studies, such as panel, case-crossover, time-series,
case-control studies, cohort, cross-sectional studies, and quasi-experimental with
appropriate timing of exposure for the health outcome of interest.
Long-term exposure
Population (P): Any human population, including populations or lifestages that might
be at increased risk;
Exposure (E): Long-term exposure to the pollutant(s) under evaluation (e.g., longer
than 30 days) at concentrations relevant to ambient air in the U.S.;
Comparison (C): Per unit increase in pollutant exposure (in ppb or pg/m3) or humans
exposed to lower concentrations compared to higher concentrations (e.g., categorical
comparisons between different exposure metric quantiles) within or across
communities;
Outcome (0): Change or difference in risk (incidence/prevalence) of health outcome
per increase in exposure;
Study Design (S): Epidemiologic studies, such as panel, case-crossover, time-series,
case-control studies, cohort, cross-sectional studies, and quasi-experimental, with
appropriate timing of exposure for the health endpoint of interest.
A-9
-------
Table A-2. Generic PECOS statement to define the criteria and framework for
identifying relevant controlled human exposure studies.
Exposure Duration
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Single or repeated
short-term exposures
Population (P): Human volunteers enrolled in controlled exposure studies, including
volunteers representing populations or lifestages that might be at increased risk of
pollutant-related health effects;
Exposure (E): Controlled pollutant exposure - pollutant exposures must be controlled
by the experimenters and not simply a measure of ambient air or occupational
exposure;
Comparison (C): An appropriate control exposure to filtered air or room air for each
study participant or an appropriately matched comparison group exposed to filtered air
or room air;
Outcome (0): Outcomes of interest are those that relate to human health, including
effects on the respiratory system, cardiovascular system, immune system, nervous
system, diabetes, cancer, or reproduction and development. Effects of interest include
changes in indicators or measures of physiological function, health-relevant
biomarkers, and organ structure. Effects can be directly measured in exposed study
participants or in cells, tissues, or fluids isolated from study participants;
Study Design (S): Studies that perform controlled human exposures meeting the
above criteria or that analyze data from previously conducted controlled human
exposures (e.g., reanalysis, meta-analysis).
A-10
-------
Table A-3. Generic PECOS statement to define the criteria and framework for
identifying relevant animal toxicological studies.
Exposure Duration
Population, Exposure, Comparison, Outcome, Study Design (PECOS)
Short-term exposure
Population (P): Laboratory nonhuman mammalian animal species (e.g., nonhuman
primate, mouse, rat, guinea pig, minipig, rabbit, cat, dog) of any lifestage including
models of increased susceptibility;
Exposure (E): Short-term (up to 30 days) exposure to relevant pollutant concentrations
(i.e., one or two orders of magnitude greater than ambient air concentrations);
Comparison (C): An appropriate control group exposed to clean air (room air, filtered
air) control;
Outcome (0): Outcomes of interest are those that relate to human health, including
effects on the respiratory system, cardiovascular system, immune system, nervous
system, diabetes, cancer, reproduction and development, or other human health effects
(e.g., nervous system). Effects of interest include changes in indicators or measures of
physiological function, health-related biomarkers, and organ structure. Effects can be
directly measured in exposed animals or in cells, tissues, or fluids isolated from
animals;
Study Design (S): Controlled exposure studies of animals in vivo meeting the above
criteria.
Long-term exposure
Population (P): Laboratory nonhuman mammalian animal species (e.g., nonhuman
primate, mouse, rat, guinea pig, minipig, rabbit, cat, dog) of any lifestage including
models of increased susceptibility;
Exposure (E): Long-term (longer than 30 days) exposure to relevant pollutant
concentrations (i.e., one or two orders of magnitude greater than ambient air
concentrations);
Comparison (C): An appropriate control group exposed to clean air (room air, filtered
air) control;
Outcome (0): Outcomes of interest are those that relate to human health, including
effects on the respiratory system, cardiovascular system, immune system, nervous
system, diabetes, cancer, reproduction and development, or other human health effects
(e.g., nervous system). Effects of interest include changes in indicators or measures of
physiological function, health-related biomarkers, and organ structure. Effects can be
directly measured in exposed animals or in cells, tissues, or fluids isolated from
animals;
Study Design (S): Controlled exposure studies of animals in vivo meeting the above
criteria.
A.3.2. Atmospheric and Exposure Sciences Studies
To be included in the ISA, relevant atmospheric science and exposure studies must have
undergone scientific peer review and been published or accepted for publication within the
predefined literature search cutoff dates.33 Consistent with the health evidence, the ISA uses
discipline-specific scoping statements to identify potentially relevant atmospheric and exposure
science studies. These scoping statements include consideration of pollutant sources, transport
33 In the atmospheric science chapter, results of published studies are often supplemented by targeted air quality
analyses conducted by the EPA.
A-ll
-------
and transformation, exposure/extent, and measurement and modeling (STEM). The STEM
statements define the objectives of the atmospheric science and exposure evidence assessments
and establish criteria that should be met to consider a study for inclusion in the ISA. A study
meeting any of the four aspects of the STEM statement could be considered for inclusion in the
atmospheric science or exposure assessment.34
For atmospheric science, flexibility is built into STEM statement application by
maintaining a broad scope for subject areas with few studies identified, but dynamically
narrowing the scope as study selection progresses. This is accomplished by iteratively adjusting
relevance criteria to account for scientific findings, geographic similarity to the United States,
representativeness or diversity of environmental conditions, quality of measurement or modeling
method used, or other refinements if the number of identified studies becomes impractical to
include.
For most subject areas, study selection is carried out by application of the STEM
statement to individual studies. However, if a very large number of relevant studies are identified
for a subject area, alternative search strategies can be applied to identify only the most relevant
and influential studies. This general approach to focusing on the most influential and relevant
atmospheric science studies is consistent with the EPA's approach to inclusion of such studies in
the Integrated Risk Information System (IRIS) and Provisional Peer Reviewed Toxicity Value
(PPRTV) programs (Thayer et al., 2022).
The STEM statements for a specific ISA are informed by the body of evidence from the
previous ISAs and AQCDs and by expert knowledge of the relevant scientific literature. These
scoping statements can serve to highlight well-established areas of research as well as gaps in the
literature and uncertainties from previous assessments (U.S EPA, 2020a; U.S. EPA, 2016).
Importantly, application of STEM statements is consistent with current best practices for
reporting or evaluating health science data as recommended by the NASEM (NASEM, 2022)
Generic STEM statements for atmospheric science and exposure studies are provided below (see
Table A-4 and Table A-5, respectively). These generic statements provide a scoping framework
that can be modified as appropriate for specific ISAs.
34 This contrasts with the PECOS statements used for health effects studies that require all of the listed criteria to be
met.
A-12
-------
Table A-4. Generic STEM statement to define the criteria and framework for
identifying relevant atmospheric science studies.
Statement
Description
Source (S)
Studies reporting quantitative emissions estimates as well as observations of physical
and chemical characteristics that add to our understanding of pollutant sources and
emissions.
Transport and
Transformation (T)
Studies investigating atmospheric fate and transport, transformation, and deposition
processes, including transport of air pollutants at various scales (i.e., national/global,
regional, urban, neighborhood), atmospheric chemical transformations in the
atmosphere, and estimates of atmospheric deposition that add to our understanding
of atmospheric processes.
Exposure/Extent (E)
Studies reporting observations and estimates of ambient air concentrations and their
trends, including spatial variability on various scales (i.e., national/global, regional,
urban, neighborhood); temporal trends such as diurnal, weekday/weekend, seasonal,
and long-term trends; or characteristics, such as composition or relationship with
atmospheric properties that provide up to date concentrations and estimates or add to
our understanding of spatiotemporal concentration trends.
Measurement and
Modeling (M)
Studies describing methods of measurement by federal reference and equivalency
methods, satellite estimates, low-cost sensor estimates, or research methods, and
modeling techniques (e.g., chemical transport modeling) for characterizing pollutant
concentrations in ambient air, including the evaluation of measurement principles and
modeling assumptions, examination of potential bias and uncertainties, and method
intercomparisons that are relevant to the NAAQS or to studies in this ISA.
A-13
-------
Table A-5. Generic STEM statement to define the criteria and framework for
identifying relevant exposure science studies.
Statement
Description
Source (S)
Emissions from outdoor or indoor sources (e.g., traffic or cookstove emissions),
anthropogenic sources or natural sources (e.g., industrial emission or wildfire).
Transport and
Transformation (T)
Atmospheric and environmental processes, including the transport of air pollutants at
various scales (i.e., national/global, regional, urban, neighborhood, middle, micro
scales, and microenvironments), and advances in chemical transformations and
deposition from the atmosphere (e.g., photochemical reactions) and
microenvironments (e.g., indoor chemistry).
Exposure/Extent (E)
Exposure levels characterized by various surrogates (e.g., ambient air measurement,
near-source measurement, microenvironmental measurement, personal exposure
measurement and modeling, biomarkers of exposure) and exposure determinants
(i.e., factors leading to differential exposures, such as proximity to sources, activity
patterns, and socioeconomic status), including characterizing concentrations and
spatiotemporal temporal trends of various exposure surrogates and examining
populations experiencing elevated exposures or the exposure patterns (e.g., exposure
level, duration, and frequency) experienced by populations identified in health studies
as being at increased risk of effects.
Measurement and
Modeling (M)
Measurement (e.g., federal reference and equivalent methods, passive samplers,
remote sensing, and biomonitoring approach) and modeling techniques (e.g.,
Stochastic Human Exposure and Dose Simulation) characterizing ambient air,
indoor/microenvironmental air, and personal exposures, including the evaluation of
measurement principles and modeling assumptions, examination of potential bias and
uncertainties, and comparison of different techniques.
A.4. APPROACH TO LITERATURE SEARCH AND LITERATURE
SCREENING
A.4.1. Literature Search
The EPA uses a multipronged approach to identify scientific studies that may be relevant
for inclusion in a particular ISA. As described below, this approach employs both targeted and
broad literature search strategies (e.g., forward citation searches and keyword searches,
respectively). It also includes consideration of studies identified by public commenters, studies
recommended during the peer input workshop or during the CASAC review/consultation process
(A.7 and A.8), and studies identified by EPA scientists based on professional expertise.
One targeted approach to literature identification that may be employed is forward
citation searching. For this approach, a set of relevant published references are identified as a
seed set and then databases (e.g., PubMed and Web of Science) are queried to identify recently
published literature that has cited any of the references in the seed set. The seed set for the new
ISA literature search is comprised of data-containing peer-reviewed references cited in the
previous ISAs or AQCDs. Each topic area (e.g., atmospheric science, epidemiology) has its own
A-14
-------
seed set(s). Because the seed set is highly relevant to the topic of interest, this targeted approach
to reference identification can be more precise than keyword searches, and it further allows for
relevance ranking based on the number of references in a bibliography that match references in
the seed set.
A broader approach to literature identification is the use of keywords to query literature
databases. When this approach is applied, a set of broad keywords is carefully curated for each
pollutant and/or topic and then used to search relevant databases (e.g., PubMed and Web of
Science). The results of the broad keyword search can then be further categorized by scientific
discipline (e.g., epidemiology, toxicology) using automatic topic classification. This step
employs machine learning where positive and negative seed references for a particular discipline
are used to train an algorithm to identify discipline-specific references based on word use and
frequency in titles and abstracts. This method varies in effectiveness across disciplines due to the
broad range of topics and variability in term usage in some evidence bases. Discipline-specific
keyword searches (e.g., related to pollutant sources, atmospheric science, exposure assessment,
dosimetry/toxicokinetics, epidemiology, controlled human exposure, or animal toxicology) may
also be used to capture literature pertinent to the pollutant of interest in citation databases (i.e.,
PubMed and Web of Science). Results of the keyword search and forward citation search are
combined and deduplicated to form the set to be screened as described below.
A.4.2. Literature Screening
Studies identified during the literature search described above are evaluated for potential
relevance using a multipronged literature screening approach designed to maximize efficiency
while simultaneously ensuring relevant studies are identified. Studies are initially evaluated for
potential relevance by comparing their titles and abstracts to the discipline-specific scoping
criteria defined by PECOS or STEM statements (A.3). Because the number of criteria pollutant-
related studies identified in initial literature searches can be very large (e.g., typically hundreds
of thousands of studies across disciplines and outcomes), machine ranking literature review
software tools (e.g., DistillerSR [Ottawa, Ontario, Canada] Hamel et al., 2020, SWIFT-Active
Screener [Sciome, RTP, NC, USA] Howard et al., 2020, Living Literature Review [U.S. EPA,
Durham, NC, USA] (U.S EPA, 2023a) are employed to maximize screening efficiency. Final
study inclusion and exclusion decisions reached while using these tools are documented in the
HERO database.35 Studies that appear to meet the ISA scoping criteria based on the title and
abstract screen, together with studies that cannot be definitively identified as out of scope, are
retained for further evaluation of the full text. For atmospheric science, a second round of title
and abstract screening is often required to accommodate the iterative revision of STEM criteria
35 https://hero.epa.gov
A-15
-------
or to further restrict the form of the STEM statement if the initial number of studies is too large
for effective full text screening.
Following title and abstract screening, ISA-relevance is further evaluated by comparing
the full text of documents to the predefined PECOS and STEM scoping criteria described above.
Studies not meeting the scoping criteria are excluded from further consideration and are tagged
in HERO as not PECOS- or STEM-relevant. Retained studies are tagged in HERO as
"considered for inclusion" in the ISA.
Studies identified as PECOS- or STEM-relevant are further evaluated for individual
study quality as described below (A. 5). The approach described below can be adapted as
appropriate for an individual ISA, and the specific study quality criteria for a given assessment
are informed by expert knowledge of the literature from previous reviews and by
recommendations from a peer input workshop, the CASAC, and members of the public during
initial phases of planning and ISA development. Studies that are determined to meet scoping
criteria and that are judged of sufficient quality are tagged in HERO for inclusion in the ISA.
Studies that do not meet study quality criteria are tagged in HERO for exclusion.
Results of the literature search and screening efforts and the evaluation of individual
study quality can be captured in flow diagrams that document the number of references identified
from each database searched (i.e., PubMed and Web of Science), the number of references
evaluated in each screening step, and general reasons for reference exclusion (see Figure A-2 for
an example of anticipated formatting). The specific types of studies selected for inclusion in an
ISA may additionally be visualized using evidence maps (see Figure A-3 for anticipated
content). Evidence maps are commonly used to visualize evidence derived from systematic
literature search and screening approaches. Evidence maps can also be used to identify emerging
areas of research, knowledge gaps, and to inform staffing needs.
A-16
-------
Figure A-2, Literature flow diagram. Studies are initially evaluated for potential
relevance by comparing their titles and abstracts to the discipline-specific scoping
criteria defined by PECOS or STEM statements. Following title and abstract
screening, the ISA relevance is further evaluated by comparing the full text of
documents to the predefined PECOS and STEM scoping criteria. Studies not meeting
the scoping criteria are excluded from further consideration and tagged in HERO as
not PECOS or STEM relevant. Retained studies are tagged in TIERO as "considered
for inclusion" in the ISA. Detailed literature screening results and include/exclude
decisions can be found 011 the ISA-specific HERO project page
(https:7/hero. epa.gov). CAS AC, Clean Air Scientific Advisory Committee; Ti/Ab,
title and abstract; WoS, Web of Science.
A-17
-------
Health Outcome
* o
o<5 si
o3
O)
QC
i/i
GS Wi
>
+-»
u £
3 Q.
¦g °
il a>
a >
x
&o
o
o
c
3
E
E
c
-------
Effective evaluation of study quality relies most fundamentally on transparency,
including the clear reporting of key data, assumptions, methods, formulas, input parameters,
QA/QC procedures, statistical models/coding, reasoning process, and limitations. Such
transparency can increase confidence in individual study results (NASEM, 2022, p. 6-7). The
sufficiency of study documentation and reporting can be evaluated based on (1) whether a person
with a general knowledge of the research area can understand the described approach, (2)
whether the study can, in principle, be independently verified/replicated based on the reported
methodology, and (3) whether the limitations can be characterized based on the reported
assumptions and uncertainties (WHO, 2008). For studies of health effects in particular, the
specific attributes considered in evaluating study quality include (1) study design, (2) study
population or test model, (3) pollutant, (4) exposure assessment or assignment, (5) outcome
assignment evaluation, (6) potential confounding (i.e., for epidemiology), and (7) statistical
methodology. These attributes, described further below, are informed by existing EPA guidelines
related to cancer, neurotoxicity, reproductive toxicity, developmental toxicity, and exposure
assessment (U.S EPA, 2005; U.S EPA, 1998; U.S EPA, 1996; U.S EPA, 1991; U.S EPA,
2019b); and are consistent with current best practices for reporting or evaluating health science
data as recommended by the NASEM (NASEM, 2022). The ISA assessment of the scientific
quality of individual studies is framed by the following general questions:
• Were the study design, study groups, methods, data, and results adequately justified and
clearly presented in relation to the study objectives to allow for study evaluation,
including evaluating underlying assumptions and study limitations?
• Are the air quality, exposure, and/or dose metrics sufficiently representative of, or
pertinent to, ambient air and are they adequately documented?
• Do the analytical methods used in the study provide adequate sensitivity and precision?
• Are the statistical analyses appropriate, properly performed, and appropriately interpreted?
• For studies of health effects specifically:
- Were the study populations, participants, or organism model systems
appropriately selected and sufficiently well-defined to allow for meaningful
comparisons between study or exposure groups?
- Are the effect measurements meaningful and valid?
- Were likely confounders controlled for and modifying factors examined in the
study design and/or statistical analysis?
- Were the studies conducted with appropriate oversight by ethics boards or
committees (e.g., documenting approval from an Institutional Review Board (IRB)
for human studies or from an Institutional Animal Care and Use Committee
(IACUC) for animal studies)?
A-19
-------
Answers to these questions, and the presence or absence of particular study attributes, are
not used as a checklist in evaluating study quality, and the identification of uncertainties that may
influence study interpretation does not necessarily lead to a conclusion that the study should be
excluded from the ISA. Rather, strengths and limitations identified during study quality
evaluation provide context for the broader process of evidence integration and inform the
development of ISA conclusions such as causality determinations. Final study inclusion and
exclusion decisions reached as a result of study quality evaluation are documented in the HERO
database or HAWC.36 The sections below provide a general discussion of the study attributes
considered in the ISA's evaluation of individual study quality for epidemiology (A. 5.1),
controlled human exposure (A5.2), animal toxicological (A.5.3), and atmospheric science and
exposure science studies (A. 5.4). These attributes can be adapted for specific IS As, as
appropriate.
A.5.1. Epidemiology
Epidemiologic studies report associations between pollutant exposures and a spectrum of
health effects (e.g., changes in heart or lung function) and outcomes (e.g., hospital admissions,
mortality) in the general population and in groups potentially at increased risk of pollutant-
related effects. When integrated with other lines of scientific evidence (i.e., controlled human
exposure and animal toxicology), epidemiologic studies can provide information that supports
the elucidation of the exposure to effect continuum, including the lag structure of associations
(i.e., lag time between exposure and effect) and the concentration-response relationships across
the range of ambient air pollutant concentrations experienced by populations. When evaluating
epidemiologic studies, inference is stronger for studies that clearly describe the primary and any
secondary aims of the study, or specific hypotheses being tested. In evaluating the quality of
epidemiologic studies, the IS As additionally consider the appropriateness of the (1) study design,
(2) study population, (3) pollutant, (4) exposure assessment or assignment, (5) outcome
assignment evaluation, (6) potential confounding, and (7) statistical methodology. Each of these
study attributes is discussed below.
A.5.1.1. Study Design
Study designs that focus on environmental exposures, such as air pollution, vary
depending on the data available and the exposure duration examined, but broadly can encompass
cohort, cross-sectional, time-series, case-crossover, panel, and ecological designs. In addition,
quasi-experimental designs have been used to mimic randomized experiments and reduce
potential bias. Such studies can be informative in the assessment of causality and can include
36 https://hero.epa.gov and https://hawc.epa.gov/
A-20
-------
intervention studies, studies of natural experiments, and accountability studies. Across designs,
studies with larger sample sizes and those conducted over longer time periods reduce selection
bias among the study population and increase generalizability, and such studies are therefore
considered to produce more reliable results than studies with smaller sample sizes. Because
multicity studies examine associations of health effects across cities and in larger and broader
populations, the ISAs generally emphasize multicity studies when available.
Air pollution epidemiologic studies examining short-term exposure (i.e., up to 30 days)
employ time-series, case-crossover, or panel designs to evaluate the relationships between short-
term (e.g., day-to-day) changes in air pollution exposures and specific health outcomes at the
population level (e.g., mortality, hospital admissions, or emergency department visits). Some of
these study designs examine temporal relationships (e.g., time-series studies) and others allow
individuals or populations to serve as their own controls (e.g., a panel study with repeated
measurements or a case crossover study), which make them less prone than cross-sectional
studies to confounding by factors that differ between individuals (e.g., SES, age, smoking
status). Therefore, among epidemiologic studies of short-term exposures, case-crossover and
panel designs in which individuals or populations serve as their own controls are emphasized.
Inference from these study designs is stronger when they include appropriate controls for factors
that vary temporally and that are correlated with the exposure of interest and the health outcome
(e.g., daily temperature, day of the week, seasonal illness rates, etc.). Panel studies can be
particularly informative if they employ clearly defined scripted activity patterns (e.g., study
participants walk the same routes at the same times of day), measure personal exposures to the
ambient air pollutant, and measure outcomes at consistent, well-defined lags after exposures.
Air pollution epidemiologic studies examining long-term exposures (i.e., longer than 30
days) and specific health outcomes at the population level (e.g., disease incidence or progression,
mortality) often employ cross-sectional or ecological study designs, which measure exposures
and outcome(s) at a single point in time, or prospective cohort designs, which assess exposure
before the outcome(s) occur. Because prospective designs can better inform the temporality of
the relationship between long-term pollutant exposures and outcomes, inference is generally
stronger for prospective cohort studies, including case-control studies nested within a prospective
cohort (e.g., for rare diseases), than for cross-sectional studies, ecological studies, or case-control
studies not nested within a prospective cohort. Long-term exposure studies that do not employ a
prospective cohort design can have uncertainties related to potential reverse causality (i.e., cross-
sectional studies), the appropriateness of the control group, and the validity of inference about
individuals from aggregated or group-level data (i.e., ecologic studies). Because studies of long-
term exposures evaluate associations based on spatial and/or temporal variation, inference from
A-21
-------
cohort studies is stronger when they address the potential for confounding by factors that vary
spatially across populations (e.g., smoking rates, SES, etc.) and temporally (e.g., trends for time).
Additionally, some epidemiologic studies employ study designs and/or statistical
approaches that, compared to traditional regression models, are intended to better account for
confounders and to mimic randomized experiments and reduce potential bias. In the peer-
reviewed literature, these epidemiologic studies are often referred to as causal inference
studies,37 studies that use causal modeling methods, or quasi-experimental studies. Examples of
such statistical approaches include, but are not limited to, general propensity scores, inverse
probability weighting models, and instrument variables. When evaluating studies that employ
these alternative methods for confounder control, the ISA uses an approach consistent with that
used for evaluating traditional regression models. As described further below (Section A.5.1.8),
this approach considers the clarity, plausibility, internal consistency, and validity of study
assumptions; the degree to which confounding has been appropriately considered and addressed;
and the degree to which statistical uncertainties are appropriately characterized and quantified.
A.5.1.2. Study Population
The population evaluated in an epidemiologic study can impact the strength of inference
drawn from that study. The bullets below describe the attributes of epidemiologic study
populations considered as part of the ISA study quality evaluations.
• Representativeness: There is greater confidence in results for study populations that are
recruited from, and representative of, the target population. Selection bias can influence
the results in either direction and may not affect the internal validity of results but rather
reduce the generalizability (external validity) of findings to the target population.
• Inclusion and exclusion criteria for participants: Clearly specified criteria for
including and excluding subjects and the reporting of baseline information on participants
that are lost to follow-up can aid in evaluating potential selection bias.
• Participation rates: Studies with high participation and low loss to follow-up over time
that is not dependent on exposure or health status are considered to have low potential for
selection bias.
• Health conditions: For studies that evaluate populations with underlying health
conditions, independent clinical assessment of the health condition is considered to be the
most reliable approach to identifying the study population, though self-report of
physician diagnosis could also be considered a reliable approach for some conditions
(e.g., respiratory and cardiovascular diseases). Comparison groups with and without an
underlying health condition are more informative if groups are from the same source
population.
37 To prevent confusion with the main scientific conclusions presented within an ISA (i.e., the causality
determinations), this document refers to such studies as employing alternative methods for confounder control.
A-22
-------
A.5.1.3. Pollutant
The primary NAAQS are set to protect public health against exposures to the "criteria"
air pollutants. To inform decisions on the NAAQS, each ISA focuses on studies that evaluate the
effects of exposures to the particular criteria pollutant(s) under evaluation. Thus, emphasis is
placed on epidemiologic studies that evaluate associations with individual criteria pollutants, or
with components of a criteria pollutant that are particularly relevant for reviewing the NAAQS
(e.g., fine particulate matter (PM2.5) in the case of the criteria pollutant particulate matter).
Studies only reporting associations with undefined mixtures (e.g., diesel exhaust) or their
surrogates (e.g., distance to roadway) are generally not used to inform ISA conclusions.
Studies of mixtures can be informative in an ISA if health effect associations with the
particular criteria pollutant under evaluation are also presented. Such studies reflect real-world
exposures and can provide insight into combinations of effects or potential modification of the
criteria pollutant's effect by the broader pollutant mixture. Inference in the context of the ISA is
generally strongest from mixtures studies that present (1) independent effect estimates for the
pollutant of interest that adequately control for, or otherwise address, the potential for
confounding by co-occurring pollutants (Section A.5.1.6, below) and (2) formal analyses
examining how co-occurring pollutants and/or the broader pollution mixture may modify the
independent effects of the pollutant of interest. In contrast, studies only presenting associations
with mixtures (i.e., no independent effect estimates for the criteria pollutant under evaluation) are
typically not policy-relevant in the context of the ISA.
A.5.1.4. Exposure Assessment or Assignment
The primary NAAQS are intended to protect the public health against effects of
exposures to criteria pollutants in ambient air. However, information about ambient air exposures
is rarely available for individual study participants. Often, epidemiologic studies use surrogates
for personal exposures. For most criteria pollutants, ambient air concentrations may be used.
Other exposure surrogates commonly used in epidemiologic studies include indoor pollutant
concentrations, total personal exposures, and biomarker concentrations. The ISAs emphasize
epidemiologic studies with clear justifications for their exposure surrogates (e.g., in terms of
capturing spatiotemporal variation in exposures), as well as studies that compare results across
multiple valid exposure assessment methods. The bullets below describe attributes of
epidemiologic study exposure estimates that are considered as part of the ISA study quality
evaluations.
• Spatiotemporal variability: Ambient air concentrations of criteria pollutants vary
spatially and temporally. Exposure estimates typically have smaller biases and smaller
reductions in precision for pollutants with less spatial variation, compared with spatially
heterogeneous pollutants (Sheppard et al., 2005). For pollutants with lower spatial
A-23
-------
variability, exposure measurement error typically causes health effect estimates to be
underestimated (Zeger et al., 2000). In these situations, biases and decreases in the
precision of the association (i.e., wider 95% CIs) tend to be relatively small (Rothman
and Greenland, 1998; Zeger et al., 2000). In the ISAs, studies with validated exposure
estimation methods that capture spatiotemporal variability appropriate for the study
design and location carry greater weight, especially for studies involving pollutants with
relatively large spatiotemporal variabilities (e.g., traffic-related pollutants, such as NO2).
• Exposure duration: Regardless of the approach to estimating pollutant exposures,
inference is stronger when the exposure duration corresponds with the time course for
physiological changes in the outcome (e.g., short-term exposures can result in respiratory
symptoms while multi-year exposures may be appropriate for cancer and other diseases
elicited by long-term exposures).
• Lag Time: When examining outcomes associated with short-term exposures, evaluation
of the evidence may focus on specific lags between exposure and outcome based on the
evidence related to the health outcome being analyzed. When the existence of a lag
between exposure and outcome is supported, the following hierarchy is used in the
process of selecting results from individual studies of short-term exposures:
(1) Distributed lag models;
(2) Average of multiple days (e.g., 0-2);
(3) If a priori lag days were used by the study authors, these are the effect estimates
presented; or
(4) If a study focuses on only a series of individual lag days, expert judgment is applied
to select the appropriate result to focus on considering the time course for
physiologic changes for the health effect or outcome being evaluated.
• Source of exposure estimates: Epidemiologic studies often estimate exposures to
ambient air pollutants using fixed-site ambient air monitors, remote sensing approaches,
modeling, or using a combination of inputs from multiple sources. For some pollutants,
studies use biomarker concentrations as estimates of exposure (e.g., blood lead
concentrations). Each of these is discussed below.
- Fixed-site monitors: Concentrations reported from fixed-site monitors are most
informative if they are correlated with personal exposures; if monitors are located
close to study subjects; if monitored concentrations within a location are
correlated; or if monitored concentrations are combined with time-activity
information. Some studies use (low-cost) sensors to estimate pollutant exposures.
Sensors can provide relatively high spatial and temporal resolution, though sensor
validation and calibration can be a challenge, and both should be considered when
interpreting studies that use them.
- Personal monitoring: Personal monitoring characterizes exposures at the
individual level and provides exposure data attributable to both ambient air and
non-ambient-air sources. Measurement errors (e.g., instrument error and
A-24
-------
representativeness of exposures to ambient air) associated with personal
monitoring should be documented.
- Remote sensing: Remote sensing-based approaches can use data from satellites
to estimate ground-level concentrations of some pollutants (e.g., PM2.5, NO2, and
O3). Remote sensing approaches should be calibrated using ground-level
monitoring networks and their accuracy, precision, and any interferences should
be clearly documented.
- Models: Some epidemiologic studies use atmospheric models to estimate
exposures, either in place of or to supplement measurements from ambient air
monitors. These models may include land use regression models, chemical
transport models (e.g., EPA's Community Multiscale Air Quality (CMAQ)
modeling system), dispersion models, geostatistical models (e.g., kriging),
population stochastic models, and hybrid models that use inputs from multiple
sources of information (e.g., including models, satellites, and monitors).
Uncertainty in exposure predictions from these models is largely influenced by
model formulations and the quality of model input data pertaining to precursor
emissions or meteorology, which tends to vary on a study-by-study basis. Studies
that use these models to estimate exposures should clearly document the
procedures for model development and validation, as well as model performance
under the conditions of the study.
- Biomarkers: For some pollutants, epidemiologic studies use biomarkers to
estimate exposures. As noted above for personal monitoring, biomarkers provide
exposure estimates at the individual level and that are attributable to both ambient
air and non-ambient-air sources. Depending on the biomarker used, exposure
biomarkers may be quite limited with regard to the specific timing and duration of
the exposure represented. When used, biomarkers should be clearly justified and
measured using valid, reliable methods with appropriate characterization of
variability.
A.5.1.5. Outcome Assessment and Evaluation
Confidence in study results is greater when outcomes are based on independent clinical
assessments of the health condition, without knowledge of exposure status. This can include data
from clinical examinations conducted as part of the study as well as data obtained from hospitals,
insurance providers or other health care organizations. Outcomes based on interviews, self-
reports, or analysis of biological indicators, and defined by consistent criteria and collected by
validated, reliable methods without knowledge of exposure status are generally viewed with
relatively high confidence. For example, independent, clinical assessment is valuable for
outcomes like lung function or incidence of disease, but self-report of physician diagnosis may
be adequate for outcomes for which validation studies have been conducted that demonstrate
good reliability of self-reported health outcomes (Murgia et al., 2014; Weakley et al., 2013; Barr
et al., 2002; Muhajarine et al., 1997; Toren et al., 1993). For biological samples, the stability of
the biomarker(s) of interest and the sensitivity and precision of the analytical method is
A-25
-------
considered. In general, if errors in the outcome assessment are not correlated with exposure
status, those errors tend to bias results toward the null.
A.5.1.6. Potential Confounding
Confounding is ".. .a confusion of effects. Specifically, the apparent effect of the
exposure of interest is distorted because the effect of an extraneous factor is mistaken for or
mixed with the actual exposure effect (which may be null)" (Rothman and Greenland, 1998). A
confounder is associated with both the exposure and the outcome. Factors are considered to be
potential confounders if demonstrated in the scientific literature to be associated with both the
exposure and outcome being evaluated. Not accounting for confounders can introduce bias and
may produce artifactual associations, thus emphasizing the importance of evaluating the
approaches used in individual studies to account for potential confounders and how the strengths
of those approaches might influence weight of evidence considerations in causality
determinations (NASEM, 2022, pp. 5-6). Inference is stronger for epidemiologic studies that
explicitly identify confounders and their potential impacts on results and that take steps to
minimize those impacts, often through statistical adjustment and/or study design. In assessing
studies that have considered confounders, ISAs give preference to those studies that clearly
document the appropriateness of models selected, the validity of methods employed, and the
assumptions underlying those methods. Studies that only provide unadjusted effect estimates are
not considered.
Potential confounders vary by study population, exposure, and outcome. To control for
potential confounders, a variety of statistical methods can be employed. Potential confounders
included in statistical analyses should be based on a thoughtful review of published literature and
the evidence of potential relationships between variables. In considering the issue of potential
confounding on study conclusions, studies in the ISA may utilize different approaches for
identifying potential confounders, controlling for the role of such confounders, and accounting
for unknown confounders, as described below.
• Identification of potential confounders: Strategies for identifying potential confounders
can include a priori biological considerations, published literature, causal diagrams (e.g.,
directed acyclic graphs), and/or statistical analyses.
• Approach to controlling for confounders: Scientific judgment is needed to identify the
likely sources and extent of confounding and to determine how effectively selected study
designs and analyses control for potential confounders. Multivariable regression models
are often used to detect and control for potential confounders by adjusting for factors that
might confound results. These models attempt to control for characteristics that may
differ systematically between exposed and unexposed study participants. Other
approaches that have been applied include matching and weighting methods Stuart, 2010,
g-computation and substitution estimators Keil et al., 2020, doubly robust estimators
(Diaz, 2019), bounding approaches Richardson et al., 2014, quantitative bias analysis
A-26
-------
approaches (Lash et al., 2014, Weuve et al., 2018), and sensitivity analyses. The approach
selected should avoid post-treatment confounding caused by inappropriate adjustment for
post-treatment variables (e.g., collider bias).
• Unknown confounders: Approaches to handling unknown confounders include, but are
not limited to, instrumental variables (Greenland, 2000), bounding approaches
Richardson et al., 2014, quantitative bias analysis approaches (Lash et al., 2014, Weuve
et al., 2018, and sensitivity analyses. In prioritizing studies for inclusion in the ISA,
preference is given to studies with clear discussions of the assumptions underlying
selected approaches, the robustness of those assumptions, and any sensitivity analyses
conducted. Confidence that unmeasured confounders are not responsible for study
findings is increased when multiple studies are conducted in various settings using
different subjects or exposures, each of which might eliminate different sources of
confounding. Multicity studies can provide insight into the potential impact of unknown
confounders on study results through the use of a consistent method to analyze data from
across locations with different concentrations of copollutants and other covariates.
For studies of short-term exposure, concern is greatest for potential confounders that vary
temporally on time scales similar to the variation in exposures and health outcomes. Confounders
commonly considered in studies of short-term exposures include, but are not limited to, the
following:
• Meteorology,
• Copollutants, particularly those arising from the same source(s) as the pollutant of
interest,
• Day of week and season,
• Medication use,
• Stress and noise,
• Allergen exposure, and
• Long-term temporal trends.
For studies of long-term exposures, concern is greatest for potential confounders that
vary spatially. Confounders commonly considered in studies of long-term exposures include, but
are not limited to, the following:
• SES, race/ethnicity, and age,
• Smoking rates,
• Stress and noise,
• Historic sources,
• Residential housing age,
• Occupational exposures, and
• Short-term exposures.
A-27
-------
Additionally, studies of long-term exposure that do not also appropriately address relevant time-
varying confounders (e.g., societal patterns and trends in smoking rates, medication use) can lead
to erroneous conclusions regarding support for causal relationships. A number of methods have
been employed to handle time-varying confounders in studies of long-term exposure, including
but not limited to marginal structural models (Robins et al., 2000), regression adjustment using
the longitudinal g-computation formula (Bang and Robins, 2005; Hernan and Robins, 2016)
longitudinal inverse probability weighting (Ertefaie and Stephens, 2010), doubly robust
sequential regression estimators (Diaz et al., 2023; Stitelman et al., 2012), and difference-in-
differences designs (Wing et al., 2018).
For epidemiologic studies that utilize biomarkers (e.g., blood Pb) to estimate exposures,
there is additional concern regarding the representation of the specific timing, duration, and/or
frequency of exposures. Depending on the biomarker used, there may be limitations in the
conclusions that can be drawn about the relationship between the exposure and outcome.
Across exposure durations, the ISAs additionally evaluate the degree to which co-
occurring ambient air pollutants may confound health effect associations with the criteria
pollutant(s) under evaluation. An emphasis on epidemiologic studies that attempt to minimize
such potential "copollutant" confounding may increase confidence in conclusions regarding
outcomes associated with the pollutant under evaluation. Copollutant modeling (i.e., two-
pollutant models) can reduce concern for this confounding, particularly when correlations
between copollutants are relatively low (e.g., r < 0.4). However, when correlations are high (e.g.,
r > 0.7), collinearity between pollutants makes copollutant modeling less informative, leading to
greater uncertainty in the degree to which reported associations reflect health effects of exposure
to the specific criteria pollutant(s) under evaluation.
A.5.1.7. Effect Measure Modification
Effect measure modification occurs when the effect of a pollutant exposure on a health
outcome of interest differs between subgroups or strata of risk factors (Rothman and Greenland,
1998). When a risk factor is an effect modifier, it changes the magnitude of the association
between the pollutant exposure and the health outcome in stratified analyses. For example, the
presence of a pre-existing disease or indicator of low SES (e.g., educational attainment,
household income) may act as an effect modifier if it is associated with increased risk of effects
from air pollution exposure. It is often possible to stratify the association between health
outcome and exposure by one or more of these potential effect modifiers. For risk factors that
modify the association, effect estimates for each stratum will differ from one another and from
the overall estimate, indicating there to be different quantitative relationships between exposure
metric and outcome for populations represented by these variables.
A-28
-------
Inference can be particularly strong from studies that consider the potential impacts of
effect measure modification, especially when the modifying factors are coherent with
information from other lines of evidence regarding the biological pathways connecting pollutant
exposures with particular health effects (e.g., larger pollutant-related effects in human
populations with pre-existing cardiovascular disease would be coherent with animal evidence of
cardiovascular effects). Such studies can also be important for quantitatively assessing risks to
populations and lifestages at increased risk of health effects for the criteria pollutant evaluated
(section A.7.2.3). Uncertainty in inference is lower from studies that articulate and justify
assumptions of treatment effect heterogeneity and that include appropriate diagnostics (e.g.,
multiple comparisons) to account for potentially spurious findings.
Evaluation of effect measure modification in the evidence base informs causality
determinations in several ways. First, the presence of effect measure modification can help
identify potentially at-risk populations. Consistent evidence that at least one population subgroup
is at risk is a category of pollutant-related health effects provides strong support for a causality
determination for that health effect category, though the lack of evidence for effect measure
modification where there is otherwise evidence of a pollutant-related health effect in the general
population does not weaken the support for a causality determination. Evidence for effect
measure modification can also explain heterogeneity in results across studies, which could
reduce uncertainties regarding inconsistent evidence. Finally, effect measure modification can
provide supporting information on mechanisms (e.g., genetic polymorphisms or microbiome
profiles) contributing to pollutant-related health effects.
A.5.1.8. Statistical Methodology
Statistical assumptions should be articulated in the context of a study design description.
Statistical methods that are appropriate for the power of the study carry greater weight. For
example, categorical analyses with small sample sizes can be prone to biasing results toward or
away from the null. Statistical tests such as correlations, t-tests and chi-squared tests are not
considered sensitive enough for adequate inferences regarding health effect associations. For all
methods, the pattern of effect estimates and precision of the estimates (i.e., width of 95% CI)
across studies are important considerations for assessing the strength and/or patterns of
associations, rather than relying only on statistical significance which is highly dependent on
study design (Gelman and Greenland, 2019; Greenland et al., 2016; Wasserstein and Lazar,
2016; Wasserstein et al., 2019). Appropriateness and limitations in statistical approaches should
be clearly articulated. Sensitivity analyses with alternative statistical models or specifications can
inform the stability of findings and aid in judgments of the strength of inference that can be
A-29
-------
drawn from results. In the case of multiple comparisons, consistency in the pattern of association
can increase confidence that associations are not due to chance alone.
Some epidemiologic studies employ alternative methods for confounder control that are
intended to better account for confounders and to mimic randomized experiments and reduce
potential bias. As discussed above (A.5.1.1), in the peer-reviewed literature these studies are
often referred to as causal inference studies or studies that used causal modeling methods.38 In its
review of the ISA weight-of-evidence framework, the NASEM committee noted that "the utility
of a causal inference framework in an individual study depends on its appropriate use" and "[t]he
ability to show mathematical equivalence between statistical and causal parameters does not
make the assumptions required for such equivalence true" (NASEM, 2022). Furthermore, the
NASEM noted that well-designed studies "1) articulate the scientific question in terms of
potential and counterfactual outcomes, 2) specify available data and a causal model, 3) articulate
a set of assumptions on the causal model that allow the identification of the causal parameter of
interest as an observable statistical quantity, 4) analyze the data to estimate the identified
statistical quantity, and 5) interpret the statistical results as causal relations according to the
validity of the assumptions made in steps 3 and 4, and quantify the statistical uncertainties"
(NASEM, 2022, pp. 163-164). Thus, consistent with the approach to evaluating epidemiologic
studies that employ traditional regression models, when evaluating studies that use such
alternative methods for confounder control the ISA considers whether study assumptions are
clear and plausible, whether confounding has been carefully considered and addressed with
appropriate approaches, whether study assumptions are internally consistent and valid, and
whether statistical uncertainties are appropriately quantified.
A.5.2. Controlled Human Exposure
Controlled human exposure studies (also known as human clinical studies) evaluate the
health effects of experimental exposures in human volunteers under highly controlled and
carefully regulated environmental conditions and activity levels. These studies provide direct
evidence of physiological and/or biomolecular effects following air pollution exposures and help
to identify the biological pathways linking exposures to health effects in humans. They can
provide strong support for the biological plausibility of relationships between air pollutant
exposures and health effects that may be indicated by epidemiologic study associations, and
precise information on exposure- or dose-response relationships in homogeneous populations,
often at exposure concentrations at or near those common in ambient air. Thus, data from
controlled human exposure studies can provide direct evidence of cause-and-effect relationships
38 To prevent confusion with the main scientific conclusions presented within an ISA (i.e., the causality
determinations), this document refers to such studies as employing alternative methods for confounder control.
A-30
-------
in humans and can help compensate for some of the limitations in epidemiologic studies (e.g.,
potential confounding by co-occurring pollutants or other factors, exposure error, etc.) (NASEM,
2017). For ethical and practical reasons, controlled human exposure studies are generally limited
to examining relatively healthy people or those with mild or moderate diseases; short exposure
durations; and exposure concentrations expected to elicit no more than mild, transient effects. As
a result, these studies do not include individuals who may be at the highest risk of pollutant-
related health effects (e.g., children, older adults, people with more severe disease or
comorbidities).
Controlled human exposure studies should clearly describe the primary and any
secondary aims of the study, or specific hypotheses being tested. In evaluating the quality of
these studies, the ISAs additionally consider the appropriateness of the study design, study
population, pollutants examined, approach to assigning exposures, outcome assessment, control
for potential confounders and statistical analysis. Each of these study attributes is discussed
below.
A.5.2.1. Study Design
In prioritizing studies for inclusion in the ISA, preference is given to balanced crossover
(repeated measures) or parallel study designs that include control exposures to clean air. In a
crossover design, each study participant is exposed to both the air pollutant under evaluation and
to clean air, under the same conditions (e.g., ventilation rate), as a control. In this design, each
study participant serves as their own control, minimizing inter-individual confounders. In
crossover studies, a sufficient and specified time between exposure days should be provided to
avoid carry over effects from prior exposure days. In studies using a parallel design, all arms
should be matched for individual characteristics, such as age, sex, race/ethnicity, anthropometric
properties, and health status. In studies evaluating effects of disease, appropriately matched
healthy controls are preferred to aid interpretation of study results. The evaluation of study
design generally includes consideration of factors that minimize bias in results such as
randomization; blinding; allocation concealment of study subjects, investigators, and research
staff; and withdrawal/exclusion of subjects. Studies must include appropriate control groups to
allow for accurate interpretation of results relative to criteria pollutant exposure.
A.5.2.2. Study Population
Depending on the study design, subjects recruited into study groups should be matched
for age, sex, race/ethnicity, anthropometric properties, and health status. In studies evaluating
effects of specific subject characteristics (e.g., disease, genetic polymorphism, etc.),
appropriately matched healthy controls are preferred. Relevant characteristics and health status
should be reported for each experimental group. For the examination of populations with an
A-31
-------
underlying health condition (e.g., asthma), independent, clinical assessment of the health
condition is ideal, but self-report of physician diagnosis generally is considered reliable for
respiratory and cardiovascular disease outcomes. Criteria for including and excluding subjects
should be indicated clearly, and the loss or withdrawal of recruited subjects during the course of
a study should be reported.
A.5.2.3. Pollutant
As described above for epidemiologic studies, each ISA focuses on studies that evaluate
the effects of exposures to the particular criteria pollutant(s) under evaluation. Thus, emphasis is
placed on controlled human exposure studies that evaluate individual criteria pollutants or
components of a criteria pollutant that are particularly relevant for reviewing the NAAQS (e.g.,
PM2.5 in the case of particulate matter). Studies of pollutant mixtures (e.g., ozone as part of an
oxidant mixture) can be informative if health effects of exposure to the criteria pollutant under
evaluation, presumably a component of the mixture, are also examined separately. Such studies
can provide insight into potential modification of the effect of the criteria pollutant by other
individual pollutants or by a broader pollutant mixture. Ideally, studies should report the source,
purity, and form of the pollutant(s) examined.
A.5.2.4. Exposures
Controlled human exposure studies that approximate expected human exposures in terms
of concentration, duration, exercise level, ventilation rate and method of exposure are of
particular interest. In prioritizing studies for inclusion in the ISA, preference is given to studies
that evaluate pollutant exposure concentrations close to existing ambient air concentrations,
though studies that use higher exposure concentrations may still provide information relevant to
consideration of the biological plausibility of effects associated with lower exposures and/or
dosimetry. Studies should have measures in place to adequately monitor and control the exposure
conditions, including pollutant concentrations, temperature, and relative humidity. The method
of exposure (e.g., chamber, facemask, etc.) should be specified, and activity level of subjects
during exposures should be well-characterized. Preference is also given to studies that include
control exposures (e.g., to filtered air or room air). Study subjects should be randomly exposed
without knowledge of the exposure condition, and exposure metrics should be well
characterized, including external exposure, intake dose, dosing regimen, and exposure route.
A.5.2.5. Outcome Assessment and Evaluation
For each experiment and each experimental group, including controls, precise details
should be provided describing the endpoints examined, how they are measured, and when and
where they are evaluated. Endpoints should be assessed in the same manner for control and
exposure groups using valid, reliable methods. This includes using the same procedures in terms
A-32
-------
of time after exposure, methods, endpoint evaluator, etc. Blinding of endpoint evaluators is ideal,
especially for qualitative endpoints such as histopathology. Time of the endpoint evaluations is a
key consideration that will vary depending on endpoint evaluated. Endpoints should be assessed
at time points that are appropriate for the research questions, and those time points should be
clearly justified.
A.5.2.6. Potential Confounding
To limit potential for confounding, studies included in the ISA use either a crossover
repeated measures design, in which each study participant serves as their own control, or a
parallel exposure design in which experimental and control groups are matched for individual
level characteristics (e.g., race/ethnicity, sex, body weight, smoking history, age) and time-
varying factors (e.g., seasonal and diurnal patterns). Exposures should be well-characterized to
evaluate independent effects of the pollutant(s) under study.
A.5.2.7. Statistical Methodology
Statistical methods should be described clearly and be appropriate for the study design
and research question, including correction for multiple comparisons when appropriate.
Statistical significance is generally used to evaluate the findings of controlled human exposure
studies. Detection of statistical significance is influenced by a variety of factors including, but
not limited to, the size of the study, exposure and outcome measurement error, and statistical
model specifications. Sample size is not a criterion for exclusion, though the sample size should
provide adequate power to detect hypothesized effects. Because statistical tests have limitations,
consideration is also given to trends in data and reproducibility of results. Consistent trends
across studies can informative, even if results of individual studies are not statistically
significant.
A.5.3. Experimental Animal Studies and Emerging Approaches in
Toxicology
Animal toxicological studies evaluate the health effects of controlled pollutant exposures
in animal models (e.g., non-human primates, mice, rats, guinea pig). Investigators expose non-
human mammalian animal species to known concentrations of air pollutants under carefully
regulated laboratory conditions. Experimental animal studies provide critical information on
potential human health effects, exposure- and dose-response relationships, and underlying
toxicological pathways and mechanisms of action. One major uncertainty associated with animal
studies is the representativeness of responses in animals to humans, given the potential
differences in metabolism, hormonal regulation, breathing patterns, lung structure, physiology,
and anatomy. When these differences are appropriately characterized, experimental animal
studies can inform and improve our understanding of biological mechanisms, providing support
A-33
-------
for biological plausibility of relationships between air pollutant exposures and health effects that
may be indicated by epidemiologic studies, and address uncertainties in other lines of evidence
(e.g., confounding in epidemiologic studies). In addition, depending on the animal model used
and the study design employed, experimental animal studies may help inform an understanding
of the potential for biological responses and health effects in certain populations that may be at
increased risk of a criteria pollutant-related health effect.
Another emerging line of evidence in toxicological studies comes from new approach
methodologies (NAMs), a term which refers to any technology, methodology, approach, or
combination of these tools that can inform chemical hazard and risk assessment while avoiding
the use of animal testing, including in silico, in chemico, in vitro, and ex vivo approaches
(ECHA, 2016; U.S EPA, 2018; U.S EPA, 2021b).39 While the EPA has historically relied on in
vitro and/or in vivo studies to provide mechanistic insight, data collected using well-validated
NAMs may also provide insight into the occurrence of health effects when guidelines are
established for use of these methods in air pollution studies.
Toxicological studies should clearly describe the primary and any secondary aims of the
study, or specific hypotheses being tested. In evaluating the quality of these studies, the ISAs
additionally consider the appropriateness of the study design, test model, pollutant(s) examined,
exposure assignment and approach, outcome assessment, variable control, and statistical
analysis. Each of these study attributes is discussed below.
A.5.3.1. Study Design
Toxicological studies should include appropriately time-matched control exposures,
should randomize assignment to exposure groups and, where possible, should blind research
personnel from endpoint evaluation and analysis. Blinding of research personnel to study group
may not be possible due to animal welfare and experimental considerations; however,
differences in the monitoring or handling of animals across groups by research personnel should
be minimized. Studies should use methods to limit differences in baseline characteristics of
control and exposure groups, and groups should be subjected to identical experimental
procedures and conditions to the extent possible (e.g., in animal care including housing,
husbandry, etc.). The evaluation of study design generally includes consideration of factors that
minimize bias in results, such as randomization, blinding, and unexplained loss of animals.
39 Experimentation performed using a computer are referred to as an in silico approach (Cronin, 2009). Chemical
reactivity tests that are performed in the absence of biological materials are referred to as in chemico approaches
(Cronin, 2009). In vitro tests are performed outside of a living organism using established cell lines while ex vivo
approaches are those that involve the collection of cells/tissues/organs from living organisms for use in
experimentation (European Commission et al., 2020).
A-34
-------
Where applicable, approval of study protocols by appropriate institutional animal care and use
committees must be obtained (European Commission et al., 2020; Cronin et al., 2009).
A.5.3.2. Test Model
Toxicological studies should provide a clear justification for their chosen model system.
Unless data indicate otherwise, laboratory nonhuman mammalian animal species of any
lifestage, stock, and strain are considered appropriate for evaluating effects of pollutant
exposure. Ideally, studies should report species, strain, substrain, genetic background, age, sex,
and weight. It is preferred that the authors test for effects in both sexes and multiple lifestages
and report the result for each group separately. All animals used in a study should be accounted
for, and rationale for exclusion of animals or data should be specified.
A.5.3.3. Pollutant
As described above for other disciplines, each ISA focuses on studies that examine the
effects of exposures to the particular criteria pollutant(s) under evaluation. Thus, emphasis is
placed on toxicological studies that examine individual criteria pollutants or components of a
criteria pollutant that are particularly relevant for reviewing the NAAQS (e.g., PM2.5 in the case
of particulate matter). Studies of pollutant mixtures can be informative if health effects of
exposure to the criteria pollutant under evaluation, presumably a component of the mixture, are
also examined separately. Such studies can provide insight into potential modification of the
effect of the criteria pollutant by other individual pollutants or by a broader pollutant mixture.
Ideally, studies should report the source, purity, and form of the pollutant(s) examined.
A.5.3.4. Exposures
Animal toxicological studies should have measures in place to adequately control
exposure conditions, including the identity of the target pollutant, exposure concentrations,
temperature, and relative humidity. Studies that approximate expected human exposures in terms
of concentration, duration, timing of exposure, and method of pollutant administration are of
particular interest, though pollutant exposures within two orders of magnitude of recent ambient
air concentrations may be considered relevant, e.g., if they assess a previously unreported effect
or mode of action for an observed effect or examine multiple concentrations to elucidate
exposure-response relationships. This range in relevant exposures is meant to account for
differences in dosimetry, toxicokinetics, and biological sensitivity between humans, including
groups at increased risk, and the various animal species and strains used in toxicological studies.
Studies using exposure concentrations or doses at the higher end of this range may be considered
to the extent that they provide information relevant to understanding mode of action or
mechanisms, inter-species variation, or factors that may confer increased risk in human
populations.
A-35
-------
The focus is generally on inhalation exposure, but oral and intravenous exposures may
also be informative in studies that examine a relevant biomarker (e.g., blood lead
concentrations). Non-inhalation exposure experiments that target the respiratory tract (e.g.,
intratracheal instillation [IT]) may be informative if they provide information relevant to
biological plausibility and dosimetry. As discussed above, in vitro studies in validated model
systems may be considered for inclusion in an ISA, though the relevance of in vitro exposure
concentrations to human inhalation exposures are often an additional source of uncertainty. All
studies should include exposure control groups (e.g., filtered air or room air).
A.5.3.5. Outcome Assessment and Evaluation
For each treatment group, including controls, precise details should be provided
describing the endpoints examined, how they are measured, and when and where they are
evaluated. Endpoints should be assessed in the same manner for control and exposure groups
using valid, reliable methods. This includes using the same procedures in terms of time after
exposure, methods, endpoint evaluator, etc. Limits of detection should be provided for
quantitative assays, when available. Blinding of endpoint evaluators is ideal, especially for
qualitative endpoints such as histopathology. Timing of the endpoint evaluations is a key
consideration that will vary depending on the endpoint under investigation. Endpoints should be
assessed at time points that are appropriate for the research questions.
A.5.3.6. Variable Control
To limit potential impact of other variables on study results, studies included in the ISA
match experimental and control groups for individual-level characteristics and time-varying
factors. Individual characteristics include strain, sex, body weight, litter size, feed, and water
consumption. Exposures should be well characterized to evaluate independent effects of the
pollutant(s) under study.
A.5.3.7. Statistical Methodology
Statistical methods should be described clearly and should be appropriate for the study
design and research question, including correction for multiple comparisons when appropriate.
Results should be reported with sufficient detail to allow for independent interpretation (e.g.,
quantitatively with a measure of variance). Statistical significance is generally used to evaluate
study findings. Detection of statistical significance is influenced by a variety of factors including,
but not limited to, the sample size used in the study, exposure and outcome measurement error,
and statistical model specifications. Sample size is not a criterion for exclusion, though the
sample size should provide adequate power to detect hypothesized effects. Because statistical
tests have limitations, the ISAs also consider trends in results across studies and reproducibility
A-36
-------
of study results. Consistent trends across studies can be informative for weight-of-evidence
evaluations, even if results of individual studies are not statistically significant.
A.5.4. Atmospheric, Environmental, and Exposure Science
Information from atmospheric and exposure sciences can inform the ISA interpretation of
health effects evidence. Atmospheric sciences provide information on air pollutant sources,
atmospheric transport and transformation, ambient air concentrations, and techniques for
measuring or modeling air pollutants and their precursors. Human exposures to air pollutants is
contact with the pollutant(s) at the interface of the breathing zone over a specified length of time
(U. S EPA, 2019b).
Atmospheric and exposure sciences provide health effects studies with information on
emissions, ambient air concentrations, properties, processes, and constituents of criteria air
pollutants (e.g., in the case of photochemical oxidants and oxides of nitrogen); spatiotemporal
variations; various exposure surrogates (e.g., ambient air concentration, indoor concentration,
and total personal exposure); and variability and uncertainty associated with exposure estimates.
Of particular interest is understanding the strengths and limitations of exposure surrogates
commonly used in studies of health effects, including information on errors in measuring or
modeling the exposure surrogate and errors resulting from using the surrogate to approximate
true exposure.
The ISA study quality evaluation for atmospheric and exposure science studies considers
the applicability and utility of the study, the soundness of the study approach, clarity and
completeness of the study, and its treatment of uncertainty and variability. These study attributes
are adopted from the EPA exposure assessment guidelines (U.S EPA, 2019b).40 They are not
mutually exclusive and can be used collectively to inform conclusions on study quality.
A.5.4.1. Applicability and Utility
Applicability and utility refer to the extent to which information presented in a study is
relevant for its intended use (U.S EPA, 2019b). To be considered for inclusion in an ISA,
atmospheric and exposure science studies should evaluate the pollutant in ambient air (and in
other media, as relevant), and/or exposure pathways (e.g., emissions, transformation, transport)
pertinent to U.S. populations, with a focus on the policy-relevant issues that help frame the
assessment of the health effects evidence. These issues can vary across the ISAs and generally
include characterizing emissions sources, transport and transformation processes, measurement
and modeling methods, temporal and spatial patterns of ambient air pollutant concentrations and
40 Analogous study quality evaluation guidelines have not previously been developed for atmospheric science. Thus,
the ISA study quality evaluation of atmospheric science studies is based on adaptation of the study attributes
described in the EPA exposure assessment guidelines (U.S EPA, 2019b).
A-37
-------
exposures, meteorological impacts on pollutant concentrations, relationships and correlations
between pollutants within complex mixtures, infiltration to indoor environments, transfers to
other media, as relevant, and the strengths and limitations of various exposure estimation
approaches. For studies examining exposure surrogates, the ISA examines various aspects of
exposure measurement error, some of which have been discussed above in the context of the
health evidence (e.g., errors in measurements, inherent assumptions in exposure modeling,
uncertainties in model input parameters, spatiotemporal variations in pollutant concentrations,
activity patterns and building ventilation). The ISA assessment of applicability and utility is
aided by clear documentation of study design, data collection/generation techniques, and any
underlying assumptions.
A.5.4.2. Soundness
Soundness refers to the extent to which the scientific and technical procedures, measures,
methods, or models employed to generate information are reasonable for, and consistent with,
the intended application (U.S EPA, 2019b). In evaluating soundness, the IS As focus on
atmospheric or environmental science and exposure studies that use quality-assured
measurement and/or modeling techniques. Studies should include or reference clear and
comprehensive descriptions of the measurement or modeling techniques used, evaluation
procedures and performance metrics, strengths and limitations of techniques, and quality-control
procedures. The ISAs place more weight on studies that use methodologies appropriate for
meeting study goals and that clearly justify those methodological decisions. For example,
considerations when evaluating soundness of measurement- and model-based atmospheric
science or exposure studies are described below.
• Measurement studies: The ISAs place more weight on studies that document or
reference high quality measurement data, including uncertainty, bias, sensitivity,
specificity, stability, repeatability, and data management practices consistent with best
practices among similar measurement studies. Also included are innovative, newly
developed methods with less developed data quality documentation that show promise as
potential alternative methods that could have advantages over current methods, and
methods that contribute unique insight into atmospheric processes and concentration
trends that complement established, high quality measurement methods.
• Modeling studies: The ISAs place more weight on studies that include or reference clear
descriptions of acceptable rationales for model selection and model assumptions, model
input parameters (e.g., data source for input parameters, rationale for input data selection,
quality of input data (including accuracy, precision, representativeness, completeness,
and consistency)), model calibration/validation procedures (e.g., goodness-of-fit criteria
for acceptance of the parameter value, the procedure to handle outliers, procedures to
validate results, model parameter sensitivity analysis, impact of parameter uncertainty on
results), model evaluation results (e.g., for chemical transport models) as well as data
transfer, transformation and storage procedures.
A-38
-------
A.5.4.3. Clarity and Completeness
Clarity and completeness refer to study transparency and is a crosscutting attribute
(WHO, 2008). Transparency is essential for evaluating other attributes of data/study quality. For
example, documenting and reporting key assumptions, methods, formulas, input parameters,
reasoning processes, and limitations would increase the transparency of a study. Key aspects of a
study for purposes of evaluating transparency are summarized below.
• Study Rationale: Study transparency is greater for studies presenting clear rationales for
study design, location, population selection (in the case of exposure studies), method
selection, and conclusions.
• Assumptions: The most informative studies clearly articulate assumptions related to
study design and methods used (e.g., dispersion model assumptions, and assumptions
associated with using central monitoring sites for exposure assignment), as well as
assumptions associated with chemistry, transport, dispersion, and/or exposure modeling
techniques, and strengths and limitations of a study.
• Data and procedures: Studies that clearly describe QA and QC procedures, key data,
data sources and methods, and statistical methods (e.g., methods for missing data and
imputation, and quantitative relationships between and within pollutant measurements,
such as regression slopes, intercepts, and fit statistic), are considered most informative.
A.5.4.4. Uncertainty and Variability
Uncertainty in environmental or exposure assessment studies comes from incomplete or
incorrect information about an environmental measurement or true exposure and variability
describes the natural heterogeneity of measurements/estimates. Uncertainty and variability are
associated with each element of atmospheric/environmental analyses and exposure assessments.
Evaluation of uncertainty and variability can increase our understanding of the reliability of
analyses and the data needs for improving them.
In considering studies for inclusion in the ISA, preference is given to studies that
characterize uncertainties in measurement and/or modeling approaches and the factors
contributing to uncertainties in these analyses. Studies can be of particular use in the ISAs if they
characterize exposure errors relevant to interpreting epidemiologic study designs or if they
examine the potential for differential exposures across various populations. For example,
classical error in the exposure surrogate of an epidemiologic study, defined as error scattered
about the true exposure and independent of the true exposure, is generally expected to reduce
precision and to negatively bias health effect associations between air pollution and health
effects. In contrast, Berkson error, in which the true value varies randomly around the measured
value and the measurement error is independent of the measured value, reduces precision but is
not expected to bias the health effect estimate (Goldman et al., 2011).
A-39
-------
Studies of differential exposures can identify populations with elevated exposures,
characterize exposures for populations at higher risk of pollutant-related effects (e.g., due to pre-
existing disease), and can evaluate the factors contributing to variability in exposures (e.g.,
activity patterns, proximity to sources, etc.).
A.6. EXTRACTING DATA FROM RELEVANT STUDIES
For studies determined to be appropriate for inclusion in an ISA, relevant study data are
extracted into evidence tables.41 Data are extracted using literature review software tools (e.g.,
DistillerSR) or directly into spreadsheets by the EPA scientists or other personnel trained to
perform this task. In all cases, the integrity of extracted data is confirmed by quality control
checks performed as described below (A.9). Extracted study summary data are compiled in
evidence tables that provide a high-level overview of the evaluated studies and are used to assist
with data analysis, evidence synthesis, and formulation of conclusions. Example formats for
example data extraction tables are provided below (see Tables A-7 through A-9).
Table A-6. Example Data Extraction Table - Epidemiologic Studies.
Concentration
Effect
Estimates
95% CI
Study
Study
Exposure
and Co-
Outcome
Statistical
Reference
Population
Assessment
pollutant
Assessment
Methods
Examination
HERO ID,
Population
Exposure
Mean/median
Description of
Model type
ORs, RRs,
author(s), year
details
model,
monitor data
value (reported
by study and/or
outcome
and list of
confounders
P
N
standardized)
Min, Max
Location
Annual,
monthly
Pearson or
Years
Model
Spearman rho
(recruitment)
development
value(s) for
(follow-up)
and validation
each co-
pollutant
Study design
Max, maximum; min, minimum; N, sample size; ORs, odds ratio; RRs, risk or rate ratios.
41 If a potentially informative study is missing information relevant to extraction endpoints, the EPA may contact the
corresponding author to request that information. The request and author's response will be included in the docket
for that ISA. If the missing information is not available, the study may be deemed inappropriate for inclusion in
the ISA.
A-40
-------
Table A-7. Example Data Extraction Table - Animal Toxicological Studies.
Study
Reference
Study
Population
Exposure Details
Endpoints Examined
HERO ID,
author(s), year
Species/strain,
age, & sample
size
Exposure route, concentration, duration,
timing of exposure, control group
composition, exposure biomarker (if
relevant)
Outcome(s) measured,
timing of outcome
measurement, results
summary
Table A-8. Example Data Extraction Table - Controlled Human Exposure
Studies.
Study
Reference
Study
Population
Exposure Details
Endpoints Examined
HERO ID,
author(s), year
Population details
(i.e., Sample size,
age, sex, disease
state, etc.)
Target pollutant concentration or range,
subject activity, duration of exposure,
frequency of exposure
Outcome(s) measured, timing
of outcome measurement,
results summary
Table A-9.
Example Data Extracted Table - Exposure Science Studies.
Study
Reference
Study Rationale
Sampling and
Modeling
Data Analysis
Exposure
Factors
Results
HERO ID,
author(s), year
Hypothesis; Study
location selection;
Inclusion and
exclusion criteria for
study participants
(age, gender, ethnic
group, health status,
SES, etc.); Sample
size determination;
Data quality
objectives
Exposure
indicators;
sampling
frequency and
duration;
Sampling
methods; model
input parameters
Statistical
analyses
methods;
summary
statistics;
uncertainty and
variability in the
findings; model
performance
Exposure
factor data and
the basis for
choosing
certain values
of exposure
factors
Exposure
levels for
various
populations;
sources and
extent of
exposure
errors;
correlations
with
copollutants
exposures
A.7. DRAFTING ISA SECTIONS
A.7.1. Draft Chapters and Obtain Peer Input
After completing data extraction, the ISA authors draft topic-specific chapters in two
stages. During the first stage, initial drafts are developed and used to inform discussions during a
peer-input workshop as described below. Initial draft chapters typically include an introduction, a
summary of the previous ISA conclusions, an overview of the scope of the assessment in the
A-41
-------
current ISA, and a review of the available science. Initial drafts of human health outcome
chapters often additionally include a biological plausibility section. The integrated summary and
policy-relevant conclusions (e.g., causality determinations, conclusions on at-risk populations)
are not developed in this initial drafting stage.
The peer input workshop brings together the EPA and external subj ect matter experts from
a variety of disciplines to review the draft chapters. The purpose of the workshop is to obtain
early feedback from experts in relevant fields to ensure that the draft ISA reflects the most up-to-
date policy-relevant science. The discussion at the peer-input workshop also can provide a
foundation for initial integration of evidence within and across disciplines. During the peer input
workshop, expert panelists are asked to address several overarching questions, often including,
but not limited to the following:
• To what extent do the initial draft materials capture the key studies published since the
cutoff date of the prior ISA?
• What are panelists' views on the specific issues that should be considered or highlighted
and that will be important for integrating evidence across disciplines?
• To what degree are study results accurately reported and appropriately interpreted?
The EPA staff may ask workshop panelists additional questions related to the
organization and content of specific draft chapters. The public is invited to listen in on the
discussion during the workshop, but no draft ISA materials are shared publicly, and comments
are not solicited from the public at this time. After the workshop, the ISA authors update the
initial draft chapters, integrate evidence across disciplines, and develop policy-relevant scientific
conclusions including causality determinations and conclusions on populations that may be at
increased risk. The approaches to integrating evidence and developing these conclusions are
discussed in detail in the following sections.
A.7.2. Develop Policy-Relevant Scientific Conclusions
Drawing upon the results of studies determined to be relevant and of adequate quality, the
ISAs reach a number of policy-relevant conclusions intended to inform the NAAQS review.
Most prominently, the ISAs use structured frameworks to reach conclusions on the strength of
the scientific evidence supporting cause-effect relationships between pollutant exposures and
adverse health effects (i.e., causality determinations) and on the populations that may be at
increased risk of such adverse health effects (i.e., at-risk populations).42 While discussed
42 The ISAs also typically present findings on other policy-relevant issues, which vary by pollutant and discipline
and can include concentration-, exposure-, and/or dose-response relationships; the exposure concentrations below
which evidence for effects is limited; on the potential adversity of particular effects; the effects of particular
pollutant components (e.g., for PM2 5); etc. These are described more fully in individual assessments.
A-42
-------
separately, these frameworks are carefully integrated within the ISA, with the evidence for
increased risk in certain populations often providing substantial support for causality
determinations. The ISA frameworks for reaching conclusions on causality and at-risk
populations are described further below.
A. 7.2.1. Causality Determinations
The 1964 Surgeon General's report on tobacco smoking defined "cause" as a
"significant, effectual relationship between an agent and an associated disorder or disease in the
host" (Hew, 1964). More generally, a cause is an agent that brings about an effect or a result.
Unlike an association, a causal claim supports the creation of counterfactual claims; that is, a
claim about what the world would have been like under different or changed circumstances
(IOM, 2008).
The ISAs evaluate and integrate scientific evidence on health effects of criteria pollutant
exposures, and based on this integration and the weight of the evidence in support of causation,
present a determination regarding the existence of a causal relationship between pollutant
exposure and various health effects. Evaluating the potential for cause-effect relationships
between criteria pollutant exposures and health effects is made more challenging by the fact that
many of the outcomes evaluated in the ISAs have complex etiologies. Diseases such as asthma,
coronary heart disease, and cancer are typically initiated by multiple environmental and
biological factors. Outcomes in a given individual can depend on factors such as age, genetic
background, nutritional status, immune competence, and social factors (IOM, 2008; Gee and
Payne-Sturges, 2004). Further, exposure to a combination of agents could cause synergistic or
antagonistic effects. Thus, the observed risk may represent the net effect of many actions and
counteractions.
The ISA framework for reaching causality determinations recognizes that, compared to
any single study, the availability of multiple studies evaluating a particular topic, each with
different strengths and limitations, provides a more robust foundation for evaluating the overall
strength of the evidence. The existing framework is described in the 2015 Preamble to the ISAs
(U.S. EPA, 2015). That framework was informed by the weight-of-evidence approaches
formulated by other regulatory and science agencies, including the National Academy of
Sciences (NAS) Institute of Medicine IOM, 2008 , the International Agency for Research on
Cancer (IARC, 2006), the EPA Guidelines for Carcinogen Risk Assessment (U.S EPA, 2005),
the Centers for Disease Control and Prevention (CDC, 2004), Integrated Risk Information
System [IRIS] (U.S EPA, 2022), and World Health Organization (WHO, 2021). The frameworks
used by each of these organizations are similar in nature, although adapted to different purposes,
and have proven effective in providing uniform structure and language for causality
A-43
-------
determinations. The framework described below builds on that in the 2015 Preamble to the IS As
(U.S. EPA, 2015), with updates reflecting CAS AC feedback over multiple IS As since 2015 and
reflecting consideration of recent NASEM recommendations (NASEM, 2022).
A.7.2.1.1. Aspects of the evidence important to judging
causality
In making judgments regarding the potential for causal relationships, the IS As consider
various aspects of causality drawn from previous efforts focused largely on epidemiology
studies. The 1964 Surgeon General's report on tobacco smoking discussed criteria for the
evaluation of epidemiologic studies, focusing on consistency, strength, specificity, temporal
relationship, and coherence (Hew, 1964). Sir Austin Bradford Hill (Hill, 1965) articulated similar
aspects of causality in epidemiology and public health that have been widely adopted (IOM,
2008; IARC, 2006; U.S EPA, 2005; CDC, 2004). The EPA has adapted this list of characteristics
for use in the ISA causality determinations specific to effects of criteria pollutant exposures (U.S.
EPA, 2015).
Table A-10 and the accompanying text describe the key aspects of the evidence base that
the ISA considers in judging causality. Although the list of aspects provides a framework for
assessing the evidence, it does not lend itself to being considered in terms of simple formulas or
fixed rules leading to conclusions about causality (Hill, 1965). For example, one cannot simply
count the number of studies reporting statistically significant results or statistically
nonsignificant results and reach credible conclusions about the relative weight of evidence and
the likelihood of causality. The aspects cannot be used as a strict checklist, and not meeting one
or more of the principles does not automatically preclude a determination of causality [see
discussion in (CDC, 2004). Rather, these aspects provide a framework for systematic appraisal of
a body of evidence, informed by peer input and public comment, which includes weighing
alternative views on controversial issues. Additional context for interpreting the aspects in Table
A-10 is provided in subsequent sections.
A-44
-------
Table A-10. Aspects of the evidence important to judging causality.
Aspect
Description
Consistency
An inference of causation is strengthened when a pattern of elevated risks is
observed across multiple independent studies. The reproducibility of findings in
different groups and using different study designs constitutes one of the strongest
arguments for causation. Statistical significance is not the sole criterion by which
the presence or absence of an effect is determined. If there are discordant results
among investigations, possible reasons such as differences in exposure,
confounding factors, and the power of the study are considered.
Coherence
An inference of causation is strengthened when multiple lines of evidence
independently support the occurrence of related effects following pollutant
exposure. Such coherence can be demonstrated by evidence across various
disciplines and/or across a variety of study designs.
Biological plausibility
An inference of causation is strengthened by results from experimental studies or
other sources demonstrating biologically plausible mechanisms by which pollutant
exposures could lead to adverse outcomes.
Biological gradient
(i.e., exposure-
response relationship)
A well-characterized exposure- or dose-response relationship (e.g., larger, more
serious effects associated with increasing exposure/dose) can strongly support
cause and effect, especially when such relationships are observed across multiple
disciplines and durations of exposure, including the observation of larger and/or
more serious effects following longer exposures.
Strength of the
observed association
The finding of large, precise risks increases confidence that an association is not
likely due to chance, bias, or other factors. However, an effect estimate that is
small in magnitude does not necessarily indicate a lack of causation.
Experimental
evidence
Strong evidence supporting causation can be provided by experimental studies
(i.e., controlled human exposure, animal toxicological) and by studies of "natural
experiments" when a change in exposure is found to result in a change in
occurrence or frequency of health effects.
Temporality of the
observed association
Evidence that pollutant exposure precedes the appearance of the effect, and that
the interval between exposure and effect are reasonable based on available
science, supports causation.
Specificity of the
observed association
Evidence linking exposure to a specific outcome can provide strong support for
causation. However, lack of specificity does not necessarily indicate a lack of
causation since it is rarely expected that a pollutant exposure will invariably predict
the occurrence of an outcome, and since a given outcome may have multiple
causes.
A.7.2.1.1.1. Consistency
In assessing the consistency of findings across studies for evaluating the weight of the
evidence, the ISAs emphasize the pattern of results in a body of epidemiologic or experimental
studies examining related outcomes. Consistency of findings is informed by the repeated
observation of effects or associations across multiple independent studies. Results that are
replicable across variations in study designs or analytic choices "can be viewed as more robust
and as stronger evidence for a causal relationship" (NASEM, 2022, p. 7). Thus, the strength of
the evidence is increased when similar findings are reported in different populations under
A-45
-------
different circumstances. For epidemiologic studies, the ISA's evaluation of consistency includes
consideration of the direction and magnitude of associations across independent studies with
greater emphasis on the pattern of results across studies than on the statistical significance of
results in individual studies. Statistical significance is influenced by a variety of factors
including, but not limited to, the size of the study population, exposure and outcome
measurement error, and statistical model specifications. Statistical significance may be
informative; however, it is just one of the means of evaluating confidence in the observed
relationship and assessing the probability of chance as an explanation. As statistical inferences
may result in both false positives and false negatives, the ISAs emphasize the pattern of
associations across epidemiologic studies. Statistical significance of results is traditionally given
greater emphasis on the evaluation of consistency across controlled human exposure and animal
toxicological studies, though the pattern of results across such experimental studies with similar
designs and examining related effects can also be informative. Discordant results among
independent studies may be explained by differences in study methods, random errors, exposure
errors, confounding factors, or study power, and the ISAs explore such potential explanations for
studies with results that are not consistent.
A.7.2.1.1.2. Coherence
In evaluating coherence of the evidence base, the ISAs examine the degree to which
studies from different disciplines, or studies from the same discipline with fundamentally
different designs, support the occurrence of effects as a result of pollutant exposures. For
example, evidence base may include epidemiologic evidence reporting positive associations
between pollutant exposures and cardiovascular events which are coherent with controlled
human exposure and/or experimental animal studies demonstrating changes in cardiac or
vascular function following exposures. Evidence that is coherent across disciplines and/or study
designs provides stronger support for a causality determination than any of the individual lines of
evidence alone.
A.7.2.1.1.3. Experimental Evidence and Biological
Plausibility
In making judgments regarding causality, the ISAs specifically consider the extent to
which experimental studies provide evidence of effects resulting from air pollutant exposures,
which can also provide biological plausibility for associations reported in epidemiologic studies
to be indicative of causal relationships. Experimental studies provide valuable information on the
relationship between exposures and observed effects under well-defined conditions. Biological
plausibility for a causal relationship between pollutant exposure and a particular type of effect
can be supported by experimental studies, including those that provide an understanding of the
A-46
-------
mode of action through which pollutant exposures lead to health effects. This understanding may
span multiple levels of biological organization including, but not limited to, molecular and
cellular events in the pathways leading to disease. A complete understanding of the mode of
action is rarely available and is not necessary for it to be biologically plausible that an exposure-
effect relationship reflects a causal relationship. Rather, experimental studies demonstrating a
pollutant to elicit key physiological events in the pollutant exposure-to-response pathway, and
the relationships between those events, can provide strong support for cause-effect
interpretations of health effect associations reported in epidemiologic studies. Due to the
complexities and uncertainties related to extrapolation from non-human experimental model
systems, the ISA may draw upon studies that use exposure concentrations higher than those
considered relevant for typical ambient air exposures in human populations to inform
consideration of biological plausibility of relationships between pollutant exposures and various
health effects.
A.7.2.1.1.4. Biological Gradient
The presence of concentration-, exposure-, and/or dose-response relationships in the
study dataset can increase confidence in a finding that exposure may be causative, particularly
when such relationships are demonstrated across multiple independent studies or across
disciplines and potential confounders have been addressed. The shapes of concentration-,
exposure-, or dose-response curves, and whether those curves are linear across the range of
ambient air exposures, can also be an important consideration in characterizing the public health
impacts associated with pollutant exposures. The shapes of these curves across ambient air
exposures occurring under air quality conditions that meet existing standards may be of
particular interest in the NAAQS reviews. Sources of variability and uncertainty in interpreting
concentration-, exposure-, and dose-response relationships can include a limitation in the data
available toward the lower and upper ends of the concentration range, the possible influence of
exposure measurement error over the range of concentrations, and variability in response among
individuals with respect to air pollution health effects. These sources of uncertainty and
variability tend to smooth and "linearize" concentration-, exposure-, and dose-response functions
and thus can obscure the existence of nonlinear relationships and thresholds. These sources of
variability and uncertainty may explain why the available exposure-response data from
epidemiologic studies of ambient air concentrations for some environmental pollutants (e.g., PM,
O3, Pb, environmental tobacco smoke, radiation) do not exhibit population-level thresholds for
cancer or noncancer health effects, even though likely mechanisms include nonlinear processes
for some key events.
A.7.2.1.1.5. Strength and Specificity of Associations
A-47
-------
In evaluating the strength of an observed association in epidemiologic studies, the ISAs
consider the magnitude, statistical precision (i.e., informed by width of confidence intervals), and
the specificity of health outcomes across multiple studies. In large studies that adequately
account for potential confounding factors, strong associations can serve to increase confidence
that findings are not due to a weak unmeasured confounder, chance, or other biases. However,
the health effects evaluated in the ISAs tend to have multiple contributing factors (e.g., genetics,
disease, lifestyle, environmental), and the magnitude of the contribution from air pollution
exposures will depend on the prevalence of other risk factors in the study population. Thus, in
studies that appropriately account for potential confounding factors and other sources of bias a
small effect size does not rule out there being a causal relationship with the air pollutant, and
such an effect can be important from a public health perspective if it impacts large segments of
the population. A small effect can represent a shift in the distribution of responses in the study
population and may increase the proportion of individuals with clinically important outcomes.
A.7.2.1.1.6. Temporality of the Observed Association
Temporality of an observed association refers to the temporal sequence of exposure and
observed effects. For a causal relationship, pollutant exposure must happen before effects.
Experimental animal and controlled human exposure studies demonstrating exposure-induced
health effects provide strong support for appropriate temporal relationships between exposures
and effects reported in observational epidemiologic studies. Not all observational studies provide
evidence of temporality. For example, cohort studies, by design, generally better suited to
address the consideration of the temporal sequence of exposure and effect than cross-sectional
studies.
A.7.2.2. ISA framework for making causality determinations
Using the aspects of the evidence described above to make judgments related to causality,
the ISAs assess the relevant scientific literature to draw conclusions on the causal nature of the
relationships between relevant pollutant exposures and health effects. These "causality
determinations" reflect overall confidence in cause-effect relationships based on the strengths
and limitations of the full body of evidence, integrated within and across disciplines. In its
review of the current ISA causality framework, the NASEM supported this approach, noting that
"[a] weight of evidence approach, which combines assessment of the scientific literature with
expert judgment to weigh that complex literature, is a scientifically defensible approach for the
ISA causal determination framework" (NASEM, 2022, p. 126). The ISAs evaluate evidence for
major health outcome categories or groups of related endpoints (e.g., respiratory effects),
characterizing the strengths and limitations of evidence for individual endpoints within the
broader category. Limitations in the evidence base can result from the consistent presence of
A-48
-------
uncertainties within a group of studies (e.g., studies similarly affected by confounding, exposure
error, species extrapolation, etc.) or uncertainties that exist across the broader body of evidence
(e.g., inconsistent evidence across disciplines, lack of coherence). The ISAs generally rely on
qualitative uncertainty evaluations, though quantitative analysis approaches such as meta-
regression are used in some situations. Rigorous external peer-review by the CASAC and
pollutant-specific expert panels is critical to informing ISA conclusions on uncertainty and/or
bias in the body of evidence supporting causality determinations.
The ISA causality determinations are articulated using a framework with a five-level
hierarchy based on the weight of the evidence for causation (Table A-l 1). The NASEM endorsed
this approach, noting that the five categories for classifying causality determinations "are
scientifically defensible given the precautionary nature of the CAA" (NASEM, 2022, p. 3). The
standardized language used in the framework to describe specific determinations is adapted from
sources across the federal government and the wider scientific community, especially the EPA
Guidelines for Carcinogen Risk Assessment (U.S EPA, 2005), U.S. Surgeon General's report,
The Health Consequences of Smoking CDC, 2004, and NAS IOM document, Improving the
Presumptive Disability Decision-Making Process for Veterans (IOM, 2008). Table A-ll presents
the human health descriptors for each of the determinations in the ISA causality framework.43
43 Characteristics of the ecological and other welfare effects evidence supporting each of the five causal
determinations are presented in the 2015 Preamble to the ISAs (U.S. EPA, 2015).
A-49
-------
Table A-ll. Causality determinations for health outcomes.
Descriptor
Evidence Characteristics
Causal relationship
Evidence is sufficient to conclude that there is a causal relationship with relevant
pollutant exposures. That is, the pollutant exposures have been shown to result in
health effects in studies in which chance, confounding, and other biases can be
ruled out with reasonable confidence. A "causal" relationship is generally based on
multiple high-quality studies conducted by different research groups. Evidence
supporting this determination can include controlled human exposure studies that
consistently demonstrate effects and/or observational studies reporting consistent
health effect associations that, when considered in light of study quality and
coherence with other lines of evidence (i.e., controlled human exposure studies,
animal toxicological studies, and mode of action information), cannot be explained
by plausible alternatives.
Likelytobeacausal
relationship
Evidence is sufficient to conclude that a causal relationship is likely to exist with
relevant pollutant exposures. That is, the pollutant exposures have been shown to
result in health effects in studies where results are not explained by chance,
confounding, and other biases, but uncertainties remain in the evidence overall. A
"likely to be causal" relationship is generally based on multiple high-quality studies
conducted by different research groups. Evidence supporting this determination can
include 1) multiple high-quality observational studies consistently reporting health
effect associations, but with uncertainty remaining related to potential confounding
and/or limited coherence with other lines of evidence (i.e., controlled human
exposure studies, animal toxicological studies, mode of action information) or 2)
consistent evidence in animal models and/or in vitro models (e.g., for cancer-related
effects) that can be reasonably extrapolated to human health, but limited availability
of human data.
Suggestive of, but
not sufficient to infer,
a causal relationship
Evidence is suggestive of, but not sufficient to infer, a causal relationship with
relevant pollutant exposures. That is, the pollutant exposures have been shown to
result in health effects, but chance, confounding, and bias cannot be confidently
ruled out. Evidence supporting a "suggestive" relationship can be comprised of
studies of varying quality that may be generally supportive of pollutant-related
effects, but not entirely consistent and with limited coherence across lines of
evidence. A suggestive determination can be reached with relatively small bodies of
evidence, or, in rare cases, one high quality study.
Inadequate to infer a
causal relationship
Evidence is inadequate to determine that a causal relationship exists with relevant
pollutant exposures. That is, the evidence supporting an "inadequate" relationship is limited
and available studies are of insufficient quantity, quality, consistency, and/or statistical
power to permit a conclusion regarding the presence or absence of an effect.
Not likely to be a
causal relationship
Evidence indicates there is no causal relationship with relevant pollutant exposures.
Several adequate studies, covering the full range of exposures that human beings
are known to encounter and considering at-risk populations and lifestages, are
consistent in not showing an effect at any level of exposure concentration.
Each level of the causality hierarchy is delineated by the degree to which chance,
confounding, and other biases can be ruled out as explanations of study results with reasonable
confidence. A conclusion on which level within the hierarchy best fits a particular body of
A-50
-------
evidence is informed by considering the aspects of the evidence described above (i.e.,
consistency, coherence, biological plausibility, biological gradient, strength of associations,
experimental evidence, temporality, and specificity). ISAs use these aspects to guide evidence
integration both across populations and within groups that share characteristics with the potential
to modify exposure-response relationships, particularly characteristics that may place a group at
higher risk. As noted by the NASEM, "[heightened human response can be due to age,
comorbidities, or other environmental, socio-economic, behavioral, epigenetic or genetic factors"
NASEM, 2022, p. 4). Given this, the NASEM cautioned that "considering, or highlighting, only
overall average population or broad ecosystem effects can obscure causal relationships that exist
for more sensitive subgroups, subspecies, communities, or ecosystems" (NASEM, 2022, p. 128).
Consistent with this advice, the ISAs take into consideration the heterogeneity in exposure-
response relationships often demonstrated between individuals and populations. When studies
informing causality determinations consistently demonstrate elevated pollutant-related risks in
populations with particular characteristics, the resulting determinations reflect the strength of the
evidence for effects in those populations.
A.7.2.3. At-risk populations
As noted above, some populations and lifestages may be at greater risk of criteria
pollutant-related health effects than the general population. Higher risks could be due to intrinsic,
extrinsic, and/or exposure-related factors. Intrinsic factors such as genetics, lifestage, or disease
status can contribute to larger or more serious responses to a particular pollutant exposure, and
physiological differences between groups (e.g., differences in breathing patterns between
children and adults) can contribute to higher internal pollutant doses in some populations.
Extrinsic factors (e.g., nutritional status) and exposure-related factors (e.g., working outdoors,
living near roadways or other pollution sources) can also contribute to increased risk, and many
of the characteristics commonly used to classify populations potentially at increased risk (e.g.,
race, SES, educational attainment) are surrogates for the combined impact of several intrinsic,
extrinsic, and exposure-related factors. The co-occurrence of risk factors in some populations,
including those with environmental justice concerns or minority groups who may have legacy
impacts for some risk factors, presents a complex public health challenge. A critical part of
characterizing the public health impacts of criteria pollutant exposures under the NAAQS is
identifying the specific populations that are at greater risk of pollutant-related health effects and
understanding, where possible, the combinations of intrinsic, extrinsic, and exposure-related
factors that confer the greatest risk.
The scientific community has used a variety of terms to classify the populations that may
be at increased risk of pollutant-related health effects. These terms, which have been defined
A-51
-------
inconsistently across the scientific literature, include susceptible, vulnerable, and sensitive
(Vinikoor-Imler et al., 2014; Sacks et al., 2011; U.S. EPA, 2009; U.S EPA, 2010). The lack of
consensus in terminology across the scientific community led previous reviews and early ISAs to
adopt the term "susceptible populations" to encompass the various factors that could confer
increased risk (Vinikoor-Imler et al., 2014; Sacks et al., 2011; U.S. EPA, 2009; U.S EPA, 2010).
However, this terminology proved problematic because the broader scientific community often
describes susceptible populations as those at increased risk specifically due to biological or
intrinsic factors such as pre-existing disease or lifestage. Therefore, starting with the 2013 ISA
for Ozone and Related Photochemical Oxidants (U.S. EPA, 2013b) the term "at-risk" was
adopted to encompass the broad range of intrinsic, extrinsic, and exposure-related factors that
may confer increased risk of criteria pollutant-related health effects in particular populations and
lifestages.
In assessing the overall public health impact of criteria pollutant exposures, the ISA
identifies, evaluates, and characterizes risk factors to inform conclusions on the populations and
lifestages that may be at increased risk. As described further below, the ISA uses a structured
framework to characterize potential risk factors and guide evaluation of the evidence across
scientific disciplines to assess the overall confidence that a specific factor may result in a
population or lifestage being at increased risk of an air pollutant-related health effect. In doing
so, the ISA draws from the evidence integration underlying causality determinations, with a
focus on the epidemiologic, controlled human exposure, and animal toxicological studies that
provide information on pollutant-related effects in particular populations or lifestages as well as
available information on differential exposures and dosimetry.
Regarding epidemiologic studies, the ISA focuses particularly on studies that include
stratified analyses, or analyses of effect measure modification, and on studies that examine
effects that are overwhelmingly or exclusively present in specific populations. Stratified analyses
and analyses of effect measure modification can compare various populations or lifestages
exposed to similar air pollutant concentrations within the same study design. Studies that
evaluate effects in specific populations or lifestages can provide evidence of increased risk in
populations that are uniquely affected (e.g., lung function development in children; heart failure,
heart attacks, or strokes in people with pre-existing cardiovascular disease). When evaluating
results across epidemiologic studies, consistent with the approach to informing causality
determinations, emphasis is placed on patterns or trends in results in the various populations
evaluated.
Some controlled human exposure and animal toxicological studies evaluate potential risk
factors, such as genetic background or health status (e.g., pre-existing asthma), though study
participants with serious health conditions are usually excluded from controlled human exposure
A-52
-------
studies and limitations in animal models of human disease often result in important uncertainties.
However, when available, these experimental studies are important for establishing coherence
across disciplines. They can provide information about the independent effects of the air
pollutant under evaluation as well as the biological plausibility of effects observed in
epidemiologic studies examining specific populations. Additionally, dosimetry studies can
further inform the plausibility of a population being at increased risk by demonstrating whether
the deposition and distribution of an air pollutant within the body varies across populations or
lifestages.
An important consideration in evaluating the health evidence for potential at-risk
populations or lifestages is variability across studies in how those groups are defined. For
example, risk in populations with well-controlled pre-existing disease (e.g., asthma,
hypertension) could be substantially different from the risk in populations with uncontrolled
disease. Variability across studies in how potential at-risk populations are defined can similarly
exist for other factors (e.g., body mass index vs. other indicators of body composition, various
indicators of SES, and various age ranges used to define lifestages). A related consideration is
variability within populations or lifestages, such as behavioral differences, biological differences,
and adherence to medical treatments. ISAs consider such sources of variation where relevant
because they may affect the extent to which studies can reliably identify a population or lifestage
that may be at increased risk of pollutant-related effects.
In addition to the health evidence, ISAs consider evidence for differential exposures
when evaluating support in the evidence for the identification of populations and lifestages that
may be at increased risk. When available, data from studies examining pollutant exposures in
specific populations can be integrated with data for health effects in those populations. Such
combinations of exposure and health data can inform the ISA's evaluation of the intrinsic,
extrinsic, and exposure-related factors that may confer the greatest risk to criteria pollutant-
related health effects.
The ISA's characterization of risk factors consists of evaluating the evidence across
scientific disciplines and assessing overall confidence that a specific factor may result in a
population or lifestage being at increased risk of an air pollutant-related health effect. The ISA
uses a structured framework with four categories to characterize the evidence for at-risk
populations. Categories are "adequate evidence," "suggestive evidence," "inadequate evidence,"
and "evidence of no effect." These categories are described below in Table A-12.
A-53
-------
Table A-12. Characterization of evidence for factors potentially increasing the risk
of pollutant-related health effects.
Classification
Health Effects
Adequate evidence
There is substantial, consistent evidence within a discipline to conclude that a factor
results in a population or lifestage being at increased risk of air pollutant-related
health effect(s) relative to some reference population or lifestage. Where applicable,
this evidence includes coherence across disciplines. Evidence includes multiple high-
quality studies.
Suggestive evidence
The collective evidence suggests that a factor results in a population or lifestage
being at increased risk of air pollutant-related health effect(s) relative to some
reference population or lifestage, but the evidence is limited due to inconsistency
within a discipline or, where applicable, a lack of coherence across disciplines.
Inadequate evidence
The collective evidence is inadequate to determine whether a factor results in a
population or lifestage being at increased risk of air pollutant-related health effect(s)
relative to some reference population or lifestage. The available studies are of
insufficient quantity, quality, consistency, and/or statistical power to permit a
conclusion to be drawn.
Evidence of no effect
There is substantial, consistent evidence within a discipline to conclude that a factor
does not result in a population or lifestage being at increased risk of air pollutant-
related health effect(s) relative to some reference population or lifestage. Where
applicable, the evidence includes coherence across disciplines. Evidence includes
multiple high-quality studies.
A.7.3. Develop the Integrated Synthesis
The Integrated Synthesis draws from the detailed assessment of the evidence in the ISA
chapters. It provides a concise synopsis of the ISA conclusions and synthesis of key information
and findings considered in characterizing pollutant exposures and relationships with health
effects. The Integrated Synthesis includes summaries of policy-relevant information for each
topic area covered in the ISA chapters, including atmospheric science, sources, and
environmental distribution; exposure, biomarkers, and toxicokinetics; the nature of health effects
associated with pollutant exposure, including causality determinations; and the human
populations and lifestages at increased risk of the effects of pollutant exposure. The Integrated
Synthesis also summarizes the evidence and conclusions for other policy-relevant issues. These
vary across assessments and can include the exposure durations, metrics, and concentrations
eliciting health effects; the shapes and statistical precision of concentration-, exposure-, or dose-
response functions; and the potential adversity and public health significance of certain health
effects.
A-54
-------
A.8. PEER REVIEW OF AND PUBLIC COMMENT ON THE
DRAFT ISA
Section 109(d)(2) of the Clean Air Act (CAA) addresses the appointment and advisory
functions of an independent scientific review committee. Section 109(d)(2)(A) requires the
Administrator to appoint this committee, which is to be composed of "seven members including
at least one member of the National Academy of Sciences, one physician, and one person
representing state air pollution control agencies." Section 109(d)(2)(B) provides that the
independent scientific review committee "shall complete a review of the criteria... and the
national primary and secondary ambient air quality standards.. .and shall recommend to the
Administrator any new... standards and revisions of existing criteria and standards as may be
appropriate ..." Since the early 1980s, this independent review function has been performed by
the CAS AC of the EPA's Science Advisory Board.
Most preliminary work of the CASAC in reviewing the draft ISA is done by an ad hoc
panel of independent subject matter experts. Ad-hoc panels for each review consist of members
of the CASAC supplemented by additional independent experts in the subject matter for that
review. CASAC panels are convened to provide broad expertise related to the particular criteria
pollutant under evaluation and the science-policy issues important for the review. Given the
breadth of scientific and technical information evaluated during NAAQS reviews, CASAC
panels reflect a wide range of expertise. The specific expertise varies across panels, but typically
includes expert knowledge of atmospheric science, human exposure, dosimetry, toxicology,
epidemiology, medicine, public health, biostatistics, and risk assessment. Consistent with
NASEM recommendations (NASEM, 2022, p. 8), critical disciplines are often represented by
multiple panel members in order to facilitate advice from a range of perspectives. Ad-hoc
CASAC panels are generally chaired by a CASAC member. The CASAC panels convene at
public meetings that are announced in the Federal Register and that provide an opportunity for
public comment. Draft advisory reports, conveying recommendations to the EPA on the draft
ISA, are prepared by the ad-hoc panels and are transmitted to the full CASAC for discussion and
deliberation. These reports convey advice to the EPA regarding the ISA conclusions (e.g.,
causality determinations, at-risk populations, exposure-response relationships, etc.) and the
ISA's approaches to evaluating, weighing, and integrating evidence to reach those conclusions,
and they often identify additional studies that the CASAC believes should be included in the
ISA. If the full CASAC determines the contents of a report are appropriate, the CASAC will
adopt the report and transmit it to the EPA to reflect its statutorily mandated advice to the
Agency.
The EPA carefully considers advice received from the CASAC and comments from the
public in revising and updating the draft ISA document. This may include consideration of
A-55
-------
additional studies identified during peer review that meet the ISA's scoping and study quality
criteria. After appropriate revisions are made, the final document is made available on the EPA
website. A notice announcing the availability of the final ISA is published in the Federal
Register. More information on EPA's peer review practices can be found in EPA's Peer Review
Handbook and via the EPA CASAC peer review website
{https://casac. epa. gov/ords/sab/r/sabapex/casac/home).
A.9. QUALITY MANAGEMENT
The EPA has an agency-wide quality assurance (QA) policy outlined in the EPA Quality
Manual for Environmental Programs (see CIO 2105-P-01.1N) and follows the specifications
outlined in EPA Order CIO 2105.1. As required by CIO 2105.1, the EPA Office of Research and
Development (ORD) maintains a Quality Management Program, which is documented in an
internal Quality Management Plan (QMP). All environmental information operations (EIO),
including the IS As, are subject to the EPA's Quality Management Program requirements for a
Quality Management Plan (QMP) and a Quality Assurance Project Plan (QAPP). Adherence to
the ORD QMP and the ISA program-level QAPP, ensures that all data generated, collected,
evaluated, or used in an ISA are "of the type and quality needed and expected for their intended
use" and that all information disseminated by the ISAs adhere to a high standard for quality
including objectivity, utility, and integrity. The EPA's Center for Public Health and
Environmental Assessment (CPHEA) QA managers (QAMs) are responsible for the review and
approval of quality-related documentation. The CPHEA ISA scientists are responsible for the
evaluation of all inputs to the ISAs, including primary (new) and secondary (existing) data from
others, to ensure their quality is appropriate for use in the ISAs. CPHEA adheres to Data Quality
Objectives, which identify the most appropriate inputs to the science assessment, and CPHEA
provides QA instruction to researchers involved with environmental information operations.
The approaches utilized to search the literature and to select and evaluate studies were
detailed in the preceding subsections. Generally, the ISA scientists rely on scientific information
found in peer-reviewed journal articles, books, and government reports. The ISAs can integrate
information that is extracted from multiple sources to create new figures, tables, or summation,
which is subject to rigorous quality assurance measures to ensure their accuracy. Documentation
of the quality of extracted information includes the types of QA/QC checks performed and the
approach to verifying information extractions (e.g., verification by a second individual). The
QA/QC checks for extracted information include comparison of entries to information from the
original publication, checking conversions (e.g., ppm to |ig/m3), confirming effect levels, and
inserting and verifying electronic citations that are converted to HERO links. In addition, QA
A-56
-------
reviews of key information from all types of health effect studies are performed. Furthermore,
publicly available databases (e.g., HERO) have their own QA processes.
The ISAs are designated as Highly Influential Scientific Assessments (HISA) and
classified as ORD QA Category A. Category A designations require reporting of all critical QA
activities, including audits. During assessment development, the ISAs undergo periodic quality
audits. A Technical Systems Audit (TSA) of each ISA is conducted by the EPA or an
independent contractor to verify that all QA/QC procedures were adequately performed and
documented. The ISAs are subjected to management and QA clearance review, and during this
step, the CPHEA QA Manager verifies that the EPA QA requirements are met.
The EPA is committed to providing public access to environmental information. The
EPA's Information Quality Guidelines for Ensuring and Maximizing the Quality, Objectivity,
Utility, and Integrity of Information Disseminated by the Environmental Protection Agency
reflects our commitment to the quality of the information we disseminate.
A-57
-------
A. 10. REFERENCES
Bang, H and Robins, JM (2005). Doubly robust estimation in missing data and causal inference
models. Biometrics 61(4): 962-972.
Barr, RG, Herbstman, J, Speizer, FE and Camargo, CA, Jr. (2002). Validation of self-reported
chronic obstructive pulmonary disease in a cohort study of nurses. Am J Epidemiol
155(10): 965-971.
CDC (2004). National Center for Chronic Disease Prevention and Health Promotion (US)
Office on Smoking and Health. The Health Consequences of Smoking—50 Years of
Progress: A Report of the Surgeon General. U.S. Department of Health and Human
Services. Centers for Disease Control and Prevention Atlanta (GA). Available at:
https://pubmed. ncbi. nlm. nih.gov/24455 788/.
Cronin, MTD, Bajot, F, Enoch, SJ, Madden, JC, Roberts, DW and Schwobel, J (2009). The In
Chemico-In Silico Interface: Challenges for Integrating Experimental and Computational
Chemistry to Identify Toxicity. Alternatives to Laboratory Animals 37(5): 513-521.
Diaz, I (2019). Machine learning in the estimation of causal effects: Targeted minimum loss-
based estimation and double/debiased machine learning. Biostatistics 21(2): 353-358.
Diaz, I, Williams, N, Hoffman, KL and Schenck, EJ (2023). Nonparametric causal effects based
on longitudinal modified treatment policies. Journal of the American Statistical
Association 118(542): 846-857.
ECHA (2016). New approach methodologies in regulatory science: Proceedings of a scientific
workshop, Helsinki, 19-20 April 2016. Helsinki, Finland.
http://dx. doi. org/10.2823/543644.
Ertefaie, A and Stephens, DA (2010). Comparing approaches to causal inference for longitudinal
data: Inverse probability weighting versus propensity scores. The International Journal of
Biostatistics 6(2): 14.
European Commission, Joint Research Centre, Adcock, I, Novotny, T, Nic, M, Dibusz, K,
Hynes, J, Marshall, L and Gribaldo, L (2020). Advanced non-animal models in
biomedical research - Respiratory tract diseases. Publications Office of the European
Union. Available at: https://data.europa.eu/doi/10.2760/725821.
Gee, GC and Payne-Sturges, DC (2004). Environmental health disparities: A framework
integrating psychosocial and environmental concepts. Environ Health Perspect 112(17):
1645-1653.
Gelman, A and Greenland, S (2019). Are confidence intervals better termed "uncertainty
intervals"? BMJ (Clinical Research Edition) 366: 15381.
A-58
-------
Goldman, GT, Mulholland, JA, Russell, AG, Strickland, MJ, Klein, M, Waller, LA and Tolbert,
PE (2011). Impact of exposure measurement error in air pollution epidemiology: Effect
of error type in time-series studies. Environ Health 10: 61.
Greenland, S (2000). An introduction to instrumental variables for epidemiologists. Int J
Epidemiol 29: 722-729.
Greenland, S, Senn, SJ, Rothman, KJ, Carlin, JB, Poole, C, Goodman, SN and Altman, DG
(2016). Statistical tests, P values, confidence intervals, and power: a guide to
misinterpretations. Eur J Epidemiol 31(4): 337-350.
Hamel, C, Kelly, SE, Thavorn, K, Rice, DB, Wells, GA and Hutton, B (2020). An evaluation of
DistillerSR's machine learning-based prioritization tool for title/abstract screening -
impact on reviewer-relevant outcomes. BMC Medical Research Methodology 20(1): 256.
Hernan, MA and Robins, JM (2016). Using big data to emulate a target trial when a randomized
trial is not available. Am J Epidemiol 183(8): 758-764.
Hew (1964). Smoking and health: Report of the advisory committee to the surgeon general of the
public health service. U.S. Department of Health, Education, and Welfare. Washington,
DC. http://profiles.nlm.nih.gov/ps/retrieve/ResourceMetadata/NNBBMQ.
Hill, AB (1965). The environment and disease: Association or causation? Proceedings of the
Royal Society of Medicine 58(5): 295-300.
Howard, BE, Phillips, J, Tandon, A, Maharana, A, Elmore, R, Mav, D, Sedykh, A, Thayer, K,
Merrick, BA, Walker, V, Rooney, A and Shah, RR (2020). SWIFT-Active Screener:
Accelerated document screening through active learning and integrated recall estimation.
Environ Int 138: 105623.
IARC (2006). IARC monographs on the evaluation of carcinogenic risks to humans: Preamble.
World Health Organization. Lyon, France. Available at:
http://monographs.iarc.fr/ENG/Preamble/.
IOM, Ed. (2008). Improving the presumptive disability decision-making process for veterans.
National Academies Press Washington, DC.
Keil, AP, Buckley, JP, Katie, MO, Ferguson, KK, Zhao, S and White, AJ (2020). A quantile-
based g-computation approach to addressing the effects of exposure mixtures. Environ
Health Perspect 128(4): 47004.
Lash, TL, Fox, MP, MacLehose, RF, Maldonado, G, McCandless, LC and Greenland, S (2014).
Good practices for quantitative bias analysis. Int J Epidemiol 43(6): 1969-1985.
Muhajarine, N, Mustard, C, Roos, LL, Young, TK and Gelskey, DE (1997). Comparison of
survey and physician claims data for detecting hypertension. J Clin Epidemiol 50(6):
711-718.
A-59
-------
Murgia, N, Brisman, J, Claesson, A, Muzi, G, Olin, AC and Toren, K (2014). Validity of a
questionnaire-based diagnosis of chronic obstructive pulmonary disease in a general
population-based study. BMC Pulmonary Medicine 14: 49.
NASEM (2017). National Academies of Sciences, Engineering, and Medicine 2017. Controlled
Human Inhalation-Exposure Studies at EPA. National Academies Press (US).
Washington (DC). Available at: http://dx.doi.org/10.17226/24618.
NASEM (2022). National Academies of Sciences, Engineering, and Medicine 2022. Advancing
the Framework for Assessing Causality of Health and Welfare Effects to Inform National
Ambient Air Quality Standard Reviews. The National Academies Press. Washington,
DC. Available at: https://doi.org/10.17226/26612.
Richardson, A, Hudgens, MG, Gilbert, PB and Fine, JP (2014). Nonparametric bounds and
sensitivity analysis of treatment effects. Statistical Science 29(4): 596-618.
Robins, JM, Hernan, MA and Brumback, B (2000). Marginal structural models and causal
inference in epidemiology. Epidemiology 11(5): 550-560.
Rothman, KJ and Greenland, S (1998). Modern Epidemiology. Lippincott-Raven. Philadelphia,
PA.
Sacks, JD, Stanek, LW, Luben, TJ, Johns, DO, Buckley, BJ, Brown, JS and Ross, M (2011).
Particulate-matter induced health effects: Who is susceptible? Environ Health Perspect
119(4): 446-454.
Savitz, DA and Forastiere, F (2021). Do pooled estimates from meta-analyses of observational
epidemiology studies contribute to causal inference? Occup Environ Med 78(9): 621-622.
Savitz, DA, Wellenius, GA and Trikalinos, TA (2019). The Problem With Mechanistic Risk of
Bias Assessments in Evidence Synthesis of Observational Studies and a Practical
Alternative: Assessing the Impact of Specific Sources of Potential Bias. Am J Epidemiol
188(9): 1581-1585.
Sheppard, L, Slaughter, JC, Schildcrout, J, Liu, JS and Lumley, T (2005). Exposure and
measurement contributions to estimates of acute air pollution effects. J Expo Anal
Environ Epidemiol 15(4): 366-376.
Stitelman, OM, De Gruttola, V and van der Laan, MJ (2012). A general implementation of
TMLE for longitudinal data applied to causal inference in survival analysis. The
International Journal of Biostatistics 8(1): 26.
Stuart, EA (2010). Matching methods for causal inference: A review and a look forward.
Statistical Science 25(1): 1-21.
Thayer, KA, Angrish, M, Arzuaga, X, Carlson, LM, Davis, A, Dishaw, L, Druwe, I, Gibbons, C,
Glenn, B, Jones, R, Kaiser, JP, Keshava, C, Keshava, N, Kraft, A, Lizarraga, L, Persad,
A, Radke, EG, Rice, G, Schulz, B and Vetter, N (2022). Systematic Evidence Map
A-60
-------
(SEM) template: Report format and methods used for the US EPA Integrated Risk
Information System (IRIS) program, Provisional Peer-Reviewed Toxicity Value
(PPRTV) program, and other "fit for purpose" literature-based human health analyses.
Environ Int 169(Elsevier): 107468.
Toren, K, Brisman, J and Jarvholm, B (1993). Asthma and asthma-like symptoms in adults
assessed by questionnaires: A literature review. Chest 104(2): 600-608.
U.S. EPA (1982). Air quality criteria for oxides of nitrogen. U.S. Environmental Protection
Agency, Environmental Criteria and Assessment Office. Research Triangle Park, NC.
EPA/600/8-82-026.
U.S. EPA (1991). Guidelines for Developmental Toxicity Risk Assessment. U.S. Environmental
Protection Agency. Available at:
https://nepis. epa.gov/exe/ZyPURL. cgi?Dockey=30002L 7L. txt.
U.S. EPA (1993). Air Quality Criteria for Oxides of Nitrogen. Office of Health and
Environmental Assessment. Environmental Criteria and Assessment Office. Washington,
DC U.S. EPA. EPA-600/8-91-049aF-cF. August 1993. Available at:
https://cfpub. epa.gov/ncea/risk/recordisplay. cfm ?deid=40179.
U.S. EPA (1996). Guidelines for Reproductive Toxicity Risk Assessment. U.S. Environmental
Protection Agency, Risk Assessment Forum. Washington, DC. EPA/630/R-96/009.
Available at: https://nepis.epa.gov/Exe/ZyPURL.cgi?Dockey=30004YQB. txt.
U.S. EPA (1998). Guidelines for neurotoxicity risk assessment. U.S. Environmental Protection
Agency, Risk Assessment Forum. Washington, DC. EPA/630/R-95/001F. Available at:
http://www.epa.gov/risk/guidelines-neurotoxicity-risk-assessment.
U.S. EPA (2005). Guidelines for carcinogen risk assessment. Risk Assessment Forum.
Washington, DC. EPA630P03001F. Available at:
https://www.epa.gov/sites/production/files/2013-
09/documents/cancer guidelines final 3-25-05.pdf.
U.S. EPA (2008). Integrated Science Assessment for Oxides of Nitrogen - Health Criteria.
Office of Research and Development, National Center for Environmental Assessment.
Research Triangle Park, NC. U.S. EPA. EPA/600/R-08/071. July 2008. Available at:
http://cfpub. epa.gov/ncea/cfm/recordisplay. cfm?deid=194645.
U.S. EPA (2009). Integrated Science Assessment for Particulate Matter (Final Report). Office of
Research and Development, National Center for Environmental Assessment. Research
Triangle Park, NC. U.S. EPA. EPA-600/R-08-139F. December 2009. Available at:
https://cfpub. epa. gov/ncea/risk/recordisplay. cfm ?deid=216546.
U.S. EPA (2010). Integrated Science assessment (ISA) for Carbon Monoxide (Final report). U.S.
Environmental Protection Agency. Washington, DC. EPA/600/R-09/019F. Available at:
http://cfpub. epa.gov/ncea/cfm/recordisplay. cfm?deid=218686.
A-61
-------
U.S. EPA (2013a). Integrated Science Assessment (ISA) for Lead (Final Report). Office of
Research and Development, National Center for Environmental Assessment.
Washington, DC. U.S. EPA. EPA/600/R-10/075F. July 2013. Available at:
https: //assessments. epa.gov/isa/document/&deid=255 721.
U.S. EPA (2013b). Integrated Science Assessment of Ozone and Related Photochemical
Oxidants (Final Report). Office of Research and Development, National Center for
Environmental Assessment. Research Triangle Park, NC. U.S. EPA. EPA-600/R-10-
076F. February 2013. Available at:
https: //nepis. epa.gov/Exe/ZyPURL. cgi?Dockey=P 100KETF. txt.
U.S. EPA (2015). Preamble To The Integrated Science Assessments (ISA). Office of Research
and Development, National Center for Environmental Assessment. Washington, DC.
U.S. EPA. EPA/600/R-15/067. Available at:
https://ordspub.epa.gov/ords/eims/eimscomm.getfile7p downloadid=526136.
U.S. EPA (2016). Integrated Science Assessment for Oxides of Nitrogen-Health Criteria. U.S.
Environmental Protection Agency, Office of Research and Development, National Center
for Environmental Assessment. Research Triangle Park, NC. EPA/600/R-15/068.
Available at: https ://cfpub. epa.gov/ncea/isa/recordisplay. cfm ?deid=310879.
U.S. EPA (2018). Strategic plan to promote the development and implementation of alternative
test methods within the TSCA program. Office of Chemical Safety and Pollution
Prevention. Washington D.C. EPA-740-R1-8004. Available at:
https://www.epa.gov/sites/production/files/2018-06/documents/epa alt stratjplan 6-20-
18 clean final.pdf.
U.S. EPA (2019a). Integrated Science Assessment (ISA) for Particulate Matter (Final Report).
U.S. Environmental Protection Agency, Office of Research and Development, National
Center for Environmental Assessment. Washington, DC. U.S. EPA. EPA/600/R-19/188.
December 2019. Available at: https://www.epa.gov/naaqs/particulate-matter-pm-
standards-i ntegraled-science-assessmenls-current-re view.
U.S. EPA (2019b). Guidelines for Human Exposure Assessment. Risk Assessment Forum.
Washington, DC. EPA/100/B-19/001. Available at:
https://www. epa. gov/sites/production/files/2020-
01/documents/guidelines for human exposure assessment Jinal2019.pdf.
U.S. EPA (2020a). Integrated Science Assessment for Oxides of Nitrogen, Oxides of Sulfur, and
Particulate Matter—Ecological Criteria (Final). U.S. Environmental Protection Agency,
Office of Research and Development, Center for Public Health and Environmental
Assessment. Research Triangle Park, NC. EPA/600/R-20/278. Available at:
https ://cfpub. epa.gov/ncea/isa/recordisplay. cfm ?deid=349473.
U.S. EPA (2020b). Integrated Science Assessment (ISA) for Ozone and Related Photochemical
Oxidants (Final). U.S. Environmental Protection Agency, Office of Research and
Development. Washington, DC. EPA/600/R-20/012. Available at:
https://ordspub.epa.gov/ords/eims/eimscomm.getfile?p_download_id=540022.
A-62
-------
U.S. EPA (2021a). Health Assessment Workspace Collaborative (HAWC) Washington, DC.
U.S. EPA (2021b). EPA new approach methods work plan: reducing use of vertebrate animals in
chemical testing. Washington D.C. Available at: https://www.epa.gov/chemical-
research/epa-new-approach-methods-work-plan-reducing-use-vertebrate-animals-
chemical.
U.S. EPA (2022). ORD staff handbook for developing IRIS assessments. U.S. Environmental
Protection Agency, Office of Research and Development, Center for Public Health and
Environmental Assessment. Washington, DC. EPA 600/R-22/268. Available at:
https://cfpub. epa.gov/ncea/iris drafts/recordisplay. cfm?deid=3563 70.
U.S. EPA (2023a). LLR Online (pre-release Version 007EC800) Washington, DC.
U.S. EPA (2023b). Overview of nitrogen dioxide (N02) air quality in the United States,
Updated: June 29, 2023. Washington, DC. Available at:
https://www.epa.gov/system/files/documents/2023-06/N02_2022.pdf.
U.S. EPA (2024). Integrated Science Assessment (ISA) for Lead (Final Report). Office of
Research and Development, Center for Public Health and Environmental Assessment.
Washington, DC. U.S. EPA. EPA/600/R-23/375. January 2024. Available at:
https://ordspub.epa.gov/ords/eims/eimscomm.getfile7p downloadid=548355.
Vinikoor-Imler, LC, Owens, EO, Nichols, JL, Ross, M, Brown, JS and Sacks, JD (2014).
Evaluating potential response-modifying factors for associations between ozone and
health outcomes: A weight-of-evidence approach. Environ Health Perspect 122(11):
1166-1176.
Wasserstein, RL and Lazar, NA (2016). The ASA's statement on p-values: Context, process, and
purpose. American Statistician 70(2): 129-133.
Wasserstein, RL, Schirm, AL and Lazar, NA (2019). Moving to a world beyond "p<0.05".
American Statistician 73(S!): 1-19.
Weakley, J, Webber, MP, Ye, F, Zeig-Owens, R, Cohen, HW, Hall, CB, Kelly, K and Prezant,
DJ (2013). Agreement between obstructive airways disease diagnoses from self-report
questionnaires and medical records. Preventive Medicine 57(1): 38-42.
Weuve, J, Sagiv, SK and Fox, MP (2018). Quantitative bias analysis for collaborative science.
Epidemiology 29(5): 627-630.
WHO (2008). Uncertainty and Data Quality in Exposure Assessment. The Internaltional
Programme on Chemical Safety. Geneva. WHO.
https://apps.who. int/iris/handle/10665/44017.
WHO (2021). World Health Organization global air quality guidelines: particulate matter
(PM2.5 andPM10), ozone, nitrogen dioxide, sulfur dioxide and carbon monoxide. World
A-63
-------
Health Organization. Geneva. Available at:
https://apps.who. int/iris/handle/10665/345329.
Wing, C, Simon, K and Bello-Gomez, RA (2018). Designing difference in difference studies:
Best practices for public health policy research. Annu Rev Public Health 39(1): 453-469.
Zeger, SL, Thomas, D, Dominici, F, Samet, JM, Schwartz, J, Dockery, D and Cohen, A (2000).
Exposure measurement error in time-series studies of air pollution: Concepts and
consequences. Environ Health Perspect 108(5): 419-426.
A-64
-------
Development Team for Appendix A of Volume 2 the Integrated Review Plan
Executive Direction
Dr. Steven J. Dutton (Division Director)—Center for Public Health and
Environmental Assessment, Office of Research and Development, U.S. Environmental
Protection Agency, Research Triangle Park, NC
Dr. Scott Jenkins (Branch Chief)—Center for Public Health and Environmental
Assessment, Office of Research and Development, U.S. Environmental Protection
Agency, Research Triangle Park, NC
AUTHORS, CONTRIBUTORS, AND REVIEWERS
Authors
Dr. David M. Lehmann—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Qingyu Meng—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Dr. Scott Jenkins—Center for Public Health and Environmental Assessment, Office of
Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Dr. Alison Krajewski—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Ms. Christine Alvarez—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Contributors
Dr. Anne Barkley—Center for Public Health and Environmental Assessment, Office of
Research and Development, U.S. Environmental Protection Agency, Research Triangle
Park, NC
Dr. James Brown—Center for Public Health and Environmental Assessment, Office of
Research and Development, U.S. Environmental Protection Agency, Research Triangle
Park, NC
Ms. LaShonda Bunch—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
A-65
-------
Dr. Peter Byrley—Center for Public Health and Environmental Assessment, Office of
Research and Development, U.S. Environmental Protection Agency, Research Triangle
Park, NC
Dr. Catheryne Chiang—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Mr. Evan Coffman—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Dr. Parker Duffney—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Ms. Amanda Haddock—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Mr. Maxwell Hatala—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Kirstin Hester—Center for Public Health and Environmental Assessment, Office of
Research and Development, U.S. Environmental Protection Agency, Research Triangle
Park, NC
Ms. Cheryl Itkin—Center for Public Health and Environmental Assessment, Office of
Research and Development, U.S. Environmental Protection Agency, Research Triangle
Park, NC
Ms. Ila Kanneboyina—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Ms. Haesoo Kim—Center for Public Health and Environmental Assessment, Office of
Research and Development, U.S. Environmental Protection Agency, Research Triangle
Park, NC
Dr. Ellen Kirrane—Center for Public Health and Environmental Assessment, Office of
Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Ms. Nichole Kulikowski—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Dr. Stephen McDow—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
A-66
-------
Dr. Natalia Neal-Walthall—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC44
Dr. Nicole Olson—Center for Public Health and Environmental Assessment, Office of
Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Ms. Mira Sanderson—Center for Public Health and Environmental Assessment,
Office of Research and Development, U.S. Environmental Protection Agency,
Research Triangle Park, NC
Dr. Adrien Wilkie—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Reviewers
Dr. Britta Bierwagen—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Washington, DC
Dr. Laura Carlson—Center for Public Health and Environmental Assessment, Office of
Research and Development, U.S. Environmental Protection Agency, Research Triangle
Park, NC
Dr. Wayne Cascio—Center for Public Health and Environmental Assessment, Office of
Research and Development, U.S. Environmental Protection Agency, Research Triangle
Park, NC
Dr. Tara Greaver—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Dr. Nicole Hagan—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Dr. Mary Hutson—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Dr. Iman Hassan—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Dr. Samantha Jones—Center for Public Health and Environmental Assessment, Office
of Research and Development, U.S. Environmental Protection Agency, Research
Triangle Park, NC
Dr. Dierdre Murphy—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Dr. Liz Naess—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
44 Dr. Neal-Walthall was affiliated with U.S. EPA at the time of drafting this document.
A-67
-------
Dr. Jennifer Nichols—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Dr. Erika Sasser—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Mr. Dahnish Shams—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
Ms. Karen Wesson—Office of Air Quality Planning and Standards, Office of Air and
Radiation, U.S. Environmental Protection Agency, Research Triangle Park, NC
A-68
-------
United States Office of Air Quality Planning and Standards EPA-452/R-24-010b
Environmental Protection Health and Environmental Impacts Division March 2024
Agency Research Triangle Park, NC
------- |