Climate Change and Children's Health and Well-Being in the United States

Appendix G: Information Quality and Peer
Review Procedures

This appendix describes the information quality assurance as well as the peer review
process undertaken for the report.

ENSURING INFORMATION QUALITY

The report and its underlying analyses were conducted in accordance with the Environmental
Protection Agency's (EPA) Guidelines for Ensuring and Maximizing the Quality; Objectivity, Utility, and
Integrity of Information Disseminated by the Environmental Protection Agency,1 which follows Office
of Management and Budget (OMB) guidelines2 and implements the Information Quality Act (IQA)
(Section 515 of Public Law 106-554).* The following section in this Appendix describes the
independent, external peer review that was performed on the report.

In accordance with OMB definitions, EPA defines the basic standard of information "quality" by the
attributes of objectivity, integrity, utility, and transparency. For products meeting a higher standard
of quality, like this report, the Agency requires an appropriate level of transparency regarding data
and methods to facilitate the reproducibility of information by qualified third parties. The EPA uses
various established Agency processes (e.g., the Quality System, peer review requirements and
processes) to ensure the appropriate level of objectivity, utility, integrity, and transparency for its
products based on the intended use of the information and the resources available.

Objectivity focuses on whether the disseminated information is being presented in an accurate,
clear, complete, and unbiased manner, and as a matter of substance, is accurate, reliable, and
unbiased. The report meets the standard for objectivity, due to activities described in the following:

a)	The information disseminated was determined to be complete, accurate, and reliable based
on internal quality control measures adopted by the expert modeling teams. This included
quality checks throughout the chain of analytic steps, including developing and processing
climate projections, calibrating and validating the sectoral impact models, and checking data
to ensure that no errors occurred in the process to compile and summarize results.

b)	The information disseminated was determined to be clear, complete, and unbiased based on
multiple rounds of independent review. Consistent with guidelines described in EPA's Peer
Review Handbook,3 the underlying sectoral modeling methodologies were peer-reviewed

* The IQA requires OMB and Federal agencies to issue guidelines that "ensur[e] and maximize[e] the quality,
objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies"
(Public Law 106-554; 44 U.S.C. 3516, note). The IQA does not impose its own standard of "quality" on agency
information; instead, it requires only that an agency "issue guidelines" ensuring data quality. Following guidelines
issued by OMB, EPA released its own guidelines to implement the IQA: Guidelines for Ensuring and Maximizing the
Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency.

April 2023

1

-------
Climate Change and Children's Health and Well-Being in the United States

through scientific journal publication processes. Citations for these publications can be found
throughout the main sector chapters of the report.

The report in full was also subject to an independent, external peer review to ensure that the
information summarized by EPA was technically supported, competently performed, properly
documented, consistent with established quality criteria, and communicated clearly.

Integrity refers to security of information, such as the protection of information from unauthorized
access or revision, to ensure that the information is not compromised through corruption or
falsification. The report and its underlying analyses meet the standard for integrity due to the
strategic steps taken to ensure that the data and information remained secure. These steps included
the use of password-protected data storage repositories, password protected data transfer
technology, and multiple layers of data validation checks to ensure that the integrity was not
compromised.

Utility is the usefulness of the information to the intended users. The report and its underlying
analyses meet the standard for utility because the information disseminated provides insights
(quantitative estimates in physical and economic terms) regarding the potential direction and
magnitude of the impacts of climate change on children of the U.S. Understanding the risks posed by
climate change can inform broader assessment reports and policy decisions designed to address
these risks.

Transparency ensures access to and description of (1) the source of the data, (2) the various
assumptions employed, (3) the analytic methods applied, and (4) the statistical procedures used. The
report and its underlying analyses meet the standard for transparency for the following reasons:

a)	The technical approaches and results of the sectoral impact analyses have been published
with open access in the peer-reviewed scientific literature and are cited throughout the
report. These papers, along with their online supplementary materials, provide detailed
information on the sources of data used, assumptions employed, the analytic and statistical
methods applied, and important limitations regarding the approaches and/or how the results
should be interpreted.

b)	Each sector of the report contains a detailed technical appendix providing descriptions of the
methodologies used in estimating impacts, assumptions used, results, limitations, and
citations to the underlying literature where the reader can go for more information.

c)	Data sources for each sectoral analysis are listed at the end of the sector-specific technical
appendices.

d)	All data output from the analyses produced for this report have been posted on the report's
website. See https://www.epa.gov/cira/climate-change-and-childrens-health-and-well-being-
united-states-report

e)	Responses to comments received from the expert peer review have been posted to EPA's
Science Inventory. See https://cfpub.epa.gov/si/

April 2023

2

-------
Climate Change and Children's Health and Well-Being in the United States

CONSIDERATION OF ASSESSMENT FACTORS

When evaluating the quality, objectivity, and relevance of scientific and technical information, the
considerations that EPA takes into account can be characterized by five general assessment factors,
as found in "A Summary of General Assessment Factors for Evaluating the Quality of Scientific and
Technical Information, and the Guidance for Evaluating and Documenting the Quality of Existing
Scientific and Technical Information."4 Table 1 lays out how the assessment factors are considered to
determine whether models and data are acceptable for their intended use in this report.

Table 1: Assessment Factors for Models and Data Used in This Report

Factor

Definition

How the Factor was Considered

Soundness

The extent to which the
scientific and technical
procedures, measures,
methods, or models
employed to generate
the information are
reasonable for, and
consistent with, the
intended application.

•	Used publicly available (to the maximum extent
practicable) data reviewed for quality and accuracy with
complete metadata available. Used data included in
peer-reviewed publications. Ensured evaluation of the
scientific and technical procedures, measures, and
methods employed to generate the estimates produced
by the sectoral impact models.

•	Considered the capabilities of integrated assessment,
simple climate model, and sectoral impacts models to
examine the key analytical questions of this report (i.e.,
physical effects, economic damages, and changes in risk
from climate change) in a manner consistent with sound
scientific theory and accepted approaches.

•	Considered the extent to which the models had been
previously applied in projects of similar scope as the
Climate change Impacts and Risk Analysis (CIRA) project.
For example, the BenMAP model has been used in
similar climate and health impact analyses, and both the
National Coastal Property Model and the aeroallergen
modeling have been employed in state-level climate
vulnerability assessments.

•	Considered whether the data and code are available,
made available by EPA, or determined to not be feasible
as it is claimed as proprietary by a nonfederal business.

•	Ensured soundness by selecting sectoral impacts models
with the following criteria: sufficient understanding of
how climate change affects the sector; the existence of
data to support the methodologies; availability of
modeling applications that could be applied in this
report; based on peer reviewed literature and datasets;
and the health or economic significance of impacts and
damages in the sector to children in the U.S.

Applicability and
Utility

The extent to which the
information is relevant
for the Agency's
intended use.

• Ensured that this report uses applicable and relevant
inputs and considers the capabilities of the integrated
assessment, simple climate model, and sectoral impacts
models to examine the key analytical questions of CIRA
(i.e., changes in physical effects, economic damages, and
risk associated with climate change).

April 2023

3

-------
Climate Change and Children's Health and Well-Being in the United States

Factor

Definition

How the Factor was Considered





•	Ensured that this report and its underlying analyses are
relevant to their intended use so that the information
disseminated provides insights and methods for
quantifying the physical and economic impacts of
climate change on children at national and regional
levels.

•	Ensured sectoral impacts models are reasonable for, and
consistent with, the intended application by being
sufficiently flexible to ensure consistency in inputs and
quantifying physical impacts.

•	Ensured that models have been applied in peer-
reviewed, published studies of similar scope and rigor as
CIRA, including those described in the Fourth National
Climate Assessment.

Clarity and
Completeness

The degree of clarity
and completeness with
which the data,
assumptions, methods,
quality assurance,
sponsoring
organizations and
analyses employed to
generate the
information are
documented.

•	Ensured use of clear and complete inputs by considering
the extent to which sectoral impacts models
documented their key methods, assumptions, parameter
values, limitations, sponsoring organizations/author
affiliations, and funding information.

•	Ensured publications clearly and comprehensively
describe analytic methods used and how they apply and
build off existing bodies of research and underlying
scientific and/or economic theories.

Uncertainty and
Variability

The extent to which the
variability and
uncertainty
(quantitative and
qualitative) in the
information or in the
procedures, measures,
methods, or models are
evaluated and
characterized.

•	Ensured inputs that appropriately characterize
uncertainty and variability by considering the capabilities
of sectoral impacts models to evaluate and characterize
key sources of variability and uncertainty. Results of
these analyses are described in the underlying journal
articles, and also demonstrated in this report.

•	Reviewed the model documentation and peer-reviewed
publications and determined if a model is sufficiently
flexible and capable of evaluating important sources of
uncertainty for climate change impacts analysis.

•	Addressed key sources of uncertainty such as: projected
emissions (high versus low); regional climate variability
(uncertainty across general circulation models); ability to
capture variability in temperature and precipitation
outcomes; and effects that increasing population and
income can have on impact estimates.

Evaluation and
Review

The extent of
independent
verification, validation,
and peer review of the
information or of the
procedures, measures,
methods or models.

•	Ensured use of independently verified and validated
inputs by considering the extent to which models have
been independently peer reviewed.

•	Reviewed the documentation associated with each
model and determined if they have been independently
peer reviewed and published in scientific journals with
procedures to ensure that the methods are technically
supportable, properly documented, and consistent with
established quality criteria.

April 2023

4

-------
Climate Change and Children's Health and Well-Being in the United States

Factor

Definition

How the Factor was Considered





• Used scenarios and projections that have been

independently verified and validated (e.g., scenarios and
projections developed for the Intergovernmental Panel
on Climate Change (IPCC) and its assessments, and then
downscaled for the U.S. for use in the Fourth National
Climate Assessment by the USGCRP Scenarios Working
Group).

PEER REVIEW OF THE TECHNICAL REPORT

Consistent with guidelines described in EPA's Peer Review Handbook,5,+ this report was subject to an
independent, external peer review. The purpose of this peer review by independent, qualified, and
objective experts was to ensure that the information summarized by EPA was technically supported,
competently performed, properly documented, consistent with established quality criteria, and
communicated clearly. The methods and applications underlying the sectoral impact modeling of the
report were previously peer reviewed and published in the research literature. The application of
these methods to investigate how children may be affected by climate impacts generally is novel,
and therefore is the primary focus of the peer review. The reviewers were also asked to provide
review and feedback on whether EPA appropriately summarized results across impact, regions, and
populations.

The review was managed by a contractor (ERG) under the direction of a designated EPA peer review
leader, who prepared a peer review plan, the scope of work for the review contract, and the charge
for the reviewers. Importantly, the EPA peer review leader played no role in producing any portion of
the report. Reviewers worked individually (i.e., without contact with other reviewers, colleagues, or
EPA) to prepare written comments in response to the charge questions.

The contractor identified, screened, and selected four reviewers who had no conflict of interest in
performing the review, and who collectively met the technical selection criteria provided by EPA.

The peer review charge directed reviewers to provide responses to the following questions during
the review:

1. Does the introductory chapter clearly explain the purpose of the report and provide

appropriate context for the subsequent chapters? If not, please provide recommendations
for improvement.

f EPA has determined that this report falls under the classification of "Other Scientific and/or Technical Work Products." The
report does not meet the criteria for "influential scientific information," as defined by OMB and further described in the EPA
Peer Review Handbook, since it is not being used to support a regulatory program or policy position and does not meet one or
more of the factors listed in Section 2.2.3 of the EPA Peer Review Handbook for consideration as influential scientific
information. As a corollary, the report also cannot be considered a "highly influential scientific assessment," as defined by OMB.
This product is for science dissemination and communication purposes only and does not reflect analysis of nor
recommendations regarding any particular policy.

April 2023

5

-------
Climate Change and Children's Health and Well-Being in the United States

2.	The report has been written for an educated but general audience. In your opinion, are the
writing level and graphics appropriate for these audiences?

3.	Are the inputs and scenarios clearly explained and documented in the main report and
appendices? If not, please explain how they may be improved.

4.	Do the text, figures, and tables in the sector report chapters clearly communicate the
modeling results? If not, please explain how they may be improved.

5.	As described in the report, the technical appendices for each analysis chapter contain
detailed information regarding the methodology, assumptions, data, and full sets of
modeling results. The report chapters are intended to summarize their respective
appendices forthe more general audience of the report. Do the main sectoral chapters
properly summarize the underlying information? If not, please explain how they may be
improved.

6.	This report contains projections of whether overburdened and underserved children are
more or less likely to experience greater risks from each climate change impact compared
to children in a reference population. The full range of disproportionality (social
vulnerability) results are presented in technical appendices for the analyses where data are
sufficient to support the method. These results are brought into the chapters of the main
report in select locations to provide additional context regarding potential risks. Is the
presentation of this information in the main report and appendix appropriate, clear, and
logical?

7.	a) Does the report, including the executive summary, draw appropriate findings and
conclusions from the modeling results? If not, why? b) Does the executive summary
provide sufficient context to understand the synthesized results? If not, why? c) Can you
please provide any missing important findings or messages in the report that you have
identified in your review?

8.	Similar to other analyses presented in the report, the "heat and learning" and "heat and
emergency department visits" analyses in Chapter 3 (extreme heat) rely on peer-reviewed
and published assessments. This EPA report, however, represents the first time these
published papers have been used to project future impacts for children under climate
change. Given this will be the first time this information is made available, please provide a
close review of the methods, assumptions, and findings forthose two analyses in particular
(see yellow highlighted publications in Attachment A).

9.	Related to question #8, EPA chose to highlight the "heat and learning" analysis in Chapter 3
as the detailed analysis and offer a simpler assessment for the "heat and emergency
department visits" impact. Do you agree that the "heat and learning" results should be
showcased in the chapter? If not, please explain why.

Re-Review Charge Questions: Chapter 7 - Infectious Diseases

1. Does the Infectious Disease chapter draw appropriate findings and conclusions from the
new modeling results? Are the revised methods clearly explained? If not, please provide
recommendations for improvement.

April 2023

6

-------
Climate Change and Children's Health and Well-Being in the United States

2. Is the treatment and coverage of the social vulnerability analysis results presented

appropriately in the main chapter and in the technical appendix? We are interested in your
perspective on whether this chapter and supporting appendix maintains consistency with
the approach used throughout the report and provides sufficient detail and relevant
references. If not, please provide recommendations for improvement.

April 2023

7

-------
Climate Change and Children's Health and Well-Being in the United States

REFERENCES

1	U.S. Environmental Protection Agency. 2002. "Guidelines for ensuring and maximizing the quality, objectivity,

utility, and integrity of information disseminated by the Environmental Protection Agency." EPA/260R-02-008.
Available online at https://www.epa.gov/quality/guidelines-ensuring-and-maximizing-quality-objectivity-utility-
and-integrity-information

2	U.S. Office of Management and Budget. 2002. "Office of Management and Budget Information Quality

Guidelines." Available online at https://www.whitehouse.gov/omb/information-regulatory-affairs/information-
policy/

3	U.S. Environmental Protection Agency. 2015. "Peer Review Handbook, 4th Edition, 2015." Programs of the Office

of the Science Advisor. Available online at https://www.epa.gov/osa/peer-review-handbook-4th-edition-2015

4	U.S. Environmental Protection Agency. 2003. "A Summary of General Assessment Factors for Evaluating the

Quality of Scientific and Technical Information, and the Guidance for Evaluating and Documenting the Quality of
Existing Scientific and Technical Information." Science Policy Council. EPA 100/B-03/001

5	U.S. Environmental Protection Agency. 2015. "Peer Review Handbook, 4th Edition, 2015." Programs of the Office

of the Science Advisor. Available online at https://www.epa.gov/osa/peer-review-handbook-4th-edition-2015

April 2023

8

-------