Climate Change and Children's Health and Well-Being in the United States Appendix G: Information Quality and Peer Review Procedures This appendix describes the information quality assurance as well as the peer review process undertaken for the report. ENSURING INFORMATION QUALITY The report and its underlying analyses were conducted in accordance with the Environmental Protection Agency's (EPA) Guidelines for Ensuring and Maximizing the Quality; Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency,1 which follows Office of Management and Budget (OMB) guidelines2 and implements the Information Quality Act (IQA) (Section 515 of Public Law 106-554).* The following section in this Appendix describes the independent, external peer review that was performed on the report. In accordance with OMB definitions, EPA defines the basic standard of information "quality" by the attributes of objectivity, integrity, utility, and transparency. For products meeting a higher standard of quality, like this report, the Agency requires an appropriate level of transparency regarding data and methods to facilitate the reproducibility of information by qualified third parties. The EPA uses various established Agency processes (e.g., the Quality System, peer review requirements and processes) to ensure the appropriate level of objectivity, utility, integrity, and transparency for its products based on the intended use of the information and the resources available. Objectivity focuses on whether the disseminated information is being presented in an accurate, clear, complete, and unbiased manner, and as a matter of substance, is accurate, reliable, and unbiased. The report meets the standard for objectivity, due to activities described in the following: a) The information disseminated was determined to be complete, accurate, and reliable based on internal quality control measures adopted by the expert modeling teams. This included quality checks throughout the chain of analytic steps, including developing and processing climate projections, calibrating and validating the sectoral impact models, and checking data to ensure that no errors occurred in the process to compile and summarize results. b) The information disseminated was determined to be clear, complete, and unbiased based on multiple rounds of independent review. Consistent with guidelines described in EPA's Peer Review Handbook,3 the underlying sectoral modeling methodologies were peer-reviewed * The IQA requires OMB and Federal agencies to issue guidelines that "ensur[e] and maximize[e] the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies" (Public Law 106-554; 44 U.S.C. 3516, note). The IQA does not impose its own standard of "quality" on agency information; instead, it requires only that an agency "issue guidelines" ensuring data quality. Following guidelines issued by OMB, EPA released its own guidelines to implement the IQA: Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency. April 2023 1 ------- Climate Change and Children's Health and Well-Being in the United States through scientific journal publication processes. Citations for these publications can be found throughout the main sector chapters of the report. The report in full was also subject to an independent, external peer review to ensure that the information summarized by EPA was technically supported, competently performed, properly documented, consistent with established quality criteria, and communicated clearly. Integrity refers to security of information, such as the protection of information from unauthorized access or revision, to ensure that the information is not compromised through corruption or falsification. The report and its underlying analyses meet the standard for integrity due to the strategic steps taken to ensure that the data and information remained secure. These steps included the use of password-protected data storage repositories, password protected data transfer technology, and multiple layers of data validation checks to ensure that the integrity was not compromised. Utility is the usefulness of the information to the intended users. The report and its underlying analyses meet the standard for utility because the information disseminated provides insights (quantitative estimates in physical and economic terms) regarding the potential direction and magnitude of the impacts of climate change on children of the U.S. Understanding the risks posed by climate change can inform broader assessment reports and policy decisions designed to address these risks. Transparency ensures access to and description of (1) the source of the data, (2) the various assumptions employed, (3) the analytic methods applied, and (4) the statistical procedures used. The report and its underlying analyses meet the standard for transparency for the following reasons: a) The technical approaches and results of the sectoral impact analyses have been published with open access in the peer-reviewed scientific literature and are cited throughout the report. These papers, along with their online supplementary materials, provide detailed information on the sources of data used, assumptions employed, the analytic and statistical methods applied, and important limitations regarding the approaches and/or how the results should be interpreted. b) Each sector of the report contains a detailed technical appendix providing descriptions of the methodologies used in estimating impacts, assumptions used, results, limitations, and citations to the underlying literature where the reader can go for more information. c) Data sources for each sectoral analysis are listed at the end of the sector-specific technical appendices. d) All data output from the analyses produced for this report have been posted on the report's website. See https://www.epa.gov/cira/climate-change-and-childrens-health-and-well-being- united-states-report e) Responses to comments received from the expert peer review have been posted to EPA's Science Inventory. See https://cfpub.epa.gov/si/ April 2023 2 ------- Climate Change and Children's Health and Well-Being in the United States CONSIDERATION OF ASSESSMENT FACTORS When evaluating the quality, objectivity, and relevance of scientific and technical information, the considerations that EPA takes into account can be characterized by five general assessment factors, as found in "A Summary of General Assessment Factors for Evaluating the Quality of Scientific and Technical Information, and the Guidance for Evaluating and Documenting the Quality of Existing Scientific and Technical Information."4 Table 1 lays out how the assessment factors are considered to determine whether models and data are acceptable for their intended use in this report. Table 1: Assessment Factors for Models and Data Used in This Report Factor Definition How the Factor was Considered Soundness The extent to which the scientific and technical procedures, measures, methods, or models employed to generate the information are reasonable for, and consistent with, the intended application. • Used publicly available (to the maximum extent practicable) data reviewed for quality and accuracy with complete metadata available. Used data included in peer-reviewed publications. Ensured evaluation of the scientific and technical procedures, measures, and methods employed to generate the estimates produced by the sectoral impact models. • Considered the capabilities of integrated assessment, simple climate model, and sectoral impacts models to examine the key analytical questions of this report (i.e., physical effects, economic damages, and changes in risk from climate change) in a manner consistent with sound scientific theory and accepted approaches. • Considered the extent to which the models had been previously applied in projects of similar scope as the Climate change Impacts and Risk Analysis (CIRA) project. For example, the BenMAP model has been used in similar climate and health impact analyses, and both the National Coastal Property Model and the aeroallergen modeling have been employed in state-level climate vulnerability assessments. • Considered whether the data and code are available, made available by EPA, or determined to not be feasible as it is claimed as proprietary by a nonfederal business. • Ensured soundness by selecting sectoral impacts models with the following criteria: sufficient understanding of how climate change affects the sector; the existence of data to support the methodologies; availability of modeling applications that could be applied in this report; based on peer reviewed literature and datasets; and the health or economic significance of impacts and damages in the sector to children in the U.S. Applicability and Utility The extent to which the information is relevant for the Agency's intended use. • Ensured that this report uses applicable and relevant inputs and considers the capabilities of the integrated assessment, simple climate model, and sectoral impacts models to examine the key analytical questions of CIRA (i.e., changes in physical effects, economic damages, and risk associated with climate change). April 2023 3 ------- Climate Change and Children's Health and Well-Being in the United States Factor Definition How the Factor was Considered • Ensured that this report and its underlying analyses are relevant to their intended use so that the information disseminated provides insights and methods for quantifying the physical and economic impacts of climate change on children at national and regional levels. • Ensured sectoral impacts models are reasonable for, and consistent with, the intended application by being sufficiently flexible to ensure consistency in inputs and quantifying physical impacts. • Ensured that models have been applied in peer- reviewed, published studies of similar scope and rigor as CIRA, including those described in the Fourth National Climate Assessment. Clarity and Completeness The degree of clarity and completeness with which the data, assumptions, methods, quality assurance, sponsoring organizations and analyses employed to generate the information are documented. • Ensured use of clear and complete inputs by considering the extent to which sectoral impacts models documented their key methods, assumptions, parameter values, limitations, sponsoring organizations/author affiliations, and funding information. • Ensured publications clearly and comprehensively describe analytic methods used and how they apply and build off existing bodies of research and underlying scientific and/or economic theories. Uncertainty and Variability The extent to which the variability and uncertainty (quantitative and qualitative) in the information or in the procedures, measures, methods, or models are evaluated and characterized. • Ensured inputs that appropriately characterize uncertainty and variability by considering the capabilities of sectoral impacts models to evaluate and characterize key sources of variability and uncertainty. Results of these analyses are described in the underlying journal articles, and also demonstrated in this report. • Reviewed the model documentation and peer-reviewed publications and determined if a model is sufficiently flexible and capable of evaluating important sources of uncertainty for climate change impacts analysis. • Addressed key sources of uncertainty such as: projected emissions (high versus low); regional climate variability (uncertainty across general circulation models); ability to capture variability in temperature and precipitation outcomes; and effects that increasing population and income can have on impact estimates. Evaluation and Review The extent of independent verification, validation, and peer review of the information or of the procedures, measures, methods or models. • Ensured use of independently verified and validated inputs by considering the extent to which models have been independently peer reviewed. • Reviewed the documentation associated with each model and determined if they have been independently peer reviewed and published in scientific journals with procedures to ensure that the methods are technically supportable, properly documented, and consistent with established quality criteria. April 2023 4 ------- Climate Change and Children's Health and Well-Being in the United States Factor Definition How the Factor was Considered • Used scenarios and projections that have been independently verified and validated (e.g., scenarios and projections developed for the Intergovernmental Panel on Climate Change (IPCC) and its assessments, and then downscaled for the U.S. for use in the Fourth National Climate Assessment by the USGCRP Scenarios Working Group). PEER REVIEW OF THE TECHNICAL REPORT Consistent with guidelines described in EPA's Peer Review Handbook,5,+ this report was subject to an independent, external peer review. The purpose of this peer review by independent, qualified, and objective experts was to ensure that the information summarized by EPA was technically supported, competently performed, properly documented, consistent with established quality criteria, and communicated clearly. The methods and applications underlying the sectoral impact modeling of the report were previously peer reviewed and published in the research literature. The application of these methods to investigate how children may be affected by climate impacts generally is novel, and therefore is the primary focus of the peer review. The reviewers were also asked to provide review and feedback on whether EPA appropriately summarized results across impact, regions, and populations. The review was managed by a contractor (ERG) under the direction of a designated EPA peer review leader, who prepared a peer review plan, the scope of work for the review contract, and the charge for the reviewers. Importantly, the EPA peer review leader played no role in producing any portion of the report. Reviewers worked individually (i.e., without contact with other reviewers, colleagues, or EPA) to prepare written comments in response to the charge questions. The contractor identified, screened, and selected four reviewers who had no conflict of interest in performing the review, and who collectively met the technical selection criteria provided by EPA. The peer review charge directed reviewers to provide responses to the following questions during the review: 1. Does the introductory chapter clearly explain the purpose of the report and provide appropriate context for the subsequent chapters? If not, please provide recommendations for improvement. f EPA has determined that this report falls under the classification of "Other Scientific and/or Technical Work Products." The report does not meet the criteria for "influential scientific information," as defined by OMB and further described in the EPA Peer Review Handbook, since it is not being used to support a regulatory program or policy position and does not meet one or more of the factors listed in Section 2.2.3 of the EPA Peer Review Handbook for consideration as influential scientific information. As a corollary, the report also cannot be considered a "highly influential scientific assessment," as defined by OMB. This product is for science dissemination and communication purposes only and does not reflect analysis of nor recommendations regarding any particular policy. April 2023 5 ------- Climate Change and Children's Health and Well-Being in the United States 2. The report has been written for an educated but general audience. In your opinion, are the writing level and graphics appropriate for these audiences? 3. Are the inputs and scenarios clearly explained and documented in the main report and appendices? If not, please explain how they may be improved. 4. Do the text, figures, and tables in the sector report chapters clearly communicate the modeling results? If not, please explain how they may be improved. 5. As described in the report, the technical appendices for each analysis chapter contain detailed information regarding the methodology, assumptions, data, and full sets of modeling results. The report chapters are intended to summarize their respective appendices forthe more general audience of the report. Do the main sectoral chapters properly summarize the underlying information? If not, please explain how they may be improved. 6. This report contains projections of whether overburdened and underserved children are more or less likely to experience greater risks from each climate change impact compared to children in a reference population. The full range of disproportionality (social vulnerability) results are presented in technical appendices for the analyses where data are sufficient to support the method. These results are brought into the chapters of the main report in select locations to provide additional context regarding potential risks. Is the presentation of this information in the main report and appendix appropriate, clear, and logical? 7. a) Does the report, including the executive summary, draw appropriate findings and conclusions from the modeling results? If not, why? b) Does the executive summary provide sufficient context to understand the synthesized results? If not, why? c) Can you please provide any missing important findings or messages in the report that you have identified in your review? 8. Similar to other analyses presented in the report, the "heat and learning" and "heat and emergency department visits" analyses in Chapter 3 (extreme heat) rely on peer-reviewed and published assessments. This EPA report, however, represents the first time these published papers have been used to project future impacts for children under climate change. Given this will be the first time this information is made available, please provide a close review of the methods, assumptions, and findings forthose two analyses in particular (see yellow highlighted publications in Attachment A). 9. Related to question #8, EPA chose to highlight the "heat and learning" analysis in Chapter 3 as the detailed analysis and offer a simpler assessment for the "heat and emergency department visits" impact. Do you agree that the "heat and learning" results should be showcased in the chapter? If not, please explain why. Re-Review Charge Questions: Chapter 7 - Infectious Diseases 1. Does the Infectious Disease chapter draw appropriate findings and conclusions from the new modeling results? Are the revised methods clearly explained? If not, please provide recommendations for improvement. April 2023 6 ------- Climate Change and Children's Health and Well-Being in the United States 2. Is the treatment and coverage of the social vulnerability analysis results presented appropriately in the main chapter and in the technical appendix? We are interested in your perspective on whether this chapter and supporting appendix maintains consistency with the approach used throughout the report and provides sufficient detail and relevant references. If not, please provide recommendations for improvement. April 2023 7 ------- Climate Change and Children's Health and Well-Being in the United States REFERENCES 1 U.S. Environmental Protection Agency. 2002. "Guidelines for ensuring and maximizing the quality, objectivity, utility, and integrity of information disseminated by the Environmental Protection Agency." EPA/260R-02-008. Available online at https://www.epa.gov/quality/guidelines-ensuring-and-maximizing-quality-objectivity-utility- and-integrity-information 2 U.S. Office of Management and Budget. 2002. "Office of Management and Budget Information Quality Guidelines." Available online at https://www.whitehouse.gov/omb/information-regulatory-affairs/information- policy/ 3 U.S. Environmental Protection Agency. 2015. "Peer Review Handbook, 4th Edition, 2015." Programs of the Office of the Science Advisor. Available online at https://www.epa.gov/osa/peer-review-handbook-4th-edition-2015 4 U.S. Environmental Protection Agency. 2003. "A Summary of General Assessment Factors for Evaluating the Quality of Scientific and Technical Information, and the Guidance for Evaluating and Documenting the Quality of Existing Scientific and Technical Information." Science Policy Council. EPA 100/B-03/001 5 U.S. Environmental Protection Agency. 2015. "Peer Review Handbook, 4th Edition, 2015." Programs of the Office of the Science Advisor. Available online at https://www.epa.gov/osa/peer-review-handbook-4th-edition-2015 April 2023 8 ------- |