PEER REVIEW DRAFT, DO NOT CITE OR QUOTE United States Office of Chemical Safety and Environmental Protection Agency Pollution Prevention Systematic Review Supplemental File: Updates to the Data Quality Criteria for Epidemiological Studies October 2019, DRAFT 1 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE EPA's Office of Pollution Prevention and Toxics (OPPT) developed data quality criteria for epidemiological studies. The first version of the criteria was documented in the Application of Systematic Review in TSCA Risk Evaluations document (EPA Document#740-P 1-8001). The initial criteria were updated after considering EPA/OPPT's practical experience and comments from the public. This systematic review supplemental document describes the updated data quality criteria for epidemiological studies that EPA/OPPT intends to apply for the TSCA risk evaluations. Refer to Appendix H of the Application of Systematic Review in TSCA Risk Evaluations document for details about the data quality evaluation tool. Evaluation Criteria for Epidemiological Studies: General (0111'icloiico l.c\cl (Score) Description Selected Score Doniiiin 1. Siuil\ Participation Metric 1. Participant selection (selection, performance biases) Instructions: To meet criteria for confidence ratings lor metrics where *.\M)' is included, studies must address Ixtth conditions where "AND" is stipulated. To meet criteria for confidence ratings for metrics where "OK* is included studies must address at least one of the conditions stipulated. High (score =1) For all studv tvves: All kev elements of the studv desisn are reported (e.s.. setting, participation rate described at all steps of the study, inclusion and exclusion criteria, and methods of participant selection or case ascertainment) AND The reported information indicates thai selection in or out of the study (or analysis sample) and participation was not likely to be biased (i.e., the exposure-outcome distribution of the participants is likely representative of the exposure-outcome distributions in llie population of persons eligible for inclusion in the study.) Medium (score = 2) For all studv tvves: Some kev elements of the studv desisn were not present but available information indicates a low risk of selection bias (i.e., the exposure-outcome distribution of the participants is likely representative of the exposure-outcome distributions in the population of persons eligible for inclusion in the study.) Low (score = 3) For all studv tvves: Kev elements of the studv desisn and information on the population (e.g., setting, participation rate described at most steps of the study, inclusion and exclusion criteria, and methods of participant selection or case ascertainment) are not reported [STROBE checklist 4, 5 and 6 (Von Elm etal.. 2008)1. Unacceptable (score = 4) For all studv tvves: The reported information indicates that selection in or out of the study (or analysis sample) and participation was likely to be significantly biased (i.e., the exposure-outcome distribution of the participants is likely not representative of the exposure-outcome distribution of the population of persons eligible for inclusion in the study.) Not rated/ not applicable (NA) Do not select for this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] 2 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE Confidence l.c\cl (Score) Description Selected Sen iv Med ic 2. Allrilion (missing 1 rilioii/o\cliision. reporting hinscs) High (score =1) For cohort studies: There was minimal subiect loss to follow uo durine the study (or exclusion from the analysis sample) and outcome and exposure data were largely complete. OR Any loss of subjects (i.e., incomplete outcome data) or missing exposure and outcome data were adequately* addressed (as described below) and reasons were documented when human subjects were removed from a study (NTP. 2015). AND Missing data have been imputed using appropriate methods (e.g., multiple imputation methods), and characteristics of subjects lost to follow up or with unavailable records are not significantly different from those of the studv rarticirants (NTP. 2015). For case-control studies and cross-sectional studies: There was minimal subject withdrawal from the study (or exclusion from the analysis sample) and outcome data and exposure were largely complete. OR Any exclusion of subjects from analyses was adequately* addressed (as described below), and reasons were documented when subjects were removed from the studv or excluded from analvses (NTP. 2015). *NOTE for all studv tvves: Adcaualc handling of subiect attrition can include: Use of imputation methods for missing outcome and exposure data; reasons for missing subjects unlikely to be related to outcome (for survival data, censoring was unlikely to introduce bias); missing outcome data balanced in numbers across study groups, with similar reasons for missing data across groups. Medium (score = 2) For cohort studies: There was moderate subiect loss to follow lid durine the study (or exclusion from the analysis sample) or outcome and exposure data were nearly complete. AND Any loss or exclusion of subjects was adequately addressed (as described in the acceptable handling of subject attrition in the high confidence category) and reasons were documented when human subjects were removed from a study. For case-control studies and cross-sectional studies: There was moderate subject withdrawal from the study (or exclusion from the analysis sample), but outcome and exposure data were largely complete AND Any exclusion of subjects from analyses was adequately addressed (as described above), and reasons were documented when subjects were removed from the study or excluded from analvses (NTP. 2015). Low (score = 3) For cohort studies: The loss of subiects (e.e.. loss to follow lid. incomplete outcome or exposure data) was moderate and unacceptably handled (as described below in the unacceptable confidence catcaorv) (NTP. 2015). OR Numbers of individuals were not reported at important stages of study (e.g., numbers of eligible participants included in the study or analysis sample, completing follow-up, and analyzed). Reasons were not provided for non- participation at each stage (Von Elm et al.. 2008). 3 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE ( onlidcncc l.i'M'l (Score) Description For case-control and cross-sectional studies: The exclusion of subjects from analyses was moderate and unacceptably handled (as described below in the unacceptable confidence category). OR Numbers of individuals were not reported at important stages of study (e.g., numbers of eligible participants included in the study or analysis sample, completing follow-up, and analyzed). Reasons were not provided for non- participation at each stage (Von Elm et al.. 2008). Unacceptable (score = 4) For cohort studies: There was large subject attrition during the study (or exclusion from the analysis sample). OR Unacceptable handling of subject attrition: reason for missing outcome data likely to be related to true outcome, with either imbalance in numbers or reasons for missing data across study groups; or potentially inappropriate application of imputation (NTP. 2015). For case-control and cross-sectional studies: There was large subject withdrawal from the study (or exclusion from the analysis sample). OR Unacceptable handling of subject attrition: reason for missing outcome data likely to be related to true outcome, with either imbalance in numbers or reasons for missing data across study groups; or potentially inappropriate application of imputation. Not rated/applicable Do not select for this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Mcli ic 3. ( omp;n ison Group (selection, perform;! ncc biiiscs) High (score =1) For ALL study types: Any differences in baseline characteristics of groups were considered as potential confounding or stratification variables and were thereby controlled by statistical analysis (NTP. 2015). OR For cohort and cross-sectional studies: Key elements of the study design are reported (i.e., setting, inclusion and exclusion criteria, and methods of participant selection), and indicate that subjects were similar (e.g., recruited from the same eligible population with the same method of ascertainment and within the same time frame using the same inclusion and exclusion criteria, and were of similar age and health status) (NTP. 2015). For case-control studies: Key elements of the study design are reported indicate that that cases and controls were similar (e.g., recruited from the same eligible population with the number of controls described, and eligibility criteria and are recruited within the same time frame (NTP. 2015). For studies reporting Standardized Mortality Ratios (SMRs) or Standardized Incidence Ratios (,SIRs): Age, sex (if applicable), and race (if applicable) adjustment or stratification is described and choice of reference population (e.g., general population) is reported. 4 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE Confidence l.c\cl (Score) Description Selected Score Medium (score = 2) For cohort studies and cross-sectional studies: There is onlv indirect evidence (e.g., stated by the authors without providing a description of methods) that groups are similar (as described above for the high confidence rating). For case-control studies: There is indirect evidence (i.e., stated bv the authors without providing a description of methods) that cases and controls are similar (as described above for the high confidence rating). For studies revortins SMRs or SIRs: Ase. sex (if aDDlicablc). and race (if applicable) adjustment or stratification is not specifically described in the text, but results tables are stratified by age and/or sex (i.e., indirect evidence); choice of reference population (e.g., general population) is reported. Low (score = 3) For cohort and cross-sectional studies: There is indirect evidence (i.e.. stated by the authors without providing a description of methods) that groups were not similar (as described above for the high confidence rating). AND Control for differences in exposure groups is not adequately controlled for in the statistical analysis. For case-control studies: There is indirect evidence (i.e.. stated bv the authors without providing a description of methods) that cases and controls were not similar (as described above for the high confidence rating). AND The characteristics of cases and controls arc not reported (NTP. 2015). AND Control for differences in the case and control groups is not adequately controlled for in the statistical analysis. For studies revortins SMRs or SIRs: Indirect evidence of a lack of adjustment or stratification for age or sex (if applicable); indirect evidence that choice of reference population (e.g., general population) is appropriate. Unacceptable (score = 4) For cohort studies: Subiects in all exposure erouos were not similar OR Information was not reported to determine if participants in all exposure aroiiDS were similar 1 STROBE Checklist 6 (Von Elm et al.. 2008) AND Potential differences in exposure groups were not controlled for in the statistical analysis. OR Subiects in the exposure groups had very different participation/response rates fNTP. 20151 For case-control studies: Controls were drawn from a verv dissimilar DODiilation than cases or recruited within verv different time frames (NTP. 2015). AND Potential differences in the case and control groups were not controlled for in the statistical analysis. 5 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE Confidence l.c\cl (Score) Description Selected Score OR Rationale and/or methods for case and control selection, matching criteria including number of controls per case (if relevant) were not reported rSTROBE Checklist 6 (Von Elm et al.. 2008)1. For cross-sectional studies: Subiects in all exposure groups were not similar, recruited within very different time frames, or had very different participation/response rates (NTP. 2015). AND Potential differences in exposure groups were not controlled for in the statistical analysis. OR Sources and methods of selection of participants in all exposure groups were not reported 1 STROBE Checklist 6 (Von Elm et al.. 2008)1. For studies reporting SMRs or SIRs: Lack of adjustment or stratification for both age and sex (if applicable); choice of reference population (e.g., general population) is not reported. Not rated/applicable Do not select for this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments lhal may highlight study strengths or important elements such as relevance] Doniiiin 2. I'.xposiirc Cli;ir;iclcri/;i(ioii Metric 4. Measurement of r.xposnrc (Dolect ioii/me;isii reinoii l/i n To rm;i I ion. performance buses) 11 lull (score =1) For all study tines: 1 aiiosiiiv was a>iisisienll\ assessed no. using 1 he same method and sampling time-frame) using well-established methods (e.g., personal and/or industrial hygiene data used to determine levels of exposure, a frequently used biomarker of exposure) that directly measure exposure [e.g., measurement of the chemical in the environment (air, drinking water, consumer product] or measurement of the chemical concentration in a biological matrix (e.g.. blood, plasma, urinc) CNTP. 2015). OR For an occupational population, contains detailed employment records which allows for construction of a job-matrix for entire work history of exposure (i.e., cumulative or peak exposures, and time since first exposure). Medium (score = 2) For all studv tvves: Exposure was directly measured and assessed using a method that is not well-established (e.g., newly developed biomarker of exposure), but is validated against a well-established method and demonstrated a high agreement between the two methods OR For an occupational study population, contains detailed employment records for only a portion of participant's work history, (i.e., only early years or later years), such that extrapolation of the missing years is required. Low (score = 3) For all study types: A less-established method (e.g., newly developed biomarker of exposure) was used and no method validation was conducted against well-established methods, but there was little to no evidence that the method had poor validity and little to no evidence of significant exposure misclassification (e.g.. differential recall of sclf-rcDortcd exposure) (NTP. 2015). 6 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE Confidence l.c\cl (Score) Description Selected Score OR For an occupational study population, exposure was estimated solely using professional judgement. Unacceptable (score = 4) For all studv tvves: Methods used to auantifv the exposure were not well defined, and sources of data and detailed methods of exposure assessment were not reported [STROBE Checklist 7 and 8] OR Exposure was assessed using methods known or suspected to have poor validity fNTP. 2015). OR There is evidence of substantial exposure misclassification that would significantly bias the results. Not rated/applicable Do not select for this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Metric 5. I-1\|)osiii c lc\cls (l)cU'C(ion/nic;iNurciiH'ii(/inl'orin;i(ion hhiscs) High (score =1) Do not select for this metric. Medium (score = 2) For all studv tvves: The ranee and distribution of exposure is sufficient or adeauate to develop an exposure-response estimate (Cooper et al.. 2016). AND Reports 3 or more levels of exposure (i.e.. referent group and 2 or more) or an exposure-response model using a continuous measure of exposure. Low (score = 3) For all studv tvves: The ranee of exposure in the population is limited OR Reports 2 levels of exposure (e.e.. cxposcd/uncxposcd)) (Cooper et al.. 2016) Unacceptable (score = 4) For all studv tvves: The ranee and distribution of exposure are not adeauate to determine an exposure-response relationship (Cooper et al.. 2016). OR No description is provided on the levels or range of exposure. Not rated/applicable Do not select for this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Metric (>. Tcmpor jlilv (Dclcclioii/iiicasurciiiciil/iiiroriiiiilioii hinscs) High (score =1) For all studv tvves: The studv presents an appropriate temporalis between exposure and outcome (i.e. the exposure precedes the disease). \ND The interval between the exposure (or reconstructed exposure) and the outcome has an appropriate consideration of relevant exposure windows (Lakindetal.. 2014). Medium (score = 2) For all study types: Temporality is established, but it is unclear whether exposures fall within relevant exposure windows for the outcome of interest (Lakind et al.. 2014). Low (score = 3) For all studv tvves: The temporality of exposure and outcome is uncertain 7 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE Confidence l.c\cl (Score) Description Selected Score Unacceptable (score = 4) For all studv tvves: Studv lacks an established time order, such that exposure is not likelv to have occurred orior to outcome (Lakind et al.. 2014). OR There was inadequate follow-up of the cohort for the expected latency period. OR Sources of data and details of methods of assessment were not sufficiently reported (e.g. duration of follow-up, periods of exposure, dates of outcome ascertainment) [STROBE Checklist 8 (Von Elm et al.. 2008)1. Not rated/applicable Do not select for this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Doniiiin 3. Outcome Assessment Mcli'ic 7. Outcome iiiciisiiiviiiciil or cli;ir;ictcri/iilion (delect ion/iiie;isu renien l/i n I'o rm ;i I ion. |)crforni;mcc. reporting l»i;ises) High (score =1) For cohort studies: The outcome was assessed using well-established methods (e.g., the "gold standard"). For case-control studies: The outcome was assessed in cases (i.e., case definition) and controls using well-established methods (the gold standard). Subjects had been followed for the same length of lime in all study groups fNTP. 2015). For cross-sectional studies. There is direct evidence that the outcome was assessed usins well-established methods (the sold standard) (NTP. 2015). *Note: Acceptable assessment methods will depend on the outcome, but examples of such methods may include: objectively measured with diagnostic methods, measured by trained interviewers, obtained from registries (NTP. 2015; Shamlivan et al., 2010). Medium (score = 2) For all study types: A less-established method was used and no method validation was conducted against well-established methods, but there was little to no evidence that that the method had poor validity and little to no evidence of outcome misclassification (e.g., differential reporting of outcome by exposure status). Low (score = 3) For cohort studies: The outcome assessment method is an insensitive instrument or measure. OR The lensth of follow ut> differed bv studv srout) (NTP. 2015). For case-control studies: The outcome was assessed in cases (i.e., case definition) usins an insensitive instrument or measure (NTP. 2015). For cross-sectional studies: The outcome assessment method is an insensitive instrument or measure (NTP. 2015). Any self-reported information Unacceptable (score = 4) For all study types: Diagnostic criteria were not defined or reported rSTROBE Checklist 15 (Von Elm et al.. 2008)1. Not rated/applicable Do not select for this metric 8 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE Confidence l.c\cl (Score) Description Selected Score kc\ lOWCI's comments [Document concerns, uncertainties, lint it at ions, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Metric 8. Reporting Bins High (score =1) For all studv tvves: A description of measured outcomes is reported in the methods, abstract, and/or introduction. Effect estimates are reported with a confidence interval and/or standard errors; number of cases/controls or exposed/unexposed reported for each analysis, to be included in exposure- response analysis or fully tabulated during data extraction and analyses (NTP. 2015). Medium (score = 2) For all study types: All of the study's measured outcomes (primary and secondary) outlined in the methods, abstract, and/or introduction (that are relevant for the evaluation) are reported, but not in a way that would allow for detailed extraction (e.g., results were discussed in the text but accompanying data were not shown). Low (score = 3) For all study types: All of the study's measured outcomes (primary and secondary) outlined in the methods, abstract, and/or introduction (that are relevant for the evaluation) have not been reported. *Note: In addition to not reporting outcomes, this would include reporting outcomes based on composite score without individual outcome components or outcomes reported using measurements, analysis methods, or unplanned analyses were included that would appreciably bias results (NTP. 2015). Unacceptable (score = 4) Do not select for this metric. Not rated/applicable Do not select for this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Doniiiin 4. Potentwl Conl'oiindini;/Y;i ruble Control Metric *). Co\;ui;itc Adjustment (conroiindiii") High (score =1) For all studv tvves: ADDrooriatc adiustments or explicit considerations were made for potential confounders (e.g. age, sex, socioeconomic status) (excluding co-exposures, which are evaluated in metric 11) in the final analyses through the use of statistical models to reduce research-specific bias, including matching, adjustment in multivariate models, stratification, or other methods that were aoorooriatelv justified (NTP. 2015). For Studies revortins SMRs or SIRs: Adiustments are described and results are age-, race-, and sex-adjusted (or stratified) if applicable.. Medium (score = 2) For all studv tvves: There is indirect evidence that aDDrooriatc adiustments were made [i.e., considerations were made for potential confounders (excluding co-exposures)] without providing a description of methods. OR The distribution of potential confounders (excluding co-exposures) did not differ significantly between exposure groups or between cases and controls. OR The major potential confounders (excluding co-exposures) were appropriately adjusted (e.g., SMRs, SIRs) and any not adjusted for are considered not to appreciably bias the results 9 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE ( onlidcncc l.c\cl (Score) Description Selected Score For Studies revortins SMRs or SIRs: Indirect evidence that results are age- and sex-adjusted (or stratified) if applicable. Low (score = 3) For all studv tvves: There is indirect evidence (i.e.. no description is provided in the study) that considerations were not made for potential confounders adiustment in the final analyses (NTP. 2015). AND The distribution of primary covariates (excluding co-exposures) and potential confounders was not reported between the exposure groups or between cases and controls (NTP. 2015). For Studies revortins SMRs or SIRs: Results are aee-. race-. OR sex- adjusted (or stratified) if applicable (i.e., if both should have been adjusted). Unacceptable (score = 4) For all studv tvves: The distribution of potential confounders differed significantly between the exposure group. AND Confounding was demonstrated and was not appropriately adjusted for in the final analyses (NTP. 2015). For Studies revortins SMRs or SIRs: No discussion of adjustments. Results are not adjusted for both age and sex (or stratified) if applicable. Not rated/applicable Do not select for this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Mclric 10. (<>\;iri;ilc (h;ir;idcri/;ilion (mc;iMircmcnl/inrnrm;ilinn. cnnlniinriin^ hhiscs) High (score =1) For all studv tvves: Potential confounded (excluding co-exDosjiuesj. e.g. age, sex, SES) were assessed using valid and reliable methodology where appropriate (e.g., validated questionnaires, biomarker). Medium (score = 2) For all studv tvves: A less-established method was used to assess confounders (excluding co-exposures) and no method validation was conducted against well-established methods, but there was little to no evidence that that the method had poor validity and little to no evidence of confounding. Low (score = 3) For all studv tvves: The confounder (excluding co-cxDOSiircs) assessment method is an insensitive instrument or measure or a method of unknown validity. Unacceptable (score = 4) For all studv tvves: Confounders were assessed using a method or instrument known to be invalid. Not rated/applicable For all studv tvves: Covariates were not assessed. Reviewer's aimmciils [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevancej Metric 11. ( o-c\|)(>Niirc ( onl'oiimlin" (iiiciisuicmciK/iiildi iiiiilidii. coiilouiidinu hinscs) High (score =1) Do not select for this metric. Medium (score = 2) For all studv tvves: Any co-cxDOSiircs to Dollutants that are not the target exposure that would likely bias the results were not likely to be present. OR Co-exposures to pollutants were appropriately measured or either directly or indirectly adjusted for. 10 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE ( onlidcncc l.c\cl (Score) Description Selected Score Low (score = 3) For cohort and cross-sectional studies: There is direct evidence that there was an unbalanced provision of additional co-exposures across the primary study groups, which were not appropriately adjusted for. For case-control studies: There is direct evidence that there was an unbalanced provision of additional co-exposures across cases and controls, which were not appropriately adjusted for, and significant indication a biased exposure-outcome association. Unacceptable (score = 4) Do not select for this metric. Not rated/applicable Enter 'NA' and do not score this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] l)om;iin 5. An;il\sis Metric 12. Siuih Desiiin ;ni(l Methods High (score =1) Do not select for this metric. Medium (score = 2) For all studv tvves: The studv desisn chosen was aDDroDriatc for the research question (e.g. assess the association between exposure levels and common chronic diseases over time with cohort studies, assess the association between exposure and rare diseases with case-control studies, and assess the association between exposure levels and acute disease with a cross-sectional studv design). AND The study uses an appropriate statistical method to address the research questions) (e.g., repeated measures analysis for longitudinal studies, logistic regression analysis for case-control studies, or mean, median for descriptive studies) Low (score = 3) Do not select for this metric. Unacceptable (score = 4) For all studv tvves: The studv desisn chosen was not aDDroDriatc for the research question. OR Inappropriate statistical analyses were applied to assess the research questions. Not rated/applicable Do not select for this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Metric 13. Sliiiisliciil power (scnsilix it>) High (score =1) Do not select for this metric. Medium (score = 2) For cohort and cross-sectional studies: The number of Darticirants are adequate to detect an effect in the exposed population and/or subgroups of the total population. OR The paper reported statistical power is high enough (> 80%) to detect an effect in the exposure population and/or subgroups of the total population. 11 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE ( onlidcncc l.c\cl (Score) Description Selected Score For case-control studies: The number of cases and controls are adeauate to detect an effect in the exposed population and/or subgroups of the total population. OR The paper reported statistical power is high enough (> 80%) to detect an effect in the exposure population and/or subgroups of the total population. Low (score = 3) Do not select for this metric. Unacceptable (score = 4) For cohort and cross-sectional studies: The number of Darticirants is inadequate to detect an effect in the exposed population and/or subgroups of the total population For case-control studies: The number of cases and controls is inadeauate to detect an effect in the exposed population and/or subgroups of the total population Not rated/applicable Do not select for this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] .Metric 14. KenmdiicihiliM of ;iii;iI\ncn Indented 1'rom ISIellner c( ;il. (201)1)1 High (score =1) Do not select for this metric. Medium (score = 2) For all studv tvpes: The description of the analvsis is sufficient to understand precisely what has been done and to be conceptually reproducible with access to the analvlic data. Low (score = 3) For all studv tvpes: The description of the analvsis is insufficient to understand what has been done and to be reproducible OR a description of analyses are not present (e.g., statistical tests and estimation procedures were not described, variables used in the analysis were not listed, transformations of continuous variables (e.g., logarithmic) were not explained, rules for categorization of continuous variables were not presented, exclusion of outliers was not elucidated and how missing values are dealt with was not mentioned). Unacceptable (score = 4) Do not select for this metric. Not rated/applicable Do not select for this metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Mcli'ic 15. Sliiiisiiciil Models (conl'oundinii hi;is) High (score =1) Do not select for this metric. Medium (score = 2) For all studv tvves: The model or method for calculating the risk estimates (e.g., odds ratios, SMRs, SIR) is transparent (i.e., it is stated how/why variables were included or excluded). AND Model assumptions were met. Low (score = 3) For all studv tvves: The statistical model buildine oroccss is not fullv appropriate OR model assumptions were not met OR a description of analyses are not present [STROBE Checklist 12e (Von Elm et al.. 2008)1. 12 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE ( onlidence l.e\el (Score) Description Selected Score Unacceptable (score = 4) Do not select for this metric. Not rated/applicable Enter 'NA' if the study did not use a statistical model. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Domain (». Other (if applicable) Considerations lor liiomarker Selection iind Measurement (l.akind el al. 2<>I4) Metric 10. Lse of Biomarker of Lxposure (detecti«jii/iiie;isiireineiil/iiiroriiialion biases) High (score =1) Biomarker in a specified matrix has accurate and precise quantitative relationship with external exposure, internal dose, or target dose. AND Biomarker is derived from exposure to one parent chemical. Medium (score = 2) Biomarker in a specified matrix has accurate and precise quantitative relationship with external exposure, internal dose, or target dose. AND Biomarker is derived from multiple parent chemicals. Low (score = 3) Evidence exists for a relationship between biomarker in a specified matrix and external exposure, internal dose or target dose, but there has been no assessment of accuracy and precision or none was reported. Unacceptable (score = 4) Biomarker in a specified matrix is a poor surrogate (low accuracy, specificilv, and precision) for exposure/dose. Not rated/applicable Enter "N A" and do not score the mclric if no biomarker of exposure was measured. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Metric 17. Effect biomarker (dclcclioii/mcasiircmcnl/inrormalion biases) High (score =1) Effect biomarker measured is an indicator of a ke\ cv cni in an adv er^e outcome pathway (AOP). Medium (score = 2) Biomarkers of effect shown to have a relationship to health outcomes using well validated methods, but the mechanism of action is not understood. Low (score = 3) Biomarkers of effect shown to have a relationship to health outcomes, but the method is not well validated and mechanism of action is not understood. Unacceptable (score = 4) Biomarker has undetermined consequences (e.g., biomarker is not specific to a health outcome). Not rated/applicable Enter 'NA' and do not score the metric if no biomarker of effect was measured. Reviewer's comments Mclric IS. Method sensili\il\ (delectioii/measiiremeiil/iiiTormation biases) High (score =1) Do not select for this metric. Medium (score = 2) Limits of detection are low enough to detect chemicals in a sufficient percentage of the samples to address the research question. Analytical methods measuring biomarker are adequately reported. The limit of detection (LOD) and limit of quantification (LOQ) (value or %) are reported. Low Frequency of detection too low to address the research hypothesis. 13 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE ( onlidcncc l.c\cl (Score) Description Selected Score (score = 3) OR LOD/LOQ (value or %) are not stated Unacceptable (score = 4) Do not select for this metric. Not rated/applicable Enter 'NA' and do not score the metric. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Metric IV. Iiinm;irker s(;il)ilil\ ((letoclion/me;isiiremeiit/inIVirm;i 1 ion l>i;iscs) 11 lull (score =1) Samples \\ nil a know n storage histors and documental stability dala or those using real-time measurements. Medium (score = 2) Samples have known losses during storage, but the difference between low and high exposures can be qualitatively assessed. Low (score = 3) Samples with either unknown storage history and/or no stability data for target analytes and high likelihood of instability for the biomarker under consideration Unacceptable (score = 4) Do not select for this metric. Not rated/applicable Enter 'NA' and do not score the metric if no biomarkers were assessed. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance/ Metric 20. Siimplc conliimiiiiilion Mlclcclinn/mc;isiircmcnl/inrnrm;ilinn hhiscs) High (score =1) Samples are co iitaimiiauon-free from die unie of collection to die lime of measurement (e.g., by use of certified analyte free collection supplies and reference materials, and appropriate use of blanks both in the field and lab). AND Documentation of the steps taken to provide the necessary assurance that the study data are reliable is included. Medium (score = 2) Samples are stated to be contamination-free from the time of collection to the time of measurement. AND There is incomplete documentation of the steps taken to provide the necessary assurance that the study data are reliable. Low (score = 3) Samples are known to have contamination issues, but steps have been taken to address and correct contamination issues. OR Samples are stated to be contamination-free from the time of collection to the time of measurement, but there is no use or documentation of the steps taken to provide the necessary assurance that the study data are reliable. Unacceptable (4) There are known contamination issues and no documentation that the issues were addressed. Not rated/applicable Enter 'NA' and do not score the metric if no samples were collected. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] 14 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE ( onlidcncc l.c\cl (Score) Description Selected Score Metric 21. Method miuircnicnls ((IcIcclion/iiiciisurciiiciil/inldniiiKion hinscs) High (score =1) Instrumentation that provides unambiguous identification and quantitation of the biomarker at the required sensitivity [e.g., gas chromatography/high- resolution mass spectrometry (GC-HRMS); gas chromatography with tandem mass spectrometry (GC-MS/MS); liquid chromatography with tandem mass spectrometry (LC-MS/MS)]. Medium (score = 2) Instrumentation that allows for identification of the biomarker with a high degree of confidence and the required sensitivity [e.g., gas chromatography mass spectrometry (GC-MS), gas chromatography with electron capture detector (GC-ECD)]. Low (score = 3) Instrumentation that only allows for possible quantification of the biomarker, but the method has known interferants [e.g., gas chromatography withflame-ionization detection (GC-FID), spectroscopy]. Unacceptable (score = 4) Do not select for this metric. Not rated/applicable Enter 'NA' and do not score the metric if bio markers were not measured. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] Metric 22. M;ilri\ ;uljus(nicnl ulclcclion/iiic;isiirciiH-nl/inl'oriiiiilion l)i;iscs) High (score =1) If applicable for the biomarker under consideration, study provides results, either in the main publication or as a supplement, for both adjusted and unadjusted matrix concentrations (e.g., creatinine-adjusted or specific gravity-adjusted and non-adjusted urine concentrations) and reasons are given for adjustment approach. Medium (score = 2) If applicable for the biomarker under consideration, study only provides results using one method (matrix-adjusted or not). Low (score = 3) If applicable for the biomarker under consideration, no established method for matrix adjustment was conducted. Unacceptable (score = 4) Do not select for this metric. Not rated/applicable Enter 'NA' and do not score the metric if not applicable for the biomarker or no biomarker was assessed. Reviewer's comments [Document concerns, uncertainties, limitations, and deficiencies and any additional comments that may highlight study strengths or important elements such as relevance] 15 ------- PEER REVIEW DRAFT, DO NOT CITE OR QUOTE References Blettner. MH. C. Razum. O. (2001). Critical reading of epidemiological papers. A guide. Eur J Public Health. 11(1): 97-101. Cooper. GL. R. Agerstrand. M. Glenn. B. Kraft A. Luke. A. Ratcliffe. J. (2016). Study sensitivity: Evaluating the ability to detect effects in systematic reviews of chemical exposures. Environ Int. 92-93: 605-610. http://dx.doi.Org/10.1016/i.envint.2016.03.017. Lakind, JSS, J. Goodman, M. Barr, D. B. Fuerst, P. Albertini, R. J. Arbuckle, T. Schoeters, G. Tan. Y. Teeguarden. J. Tornero-Velez. R. Weisel. C. P. (2014). A proposal for assessing study quality: Biomonitoring, Environmental Epidemiology, and Short-lived Chemicals (BEES-C) instrument. Environ Int. 73: 195-207. http://dx.doi.Org/10.1016/i.envint.2014.07.011: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4310547/pdf/nihms-656623.pdf. NTP. (2015). Handbook for conducting a literature-based health assessment using OHAT approach for systematic review and evidence integration. U.S. Dept. of Health and Human Services, National Toxicology Program. http://ntp.niehs.nih.gov/pubhealth/hat/noms/index-2.html. Shamliyan. TK. R. L. Dickinson. S. (2010). A systematic review of tools used to assess the quality of observational studies that examine incidence or prevalence and risk factors for diseases [Review], J Clin Epidemiol. 63(10): 1061-1070. http://dx.doi.Org/10.1016/i.iclinepi.2010.04.014. Von Elm. EA. D. G. Egger. M. Pocock. S. J. G0tzsche. P. C. Vandenbroucke. J. P. (2008). The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 61(4): 344- 349. https://hero.epa.gov/heronet/index.cfin/reference/download/reference id/4263036. 16 ------- |