SEPA
United States
Environmental Protection
Agency
Evaluating the consistency of heterogeneous results:
important determinants of inconsistency
Barbara S. Glenn, Ph.D.; Elizabeth Radke, Ph.D.; and Andrew Kraft, Ph.D.
National Center for Environmental Assessment, U.S. Environmental Protection Agency, Washington, D.C., USA
Barbara Glenn I
202-564-5483
Background and Methods
Analysis of Confounding
Background/Aim
The analysis of study results across a set of studies is a powerful tool that can help with
decisions about whether a potential bias is an important concern for an individual study, and to
illuminate a pattern within apparently inconsistent effect estimates. The heterogeneity may
stem from differing study designs examining varying outcome and exposure definitions and be
influenced to varying degrees by sources of bias and other factors that affect the magnitude,
direction, and precision of effect estimates. Influential aspects include potential bias (e.g.,
selection, information, confounding) and other quality aspects (e.g., sensitivity, precision).
This type of analysis also can include factors, such as exposure levels, that are important for
the interpretation of results. Studies of the association between indoor formaldehyde exposure
and current asthma and pulmonary function were used as a case study to illustrate the impact
of bias and other study attributes on the analysis of consistency across studies.
Current Asthma in Children and Adults
Methods:
Analyses of current asthma and pulmonary function endpoints were performed as
part of a systematic evaluation of the literature database on studies examining the
potential for respiratory and immune-mediated conditions in relation to
formaldehyde exposure that was conducted through October 2016.
Criteria to evaluate risk of bias and sensitivity for the selected endpoints were
developed using expert consultation or methodological reviews by professional
organizations. The IRIS study evaluation tool included domains for participant
selection, exposure, outcome, confounding, analysis, and sensitivity.
The consistency of results for current asthma was examined via forest plots
presenting effect estimates (e.g., risk ratios, odds ratios) stratified by exposure
levels (low vs high) and overall study confidence, and an analysis of potential
confounding looking across study results was conducted for current asthma and
pulmonary function endpoints.
Current asthma is defined as a report of asthma symptoms during the last 12 months. Studies limited to "ever had asthma" were not included because the formaldehyde measures available did not
reflect cumulative exposures that could be related to cumulative risk. The population relevant to the PECO for this analysis included children and adults in 22 studies of residential or school exposures
and 5 occupational studies involving manufacture of pressed wood products, chemical production and embalming. These studies analyzed the variation in risk of prevalence of current asthma,
incidence of asthma or asthma control or severity in relation to variation in formaldehyde at exposures above 0.010 mg/m3 across a range spanning at least 0.01 mg/m3. Of the 22 studies of residential
or school settings, 4 were considered "not informative" for current asthma because the target population was under 5 years of age, an age range when asthma diagnoses are not specific. Three studies
reported comparisons of mean formaldehyde concentrations in cases and controls and could not be included in the forest plots.
Sorting by Exposure Setting, Level, and Confidence for Studies of Current Asthma
Studies ordered alphabetically
Adults
Group	OR
Studies ordered by exposure setting, level, and confidence
Adults
Limitations of Low Confidence Studies
Adults
Reference
Billionnet et al 2001
Fransman et al 2003
Confidence Group_
Norback et al,
1995
Holness et al 1989
Krzyzanowski et al 1990
2.0
NS
¦
Matsunaga et al 2003
1 o.8 *	—





3 2.65

Neghab etal 2011
10.4
	"	
Norback et al 1995
12.5
	¦		
Palczynski et al 1999
1 0.71	 ¦

Yeatts et al 2012
143 ¦
I
Zhai et al 2013
NR (0 in referent)


0.50 1.0 2.0 4 0 80 160
Odds Ratio, 95% CI

Children

Reference
Group OR

Annesi Maesano et al 2012
1	1.1 I
2	0.9 |
¦
I
Garrett et al 1999
NS

Hulin et al 2010
1.7
Lift—
Kim etal 2011
1.04 per 0.01 mg/cu m 1
¦
Krzyzanowski et al 1990
2.4
——
Mi et al 2006
Palczynski et al 1999
1.3 per 0.01 mg/cu m
¦
0.99 ¦

Smedje and Norback 2001
Tavernier et al 2006
1.2 per 0.01 mg/cu m
¦



2 0.99 ¦

Venn et al 2003
1	1.14
2	1.08
¦
¦—
Yeatts etal 2012
1.43
.
Zhai etal 2013
12.4

ntial, < 0.05 mg/cu m
et et al 2001
aga et al 2003
lski et al 1999
ketal 1995
st al 2012
Yeatts et a I
2012
Residential > 0.05 mg/cu m


Krzyzanowski et al 1990
M
NS
Matsunaga etal 2003
M
3 2.65 ¦
Palczynski etal 1999
M
2 3.6	¦	
Zhai etal 2013
L
NR
(0 in referent)
al 2003 (All)
al 2003 (High)
High
>6.5yr
Odds Ratio, 95% CI
Information bias: Most values < LOQ for formaldehyde
Confounding: Unable to distinguish RR for VOCs with
formaldehyde
Information bias: Analyses combined children and adults;
mothers responded for children
Confounding: Unable to distinguish RR for S02with
formaldehyde
Zhai et al, 2013 Selection bias: Participation rates not reported, but
selection criteria were reported
Information bias: Sampling period and protocol details not
reported
Confounding: Univariate, but magnitude of OR not likely
explained by confounding
Analysis: Small number of cases for analysis
Selection bias: Lead time bias, Left truncation
Information bias: Short formaldehyde sampling period;
Asthma definition imprecise
Confounding: Possible residual confounding for smoking
Selection bias: Lead time bias, Left truncation
Information bias: Asthma definition imprecise
Confounding: Univariate analysis
Children
Neghab et al,
2011
Holness et al.
1989
Children
oup_ __ 		 OR_
Kim et al 2011
Mi et al 2006
Venn et al 2003
1.04 per 0.01 mg/cu m
1.3 per 001 mg/cu m
Residential < 0.05 mg/cu m
Garrett et al 1999
Hulin et al 2010
Smedjeand Norback 2001
Tavernier et al 2006
Yeatts etal 2012
Odds Ratio, 95% CI
Conclusion
When studies are ordered alphabetically, results appear heterogenous, but when exposure levels and study
confidence are considered, a pattern of increasing risk with increasing exposure levels is apparent among the high
and medium confidence studies. No single domain limitation was a primary reason for the low confidence
determinations, but collectively results of these studies are more variable.
Smedje and
Norback, 2001
Garrett et al,
1999
Hulin et al,
2010
Tavernier et al,
2006
Yeatts et al,
2012
Hsu et al, 2012
Hwang et al,
2011
Information bias: Exposure, uncertain concentration
distribution, high proportion < LOD
Confounding: No adjustment for coexposures, but results
varied among exposures
Selection bias: Potential household correlation of cases and
controls
Information bias: Asthma definition imprecise
Analysis: Adjusted results reported as not signif
Analysis: Small sample size; uncertain interpretation of
urban/ rural stratified analyses
Selection bias: Missing data for 50% cases; not reported for
controls
Exposure: Distribution not reported
Information bias: Asthma definition included questions not
specific to asthma
Analysis: Exposure levels by tertile not reported
Information bias: Analyses combined children and adults
Confounding: Unable to distinguish RR for S02with
formaldehyde
Not plotted.
Selection bias: Low, differential participation rate
Information bias: Short formaldehyde sampling period and
protocol not reported
Confounding: Univariate analysis
Analysis: Limited
Not plotted. Selection bias: High prevalence family history
asthma in both groups
Information bias: Asthma definition imprecise
Analysis: Questions about analysis and distribution
U.S. Environmental Protection Agency
Office of Research and Development
0^ Printed on 100% recycled/recyclable paper
% with a minimum 50% post-consumer
fiber using vegetable-based ink.
Current Asthma: Sorting by Rating for Confounding


Adults
Reference
OR

Residential, < 0.05 mg/cu m
Adequate
Matsunaga et al 2003


o°7«

Billionnet etal 2001
1.43
	¦	
Deficient:Univariate analysis
Palczynski et al 1999
Deficient: Potentially inflated OR
Norback et al 1995
Yeatts et al 2012
0.71 -
12.5
1.43
¦
¦
Residential, > 0.05 mg/cu m
Adequate
Matsunaga etal 2003
Krzyzanowski et al 1990


NS

Deficient: Univariate analysis
Palczynski et al 1999
Zhai etal 2013


NR (0 in referent)

Children
Mi etal 2006	1.3 per 0.01 mg/cu m
Kim et al 2011	1.04 per 0.01 mg/cu m
Occupational exposure
Heitertetal 1994
Fransman et al 2003 (All)
Fransman et al 2003 (High)
Duration > 6.5 yr
Malaka & Kodama 1990
Neghab et al 2011
Deficient: Univariate analysis
Holness et al 1989
:k 2001 1.2 per 0.01 mg/cu m
Odds Ratio, 95% CI
Odds Ratio, 95% CI
Conclusion
For studies in residential or school settings with lower exposure levels, a deficient rating for confounding with a predicted direction
away from the null provides a potential explanation for some of the heterogeneity in odds ratios.
Occupational Studies of Pulmonary Function: Confounding
Forced expiratory volume in 1 second (FEV,) and forced vital capacity (FVC)
were the most common measures analyzed by the studies of formaldehyde
exposure. Fhe population relevant to the PECO for this analysis included
workers with occupational exposure to formaldehyde in studies where
exposure was confirmed by air measurements, or involved professional
categories of embalmers and anatomists/pathologists. A total of 21 studies
involving manufacture of wood products, chemical production, embalming or
offices in mobile trailers were identified, which analyzed variation in
pulmonary function values in relation to variation in formaldehyde at
exposures above 0.010 mg/m3 across a range spanning at least 0.01 mg/m3. Of
the 21 studies, 5 were determined "not informative" because of one or more
critical deficiencies; one reported additional analyses in the same cohort, and
three presented longitudinal analyses or cross-shift changes, which are not
shown in this example.
Fhe occupational studies were limited by low sensitivity due to healthy
worker survivor bias resulting from the cross-sectional analyses and loss of
sensitive individuals before the studies began. Different analytic approaches
were used making it challenging to examine results across studies in a graph
using a single metric. However, most of the studies provided a mean value for
exposed and referent groups as a percent of predicted adjusting for age, sex
and height, and some expression of error. For these studies, forest plots were
constructed using the mean difference of percent of predicted FEV, or FVC
and confidence intervals. Fhe mean difference is adjusted for smoking in
these graphs only for three of the studies (Malaka et al.. 1999; Holmstrom et
al., 1988; Levine et al., 1984). However, most of the studies addressed
smoking either in their designs or analyses, and the plotted results are in the
same direction as the reported study results.
Cross-sectional analyses of pulmonary function endpoints in relation to occupational exposure
FEV,
FVC
Author, Year
Wood Products
Malaka (high), 1990
Malaka (low), 1990
Alexandersson, 1982
Horvath, 1988
il Manufacture
ib, 2011
Lofstedt, 2009
Schoenberg, 1975
Holness, 1989
Levine, 1984
Mobile Office
Main, 1983
HCHO Ref Conf
47 20
109 254
84 38 L
12 12 M
14 17 L
Mean Difference [95% CI]
-3.6 [-8.9, 1.7]
-7.0 [-12.2,-1.8]
-9.5 [-16.9, -21]
-2.0 [-4.9, 0.9]
-12.2 [-18 9, -5.5]
-1.9 [-5.4, 1.6]
-1.7 [-12.8, 9.4]
-1.5 [-6.4, 3.4]
—i	4.7 [-5.6,15.0]
Author, Year
HCHO Ref Conf
Mean Difference [95% CI]
Wood Products


Malaka (high), 1990	5<
Malaka (low), 1990	3'
Holmstrom, 1988.2	91
Alexandersson, 1982	4"
Horvath, 1988	10
Chemical Manufacture
Neghab, 2011	71
Lofstedt, 2009	6-
Holmstrom, 1988.1	71
Schoenberg, 1975	1!
Embalming
Holness, 1989	8-
Levine, 1984	1;
Mobile Office
Main, 1983
-0.3 [-3.6, 3.0]
-4.9 [-8.2, -1.6]
-8.1 [-12.5,-3.7]
-5.61-11.7, 0.5]
-2.0 [-4.8, 0.8]
-13.9 [-20.6, -7.2]
-0.6 [-4.1, 2.9]
-6.6 [-11.3, -1.9]
7.9 [-1.5,17.3]
-0.4 [-4.9, 4.1]
-1.6 [-10.6, 7.4]
-10.0	0.0	10.0
Mean Difference, FEV 1 (%)
-10.0	0.0	10.0
Mean Difference, FVC (%)
Notes on forest plots
•	Difference in means of spirometry measurements taken before shift comparing exposed to unexposed. Means adjusted for age, height and sex.
•	Study regression analyses also adjusted for smoking, and coefficients were in the same direction as those in forest plot.
•	Two studies were not plotted because the studies reported only means of the unadjusted absolute values. Herbert et al. (1984) found a
statistically significant decrease in FEV,/FVC in analyses adjusted for age, height, sex and smoking, and Khamgaonkar et al. (1991) found a
statistically significant decrease in FEV, and FVC in analyses adjusted for age, height, weight and sex.
Fhe views expressed in this abstract are those of the authors and do not necessarily reflect the views or policies of the U.S. Environmental Protection Agency.
Conclusion
Overall, mean values of pulmonary function among exposed workers were lower than those of unexposed comparison groups. The
difference is not large, but is consistently observed in most of the studies, which were limited by a healthy worker survival bias, which
may have attenuated the size of the observed difference. Concern for residual confounding is lessened because findings were consistent
between the wood products and chemical manufacturing industries, which involve different coexposures.

-------