Document dn9Kq2Me3Xo819zJQa96be645

American Journal of Industrial Medicine 173269-27 I ( 1990) LETTER TO THE EDITOR Limitations of Meta-Analysis: Cancer in the Petroleum Industry Key words: occupational epidemiology, risk assessment, standardized mortality ratio Meta-analysis is a methodology that has gained recent popularity as a way to summarize the results of multiple studies and make quantitative conclusions from them. At its best, it provides a framework with which to systematically assess the results of many studies addressing a single hypothesis, giving guidance to those wishing to weigh the evidence from many sources to reach a single conclusion. But just as meta-analysis represents a compounding of many individual studies, it cannot ignore the pitfalls and limitations of these individual studies. Nor can it avoid the minimum requirements of any scientific study: a clearly stated hypothesis; systematic, clearly described methods; and conclusions tempered by the limitations in the data. Although seemingly minor, the latter caveat probably represents meta-analysis' single greatest danger: that in attempting to capture many studies (the complete as well as the weak or limited studies) in a single quantitative summary, the authors may allow the reader to lose sight of the limitations of the studies on which the metaanalysis is based. Wong and Raabe may have fallen into this trap in their recent meta-analysis of cancer risks in the petrochemical industry [Wong and Raabe, 19891. It appears that ' their analysis combines a number of epidemiologic studies whose weaknesseschiefly, poor or entirely lacking exposure assessments and the healthy worker effect-make interpretation of the summary SMR measures impossible. The stated purpose for the meta-analysis was to provide a quantitative estimate of the human cancer risks from exposure to the various gasoline hydrocarbons. One can proceed from such a broad objective in one of two ways: either by enumerating a series of hypotheses, each based on a specific chemical exposure, or by sidestepping the problem of chemical specificity through pooling studies with different exposures. While Wong and Raabe do identify the limited evidence suggestive of certain sitespecific cancer risks in certain occupational groups (the former approach mentioned above), the major thrust of their analysis is a pooling over many different, often poorly specified exposure conditions. We believe that this approach (the second alternative above) has severe methodologic limitations and may result in misleading conclusions. Greenland has pointed out that the pooling of risk estimates from multiple Address reprint requests to Dr. D. Kriebel, Department of Work Environment, University of Lowell, 1 University Avenue, Lowell, MA 01854. -Accepted for publication August 7, 1989. 0 1990 W h y - L i s , Inc. 270 Kriebel et al. TABLE 1. Leukemia Mortality in Petrochemical Job Groups* Specific job > I Year >5 Years !PUP N SMR 95% c1 N SMR 95% CI OfficelManag. 18 I24 (74- 197) II 93 (46-167) Operators 31 I06 (93-194) 21 120 (74-184) Maintenance 27 I15 (76- 167) 14 93 (5 1-156) Laboratory 3 74 (15-218) 3 94 ( 19-274) Pipefittersl boilermakers 17 212 ( 123-339) 12 285 (147-498) All subjects 48 1 I 8 (87-156) *Data on specific job groups are from Divine and Barron [ 19861. Data on "all subjects" are from Divine et al. (19851. studies should only be done if one assumes that the only source of variability between studies is sampling variability [Greenland, 19871. That is, all effect estimates from the different studies must be considered unbiased estimators of a single "true" effect. To pool estimates of risk for petrochemical workers, therefore, it must be determined that the available risk estimates can reasonably be assumed to result from a common exposure experience. The petrochemical industry, however, constitutes not a single hazard, but a large collection of exposures to gasoline hydrocarbons varying by chemical mixture, intensity, and duration. Dilution of true risk estimates would, therefore, result from pooling of findings from studies of qualitatively and quantitatively different exposure experiences. For example, the authors chose to include in the meta-analysis for leukemia [Table XI11 in Wong and Raabe, 19891 the study conducted by Divine and colleagues [Divine et al., 19851 summarizing mortality in all Texaco refinery petrochemical and research workers combined, despite the existence of a paper examining specific.job classifications in the same plants [Divine and Barron, 19861. When the same population was examined by job, SMRs ranged from 93 to 285, with a significant excess for pipefitters and boilermakers (see Table I). By using only the summary SMR for leukemia, Wong and Raabe are implicitly assuming that these differences are not due to differences in exposure, but rather represent sampling variability. This assumption is not justified by the authors. The effect of including only the combined SMR for these different groups, then, may be to mask potentially important differences in leukemia risks associated with more narrowly defined refinery activities and more homogeneous exposure profiles. The healthy worker selection effect (HWSE) is another important limitation of many of the individual studies, an effect which is likely to be compounded in a meta-analysis. While it has been argued that the HWSE is less important for cancer mortality than for certain other causes (respiratory disease, for example), it has been observed in other studies [Gilbert, 1982; Sterling and Weinkam, 1986; Weed et al., 19871 and may explain a number of SMRs of less than 100 observed in the petrochemical industry. This known bias, when unaccounted for in a meta-analysis, may overwhelm studies showing small excess risks (which may themselves be biased toward the null by the HWSE). The resulting meta-SMR would have to be considered biased by some average amount, a concept difficult to grasp and therefore a result difficult to interpret. One might counter that the individual studies were published Limitations of Meta-Analysis 271 with HWSE-biased results and so there is nothing wrong with doing this in a metaanalysis as well, but we find this argument unconvincing. The reader of the metaanalysis is less able to consider the implications of such problems as the HWSE when so many studies are included in a single summary paper (and summary risk estimates), with little descriptive information on each one. We would suggest that, in the future, meta-analysis of the petroleum industry or any other complex environment should proceed as follows: 1. studies would be reviewed to identify a set judged to be reasonably alike with respect to the quality, duration, and intensity of exposure to potential toxic chemicals; 2. if necessary, adjustments to individual results could be made to attempt to remove confounding effects or make the individual study results more closely estimate the same exposure effect [Greenland, 19871; 3. the degree of potential bias in the results would be evaluated, and certain studies would be eliminated if biases are judged to be severe [Greenland, 19871; 4. the adjusted results of the accepted studies would be pooled, and a single effect estimator would be calculated. The meta-analysis of Wong and Raabe gives the impression that available studies suggest the petroleum industry has few carcinogenic hazards. We think that this conclusion is premature. David Kriebel, SCD, David H.Wegman, MD. MS. Rafael Moure-Eraso, PhD, and Laura Punnett, SCD. Department of Work Environment, University of Lowell, Lowell, MA REFERENCES Divine BJ, Barron V (1986): Texaco mortality study: 11. Patterns of mortality among white males by specific job groups. Am J Ind Med 10:371-381. Divine BJ, Bamn V, Kaplan SD (1985): Texaco mortality study: I. Mortality among white male refinery, petrochemical, and research workers. J Occup Med 27:445-447. Gilbert FS (1982): Some confounding factors in the study of mortality and occupational exposures. Am J Epidemiol 1 16:177-188. Greenland S (1987): Quantitative methods in the review of epidemiologic literature. Epidemiol Rev 9: 1-30. Sterling TD, Weinkam JJ (1986): Extent, persistence and constancy of the healthy worker or healthy person effect by all and selected causes of death. J Occup Med 28:348-353. Weed DL, Tyroler HA, Shy C (1987): The healthy worker effect in actively working communications .workers. J Occup Med 29:335-339. Wong 0, Raabe GK (1989): Critical review of cancer epidemiology in petroleum industry employees, with a quantitative meta-analysis by cancer site. Am J lnd Med 15:283-310.