Document bO7G7jJJRYgmq1yQ5gjYvzzLk

Sheila Jasanoff <sjasan@fas.harvard.edu> 12/16/2003 07:17:30 AM Record Type: Record To: Mabel E. Echols OMB_Peer_Review/OMB/EOP@EOP cc: Subject: Comments PLEASE NOTE: THESE COMMENTS WERE SUBMITTED LAST NIGHT (DECEMBER 15) AND WERE NOT DELIVERED DUE TO PROBLEMS AT THE RECIPIENT'S MAILBOX. I AM TRYING AGAIN AND WILL ALSO FAX THEM LATER TODAY. - OMB Peer Review Comment.fin.doc ______________________________ Pforzheimer Professor of Science and Technology Studies Harvard University John F. Kennedy School of Government 79 John F. Kennedy Street Cambridge, MA 02138 Phone: (617) 495-7902 Fax: (617) 496-5960 E-mail: sheila_jasanoff@harvard.edu Comment on Office of Management and Budget (OMB) Proposed Bulletin on Peer Review and Information Quality Sheila Jasanoff Pforzheimer Professor of Science and Technology Studies Harvard University John F. Kennedy School of Government 79 John F. Kennedy Street Cambridge, MA 02138 Tel: (627) 495-7902 E-mail: sheila_jasanoff@harvard.edu On August 29, 2003, OMB's Office of Information and Regulatory Affairs (OIRA) issued a Proposed Bulletin on Peer Review and Information Quality (hereafter, the Bulletin). The purpose of the Bulletin is to ensure "meaningful peer review" of science pertaining to regulation, as part of an "ongoing effort to improve the quality, objectivity, utility, and integrity of information disseminated by the federal government."1 Specifically affected would be the category of "significant regulatory information," which includes information that could have "a clear and substantial impact on important public policies or important private sector decisions with a possible impact of more than $100 million in any year." This proposal would have far-reaching impacts across the federal agencies, requiring 200 or more draft technical documents to be subjected annually to OMB-supervised "formal, independent, external" peer review.2 Accordingly, it is extremely important that this proposal itself should rest on an adequate understanding of (1) the character of regulatory science and (2) the nature and limitations of peer review as an instrument for securing the "quality, objectivity, utility, and integrity" of regulatory science. In the light of scholarship in the social studies of science and of science in public policy, the proposal falls short of meeting these tests. The Bulletin's principal intellectual justification is the proposition that the quality of all science depends on the process of peer review. Thus, the Bulletin states: A "peer review," as used in this document for scientific and technical information relevant to regulatory policies, is a scientifically rigorous review and critique of a study's methods, results, and findings by others in the field with requisite training and expertise. Independent, objective peer review has long been regarded as a critical element in ensuring the reliability of scientific analyses. For decades, the American academic and scientific communities have withheld acknowledgment of scientific studies that have not been subject to rigorous independent peer review.3 Peer review is advanced as "one of the reasons why American science has done so well." These statements (as indeed the thrust of the entire Bulletin) assume that science is a 1 Bulletin, Summary, 68 Fed. Reg. 54023, September 15, 2003. 2 Frederick R. Anderson, "Peer Review of Data," The National Law Journal September 29, 2003. 3 Bulletin, Supplementary Information, 68 Fed. Reg. 54024. S. Jasanoff OMB Peer Review Comments p. 1 singular type of activity, that peer review is likewise a singular and well-defined process, and that the application of peer review to all forms of science--including regulatory or policy-relevant science--can therefore be viewed as unproblematic. Peer review, as conceived by the Bulletin, can be applied as a backstop at the end of scientific production to guarantee the quality of the product. It can serve, in other words, as a kind of audit mechanism for regulatory science. However, available knowledge from more than a decade of research on regulatory science indicates that these assumptions are not well-founded. The audit model corresponds more to OMB's institutional capabilities and administrative approach as a body primarily responsible for economic efficiency than to the needs of science or public policy, both of which call for greater flexibility and discretion. In asserting that peer review has been essential to securing the reliability and success of American science--and in extrapolating from this statement to a generalized demand for peer review of regulatory science--the Bulletin fundamentally misconceives the nature of science and the function of peer review. Specifically: 1. Research science, which is investigator-initiated or "curiosity-driven," differs in important respects from regulatory science. For example, the efficacy of regulatory science depends in part on its capacity to provide timely answers to pressing policy questions or, put differently, to produce "serviceable truths."4 Research science operates under no comparable time pressures; in principle, it can wait indefinitely to produce results. Accordingly, the meanings of reliability and "doing well" are legitimately different for regulatory and research science. The reliability of regulatory science cannot and should not necessarily be measured according to the same criteria as the reliability of research science. Correspondingly, the procedures used to ensure reliability may reasonably differ from the one scientific context to the other. 2. Even in basic research contexts, peer review refers to a heterogeneous cluster of processes that are tailored to specific types of scientific practice. Journal peer review is organized differently and serves different purposes from grants peer review, for example. Moreover, peer review alone is never sufficient to secure the reliability or success of science. Rather, it is always peer review in conjunction with other appropriate processes (such as replication and use) that ultimately ensure quality in science. It is thus a mistake to place too much weight on peer review alone as the driver of scientific quality. In elaborating on the above points, this comment is organized under the following four propositions: Regulatory science--the target of the Proposed Bulletin--is not the same as research science. 4 Sheila Jasanoff, The Fifth Branch: Science Advisers as Policymakers (Cambridge, MA: Harvard University Press, 1990). S. Jasanoff OMB Peer Review Comments p. 2  Identification of peers and defining their role poses special problems in the context of regulatory peer review. Appropriate review of regulatory science demands more flexibility and less standardization than the Bulletin allows. The reliability and credibility of regulatory peer review call for improved public as well as peer review Regulatory Science and Research Science Regulatory science serves fundamentally different purposes from research science and hence is conducted under radically different constraints. As summarized in the table on the following page, these differences include: Goals of scientists and scientific assessors Institutions in which science is done Products of scientific activity Incentives and rewards for producing science Time-frame within which science must be done Options for action available to decisionmakers Accountability for results Given these differences in the context of inquiry, the quality of regulatory science cannot be guaranteed simply through a standardized, end-of-pipe peer review process. A rigid audit mentality will not serve the production of high-quality regulatory science. Rather, attention has to be paid to the totality of the research and assessment process, taking into account the kinds of tensions and conflicts that arise in the course of producing knowledge for policy. To take but one example, science conducted for purposes of public-health standardsetting tends to be generated under skeptical and adversarial conditions that do not obtain in usually more trusting research science environments.5 Highly skeptical environments are not likely to promote the balance and objectivity--that is, the absence of a tilt by reviewers toward given, predetermined, interpretations of the results--that characterize idealized forms of scientific peer review. In adversarial regulatory settings, review is more likely to be polarized, and reviewers are more likely to attack results they do not like than to provide constructive criticism, as is the norm in journal peer review. The progress of regulatory science may therefore depend more on getting stakeholders to agree in advance on appropriate methodologies and investigative protocols than on subsequent critical peer review. Just such prior negotiation has enabled the Massachusetts-based Health Effects Institute for two decades to play an effective role in 5 For the argument that research scientists review each other's work with a fundamental assumption of trust, see William Broad and Nicholas Wade, Betrayers of the Truth (Oxford: Oxford University Press, 1982). In settings where scientific methods are relatively well understood and worked out, such trust may be both warranted and, on the whole, conducive to progress. In regulatory science, by contrast, trust needs to be generated through appropriately designed processes. S. Jasanoff OMB Peer Review Comments p. 3 Source: Sheila Jasanoff, The Fifth Branch: Science Advisers as Policymakers (Cambridge, MA: Harvard University Press, 1990), p. 80. S. Jasanoff OMB Peer Review Comments p. 4 generating high-quality data on air pollution. In HEI's case, a priori collaboration between the Environmental Protection Agency (EPA) and the affected industry established methodological parameters for work that could subsequently be meaningfully peer reviewed. Without such early collaboration, later review would very likely not have resulted in a strengthened and improved knowledge base. Instead, review would have generated protracted controversy and purposeless deconstruction of scientific results-- hindering the progress of both science and public policy.6 Peer Selection and Regulatory Science Peer review in research science promotes the careful application of standards and criteria that are widely agreed upon within the relevant research community. When science is "normal" or paradigmatic in the sense described by the philosopher of science Thomas Kuhn,7 independent review can help ensure that researchers are applying the standards of their field rigorously, consistently, and without bias or deception. In these circumstances, there is ordinarily little doubt who counts as a peer. Peers are the recognized members of the scientific specialty or subspecialty within which normal science is conducted. Such peers share a common culture of scientific practice, with a shared commitment to the goals and methods of inquiry in their field. Regulatory science, however, is not normal science. It may cross disciplinary lines, enter into previously unknown investigative territories, and require the deployment of new methods, instruments, protocols, and experimental systems. Correspondingly, the "peers" for reviewing regulatory science are likely to come from disparate technical backgrounds and not form part of a single, tightly-knit research community. In general, peers for conducting peer review will be hardest to identify and are most likely to produce inconsistent or unhelpful reviews when the science in question has the following properties: It is emergent. That is, neither the knowledge base nor the methods for producing it are firmly established in advance. It is characterized by high uncertainty (with respect to data) and low consensus (with respect to methods). It is the product of interdisciplinary methods. It is politically sensitive. All these characteristics are frequently present in regulatory science (particularly so in the case of significant regulatory information), making the identification of independent, objective peers both difficult and controversial. 6 On the unproductive deconstruction of policy-relevant science in adversarial settings, see Sheila Jasanoff, Risk Management and Political Culture (New York: Russell Sage Foundation, 1986); David Collingridge and Colin Reeve, Science Speaks to Power: The Role of Experts in Policy (New York: St. Martin's Press, 1986); H.M. Collins, Changing Order: Replication and Induction in Scientific Practice (London: Sage Publications, 1985). 7 Thomas Kuhn, The Structure of Scientific Revolutions, (Chicago: Chicago University Press, 1962). S. Jasanoff OMB Peer Review Comments p. 5 Further, allowing "peers" to import standards from one context of scientific practice to another may not be warranted. Methods suited to one investigative context may not be suited to another, even when the phenomena under investigation are similar. For example, practices for forensic DNA analysis (involving smaller samples, higher potential for contamination, different collection methods, etc.) are different from those for DNA typing in hospitals or laboratories. Crime lab researchers might legitimately apply different quality control standards from hospital or lab researchers. The Problems of Standardization The Bulletin contemplates a uniform peer review process designed to correct what are seen as deviations from the gold standard of independent, objective peer review. The following citations are indicative of this standardizing mindset8: 54024: "Existing agency peer review mechanisms have not always been sufficient to ensure the reliability of regulatory information disseminated or relied upon by federal agencies." 54024: "Even when agencies do conduct timely peer reviews, such reviews are sometimes undertaken by people who are not independent of the agencies." 54025: "When an agency does initiate a program to select outside peer reviewers for regulatory science, it sometimes selects the same reviewers for all or nearly all of its peer reviews on a particular topic." 54025: "it is also essential to grant the peer reviewers access to sufficient information..." 54025: "the results are not always available for public scrutiny or comment." 54025: "experience has shown that they are not always followed by all of the federal agencies, and that actual practice has not always lived up to the ideals underlying the various agencies' manuals."9 While the goal of assuring consistency in agency practices is laudable, it is questionable whether public health and safety interests are well served by insisting upon a single standardized set of procedures for all significant regulatory information.10 Agencies entrusted with primary responsibility for public health and safety will need sufficient discretionary room to decide when rapid responses are warranted; when standards of proof should be lowered in the interests of precaution (e.g., in situations involving children's health or threats of catastrophic harm, such as stratospheric ozone depletion); which mechanisms or combination of mechanisms are best suited for quality assurance purposes (e.g., scientific advice, public review, peer review); 8 All page citations are to the Federal Register, vol. 68, no. 178 (September 13, 2003). 9 I am indebted to John Mathew and John Price for identifying these extracts. 10 See, for instance, Sharon Begley, "White House Seeks Peer Review Standard for Range of Studies," Wall Street Journal, December 5, 2003. S. Jasanoff OMB Peer Review Comments p. 6  and when repeated public consultation is needed in order to strengthen the credibility of regulatory science.11 Even the editors of leading scientific journals have long recognized that their ordinarily less consequential review processes may need to be modified when applied to studies of potentially high policy relevance.12 Regulatory agencies, bearing far more responsibility for public health and welfare, need to retain all the more room for discretion in the production and review of scientific information. Public Review and Peer Review As noted above, the Bulletin's conception of peer review conforms to an audit mentality that is poorly suited to achieving progress in, or quality assurance of, regulatory science. In the past few years, several expert committee reports have concluded that, far from relying on a single, end-of-pipe review process, regulatory science should be developed through significant components of public participation and comment. Thus, a 1996 report of the National Research Council determined that the quality of risk information disseminated by federal regulators will be improved if the risk analytic process develops through coupled procedures of analysis and deliberation.13 A presidential and congressional commission on risk assessment and management came to very similar conclusions with regard to the need for wide stakeholder participation in the development and critique of regulatory science.14 Many academic observers have suggested that regulatory science in areas of high uncertainty should be subject to wider and more public critique--sometimes termed "extended peer review"15--rather than to traditional peer review by technical experts alone. Arguably, the Bulletin's call for a public comment period before peer review could serve this public review function, but the value of such an open process might be vitiated if it is adopted only as an input to a standardized, closed, technical peer review. The strongest argument in favor of broad public comment on science is that orthodox peer review is too narrow and constrained to permit effective discussion of important uncertainties and indeterminacies.16 By the same token, there may be less need for wide 11 This argument is not new. Just such considerations led the National Research Council in 1983 to advise against a single, centralized risk assessment process for federal regulatory agencies. See NRC, Risk Assessment in the Federal Government: Managing the Process (Washington, DC: National Academy Press, 1983). 12 See, for example, Sheila Jasanoff, The Fifth Branch: Science Advisers as Policymakers (Cambridge, MA: Harvard University Press, 1990), Chapter 4. 13 Paul Stern and Harvey Fineberg, eds., Understanding Risk (Washington, DC: National Academy Press, 1996). 14 Presidential/Congressional Commission on Risk Assessment and Risk Management, Final Report (1997), http://www.riskworld.com/Nreports/1996/risk_rpt/Rr6me001.htm. 15 Silvio O. Funtowicz and Jerome R. Ravetz, "Three Types of Risk Assessment and the Emergence of Post Normal Science," in Sheldon Krimsky and D. Golding, eds., Social Theories of Risk (London: Praeger, 1992), 251-273. 16 The long-term environmental impacts of genetically modified crops offer one example of such a case. This reasoning led the British government recently to conduct a national deliberation on the risks of commercializing such crops. S. Jasanoff OMB Peer Review Comments p. 7 public debate in those cases where science is "normal" and technical peer review therefore is uncontroversial and serves an adequate check on quality. Combining the two processes of public review and peer review accordingly calls for the exercise of judgment. Mechanically using the former as an input to the latter could prove impossibly time-consuming and burdensome for policymaking aimed at protecting the nation's health, safety, and environmental quality. The considerations laid out above suggest that the reliability and credibility of peer review are every bit as significant from the standpoint of policy legitimacy as the reliability and credibility of the information to be reviewed. In regulatory settings, the choice of peer reviewers is less constrained by disciplinary norms and may lead to a higher potential for bias and manipulation than in research contexts. The Bulletin is sensitive to one possible source of bias: ongoing financial ties between the regulatory agency and possible reviewers. However, such ties do not necessarily lead to inappropriate cognitive bias (a point that industry consultants in the biotech industry have frequently made), and peers may display many other kinds of bias that the Bulletin does not directly address, for instance: Financial ties to particular industry interests. Framing of research by specific policy commitments (e.g., opposition to the idea of tobacco-induced cancer or anthropogenic climate change) Commitment to marginal or discredited scientific viewpoints (e.g., rejection of the HIV-AIDS hypothesis). Intellectual biases in favor of methodologies that systematically under- or overstate the gravity of problems (e.g., bias in favor of epidemiology as opposed to structureactivity information, mutagenicity studies or animal toxicity studies). The best way to guard against all these forms of bias is to maintain open and transparent review procedures, holding agencies accountable to expert advisers and the public for their choice of reviewers and their overall conduct of science. OMB oversight under rigidly defined procedural rules is not ideally suited to promoting such transparency. Summary and Conclusions The Bulletin seeks to impose on federal agencies a uniform, standardized approach to peer review. Through its wide-ranging application and the stringency of its requirements, the proposal is likely to have significant impact on the time and cost of policy development--and, by extension, on the capacity of regulators to effectively protect public health, safety, and the environment. It is therefore imperative that the proposal be grounded in the best available knowledge about the nature of scientific research and the function of peer review. The proposal, as currently drafted, fails to meet basic standards of scholarly accountability. In the light of relevant scholarship, I have identified several important flaws in the proposal. These include: S. Jasanoff OMB Peer Review Comments p. 8  Adherence to an inappropriate audit model of peer review. Failure to distinguish the contexts, needs, and purposes of regulatory science from those of research science. Failure to acknowledge the difficulties associated with peer selection in fluid, open- ended, and political areas of inquiry. Inappropriate insistence on standardized procedures in areas marked by variable and divergent legal mandates, practical constraints, and societal needs. Imposition of an OMB-supervised peer review process in situations where previous expert inquiry has emphasized the need for transparency, broad participation, and significant deliberative involvement by stakeholders. These criticisms suggest that the proposal in its present form will not achieve the goals of the Information Quality Act and may undermine the goals and purposes of public health and environmental legislation. The proposal should be withdrawn or else radically revised so as to leave much greater discretion within the expert regulatory agencies to tailor their review practices consistently with their legal mandates and policy missions. S. Jasanoff OMB Peer Review Comments p. 9