Document V3yJLdyzqjyNydeOpLK18eQyg

"Colglazier, Bill" <bcolglazier@nas.edu> 12/15/2003 12:38:50 PM Record Type: Record To: Mabel E. Echols OMB_Peer_Review/OMB/EOP@EOP cc: "Alberts, Bruce" <BAlberts@nas.edu> Subject: Comments from National Academy of Sciences Dear Dr. Schwab: I have enclosed and attached the comments from Dr. Bruce Alberts, President, National Academy of Sciences, 500 Fifth Street, NW, Washington, DC 20001, telephone 202-334-2101, email balberts@nas.edu, on OMB's Proposed Bulletin on Peer Review and Information Quality. Sincerely, Bill Colglazier, 202-334-3000, bcolglaz@nas.edu <<OMB.Final.doc>> - OMB.Final.doc December 15, 2003 Dr. John D. Graham, Administrator Office of Information and Regulatory Affairs Office of Management and Budget c/o Dr. Margo Schwab New Executive Office Building, Room 10201 725 17th Street, NW Washington, D.C. 20503 Dear Dr. Graham: We appreciate the opportunity to submit comments on the proposed OMB Bulletin on Peer Review and Information Quality. We were also pleased, at your request, to host the workshop on Peer Review Standards for Regulatory Science and Technical Information in the Main Auditorium of the National Academies on November 18 at which a distinguished group of speakers and a substantial audience of interested and knowledgeable persons had an opportunity to discuss the proposed bulletin in some detail and to provide individual views. We hope that you found this meeting useful. We endorse the fundamental objective of strengthening the federal government's peer review processes for the dissemination of scientific and technical information relevant to regulatory polices, particularly with respect to especially significant regulatory information. We agree that greater use of peer review can improve the technical quality of such information, enhance the credibility of government information, and help ensure that all federal agencies use the best available science. However, the highly prescriptive type of peer review that OMB is proposing differs from accepted practices of peer review in the scientific community, and if enacted in its present form is likely to be counterproductive. Our institution has decades of experience, most notably through the National Research Council and the Institute of Medicine, in providing independent external analysis, recommendations, and advice to the federal government. Drawing upon our collective institutional experience, I offer the following comments on the proposed bulletin. Different Types of Peer Review Exist that Are Applicable to Different Purposes Peer review is a well-established concept widely practiced within the scientific community. However, it takes different forms and uses different procedures depending upon the circumstances and the functions to be performed. The peer review methods practiced at scientific journals, for example, differ somewhat from each other, and the same is true of the methods used to review grant applications by funding agencies. Different types of peer review are used because the circumstances and the functions to be performed are different. As an illustration, the peer review conducted by a typical scientific journal involves experts selected by that journal who review articles submitted for publication. The journal editor considers these reviewers' comments in deciding whether or not to publish the article. The author receives reviewers' comments anonymously, and the reviewers' comments are not made public. Each journal has its own screen on potential conflicts of interest of reviewers. This type of peer review would not be consistent with the prescriptions in the draft OMB guidance if this guidance were applied to scientific journals. It is also worth noting that very high quality, peer reviewed, scientific research articles and reports by highly respected research teams can, and sometimes do, reach differing conclusions and results on substantially the same research subjects. That is not a weakness of science, of the scientists performing the research, or of the peer review process. It is simply characteristic of the initial difficulties often encountered in charting the unknown. Although peer review is an important tool of the scientific process, science primarily advances through further research and testing and the challenges from new data -- all this leading to further scientific debate, discovery and insight. Over time science is self-refining and self-correcting, as well as always open to the possibilities of unanticipated discovery and change. Thus, the fact that a particular scientific finding has been published in the peer reviewed literature is certainly not sufficient grounds for accepting it as correct. The reports that are produced through the National Research Council and the Institute of Medicine should be considered as a further stage of peer review, and one reason that agencies seek such overarching reviews is that uncertainties exist in the current scientific knowledge published in the peer reviewed scientific literature. The National Academies conduct reviews of such scientific information for many agencies. The appointment of our committees conducting such reviews is totally controlled by our institution, and we comply with the special provisions of Section 15 of the Federal Advisory Committee Act that ensure our reports are independent of the sponsoring agencies. We have our own rigorous process for screening for potential "conflicts of interest," and we carefully balance the scientific views on each committee in a way judged to be appropriate for the particular task under investigation. We also submit our draft committee reports to a rigorous internal peer review process where we appoint additional experts to review the draft and to make written comments that we provide anonymously to the study committee. Only when the study committee has responded satisfactorily to these reviewers' comments, in the view of the chair of our Report Review Committee and the Chair of the National Research Council, is a final report approved for release to the sponsoring agency and the general public. Neither the preliminary drafts nor the comments of reviewers are made public in order for us to ensure that both our committees and our reviewers are free from pressure from sponsoring agencies and interest groups. The names of our reviewers, however, are made public when a report is released. In the way just described, we seek to provide independent, high quality, and objective assessments of scientific knowledge relevant to many important public policy issues. Nevertheless, it is important to note that, if the draft OMB guidance were put into effect, federal agencies may feel that, by continuing to rely on National Research Council and Institute of Medicine reports, they may not be in compliance inasmuch as neither the draft report that entered review nor the comments of our reviewers are made public. The Proposed Bulletin Is Too Prescriptive Based on our extensive experience with peer review, we conclude that the proposed bulletin lays down a variety of specific requirements for the proposed peer review process that will in some cases be counterproductive. Most notably perhaps, these requirements are not consistent with many types of peer review that have been successfully used for decades in the scientific community. In addition, many federal agencies that deal with scientific information have developed reasonable peer review processes applicable to their own situation that also appear to be inconsistent with the prescriptive approach taken in the OMB bulletin. Many federal agencies are quite knowledgeable about what peer review can and cannot do. In addition, there are concerns that any overly prescriptive approach will become so onerous and time consuming that it will reduce the amount of sound scientific information and analysis that is available as input to important public policy decisions and to an informed public. The proposed bulletin, for example, reflects considerable concern and has fairly specific requirements regarding possible conflicts of interest on the part of peer reviewers. At the National Academies, we strongly agree that our consensus committees and panels must be balanced and free of conflicts of interest, but that same principle does not extend to our report reviewers. We have found that the sharpest and most interested critics can sometimes serve as the best reviewers because their critiques test the soundness of draft reports. In the end, it is our institution, in consultation with the members of each authoring committee, that makes the final decisions about a report based on the facts and on an independent judgment and analysis without any further participation by the reviewers. But the extensive reviewer comments that are received through our report review process are extremely valuable in testing ideas and for exposing weaknesses in data, reasoning, and analysis. Given the relatively narrow and specific function of peer review, it is also not clear why the proposed OMB bulletin should have such extensive conflict of interest requirements. So long as the group of experts consulted is sufficiently broad, diverse, and balanced to fairly reflect all points of view, adoption of unnecessarily rigid conflict of interest requirements may exclude some of the most useful and knowledgeable experts, directly conflicting with the proposed bulletin's underlying goal of enhancing the quality of information disseminated by the federal government. Thus, for example, were we to exclude as committee members all those scientists whose research is or has been funded by the National Institutes of Health when providing advice to that agency, we would severely weaken the expertise needed to provide them with sound advice. The proposed bulletin also places particular emphasis on the public transparency of the proposed peer review process. At the National Academies, for both legal and strong institutional policy reasons, we make our reports available to the public on the worldwide web to the maximum extent possible. So we clearly recognize the general value of transparency, and our final reports that provide peer review of scientific information for federal agencies are made available for anyone to comment upon and critique. However, we do not make public the comments of reviewers who comment on our draft reports in order to ensure that these reviewers can be candid and that our final reports are truly independent and represent our best effort to provide high quality objective analysis. Similarly, while the proposed bulletin is careful to note the importance of protecting the confidentiality of federal agency deliberative processes, it does not address how that confidentiality can be reconciled with a completely transparent peer review process for scientific information that may be central to the agency deliberative process. For these and other reasons, we suggest that the proposed bulletin be framed in terms of general principles and objectives rather than rigid procedures, and that OSTP and OIRA work with the individual agencies in specific cases to craft peer review processes that are tailored to achieve those objectives within the context of the particular circumstances involved. In addition, OMB should state explicitly that reports from the National Academies (National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council) are generally presumed to be adequately peer reviewed, as the draft guidance has stated for publications in scientific journals, as long as we comply with the special provisions of Section 15 of FACA. It has been suggested to us that some agencies may be restricted from seeking our reports reviewing scientific information because of the prescriptive nature of the draft OMB guidance. As noted above, our reports are independent of the sponsoring agencies, and they are viewed by many audiences as being among the most highly credible and objective assessments of current scientific knowledge. We would be happy to provide you with additional views as OMB and OSTP work to revise the guidance for the laudable purpose of improving the quality of the scientific information that is used in important public policy decisions. Sincerely, Cc: John Marburger Bruce Alberts President, National Academy of Sciences Chair, National Research Council