Document zdkE7zKNy6xDZb4baz7KNayBB

Anne Taylor <ataylor@cogr.edu> 12/12/2003 02:05:58 PM Record Type: Record To: Mabel E. Echols OMB_Peer_Review/OMB/EOP@EOP cc: Subject: Comment Letter on Peer Review and Information Quality - Schwab.doc COGR COGR an organization of research universities RELATIONS BOAaRnDoOrFgaDnIiRzaEtCioTnOoRf Sresear MARY ELLEN SHERIDAN CHAIR Unive(rFsiAtyXof)Chicago JANET ACKERMAN Yale UCnHivReIrSsiTtyOPHER MC CRUDD CHAIR WENDPrYinBcAetLonDUWnIiNversity University of Kentucky JANET ACKERMAN MARKYaBleRUEnNivNeErsRity Indiana University (IUPUI) MARK BRENNER JERRY BRIDGES The JoWhNnsInHdoiapnkainUs nUinveivresritsyit(yIUPUI) PETEJREDRURNYNBRIDGES PurduTehUenJiovhenrsitHyopkins University JERRPYEFTIEFRE DUNN VandePrubridltuUenUivneivrseirtsyity JILDACADRIELHALFGISAHRMTAONN GeorgTiaulIannsetitUunteivoefrsTiteychnology TODDJIGLUDTATDMIEAHNL GARTON The OGheioorSgtiaateInUsntiitvuetresoitfyTechnolog ALBEERRTICHAOHRV. KARTOHPP CalifoUrniaveIrnssitiytuotfeMofaTryelcahnndology KATHJOLESEPNHIRMWUILNLINIX UniveUrsnitiyveorfsiWtyisocfoCnasilnif-oMrnaidaisSoynstem ERICANHD. RKERWOPNPEIGHBOUR UniveUrsnitiyveorfsiMtyaorfylCaanldifornia, Los A GUNTAANLDIRDEEWRSB. RUDCZYNSKI UniveUrsnitiyveorfsiRtyocohfePsetenrnsylvania JOSEJPAHMMEUSLSLEIVNEIXRSON UniveCrsoitryneolfl CUanliivfoersniitaySystem ANDRMEAWRNYEEILGLHEBNOSUHRERIDAN UniveUrsnitiyveorfsiCtyaloifoCrnhiac,aLgoos Angeles ALICE A. TANGREDI-HArsi MARVIN PARNES UniveDrsOitUy GofLMAiSchKig. aTnRUE University of Iowa YOKE SAN REYNOLDS UniveVrs'iEtyLoLfAVWirgAinRiaREN University of Washington ANDREW B. RUDCZYNSKI UniveJrAsitNyEofYPOeUnnNsGylEvaRnSia University of Texas Health Scie V'ELLA WARRCEeNnter at San Anton University of Washington JANE YOUNGERS UnivePrsrietsyidoefnTtexas Health Science Center at San Antonio KATHARINA PHILLIPS President COUCNOCUILNCOINLGOONVGEORVNEMRENNMTEANLTRAELLATIONS 1200 New York Avenue, N.W., Suite 320, Washington, D.C. 20005 (202) 289-6655/(202) 289-6698 (FAX) 1200 New York Avenue, N.W., Suite 320, Washington, D.C. 20005 (202) 289-6655/(202) 289-6698 December 15, 2003 Dr. Margo Schwab Office of Information and Regulatory Affairs Office of Management and Budget 725 17th Street, N.W. New Executive Office Building, Room 10201 Washington, D.C. 20503 OMB_peer_review@omb.eop.gov Re: Proposed Bulletin on Peer Review and Information Quality 68 FR 54023-29 Dear Dr. Schwab: The Council on Governmental Relations is an association of over 150 of the leading research universities in the United States, as well as several affiliated hospitals and research centers. During OMB's development of guidance to federal agencies on how to meet the standards for information dissemination set by P.L. 106-554 and Executive Order 12866, as amended, COGR commented on the proposed standards of objectivity, utility, quality and integrity which federal agencies must meet. COGR strongly endorsed peer review as the appropriate process for meeting these standards. While OMB acknowledged that the academic peer review of publications is the "gold standard" for academic science, it nevertheless mandated that agencies must perform additional peer review in cases of specific sensitive technical and scientific information. In the subject guidance, OMB has now proposed detailed peer review criteria to be adopted by the federal agencies. Although we supported the goals of the information quality standards, we find that the peer review which OMB is proposing for federal agencies for regulatory inforKmAaTtiHonAdRiIfNfeArsPfHroImLLpIePeSr review which has been successful as a publication standard for journal review of submitted manuscripts, for federal agency review of grant proposals and as an assessment tool for promotion and tenure in academia. This "regulatory" peer review is likely to be counter productive for the following reasons. Dr. Schwab December 15, 2003 Page Two The scope of the proposed guidance is broad and not justified in comparison to the resources it will consume both on the part of the federal agencies and the research community. The proposed processes are quite prescriptive and likely to diminish the quality of peer review by preventing the most capable scientists from participating in the peer review process. Imposition of these requirements will increase the cost and time needed for peer review and will change federal rulemaking to a much broader extent than implied. In short, the guidance as proposed could result in peer reviews of lesser quality than currently typically performed at the federal level and in reducing rather than enhancing quality as expected. Our reasons for these conclusions are based on our evaluation of the impact of the guidance on three main areas: the scope of review, the quality of the peer review process and the impact on future regulatory requirements. They are summarized below. The Appendix discusses our concerns in more detail. COGR is concerned that substantial federal and research community resources will be spent on unnecessary peer review because the scope of "significant regulatory information" is not well defined and therefore likely to be interpreted broadly by federal agencies. To avoid unnecessary expenditures, OMB should establish objective criteria, which would limit "regulatory" peer review to truly significant or exceptionally significant regulatory information. After reviewing the various criteria to be observed by federal agencies in the peer reviewer selection process, we conclude that OMB would be better served in describing the desired outcome rather than prescribing the process. In particular, by potentially excluding from peer review of "especially significant regulatory information" all scientists who are currently funded or have applied for funding from the agency responsible for the research under peer review, OMB is establishing a basis for excluding the most capable and knowledgeable scientific experts from serving the government's needs, particularly federally funded scientists at universities. OMB has been zealous in introducing the opportunity for public comment at multiple stages, as the agencies prepare their regulatory information for public dissemination. While this is laudable in principle, in practice it may hamstring the process unduly. We provide three examples of likely outcomes in the Appendix and hope that OMB will not overly constrain the process. OMB asked for comments from the affected agencies on the expected benefits and burdens of the proposed guidance. We believe OMB should not overlook the potential burden on the research community. Scientists typically serve as peer reviewers for federal agencies on a volunteer basis. The proposed guidance appears to assume that researchers will be willing to also serve as regulatory peer reviewers but the same incentives may not apply as in the case of research peer review, and there is no indication that OMB envisions providing reviewers with any compensation for their service. Dr. Schwab December 15, 2003 Page Three In conclusion, COGR finds this proposed guidance to be inconsistent with the federal government's increasing transition to less prescriptive, performance based requirements which enable each agency to apply its expertise in the manner best suited to its mission. Given the resource intensive nature of the proposed requirements, we also do not believe that OMB has clearly justified the need for the proposed guidance from a cost/benefit standpoint. Accordingly, COGR recommends that OMB revise this guidance to focus it much more narrowly and address the concerns identified. We appreciate this opportunity to provide comment and are available to answer further questions. Sincerely, Attachment Katharina Phillips Appendix 1. Scope of the Review Critical to assessing the scope of this guidance is the understanding of the term "regulatory information", which is defined as any scientific or technical study that might be used by domestic or international regulatory bodies. "Study" includes any research report, data, finding or other analysis. Since no criteria are proposed to guide agency interpretation, we believe that the deliberate breadth of the term study when combined with the term might, could result in the agency's identification of a large volume of material as regulatory information. "Significant regulatory information" is regulatory information that satisfies OMB's influential test in the Information Quality Guidelines. Information is influential if an agency can reasonably determine that dissemination of the information will have or does have a clear and substantial impact on important policies or on important private sector decisions. Again, we believe that federal agencies will err on the side of being overly cautious in their assessment of what is reasonable and important and thereby subject to the proposed guidance, especially since they are to report to OMB on their peer review activities. This same logic applies to the proposal to exclude from peer review any studies that have already been adequately and independently peer reviewed or science that is not directed toward regulatory issues. The term adequate is subject to broad interpretation. Moreover, the adequacy of a prior peer review, even a peer review by a reputable scientific journal, may be challenged. The bulletin establishes only a rebuttable presumption of adequacy of prior peer review that may be challenged on the basis of a persuasive showing. The term independent is defined only for the category of "especially significant regulatory information" in Section 3, from which it could be extrapolated. That would mean importing into the review of "significant regulatory information" in Section 2 all the selection criteria for peer reviewers proposed primarily for only "especially significant regulatory information." We are not assured that prior peer reviews will be recognized as sufficiently able to reduce the number of studies that are subject to additional peer review under these guidelines. It is also unclear whether the Section 2 exemption for studies that have already been subjected to adequate independent peer review also applies to Section 3, "especially significant regulatory information", or whether an additional level of per review is required regardless. If required, it is not clear what the purpose of this additional peer review should be. Finally, the exclusion for science that is not directed toward regulatory issues is not contained in the proposed bulletin but only in the accompanying supplementary information. As a result, COGR believes that scientific research that is not intentionally directed toward regulatory issues but which produces results that affect regulatory issues, could be subject to the bulletin's peer review requirements for significantly regulatory information, including research supported by NIH, NSF and other funding agencies. COGR recommends that the exclusion for scientific research be incorporated into the bulletin. Appendix Dr. Schwab December 15, 2003 Page 2, 2003 2. Quality of the Peer Review Process OMB should not presume that scientists are tainted by receipt of research funding from a particular agency. A history of peer reviewed federal support demonstrates expertise recognized by other experts in the fields and thus should be a positive attribute of potential reviewers rather than a basis for exclusion. More appropriate would be a focus on excluding potential reviewers with professional or personal financial interests in the industry which is the subject of the anticipated regulations. It would be much more productive to call for the best available expertise, vetted through conflict of interest disclosures. Such policies are already in existence at two major funding agencies, NIH and NSF, and at respected peer reviewed journals and should be recommended to those agencies that currently have neither peer review nor conflict of interest policies. Scientists should also not be subjected to potential exclusion if they have served for the same agency on specific matters before, because in some specialized areas it is difficult to find the requisite expertise. Finally, while avoiding bias is a goal we share, the remedy of appointing a person with a different bias for the sake of balance does not address the core problem: avoiding bias altogether, by identifying is as non-scientific views, which should have no place in scientific deliberations. OMB calls for per review reports, in order to provide transparency of the process as well as permanent records of all deliberations and conclusions. This goal conflicts with another goal, which OMB has set for itself: obtaining frank assessments. It is the reason that OMB reserves for itself the right not to share the deliberations between it and the funding agencies on some peer review decisions. Peer reviewers need that same protection in order to provide frank assessments. Grant making agencies like NIH and NSF have well-established practices in this area that OMB may wish to look to for guidance. While listing the names, association and qualification of the peer reviews is legitimate, individual assessments should be provided without individual attribution, only as part of the overall agency group review. Such procedure would not conflict with the original guidance's transparency standards. 3. Impact on Future Regulatory Requirements The OMB proposal would require that each federal agency provide an annual summary of any existing, ongoing or contemplated scientific or technical study that might, in whole or in part, constitute or support significant regulatory information, which the agency intends to disseminate within the next year. In addition, OMB requires that the federal agency provide detail about the proposed review for such studies. COGR believes that the current semiannual reports that agencies send to OMB already fulfill this purpose and that this additional requirement should not be imposed. Even if drawn more narrowly, it would require the expenditure of federal resources that are in short supply. Appendix Dr. Schwab December 15, 2003 Page Three By providing opportunity for public comment on the regulatory material and for such comments to be considered by peer reviewers, scientists will be expected to spend considerable time to review comments provided by other parties that may be interested but not qualified to participate in the discussion. This will result in the unnecessary expenditure of resources on matters that do not require the expertise of peer reviewers. This also may raise issues as to potential documentation and record keeping responsibilities and burdens for reviewers. We believe that the rulemaking process provides adequate opportunity for public input. Finally, OMB has changed the February 2002 guidance with respect to the handling of correction requests under the Information Quality Act. While originally required to report only the number and frequency if complaints, agencies now have to provide a copy of their draft response for correction requests or appeals at least seven days prior to the intended issuance and the agency will be expected to consult with OMB regarding the consistency of its response with the OMB guidance and its own policies. We believe that OMB should step back from imposing such control over the federal agencies and should also advocate no more than minimal use of interagency comments in addition to agency peer reviews, especially if such reviews might inject even additional third parties into the already complex peer review process. ---