Document RjDO3ERY6NRDJappE23xQzaNk

Stuart Shapiro <stuartsh@rci.rutgers.edu> 12/12/2003 05:50:11 PM Record Type: Record To: Mabel E. Echols OMB_Peer_Review/OMB/EOP@EOP cc: guston@rci.rutgers.edu Subject: comments As requested, please find comments both below in the body of this message and attached (note: footnotes did not copy to the body of the message, they are only in the attachment). If they are not transmitted adequately or if you have any other questions, please contact Stuart Shapiro at stuartsh@rci.rutgers.edu or 732-932-2499 ext 870. Thank you for your attention and consideration. - OMB public comments..doc Stuart Shapiro Ph.D.1 Assistant Professor Edward J. Bloustein School of Planning and Public Policy Rutgers. The State University of New Jersey 33 Livingston Avenue New Brunswick NJ 08901 732-932-2499 ext 870 stuartsh@rci.rutgers.edu David Guston Ph.D.2 Associate Professor Edward J. Bloustein School of Planning and Public Policy Rutgers. The State University of New Jersey 33 Livingston Avenue New Brunswick NJ 08901 732-932-2499 ext 707 guston@rci.rutgers.edu Comments on The Office of Management and Budget's Proposed Bulletin on Peer Review and Information Quality. OMB has requested comments on its proposal to require agencies to conduct peer review of significant regulatory information (68 FR 54023). These comments are in response to that request. OMB's goals of promoting sound science in the regulatory process are to be applauded, but the proposed bulletin appears to do little to achieve these goals. At the same time, these guidelines will impose significant costs on society. While there are significant drawbacks to any requirement of peer review on the regulatory process, the particulars of OMB's proposal exacerbate these drawbacks. Early in Dr. John Graham's tenure as OIRA Administrator, he announced that OIRA would give added deference in review of regulations under Executive Order 12866 to those regulations supported by peer reviewed analyses.3 This flexible incentive-based approach is likely to work much better than the proposed guidelines, which impose a uniform set of requirements on a diverse collection of regulatory agencies. The comments below are intended to highlight the problems with the proposed guidelines. We support both OIRA's goals of ensuring that regulations are based on sound science and its longstanding effort to push agencies to analyze the economic 1 Dr. Shapiro is an assistant professor of public policy and has written on, researched and teaches a class on the regulatory process. Prior to his appointment at Rutgers University he was a desk officer in OIRA for five years reviewing regulations primarily in labor and health policy. 2 Dr. Guston is an associate professor of public policy. He is an expert on the role of experts in the public policy process. He is also the North American Editor of Science and Public Policy and a fellow at the American Association for the Advancement of Science. 3 See for example John Graham speech on May 8, 2002 before Organization Resource Counselors. impacts of regulatory alternatives.4 It is unlikely however that the proposed bulletin will further either of those efforts. We urge OMB to reconsider the regulatory peer review requirements and consider more flexible alternatives to achieve its goals. I. Analyzing the Peer Review Guidelines Using OMB's Standards The first thing that OMB's guidelines on regulatory analysis require of agencies proposing new regulations is a statement of need of the proposed action.5 Such a statement of need is absent from OMB's proposed guidelines. There are assertions that peer review is critical in ensuring the reliability of scientific analyses and "existing agency peer review mechanisms have not always been sufficient to ensure the reliability of regulatory information." There is no evidence in the proposal supporting these claims however. Such evidence would need to show two things. First, it would need to demonstrate that current agency scientific and economic analyses are not sufficiently reliable or objective. Examples of agency analyses that fail to meet these criteria would be critical evidence of this nature. Second, a demonstration that peer review would improve the reliability or objectivity of agency work is necessary. It should be noted that agencies are often under strict legislative or judicial deadlines to produce regulations and in these circumstances peer review is unlikely to make any difference in the quality of analyses. There are other reasons to question whether peer review would improve agency scientific analyses. Peer reviews, whenever received by agencies, might not lead to consensus about the soundness of the science. As the recent case of gross error in research on the neurological effects of the drug Ecstasy shows, peer review is no guarantor of truth. This problem will be even greater for economic analysis (also subject to the guidelines) since there is less consensus in the social sciences than the natural sciences about what constitutes appropriate work. OMB also requires agencies to measure the costs and the benefits of its regulatory actions. While this bulletin is not a regulation, by impacting the regulatory process, it will have impacts of the same nature as regulations. In the appendix of this document, we have done an extremely rough calculation of the costs of these guidelines and discuss whether the benefits are likely to justify the costs. The largest cost of these guidelines is the delay that they will create in the promulgation of rules with net benefits. We give a most likely estimate of these costs as $325 million per year and note that this is likely to be low if OMB interprets the guidelines to apply to a broad array of regulations. The benefits that OMB anticipates are largely based on how peer review works in the academic setting. There are likely to be significant differences between regulatory peer review and academic peer review however. These differences are discussed in the next section of these comments. Certain 4 The lead author was an OIRA desk officer for five years and happily pursued those goals. 5 OMB Circular A-4 aspects of OMB's proposal will further increase the costs and decrease the benefits. These are discussed in section III. Because of these factors, we find it extremely unlikely that the benefits will exceed or even justify the costs. Therefore this bulletin would likely be a candidate for a return letter if it were reviewed by OMB. II. Why Regulatory Peer Review is Different than Academic Peer Review The organized scrutiny of peer review in journal articles and of funding proposals is believed to reduce error, subjectivity, and self-delusion in descriptions of research and increase the quality of work for publishing or funding. Peer review has proven a useful tool to help journal editors and program managers determine the allocation of scarce funds and limited journal space. Traditional peer review preserves some discretion for editors to publish or program managers to fund research. This discretion reflects the function of academic peer review as providing information to editors and program managers rather than determining the outcome of those decisions. Regulatory agencies, however, will face political threats and endless legal challenges in completing regulations that receive any negative peer reviews.6 Imposing these guidelines would mean transferring some of the discretion held by regulators responsible to the President and Congress to unaccountable technocrats. Any individual peer reviewer will have the potential power to derail a rulemaking effort by providing a negative peer review. This is not the case in academic peer review, where editors and program managers are free to ignore negative reviews of articles or proposals. The context of regulatory peer review is very different however. It is very easy to envision a court using a negative peer review as evidence that an agency was arbitrary and capricious in promulgating a regulation. It is also very easy to envision political actors hostile to a regulation using a negative peer review to attempt to derail regulatory initiatives for purely political reasons. And such negative peer reviews need not (and indeed are likely not to) come from peer reviewers with particular anti-regulatory agendas. As discussed above, disagreement in the sciences is common. Disagreement in economics is rampant. It is unlikely that very many analyses that agencies submit for regulatory peer review will result in unanimous endorsement. Such a lack of consensus, which is useful in the academic setting may provide a death blow for regulatory efforts in the policymaking setting. This may be true even for regulations with large net benefits. Derailment of such regulations would raise the cost of these guidelines significantly above the $325 million estimate. 6 Unless the regulation has already been required by a court in which case, as discussed above, peer review is unlikely to yield any benefit. III. Problems with the particulars of the OMB Proposal While we find it very unlikely that any requirement for peer review of regulatory documents will have benefits that justify its costs, the particulars of OMB's bulletin make it even less likely that such requirements would pass this test. Three of these requirements in particular are discussed below: Exclusions Many journals and funding agencies bar scientists who have financial interests in a study from reviewing the study. OMB's proposed guidelines would similarly bar those who have financial interests in the regulation, as well as those who have previously been advocates on the matter. But the guidelines would also bar those who have previously conducted peer reviews for the agency and, most troublingly, those who receive funding from the agency to conduct research. Funding agencies use neither of these latter exclusions in peer review. Indeed one of the goals of agencies like EPA, who sponsor research but also have regulatory missions, is to create a cadre of experts to contribute to informed policy debate. With these exclusions, much of the community that could contribute expertise to the issues in an agency analysis will not be permitted to be peer reviewers. Agencies will be forced to go far afield to find individuals who are not subject to the exclusions. Furthermore an agency economic analysis or risk assessment is often longer and more complicated than the occasional journal article that many scientists and economists are asked to review. Many academics will not have the time to contribute meaningful input as peer reviewers particularly if they have to overcome a learning curve because they are reviewing something outside their area of expertise. Anonymity The proposal also fails to protect the integrity and reputation of regulatory peer reviewers with the anonymity that journal and funding peer reviewers receive. This anonymity is something that many journals find essential in order to produce peer reviews that are honest and unbiased. There is a tradeoff between the incentives that anonymity gives peer reviewers to provide impartial feedback and the government's need for openness and transparency. OMB must understand that by deciding this tradeoff against anonymity, that a disincentive for participation as a regulatory peer reviewer has been created. Coupled with the exclusions discussed above this disincentive will force agencies to recruit reviewers who are more peons than peers. Public comment period The guidelines require the agency to "provide an opportunity for other interested agencies and persons to submit comments." This appears to be a public comment period on the analyses supporting a rule that will be in addition to the public comment period on a proposed rule required by the Administrative Procedure Act. The purpose and anticipated benefit of this additional comment period is not clear and OMB does not discuss it anywhere in the proposed bulletin. Since agencies are required to hold this comment period to "ensure that such comments are provided to peer reviewers" one can infer that the purpose is to allow peer reviewers to see opposing viewpoints on the analysis when conducting their review. This is a peer review process unlike any in the academic setting. If the purpose of peer review is the application of objective expertise to scientific questions, then requiring agencies to provide subjective public comments to reviewers appears to be directly contrary to this purpose. The public already has an opportunity to assess the documents supporting a rulemaking through the APA. This additional comment period appears to both compromise the ostensible benefits of peer review and increase its costs by increasing the delay that peer review will impose upon the regulatory process. IV. The Impact on the Public Comment Process In this Administration, the Office of Management and Budget has shown a praiseworthy commitment to openness and transparency in the regulatory process. The efforts to expand e-rulemaking are particularly impressive. Such efforts are likely to increase the use of the public comment process and will hopefully lead to agencies hearing and responding to a wider array of views on their regulatory proposals. Unfortunately the peer review initiative may counteract much of the Administration's work in opening up the regulatory process. As we read the guidelines, they require peer review of all significant regulatory information. The requirements are even more specific for "especially significant regulatory information." It makes no distinction as to whether information in these categories is generated by the agency or by outside parties. One of the most important purposes of the public comment process is to give agencies information that they might not have otherwise. However, if any public comments count as "significant regulatory information," then agencies may be forbidden to rely upon them unless the agency peer-reviews them. It is hard to imagine an agency delaying a rulemaking further in order to peer review public comments. Trade associations, individual firms, environmental groups, and even academic experts may find their public comments ignored for the very reason that they are technical, but have not met OMB's burdensome standard of peer review. This potentially calls into question the Administration's initiative on erulemaking. If the Administration is indeed interested in encouraging a greater diversity in public comment and greater access to the government, then agencies must be allowed and encouraged to listen to the new voices they will be hearing. If instead, agencies are given incentives to ignore these voices, then the sense of participation is a sham. Many already deride the public comment process as insufficiently responsive to public concerns. These guidelines threaten to formalize this by giving agencies OMB sanctioned reasons not to be responsive. V. Conclusion The proposed bulletin raises many troubling questions. While the goal of promoting sound science and economics is a laudable one, the requirements for regulatory peer review are unlikely to achieve this goal and will create numerous other problems in the regulatory process. As discussed above, the idea of requiring regulatory peer review at all is likely to cause greater harm than good. Requiring it in the manner proposed by OMB increases the costs while decreasing its benefits. OMB should revisit this approach and conduct a careful analysis before considering whether to finalize these guidelines. The flexible approach that OMB has adopted, rewarding agencies that conduct peer review with more deference under E.O. 12866 is a sound one. This frees agencies to make the decisions as to when peer review is appropriate and as to what process is appropriate. It also allows agencies to be responsive to their stakeholders in a way that a requirement for peer review does not. Appendix: Rough Estimate of Costs and Benefits of Proposed OMB Peer Review Guidelines. OMB Circular A-4 issued in September 2003, states that a regulatory impact analysis should have three elements: 1. A statement of the need for the proposed action. 2. An examination of alternative approaches. 3. An evaluation of the benefits and costs of the proposed action.7 OMB's draft guidelines, also issued in September 2003 had none of these three elements associated with them. While the guidelines do not reflect regulatory requirements, they will have profound impacts on regulatory policy and by extension will create social benefits and costs. The discussion below is a very rough attempt at the third of the three elements that OMB suggests for regulatory impact analysis. We urge OMB to put together a comprehensive regulatory impact analysis, which contains all three elements. Costs The primary cost of the OMB guidelines is not likely to be the actual cost of implementing the peer review requirements but rather the delay that will result from subjecting proposed and final regulations to additional procedural requirements. If this delay just impacted rules with net social costs, then the costs of the OMB guidelines would be minimal. However the scope of the guidelines makes clear that they are 7 OMB Circular A-4 intended to apply to all economically significant regulations as defined in Executive Order 12866. Many of these rules have net benefits for society. Some have net benefits that are quite large. In its 2003 Report to Congress on the Costs and Benefits of Federal Regulations, OMB tallied the net benefits of a subset of these economically significant rules. OMB looked at 107 rules issued between1992 and 2002 and found that the benefits of these rules totaled $146,812 to $230,896 million and the costs totaled $36,625 to $42,813 million.8 In attempting to calculate a net benefit per economically significant rule there are several assumptions that must be made. As OMB notes in its report to Congress the above figures do not include regulations where there was no reliable information allowing the quantification of benefits or costs. This likely biases the estimates of net benefits upward. OMB also notes that a large portion of the net benefits come from only four EPA rules. Without these four EPA rules "the aggregate benefits and costs of the other 103 rules range from $41 to 107 billion and $29 to34 billion respectively."9 We therefore use three estimates of the total net benefits of regulations between 1992 and 2002 in order to derive an estimate of net benefits/ rule. 1. The low end of the range of benefits and the high end of the range of the costs of all 107 rules gives an estimate of $104 billion in net benefits. 2. The low end of the range of benefits and the high end of the range of the costs of the 103 rules (exempting the 4 EPA rules) gives a figure for total net benefits of $7 billion. 3. The midrange of both the benefits and the costs of the 103 rules gives a figure for total net benefits of $42.5 billion. These figures should then be divided by the number of rules to give a result for net benefits/rule. Doing so gives a value of $971 million per rule for method 1, $65 million per rule for method 2, and $412 million per rule for mehtod 3. How much delay will the OMB guidelines create in the production of regulations? A reasonable minimum seems to be three months. It seems impossible that an agency can send a document to a peer reviewer, the peer reviewer can review it, and the agency can respond to the comments in less than three months. Any shorter peer review would be likely to produce none of the benefits that OMB claims these guidelines will have. A more realistic estimate is likely to be six months as many peer reviewed journals warn authors of academic articles to expect a delay of this magnitude. A third possible estimate is a year. If an agency has to go through multiple rounds of peer review or has to include an additional public comment period prior to peer review of studies, it is easily imaginable that delays of at least a year could be added to the regulatory process. 8 Informing Regulatory Decisions: 2003 Report to Congress on the Costs and Benefits of Federal Regulations and Unfunded Mandates on State, Local and Tribal Entities 9 Ibid. The table below takes the three values for net benefits per rule and the three estimates for the amount of time peer review will delay the rule to arrive at nine possible values for the annual costs of peer review. The table assumes that there will be 10 rules per year subject to the OMB guidelines (based on the 107 economically significant rules in the 11 year period spanning 1992 to 2002). The figure marked with an asterisk will be used as a most likely cost figure since we believe that a delay of six months per rule is a reasonable expectation, and we use the lowest net benefits per rule number to give the most conservative cost estimate. Netbenefits $65 million per rule $412 million per rule $971 million per rule Delay of 3 months $162.5 million $1.03 billion $2.42 billion Delay of 6 months $325 million* $2.06 billion $4.85 billion Delay of 1 year $650 million $4.12 billion $9.7 billion As can be seen the most likely value for costs is $325 million per year. This obviously leaves out a great deal of information about the costs of the OMB guidelines. Are the omitted factors likely to lead to a higher or lower cost estimate? There are reasons to assume bias in either direction. There are three relatively obvious reasons that the estimate may be low. 1. The estimate does not count the administrative costs for an agency of running a peer review process or the value of the time spent by scientists or economists reviewing the documents. This cost is likely to be small relative to the cost of delayed net benefits in the table above. 2. The number above assumes that there will only be ten rules per year subject to these requirements. The guidelines however give OIRA considerable discretion to subject significantly more rules to peer review requirements then those that are economically significant. Indeed, many economically significant rules (including those rules which transfer more than $100 million, such as the rules governing the Medicare program) are not counted in the totals above. While rules besides those making up the $325 million annual cost are likely to have smaller net benefits than the rules that are tallied above, collectively they will likely also include a significant amount of net benefits. Also while delayed transfers do not have true economic costs, they do represent a delay in the policymaking process that presumably has policy consequences. 3. Finally as noted in the discussion of the costs of delay, the value highlighted, $325 million, does not include the EPA rules with the largest net benefits. A delay in one such EPA rule would increase the costs to billions in any given year. There are also several reasons to believe that the $325 million estimate might be high. 1. Some of the rules covered by the guidelines may get exemptions from the peer review requirements. Given that most of the rules covered by the reports do not fall into the national security or homeland security exemptions, it is unlikely that such exemptions will be common for these rules. 2. The numbers above were generated using rules from the past decade. Presumably these rules were "low hanging fruits;" the reasons that agencies pursued them were the significant net benefits. Future rules therefore will have lower benefits than current ones and therefore the costs of delaying such rules will be lower than the costs of delaying rules promulgated in the past decade. 3. As OMB notes, its tally of net benefits only includes rules for which good data on benefits and costs were available. It is reasonable to suppose that the rules in the OMB totals are the rules promulgated in the last decade with the greatest net benefits. If so, then the estimates of the costs of delaying economically significant rules are biased upward since the average rule is likely to have fewer net benefits than is reflected in the numbers derived from the OMB reports. Which of these factors is the most important? Without detailed research it is impossible to know. We tend to think that the guidelines are intended to apply to more than 10 rules per year and so factor number 2 of the reasons the estimate is low may be the largest factor. However, this uncertainty argues for a detailed examination of the costs of these guidelines. One other factor should be noted. The costs above would be reduced if agencies are already undertaking peer review in the fashion envisioned by the guidelines. Some agencies have claimed that they already undertake peer review of their analyses.10 While this is possible, it is unlikely particularly that economic analyses undergo the type of peer review specified here (and all of the rules in the $325 million calculation would have required economic analysis in addition to whatever scientific analyses were conducted) and that agencies follow the specific requirements of the guidelines such as excluding large numbers of scientists as peer reviewers and holding a public comment period as part of peer review. It should be noted that if this assertion is incorrect, then the costs would be reduced by the percentage of rules already undergoing peer review. However, if the peer review is already occurring, then the benefits of these guidelines would also have to be reduced accordingly. Benefits As any economist knows, calculating benefits is usually much harder than calculating costs. The primary benefit of these guidelines presumably is to stop the promulgation of rules that are not justified by their science or their economics. How many such rules will be prevented by these guidelines? More relevantly, will a 10 Paul Gilman of EPA at the NAS workshop on peer review on November 18, 2003 argued that EPA already does extensive peer review. minimum of $325 million worth of bad rules be prevented annually in order to justify the costs of these guidelines? There are many reasons to be skeptical that the guidelines will have this impact. Foremost among these reasons is that OMB did not cite any examples of rules that would have benefited from peer review in the preamble to their proposed guidelines. This does not mean that such examples do not exist but it certainly casts doubts on their existence. There are three other reasons to be skeptical that the peer review guidelines will have annual benefits approaching $325 million. First, if one examines the OMB reports on the costs and benefits of federal regulations, there are few rules promulgated with net costs of this magnitude. For example in the 2002 report, which covers an 18 month period, there are two rules that have net costs, EPA's arsenic rule, and Interior's Roadless Conservation rule. Several other rules do not estimate the net benefits (these rules may have net benefits but agencies did not estimate them).11 Second, assuming that a rule does have net costs, there is no reason to think that peer review will force the agency to change the rule. Peer reviewers may not criticize the scientific or economic studies supporting the rule, either because they didn't find particular problems, or the studies are not the sources of the net costs. Even if peer reviewers do criticize the scientific or economic basis for the rules, many of the rules that have the greatest net costs are driven by legal requirements12 or are political imperatives.13 It is unlikely in the case of political imperatives and impossible in the case of legal requirements for agencies to defer to peer reviewers and to decide not to promulgate a rule. Finally there is some possibility that peer review will lead to not just the delay of rules with net benefits but will stop their promulgation altogether. As noted in the body of this comment, scientists often disagree and economists very often disagree. Such disagreement may lead to the wrong course of action (from a welfare economics perspective) being taken and a rule with net benefits to society being stopped.14 If one regards this as an offset to the benefits of stopping bad rules (one can also treat it as an additional cost) then the benefits become even smaller. In short, the benefits of the peer review guidelines appear to be very unlikely to justify the considerable costs. Once again, any government action with costs of this magnitude deserves a much closer economic analysis of its consequences. We won't be inconsistent and suggest that OMB should have any such analysis peer reviewed, only 11 Indeed these guidelines may give agencies incentives to be as vague as possible in their economic analyses. Why make difficult benefit assumptions when such assumptions are likely to be criticized for methodological flaws when vague assertions about benefits are much harder to find fault with. 12 FDA's bioterrorism rules on registration and recordkeeping requirements for food producers are an example. 13 The roadless rule cited above is an example of a high priority of the Clinton Administration. 14 If these "good" rules are legally required or have widespread political support, then peer review will be less likely to have this effect. that they hold themselves to the same standard that they hold the agencies which they oversee.