Document 719ZewMbbynDLeORbM5b4VDkV

NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. THE NATIONAL ACADEMIES THE NATIONAL RESEARCH COUNCIL POLICY AND GLOBAL AFFAIRS DIVISION SCIENCE, TECHNOLOGY, AND LAW PROGRAM PEER REVIEW STANDARDS FOR REGULATORY SCIENCE AND TECHNICAL INFORMATION Tuesday, November 18, 2003 The National Academies Main Auditorium 2100 C Street, NW Washington, DC Proceedings By: CASET Associates, Ltd. 10201 Lee Highway, Suite 160 Fairfax, VA 22030 (703) 352-0091 NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. TABLE OF CONTENTS Page Welcome and Introductions - Dr. Kennedy 1 Keynote Address - Dr. Graham 3 Questions 22 OSTP's Role - Dr. Olsen 32 Questions 41 What is Peer Review? A Comparison of the Bulletin's Model of Peer Review with Other Types of Peer Review Moderator: Dr. Meserve 52 Three Models: FDA, HEI, Science - Dr. Kennedy 54 The Conceptual Framework - Dr. Jasanoff 62 Questions 73 The Range of Agency Utilized, Generated, and Disseminated Scientific Information: Academic Research; Agency Research; Warnings to the Public; CBI; Third Party Submissions Moderator: Mr. Anderson 81 Speakers: Dr. Gilman 83 Mr. Scanlon 97 Dr. Lenfant 109 Questions 114 The Boundaries of the Bulletin: Scope, OIRA Involvement Speaker - Mr. Morrison 126 Questions 141 NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Peer Review of Especially Significant Regulatory Information: Implications for Agency Peer Review Procedures -- Changes in Practices: Costs and Delays; Pre-Peer Review Public Comment Moderator: Dr. Kennedy 152 Speakers: Dr. Mahoney 153 Dr. Schaub 166 Dr. Michaels 175 Mr. Taylor 189 Questions 200 Peer Reviewers' Perspective: Implications for Scientists -- Transparency; Conflict of Interest; Burden; Joint Review; Agency Response; Public Comments Moderator: Dr. Bailar 207 Speakers: Dr. Washington 208 Dr. Omenn 213 Dr. Louis 225 Questions 231 Integration of Peer Review with the Requirements of the FACA and Administrative Procedure Act Speaker: Prof. Shapiro 253 Questions 262 Closing Remarks - Dr. Meserve 265 NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. P R O C E E D I N G S (9:02 am) Agenda Item: Welcome and Introductions - Donald Kennedy, Editor in Chief, Science; Co-chair, Science, Technology, and Law Program DR. KENNEDY: Ladies and gentlemen, good morning and welcome to the home of the National Academies. I'm Don Kennedy. I'm co-chair with Prof. Dick Merrill at the University of Virginia Law School of the Science, Technology, and Law Program at the Academies. We have been very interested in exploring the federal regulations and policies that govern access to, and evaluation of, and dissemination of information by the government, especially scientific research results used in formulating regulation. We began that exploration in 2001, with a working on the Shelby Amendment, and the growing tension between those who do research, and those who want to gain access to the underlying data supporting the conclusions of that research. It suddenly had become a matter in which people were eager to see the data books, as well as the paper. In 2002, at the suggestion of the Office of Management and Budget, we held three workshops which had the purpose of exploring with agency representatives, the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. meaning and implications of the Data Quality Guidelines. And today, we are taking another step in the same direction to consider the recently released OMB Peer Review Bulletin for Scientific and Technical Information. By the size of the list of people interested, and the welcome size of this audience, I think we can assume that there is a good deal of interest in this topic among the agencies and the broader academic and industry communities that depend on science. We believe these discussions draw on the strength of the Academies to convene diverse groups, to discuss, evaluate, and develop conclusions from those deliberations. Before I introduce the morning speakers, let me first thank Dick Meserve, president of the Carnegie Institution of Washington, who chaired the committee that is responsible for organizing today's workshop session. Dick was joined on that committee by Dick Merrill, Dave Korn, Artie Bienenstock, Jonathan Samet, Fred Anderson, Alan Morrison, and Sheila Jasanoff. You are going to hear from many of those named later during the day. First a little housekeeping. I remind you, as though it weren't already obvious, that this meeting is open to the public. There are members of the press present. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. When you come forward to ask a question after any one of the speeches or panels, we ask that you state your name and your affiliation. And we also ask that you please be economical in your expression. A transcript of the meeting is going to be posted on the Science, Technology, and Law Program's Web site early next month. Now, to our speakers. It's a great pleasure for me to introduce Dr. John Graham, administrator of the Office of Information and Regulatory Affairs, commonly known to us as OIRA. You have a short biography in your folder. Many of you will doubtless know that John had a distinguished academic career at Harvard before giving that up, at least temporarily, to serve the government. Please welcome John Graham. Agenda Item: Keynote Address - John D. Graham, Administrator, Office of Information & Regulatory Affairs, Office of Management and Budget DR. GRAHAM: Thank you very much, Don. It's delightful to see such a large turn out. I must confess, I had not predicted that the little topic of peer review would generate several hundred eager faces at nine o'clock in the morning, but I'm eager to have this dialogue. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. A special thanks goes to the National Academy of Sciences for agreeing on very short notice to put together this program. And for those of you who have criticized the NAS for being an organization -- as I have -- that tends to do things at two years of time and $2 million and nothing less, this exercise is a good illustration of a more nimble, and I think very responsive National Academy of Sciences. A particularly special thanks to a couple of people, to Don Kennedy, for his intellectual energies putting into this program. And certainly a person who has been both FDA commissioner, editor of Science magazine, and a long time academic at Stanford brings, I think, a lot of perspective to this topic. And a special thanks to Anne-Marie Mazza and her colleagues in the National Research Council for helping working with my staff, lots of e-mails and phone calls over the last couple of days, trying to make this all work. Moving to substance, the topic that we are going to be discussing today is one that is a major priority for this administration, and that is promoting the use of appropriate peer review mechanisms in the development of what I'll call regulatory science, and then maybe allow Prof. Jasanoff to flesh exactly what that phrase exactly NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. means. We are not discussing today, any particular form of legislation, or there is not any executive order on the table. What we are discussing is a draft OMB bulletin. For those of you who are not familiar with OMB as a bureaucracy, the one thing I would say is you might think at first blush that legislation and executive orders are more important than OMB bulletins. Believe it or not, there are a few people on my staff who think in the long run sometimes little documents like OMB bulletins have bigger and more durable impacts on the federal government than do legislation and executive orders. So, I don't think we should underestimate the potential significance of the activity we are engaged in today. Now, the lawyers in the White House always remind me when I am giving speeches to start by providing the legal authority under which we at OMB are engaging in this activity. So, I will cite the Information Quality Act, which was passed at the end of the previous administration. And it provides for the development of procedures in the government to improve the quality of information disseminated to the public. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. It calls for both development of pre-dissemination procedures to insure the quality of the information disseminated by agencies, and also the development of a correction mechanism, whereby affected parties can request that agencies correct any poor quality information that happens to get disseminated. And our effort at OMB through this draft bulletin, is to try to improve the predissemination quality control mechanisms. Now, I have mentioned the Information Quality Law, but quite frankly we believe at OMB that even in the absence of the Information Quality Law, the general authorities at OMB, statutory authority in the area of improving the management and functioning of government are adequate to support the development of the bulletin we have before us. So, you shouldn't think that this initiative is in any way contingent on the particular passage of the Information Quality Law. Let me just make a few observations about why peer review is, we think, useful and it's something that should be promoted in an appropriate way. One is we believe in a quite straightforward way that the examination of draft technical documents by qualified peers in the various fields of science, engineering, or economics improves ultimately NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the technical quality of those documents. Now, having said that, and I want to be very clear about this, this is in no way a reflection of anyone's views of the technical qualifications or expertise of the people who are involved in preparing a document in the first instance. And to give some sharp and more provocative character to that comment, I served on the faculty for 17 years, as Dr. Kennedy noted, at the Harvard School of Public Health before joining this administration. Certainly, there were a lot of qualified and talented people at the Harvard School of Public Health. But if someone were to say to me we would like to have their work disseminated to the public without peer review, I would think that would be a troubling suggestion. So, to argue for peer review does not in any way question the expertise or talent of the people who are developing these important information products. I think the second point I would like to make is that peer review also has an important legitimating function in government by improving the credibility of governmental information. There are obviously among mixed stakeholders, there will always be concerns about whether a particular technical product should be accepted as the source for NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. governmental action, and whether it has been developed through an appropriate process. And the mere existence of serious peer review mechanisms has the effect of enhancing the credibility of government information. I'm sure you will hear throughout today at least some concerns about whether peer review is really in some way some effort to slow down the development of regulations. And we in the administration don't see it that way. In fact, we believe in the long run that appropriate peer review mechanisms will actually cause the development of smarter regulations, and regulations that will be more durable, because they can survive the challenge of political and legal attack. So, we see peer review as an important legitimating function in the development of governmental information. Now, looking back at the history of current agency practices regarding peer review, I enter this territory with some trepidation, because there is in fact a substantial academic body of literature in this territory in what agencies are currently doing in the area of peer review. One of the authors of those books is with us this morning, Prof. Sheila Jasanoff of the Kennedy School, and I love the title of her book, The Fifth Branch, referring of course to NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the fact that the growing emergence of science advisors in the government is creating an influence for the scientific community that rivals those of the other branches of government. But having said that, I love the title of that book. I have to acknowledge that many of the lawyers in the White House haven't even acknowledged the fourth branch yet, so I'm not sure they are going to go for the fifth branch. Other contributions to this literature include a nice book by Resources for the Future entitled, Science at EPA, which focuses specifically on the peer review practices at the Environmental Protection Agency. And there are also several General Accounting Office reports over the last several years that look into this area. What we take from this available literature is that some agencies currently have no formal peer review policies. Other agencies have quite significant and well developed peer review policies. But even in those cases, if you look carefully at the actual practice of peer review, even agencies that have these guidelines or policies, they don't always actually use these in specific instances of the development of regulatory information products. And indeed, there is currently no oversight mechanism in the federal NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. government to make sure that agencies are practicing appropriate peer review. The draft bulletin, which is now out for public comment, technically applies to all agencies covered by the Paperwork Reduction Act. And for those of you who are students of the legalities of this matter, it turns out that the Paperwork Reduction Act also applies to the so-called independent agencies, as well as the cabinet agencies, and the Environmental Protection Agency. However, I don't have any obvious solution to the question of how OMB will, practically speaking, deal with the independent agencies in this area. As you all know, they are currently treated very differently than the cabinet agencies in their relationships with the executive office of the president. The coverage of the bulletin also does not apply to third party information. We are envisioning here peer review of information disseminated by the government, representing the official position of the government in these various settings. And hence, information for federally funded research activities, or for contractor activity that does not represent the official position of the federal government is not covered by the bulletin. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. A few comments on definitions. You might start by asking, for those of you who have read the definition, why do we need to get into all these various formulations of phrases -- regulatory information, significant regulatory information, especially significant regulatory information. And there is a very simple reason for that, at least we believe that at OMB, and that is that a one size fits all approach to peer review simply is not going to work. We are going to need to have mechanisms for deciding that some types of information is more important than other types of information. And that will cause more stringent peer review requirements for the more important information. So, we start with the phrase "regulatory information," meaning any scientific or technical study that is relevant to regulatory policy. Let me emphasize when we say relevant to regulatory policy, we do not mean simply relevant to a federal rulemaking. We have in mind information that can reasonably be anticipated to influence state and local regulatory activities, the activities of international regulatory bodies, and other examples which are provided in the bulletin itself. Adding the phrase significant to regulatory NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. information simply brings in the definition of influential, which is already adopted within OMB's Information Quality Guidelines, adopted several years ago. I think the important thing about the word "influential" is that it implies that one could reasonably determine that information, if it were released, will have or does have a clear and substantial impact on important public policies, or on important private sector decisions. Let me just elaborate briefly on that. We at OMB believe that release of governmental information that has important impacts on the private sector, is in itself in some ways, a form of regulation. And indeed, many economists would argue that we should more frequently use dissemination of information, for example about a product through a warning label or an educational activity, rather than a traditional command and control regulation. When we engage in information dissemination officially by the federal government, we are in a sense engaging in a form of regulatory or quasi-regulatory activity. Now, in the area of especially significant information, and this is the category of information that the draft bulletin applies the most stringent standard of NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. peer review to, we are envisioning something that will either: (1) satisfy the economic significance test in Presidential Executive Order 12866, which is roughly the $100 million impact test; or (2) that it's not a regulatory action as defined narrowly, but it is one of those actions that is defined in the influential test to also have a possible impact of $100 million; or (3) it's an issue of significant interagency interest, or one that the administration deems to be a priority. Now, the draft bulletin, and I'll focus here on the especially significant regulatory information, and keep in mind that we are envisioning that this might cover on the order of several hundred information products per year throughout the federal government. So, we are not talking about covering in this particular area, necessarily thousands of information products, but certainly more than several dozen if you look at all the agencies who are affected. We are looking for external peer reviewers who have the necessary experience and independence. And the word "external" refers to external to the agency that is sponsoring and developing the information product. Agencies must provide the peer reviewers with NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. sufficient information, and an appropriately broad charge. And the agencies must publicly respond to the peer reviewers' written reports, and make other appropriate disclosures, like the names and qualifications of these reviewers. I want you to keep in mind that requirement with regard to public disclosure, because it's something that I think deserves considerable discussion and attention. Types of information to be covered. We have in mind very broadly, all scientific or technical reports, studies, findings, determinations, not simply limited to the physical sciences, but including the social sciences as well; reviews of the literature with a strong regulation implication to them; risk assessment documents; regulatory analysis documents, including the science, engineering, and economic inputs that go into these documents. Types of information not covered: grant applications; material already peer reviewed through an appropriate mechanism; internal documents that are not disseminated outside the federal government; information involved in individual adjudications and permit applications; national security documents; routine financial and statistical information; and waiver for emergencies involving for example imminent hazards. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. I think one of the most important features of the draft bulletin, although it takes only a few paragraphs of discussion in the bulletin, is the concept of advanced notice of agencies to OMB and OSTP and the public that they are developing an information product that might meet some of these standards, and what they plan to do in terms of peer review. The proposed bulletin asks agencies to develop a notification mechanism that they would periodically publish forthcoming reports or findings, and what their peer reviewed plans for those documents are. This type of mechanism would allow OIRA and the Office of Science and Technology Policy to get an advanced notice of these types of reports, to look into whether or not the peer review plans seem to make sense, and to allow the public to ask the same kinds of questions. Topics to be addressed in the agency guidelines. As a practical matter, the single most important thing the bulletin does, even though it's of a very general sort, is it requires agencies to develop guidelines in the area of peer review. And some of the difficult questions that will be addressed are how to handle real and perceived conflicts of interest. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. For example, potential ties of a reviewer to a business or another commercial interest in the activity that is involving the peer review; entanglements with the sponsoring agency, for example, an academic who gets a substantial amount of research funding from a particular agency, and for example, may also have consulting relationships with that particular agency. Issues around bias -- these are issues that for example, selection activities here at the National Academy of Sciences have struggled with for years, and there are evolving standards on how to address these issues. While all of that is important, we at OMB think that the single most important thing we do is make sure that we identify and select reviewers who are knowledgeable, who have expertise in the relevant subject areas, and in many cases those people are going to have some relationships to the sponsoring agency, and to the affected regulating entities. In order to give some practical feel for what the impact of this proposed bulletin might do, I have been working with my staff in preparing these remarks on some case examples of recent reports issued in this administration that presumably would have been covered by this bulletin if it had been in effect at the time they were NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. released. And those are: a National Highway Traffic Safety Administration study of the safety of sport utility vehicles, light trucks, and mini-vans; an OMB report, Circular A4 on guidelines for regulatory impact analysis -that is our guidance document on how to do cost-benefit type of work in the context of major rules; the Food and Drug Administration's recent final rule on consumer labeling for the trans-fat content of foods; and then two EPA reports, one on the cancer inhalation risk value for benzine, and the other on the carcinogenicity and non-cancer health effects of diesel engine emissions. I want to make a few remarks about each of these, just as a way for context-setting, and to provide an ability to discuss concrete examples throughout the day. We have chosen them from multiple agencies that have different types of peer review, and that I think give a sense of the stateof-the-art of peer review practices within the agencies, though these five examples certainly are not a random sample of current agency activity. Indeed, my staff would argue they probably represent more the cutting edge of more intensive peer review than exists compared to a lot of documents in the federal government at this time. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. The first report was on sport utility vehicles by NHTSA. A couple of comments on it. Its regulatory significant is important in several ways. It can influence the development of rules for the compatibility of light trucks and passenger cars. And it could influence the future evolution of fuel economy standards, the Corporate Average Fuel Economy Program. What is interesting about this particular report that I want to note is that there were three peer reviewers selected by the agency, and they gave their comments as individual reviewers, rather than as a group or as a panel. And those of you who have read the draft bulletin carefully will note that that activity is permissible, even for the most significant information covered by the bulletin. In some cases, an agency may choose individual peer reviewers, rather than a group or a panel peer review. Another point I would like to make is that in this particular report, the agency did not provide a written response to the peer review document, though they indicated they made some changes to the report in light of these peer review comments. If you try to get a copy of the peer reviewers comments, my understanding is as of today you might not be able to do so, but we understand that the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. intention is that they will be in the public docket at NHTSA. What that implies is that the peer reviewer comments are not as publicly available as the report itself, which would not be consistent with the standards in the draft proposed bulletin. By the way, I happen to have read the three peer review comments, which will be publicly available, and they are a pretty solid piece of work if you read those peer review comments. The second example is the peer review that we initiated at OMB on our own guidance to agencies on the development of regulatory impact analysis and cost benefit analysis. Just a couple of comments about this document. We did not supply a written charge to our peer reviewers. We simply gave them the draft regulatory analysis document, and asked them to peer review it. And that would not comply with the terms of the draft bulletin, which requires a written charge for especially significant information. We did not seek public comments in advance of the peer review. We held a public comment process concurrently with the peer review, and in at least some readings of the draft bulletin, a public comment process must precede, and be provided information to the reviewers, and in this sense NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. this OMB activity would not have complied with the draft position in the bulletin. An interesting feature of the OMB exercise is that it does have a formal response to comments document that was published and is available in the same way as the guidance itself. And you can access that information on OMB's Web site. The third example is the regulatory impact analysis for the Food and Drug Administration's final rule on consumer labeling of foods for trans-fat content. I think this is an interesting example of peer review activity, because in this case the peer reviewers, while external to the Food and Drug Administration, were predominantly inside the federal government at other federal agencies. And this is a practice that would be consistent with the terms of the peer review bulletin. So, here you have in this particular case, some agency economists at one agency peer reviewing the work of agency economists at another federal agency. I think it also raises an interesting point about how peer review, if occurring inside the federal government, will ultimately have more deliberative protections than activities that occur outside NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the federal government. The fourth report is the EPA's benzine cancer inhalation report. I want to make just a couple of comments on this particular example, and that is I have been providing estimates of the budgetary costs of peer review in each of the examples, and you notices a couple of zeros on the previous ones. That's the budgetary cost to the federal government, no incremental cost in those cases, which means the peer reviewers were basically persuaded to do this on a volunteer basis. This is an example of EPA actually commissioning an outside contractor, a private firm, to organize a workshop for peer review where some compensation, travel and per diem was provided to the peer reviewers. And the selection, as I understand it of the peer reviewers, was performed by the contractor, rather than by the agency itself. And you can get a sense that the cost and time frame for this type of activity is somewhat longer than the individual peer review model that I described to you earlier. The last example I want to discuss is EPA's diesel engine health assessment report. Let me start by saying while this slide may not convey what to me is sort of the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. obvious implication. This is something to be worried about. This is a process that was as a part of EPA's Clean Air Science Advisory Committee, and as we computed -- my staff - it's a 10 year process of peer review; 5 panel reviews, each of them with succeeding drafts proceeded by the agency; 5 public comment periods; hundreds of thousands of dollars and a very long time to get a work product out. There may be people in the audience who know this particular example better than I do, but I think it does raise significant questions of do we need to have some oversight mechanisms for peer review activities that seem to be onward and onward? Lessons from the report that I have examined are first of all, there is much peer review already occurring at various federal agencies, and to some extent, it would comply with the requirements in the peer review bulletin. But in other areas, there are deviations from practice that are specified in the draft bulletin. These peer review practices are variable. And one of the most interesting features I think is the prevalence of individual peer review, as opposed to group or panel review. And we'll be interested in your comments on whether you think both of those models should be permitted in the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. context of the OMB policy. It also illustrates that peer review does not necessarily have to be a highly expensive or time consuming activity. And I can assure you that my budget colleagues at OMB are very pleased to hear that particular result. Looking forward for the process that we are engaged in now, we have a public comment period on the draft bulletin which will run through December 15, 2003. We have extended that comment period an extra 30 days to allow additional time for comment, and for this workshop and the ideas that it will stimulate. We have also extended the comment period for agencies inside the federal government to January 16, so they will have the benefit of the public comment process and this workshop as they develop agency comments to OMB. Let me conclude by thanking you all for coming, and i look forward to comments and discussion. I intend to stay for most of the morning and listen to some of the dialogue. And I also have my staff here, who will be briefing me at the end of the day on the full day's activities. We care very deeply about this issue in the administration, and we are looking forward to your comments on all aspects of the peer review initiative. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Thank you very much. Agenda Item: Questions DR. KENNEDY: We have time for questions, so please proceed to the microphones. MR. CONRAD: Jamie Conrad with the American Chemistry Council. I know that you described this peer review process as part of the pre-dissemination review under the Information Quality Act, which would suggest by implication that documents being peer reviewed wouldn't be subject to the correction mechanism, because that comes after dissemination. And while on its face I think that makes a certain amount of sense, the history of the diesel peer review going on for 10 years suggests a situation in which documents are actually out in circulation for years and years, and yet would never be disseminated under that process. So, I wondered if you could offer some thoughts on when something would be disseminated? And if there would ever be a case where even though it's still undergoing peer review, it's been out long enough that it ought to be subject to correct? DR. GRAHAM: I think that's an excellent question. The only comment I would make is I think that in the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Information Quality Guidelines that we have developed, we have tried to develop the principle that a comment process that is developed on a document is the appropriate forum for those comments. And as a result, a correction request would be not really appropriate if an agency is already in a comment process on that document itself. Now, if an agency for some reason, were to develop a draft comment and take comments, and then just let this document hang out there indefinitely, then I think it's a much more difficult situation as to whether it constitutes a dissemination or not. MR. BELZER: Richard Belzer, Regulatory Checkbook. I think it's a follow-up on the previous question, but the purpose of the peer review policy of course, as I understand now is to give force to pre-dissemination procedure. But the act of securing peer review is an act of dissemination. And I have trouble squaring that circle. Can you explain more how we would distribute documents for peer review without actually disseminating them? DR. GRAHAM: Yes. I think that as I tried to illustrate in the comments that I made, a lot of agency practices in peer review right now do not involve public dissemination of documents. They do involve sharing a NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. document with individual technical experts, asking them to perform letter review, and then providing those, or in some cases, interagency review of documents. However, the key question is, if we are going to have a public comment process prior to peer review, that is going to engage and force this dissemination of draft material, and it is going to raise I think, some of the issues that you just described. So, we are open to comment in the context of the draft bulletin about how OMB should weigh on the one hand, the need to provide an opportunity for public participation in information to peer reviewers as they discharge their work. But on the other hand, not having a lot of draft documents released publicly that aren't really of sufficient stature or haven't been well enough developed to justify that type of public dissemination. That's a difficult balance. MS. SHERWIN: Joanne Sherwin from the IHO(?) National Engineering and Environmental Lab. I have a question about your cost data. You have other categories on your slides, were yes or no comments provided to the reviewers, yes or no. On your cost data where you have zero cost, it seems to me there are always NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. administrative costs at least. And so, in no way can you come up with a zero cost for peer review. And it is not clear to me what types of things you have included in those costs. DR. GRAHAM: First of all, just to clarify, there were zero budgetary costs. As someone with a little economics background, there are a lot more overall costs than that. There is the value of the time of the reviewers themselves. There is activity by, for example in the case of the OMB report, our own staff in developing the documents and responding to the reviewers' comments. But there was no incremental appropriation or budgetary allocation provided for those activities. So, they were basically within the context of current budgetary allocations. DR. KENNEDY: Bootlegging is not an inappropriate term. DR. GRAHAM: That's right. MR. ELKINS: Chuck Elkins. I'm an environmental consultant. You mentioned that you thought there might be maybe 200 or 300 especially significant information documents which would be peer reviewed across the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. government, and that you have a number of fairly interesting and strict requirements for those, and I'll sure you will get comments on how they can be made more strict or less strict. But for all the other technical documents you have essentially, if I read this correctly, two paragraphs that deal with them. But yet those documents include all the influential documents under the Information Quality Guidelines. And so, am I correct in reading this that for those influential documents which don't rise to the $100 million category for especially significant, that all the bulletin does is basically say that they should have peer review? DR. GRAHAM: I think that's pretty much accurate. And it requires that agencies development peer review policies for those documents and other documents. So, in a sense at OMB we have done what is frankly pretty classic at OMB. We have tried to address what we think are maybe some of the most important documents with some specificity. But then we have deferred to the agencies to develop guidelines that would apply to the whole universe of documents. MR. ASHBY: Bob Ashby, DOT. A comment on the procedure to be followed in NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. finalizing the bulletin. This bulletin appeared, the agencies had notice of it when the public had notice of it. I think that that may not be appropriate, particularly as we work toward a final document. I think that this process should be a collaborative, collegial process in which OMB and agencies working together create a final product that will be an improvement on the proposal. It should not be simply a matter of the agency commenting just like one more special interest group. And I fear that at times the document treats agencies just like one more special interest group. When we come to January, I would recommend for example, a group or panel of agency peers of OMB if you like, who would work closely with OMB to develop the final document, rather than simply putting the agencies in the position of receiving an edict from the rarified heights of OIRA sometime in the winter. DR. GRAHAM: We do intend a formal interagency review process. And we also are engaged, as you know, in some informal outreach as well. MR. WARNER: My name is Leo Warner. My interested group is old citizens. If the devil is in the details, the danger is in NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the exceptions. And I noticed that you excepted national security considerations -- we don't need to discuss the awful overuse of that umbrella -- but also emergencies. Now, I'm mostly concerned, because when people think of an emergency, that's when they are the least practical and in most need of perhaps a quick peer review. In my experience, in 1952, the secretary of agriculture came on television and said something about cranberries which nearly destroyed the New Jersey cranberry industry, and it was wrong. In 1970, there was a wonderful sugar substitute, and that was cursed. The soft drink industry suffered a terrible shot, and that was wrong. DR. GRAHAM: We don't want to go there, do we? Dr. Kennedy was in the government at that time. MR. WARNER: What I'm worried about is when an emergency occurs, that's just when somebody loses their head. And perhaps you ought to have a little section that doesn't make that in an emergency, but requires at least a two day peer review before some big government person comes and says watch out for your beef or whatever. I'm done, but I'm just worried about that exception. DR. GRAHAM: I accept that as a comment which we NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. will definitely look at. It's very thoughtful. MS. DE PALMA: Winifred DePalma with Public Citizen. Is it a correct reading that the question of whether to disclose personal or institutional funding of a potential reviewer is left to the discretion of the agency? And if so, what's the rationale behind that? DR. GRAHAM: I think the OMB bulletin is expecting that agencies will develop guidance on sources of financial support, whether it be private sector support, whether it be public sector support. The OMB draft doesn't go into details on how that is done, but we are expecting that all agency guidance would have that, frankly as many agencies already have. MS. DE PALMA: Can I ask just a follow-up? Would an agency be in contravention of these guidelines if it didn't require public disclosure of potential reviewers? DR. GRAHAM: Oh, public disclosure. I'm sorry, I misunderstood the question. I think we are envisioning certainly disclosure to the agency. But the question of how much public disclosure of that information would occur, that is not specified in the bulletin itself. And as you know, I'm sure, there are issues there, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. a trade-off between transparency so people can make their own judgments about say conflict of interest, but also privacy interests of individual scientists in terms of their own sources of financial support. So, that needs to be worked out in the individual agency guidance. MR. GILAMAN(?): John Gilaman. Two details. One is the justification for continuing to use $100 million, in fact possible $100 million as the criterion which is a relic from the 1970s. And the other is what are the consequences of noncompliance? Is there a judicial review? Is there a whole new genre of litigation? These are questions that came up in the nineties when judicial review and peer review in some eyes became linked as a new target for litigation. That is, when the peer review did not satisfy the agency's guidelines, or in this case the government's OMB guidelines. DR. GRAHAM: On the latter question, the OMB bulletin, its sort of status in the government as an internal management device whereby we try to encourage agencies to do peer review. I would be the wrong person to ask the legal question about what its ramifications might potentially be. And I suspect there are experts here today who are going to do a better job than I will to try to NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. answer the consequences in that area. The $100 million test, I think you raise a good point. Gradually over time, using that same test captures a broader and broader net of rules and documents. And there will be, I think, at some point, that it needs to be adjusted. We haven't taken that step yet, but we are certainly open to comment if people have constructive suggestions on that. DR. KENNEDY: This is going to be the last question for this round, but after Kathie Olsen talks, she and John will also be able to take questions together. MR. SCHECKER: Thank you. I'm Larry Schecker. I'm with the FCC. The draft bulletin exempts the studies within the context of adjudicatory proceedings, but does not do so for rulemaking proceedings. Rulemaking proceedings are designed to test studies such as you are talking about. What is the rationale for not exempting these studies in the rulemaking context? DR. GRAHAM: I think the answer to that is fairly straightforward. We think public comment processes, which are the traditional device for input to an agency in a rulemaking, is not an appropriate substitute for a peer NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. review, but of technical issues by a qualified panel of scientists, engineers, and so forth. So, we don't regard a public comment period as a substitute for peer review. DR. KENNEDY: Please join me in thanking Dr. Graham for getting us so well started. [Applause.] It's a great pleasure to introduce Kathie Olsen, associate director of the Office of Science and Technology Policy in the White House. Kathie had a distinguished career -- still has one -- as a neuroscientist, although she is turning her talents in a different direction. She had a wonderfully productive time at the National Science Foundation, and has now moved as it were, downtown. Kathie. Agenda Item: OSTP's Role - Kathie L. Olsen, Associate Director for Science, White House Office of Science and Technology Policy DR. OLSEN: First of all, I want to thank Dr. Kennedy and the National Academy of Sciences for organizing this symposium. John Graham I think actually gave an excellent overview of the proposed standards for regulatory science. What I would like to do in my time is to follow-up with some NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. discussion of why I think that a transparent peer review process for regulatory science is good and necessary, and what I see as OSTP's role in this process. At OSTP, we believe that a strong federal peer review process for regulatory science will enhance the regulatory process. I think that we have all seen enough of the debate to know that when regulators affect a lot of people and a lot of investments, nothing goes unchallenged. These draft peer review standards for regulatory science are an attempt to bring the tough questions into the regulatory process at an earlier stage to make sure that sound, objective science not only is the basis of the federal agency regulations, but that we can demonstrate it to the satisfaction of the external community, and to head off time consuming and costly after-the-fact challenges to regulations. The philosophy behind these draft peer review standards for regulatory science, as well as the federal standards for data quality is that a little bit more work and attention to best practices up front, will result in a much better ultimate product, and fewer challenges and revisions down the road. I also want to emphasize the limits of these new NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. draft peer review standards for regulatory science. They will not affect the typical NIH or NSF grant, or other basic research programs. The new standards apply only to, and I quote, "the most important science disseminated by the federal government regarding regulatory topics." Since this is the science that affects literally every person in the country, it is definitely worth it to make sure that this critical science does conform to the very best objective standards of merit. However, I also strongly believe that we should take advantage of the knowledge gained through years of experience in the peer review process that is provided by our premiere research agencies, the National Institutes of Health, the National Science Foundation, and incorporate much of their philosophy and standards to insure the same quality for the science underlying regulatory action. Indeed, peer review is the landmark of America's research enterprise. It is a critical component of our competitive merit-based system for insuring the best use of limited resources. OSTP is charged with being a source of analysis and judgment for the president with respect to major policies, plans, and programs of the federal science and NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. technology enterprise. We also have the role of leading interagency efforts to develop and implement sound policies across the government. As such, we are already actively involved with the agencies in implementing best practices for research evaluation and funding decisions. The federal science agencies all perform peer review of research, and all have high standards for their research programs, even though the individual applications of peer review are widely variable. Yesterday, as I was preparing for the task, I informally checked four agencies - NIH, NSF, NASA, and EPA -- for definitions of their peer review standards and research opportunity announcements. Each agency documents their peer review standards, uses internal and external expert review, and evaluates a proposal by tough standards of scientific and technical merit. They clearly are all concerned about research quality, insuring objectivity, and avoiding conflicts of interest or the appearance of conflict of interest. But I also noticed that the agencies apply their review criteria differently, and they do so because they have different missions and different relationships with their research community, and they have evolved science management practices that work in these different contexts. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. The next draft standards of peer review for regulatory science preserve most of the flexibility of the individual agency to implement peer review for studies that may affect the development of regulations in ways that make sense for that agency. At the same time, the draft peer review standards provide uniform federal criteria for involvement of external researchers, insuring objectivity and insuring against conflicts of interest, and as I like to believe, the perceived conflicts of interest. The reviews of science that will support the federal regulatory process must be reliable, independent, and transparent. Independence and transparency are of increasing importance as the impact of the research increases. One of the questions concerning the draft peer review standards we are discussing today has to do with whether the names of reviewers and their details reviews should be made public. We don't have an answer to that, but I will be very interested in hearing your comments on this. As I was thinking about this subject, I thought that while peer reviewers are accustomed to some level of anonymity, anonymity has never been absolute. It never has. The names, backgrounds, and extent of reviews have always been carefully scrutinized by someone. And when the stakes NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. are the highest, more people need to be assured of the quality of the review process, and the level of anonymity generally decreases. As many of you know, I have had much experience in the peer review arena, both having my articles and grant proposals reviewed, and also serving as a reviewer or panelist for NIH, DoD, and NSF grants, as well as a reviewer for many scientific journals, and serving on a journals editorial board. At NSF I was responsible for peer review of proposals, selecting specific panels of experts to insure the appropriate diversity, to identifying ad hoc reviewers that possessed the required expertise specific to the questions being addressing, and to chairing the advisory panel, and ultimately working with the grantees or the declinees based on the action from the advisory panel. The type of review actually differs dramatically, depending upon the impact and the investment in the activities. For example, we have had proposals coming in requesting to provide funds for beginning scientists for opportunities to attend meetings. And this may only require internal agency reviews. Whereas, the National Science Foundation's science and technology centers, which is the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. centerpiece of NSF support, $40 million grants over 10 years, have an extremely detailed review process with specific criteria for evaluation that was clearly articulated prior to even the submission of the proposal. The review was multi-dimensional. First, we have a panel. And then we sort of narrow it down. Then we have a solicitation from experts in the field, followed by another panel review. Then we have site visits, a blue ribbon panel, and finally it's presented to the National Science Board. So, we have all of these steps to insure the scientific integrity and quality. It is clear that in much of this process the expert reviewers doing this evaluation are known. If any of you have been on a site visit, you know you are sitting right across from the people that you are reviewing, and they know who you are. But the site visits, blue ribbon panel, the National Science Board are indeed clearly identified. Also, individuals that review for the federal government are usually identified through available lists of names, some by request. Others are sent, for example, at the NIH, along with the review, the members of the panel. In these cases, however, you have a very large list. And I NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. guarantee you, the proposals can only think that he or she knows who did the review, and they do like to guess. The bottom line is that the type of peer review carried out is appropriate for the ultimate goal of the program or the activity. In all cases, the goal is to insure that reviewers are unbiased, unconflicted, including as I keep saying, the appearance of a conflict, and have the expertise to provide a thorough critique that is responsible to the specific criteria set up for evaluation. Actually, I see one of the lawyers at the National Science Foundation. And I want you to know before I chaired a panel, it was ingrained in terms of stating the conflict of interest rules. You are a conflict of interest if you have published or collaborated with the individuals in the last five years, even if it hasn't been published, if you have collaborated. If you are from the same institution, if you have gone to that institution and received an honorarium, regardless of if you know or do not know this individual or visited his department. You are a conflict of interest if you or either the PhD advisor, or he is or she is your PhD advisor. Interestingly, a post-doctoral advisor is not a formal conflict of interest. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. But more importantly in this discussion with each one of the reviewers is the appearance of a conflict of interest like for example, a post-doctoral advisor. Or the fact that little things, like for example you served as the best man or the maid of honor in the person's wedding. These things need to be brought to the attention, so a decision can be made in terms of the quality of the review. At NSF, each conflict or perceived conflict of interest is noted in the record of the review. These are the high standards of merit-based peer review that serves as the basis as a scientific enterprise. It works. It works, because it deals head on with balance, objectivity, diversity, and conflict of interest. So, I hope you will help us determine what makes sense in terms of public openness in this important process for peer review of regulatory science. As I said before, it is important to maintain flexibility in how peer review is performed. But it is critical for the regulatory process that all of the agencies can clearly demonstrate the same high standards for quality, objectivity, and avoiding conflicts of interest. I'm really happy to see so many people here, and I know that I'm preaching to the choir on the importance of NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. well conducted peer review for the government regulatory process. But you can see that this area is indeed very close to my heart. And OSTP has the role to consult with the agencies concerning the sufficiency of the planned policies and the implementation. In particular, in the current wording of the Federal Register notice, the associate director for science has a major role in an interagency work group on peer review policies. This group will make recommendations regarding the best peer review practices for regulatory science, and recommend steps to expedite and improve agency processes in this area. So, that kind of takes note to one of the questions that was asked about the role of the federal agencies as well. But referring back to our stellar research agencies, NIH and NSF constantly re-evaluate their peer review processes. And indeed, NIH recently went through major reorganization of their panels in scientific review branch. We need to learn from the best. We need to make sure that we implement the new standards for regulatory science as an enhancement to current processes, and to save ourselves work down the road. And we need to make sure that NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. they are not just an extra administrative burden. I also want to understand the agencies' concerns, and the concerns of the scientific community, and to make sure that we have the benefit of all of our thinking on how to make this draft federal standards for regulatory science work for us. I'm going to be here as much as my schedule permits, but I want to acknowledge Dr. Ann Carlson from my office, who will be here all day, again, hopefully interacting with you. But also, similar to John's staff, taking back to us a thorough report on the issues, the concerns, and the recommendations that were discussed. As John said, we care very deeply, and this process can be improved with your comments. So, again, thank you very, very much. Agenda Item: Questions DR. KENNEDY: As I said, what we would like to do is have you direct questions either to Kathie or to John for the next few minutes. So, please, line up and get ready. MR. HALPERN: I'm Harold Halpern. I'm with the Department of Energy. I speak for myself, and not the department of course. A question for Mr. Graham. Should the costs of NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the peer review be much higher than anticipated, will there be a relief mechanism? DR. GRAHAM: Good question. Yes, one of the questions that we get frequently on the peer review bulletin is whether agencies will be supplied with sufficient resources to engage in the contemplated peer review practices. One of the complexities that many people in the room may not be aware of is that OMB does not control all of the things that agencies ask for in terms of federal funds. Agencies may be surprised to learn that OMB has a more sympathetic attitude toward financing peer review activities than some of the other activities that agencies would prefer to have funds for. So, it's a complex question, how we get the appropriate degree of resources to this activity. MR. ANDERSON: Fred Anderson with the Panel on Science, Technology, and Law. It's a question for both of you. In light of Kathie's comments, and looking back at the matrix or the charts you put, John, should there be a line or two about the vetting or the qualifying of the reviewers in those five case studies as to whether or not they met bulletin requirements for disclosure and proper qualification to NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. serve? DR. GRAHAM: On their expertise, their qualifications in that regard? MR. ANDERSON: Independence, lack of bias, conflict, and qualification. DR. GRAHAM: I think it would be some additional work that we at OMB could get into, but we haven't done it as of yet. We didn't evaluate, for example, whether the selection procedures for the reviewers in each of those cases was reasonable and complied with the bulletin. We haven't done that. DR. OLSEN: But I was going to say in terms of the federal science agencies, they actually look at sort of the known conflicts of interest up front, and they are immediately disqualified from receiving the proposals or the applications. MR. MOULTON: Sean Moulton, OMB Watch. This is a question for either John or Kathie. You had said that one of the reasons that you wanted to have the public review or the public comment process inserted into the peer review was to get concerns raised earlier in the process. But my understanding about peer review is that it is to get a technical and an expert opinion on a technical, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. scientific document, and we are looking for objective science apparently. It seems to me that allowing a public comment process, and believe me, we are all for public comment processes, but putting it in on the peer review seems to actually increase, or insert policy concerns and special interests into what is supposed to be an objective process. The scientists are supposed to be reviewing the material on its merit alone. And adding another layer of influence to a process where the public isn't going to probably participate in every level, the people who are going to be able to participate in each level are going to be the ones with the most resources, which are going to be industry mostly. I was wondering if you could respond. DR. GRAHAM: Sure. I think the word "public" and the way you use it is a very diverse construct, and there are a wide range of interests. You mentioned business interests, and even within the business community there may be very diverse interests. And we recognize that they may have expertise. They may have data that they could submit that would be relevant to a peer review process. The same could occur for a public interest group, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. an environmental group. They may have expertise and interests. And while they may be appropriate in certain cases, for individuals from those organizations to serve on the peer review panel itself, in many cases that won't be appropriate, because of the selection criteria that are described. And as a consequence, there is merit in thinking of value in the public comment process prior to a peer review, and allowing the peer review members to see the issue laid out, even though it is a technical issue, with the expertise from those various groups. So, that I think, would be the argument in favor of preceding the peer review with a public comment process. But I think your question also illustrates some of the complexity that it introduces. MR. MOULTON: And as a brief follow-up, I'm curious that part of the process that you are outlining here, you said it is to avoid data quality challenges and other challenges further down the road. But it seems to me by again allowing kind of a public comment process during, and allowing them to insert again possibly biased data, and then having a report produced where the peer reviewers are required to outline all of the problems that were voiced or NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. brought up and what they would do to correct them, it seems to me that you are providing an easy entre to challenges. You are basically establishing all of the basis that someone would need to then challenge the data based on peer review conflicts that were literally unresolvable, because there were two issues going head to head. And we could wind up actually encouraging challenges down the road, including legal challenges. DR. GRAHAM: Is there a question in there? MR. MOULTON: I'm wondering how you expect to avoid establishing the legal basis for a challenge, and instead go with what you are saying, which is reduce challenges. DR. GRAHAM: Just briefly, I think the answer would be that a federal judge who sees that the agency has laid out its technical premises for review by an objective third party panel for example, and has revised its technical determinations as appropriate in light of those, might be granted more deference by a federal judge than an agency that had engaged in no peer review whatsoever. MS. CARROLL: Beth Carroll, and I'm with the regulated industry. I'm curious about the exemption, so to speak, of NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. NIH, NIHS, NSF from the bulletin guidelines. It seems to me that many of those studies are going to be used in a regulatory context, and all that may occur with them is publication in a journal. And from the industry's point of view, that does not pass the muster from the standpoint of the studies that we must turn in, in order to have our products regulated appropriately. At best, a peer reviewed journal article is generally done by an unpaid individual who may or may not have access to a good bit of the data, in most cases not, and can basically only make a decision on whether that paper and what is brought up in that paper is plausible. Yet, those papers that are published are being used in the regulatory decision-making process. I just wonder why they are being exempted from the bulletin requirements. DR. GRAHAM: Let me make just a brief response to that, and let Kathie add as she feels appropriate, the process whereby extramural scientists funded by a federal agency, or even intramural scientists funded by an agency, the process by which they get their scientific work published, and they have it evaluated, et cetera. That is a very important set of issues, but the draft bulletin only addresses those disseminations by an agency where it is NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. clear that the information represents the official position of the agency. There is a lot of intramural scientific work for example, by scientists at NIH and other agencies that would not represent the official position of NIH. And the bulletin is not designed or intended to cover that. MS. CARROLL: Will there be any kind of review of those that might be, or might appear to be representative of NIHS or NIH? DR. GRAHAM: Yes. In the premise to your question, I think you may be under the impression that the document simply just exempts all NSF and NIH disseminations. And technically speaking, I don't think that's what the bulletin does. There are examples within NIH, for example, of disseminations that represent the official position of the agency. And those would in fact be covered by the bulletin. And we are looking carefully at the magnitude of those, and are beginning dialogue with NIH. We haven't found an example of that sort in NSF yet, but it may exist. We are simply not aware of it. MS. CARROLL: So then NIH and NIHS will have to have guidance developed for those types of articles? DR. GRAHAM: Right. The last time I checked, they NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. are part of HHS, and HHS at a minimum, will have to have guidance. PROF. SHAPIRO: Good morning, Sid Shapiro, the University of Kansas. The draft bulletin, in discussing potential conflicts of interest advises agencies that they may wish to look at whether or not a person has received funding from an agency to do research, or may in the future receive funding to do research. And the bulletin explains that there may be some cause for concern, because that person may alter his or her advice to the agency in the hopes presumably of currying favor with the agency. At least at that point in the bulletin OMB does not refer to any similar suspicion about scientists who do research for industry, and whether or not a similar skepticism would be appropriate. Earlier in the bulletin, and in today's talk, Dr. Graham refers to a general idea of ties to business as maybe raising concerns. So, my first question is whether or not that's just a confusion, and both sets of scientists are intended to be treated equally? And my second question is whether or not that's current practice in the government, that that is automatically regarded as a conflict of interest if someone NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. is funded by the government, or a particular agency to which that person might give advice when the advice doesn't deal with the particular issue in front of the agency? DR. GRAHAM: I think you ask actually a series of pretty complicated questions, because the treatment of those issues currently in different federal agencies may in fact be subtlety different in ways that I don't fully understand. I know for example, that FDA and EPA for example, both currently have policies that address the area of your concern. I do think in the first part of your question, that there is a confusion frankly in your question, which is that we do envision that sources of private sector support, as well as public sector support should be disclosed to the agency or the entity that is involved in selecting the peer reviewers, so they are aware of that, and can weigh that in deciding whether there is a conflict of interest, whether there is a bias. So, that's the intention in the bulletin. MS. PHIBBS: Hi, Pat Phibbs. I'm the reporter with BNA. I would like both of your thoughts on how to handle particularly controversial peer reviews that go on and on and on. Some states, out of frustration, are NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. beginning to use interim values. Should agencies be required to describe draft documents as their official interim position, like EPA did with its cancer risk assessment guidelines? How should these be handled? DR. GRAHAM: I think that's an excellent question. The construct of an interim position, if it hasn't been fully developed in the context of the peer review process I think is troubling in the context of the bulletin. I don't want to give a lot of encouragement to that basic construct, but I think that there is a legitimate concern about peer review processes that take a substantial amount of time. One of the things that we have envisioned in the draft OMB bulletin is the idea that agencies would specify in the future, what documents they intend to disseminate, what peer review plans they have for those documents. And that planning process may include some suggestion of what they think the time frames are that they need to have in order to do an effective peer review. That would allow OMB and OSTP to examine for example, whether agencies are meeting their own deadlines with regard to peer review activities, whether peer review activities are taking longer in certain areas or at certain agencies. But until we get some kind of management system NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. in place to look at this issue, it's going to be a very sporadic exercise. DR. KENNEDY: You have the last question. MS. CARNEY: Joanne Carney with AAAS. My question is for Mr. Graham. The Information Quality Guidelines does not recognize peer review as meeting a high enough standard for influential information. Once the federal agencies go through this peer review guideline process, would you envision making any changes to the Information Quality Guidelines for recognizing peer reviewed information has gone through this new process? DR. GRAHAM: To be honest with you, I didn't understand the premise, the very first part of your question where you referred to something about the Information Quality Guidelines. If you could just explain that for me some more. MS. CARNEY: There is a two tier process where peer review is recognized for a certain level of information, but -DR. GRAHAM: You mean journal peer review? MS. CARNEY: Well, it doesn't specify journals specifically, just peer review in general. DR. GRAHAM: Are you referring to the presumption NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. of objectivity on behalf of information that has already been published in a peer reviewed journal? MS. CARNEY: Correct. DR. GRAHAM: And that presumption in the Information Quality Guidelines is rebuttable based upon a persuasive showing to the contrary. Now, that's in the OMB Information Quality Guidelines. And in a sense, the peer review bulletin adopts a similar standard in the way it crafts the degree of deference appropriate for information that has already been published in a journal. We understand that's a source of some contention, but we hope the workshop will dig into your question. I think there is a lot in your question that I can't really get to. I hope you will develop that further as the day goes on. DR. KENNEDY: Thanks, Joanne. Now, If you will assist Kathie Olsen and John Graham down into the audience with a well deserved round of applause, you can stretch a minute, and we'll have the next panel. [Applause.] Agenda Item: What is Peer Review? A Comparison of the Bulletin's Model of Peer Review with Other Types of NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Peer Review - Moderator: Richard A. Meserve, President, Carnegie Institution of Washington DR. MESERVE: Good morning, I'm Dick Meserve, and I have the pleasure of moderating the panel that is dealing with the question, what is peer review? Before we get started, I want to alert you all to a minor change in our schedule in order to accommodate a speaker who has to catch an airplane. We are going to truncate this panel a little bit short. That is why I have hastened to get you started, so we can give them time for their presentations. We will aim to finish a little bit after eleven o'clock. This panel is intended to provide a factual backdrop for the discussion that will occur during the course of the remainder of the day. As I think all of you know, peer review comes in a variety of different flavors, and serves a variety of different purposes. Our two speakers this morning aim to provide us with an understanding of the conceptual foundations underlying peer review, and to explain how peer review is used to assess scientific and technical information in various different arenas. Because there are some biographies that are in the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. materials that have been provided to you, I will be very brief in describing our speakers. Our first speaker is Don Kennedy, who obviously already has been up here. As has been indicated, he is the editor in chief of Science magazine, and is the former president of Stanford University. He will be followed by Sheila Jasanoff, the Pforzheimer Professor of Science and Technical Studies at the Kennedy School at Harvard. We will hear from both speakers, and then we will have questions directed at both. Don. Agenda Item: Three Models: FDA, HEI, Science Donald Kennedy, Editor in Chief, Science, President Emeritus, Stanford University DR. KENNEDY: Thank you. I have the feeling that peer review is a term that has many meanings, perhaps too many for it to carry. And so, I'm going to try to do a taxonomy of peer review as we experience it in journals, in government agencies, and in independent institutions that have tried to work out their own modes of peer review in the hopes that some comparative anatomy will help the rest of the discussion. That's a pious hope. It may not be fulfilled. You'll have to judge. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. I want to talk about three different gradients along which we can describe peer review. One of them is one that you have heard a good deal about, namely openness. Is the process transparent, or is it opaque? Are the results there and made available for everyone to see how it worked, or are they not? The second gradient is identification. Does the author or the authors of the work know who the reviewers are? And do the reviewers know who the author is? Because in various journal settings, there are different variants of identification from anonymous to completely onymous. I think I just invented the antonym of anonymous. I have been desperate for some time to do that. And third is the level. Is it individual or individuals, or is a group acting as a group or committee? So, those are the gradients that I will deploy, if I can, in trying to describe how the system works. I'm going to start with journals, because I'm deeply involved with a peer reviewed journal as editor of Science. I could start out perhaps by telling you exactly how we do it. I'll just walk you through it, but forewarn you first that it's not that way in every journal. For example, when a manuscript comes into Science, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. and we get 10,000 of them a year, it goes to an editor -- we have 24 editors, all with PhDs, all with post-doctoral experience, some with some academic expertise before they decided that it was more fun to be in the middle of this process -- there is some internal consultation among editors. And then that manuscript will go out to 2 members of our 100 member board of reviewing editors, who are generally quite senior and experienced people, who will give us a quick turnaround on that manuscript, a fast grade. This is all done electronically, and taking into account those views, we decide whether or not to send it out for indepth review, which we do to about 45 or 50 percent of our submissions. And the in-depth reviewers may be two or three or more people who are asked to share their views to us, both views intended for transmission to the author, and views not intended for transmission to the author, but for educating our editors. And that process may take several weeks. Sometimes we have to bug reviewers that are slow, and sometimes we have to ask for extra reviews, because there are problems that we didn't anticipate that the reviewers initially selected didn't seem appropriately qualified to NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. handle. There is then an exchange with the authors about the results of that review. The authors then will often submit a revised version if they are encouraged to do, or the paper may be rejected without opportunity for resubmission, or the authors may be invited to make some changes, encouraged that should they be able to meet the objections of the reviewers, the manuscript will then be published. Please note that the author's identity is available to the reviewers, and the reviewers' identity are not available to the authors at any point in the discussion. Authors love to guess the identity of the reviewers. Sometimes they tell us with an insistent note that they know damn well that we sent it to Charles Smith, because Charles Smith referred to so many of his own papers in the review. Fortunately, that happens relatively seldom, and we don't respond to these claims of successful identification. I think the process works pretty well, but on the transparency gradient, we are pretty darned opaque. And there are reasons why we operate in the way we do. With respect to anonymity, we believe that our referees are capable of not being overly impressed or overly NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. underimpressed with the status or institutional affiliation of the author. We don't ask colleagues to referee colleagues, and we are about like NSF's grant review process with respect to identifying conflicts of interest. But we think that candor is encouraged by the anonymity of the reviewer. We act like tenure committees in universities in that respect, believing that candor would be a victim if we changed the relative anonymity of reviewers. There are other journals that don't do it that way. There are other journals -- I happen to have published in one recently that conceal the identity of the author, so that it is blind double ended. They think that that encourages referees to be less impressed or more impressed maybe with who the author is. We don't do it that way. We also are opaque with respect to the result. We tell our referees that they shouldn't do what one referee has done to us recently, namely to tell The New York Times that they recommended against a paper that we in fact published. That ignores the fact of course that several other referees had made different recommendations. Our hands were tied with respect to revealing that fact. We didn't return the reporter's telephone calls in that instance, except to tell him that we weren't going to NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. tell him. And we think that the process suffers considerably when participants in the process choose to go public in that way. I want to make one other comment about peer review, and this is really quite personal. I started as a scientist in the fifties. When we had grant applications reviewed at NIH or NSF, we had the timorous, tremulous feeling of having put our work into the hands and minds of giants. It was the same with the journals. They had extraordinarily distinguished editorial boards. We trusted that process, because once again, we felt that were putting our work into the hands of giants. As peer review became more generally applied, as peer review panels became diversified, as the giants got tired of doing it, and other people had to take their place, peer review suddenly became really peer review, and the peers stopped liking it as much as they used to, and trusting it as much as they used to. It's really interesting to me that doubts and suspicions about peer review only began when it really started to be peer review. The Health Effects Institute, which is a relatively new organization, that is, it's exactly 23 years old, and that is new in this business, was established in NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. order to provide for EPA, peer reviewed work on the health effects of mobile source emissions; a pretty targeted mandate. The founding board of three people, I was one, Bill Baker of Bell Labs, and Archibald Cox, Harvard Law School were the others. And we had to put together from scratch, a process to provide what we hoped would be trusted peer review of regulatory science. That is, this is science that was going to be deployed by EPA in regulation making. It was doing exactly the kinds of things that this OMB bulletin is really talking about for the most part. So, we had a research committee that actually brokered research from universities and other kinds of institutions, that was applied to regulatory needs and interests. And we were able to recruit pretty good people to do that. And we had an independent review committee that would review the results of those research projects, and transmit their reviews along with the original research reports by the investigators to the Environmental Protection Agency and to the public, because they are published. So, this was a process that was identified, that is, the authors knew who the reviewers were. In fact, they knew before they started the project, who the reviewers were NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. going to be. And that raised an interesting problem, because one chair of the review committee asked us at one point, wouldn't it be useful if we could get into the process early on, and make suggestions about the conduct of the research? And this became a sort of sticky issue, and we eventually said no, we're not going to let you help them. But we want you to reserved as critics for the end of the process, which is the way it ultimately worked out. I think that work has been quite generally trusted by the agency, and I think it has been generally trusted by the general public. And the process, which is as transparent as it is fully identified, is the way that system continues to work. A final category is composed of standing or ad hoc committees that are called into being, or into continuing existence by federal agencies. And committees of this kind, like FDA advisory committees and the Clean Air Act Science Advisory Committee at EPA or EPA's overarching Science Advisory Board are emblematic of the kinds of committees that Sheila has written about in The Fifth Branch, which John referred to earlier this morning. And she knows much more about it than I do, but I NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. want to point out that in the way these committees function, they include various kinds of materials that are themselves peer reviewed materials. So that an advisory committee, for example, will consider a variety of articles that have appeared in peer reviewed journal literature. They will receive input from agency staff. And they will receive expertise from their own membership that walk in the door with considerable amounts of knowledge. And all of those, in the FDA case, be put together in an advisory committee ruling on for example, the approvability of a particular category of drugs, or the removal of a particular category of medical devices from some particular restriction that it has been under. The transparency of that process is considerable. That is, those committees operate out in public, or if not in public then they work under conditions in which the minutes of the meeting, fairly careful descriptions of the meeting will be made public. But I want to point out that there is something mixed in terms of transparency about the output, because a committee may give considerable weight to non-transparent peer reviewed journal articles in reaching its decision. So, the committee has to apply some judgment about whether NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. it can repose full trust, or at least partial trust in a process that does not meet the full conditions of transparency that it applies to its own work product as it is developed. So, I think there is a certain inevitability to the likelihood that if a particular federal agency works in this way, that its work product, its decisions will be a mixture of transparent and non-transparent review processes. And I think that is a problem with the OMB bulletin that needs to be worked out by the agencies as they develop their own policies. I'm going to stop there, having used I guess one minute less than my allocation, and turn it over to Sheila. Agenda Item: The Conceptual Framework - Sheila S. Jasanoff, Pfozheimer Professor of Science and Technology Studies, John F. Kennedy School of Government, Harvard University DR. JASANOFF: Thank you very much, Don, for sort of laying the groundwork for the things I'm going to talk about, and also to John for having introduced my earlier work. I want to begin though with a little autobiographical note about that earlier work to set the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. context for what I want to talk about today, and also mention quickly, a paradox that I think has been emerging over the course of this morning already, which I think will help frame my comments further. So, the autobiographical note is that some of you may actually be in the position to remember that around the early 1980s, there was a widespread call for more peer review of the science underlying federal regulations. Now, we all know that what goes around, comes around. But it is interesting that in a 20 year cycle we should be coming back to more or less the same kind of rhetoric about the need for peer review, which drove me, back in 1981 or 1982 I think it was, to apply to the National Science Foundation for a grant to study peer review in the federal government. I did get that grant, following peer review. And when I started looking at peer review in the federal government, it became very quickly clear that there was not one thing that I was looking at. And indeed, that peer review was not the interesting thing to be looking at all. That this may seem heresy to people in this audience looking at this topic today, but what turned out to be interesting was the meta-structure of peer review. That is, why people kept insisting they were doing NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. a thing called peer review, when in fact they were doing 100 different sorts of things. What was this magic bullet kind of phrase? And what kinds of actual practices in the government did this map onto? And the end result of that study was not a book called "Peer Review in the Federal Government." It was a book called The Fifth Branch, because I came to the conclusion that what was really worth studying was the complicated set of advisory processes by which people validated and made credible, the science underlying federal regulation. In a way, calling that peer review was a distraction. It was more important to be looking at what actually went on. Well, that brings me to the paradox, which I think all of the three previous speakers have done an enormous amount to both elaborate and to some degree reinforce. And that is peer review is not one thing. It is a multitude of things. And yet, when we are making a regulation, we are talking about peer review, that term. So, how is it that continuing over my now 23 years of engagement with this topic, we have this sort of endless diversity of processes. It's almost like studying the insect world to look at peer review in all of its intricate NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. diversity. And yet, we keep saying there must be at the end of the day, this silver bullet procedure with which we are going to make regulations say for science. Okay, so let's begin as my friend and colleague John Graham did with the definition and say a couple of things about it. This is text drawn from the bulletin itself. And there are some points here that are worth noticing. So, a scientifically rigorous review and critique of a study's methods, results, and findings. I'll come back to that point. And then toward the end of this introductory blurb, for decades the American academic and scientific communities have withheld acknowledgement of scientific studies that have not been subject to rigorous independent peer review. Peer review has been an essential part of the American science scene, and one of the reasons why American science has done so well. Now, the thing that I would invite you to consider is the set of assumptions underlying that. One assumption is that the American science scene is as unitary a thing as peer review itself is. And given the wealth of experience in this audience, I think you all realize that the American science scene is not one thing. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. And if American science has done so well, as indeed it has, it may be because we recognize in our actual practices that American science consists of very many different things, and we have somehow elaborated a set of ways of living with all those different sciences which together add up to an important place for science in our society. So, let's follow then with what an academic like me does when confronted with a statement like that. And that is, can we boil it down, and ask analytically significant questions which will lead to answers about whether we are approaching this problem in the right way or not? So, the first thing to ask, is peer review the same thing in all contexts? And when we are talking about peer review relevant to regulation, does it really help us, or does it confuse us to be talking about NSF and NIH as the models? Does it help us or hurt us to be talking about Science, the journal, as the model? Or should we be talking about different models of peer review in relation to different kinds of scientific activity? And then some very basic primer type questions. When we have said peer review, have we adequately addressed NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. what the stuff is that is supposed to be peer reviewed? Have we addressed who the peers are? Have we addressed what constitutes review? And have we addressed not the point that Dr. Graham eloquently, and I think correctly made that peer review conduces to the credibility of the regulatory work done, but what about the credibility of the peer review itself? That is a separate question, and it seems to me it's worth addressing. So, you won't be able to see what that table says, but I think the interesting point is this is a chart taken from a 1990 book, my own as it happens, but you know a chart that attempted to suggest ways in which we cannot simply transport regulatory science onto research science and treat the two of them as the same thing. And what I tried to do in the way that a social analyst will is get at some of the variables to operationalize in some concrete way, why research done for regulatory purposes, or relevant to regulation, or used in regulation often turns out to be a different animal from the kind of stuff that is published in leading science journals. And in that left-hand column I suggested that the goals, the institutions, the products, the incentives, the time frame, the options available for what to do with the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. science, and the methods of accountability for the science in question were all different in the regulatory domain from the research domain. And this is not an irrelevant point, because when we are trying to generate knowledge for public policy, we legitimately I think, want to serve a different set of purposes from what we are trying to do when we are trying to add to the storehouse of incontrovertible knowledge. That is, knowledge that is generated in the policy domain is meant to be serving a set of wider social purposes. Maybe they are not wider, but at any rate, a different set of social purposes from the kind of incremental knowledge generation with which we try to build up a stable base of what we absolutely want to know, and do not want to have challenged in particular domains of science. So, regulatory science, as I suggested in that book, and as I continue to believe is differently contextualized in institutions, rests on different procedures, and serves different purposes from what ordinary lab bench research science does. So, that leads to questions about what is or should be subject to peer review when we are talking about NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. research related to regulation. And I go back to the text which said, studies, methods, results, and findings. Now, we have learned a lot in the history and sociology of science and the philosophy of science over the last 30 or 35 years. And one of the things that we have learned is that the methods of science are not given in advance, they are not uniform across different domains. That they shift, and that they are variable. So, in the regulatory domain, unlike in many parts of the research domain, one of the important points to mention is that the method for doing the research is often arrived at in the very course of doing that research itself. Very often it is not the case that the scientist has learned those methods, and is simply mechanically applying them to a new problem area. And the HEI example which Don mentioned illustrates very well that the kinds of problems that people are setting out to study often have to be resting on methodologies that are developed in the course of the study itself. One reason that the HEI model has worked so well is that the people with the largest intellectual and economic stakes in developing appropriate methods have been able to negotiate out from the start, how a study's methods ought to NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. be developed and stabilized. So, those methods in the regulatory arena are often established through interdisciplinary and interinstitutional and inter-sectoral negotiations. Those of you from federal agencies will know that very often in telling an industry what kind of studies must be conducted, you may have to lay out the methods for doing that study. It is not given in advance, even what a maximum tolerated dose is in connection with different kinds of chemicals. That is a negotiated output. It is not written out there in some ten commandants of the scientific method. So, methods are often suited to the particular context of the investigation. For instance, we have had numerous court cases around the country on the issue of whether methods used by research labs in doing DNA typing are appropriately advanced as a standard for the forensic arena where the kind of sample that you are subjecting to DNA typing may be of a completely different sort. And questions of validity, and questions of appropriateness may be very different in that sort of much more dirty and untidy forensic context than they are in the research lab. Who are peers? Again, this sounds like a simple question, but I think that before we get into whether they NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. have conflicts of interest or not, we should be asking whether we are sure who peers are in a given kind of situation. So, when we are doing normal science, what the late philosopher Thomas Coon(?) called normal science. That is, we know exactly what the science is. We are operating within a straightforward paradigm, and so on and so forth. It's rather clear who the peers might be. And when NSF for instance, looks for peers in established areas of science, there is not necessarily a problem. But NSF people would be the first to tell you, and having done an enormous amount of refereeing for NSF I know this myself, that as soon as it's a novel area, or an interdisciplinary area, the question of who the peer are becomes difficult even for that premiere scientific research organization that does some of the best peer review that we know of. So, progressively it becomes harder to identify the peers when the science in question is emergent, that is we haven't done the work on that before; when it is uncertain as regards the data that is being relied on; and low consensus as regards the kind of methods to be used in that context; when it crosses disciplines; and when it is NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. politically embedded in the sense that the results matter a lot to the doing of politics. In fact, some scholars of peer review in other countries have suggested that we need a new term called extended peer review, because when science follows those kinds of characteristics, then the peer review in question has to bring in a wider set of experiences and expertises than in normal or paradigmatic science in the sense of my first bullet. So, what then is review? And it is worth perhaps going back to another historical document produced by the National Research Council actually in a report called Understanding Risk, that some of you may be familiar with, that was issued in 1996. And one of the things that that report -- I mean all you need to do is get a look at this diagram and compare it with your mental models, which I'm sure you all have, of the 1983 red book laying out the paradigm of research, risk assessment, and risk characterization. The thing to note is that this rather unfortunately missile looking diagram in fact allows for many, many feedback loops. And maybe I should have taken the report that my colleague Goloman(?) actually produced -- his committee NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. produced at about the same time, because it make a similar kind of point. If you look at that midsection of the rocket, of the launcher, you will see a mini version of the risk assessment/risk management model that the 1983 red book produced. But in 1996, the same institution, the National Research Council, suggested that there was actually a need for much more feedback, particularly in analysis, which is the straightforward scientific kind of exercise, and deliberation, and those feedback loops are meant to suggest that public input can be highly relevant at different stages of the process in which science gets from the place of its production into the places of use. So, review itself can be a very different kind of thing. And that then would help to explain why in Dr. Graham's lists of the costs of peer review we go from zero cost to hundreds of thousands of dollars of costs. Clearly, if peer review is such a variable kind of enterprise, going from the narrow disciplinary peer review of a single study to a much more complex process in which the methods and the objectives and the studies themselves are being developed in the course of a wide public deliberation, you would expect to see a great diversity of costs, of time, and everything NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. else. So, how then should we think about the validity and credibility of the peer review process itself? And obviously, one thing I'm suggesting is that there is no silver bullet. Peer review, as all of our speakers have said so far, is context dependent. We cannot really hope for a one size fits all approach. But then can we hope for a one size fits all approach to regulating peer review, to reviewing its validity, and so on and so forth? These are additional questions I would throw out. In the regulatory context, the peers are harder to identify, and accountability processes are much less straightforward than in research settings. And finally, peer review is itself a process, and hence, certain kinds of process norms are important. We have heard a lot about transparency, but I'm a lawyer by training, and I like certain other process values as well. And one of them is representation, and the other of them is deliberation. So, I think I would throw that in along with transparency as some of the values that we might want to consult or maintain in peer review. Thank you. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Agenda Item: Questions DR. MESERVE: Thank you, Don and Sheila. Both of those were excellent presentations, and I think they lay a good foundation for our discussions throughout the day. There is an opportunity now for questions from the audience. MR. MORRISON: Alan Morrison, a member of the panel. Don, could you say a word about pay? That is, are the peer reviewers paid in these various models that you have? And what is the impact of their pay or lack of it on the quality or availability of peer review? DR. KENNEDY: They are volunteers, Alan. MR. MORRISON: In all the three models? DR. KENNEDY: Oh, no, no. I thought you were talking about journal peer review. We don't pay them at Science. Thank God they are volunteers and wonderfully helpful. It is sort of understood as a public service to the scientific community. At HEI we had both the research and the review committees received annual stipends, not enough to get rich on, but we wanted their attention, and we paid them. DR. MESERVE: Other comments or questions? NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. DR. ASHFORD: Nicholas Ashford, MIT. A question for both Sheila and Don. Alvin Weinberg(?) a long time ago coined the term trans-science to be talking about the kind of science that was used in regulatory processes. No one has I think mentioned it today, but the precautionary principle is something which is receiving a lot of attention, which although the Europeans insist it was discovered in Europe, was discovered actually within the American regulatory system when courts acknowledged the agencies to be able to err on the side of caution in reaching conclusions based on science. Now, if Sheila is right that regulatory science is different than science, and initially it was, where is the role for the precautionary principle here? Because I see the precautionary principle as trying to maintain the precautionary approach, allowing data to yield public policy actions in the absence of what conventional peer review would give you. But there is a much larger attack on the precautionary principle, arguing it's not science, it's anti-science. And what I do see, and I don't know if you see the same thing, I see regulatory science and science becoming almost synonymous as efforts to err on the side of NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. caution are softened on those in the public policy arena. DR. JASANOFF: Nick, that's a very interesting and complex question. I'm not sure that the tie in to peer review is instantly clear. But I think one of the things you are pointing to is that to make public policy on the basis of scientific knowledge requires the exercise of judgment. And it requires the exercise of judgment among other things, on the question is the knowledge that we currently have strong enough to base potentially very costly, but also potentially very beneficial decisions upon? And that question of when is the science reliable enough for us to act on I think underlies some of our concerns about peer review. That is, peer review is part of the answer that people would give when they say well, clearly it has to meet at least the standards of the scientific community. Now, one of the things that peer review does necessarily and by definition is that it gives an answer to the question, how good is the science that we already have before us? It says nothing about what is the relationship of how much we know, to how much we do not know? And of those areas of ignorance in relation to how bad things could NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. become if we didn't take precautionary actions right now. So, no peer review committee can say well, we know this much with certainty. There is a whole lot else that we don't know. And given that there is all this unanswered stuff here, maybe we shouldn't go ahead with what we are planning to undertake right now. That is, to me, the essence of government discretion and governmental judgment. And to think that peer review could offset the need for public health-type judgment or governmental judgment on the behalf of people, I think would be a serious mistake. DR. KENNEDY: Just a quick point about the precautionary principle, which has now been elaborated to work in all kinds of work in foreign territory. I think it really originates from the fact that risk assessment often give you a point estimate, but there is a probability distribution around that point estimate of uncertainty, and the precautionary principle simply takes that into account. And it is a way of introducing the politics of risk aversion into the way we deal with a scientific result that is uncertain. DR. OMAYA: I'm Alex Omaya, Institute of Medicine. I have a question about public involvement in peer review. Do you think there is a role for direct public NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. involvement? In other words, lay person involvement in the peer review process. And if so, where? DR. KENNEDY: Well, in journal peer review, I think not. But obviously in the regulatory process there are a number of places at which public input, stakeholder input, as it is now commonly called, takes place. I'm not sure if it takes place within or collateral with the formal process of peer review, but certainly the public gets an opportunity to comment on the results of the one. DR. JASANOFF: The only thing I would add is that we shouldn't forget in the focus on peer review, that this is something that is being grafted onto a process that we have for developing regulations that already has many, many entry points for the public. So, the question becomes more something like what we heard earlier in the morning, is there need for yet a separate peer review-framed input point that is appropriate for the public, so that questions for the peer reviewers would already include some measure of public input different from what we have under the Administrative Procedure Act or judicial review. We already have all of those different kinds of entry points anyway. And it seems to me that that NRC proposal is one NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. kind of answer. It says when you move data from one significant moment to another, from risk characterization let's say through to risk assessment and management, that those are important points at which to get some public input. But I think I would stress what is the purpose that that public input is supposed to serve? And one of the purposes that I think most people have suggested it should serve is that scientific analysis often becomes very blinkered and starts looking at the technical issues, and not so much as the context in which those technical issues have meaning. But regulatory science has always been done for a purpose. And one reason you need the stakeholders is to remind you of that purpose. And so, a functionalist answer, when will those blinkers actually be to some extent, removed because of public input? That's the only kind of answer I think I could give you. PARTICIPANT: I heard the precautionary principle, and I'm very concerned. I think that's quite political. How do you know which side is the precautionary side? For example, is giving everyone smallpox vaccination the precautionary side, or is not giving? I'm very concerned NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. about that. DR. KENNEDY: Well, that's why I intended to comment slightingly on its application outside the domain of risk assessment. PARTICIPANT: Sheila, you mentioned a number of factors that make regulatory peer review and finding peers more complicated -- the interdisciplinary nature, the undercertainty, and the political significance. Economic regulatory analysis really exemplifies a lot of those qualities. And I was wondering if you could comment on the implications of requiring peer review for social sciences, rather than just the natural sciences, which the bulletin does do? DR. JASANOFF: Well, I think that as John Graham mentioned, it is important to remember the social sciences when one is developing regulatory knowledge. I would simply reinforce that what one expects from peer review in different areas of science in some sense has to be related to the load bearing capacity of that science itself. And one should not imagine that in a field like economics where the scuttlebutt within the community is that there are as many disagreements as binary pairs of representatives of that perfection in the room, it is NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. important to remember that one can't expect the peer review to give the right result. That is, it's not the case that because the agency has hired one expert, and another one comes along and says, no, that result is not to be accepted, that one or the other of those has primacy. You have to keep in mind what the nature of the deliberative process is. So, looking at peer review not as the final kind of justification for doing things one way, but looking at it as a moment for systematic critical inquiry to raise the relevant issues so that judgment comes back where it belongs, in our regulatory agencies, which are constitutionally authorized to make decisions on our behalf. And then thinking how we actually maintain the credibility of those regulatory agencies, not of the scientific community, which does very well for itself on public opinion polls and everything else. It is the regulatory agencies who should be concerned about it. DR. MESERVE: We have to accommodate a speaker who has got to rush to the airport. Let me suggest that we terminate this panel now, but let me suggest to the moderator of the next panel that he allow an opportunity for those few people who had some questions for Don and Sheila NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. to ask them at the end of the next panel. Please join me in thanking Don and Sheila for their excellent presentations. [Applause.] Agenda Item: The Range of Agency Utilized, Generated, and Disseminated Scientific Information: Academic Research; Agency Research; Warnings to the Public; CBI; Third Party Submissions - Moderator: Frederick R. Anderson, Jr., Partner, Cadwalader, Wickersham & Taft MR. ANDERSON: Let me just join the others in thanking the agencies, OMB, and of course the Academy all for participating, and the Academy for hosting this program, which I believe thus far has been a contribution to understanding, and then perhaps commenting on this bulletin. I hope our panel will be able to do the same. We are going to talk about the varieties of information that agencies rely upon as they develop their information documents and products, and then the varieties of the disseminations that they engage in. I think it is quite clear that the bulletin applies to federally disseminated scientific and technical information, studies, reports, documents, and not to any other information products. And this both for the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Information Quality Act, and for the bulletin under it is because of the government's power and influence with information that it places in the public domain, i.e., it disseminates. Whatever we may say about the federal government, and we have more and more to say about it, we do listen to it. Now, what is far less clear is what I have called the reach back or look back problem, where to do their jobs well under this bulletin, reviewers may want to reach back, to look back to information the agency relies upon to write its regulatory science documents. If Sheila is correct in showing some of the disparities between classically peer reviewed science and regulatory science, then the peer reviewers, let's just call them the reviewers, participating under this bulletin may have needs to look harder at information that the agencies have obtained through other channels such as the private sector, research universities, other agencies, state and federal government, and others, although I'm pressed right at this moment to think of others. So, the panel, as I say, is going to address the variety of the sources of information agencies rely upon, and then the variety of disseminations from press release NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. through to rules presumably. Paul Gilman is our first speaker, and exceptionally well qualified to address practically everything. One could imagine his being a kind of poohba of the bulletin, where he could speak from the point of view of the agencies, because he is the assistant administrator for research and development, and science advisor to the administrator at EPA. But he also was former associate director of natural resources, energy, and science at OMB, and the executive director of the Life Sciences and Agricultural Division of this institution, the National Research Council, and has been in the private sector working in the biotechnology area, an area where information products coming from third parties are certain to play heavily in federally disseminated products. Paul. Agenda Item: Speaker - Paul Gilman, Assistant Administrator, Office of Research and Development, Environmental Protection Agency DR. GILMAN: I did have some Power Point slides. I will leave them. Fred, you locked up the computer when you bumped it over there. Foiled by technology. But I will NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. leave them. I don't know whether there is a Web site for today's session, and so they can go up on the Web site as well. Let me give some context from an EPA perspective. The agency's mission is a very broad one, to protect the public health and the environment. It does so with three principal tools, the regulatory actions that folks are so familiar with. But also voluntary actions, and they can range from voluntary programs like the Energy Star Program, on through to voluntary actions by regulated parties, like recently the folks who produce pressure treated lumber voluntarily taking their product off the market at the end of this calendar year. And lastly, one of the major tools of the agency is research and development, and also the use of science throughout all of those different activities. The scope of science that the agency uses runs the gamut that Fred laid out in terms of the different parties doing the research. And the content of the research also is very wide. It ranges from ecologically-related studies through to human health, as I mentioned, but also technology development, science policy development, really truly a very NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. broad array. The use of peer review in the agency is something that has been evolving through time. Probably the first point in a chronology would be to cover when Admin. Riley put together a blue ribbon panel to look at peer review inside the agency. And in 1992, they came out with a report, Safeguarding the Future: Credible Science, Credible Decisions. He subsequently put out a policy statement. Admin. Browner did subsequent to that. And in 1995, the agency really set out a policy that all major work products, not just research activities of the agency, would be subject to peer review. The agency produced its first peer review handbook, guidance to the staff inside the agency as to how to conduct peer review, what levels of peer review to give different work products in 1998. The second edition of that was issued in the year 2000. You can access that through our Web site. And actually, just this morning we released what we call the Science Inventory, which is an inventory of all the science-related activities inside the agency, not just research, but activities at the regional level, as well as NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the programmatic level. And the peer review view of that database provides you the opportunity to look at the different work products, and see what peer review has been assigned to them, and what stage that work is. Let me see if I can give you a thumbnail sketch of peer review at the EPA. In 1995, when that policy was implemented, and major work products were to be subjected to peer review, 112 work products were identified in that year. Last year, 859 work products were identified, and about 100 of those work products were deemed to be sufficiently repetitive of prior peer reviewed activities that they weren't subject to peer review. Of the remaining 750, it was determined that 91 percent of them should be subjected to external peer review. So, of the 750, only 10 percent received internal peer review. I think that is quite a remarkable number, and that's why I took the time to tell it to you. People often ask, what about peer review at EPA? I'll read you a quote from Dr. Jean Manonowski(?) at Hopkins, who is a member of the Science Advisory Board and has been involved in a number of the activities looking at peer review in the agency. When she testified before the House Science Committee in April 2002 she said, "I think EPA NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. has taken massive steps to improve their peer review, and the Science Advisory Board is not the only place. From what we have looked at in EPA, they have done an extremely good job getting almost everything that they looked at now peer reviewed." Prior to the policy of 1995, approximately only 35 percent of the activities that would fall into these work products category were peer reviewed. Today, the average over that time since 1995 is about 71 percent. So, it has been a climbing activity. I mentioned the Web site we have opened for the Science Inventory. The URL for that is www.epa.gov/si, for science inventory, for those of you who are interested in it. As you all know, the OMB issued its Information Quality Guidelines instruction to agencies in February 2002. The EPA engaged in its look at what it wanted to do for itself, and released those in October of 2002. And one of the unique features I think of the agency's activities were what we then dubbed third party assessment factors. Those were factors that we wanted outside producers of information -- academics, industry, whatever -to understand that we used in weighing the information that NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. they produced. Subsequent to our creating those factors, and putting them out for public comment, we have really adopted them as assessment factors for both internal and external work. And our hope in creating them was to really -- to use that word again -- provide transparency to outside users for what it is that the agency values when it is looking at work products produced by academics and others. The factors sound like the scientific equivalent of motherhood and apple pie. It's about soundness, applicability and utility, clarity and completeness, uncertainty and variability, evaluation and review. They are factors that at a workshop that we asked the Academy to do here on these factors, a number of people from the outside scientific community said, gosh, this is the right way to do the business of science. It's what we do. And as one editor of an epidemiological journal said that very same thing, he also noted now of course when someone submits an article to me, I have page limitations to consider. And so, we might discourage them from fully elaborating on a method, or fully explaining the statistical treatment. And that was the very reason why we felt a need to do these assessment factors, to try and communicate to folks NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. on the outside sometimes the published literature is insufficient for the purposes of the regulatory application of a piece of information. I actually have a very nice here that puts the OMB proposal up along side EPA's guidance, and I'll give you the bottom line of that. And it is that I think the EPA guidance on peer review falls well within the guidance that is being suggested by the Office of Management and Budget. I really do believe that the infrastructure that is now in place at the agency can easily absorb and thrive in the context of the OMB guidance. There are some areas where it will help us improve. There are some programmatic areas where we can do better. That's a continuous process. And the OMB guidance gives us really the impetus to do that. But the bottom line of that slide was we are in pretty good shape. So, let me say in summary, the EPA has an established history in the area of peer review. Its peer review records are now available online. I gave you the URL. We view peer review as a pillar of information quality. And we think information quality has to be extended out side just the activities of the EPA. And that is why we have created the assessment factors. And lastly, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. we do think that EPA is well positioned to address the OMB's peer review bulletin. Thank you. MR. ANDERSON: Thanks, Paul. And because of Paul's time constraints, we are going to take a window of questioning for him. Paul, I have to say that we attorneys don't typically use audiovisual aids or Power Point, but I don't know that we have taken to actively sabotaging stuff. Questions for Paul? MS. CASANO: Pat Casano, GE. I'm just curious as to what kind of tracking mechanism or monitoring mechanism you have in place to track whether or how peer review is done in relationship to the guidelines at the agency? DR. GILMAN: The very same database that will have a public view as of today is the database that we utilize. I have a group within the Office of Research and Development that takes on the task of regularly checking that. We also have within the Office of Environmental Information, a quality assurance group that literally audits labs, centers, and programs for their compliance with the procedural aspects of it, whether the conflict of interest NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. forms have been appropriately gathered, et cetera. So, we actually have sort of two infrastructures there that are working the system in terms of both process, and ultimately quality of the review as well. DR. MESERVE: Dick Meserve, Carnegie Institution. Paul, could you say something about the aspect of the OMB bulletin that relates to public comment before the peer review? Do you undertake that now? Are you concerned about the delay that might be incident to it? DR. GILMAN: It is a very interesting question. It's one that is raised all the time as we enter into peer reviews. We have both formal situations where we solicit public comment in a fashion that is timely for the peer reviewers, and we have less formal circumstances. Some of our peer reviews put a notice out that if someone wishes to make a public comment, that here is a URL to submit it to, or here is an individual to mail it to. Our hope is that we have public comment available to the peer reviewers. A number of folks providing peer review often times look at the peer review group as the court, if you will, that will decide which of the public comments are the most valid or most important. We still take the position that it's the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. responsibility of the agency to cope with those public comments. And while we give them to the peer reviewers for their information, and encourage them to pursue whatever avenue they want coming from their thought process, we do try and keep the two processes separate, that of the public comment process and the peer review process. But we want the peer review process to be informed by the public comment. We will probably have to formalize that more, given the guidance at the OMB. DR. OMENN: Gil Omenn, University of Michigan. When you mentioned several hundred per year, sort of the question is what would be interesting cases in the numerator. It was very helpful when John Graham gave a few examples this morning. Maybe you could address two of the more salient ones of the last decade, the diesel health assessment he referred to, and dioxins. DR. GILMAN: Yes, there are those that actually received multiple peer reviews. Dioxin is a good example of it, there are others as well. And it sort of again reflects the fact that the agency will re-review an item if substantial comments are received in an initial peer review, and substantial modification is made to it. So, we try to err on the side of frankly multiple NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. peer reviews. And it is a tiered system. Of that 650 items or so, or 675 items that received external peer review, some portion of them are products that will be a so-called letter review where three individuals are asked to provide written comments on the topic. Some portion will be face-to-face panel reviews. Those panels range in size from on the order of 10 folks to 17-18 people. They also include the Scientific Review Board, an overarching independent review group for the agency. The National Research Council is also another avenue for us. I joke that they should be providing me a free parking place over here, because we do use their advice extensively. So, the subject matter ranges from new methodologies to significant health assessments like a dioxin or a diesel emissions kind of health assessment. And we try and tier the system to meet the different levels that they would necessitate. And I do think it's fair to say we err on the side of jumping to a face-to-face peer panel or a Science Advisory Board if we think there is controversy involved. MR. SIGMAN: Dave Sigman for Exxon/Mobil. Paul, do you plan presently or in the future to NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. reflect not only the comments of the peer reviewers in the public record, but also any changes or comments you make in response to those peer reviews where the regulations either have changed for some reason, or whether some judgment has been changed, or whether in fact you don't agree with the peer reviewers and have taken action based on your own discretion? DR. GILMAN: Right. The Science Advisory Board reviews currently get a letter from the administrator outlining the response of the agency to their review comments. There is another level of activity, a pretty well known database called the IRIS database, which is a place where many of the toxicological values are stored for the agency. Those activities also go through both a response to public comment and a formal response to peer review kind of document. I think we are going to be looking to make that more uniform across the agency. Hopefully, we might even be using the database that we have created to provide easier access to those kinds of comments as well. Thank you for the question. MR. HALPERN: Harold Halpern, Department of Energy. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. What is the structure of your Science Advisory Board? And how does it interact with the individual peer review that is conducted? DR. GILMAN: The Science Advisory Board is actually going through a restructuring at this point in time. It has traditionally been at a place where the agency can ask for forward-looking advice, what route to follow, or the like. But it's also been a place that oversees peer review of major decisions or major work products of the agency. The revamping of that is to both increase its size, and to increase the depth of its reviews of those work products. There will be an additional review of the review worked into the process as well. So, in contrast perhaps to the Science Advisory Board at the Department of Energy, which in my experience is much more of a forward thinking, visionary enterprise, it has traditionally been a quality component to the EPA process. MR. HALPERN: Are the peer reviewers, are they subcommittees of the Science Advisory Board? How does that work? DR. GILMAN: When the peer review is done by the Science Advisory Board, and the number of activities they do NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. is limited to maybe 30-40 per year of all kinds of the forward-looking things, as well as the more traditional as we are talking about of peer review, they sometimes do it by their executive committee, which is an overarching group representing all the different subcommittees that are specially utilized for interdisciplinary work. But more often they are done by subcommittees of the group, which has more focused expertise. So, there is a group that looks at economics. There is a group that looks at ecological issues. There is a group that looks at human health-related issues, and so on. MR. ANDERSON: Paul, just for perspective, given OMB's emphasis on transparency and you agency's prior emphasis on transparency, is there any place left for confidential peer reviews where you have reason to believe a report is sensitive, or the public impact will be great, or is that simply a thing of the past? DR. GILMAN: Well, I do think that's one of the challenges. Sheila, in one of her slides said that this isn't necessarily a magic bullet. I made a point with some colleagues from other federal agencies the other day that for many in the audience, probably the thought is that guidance such as this is aimed at agencies like the EPA, and NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. therefore it might come as a surprise that the EPA is comfortable with the guidance, and hopefully, operating pretty well within it currently. Because gosh if we were, why the heck does everybody hate EPA? And so, it does point out the fact that you can peer review the underlying information and analysis. You can have a pretty good sense of scientific consensus, but you may still not be very happy with the answer that you get in the regulatory context. And that's something I think we should all keep in mind. The question occasionally happens when EPA puts out a draft for comment. The public at large, the media grab onto that draft, and say we have the answer in hand. We have the truth. And I have recently been chastised in some of the newsletters inside the Beltway for having said that I thought that a draft that we had, that had undergone peer review by our Science Advisory Board was not sufficiently sound to form the basis of interim guidance to the agency in the intervening period between our revising it per the peer review comments. I think those are judgment decisions. How you deal with that situation where the draft becomes the focus of debate remains to be seen. How you can do a peer review NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. and have public comment, and do it without public distribution of the review, it would be a little problematic. MR. ANDERSON: All right, just coming from another field, from the law, I'm struck by the -- you see this on Perry Mason-type shows. The lawyer asks a question, and an objection is immediately registered. But the point was to ask the question and then withdraw it, because it's a presence of meaning. So, a document out there with all kinds of disclaimers nevertheless is out there. Thank you. DR. GILMAN: Thank you all for the opportunity. [Applause.] MR. ANDERSON: Let's proceed with the next presentation. Jim Scanlon is the acting deputy assistant secretary for science and data policy -- what could be more germane -- in the Office of Planning and Evaluation at HHS. Jim. Agenda Item: Speaker - James Scanlon, Acting Deputy Assistant Secretary for Science and Data Policy, Office of the Assistant Secretary for Planning and Evaluation, Department of Health and Human Services MR. SCANLON: Thanks very much, Fred. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Good morning everyone. As Fred said, I'm with the Office of Science and Data Policy within the Office of the Secretary at HHS. And it is our office that has the role of coordinating science policy and data policy across all of HHS. And I'm going to talk a little bit this morning about the range of scientific information and studies that our various scientific agencies develop, as well as the studies and information that they use and disseminate in general, but especially in connection with scientific regulatory activities. I'll also talk a little bit about the context of scientific information and peer review within HHS. I think in our case, the scope, magnitude, and complexity, and diversity forces us to take a more nuanced view of how to achieve the goals of the bulletin. And I'll talk a little bit in general terms about the models of peer review that our agencies use, but again, the detail will come this afternoon in another panel. HHS is both a producer and a user of scientific information. As you know, our agencies in many cases are household words in the public health area and the research area. And so, obviously, we develop and use scientific information broadly in connection with our statutes in NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. public health and in research. And we use such information in support of our science-based regulations as well. And most of these activities are described, both the types of information that we use and disseminate, as well as the review policies we have incorporated in our HHS Information Quality Guidelines on the HHS Web site. So, obviously, within HHS in terms of today's workshop, we have a number of scientific regulatory activities, and a number of programs that generate the science information that would be used in those regulatory activities as well. Let me at the outset, sort of parallel what my colleague at EPA said in terms of we are clearly sharing the broad goal of the bulletin of insuring that objective and high quality scientific information is available in support of our regulatory decisions. And I think as has been stated previously, we have a long standing commitment to assuring that the scientific information that we both development and disseminate directly through our intramural programs, as well as the academic research that we fund meets really the highest standards of scientific quality, objectivity, and integrity. And we have learned a lot over the years in the kinds of mechanisms that will assure us to meet those goals. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Similarly, in the scientific regulations, scientifically-based regulations that we develop and issue, again our standard is to look for the best available scientific information to support the directions and the analysis in the regulations as well. And needless to say, we employ a wide variety of scientific review and even peer review procedures to support those goals. Just in terms of status and progress, we are reviewing the OMB bulletin within HHS. And we are looking at the potential impact and the interaction, how does it interact with our current peer review and scientific review activities? And in that process, it is quite complex, as you would imagine, and we have involved not only our regulatory channels, but our science policy channels, and our legal staff, not surprisingly as well. Let me talk a little bit about more context. I think context in this area is virtually everything. And there are some aspects of HHS and the kind of an agency that we are that kind of make some things work, and other things not work. First, in terms of scope and mission, you are familiar with our mission. We are the United States government's principal agency for protecting the health of NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the population, and providing essential human services. We are a very large and diverse agency, and in fact we have 11 large operating agencies, and over 300 programs. Some of them, as I said, are household names in the health and public health area. Medicare is the largest health plan in the US, with over 900 million claims annually. Several previous speakers have mentioned the National Institutes of Health, and Dr. Lenfant will give us more detail. But NIH is the nation's largest biomedical research agency, and the largest funder of investigator-initiated awards in the United States, and in the world as well. At any given point there are about 35,000 research projects underway funded through NIH. We have other major scientific agencies. The FDA is probably the best well known scientific regulatory agency, but our CDC and our Agency for Healthcare Research and Quality really are research-oriented agencies as well, whose business is to generate research. So, again, when we look at peer review, and how to achieve the goals of peer review with the OMB bulletin across this diversity, and across this scale of complexity, typically we are looking for nuanced kind of approaches, and NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. flexible approaches, as I think some of our speakers have mentioned this morning. Let me continue the diversity theme a little bit more, because there are quite a number of differences between a scientific regulatory agency, which may use the information developed, and may in fact develop the information themselves, and the agencies such as NIH, CDC, and others that may actually be in the business of generating the scientific information and studies in epidemiology that may support some of our regulations as well. Not only do we have different program missions within HHS, but some of our very statutes are different, and they actually derive from different authorizing committees and appropriations committees in Congress. The FDA authorizing statute, for example, is more akin to the appropriations and authorizing committees for the Department of Agriculture, as Don would know, than it is to some of our other public health agencies. And the Social Security Act is the authorizing legislation for some of our grant programs and entitlement programs as well. So, this creates a fair amount of diversity in terms of the constituents, the communities, and the partners NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. that we deal with, and the kinds of mechanisms that we would use to develop information, and to insure its quality as well. In general, one size fits all approaches typically have difficulty in HHS. We typically look for an approach that includes umbrella policies across HHS, and then other approaches and policies that are tailored to the individual science agency or programmatic agency with HHS. And this was the concept reflected in our HHS information quality guidance as well. We also have a long tradition of peer review and scientific review, and assuring information quality within HHS, not only in our own internal agency research, but in the research that we fund. And I think in many ways, the NIH has kind of perfected and pretty originated much of the study section, peer review, and has contributed to much of the journal peer review as well. And those are well established and well documented policies. They are available within HHS, as well on the NIH Web site in terms of grants policy guidance. So, again, we are hoping to build on what we have already done, insuring that it comports with the principles enunciated in the bulletin as well, but these concepts NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. themselves, and the goals themselves are quite familiar concepts at HHS. We have a long tradition with them. And we think that building on those policies and procedures, we can certainly meet the goals of the bulletin. Let me turn now to kind of the range of scientific information and technical studies that our regulatory agencies might use in HHS. And again, FDA may be the major example here of scientific regulatory activities, but we do have other scientific activities of course at CDC and NIH and AHRQ and other places as well. Again, at HHS we tend to distinguish between the agency that issues the regulation, and may use scientific information that it develops or that another agency develops, or that appears in a peer reviewed literature, and the research generation and knowledge generation agencies like the NIH and other, though in any instance it could be the regulatory agency that is developing the information as well. And let me give you some examples of what kind of scientific information and studies could be utilized in any of the science-oriented regulations that we may issue. And literally, the agency in planning its regulatory approach, can really look across the panoply of existing scientific NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. information and studies. It is typically not one study or analysis. And it is often the cumulative or the consensus scientific knowledge in an area. So, with that background, let me just run through some of the types of activities. This probably fits best with the FDA model, but not solely in terms of scientificoriented regulations. First, any one of the scientificallyoriented regulations could employ published scientific research studies in the formulation of its strategy. These would the results of biomedical, behavioral, health services research, and even social science research. And these are typically drawn from the peer reviewed literature, and the scientific literature that we discussed earlier this morning. An FDA or another agency could also employ information developed and disseminated by other agencies. In the case of the FDA, for example, in the food safety area for example, it would not be unusual for FDA to rely on CDC case control studies or surveillance data, or epidemiological other studies to support its regulatory strategy. The agency could also, in its regulation, develop its own scientific information. This is not unusual. And NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. this would typically be done by internal scientists. NIH alone has over 3,000 doctorally-trained scientists and medical researchers. When this is done, when the agency develops its own information, there is typically an internal review program within the agency, and often the information is presented to a science advisory committee or some other parallel kind of a forum as well. Our agencies as well often rely on committees like the National Academy of Sciences and the National Research Council. And we are often relying on, and in fact we commission many studies for the Institute of Medicine to look at an area, and to give us recommendations. Those too could be used in the development of a scientific regulation within HHS. Many of our agencies as was alluded earlier, also have agency scientific advisory committees. And again, these are structured in a way that makes the most sense to the agency. At NIH, there is a Director's Advisory Committee. Each of the institutes has an advisory committee, and most of the agencies have boards of scientific counselors as well. And this is in addition to the study section review that the individual grant awards NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. would be subject to. Now, similarly within the agencies ourselves, within CDC, NIH, FDA, and others, we typically have a position of the science advisor who is a senior scientist within the agency, has a staff of his or her own, is separate from program operations. And the science advisor's office is often responsible for looking over in an independent fashion, the scientific work of the agency. And in many cases, reviewing the proposed scientific information that might be used in any agency action, including a proposed regulation as well. And of course, our agencies could rely on scientific literature reviews, state of the science reviews, and consensus reviews and reports. And in many cases, our agencies have actually developed the models for how these are conducted in terms of the NIH consensus conference as well. It is not unusual as well for a regulation developed by one of our agencies to at least include, and possibly rely on information from their parties. These could be industry groups, public health groups, and other organizations who have developed their own analyses and submit them to the agencies. There are a variety of ways to NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. get this information to the agency for consideration. And then there is more descriptive information, which isn't really the study model that the journals would look at. It's more descriptive epidemiology. It's information reported from our surveillance systems, from our safety monitoring systems, and so on. And this is often used in terms of adverse events, in terms of the incidence of diseases, trends in diseases and disorders, risk factors, risk assessments, and so on. And these could be used as well. And finally, we have a whole panoply of epidemiological studies that are part of the armamentarium for any agency to use. These would include formal risk assessment, toxicological studies, standardized testing for toxicological studies as well. And any of these in any given case would be employed in the development of a proposed regulation. I'm running short of time here, so let me just give a broad set of examples in terms of the kinds of peer review models employed across HHS. So, this is just within HHS, and again, I think flexibility is our standard here. As I indicated, there are a number of external review models for peer review, and often the model is simply to use a peer NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. reviewed journal studies, and that's often the place to start, or scientific reviews of the journals. The NIH study section model has been mentioned as well in terms of an external model. And the agency's scientific advisory committee model, which we have in numerous agencies across HHS, are used as well. We have internal peer review models that I mentioned in terms of internal agency scientists and senior science advisors as well. And then we have mixed models, which kind of combine I guess in many ways like the EPA process, information developed internally by agency scientists. The information would then be subject to an internal peer review. It would then be subject to an external peer review of two or three external reviewers. And then it would be subject and presented to a science advisory committee. This is actually the process in the National Toxicology Program. And even then, there is opportunity for public comment before information is disseminated. And there are candidate areas that one can nominate for inclusion in the program of toxicology studies. So, let me stop there, Fred. MR. ANDERSON: Thank you, Jim. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Our next speaker is Claude Lenfant, who was the director of the National Heart, Lung, and Blood Institute within NIH. And the institute has already been involved with the statute under which the bulletin is being developed, the Information Quality Act, because of a data correction request regarding information of the very kind that I identified earlier involving reach back or look back, that is data from university researchers that in turn was used to underpin an institute policy statement about salt consumption in the US diet. Claude. Agenda Item: Speaker - Claude Lenfant, former Director, National Heart, Lung, and Blood Institute, National Institutes of Health DR. LENFANT: Thank you very much. During the next few moments I would like to complement if you want in a way, some of the comments that have been made by James Scanlon. In lieu of this closure, let me first say that I'm officially unemployed, and therefore, I do not speak on behalf of the National Institutes of Health. But I bring to the table here, 40 years of having lived with peer reviews through the process that was described here. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. What we can say is that we are all in favor of peer review. Actually, the National Institutes' works and functions through peer review. And I would even say that in most instances, there is a legal requirement that peer review be used for most of the decisions which are made at the National Institutes of Health, especially those which entail allocation of public funds. I would like to make a few comments really reflecting on what I heard this morning as it relates to the National Institutes of Health, and time permitting, give a couple of examples of situations which I believe warrants the consideration of those who will make final decisions in this process. One of the questions that came to my mind is the following. Is biomedical research regulatory research? I personally think not, and I think it shouldn't be. However, we all have to recognize that there are some rare situations where research is commissioned to answer some very specific questions. For example, the research effort which is being developed at the National Institutes of Health for biodefense. We can very well conceive that eventually that will lead to regulations. What I heard this morning is somebody asking, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. well, does all that apply to NIH? And I think the answer was well, if there is an official position of the agency, it should go through the process that is discussed here. And I have to say that I'm a little bit concerned by the question and the answer, and let me tell you why. As you heard from James Scanlon, the role of the Department of Health and Human Services is to maintain the health of the American people. I would like to add something. It is to maintain and improve the health of the American people. And therefore, to improve the health of the American people, we have to come up with recommendations which are the result of a huge amount of research, debates and discussions. I say very shortly and very briefly, but in effect, some of these discussions go on ad nauseam until finally we come up with something that we think we can communicate to the public. Let me give you an example. I suppose everybody here lives inside the Beltway, and you must know that during the last six months there is no day that goes by without hearing something about obesity and controlling of obesity, either in the newspapers or on the radio, on the TV. Well, there are lots of positions of NIH about NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. that. Should we refrain from pronouncing these positions, and let them go through another deliberative process that would go forever before there is a firm stand which is taken up and down the department? But we've got to do something about that, or basically we are all going to spend our life in a hospital, and spending lots of money in the process. So, I think that we have to be very careful here on how those things will be applied. Lots of basic research is supported by the National Institutes of Health. And I have to say that I am not aware of any situations where basically basic research would be anywhere close to regulatory actions. In contrast, we also support lots of clinical research, and that is research about or with or on people. And the goal of this research is clearly to lead in the improvement of public health. And that may indeed in many instances lead to conclusions that other agencies, primarily our sister agency the FDA, if it comes to a new medication or devices which are going to be implanted in individuals. And the FDA has to sanction a number of points, and establish regulations which can be yes, you can do it, or no, you shouldn't do it. And that is, I believe, a very fair and appropriate NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. process. But what NIH says is usually to make a recommendation to give an opinion of advice. There are some exceptions, and let me give you an exception that actually involved me some 13 or 14 years ago. The scientific community had advised the National Heart, Lung, and Blood Institute to initiate a clinical trial to test the effectiveness of three medications which were already available in order to see if they would decrease the disorder of the heartbeat, if you want, arrhythmia -- that's the technical word. And that is a very serious disorder that can lead to all kinds of complications, not the least sudden death. So, these medications, we wanted to extend the indications of these medications to see if that would treat that condition. We initiated that study, and very quickly it became clear that these medications were more harmful than beneficial. In fact, the subjects who were taking these medications, the death rate if you want among those who were treated with these medications was twice the rate as those who were not taking the medication. So, the decision was made to stop the study. And NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the issue was how are we going to announce that? Well, we could not go with a deliberative process, because meanwhile the patients who were taking this medication would continue to be harmed. And the point of my story is that eight days after the decision was made to stop these medications, the public knew about it. A press conference had been made. The FDA had taken actions, and a paper was in preparation, and it was published one week later in the New England Journal of Medicine. So, my point here is that I think we have to be very careful on how decisions are going to be made about very specific studies and issues. And on the other hand, we have to remember that as you heard, in the Department of Health and Human Services there are extensive review processes, peer review processes. At the National Institutes of Health, as I mentioned earlier, it is a legal requirement. But what you should know is that in many of the panels which participate in the process, all the steps which are taken, are actually not scientific people. They are people who are lawyers, economists, people working with voluntary organizations. And so therefore, there is already a very strong NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. participation of the public in these processes. And I personally think that it is very important, and that it should be continued. So, these are my comments. They are really supplementary to what James said, and we'll go from there. Agenda Item: Questions MR. ANDERSON: Thank you very much. Let's do several things. While I'm speaking, let me invite Don and Sheila to come back up to the podium to the two empty chairs. And as you go to the microphones, keep in mind that Claude also has a plane to catch, therefore the initial questions should be for him, because of his time constraint. He will have to leave no later than 12:15 pm. Now, after we address questions to Claude, we will open the floor more broadly to questions to this entire reconstituted panel. Questions for Claude Lenfant? Well, then ponder this. He will be here for another 15 minutes. Then let's open the floor to questions at large. If you think of a question that you want specifically to address to Claude, he will be here for a while. MS. PHIBBS: Dr. Scanlon, Pat Phibbs, reporter with BNA. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. There is a section of OMB's peer review guidance that strongly encourages federal agencies to make information correction requests available to the public on the Internet. Will HHS agencies be doing that? MR. SCANLON: Yes. The speaker is referring to a requirement in the existing Information Quality Guidelines and peer review for requests for correction that are raised through the Information Quality Guidelines complaint procedure, that they be posted on the Web site, I guess that specifically. We have already made them available in a docket. We have received about 12 requests for correction within HHS. Dr. Lenfant received one specifically as well when he was director of the National Heart, Lung, and Blood Institute. As we complete them, we are making them available in a docket. But yes, we are planning to put them up on our HHS information quality Web site as well. DR. GASTWIRTH: Joseph Gastwirth, Department of Statistics, George Washington University. I guess my question is for Sheila, because she also wrote another book on science in the courts. And regulations typically wind up in the courtroom. So, I was wondering how you felt these new regulations for peer review NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. should fit into the Daubert framework the Supreme Court laid out roughly ten years ago for the judges to assess expert testimony? DR. JASANOFF: And you expect an on the spot, immediate answer to a technical question? Well, you know the first sort of technical answer to the question is that one of the Daubert criteria was in fact peer review. Well, a couple of things. One would need to think about the procedural context in which the challenge would have arisen. And Daubert is about civil litigation between private parties. It is not directly applicable to a challenge to the rationality of federal regulations per se. There has been some suggestions that the Daubert criteria ought to be directly applied by the courts in challenges to the strength of the basis of regulatory issues. But if agencies have complied with these peer review guidelines and criteria or whatever final version of them emerges, that would be prima facia evidence that that kind of Daubert criterion had been met. That being said, you know the stuff that I have written about this. I think that Daubert and its interpretation idealize peer review in somewhat the same sort of ways that attempts to proceduralize a single sort of NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. formula for peer review tends to do. I retain a somewhat healthy skepticism about whether going through a process which people have labeled peer review either is or is not all the time a guarantee of the fairness, the decipherability, the utility, the validity of the kind of knowledge claims that are being represented. So, while I think technically what has happened to Daubert, namely that in spite of Justice Blackman's injunction in the majority opinion that his four criteria should not be taken as a checklist, nevertheless they have been taken as a checklist. I think an agency's claim that it has complied with something called peer review is likely to satisfy that checklist mentality. This is something that I find intellectually problematic. MS. SHAH: Hello, Himani Shah from Covington and Burling. Do you believe that under the OMB bulletin, FDA's consideration of scientific information regarding pre-market approvals for medical devices and new drug approvals would be subject to the new peer review requirements? Or would they considered individual adjudications, and thus exempt from the requirements? MR. SCANLON: For Don or myself? NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. DR. KENNEDY: I think I would be glad if you answered it. MR. SCANLON: Well, again, we are still sorting through the requirements of the bulletin at HHS. But I think in general, our feeling is that individual adjudications wouldn't be subject. DR. MICHAELS: David Michaels, George Washington University. This is to Dr. Lenfant, but actually probably to the whole panel. You have raised an interesting issue in terms of the blurring of the areas of what might be covered. The ending of the anti-arrhythmia trials both had an affect on the individual adjudication of a product, but had major policy implications in terms of telling the public no longer to take these drugs that were on the market, and having obviously implications for Medicare, major international implications in the way health care is provided. Under these new guidelines, should that decision of ending the trial and telling the world to no longer take these drugs, should that have had external peer review? DR. LENFANT: That well it had. The initiation of the study was extensively peer reviewed, and published actually, the protocols and all that as we were going along. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. And once the decision was made -- well, let me rephrase that. The decision to stop the trial was not mine or one of my colleagues. It was through a process where we brought to the institute, a number of experts in the field and said, we have this situation. Without any hesitation they said you have got to stop it. It's unethical to continue it, and that was the end of it. DR. MICHAELS: So, you would say that was the peer review process, as well as the decision-making process? DR. LENFANT: Yes. DR. MICHAELS: But then the two of them are the same. You are saying that the process of ending the trial, and the peer review of that ending involved the same people. DR. LENFANT: No, different people. I should say if you allow me just two minutes more here, let me mention another study which was the release of the data from the Women's Health Initiative last July. And I'm sure my colleagues here or former colleagues I should say remember that that created quite a few ripples in the system. And Dr. Zerhouni, the current director of the National Institutes of Health, made what I think is a terrific decision, which was to say okay, NIH is not going NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. to provide a position. We know what the scientific data came up with. But we will not formulate the decision until we have convened a large group. At NIH there is an auditorium that size, which was filled with people from all points of view, and participated in this discussion for one day. And at the end of the day, a position was developed and communicated. DR. KENNEDY: I might just add one thing to the discussion of the so-called interim look, as that process is termed. Under ordinary circumstances, without a clinical trial of the level of participation and the level of public interest of this one, the way it would normally work is that a modest sized phase II trial would be negotiated between the chief medical officer of the pharmaceutical company that wishes the approval of the drug, and the principal investigator, who would line up the other investigators, and patients would be enrolled. And under most circumstances, the chief medical officer of the company and the PI would get together on the interim look to see whether there was an ethical problem here. And if the placebo recipients were doing either conspicuously better or spectacularly worse, it is over. DR. LENFANT: Well, you know it's interesting that NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. you say that. History is not the purpose of this meeting here, but there was considerable debate. And I can tell you that the manufacturers of these two medications, there were two companies, did not really agree with the decision. And they did not want the institute to go ahead with a press conference. And in fact there was even a book written about that story. MR. ANDERSON: Keep in mind that the deadline for comments to OMB is 15 December. Now, if you have ideas about emergency or exigent circumstances where an agency has to act faster, and can't come up with an appropriate sliding scale peer review mechanism, or about adjudication, how narrowly to construe adjudication, then you have an opportunity to say something about it come comment time, and to suggest to OMB how it might better handle these categories. John. PARTICIPANT: Dr. Kennedy referred to the uncertainty in point estimates about risk, but in fact there are a lot of other kinds of uncertainty that the honest peer reviewer will recognize and comment on. But there is a second distribution of uncertainty, and that is the distribution across the peer reviewers or potential peer NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. reviewers. It is the job of the decision-maker to integrate those two kinds of uncertainty, to choose the right kinds of peer reviewers, deliberately chosen with different points of view so that you will have different perspectives, cover all the different aspects. If the peer reviewers are in close agreement in detail, as well as their summary judgment, all but one of them are redundant. You don't want the peer reviewers to look at things in the same way, and come to the same conclusions. The critical thing is that the peer reviewers should be sources of information, not decision-makers. And to turn over a journal to peer reviewers to vote on articles is an error. To turn over major government decisions to peer reviewers to vote and take over the function of the decision-maker I think is also an error. We have to look at them as a kind of information, not the only kind of information, and that all has to be brought together and properly integrated in the process of coming to a decision. DR. KENNEDY: I surely agree with that. We frequently even overturn peer reviewer judgments in-house. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. DR. JASANOFF: And if I could make a comment. That was precisely the sort of point that I was making earlier about in this striving for a perfect approach to peer review, not losing sight of the fact that somebody has to make a discretionary judgment about a set of issues at the end of the day. And, John, you are absolutely right that one of those discretionary judgments is about how to weigh up the inputs from the different peer reviewers. And I'm reminded that there was a time when a group of researchers at Carnegie Mellon were attempting to conduct a process called judgmental probability analysis -- that was one of the names given to it, which was to try to find a foolproof technical way of integrating the variety of opinions that the peer reviewers came up with. Now, a very talented former colleague of mine, a physicist, when he taught physics for poets or whatever that course was called at Cornell, pointed out that there is no point getting to umpteen decimal places of accuracy in combining a set of data points if one of your data points actually is not good beyond a very crude level of approximation. Similarly, sort of trying to get a highly NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. technical and highly perfected form of enclosed peer review process, if you haven't got the right peer reviewers in there, for instance to take your own example, if you have got everybody agreeing, but it so happens that they don't represent the full spectrum of informed opinion out there that should have been consulted. Then it's not going to do you a whole lot of good to insist on the perfection of that small and perhaps erroneous cross-section of the people you have brought in. So, somebody else has to be looking out for a range of sort of contextual factors as well. DR. OMENN: I wanted to raise a point that I don't think has been asked yet, which is the provision in the bulletin that peer reviewed articles should be accepted as already having passed peer review with a rebuttal possibility. A vast majority of the scientific literature of course is comprised of articles and work never addressing the issues that are salient to the regulatory agencies. In 1978, at the Atlantic City meetings I remember the then-FDA commission Kennedy laid before the scientific community, a whole array of very challenging, interesting scientifically compelling questions that were then salient to the FDA. A similar kind of agenda was built at EPA, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. especially because on the criteria air pollutants, it is required by law that the resolution of the every five year regulatory status include a research agenda, so the agency and the nation should be better prepared five years hence. A certain amount of research is conducted on that basis, sometimes under the aegis of those regulatory agencies, one of the rationales for their having strong science. But the larger scientific community typically is not motivated by that agenda. And then the regulatory process sort of reaches out to find potentially relevant articles, often at different doses or different kinds of exposures or many parameters not explored. And I think the desire of the OMB bulletin to be accommodating probably could use some nuance on this point such that there should be some phraseology about appropriate or appropriately matched to the issues at hand in the regulatory decision-making would be necessary in order to just accept the previous peer review. MR. ANDERSON: Thanks. Yes? MR. BYRD: My question is for Dr. Scanlon. My name is Daniel Byrd. I'm from LSRO. As I understood your discussion, the point of view NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. at HHS is that panels such as we have at NTP, to which NTP delegates decision-making authority and they vote, will be considered peer reviews by HHS under the OMB bulletin. Am I correct in understanding you that way? MR. SCANLON: I think my major point was that we have a variety of peer review mechanisms, and scientific review mechanisms, some internal, external, and some mixed. And the NTP was an example of a mixed extensive scientific review and peer review process. We'll be talking to OMB about how that all shifts out, but I think that would be our position, yes, that's peer review. MR. ANDERSON: I take it then that no one is at a microphone, that either you are very hungry, or you have a plane to catch. Lunch is in the Great Hall. Thank you. We will resume at 1:15 pm. [Whereupon, the meeting was recessed for lunch at 12:16 pm, to reconvene at 1:15 pm.] NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. AFTERNOON SESSION (1:18 pm) Agenda Item: The Boundaries of the Bulletin: Scope, OIRA Involvement - Alan B. Morrison, Public Citizen Litigation Group MR. MORRISON: Since I'm the only person on the panel, and since no one is here to introduce me, I'll introduce myself. It's not clear why I'm the only person on the panel, but I'll leave that for others to decide. I'm Alan Morrison. I'm a member of the Program on Science, Technology, and Law, and I'm with Public Citizen Litigation Group in Washington, DC. I'm here to talk to you today about actually more of the nuts and bolts of how this bulletin is supposed to work, some of the things that are covered, and not covered, and many of the questions that will arise. I want to say I'm going to take some positions here, and in some cases only because it seems like what the words seem to say. But I'm also going to ask some questions about whether on the theory behind the bulletin, it ought to cover other things, or things should be exempted that are not exempted and so forth. Let's begin with who is covered. Dr. Graham mentioned earlier that an agency is covered if it is within NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the Paperwork Reduction Act definition of an agency, which is very broad, of course. It includes all the independent agencies, the executive office of the president, not the president and his immediate staff. It is roughly analogous to an agency as defined for FOIA purposes. The Paperwork Reduction Act insofar as it gives any authority, doesn't appear to be directed toward this kind of activity. And so, one wonders, as Dr. Graham wondered, how they are going to get control over the independent agencies. But in all likelihood, they are not going to be the agencies that are going to have principal responsibility for the kind of information that is subject to peer review here. The next question then is what information are we talking about? We are talking about regulatory information, which is defined as scientific or technical studies that are relevant to regulatory policy. Notice the last word, policy; not just rulemaking, but regulatory policy. It's obviously broad. The word "study" is very broadly defined in the bulletin as any research report, et cetera. Now, that gets me to the next question, however. The terms "scientific" and "technical evidence" do not appear on their face, to be very broad. Dr. Graham said NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. this morning that they covered social science information, by which I presume he meant economic information. But if you look at the data quality guidelines and the statute, that statute is very broad. It says any kind of information, including statistical information. So, if you look at the two together, and then examine scientific and technical, the natural reading of that would be, particularly since they came out of the same office, OMB, you would think that they were intended to be different. And if OMB does not intend them to be different, I would want to know why economic information should not be covered. And if they are intended to both be covered, then they ought to be done more clearly in this particular bulletin. Further refinements, they have in addition to information, if it is significant information, it is significant as defined as influential under the Information Quality Guidelines. Now, the same part of the definition also defines the word "relevant." And information is relevant in a regulatory context if, "it might be used," by any regulatory body, federal -- yes of course -- state, local, and international regulatory bodies. Obviously, the word NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. "might" is very broad. The federal has to have no control over it as long as it is federally disseminated information that might be picked up somebody else. Interestingly, the definition does not include information that could be used by private conduct, by private parties to make important decisions. The data quality regulations do pick that up, but interestingly enough, as we will see in a moment, that may come in through the back door when you get over to especially significant information. Now, this morning several people raised the question about whether science that is produced by NIH or the other non-regulatory scientific agencies is subject to the bulletin. The answer seems to be yes and no. No, because it is not regulatory policy when they produce it. The problem is that if any regulatory agency wants to use this information, it has to have been peer reviewed before that agency can disseminate it and use it as a part of the regulatory process. So that potentially, if you don't protect yourself on the front end and see that your information, you being the regulatory agency, the information that you are going to be using is peer reviewed or excluded from peer review on NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the front end, you may not be able to use it, and you may have to go through that peer review process yourself. At least, that's how I read the regulation. There is also a definition of dissemination, but for these purposes it is not particularly significant, because you don't have to worry only about what is going to be used at the front end when the report is produced, but what is going to be used at the back end. And so, almost by definition you have to assume that it's going to be disseminated and take appropriate precautions, if I may use that terminology. Section 2 then goes on to require peer review for significant regulatory information that is intended to be disseminated. The standard here is what seems to be a quite sensible one, which is the level of peer review depends upon all of the relevant factors, and that it does not have one size fits all at this stage at least. And that seems to be a sensible approach. However, there are a series of exceptions, and I want to talk about them now. It is not required for the agency to do peer review before it disseminates this information so long as there has been prior peer review by an independent scientific journal. That authority to use NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the independent scientific journal peer review is a rebuttal presumption. The difficulty is the bulletin doesn't explain on what basis the presumption can be rebutted. Does it mean that the scientific journals have to use the same standards that OMB insists upon here? Because if that is the case, and I understood Don Kennedy correctly this morning, the scientific peer reviews would fluke every one of these OMB tests right off the bat. So, what does it mean to have an ability to rebut it? And how are we going to have a proceeding under which this rebuttal is going to take place? And who is going to referee the proceeding in which it occurs? These are questions which are simply not answered in the bulletin. The press release accompanying the announcement of the bulletin, the draft bulletin I should say, says that you get the exception if the journal is a respected scientific journal. I don't know what a disrespected scientific journal is, and it rather harkens back to those lawyers in the room to the Frye debate before we had Daubert in which the question was whether there was consensus. So, maybe if there is consensus that it's a good journal, it is okay, if not, it's not. I would hate to be in the room when that NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. debate is going to go on. Okay, what else do we have? Well, we have a large exception for national defense and regulatory affairs. I kind of thought that science mattered in national security affairs, and foreign affairs as well. I mean wouldn't you like to have had some good science about for example, whether there were weapons of mass destruction in Iraq? Or didn't that matter, because nobody was going to be regulated by that? It seems to me that maybe that is a little harsh. We ought to have some ideas about regular alternatives. Of course, perhaps maybe the data quality regulations could pick those up, and we could file a challenge to the information that the Department of Defense put out on whether there were weapons of mass destruction in Iraq. Now, I want to talk next about a series of exceptions that I consider to be extremely open-ended, questionable, perhaps even problematic. And that has to do with the exceptions for agency adjudications and permits. And I'm going to consider them together, although it is not clear that there is not some overlap between them. If one reads the Administrative Procedures Act, it defines agency adjudication as a type of action taken other NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. than rulemaking. And basically, everything else an agency does is an adjudication. There are formal adjudications. The FDA used to conduct a few of them from time to time. They got smart, and they didn't do it any more. The Federal Trade Commission has formal adjudications. There are other agencies, the SEC, that have formal adjudications. It doesn't say anything about formal adjudications which are like court proceedings with cross-examination and so forth. Everything else is still called an adjudication. It is an informal adjudication. That runs from Social Security determinations to the determinations about whether you are going to grant money for a highway permit through the city of Memphis, as in the case of Overton Park, which all administrative law students will have studied. Everything there is an adjudication of one kind or another. To make it clear though that this is a broad exemption, there is also an example for permitting. We know what permitting is for EPA. When you get something, it is called a permit. But what about -- and the question was asked this morning by the woman from Covington and Burling - what about new drug applications or PMAs for medical devices? Are those permits? They are in my view, either permits or they are NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. informal adjudications. But they are sure as heck scientifically based, at least I hope they are scientifically based. And one wonders why those should not be included. Surely, they are of very great importance. The current battle going on about whether silicon gel breast implants should or should not be allowed back on the market for reconstructive purposes is a matter on which science is extraordinarily relevant. Why should that information not be subject to the same kind of level of scrutiny? I'll get to another problem with it in a second. That is true when agencies are doing other things, simply because it's not part of either regulatory policy, or part of rulemaking. All right, so far we have been talking only about peer review in the context of significant regulatory policy. Now, we want to get to the very special or very significant regulatory policy. And this is defined as the impact of $100 million over a year. All right, we understand that. Or the dissemination could otherwise "have clear and substantial impact on important public policies" -- and then this is the important addition, "or important private decisions." We now suddenly in the especially significant, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. come back into the back door, what I thought was taken out in the front door. That is, by the definition it appeared to only talk about regulatory policies. This now allows it come back in the back door in affecting private conduct. Which is of course consistent with the data quality regulations, but not consistent with everything else in this bulletin. Then on top of that, if you didn't think that was enough, it could come under especially significant. OIRA gets the right to designate anything that it says has significant interagency concern or is relevant to administration policy priorities. Translated that means anything else that we say it is, can go through this especially significant. All right, besides have a nice label to it, does it have any consequences? The answer is yes. This is the place where one size fits all no longer applies, because if you are a lucky winner in the especially significant regulatory designation category, you get the following bonuses. First, you get a written charge to the peer reviewers. The second thing you do is there is a prior public comment on the study that is being peer reviewed. I NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. want everybody to be clear about what I understand this to be. The study is done. It goes to the agency. The agency then puts the study out for public comment. The public comments come in, and they go to the peer reviewers, who then review it again. The agency comes back and does it again, puts it in final form, and then presumably puts it out as part of a regulatory analysis or a regulatory part some time. This is, I suggest to you, a potential for serious time delays, not to mention costs and everything else, and we will hear in the next panel, I hope, how the peer reviewers, and ultimately the regulators are going to react to this kind of interruption in the middle of the regulatory process, where there is going to be a lot more public participation at this stage directed toward in essence, telling the peer reviewers how to peer review the document that they are going to be put before. Third is there must be details about what must be in the final report. The agency must certify there is compliance with the peer review, and required to be put in the administrative record. And it must consult with OIRA along the way. Section 4 of the bulletin goes through some NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. procedures. Sid Shapiro will talk about the implications for FACA this afternoon, a very important topic. There are also, I noticed, the authority for OIRA to waive for health emergencies, which is obviously a good thing. And also a waiver for homeland security. Once again, I sort of thought science mattered as to how much we are going to be burdened at the airports, and how effective these various measures are going to be for which people are spending lots of money. Maybe it's taxpayer money, maybe it's private money, but somebody is spending it. Why shouldn't we have some good science with regard to that, as well as on other things? There is a requirement for annual reports. And if you didn't know about it before, Section 8 makes it clear that OIRA can request comments from other agencies about your own peer review, and can make those public, and require various things to be made public as well. I want to talk about the next thing, the effective date. I want to ask a question here. How many people have read the effective date provision? All right, it says January 1, 2004. All right, we all understand that really it's not going to be January 1, because the agencies don't have until January 15. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. What I don't think many of you recognize is that it is reflective with respect to disseminations made after the effective date. Which means, as I understand it, if you did a study four years ago, it is final getting done, and it's all completed two years ago, if you don't get around to disseminating it until after January 1 or whatever the effective date is, you've got to go back and do all the peer review on top of it. That's my understanding of dissemination. If there was ever a formula for ossification, that is surely it. Perhaps OMB doesn't mean it, but that is what it literally says right now. Tucked in Section 7, three sections from the end, is something not relating to peer review, but goes back to the data quality regulations last year, one could say general subject, I would say different. In any event, it's sort of buried in there, and it in effect, makes a change to the data quality regulations when we are barely beyond the year in which the regulations have been in effect. The annual reports from the agencies have not yet come in yet, but OMB is making the change. And it does two things. One, it says that within seven days of the receipt of a non-frivolous request for NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. correction, you have to either send it to OIRA, or put it on your Web site. I would suppose that every agency would put it on their Web site, save for one thing. By putting it on your Web site, do you concede that it's non-frivolous? Or can you put a disclaimer up there saying we put it up there, and we think it's frivolous, but we put it up there anyway. Can they ask for a correction because you now said it's frivolous, when you actually don't think it's frivolous? In any event, the OIRA also can also now require agencies to show to OIRA, draft responses to requests for correction. And they have got to give it seven days in advance. And if OIRA says you may not respond, you may not respond until OIRA says it's okay. And OIRA has the right to send the comments over to OSTP for its views about whether the agency's response the request for correction is a sensible one. If you read the background paper, the background materials in front of the bulletin, it is what I would call a fairly damning criticism of the administrative agencies and their use of science and peer review. The charge is that many agencies don't use it. The agencies that have it don't use it, even if they have it. The people they get are NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. financially and otherwise dependent upon the agency in all kinds of ways that seem inappropriate. There is a narrow focus to the inquiry of the peer reviews. The results of the peer review are not available for public scrutiny. And the effect is that peer review is, as it is alleged here, created in order to support the agency, rather than to get independent judgments. As I say, if that were true, it would be quite damning. And if they are true, is this the appropriate response? There is no mention any place in this bulletin about cost and delay. One would think that for an agency that is concerned about cost-benefit analysis, one would do a cost-benefit analysis of the burdens imposed by this bulletin. At the very least there are going to be delays. Perhaps they will be saved at the end for having better science, but perhaps they will not be. There are obviously external costs to pay the peer reviewers. There are going to be internal costs whether they appear on the cost sheet or not. And moreover, there are in the bulletin and the description, no explanations any place as to the reasons for these exemptions, the reasons for the exceptions, and are not at all consistent with some of the purposes behind the attempt to have good science. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. One question at least OMB I think ought to ask, and it is interesting, because it seems to be there in the basic peer review is whether we need to have one size fits all, or whether we could have in the areas that are now exempt, some different kind of peer review, not the same as we would have for regulations or let alone for especially influential regulations. Could there be gradations of peer review? Rather like when you go to a restaurant and you order a steak, and they say to you would you like it well done, medium rare, rare, or pink? Could there be something along that line as well? The last point I want to raise is think about the general criticisms of the science and the peer review that OMB has made of the agencies, at least implicitly. Take the word "agency" out of all of those criticisms that I gave you just a moment ago, and substitute instead the word "industry," and ask yourself, do we or should we have peer review of industry documents? Could OMB mandate that? Well, of course in one sense of the word it cannot mandate what industry or groups like Public Citizen or the environmental groups, the workers groups, or anyone else submits to federal agencies. On the other hand, could they NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. start to establish some sort of hierarchy such that when non-peer reviewed, non-transparent information is submitted to an agency, the agency can say, look, our information is peer reviewed. We have had transparency. We have independence. We have all these comments out there for everybody to see. You, the industry, whether it is in a permitting situation or in a rulemaking situation, have not done the same. And therefore, we do not have to give your kind of submission the same kind of deference that we think that we give the agency submission that is peer reviewed. I offer that as a suggestion, as a means to try to accomplish what the stated goal of this is, which is that we have good science. Nobody in this room is in favor of bad science. The question is, how do we go about doing it, and at what cost? And it seems to me that one of the concerns that I have, not reading so much the particular words, but the overall ambiance of this bulletin is that it is biased against agency science, suggesting that it is poor quality, and that by interference, other science which the agencies are receiving, often in contrast to what they are putting out, but that is somehow good science, even though there has NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. been no peer review of that as well. Thank you very much. Agenda Item: Questions MR. MORRISON: Questions, comments, objections? MS. CARROLL: Beth Carroll with Syngenta. I would like to know what is your definition of policy? And could it be any regulatory action that would set precedent? MR. MORRISON: That would set precedent? MS. CARROLL: Yes. MR. MORRISON: I'm not even sure it has to set precedent. It seems to me if the agency does something which has an impact on somebody else, it could be policy. And my concern is not at that end. I think that the concern is, as I think the concern was properly in the data quality guidelines there, that government information can have influences. The question is what we ought to do about that fact, and that's my concern. But as a general proposition, I accept the notion that government disseminated information does have an impact, even if it's not regulatory in nature. PARTICIPANT: Hi. I was listening. I wanted to understand what you are saying. Did you say this bulletin NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. is badly written, and that it ought to be more broadly applied with fewer exceptions? MR. MORRISON: I said it was unclear in a number of respects. I questioned whether the exceptions were justifiable in terms of some of the purposes that are stated behind the bulletin. I questioned in some respects the need for some of the provisions here. I don't think my comments were exactly what you said, but if you want to refer to a specific part, do I think they should go back and start again? Yes. PARTICIPANT: Well, I had one other one. And that is when industry promulgates this stuff, we call it advertising. And we all know what we think about advertising. MR. MORRISON: Well, I wouldn't go so far as to say it's advertising. It's advocacy, and they should be proud of it, as I'm proud of it. I try to be an advocate, but let's be open about it. That's all I'm trying to say. PARTICIPANT: But if this were done, would we have a better feeling about the government puts out than we have about what industry puts out? MR. MORRISON: Well, I would hope we have a different feeling already about it. Some people have a NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. different feeling one way, and some people have a different feeling the other way. MR. HEINICK(?): Hi, I'm Steve Heinick with the AAMC. Would you mind addressing this question of judicial review? Do you think it creates an expectation? MR. MORRISON: My own view about it is, and this is not based on anything here, but it is based upon my views, which I stated publicly in connection with the data quality regulations. To the extent I heard Dr. Graham this morning say that this is management only. My experience is that courts do not pay much attention when agencies say we are doing this for a management purpose only. OMB says it's for management purposes. If you are supposed to do it, and you don't do it, you are increasing the risk of having an adverse outcome on judicial review. My view is that if this kind of information arises or is submitted in connection with a rulemaking proceeding, which is let's say the paradigm of this, and let's talk about that first, the fact of noncompliance, if there was non-compliance with the OMB guidance, would certainly be something that would be called to the attention. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. I don't know any federal judge who is going to say I am going to completely disregard that. It is probably not going to be enough to get the rule overturned, but it's going to be a danger signal, and perhaps it ought to be. That's my reaction. Now, the second question is what about non-peer reviewed material that doesn't come up in the context of a regulatory rulemaking? It seems to me there that the likelihood of judicial review depends upon the likelihood of judicial review under the data quality regulations. And as I think some of you know, there is a considerable difference of opinion as to whether the failure for example to do corrections is judicially reviewable or not, assuming somebody has standing, and it is the kind of thing as to which there is not simply differences of opinion, but actually factual differences. And there is clearly no answer yet, and I'm not sure there will be one for a while, but that's my view about it. PARTICIPANT: Do you think under the Administrative Procedures Act it is possible to give levels of weight and credibility, as you suggest, to data, depending on whether it's been peer reviewed, how well it's been peer reviewed from whatever source? So, could you NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. establish criteria by modifying this regulation that basically establishes kind of a gold standard for the kind of scientific evidence that is being admitted by all source, government, industry, et cetera, so that you would have an A, B, C, D, and that would pass muster both under the APA act and under Chevron? MR. MORRISON: Well, I didn't mean to say, and I don't think I said it, but I don't want to quarrel about that is that there would actually put forth a standard in the same way like a rule. What I meant to say that it would be appropriate for agencies to announce as a matter of policy that we are going to give greater deference to peer reviewed information, than to non-peer reviewed information. And in determining within peer reviewed, we are going to look at some various factors, including the level of transparency, who the peer reviewers were, and so forth and so on, just as the agencies' conduct will be reviewed here. And the answer is do I think the agency would be upheld by doing that? Absolutely. PARTICIPANT: We were a little bit concerned when we were reading through this about the regulatory rulemaking, sort of broad thing, an internal agency rulemaking about its own operations. It doesn't sound like NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. what is meant by regulatory decisions, which you think sort of involve telling other people what to do? But it seems like it's the whole enchilada could possibly be included in this. MR. MORRISON: If by internal do you mean for example, the agency wants to promulgate new FOIA regulations? PARTICIPANT: Right. MR. MORRISON: I don't think that there are reports or scientific studies that would come within regulatory information that would likely kick in, in that situation to begin with. But I don't want to say never, but I think the answer is it would probably be not covered by it. PARTICIPANT: Well, and then that brings up the decision. Who decides if science should be used? MR. MORRISON: Well, in the first instance, I suppose the agency has to decide whether science is necessary for their decision. If the question is how many copies of something you have to file in order to be treated as a complaint with the agency, and who you have to send it to, I don't think you need science to justify that. The agency can just simply make a rule. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. But in the end of course, if somebody thinks the rule is arbitrary, they can take you to court and say you should have had science to determine the rule. And that's ultimately how it is going to get there, assuming it gets there at all. DR. ASHFORD: Nicholas Ashford, MIT. A couple of questions. In your view, what would the answers be to this question -MR. MORRISON: Can I claim the Sheila Jasanoff defense right away? DR. ASHFORD: That's fine, Alan. What was wrong with the prior decisions that this bulletin purports to cure? Secondly, with the stacking or the removal of experts from study sections that do their own peer review of science which is done by both regulatory and non-regulatory agencies? And third, isn't the real damage here not what might happen to any particular study, but the enormous chilling effect this has on the agencies for putting anything forward that isn't absolutely water tight? That is, I would argue the chilling effect this has on the agencies is far more serious than the fact that a particular review might not be viewed in a certain light. But I have given you three things to comment on. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. MR. MORRISON: Well, I know a softball when I see one. The third answer is yes, I agree with you, although I also agree that about time and cost of money. As to the first one, I was interested to hear, and I would be interested to hear what OMB said in response, that the gentleman from EPA said this morning, we are doing peer review just fine. Everything we do now would go under the bulletin. The question is does OMB agree with that? And the second is if that's the case, how come people are so much complaining about their science? Not just the results, you are partially right about the results, but about the science. I'm sorry, the second question again was? DR. ASHFORD: It has to do with the stacking of agency science groups. The removal by the White House of people from actual groups that fund research and interpret research, and how does that square with a fair peer review process? MR. MORRISON: I'm not sure that I know enough about the process you are talking about to comment on that. MS. CASANO: Pat Casano, General Electric. If I understood you correctly, it seemed to me that you indicated that you are at least concerned about NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. allowing public comment on a product that is about to be peer reviewed before the peer review occurs. It seems to me that there are probably at least two kinds of public comment. There is scientific comment, and there is non-scientific comment. And scientific comment would be things like the agency missed this batch of studies. Or the agency is citing a study for a proposition that it does not support, that kind of thing. It seems to me that if you don't allow at least scientific public comment, the peer review panel might never hear those sorts of things, because they are only going to have the agency's view of the science. So, I'm curious as to whether you think that kind of public comment in advance of a peer review is appropriate? Or if you think that there are other avenues for getting that kind of information to a peer review panel? MR. MORRISON: Well, first let me say one of my concerns was not that there was public comment as such. It was partially the question of time. A second concern was the impact upon the persons who are doing the studies, to have their studies in draft form, about which we hear so much from agencies, nobody wants to see a draft commented on publicly, is going to be commented on publicly before NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. outsiders review it, to see whether it makes any sense. Third, I don't know how you can separate out scientific comments from non-scientific. Do you have to have a PhD to be entitled to submit comments at that stage? I don't know how that would work. The last thing I would say, and this is maybe off base, but let me suggest it anyway. If part of the problem in these studies may be the question about methodology and what you take into account, that is things that go into the protocols at the front end of the studies, it may make more sense, particularly for these especially significant studies, which OMB is going to get hold of the plans for, because the agencies have to tell OMB about this in advance. For those plans to be vetting in public in a general way. And for people to say look, these are some things that the study design ought to take into account. That that would be an appropriate front end thing. It would not delay things unnecessarily. And the peer reviewers could look at those kind of comments at the end, after they get done with the study. Anyway, it was just a thought. Don. DR. KENNEDY: Alan, I need some legal guidance over the complicated terrain between the Data Quality Act NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. and this bulletin. Suppose for example that a Web site maintained by let's say the National Cancer Institute, asserts a changed relationship one year as compared to a previous year on let's say the relationship between abortion history and liability to breast cancer. Presumably, those who are concerned with the change could alternatively attack it either by alleging that the data is of bad quality, or under this bulletin, after we have reached the implementation date, on the grounds you didn't do peer review. MR. MORRISON: I think the answer is that both are available. And I as understood Dr. Graham this morning, what he said was that this is an amplification of what we mean by good science. And that therefore, to the extent that they were challengeable at all, they could be challenged and claimed that they were not objective, or they didn't meet the reliability or other standards, because in part they were especially significant, and they hadn't gone through peer review, because it surely "might" affect regulatory policy by federal, state, national, or international bodies. Anything else? Okay. [Applause.] NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Agenda Item: Peer Review of Especially Significant Regulatory Information: Implications for Agency Peer Review Procedures -- Changes in Practices; Costs and Delays; Pre-Peer Review Public Comment - Moderator: Donald Kennedy, Editor in Chief, Science, President Emeritus, Stanford University DR. KENNEDY: This discussion will involve several affected agencies and others, some agency alumni in a discussion of how the peer review bulletin has implications for agency peer review procedures. What is going to change as a consequence of what will, after the comment period over, issue with whatever changes are made in it. What changes in practice, what costs and delays, and so forth. And I'm very happy to be the moderator, and I don't plan to moderate them very much, as long as they keep on time. They have been allocated the choice between speaking from the podium, and from doing what I did and talking from right here. And from my immediate right, James Mahoney is assistant secretary and deputy administrator for Oceans and Atmosphere at NOAA; James Schaub, director of the Office of Risk Assessment and Cost-benefit Analysis at USDA. David Michaels, professor of occupational and environmental health and epidemiology at George Washington NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. University School of Public Health -- he's an alum of the Department of Energy, formerly assistant secretary for environment, safety, and health there; and Mike Taylor, a long time FDA colleague of mine, senior fellow and director of the Risk, Resource, and Environment Division at RFF, and a former deputy commissioner at FDA. So, they have their assignments. You know who they are. Jim, do you want to start? Agenda Item: Speaker - James Mahoney, Assistant Secretary and Deputy Administrator for Oceans and Atmosphere, National Oceanic and Atmospheric Administration DR. MAHONEY: Thank you, Dr. Kennedy. I'm glad to do it. I apologize for running in at the last minute. I was on another call on peer review guidelines, believe it or not. So, there are a few people in town who aren't here in the room at the moment, but I begged off that, saying I must get over here. DR. KENNEDY: You escaped just in time. DR. MAHONEY: I'll try to number one, keep our chair's strict guidance on time. So, I'll try to jump right ahead. While Don has given my daytime job title working at NOAA and in the Commerce Department, my focus here today NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. will be a little bit different to give you hopefully another dimension that may be of some special interest. That is, I have spent since I was sworn in going on a couple of years ago, I have spent a great deal of my time as director of the Federal Climate Change Science Program. That involves 13 agencies of the federal government doing our best combined job to deliver to the public and the stakeholder communities worldwide, our best views on the science issues related to climate change, causes, impacts, and the like. And because of the major multi-agency nature of this work, the whole series of issues raised in the bulletin and the underlying guidelines have some specific features. I want to call attention to them in my comments here at this time. We all recognize the high controversy that is involved in dealing with climate change issues, but I don't really make that unique. I think that is the nature of the controversy relative to all of the scientific inquiries of great importance that we face. But because we know we face this in this case, we are looking at a circumstance where of course the underlying OMB guidelines must be met. We will need to meet the requirements as they are finally adopted NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. for the further steps taken. We also have to meet specific guidelines of one or more of the sponsoring agencies for our work. But in addition to that, we have been on a track for some long time now of setting some very special standards relative to public involvement, public information in the processes that we are involved with. So, we want to call that to your attention for its possible reflective role in the more literal guideline and schedule of extension that is on track here. A note for those of you who have an interest in the climate science program, the Web site address is on the lower right-hand box on all these slides. So, we try to put everything on the publicly accessible site. So, if you are looking anytime, just go to climatescience.gov, and you will find references to all of this work. I will be very quick about just saying a word of background for those who might not be familiar. The Climate Change Science Program began in its modern form, I would argue, in the late 1980s, first by some budgetary exercises, and then by enactment of the US Global Change Research Act in 1990. The act set a series of specific requirements, creating a background for budgeting, and launched a series NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. of global change studies and measurement and data analysis and models, which have continued since that time, and which have consumed over $20 billion of taxpayer money in the research sponsored through those mechanisms since then. When the president took office at the beginning of 2001, he undertook to provide some additional funding for the Climate Change Program, and asked the program to take on in addition to its underlying activities, a focus on bringing forward those matters where our science might be able to give us better information to support decision analysis, decision-making in what was defined as the nearterm. And that was further defined as in the next two to five years, without any implication that the research would stop after that time, but rather simply that we would try to bring forward what could be done then. The president took another step in early 2002, by assigning to line cabinet departments the responsibility to oversee this program because of its size and scope of interagency activity. The science program is running on a budget level of approximately $1.75 billion a year at this time. For those in this community, this is kind of an organization diagram. You won't be able to read it all from NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the back of the room, but the point of this is that the intention is to provide a mechanism so that all of the research and related measurements and supporting activity, as well as all of the technology development activity flow up from the department's that sponsor the work in an increasingly coordinated way. Back through a very active board of directors, which is made up of the deputy secretaries of the relevant departments, and then on through them to the cabinet officers, and ultimately to the White House policy offices at the end of the day. In fact, on this very morning we had a regularly scheduled what I think of as a board of director's meeting, which runs at the deputy secretary level. Now, in the next slide I want to come over specifically to this matter of the distributed science inquiry that is undertaken in this case. That is, within the Climate Science Program there are, as I think I said, 13 agencies and departments taking part directly. What has happened over the years is that everyone who with a Washington background would know, budgets come up through departments. They have their own examiners at OMB. They have their own appropriators. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. And the matter of really getting collaboration to prioritize programs between agencies is very difficult, because of the stovepiping of the appropriations process. We put a lot of time in the last year and a half into trying to do a better job at prioritizing and strategizing so that we can in fact get the best performance out of this kind of activity. But for our purposes here, what we want to call attention to is the matter that there are increasingly within the federal sector, cases where there will be multiple departments and agencies involved. And indeed, as far as that is concerned, various collaborations between the federal government and other government units and NGOs and the like. So, we want to call attention to that somewhat in this case. And as I say, at the end of this slide, we don't think of this as an entirely unique case, but rather something that is worth citing as far as other science and technology matters are concerned as well. Now, in the next slide, the characteristics. I think I have sort of identified this already, but just to note what we are doing in the climate science area involves the whole range of disciplines that would be appropriate. I have said here atmospheric sciences, social sciences, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. engineering, but don't forget ecosystems, don't forget decision support. Don't forget the need for communication and massive measurement systems in their own right as well. Much of the work done in this program, the vast majority of the work is done as sponsored research would be anywhere. That is, a very large component of it, as I note here, is in university PIs and other research groups, not exclusively, but in very large amounts. The government labs play a fairly large role. The government science managers of course oversee a lot of this, but the great majority of the work in fact is awarded on a merit selection basis from the beginning. However, within this what we have added to the process is the concept that we need to charge the scientific community with organizing and synthesizing its results to make them more readily available to a wider audience. And while that takes a relatively small amount of the total dollar budget, and total effort in time, it is on the order of 2 or 3 percent of the budget, compared to the great majority of the funds going into the underlying sponsored research. The delivery of synthesized information is of special importance, and we have been trying to focus that. The next slide shows a picture of something that I NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. will hold up at the same time as well. We spent the last 16 months going through a very open process to produce first a draft strategic plan for this massive climate science program. And we did everything we could think of to get useful input on that process. We had a major workshop here in Washington a year ago coming in a week or two now concerning that draft plan. We had an open comment period. We received hundreds and hundreds of comments on our Web site. We asked the Academy to convene a special committee to review both that draft plan, and the updated -- I never want to use quite the word final version, and we got that comment back. This is the plan document. They are available to any of you who would like it. There is both a full plan document, and a short summary. And please inquire if you would like to get copies of this. But what we have done on this, and it's consistent with what we are trying to do within all of this area is to say how could we get our underlying science and information out so that it can be seen and challenged, and what does it mean to do that? Now, to go along with that, as I have indicated, one of the results of the plan process is that we have identified 21 areas. There is nothing magic about that NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. number. That is the number that came out of a planning and prioritization process. But we have identified 21 areas where over the next four years this program is producing scientific synthesis and assessment reports that take the same kind of approach that we put into the overall plan, which is to be explicit about the questions being raised in the first case. Get the questions down, agree on them. Agree on an approach to addressing them. And then conduct the most transparent and open process, with the best possible commitment to objectivity. I say the best possible, because I don't know any human that can said I am absolutely the gold standard in every case. But that's the sense of play. So, what we have done with the synthesis and assessment reports, which are now begun and in process, is to get the best information out. For those of you who know the climate business, you may well know that this is done at the international level too through the IPCC, the Intergovernmental Panel on Climate Change. So, we are trying to be very careful not to waste resources or reinvent the wheel, but instead to work collaboratively with our colleagues in IPCC, recognizing, however though, that with many special interests to the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. American audiences, the sense of play is that we have a responsibility to raise questions and try to address them in ways that will help the American decision-makers, and that will also not look as if somehow the US effort is trying to co-opt or slant the IPCC process, which it wouldn't at the end of the day, but rather, that we set our own standards for careful review and objectivity. Now, for just a moment, these special assessment reports, which are begun in process already now, and they will be completed between now and at various times over the next four years, reflect a series of five goals for this program that go from historic in the sense of what do we know about past climate with paleo data and other things, on through how well can we quantify things, how well can we differentiate about future projections that may relate to different interventions that might be taken, and so forth? But these are all listed out. And in fact in this chart, as much as you may be able to see it from the room here, one or more of the 13 sponsoring agencies and departments is identified as the lead in each case. And this comes again now back to the guidelines at issue. This means that we have a need to in fact assure conformance with guidelines in a large, multi-department context. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. The next slide is just the other set of these. So, it's not other independent information. Again, I certainly commend you getting the detailed document if you want to look at that more generally. Now, having said this, I have three more viewgraphs I want to show that get to a little bit more specifically to the process that we have been dealing with. So, on the next one, the review process that we are about to publish on our Web site, we have the plan out, we have assigned the work to be done, we have dealt with the fiscal 2005 budget requests. It's mostly in now, although that still has to be prioritized. But now, we are back to trying to say to all those who are interested in the scientific and the stakeholder communities, how are we going about in fact crafting these reports, which are meant to synthesize the various science and related areas here? The guidelines exist fully in draft. These are our guidelines, which in a sense go beyond the kind of thing that is in the OMB circular at this time. But they begin with an overall statement of these principles of objectivity and transparency, which are similar to the OMB statements, because after all, it's the same kind of thing. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. But here, since we are trying to lay out specific information, the first step in our procedure involves framing the document to be produced in each case, by developing it, creating a prospectus for it, identifying the questions to be raised and addressed, identifying the authors, so that the reader can understand the range of the authors and their freedom from bias, hopefully. And then also in the prospectus, identifying the review processes to be followed. That information will be made available publicly with a comment period. We'll go ahead with the drafting and preparation. The whole document will then be subject to not only scientific peer review, but to again, public comment to any who choose to respond to it. And we expect asking the Academy or the NRC to play a role in some or all of these by way of specific review as the come along. The issue is partly that of quantity, because we have so much we are dealing with in a rather fast time frame. What are just a couple of the issues I would note about this? We have of course the applicable Information Quality Act guidelines to be addressed, the subject of this special session here. And I am operating here under a presumption that the information covered in these documents NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. generally falls in the category of being of interest and relevance for regulatory decision-making. One could argue that some may not fall in that scope, but for our purposes here, I think it's sufficient to say that we are operating within that area. But by the sheer amount of work to be done here, and I can think of other cases like in Dr. Kennedy's area and the prescription drug area, I can imagine a major review consuming all of the eligible reviewers. We have somewhat of a similar issue here that even in the review of our draft plan, we had almost 250 reviewers involved. And so, when we think of applying the guidelines in the final version that gets produced here, we are concerned about the ability to find non-biased, independent, balanced reviewers to handle as much of the simple quantity that we may need to address in the near-term, and that may call for some special considerations. We also have a concern that this may impact production schedules for getting this information out. We are trying to strike the right balance with doing quality scientific requires on the one hand, and on the other hand keep the process moving as quickly as we can while accommodating the substantial open review. So, we are NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. concerned about the timing on that, but I don't cite it as a red flag stop sign, but rather as something we simply have to deal with. The last of the slides, I wanted to focus just a bit more on this question of the selection of peer reviewers when there is a large quantity of information to review over a limited time period. So, that time period in this case being activities that will be completed roughly in the next 3-3.5 years. When we recognize the Climate Change Science Program sponsors much, the majority of federal governmentsponsored work in the climate area, there is a difficult about finding reviewers who have not had involvement in the recent past, either currently as funded or seeking funds. And that leads to some questions about understanding some matters in detail on a more explicit definition of the terms like biased and contrary biased; easy to understand in general, perhaps a little more difficult to apply in specific. And the same about the questions of the differences between real and perceived conflicts of interest, all in the range of what might be largely lawyers' definitions, but they are concerns anyway as we play it out. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. So, with that, Mr. Chairman, I finish up with the concept that we are very supportive of the moves to further enhance the peer review guidelines. And we are trying to call attention to some possible special wrinkles in the case of large scale, multi-department programs, and hope that is helpful as part of the input for the program. DR. KENNEDY: Thanks very much. Jim Schaub. Agenda Item: Speaker - James D. Schaub, Director, Office of Risk Assessment and Cost-benefit Analysis, Department of Agriculture DR. SCHAUB: Thank you. I'm honored to be here, and I'm pleased to have the opportunity to discuss the implications of this draft bulletin for agency peer review procedures. I am not a lawyer, and I'm not a scientist. My presentation reflects the perspectives of an economist who has had experience with significant regulatory actions that have been based on scientific information. The Department of Agriculture, will of course, be submitting comments on this draft bulletin to OMB. And they may very well differ from what I say today. And finally, by way of prologue, any negative tone NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. that comes across in these remarks, I assure you it is not based on an opposition to the principles of peer review. It is a belief that you have to challenge things to improve them. So, we'll see how it comes out. This next slide is the overview of my remarks. I want to quickly just show you what is in the Section 3 of the bulletin dealing with especially significant information, and what it expects from the agencies. I then want to consider a baseline for peer review in federal agencies, and go on to evaluate Section 3 subsection by subsection. This will followed by some observations on likely resource demands arising from Section, and comments on public involvement and peer review. I will close with two basic, and probably provocative questions that are asked not out of meanness, but out of a sense of urging good government. So, the next slide here summarizes by title, Section 3 on especially significant information. The first part deals with applicability. And then we have these seven subsections. And my impression is that with a couple of exceptions, those last two provisions about consultation and certification probably don't merit a lot of attention in NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. this forum, and I'm not going to say much about them. The next slide is some remarks on baseline. A good approach to analyzing to develop a baseline. If you want to see what the impacts of a bulletin are, you need to know where you start from. What is obvious, and you have heard it over and over, there is great variability across agencies. My familiarity is mostly with USDA agencies, but I know a little bit about a few others. And this inconsistency is a good reason for having some sort of guidelines to get some sort of standardization, and to justify some level of consistency. And I think it would be very difficult to prepare a comprehensive baseline for all the federal establishment, and match it up against the Section 3 provisions. Absent this centralized guidance on peer review, it appears that the following factors and characteristics of regulatory information influence how agencies use peer review. And these are not operating independently, they are operating jointly. And I have a few comments on each. With respect to controversy, well, peer review can shield an agency from some criticism. That is a good reason to do peer review, and controversy is frequent. Novelty, and I think this one is easy for agencies to recognize and NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. to drive them towards peer review, that new things merit more peer review, while old information tends to be trusted. Urgency, and this gets to some of the ideas about emergencies and such. And an urgent need to act can lead an agency to minimize peer review, yet there may be arguments to go more slowly. Scope and effect, this really is the idea that the peer review effort is proportional to what is at stake and its likely impact. Budget, and this is a big concern of agencies, and peer review is not costless, either in direct expenditures or in time and resources diverted. However, investing in peer review might result in subsequent savings. And culture -- I think it's a testable hypothesis that research-oriented agencies tend to follow academic models for business practices. And they will have well developed peer review programs. Agencies do tend to put a lot of effort into clearances, and focusing on is this consistent with policy, but that doesn't insure that the information they are putting out is good information. Now, turning to the next slide, I will step through Section 3. A preamble to this is that we have NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. already had the question raised, just what is this bulletin, and what is its affect? What is its legal standing? I'm not going to elaborate on that, but that's a critical thing for understanding how significant the impacts and adjustments will be for agencies. With respect to selection of peer reviewers, because some agencies are causal and ad hoc in their selection of peer reviewers, the criteria here, and the requirement to have a peer review plan will address two critical goals. It will insure reviewers are qualified and free of conflict of interest. A more arm's length relationship will evolve. These peer review selection guidelines will limit repeated use of individual reviewers, and spur demand for paid reviewers. Now, this is partly speculation. I think there is a question if you have significant screening out because of conflict of interest broadly defined, you may run into issues of where do you get your peer reviewers, and I think that is a concern among some agencies. The real question that I couldn't come up with an answer, and I played economist here, what happens to the quality of the reviews from this? At first blush, I think it will improve the quality of reviews overall. But as you NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. are driven away from sort of the known peer reviewers, if you get into a world where you don't know, and you have to depend on a new system to identify, select, and deploy peer reviewers, you may not get the quality that you are used to where you have had a narrow pool that you have drawn from. Another issue that arises with this selection criteria is if it does in fact drive agencies towards contracting and use of paid reviewers, how does that alter the relationship with those entities that have been doing pro bono reviews? It has I think significant implications for the sustainability of review panels that are comprised of pro bono and commercial reviewers. I think there is a potential for agencies to engage in even more interagency reimbursable agreements to insure that the demands placed on their staff to assist another agency doesn't unduly remove resources. Related to the charge to reviewers, the provision will focus agencies on critical quality matters. This is good. However, if paid reviewers become the rule, care must be taken to reconcile contract specifications and the charge to reviewers. That is, the contract specifications like compensation and deadlines, must correspond to what you expect in terms of the quality of review you want. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. The bulletin stresses peer review should leave policy determinations to the agency. I think we all would agree that that's a sound distinction. However, policy and technical issues, that is science issues sometimes are confounded. Consider that in 1998, EPA began releasing science policy documents. And there we had science and policy right next to each other related to the implementation of the Food Quality Protection Act. An example here that might help understanding it is using the one-half LOD in place of non-detects for pesticide residue presents both science and policy issues. The science question is how to best characterize the distributions of residues, while taking into account the half LODs from treated materials, and the zeros on nontreated materials. It is a modeling issue, a science issue. Yet, we have a confounded policy position to use one-half LOD for a legitimate policy purpose of protecting people. Regarding information access, there are challenges here of protecting confidential business information, and protecting an agency's deliberative process. I don't see this as insurmountable. Some agencies are very good at it, and do this with special government employees and so on. But I think there is another point on this, and NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. that gets to giving reviewers and perhaps even the public -how do you deal with proprietary models and software that really is essential for evaluating a document? I don't have an answer to that, but I think it's not unfair for reviewers to say I want to see the model work, and I want to see the inside of the model. Moving on to more implications, the opportunity for public comment. I think we have all found the bulletin may be unclear regarding the timing of public opportunity for comment. My interpretation suggests the information that has not been peer reviewed will be provided to the public for comment, and that this will be completed before a proposed rule is published. Proposed economically significant rules frequently have a 60 or 90 day comment period, and extensions are frequently granted. Thus, the public may feel that this is the norm for public comment for information. This is going to add time to the overall rulemaking process. The bulletin requires public comments to be provided to peer reviewers. Now, we have already heard this. Experience has shown that public comments on regulatory matters do not include data -- let me say often do not include data and analysis to support the positions NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. expressed in those comments. So, we are going to be delivering emotion charged or subjective opinion to the reviewers, and ask them to deal with it. Of course, making regulatory information available to the public is not costless, and subsequently assembling those comments for consideration by the peer reviewers is a resource demand. Substantive reviews may assist peer reviewers, however, the public will have an opportunity if we think only in terms of the rulemaking process, there is another bite at the apple for the public during the rulemaking process. Regarding peer review reports, the bulletin appears to favor panel reports over individual reports. Panel approaches are generally more costly, and could be problematic unless you have a good plan. And I think OIRA has that in mind, that you would recognize the need for a panel leader. While individual reviews maximize peer reviewer accountability, panel reports can maximize candor. Responding agencies to respond to peer reviews reports just makes sense. It insures that agencies do not arbitrarily dismiss comments or ignore them. But getting back to someone else who has already anticipated my remarks, peer reviews don't insure that you NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. come to closure with a matter, and we have already mentioned dioxin. You can have repeated incidences of studies. I think another issue that has already been raised, and I'll go very quickly here is what kind of deference is given to the way agencies respond to peer review comments? It appears that agencies will be granted a great deal of deference in that if they can show that they are following their legitimate authorities in making decisions and implementing rules. Now, peer reviewers are intended to be experts, and peer review is one fact identified in Daubert. We have already had some mention of Daubert. So, we have done Chevron, we have done Daubert, but those things are on agency minds. The next slide is what I'm skipping over, the consultation with OIRA, the certification of the administration record. I have nothing substantive to say there. The resources arising, and I have hinted or been explicit on these. It increases demand for paid reviewers, in my opinion. I will wrap up by saying there are indirect costs of managing a peer review process. It increases the time for peer review. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Public involvement, I will skip that; we have heard from others. But the two provocative questions, asked not out of meanness, but one is will the bulletin accomplish its goals? And this is the last slide, if we get to it, which are stated as improved quality, objectivity, utility, and integrity of information disseminated by the federal government. But there also was a statement in a news release that said the goal was fewer lawsuits and a more consistent regulatory environment. And I think it's an open question, particularly with the second point. I think we have a great chance on the first point. The second is let's consider this bulletin as if it were a regulatory action, and say is there any way in the world we can come up with measurable net social benefits? It doesn't mean they are not there. It just means it is real tough. But I think it's a reasonable question to pose, will we actually get significant net benefits? That concludes my remarks. DR. KENNEDY: Thank you very much. Now, we turn to David Michaels. Agenda Item: Speaker - David M. Michaels, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Professor, Departments of Occupational and Environmental Health and Epidemiology, George Washington University School of Public Health, former Assistant Secretary for Environment, Safety, and Health, Department of Energy DR. MICHAELS: Thank you, Dr. Kennedy. The late Rep. Morris Udall, at a late point in a meeting once was heard to say, this is the point in the meeting where everything has been said, but not everybody has said it. Now, I don't think that is the case here, because I think we have a number of distinguished speakers who are going to come after me, but I want to apologize for going over the same ground and the same criticisms that several of our speakers have probably much more eloquently discussed before I have had the opportunity to speak. I was asked to be on this panel because of my experience in the complex and sometimes byzantine world of promulgating regulation. As many of you know, the nuclear weapons complex is self-regulated. My office was the office of the regulators. I, in some ways, was the regulator. I was OSHA, the NRC. EPA actually has some ability to regulate the DOE. I also ran the Nuclear Safety Enforcement Program, and had a fairly significant research portfolio, so NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. I can look at these things from a couple of different perspectives. But I am also a practicing epidemiologist, a teacher of it. I have served on numerous federal advisory panels, and I have peer reviewed quite a few journal submissions. I'm exactly the sort of person that a regulatory agency might call on to review important scientific documents. Like most scientists, peer review plays a very important role in my professional life. In the world of publish or perish, you live or die by peer review. So, I'm also talking from the perspective of a scientist who might be brought in to do the reviews proposed in this bulletin. Let me begin with my conclusions, so you have no doubt as to where I stand. I believe this proposal is fundamentally flawed. Implementation in anything resembling its current form would serve little or not value. In the currency of the Office of Management and Budget its costs will be substantial. Its benefit, at least to the public's health and environment will likely be negative. Through delay and cost it will hurt, rather than provide benefits. And to follow-up on a question raised by Dr. Schaub, another question we can ask is are fewer lawsuits a NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. reasonable goal for the government? It is not clear that that is a particularly important goal, or goal the government should have at all. The goal should be protecting the public's health and environment. If it means more lawsuits that the government wins, that's fine. It is not at all clear to me what problem the proposed bulletin is attempting to solve. In the bulletin, and in Dr. Graham's talk today, we have not heard of widespread examples of inappropriate or flawed federal regulations being promulgated as a result of failure to peer review. In fact, we have not heard of a single example. Yet, we are today considering a proposal that has onerous and expensive requirements, but -- and this is what I will primarily focus on -- is unlikely to result in a system where important documents are reviewed by the best scientists. I'll demonstrate this by using examples from a regulatory process in which I was deeply involved, the Department of Energy's Chronic Beryllium Disease Prevention Program, 10CFR Part A50. A little background for those of you who are not steeped in beryllium lore; but there are people in the audience who are, I'm pleased to see. Beryllium was an important component in nuclear NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. weapons. In the last 1940s, a significant number of workers in beryllium manufacturing facilities or facilities manufacturing beryllium components for nuclear weapons began to develop and often died from acute beryllium disease. Even people living near the plants were getting it, and in many cases dying as well. Two Atomic Energy Commission scientists literally in the back of a taxi cab on their way to a meeting at Brookhaven came up with what seemed to be a reasonable standard for exposure to beryllium. That number became the Atomic Energy Commission's standard for exposure in 1949. It was later adopted by OSHA, when that agency came into being. And it remained the DOE standard for many years. Since being self-regulated, DOE set its own standards. This 1949 standard still survives today at OSHA, although we think of it as sort of dinosaur. Although strengthening the beryllium standard has been on OSHA's regulatory agenda for years -- though no longer is -- I don't know of a single scientist who believes it is an adequate standard. It has not changed. It became clear though a number of years ago that workers exposed to levels far below the standard were developing chronic beryllium disease. By the early 1990s, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. the Department of Energy started the process which resulted in DOE issuing a rule reducing the level at which workers could be exposed by a factor of 10, from 2 to 0.2 micrograms per cubic meter. So, I want to go through the exercise of applying the proposed OMB peer review bulletin to the multi-year process of promulgating a new standard. So, the first question is what documents would have to undergo formal external peer review? The clear answer in terms of general category, as you heard from Alan Morrison this afternoon, is anything that relates to a major regulatory action, an important public policy, or anything else OMB thinks should be, particularly if it has interagency interest, and needless to say, beryllium has great interagency interest, because the Department of Defenses also has people exposed to it. The Department of Energy started studying beryllium exposure and disease in the early 1990s. There were numerous workplace surveys and inspections involving data collection -- and here is the key word, that's what links to the quality act -- dissemination. The studies looked at methods to measure exposure, methods to decontaminate buildings, the applicability of NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. medical screening tests. All of these reports were disseminated publicly, some in a widespread manner, some to small groups, like the workers involved. Some of these reports were distilled and synthesized in annual reports and other summary documents. Many are discussed in the regulation that was promulgated in the regulation's preamble. We started a beryllium disease screening program, which also issued reports that were all disseminated. All of these influenced the regulation. Should these all have been peer reviewed? They all fit the definition of significant regulatory information. The department was keenly aware of the need for outside review and comment. We issued a formal request for information in 1996, and then convened an advisory committee under FACA rules that had numerous public meetings. And of course, we had a notice of proposed rulemaking, and actually several comment periods, several public hearings, lots of comments we went through. A few of our most important studies were submitted to scientific journals and received peer review, but certainly not all, and not necessarily in time for the rulemaking process. But this clearly would not have been NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. enough under the proposed guidelines. We would have to have a different system for review in all of these supporting studies. Putting aside the issue of cost, and as many of you know, a DOE contractor doesn't open a file cabinet without charging for it, so the idea that this has minimal cost is ludicrous at best, I have no idea who we could have chosen to peer review these documents, since every single beryllium disease expert in this country works either fulltime or as a consultant for DOE, the beryllium industry, or in many cases both. I'll come back to this. Let me start with a brief parenthetical aside here though. I want address the issue of deference to studies that have already had independent peer review, presumably through publication in scientific literature. And this has already come up several times. Again, I can use an example from beryllium, although the basic story will be familiar to many of you who work in other areas. There is little debate in the scientific community that beryllium is a carcinogen. There have been numerous animal studies demonstrating its carcinogenicity, as well as several well conducted human studies, all published needless to say, in peer reviewed journals. The International Agency NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. for Research on Cancer has formally categorized it as a known human carcinogen, as has the National Toxicology Program at the National Institute for Environmental Health Sciences. Earlier this year scientists hired by the beryllium industry published a re-analysis of the study done by CDC scientists, a 10 year old study, by the way, in which by changing some parameters, the statistically significant elevation of lung cancer in beryllium exposed workers was no longer statistically significant. And it was published in a peer reviewed journal. Never mind this is a journal that doesn't publish epidemiology, but it was a peer reviewed journal, and it came to the opposite conclusion as other peer reviewed studies. The beryllium industry is now promoting the study as evidence that the National Institute for Environmental Health Science and the International Agency for Research on Cancer are wrong. Now, while the beryllium industry study was peer reviewed, an IARC monograph is not peer reviewed. And it's my understanding that we would not be able to rely on an IARC monograph, even though it has a currency in the world that is quite elevated, because it has not gone through peer review. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. It would be laughable, but you can imagine the mischief that could be caused in the type of system proposed here, especially if the beryllium industry found a sympathetic ear at the Office of Management and Budget. Now, this is not an isolated example. There is now a whole industry that has sprung up that re-analyzes data to make the results go away, and then usually publishing the results in peer reviewed journals. Not surprisingly, these re-analyses -- I don't like to call them studies -- are usually commissioned when regulation appears on the horizon. The companies that do this work are hired guns for dirty companies. They have the same relationship to epidemiology and toxicology as the Arthur Andersen Company has to accounting. But their work gets peer reviewed and published in second rate journals, but it is still peer reviewed. My point, journal peer review doesn't make our tasks any easier. In fact, blind reliance on it may make the agency tasks more difficult, or at least more confused. Now, going back to the reams of studies that the DOE is producing and turning out, who is going to peer review them? And again, putting aside the issue that if we need to hire someone who isn't working for us already, it's NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. going to be hard to find. But selection of which peers, which scientists to do the review will have a huge impact on the outcome. The first issue here is the conflict of interest restrictions. It's absurd to say that if a university scientist gets NIH funding, she can't be on the panel reviewing say a CDC study on childhood lead poisoning. But you can be on the panel if you work for a company that is directly impacted by the regulation. You can even be working on lead issues for that company, as long as you haven't taken a public position on it. That's a literal reading of this bulletin. The components of the bulletin which suggest that academic scientists are more beholden to the funding agencies than corporate employed scientists are to their employers is on its face, ludicrous. Most university scientists I know are public spirited, and willing to devote a reasonable amount of time to assisting the government in issues of science by serving on study sections or federal advisory committees. But the job envisioned by these guidelines, peer reviewing reports, many of which contain no new science, but consist of analyses and re-analyses of fairly mundane data, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. or synthesize already published studies is drudgery. Many university of scientists, especially the more junior ones, are under pressure to publish, and won't want to serve as reviewers unless they are paid very well, making this an extremely expensive endeavor, and couldn't afford to even if they wanted to. So, which peers will actually do the reviewing? I predict two types of scientists will predominate, and both are problematic. The first, scientists whose employers see it worthwhile to pay their salaries for the time spent reviewing. Needless to say, if an employer thinks they are deriving a benefit for their employee's work in peer review of government publications, we are probably facing a conflict of interest here that should preclude that employee's participation as a peer reviewer. These guidelines, however, fail to address this issue, and this is clearly a fatal flaw. The second category of reviewer is a contractor. Once agencies start using some of the usual contractors for peer review, putting peer review into the support services budget, paying at the same rate that they pay for other services, there will be no shortage of contractor-scientists willing to take this on. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. We'll start seeing full-time peer reviewers, probably people incapable of supporting themselves actually doing science. Are these the peers we are going to be relying on? A scientist whose primary work is peer review probably no longer can be characterized as a peer. Further, there is no reason to think there will be any consistency in the peer reviews, especially if you have many documents under review by many different reviewers. And you need to have many reviewers, because you don't want to have one set of scientific biases replicated over and over. As every scientist who has had a submission turned down by one scientific journal, and then accepted by another knows, success in a peer review system has a lot to do with luck of the draw. Whoever gets the paper to review has a huge influence on its fate. Scientific peer review is not a perfect system, we just don't have a better one. It works for journals. It's not clear here it will work for regulation. And finally, I have to say the obvious. I am a strong supporter of peer review, but this proposal looks to me in the way it is constructed, as an opportunity for mischief to be made. I am troubled by OIRA's attempt to NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. define itself as the arbiter of what is good science and how peer review is performed. Outside of certain defense and intelligence agencies, there are few offices more opaque than OMB. Let me be clear, this is not a criticism of this administration alone. I believe that under Dr. Graham, OMB has become less opaque than in the past, but it is still the black hole of regulation. I call your attention to a report recently issued by the GAO entitled, "OMB's Role in Reviews of Agency Draft Rules, and the Transparency of those Reviews." It came out September 2003. One example is telling. The EPA proposed listing manganese in the rule on hazardous waste reporting. The steel industry, among others, filed comments with EPA. EPA considered these comments, decided to maintain the listing of manganese. The same industry representatives then went to OMB, and behind the scenes got manganese de-listed. There was no public process here, no transparency, no discussion of how the discussion was made. It may be a legitimate decision. It may be made for budget reasons. It may be made for policy reasons, but we don't know, and OMB has stated they don't have to tell. Again, this is not a NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. criticism of the current OMB. It's all of OMB. So, it's not surprise we are sitting here in the auditorium of the National Academy of Sciences. For 150 years, when the federal government needed help on tough science issues, they turned here. This proposal is badly flawed and potentially dangerous. It's too complicated, and it's too important an issue to try to fix through the notice and comment secret system essentially. I believe that the people who really can help address this issue is the scientific community. We can help figure out if there is a problem that needs to be fixed. We can help define the scientific threshold for which peer review is appropriate. We can help construct a system to identify which things need new peer review, and which ones don't. We can design a system that doesn't dumb down peer review by making it attractive only to scientists who shouldn't be doing it. This isn't new to the Academy. They dealt with this issue over and over again. I call your attention to the red book, which in the first page said what was the purpose of this study issued 20 years ago? To strengthen the reliability and objectivity of scientific assessment that forms the basis for federal regulatory policies. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Dr. Graham, if you are listening to this, declare victory. You have raised the issue of peer review as being important. We all agree. Withdraw the bulletin. Engage the scientific community. Engage the agencies in an open, transparent process, and let's figure out a way to do peer review in a way that doesn't hurt regulation. Thank you. DR. KENNEDY: Well, I told Anne-Marie she should have given me a gavel here. I didn't succeed in keeping anybody on time. So, Mike, it's all up to you. Agenda Item: Speaker - Michael R. Taylor, Senior Fellow and Director, Risk, Resource, and Environment Division, Resources for the Future, former Deputy Commissioner for Policy, Food and Drug Administration MR. TAYLOR: Don, I will be redundant, but brief. And I do think we have to give Dr. Graham for the consistency seriously of his commitment to peer review. He's getting a heavy dose of it today, and I do think it will help the outcome. I would like to have had my remarks peer reviewed myself, but I didn't have time. I had wanted to touch on three topics, one of which has been addressed. That's the need to define the problem being solved. I'll say something brief about that. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. The second concerned just some reflections from my vantage point as an FDA alum, but not speaking for FDA about the role of peer review in public health regulatory decisionmaking, again, focusing on FDA and FDA's responsibility to insure the quality of the science it relies upon. And then I will pose some questions about the potential unintended consequences of the proposed bulletin. I don't have the advantage of knowing what really underlies this document, but I do think it is striking that it offers peer review as a solution, but it doesn't define the problem that it sets out to solve. It cites the value of peer review in insuring the reliability of scientific analyses, and we all embrace that idea. It says the agencies have not always followed their own peer review processes, but the proposal does not provide any examples, as David mentioned, of cases in which the lack of adequate peer review has had important consequences for the regulatory process or society. Now, I don't doubt that such cases exist, and I assume that there are specific cases that are of concern to OMB. But I'm equally confident that each case arises in its own complicated legal, factual, and procedural context, with competing values and interests at stake. And whether the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. process worked well or not will be in most cases, a matter of judgment and value-based opinion. My point is this, the proposed peer review bulletin is another in a long series of procedural requirements that OMB and Congress have established to govern the process through which the regulatory agencies carry out their statutorily mandated missions. Like the regulatory requirements the agencies impose on the private sector, OMB's procedural requirements confer on the regulatory process both benefits and costs. Knowing what the benefits and costs are requires analysis. Analysis of the costs and benefits of this new requirement should precede its adoption, and the analysis should begin with a definition of the problem being solved. Thus, my first suggestion for OMB is that before moving forward the peer review bulletin, it should provide specific examples of cases in which it believes the proposed peer review requirement would have improved the outcome for society. With these examples in hand, OMB, the agencies, and interested members of the public can analyze benefits and costs, and one hopes devise a solution or more likely a set of solutions well tailored to the problem, and NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. reasonably assured to confer more benefits than costs. Next, I would like to make a point about the role of peer review in public health decision-making. And I would like to suggest why OMB should be very cautious in imposing any particular model of peer review on agencies like the Food and Drug Administration. As the OMB draft bulletin points out, peer review is enshrined in the culture of American research science as the vehicle for insuring that the private work of individual researchers is thoroughly reviewed by scientifically qualified peers before being added to the open public literature on which other scientists will rely. In the context of academic research, there really is no substitute for rigorous, independent peer review. But FDA and other public health regulatory agencies are not like individual researchers, and they are not academic institutions. They are public organizations charged by Congress with achieving good public health outcomes. They rely heavily on scientific studies and information in making decisions, but the studies they rely upon and disseminate to the public are generated by the others. FDA thus, functions itself like a large peer review organization. It takes in studies and other NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. compilations of data and information from multiple sources, analyzes them, and makes public health regulatory decisions based on the available scientific information, the law, and legitimate public health policy considerations. Now, FDA is properly held accountable for the quality of its analyses and the data on which they are based. And that accountability currently comes in a number of forms. In the product approval area, for example, there often are formal public hearing processes in which the data supporting the safety and effectiveness of a product can be tested and vetted by experts. In rulemaking situations, as we have heard already today, the studies on which FDA bases its proposed rules are put in the public domain for review and critique, along with the rule itself, and the proposed justification for the rule. When important new policies are under consideration, or novel scientific issues are before FDA, the agency commonly consults with scientific experts in academia, at the National Institutes of Health or other federal agencies. It convenes formal expert advisory committees, or it refers questions to the National Academy of Sciences. On some occasions FDA publishes its own analyses of critical issues in peer reviewed scientific journals. The point I NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. would like to really make is that insuring the quality of the science on which it relies is at the heart of what the Food and Drug Administration does as an expert scientific regulatory agency. It's what the FDA commissioner and the career FDA staff do every day. It's their job. And it is FDA's success over many years in insuring the quality of the science on which it relies that has earned FDA's reputation as the world's premiere public health regulatory agency. If FDA were not doing this part of its job well, it would not be doing its job at all, in my opinion. The solution though would not be to graft on a peer review process. It would be to fix what, if it existed, would be a fundamental management problem at the Food and Drug Administration. Now, the draft bulletin does not suggest that there is a fundamental management problem at FDA. So, my suggestion to OMB is to be careful in imposing blanket new peer review procedures that imply otherwise; that imply that there is a fundamental management problem at the FDA. It should also be careful not to impose procedural requirements that undercut FDA's flexibility to manage its scientific review processes appropriately in the many and NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. diverse circumstances in which it must marshal science and act to protect public health. In doing that, OMB would be substituting its judgment about how to manage these key FDA functions for the judgment of the FDA commissioner and the agency's experienced staff. Now, let me turn to potential unintended consequences, which I think flow from the potential scope and breadth of the impact of this document. And again, I may appear to be overreacting to the document, but on its face, the document suggests that its scope and impact will be extremely broad. First, the bar for triggering formal external peer review is set quite low. Regulatory information is deemed especially significant not only if it is disseminated in support of major regulatory action, but also if its dissemination has a clear and substantial impact on important policies or private decisions with the possible impact of $100 million. Now, these terms are not defined, but could be read fairly to encompass a great many instances in which FDA communicates scientific information in pursuit of its public health mission. The proposed bulletin, as already noted earlier, also gives the OIRA administrator broad authority NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. to require independent, external peer review if there is significant interagency interest in the dissemination, or it is relevant to an administration policy priority. In these words, there are no clear limits on this authority at all. The second basis on which I see the scope and impact of this bulletin being quite broad is that it does not distinguish data generated by FDA from data generated by outside parties. Now, in cases where FDA itself conducts a potentially impactful study, a good case can be made for some form of peer review. As noted earlier though, FDA is more typically in the business of taking in, reviewing, and disseminating studies conducted by others. The case for an extra layer of peer review is much weaker in those circumstances, but the proposed bulletin makes no distinction in its call for peer review prior to dissemination. Finally, the proposed bulletin's waiver provision for imminent health hazards removes from the FDA commissioner, and places in the hands of the OIRA administrator, the critical public health decision on whether the health hazard data available to FDA are sufficient to require immediate action to protect the public. Or alternatively, should be subject to further NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. review. Now, this shift of public health decision-making authority is truly radical and unprecedented, whether the number of cases that arises annually is large or small. And it overlooks the nature of public health decision-making, which differs fundamentally from the decision about whether a new piece of academic research belongs in the scientific literature. Public health agencies act all the time on the basis of emerging information, and in the face of scientific uncertainty. That is what we want them to do. This doesn't mean they should act capriciously or without careful review of the information before them. The stakes are too high to accept anything less than great care and allegiance to sound scientific practice. But if we want our public health agencies to act like public health agencies, we should not impose formal review requirements that may preclude their doing so. Thus, any OMB review mandate should include an explicit exception for cases in which the FDA commissioner, not the OIRA administrator, determines that action must be taken to protect public health. Now, together these three features of the proposed NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. bulletin, the low bar for independent external review, no distinction between externally and internally generated studies, and the vesting of public health decision-making at OMB seem likely to cause unintended consequences. I will close my remarks by identifying a few of these possibilities. Let's suppose for example that FDA learns in the context of a foodborne disease outbreak that a certain item of fresh produce from a certain geographical source is at high risk of being contaminated with a dangerous pathogen. The agency compiles contamination data and information on the origins and distribution of the produce, and decides to release the information with an advisory to restaurants, grocery stores, and consumers not to serve or consume the product. The direct and indirect economic impact of this information release in advisory could easily exceed $100 million. Should FDA be required to conduct an independent, external peer review before acting? Or let's suppose that FDA receives through its adverse event reporting system, a rash of reports on the failure of a particular artificial heart valve, or learns through inspections that there has been a breakdown in good NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. manufacturing practices affecting the safety of a critical drug. FDA checks out the information through its normal processes, and concludes that prompt action, including an alert to the medical community and court enforcement action is required to protect the public from the unsafe medical device or drug. Would further formalized peer review at this point, be in the public interest? Now, let's suppose in another setting that FDA receive a petition from a food company for a new diseaserelated health claim. The petition includes data from dozens of clinical trials and epidemiological studies, only a few of which have been published in the peer reviewed literature. FDA carefully reviews the studies and concludes that the totality of the evidence supports the claim. On that basis, FDA initiates the public comment process by publishing a notice of proposed rulemaking to approve the claim, and placing the studies and its detailed analysis of them on public display. During the comment period FDA convenes its Foods Advisory Committee to discuss the proposed claim and its scientific basis, a common procedure at the agency. Now, the economic stakes here are also well above NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. $100 million. So, I ask, how would the OMB requirement of independent external peer review contribute to the quality of decision-making in this case? Would the benefit of an additional layer of review be sufficient to justify the delay inherent in external peer review of dozens of studies? Finally, how would the peer review bulletin apply to the ongoing efforts of many federal agencies to craft a public health strategy on obesity? FDA and other agencies are considering both educational and regulatory measures to address this problem. Literally thousands of studies arising from the natural and social sciences are relevant. And the scientific uncertainty about how to affect behavior change in this area is great. Now, this is clearly a case in which the government needs to engage the outside scientific, public health, and educational communities to craft effective interventions. Is formal peer review of the studies on which the agencies have tended to rely, the most effective way to do this? I'm not sure, but as I read the bulletin, any agency launching an obesity initiative would have to obtain formal peer review of the agency analysis underlying the initiative, and any studies disseminated in support of the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. initiative unless previously peer reviewed. Now, these are just a few of the potential consequences that I see of OMB's draft peer review bulletin. Some maybe intended, and some may be unintended. I hope that before the bulletin is finalized, OMB and the agencies will consider these and other possible consequences, and work to craft an approach that does not simply add another layer of review, but preserves and even enhances the ability of public health agencies to do their jobs. Thank you. DR. KENNEDY: Thanks, Mike. Two quick comments. I want to thank my colleagues on the panel for a very rich hour. No one went more than a couple percent over the time limit, so I apologize for harassing them a little bit. I also think that anybody who had taken careful notes in this proceeding is going to be able to file just one heck of a comment. Agenda Item: Questions DR. KENNEDY: Questions for any members of the panel, or for all comers. While you are warming up, Mike, doesn't in effect, the last problem you mentioned with respect to the public health authority deliberately, or perhaps mistakenly NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. transgress a statutory provision for the imminent hazard provision? MR. TAYLOR: Well, actually that authority exists statutorily only for the secretary of health and human services with respect specifically to drugs. But the vesting of the authority to implement the federal Food, Drug, and Cosmetics Act generally is vested in the secretary of health and human services. So, this is why I characterized this particular element of the bulletin as radical, because it does put in the hands of the OIRA administrator, the decision whether there is enough evidence to act on a public health issue. And that arguably not the job there. MS. KENDY(?): My name is Eloise Kendy. I'm a AAAS congressional science fellow. I have been on both sides of peer review. That is, I've been a reviewer and reviewed, both within federal agencies, and in the scientific literature. And what I'm hearing from you is that the major impact of this on agencies is going to be delays and increased cost. And what I'm not hearing is that there is going to be any major change in policy. Particularly, I don't hear any of you saying that NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. you are worried that your science that your agency produces might be rejected by peer review, like it can be rejected when you submit an article to a journal. Why is it that you are not addressing that? Why is it that it's not in guidelines? Is that something that is just given, that your science will not be rejected? And if so, why are we doing this additional peer review process if it's not actually going to affect the policies and regulations that result from the science? DR. KENNEDY: Anyone want to take a crack at it? MR. TAYLOR: I think the point I was trying to make, and I think probably the others were trying to make is that obviously work needs to be done. Procedures need to be gone through to insure the quality of science underlying regulatory decisions. The question is how you get there. And so, it's common within the Food and Drug Administration for example, that there will be different points of view about science. Ideas and analysis will come forward that will get essentially brutally peer reviewed internally within FDA. And so, there is a good process to insure the quality of science. Not every idea that occurs to every FDA scientist turns out to be right or to see the light of day. The NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. question is whether you get there through established internal processes supplemented at the discretion of the public health regulatory manager by external review, as opposed to requiring kind of a routinized external review. I think that's really the issue. DR. KENNEDY: Anybody want to add to that? DR. MAHONEY: I might take a little bit of a stab in the same thing. I guess I have been on both sides of this, and also been a journal editor for some long time, trying to deal with the matter of resolving reviewer comments back in on the yea or nay decision on publishing, after getting reviewer comments and getting revisions. In the case I cited in the climate area, we have the issue of developing findings perhaps on a somewhat longer time scale. And I make kind of a fundamental difference between what might be seen as a bright line regulatory decision, we need the following level to be declared as a safe level for a particular substance, whether it's a drug or a toxic substance, and the like versus a broader body of information. I don't think it's a matter of either fear of being rejected, or absence of fear of being rejected. But I think have made probably the most positive of anybody on NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. this panel, comments about a role for a great deal of scientific community peer review, and even more open review, because some types of information benefit from that kind of broad challenge. So, I believe I'm trying to address that in the comments I was making, which was probably of a different character than as I say, the bright line yea or nay decision about this is a level that we will declare safe, or will allow the following pharmaceutical to be on the market. So, it may be good to keep that kind of distinction of type in mind. I don't think it responds to your question beyond that, but I think your question is an important insight. MR. BROMBERG: Kevin Bromberg, US Small Business Administration, Office of Advocacy. I've got a comment and a short question for David Michaels. As someone who has been involved in reviewing federal agency rules, and also EPA rules for the last two decades, I can assure you from my experience, even though I heard comments from I guess the last two speakers that perhaps we don't need additional peer review, that there haven't been problems that have occurred because there has not been peer review, there is a very natural tendency for NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. an agency to defend its own work. And it is not interested in hearing skepticism from the outside. I'll just give you one experience I had. EPA in the metals, products, and machinery rule, which is a water pollution rule that they have worked on for several years, was a $2 billion a year cost of the rule. And we looked at the result of their analysis, and it struck me immediately upon looking at the draft analysis that it didn't have any peer review, and it was something that could have benefitted from peer review. It appeared to me that it was very odd that industries are already regulated by an existing set of federal water guidelines were emitting so much pollution. So much pollution that I could have a pail at the end of the pipe and pick up a lot of copper and sell it at the end of the day. So, I thought it was rather odd. Anyway, at the end of the day, EPA decided after seeing work from my office and others that they had made a mistake. And the $2 billion rule became an $18 million a year rule. And it's not necessarily an isolated example that is at the high end. But I can assure you that more peer review has its place. DR. MICHAELS: But if I can just say, it sounds NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. like the system worked, the interagency system. MR. BROMBERG: Yes, the interagency system worked, but we don't always have people out there looking at things like that. And that is the kind of rule that should have had that. What if I hadn't been there? What if people hadn't been watching? So, this mechanism is to assure that there is a minimal level for rules that are of some significance. The question I had, David, you raised in your comments is that the guidelines for peer reviewers in terms of the selection criteria and bias, et cetera. You were critical of the fact that there is a comment here about it should be noted whether a reviewer is getting agency funding. And you thought there should be the equivalent for business funding. And I think there should be too, and I suspect OMB did. But when I looked at the language under (i) that perhaps should have been better worded, which talks about among the factors considered in deciding whether a peer reviewer has a conflict of interest should be financial interest in the matter. I thought that was intended to cover whether you have a consulting relationship, or whether you are working NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. for the company that has an interest in the matter. Doesn't the financial interest test cover businesses in your mind? DR. MICHAELS: It is my understand that let's say there is a review on pesticides, and I work for Dupont on lead, I would have no financial interest in the matter. Am I wrong in that? MR. BROMBERG: But I suspect it would be nice if they explicitly said financial interest, including business relationships, et cetera. I'm sure they can work on that. DR. MICHAELS: No, I appreciate that. DR. KENNEDY: We have time for one more question, and then we must change the guard. DR. ASHFORD: Nicholas Ashford, MIT. I have been looking at this thing called regulatory science for 30 years. And what I have seen -- I think you have seen the same thing -- is the number of independent scientists dealing with environmental health and safety issues that we're able to graduate over the 30 years has gone down immensely. I think this proposal is clearly anti-science. It seeks to number one, control the peer review process. And no one has talked about the fact that people have been taken off of study sections that peer review the products of some NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. of the agency research. We had the Office of Technology Assessment, which did effectively a peer review process. But because it evaluated Star Wars the way it did, it was basically eliminated. The climate change report was not factored in and used in the construction of the energy plan by the vice president. If we really want to put science back into the regulatory and governmental policy, this administration should endorse more money for the graduation of scientists, for the creation of independent scientific research. The fact that it is not doing that, and seeks this to put a stranglehold on the process should be revealing to all the people who observe it. DR. KENNEDY: Anybody here question that? Okay, thank you, Nick. We are going to change the guard here. Agenda Item: The Peer Reviewers' Perspective: Implications for Scientists -- Transparency; Conflict of Interest; Burden; Joint Review; Agency Response; Public Comments - Moderator: John C. Bailar III, Professor Emeritus, Department of Health Studies, University of Chicago NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. DR. BAILAR: It is a pleasure to call this next session to order. I had planned to make some introductory comments. I found that everything I was going to say has already been said two or three times, so I'll go straight to the introduction of the speakers. First is Warren Washington, who is chair of the National Science Board, as well as senior scientist and head, Climate Change Research Section of the National Center for Atmospheric Research. Warren. Agenda Item: Speaker - Warren M. Washington, Chair, National Science Board, Senior Scientist and Head, Climate Change Research Section, National Center for Atmospheric Research DR. WASHINGTON: Thank you for the opportunity of having me speak. I'm going to have two different roles here. I'm going to speak as the chair of the National Science Board. Also, I'm going to speak as an individual scientist who is engaged in climate change research. The first is as the chair of the National Science Board. I did invite OMB to come over and speak to the board at our latter October meeting. So, we had a preview of some NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. of the issues that are the subject of this workshop. The first slide here that I want to show is that the integrity and the credibility of the regulatory decisions relies upon a peer review process that is clearly articulated and clearly implemented on the proposed OMB guidelines specified for a use of especially significant information. Now, I should point out that in the years previous to now that NSF guidelines have not been challenged, even a single time, so that we have not had a problem with information that is disseminated, and that information being challenged. So, for the NSF on this, it's not a serious issue. On the impact of the guidelines on NSF-funded research, we anticipate that there will be minimal impact on NSF-funded research. And NSF sort of views this as an agency in which it funds research conducted by the awardees, and not by NSF itself. And so, the burden of compliance with the act resides with agencies who are using some of that information, but not with the agency actually funding it. Conflict of interest. Extensive guidelines are already in place to avoid conflicts of interest. It's NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. essentially a voluntary system. However, I think that we have sort of a built in aspect, which I think actually works fairly well. And that is we have knowledgeable programmers who certainly know the community, and therefore they can sort of establish a balance of reviewers who can look at all sides of a proposal. Now, in some of the emerging areas of innovative research, we are finding that we only have a limited number of peers to serve as reviewers. And therefore, we have to be fairly careful about overusing some of these people. These well informed peers have only limited time to conduct reviews and sort of see a problem if we are putting extra burden on them. We view a separate review process will impose work load burdens on the active researchers in the field, especially in those areas where there are only a limited number of peers. On the burden on agencies, well, right now for this last year, the NSF included 54,000 individuals as reviewers of its proposals, both as members of panels, and as well the mail reviewers. And also, we are finding that over this last year that we had to attract 9,000 more new reviewers to our system. And we are finding it harder and NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. harder each year to sort of cope with this. It is especially true since our number of proposals has increased dramatically over the last few years. And so, we are quite seriously concerned about imposing another layer of review burden on the science community. So, we can see that especially if we get in situations where there are cycles of challenges from dissenting sources, that we can get ourselves an even more overburdened system. Challenging information. The act allows people to challenge the quality of information disseminated by the federal agencies. There is potential for some agencies to be burdened by frivolous claims petitioned under the corrective process. Who decides on whether a claim if frivolous or serious? And so, there needs to be an articulated appeals process. Now, I'm going to shift to my personal observations as a climate change researcher at the National Center for Atmospheric Research in Boulder, Colorado. As you well know, there are very serious concerns about global warming, about climate change, about environmental change, and so forth. And all climate data has errors. All climate NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. models, and I'm a climate modeler, have models that are used for future climate projection, and all of these models have some sort of problems. However, I still regard them as very useful tools about guiding and understanding on the past climates, as well as giving us information about future climates. And there is a serious question about the climate issues. And that is should the solutions to intergenerational problems such as climate change, wait for perfect understanding before we start to implement the knowledge and sort of regulations? And finally, because of worldwide concerns about climate change, we have been full partners in international aspects of science. For example, the Intergovernmental Panel on Climate Change, the IPCC, is an assessment process where scientists all over the world submit peer reviewed publications and research to this panel. And they try to draw up sort of conclusions about our basic knowledge in terms of what the climate is doing, and what it's expected to do in the future. Now, clearly, the ultimate aim of this very large international effort involving hundreds of scientists is to provide information to policymakers. And I can see that NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. under some of the aspects of this act, that there could be opportunities for mischief in terms of directing on the science in any sort of way that isn't in the public interest. I think that I just want to end at this point, and say there is a role for skeptics, especially in the area of climate change. And I think the role ought to be that skeptics ought to use the normal avenues of peer reviewed publications, and taking part in various assessments and debates in the community to make sure that their views are sort of weighed in. But this has to be done very carefully through the peer review system. So, I'll end up my remarks there. I'll be happy to answer questions later on. Thank you. DR. BAILAR: It is a pleasure to introduce the next speaker, Dr. Gilbert Omenn, who is at present professor of medicine, genetics, and public health, University of Michigan Medical School after his distinguished career in several other very important positions. Gil. Agenda Item: Speaker - Gilbert S. Omenn, Professor of Medicine, Genetics, and Public Health, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. University of Michigan Medical School DR. OMENN: Thank you, John. It's a great pleasure to be here and join many colleagues involved in the regulatory science arena for a long time. I'm a little disoriented today by so much discussion about sort of what we take for granted as routine research peer review on research grants, while I think the major intent of the OMB bulletin, and of our agenda today is on the regulatory agencies' use of peer review. So, I will focus on that, and not make further comments about the former. This is our general problem, risk communication in one cartoon. The lay public, the journalists, the regulatory officials, public elected officials, and many others decry the difficulty of getting a uniform, consistent full agreement among scientists. It is our nature to disagree, to challenge, and to find the uncertainties and the variability in various schemes, to look to do things better in the future. And we have to somehow be faithful to our values and our disciplines while working with people who are looking for crisp, clean, certain responses. It is even true after peer review that we sometimes have this result. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Now, a wonderful figure in this world was Judge David Bazelon, for some 31 years was the chief judge of the 1st District Court here in Washington, DC, which is the home of essentially all federal regulatory agencies. And in the 60s and 70s and 80s he and his colleagues had many of the most important regulatory review cases. He gave a lecture. I don't know that this was ever published, that why I put an approximate. But the lecture was in the 1979, in April at the University of Southern California with this wonderful title, "The Pearls of Wizardry." After all, the wizard is someone who divines from all kinds of complicated situations, exactly what should be done. And he felt that too many experts who were qualified as experts in the Daubert sense that came later, were seduced or volunteered to opine about what the regulatory decisions should be, instead of what could be made of the underpinnings of scientific and technical material for which their expertise was judged relevant. So, his premise was that regulatory issues tend to recur. Therefore, building a useful record on which the next case can start is a rational and efficient strategy. And to the technical experts both inside agencies and NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. outside the agencies his recommendations were stay away from the ultimate policy decision. It's not your charge. It's not your special expertise. Delineate the specific elements of the risk characterization. And then focus on those elements and build the record. For some of those, there will be agreement between experts representing opposing forces that the evidence is sound, the methods were right, the assumptions are credible. You can check that one off. For others, there will be disagreement about the quality of the evidence, the conclusion of the evidence, but there will at least be agreement that it is feasible to study those problems. And for elements it may just be beyond our current capability to even pose a critical experiment. That should be recorded also, and perhaps next time science will have advanced so that that will be feasible. And for those who participate credibly, the reward might be that you would be asked to serve again, as long as there is not an admonition against multiple services, we read here. I should say before I go onto the red book that the origins of the red book really lay with an interagency NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. regulatory liaison group in which Don Kennedy was one of the founding members, and I had a chance to work with him when I was in the Office of Science and Technology Policy. And several publications in late 70s and in 1980 set the stage for what became a very influential Academy report widely referred to as the red book. It was already held up in the last session, Risk Assessment in the Federal Government: Managing the Process. One of the questions we were asked to address very specifically was should regulatory relevant research, the research activities upon the regulatory agencies depended, be removed from, separated from the regulators, because it was charged, the regulators drove the researchers to come up with the answers that were desired. We looked at different models, because this federal government has quite a variety, as we have heard today. The one is NIOSH and OSHA, two agencies that need to work closely together, and are in totally separate cabinet department, with only a modestly good record of finding common ground on particular issues. A very small agency, the Consumer Products Safety Commission, which has quite a lot of influence, which has no scientific staff -- at least in those days had no scientific NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. staff, and under statute relied upon panels organized for them by the National Academy of Sciences. And then the two big agencies, FDA and EPA, which had extensive and still have extensive scientific capabilities. And the question is, how is it organized and managed within the agency with different program offices and the like. Well, we embraced variety, but we strongly favored more interaction rather than less, between research and regulatory programs and offices. That is to say, the FDA and EPA models. And we promulgated this framework, which has been widely used of hazard identification, risk characterization, and risk management to organize the different functions in a way that would be feasible within those agencies. A decade later under the provisions of the Clean Air Act of 1990, there were two important reports, one called, "Science and Judgment in Risk Assessment," a terrific title. Science and judgment always need to be brought together. And then the presidential and congressional risk commission, on which I served, and as requested before about disclosure here, the other members, including many who are familiar to you. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Well, our framework -- you know a picture is worth a thousand words. This is our main output, a six stages framework which has been adopted in Japan and Britain and other countries, and used extensively by US agencies. Two crucial points. The first is that with every new problem, whether it's a research report, or a neighborhood cluster of cancer cases or birth defects or some other problem, it must be addressed and put in context, preferably -- that's the second point, with the engagement of the stakeholders from the start. And I can briefly tell you a story. In 1983, when Bill Ruckelshouse(?) came back to EPA at the request of Pres. Reagan, there was a pending regulation on arsenic emissions from copper smelters, of which the biggest by far was in Tacoma, Washington, right near where I lived, but very close to where he worked at Werhouser(?) -- that is Ruckelshouse. And across the water in the prevailing winds was a place called Vashanal(?) Island. For those of you not familiar with the northwest, consider it Walden Pond West. People there thought they had gotten away from any hazard, problems of modern society. And here the winds, the forces of nature bring them arsenic from the copper smelter in NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Tacoma. Well, there was a memorable evening, the first of three public workshops, public engagement, Ruckelshouse. Two assistant secretaries of EPA, all three TV networks. It was quite something. I had the duty of being the moderator. And the reason I tell you this story is that the experts gave a typical solid, technically laden presentation about the basis for the risk assessment, and the conclusion that the risk was 2.437 times 10 to the minus something or other, if in fact you were exposed for 7 years at the maximum dose and all the rest upper bound estimate. And the people, when they had a chance to ask questions, asked reasonable questions like, is it safe to eat the vegetables from our garden? My dog died last year. Could it have been from the arsenic? Is it safe for the children to play outside? Did you say after additional controls -- they knew there were controls placed on before - that there would still be tons of arsenic emitted per year to the air? I thought a thimbleful can kill you. And unfortunately, the experts were just dumbfounded. There was no response to these questions. They hadn't been heard in the problem context days. They hadn't been considered in the risk assessment. And they NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. certainly weren't part of the risk communication, which was thought to be one-way. Risk communication has to be twoways, and it has to start early. Anyway, among the many topics we discussed, there is peer review in the middle of a long list. I recommend to you this report. Here is our position, which is frankly not too different from John Graham's today. That is, that peer reviews, if they are going to be conducted, should be conducted in a way so as to enhance the credibility of agency decisions and positions, and to improve the technical quality of the individual reports. That the peer review should evaluate quality of all types of data. And in the OMB bulletin this report is cited specifically to highlight our recommendation that economics, social science, behavioral science data be treated just as seriously as epidemiologic, toxicologic, chemical, engineering, exposure, and other data from the socalled hard sciences, because the embedded assumptions are just as important, and just as controversial, maybe more so. And the ease with which people quote cost figures, or sometimes even benefit dollar figures without attention to variability of the estimates, or gross uncertainty in the numbers is quite shocking, and quite asymmetric. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Third, that the peer review must address the credibility of assumptions and scientific interpretations. Not the policy recommendations, but the scientific interpretations. It should be an open process, informed by stakeholders and placed in broad context. We have a challenge in the OMB bulletin. We are told to be both broad and specific. It's not impossible, but it's different. And that peer review should not be overused. I think that resonates with the last panel. With regard to qualifications of peer reviewers, it's actually pretty similar to what we hear today. That the primary criterion without question should be relevant expertise. There is a secondary criterion, which I'll come to in a minute, which is willingness to address the charge. And that is quite often very vague, and not adequately discussed up front. Third, that there should be a multidisciplinary panel. People should interact, learn from each other, and not just be individual shots in the dark. But that we need to be respectful and avoid conflicts of interest. And these usually hinge on employment or financial relationship of any NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. kind, let's say a research grant, consulting, board membership, whatever, not in industry broadly, but in the regulated industry. And that led us to say that not only the environmental organization scientists, but scientists from industry not affected by this regulatory regime would be desirable if they brought necessary expertise. And with regard to academics, there had to be substantial disclosure. On the agency side there have to be clear written policies and guidelines. At the time, EPA had generated in 1994, the scheme which was referenced this morning, and we thought that was very promising. The extent of the peer review, let's say as an agency activity, should be commensurate with the importance of the scientific, economic issues, and the regulatory impact of the proposed decision. That is to say, it shouldn't be used routinely. It should be used when it would really matter, and there should be a justification, including published explanation and justification, both when it was decided to use it, and when it was decided not to use it. And especially if there was another mechanism, like a standing advisory board or National Academy relationship, or other NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. that sufficed. And very importantly, that there should be, just as in our six stage framework, a review, assessment, evaluation of the whole process; not of the individual decision, but of the whole process such that all stakeholders would be welcome participants in that review. Now, turning to my role as an individual participant many times in such peer reviews, and on the report committee that of the Academy, and the Science Advisory Board of EPA, and various other activities, here are a few observations. The first stage is receiving a request. And most people are asked, are busy and actually have paid jobs. So, to do something on the side when you've got other urgent deadlines and burdens and desires of pursuing your responsibilities and interests is not a trivial matter. The first question usually is how much time is required? And more importantly, what is the length and complexity of the document? What is the baggage of the issue? This is not an easy question to ask or get a good answer. And it's especially difficult for a person, as our bulletin proposes, with little experience in the issue. That's a big problem. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. The nature of the issues or the controversies you would like to have revealed to you, but I'm told by agency people that they are advised that this not permitted, or at least advised against. It's a very complicated matter to get the guidance to find the needles in the haystack of a very big, complicated report, sometimes having so-called gray literature, unpublished literature, or agency reports, as well as peer reviewed publications. And there is also the question of who else is on that panel. Is there an agency bias in the selection? What are you walking into? There is an awful lot asked when you serve. Pending now is this notion that the participants are supposed to learn about the expectations and the federal guidelines on information quality and regulatory analyses. This may be more burdensome than I realize. Will there be an individual response or a group process? Most people I think, find it much more efficient to do an individual shot, but much more educational and much more productive to have a good group process, if it's good process. It takes a lot of care to construct a well balanced, multidisciplinary panel roster. It always gets NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. botched up by the people who decline. Adjustments have to be made. Time is lost. Time pressures are often severe. But this is the hardest part. Among our colleagues, we need to acknowledge that the return is often weak. Many people submit perfunctory reviews. It looks fine. Sign the name. Or do a little editing, a similar result. Some people are argumentative, putting in more than you really want to read. But most limit themselves to one or two chapters out of a complex report, because they feel that they have only been asked for their most narrow expertise, not their judgment or their participation in a multidisciplinary analysis. So, a thoughtful review of the whole document is hard to come by. Thank you. DR. BAILAR: Thank you, Gil. The next speaker is Dr. Tom Louis, who is professor, Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health. Tom. Agenda Item: Speaker - Thomas A. Louis, Professor, Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. DR. LOUIS: Yes, and I have no affiliation with the New York mayor, although my school does. If you didn't think that everything had been said in the previous panel, by this time in the day you have to agree that almost everything has been said. So, I think I'll speak on FACA, which I know very little about, but at least is a topic that has yet to be discussed. I will make a few remarks that focus on peer reviewers and the role thereof. I have been on all sides of this other than a federal agency employee. I've been certainly reviewed and reviewer. I'm editor of the Journal of the American Statistical Association, and on and on like everybody else who has spoken today. So, at least I think we all have a good idea of what it is like to be a peer reviewer in a lot of different contexts. I do think that sociology actually does have some load bearing capacity at least for dealing with the issue of the role of peer reviewers. And I think that we are involved here in a situation that Gil has said it, others have said it, the supply of qualified reviewers, even if you set aside for a moment the conflict of interest issues, but just scientific qualified, let's say sociologically qualified in the sense that they have been experienced in NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. this process, is relatively small. And even at whatever size it is, most of those people are booked or overbooked, doing NSF, NIH, HEI, put in all the letters you care to put in, FDA reviews. So, one of the challenges that I see here is that inevitably if something like this advisory goes into effect, the need for peer reviewers will increase. It is also true that even if the need for peer reviewers doesn't increase, the context of peer review, even in those agencies and institutes that are doing it already, I think will change. The culture will change, at least in the context of the way this document has been written. And so, I think a very big challenge if something like this goes forward, and goes forward in a way that people find at least reasonably acceptable is to recruit and retain qualified peer reviewers in the context of many other competing demands, and possibly a change of culture even for those things that used to be very attractive for reviewers to do. People have touched on the issue of constituting a peer review panel, so I kind of labeled this level zero design, and I certainly talk about that a lot in the context of putting together a study team for a scientific study, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. primarily when they leave out statisticians. But in general, it is a very important issue. It not only requires putting in the right skill sets into the panel, but others have mentioned, and I'll just repeat, making sure within those skill sets that there is a variety of points of view. The most failed peer review is the one that has complete consensus. It is either the case that the topic is so clear that it didn't need any peer review, or in fact that the committee was just overly homogenized. To constitute a good peer review committee, and to maintain it in the sense of support, and if you would like esprit takes amazingly good staff work, not only in terms of attention to detail, but staffers that actually know the fields, know the context. And I'm not implying that those don't exist already in the agencies that are doing peer review, and doing it well. But again, if there is a dramatic increase in the need, there is going to be the need for a cadre of experienced staff people to run these panels, constitute these panels, and so on. The issue of, at least speaking for myself in terms of the attractiveness of being on a peer review panel, or submitting an individual review in this context, I think NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. it's going to be very important to have it be that reviewers are not individually identified. That they aren't nonymized or whatever word Don used for that. That our names be listed, as does the National Academy, in terms of the following people have reviewed this document, but that the individual review content is not names. And it's not because I'm embarrassed about having anybody know about my words. It's that in this regulatory context, it very well may be that by identifying individual reviewers, you are opening yourself up to all sorts of additional contact trouble and so on, that you may or may not want to have. But if you are going to retain good reviewers, I think you need to have a process that will allows people to be candid, and allows people to feel protected at least in that sense. There are a lot incentives for agreeing to do peer review or to be on study panels. Lord knows a lot of us spend a lot of time at the National Academy. Basically, you get dinner. Today we got lunch. But what do you actually get? Well, the first things you've been on maybe you feel it's prestige, and presumably it is, but that wears off very quickly. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. It's impact. It's actually learning a lot from the colleagues on the panel. It's knowing that the results of your labor is going to be treated properly. And what I mean by that is it is going to be going to be kept as a whole. It's not going to be fragmented. It's not going to be politicized at least by the Academy; it may be by others. But that your work is going to actually be treated with respect and as a whole. And so, I think again, I'm not implying that it won't happen, but I will simply suggest that it must happen, is that the process here, which is very similar in that these reviews and the results of the entire panel are going to be used for regulatory purposes in the sense at least of blessing or possibly condemning something that is going on. The integrity of the process has to be great, for a lot of reasons, but speaking in the role of reviewer, because I'll walk otherwise. It's just not worth my time here, versus the 50 other things I could be doing, if in fact the process isn't kept good. I can list a long laundry list of other incentives, and indeed, compensation in the sense of money is there, but I don't know of anybody. And I'm certainly not a person who does these reviews for money, even if in NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. some context I'm getting a little bit of fee, as I am for the Health Effects Institute. And in fact, for anybody who is actually thinking of doing reviews for that reason, they should definitely get their head examined, because you can make money much more easily than doing these reviews. And if the price goes up, others have mentioned, and I think it's true, the entire sort of sociology and demographics of the review population will change. So, John Bailar has been known to state frequently that anything not worth doing, is not worth doing well. Correct, John? DR. BAILAR: You've got it. DR. LOUIS: And I think OMB needs to be very careful here, as others have stated, about putting something in place, even that by putting it in place very well, that in fact is not worth doing. My modest suggestion is that rather than going this route, that there be data collected, information collected on all of the agencies that are now doing reviews in whatever context, whether FDA, NIH, or whatever. Build, if you would like, a collection of models. Have agencies peer review each other, possibly with outside help to peer NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. review the peer review. And have formal, but not legalistic feedback go back to say, here is a case where I think we as peers think you need to improve the process. And start by taking advantage of the very many wonderful models around. Not that they are fail safe. Everything will have its problems from time to time. Start that way. Evaluate that process. And if something else more legalistic or regulatory or at least implied regulatory is needed, put it in place, but wait and try what you have called sort of the collaborative approach first. Another point made by one other speaker, and I'll close with this is it is absolutely the case that the population of peer reviewers relative to even the current need is small. And I think we need to do two things. One is to be very proactive in pairing a younger in career person on a peer review panel in a discipline similar to an older in career person, so at least for those that exist, we aren't waiting until they are older in career to give them the experience, but have it be a mentoring process. And indeed, we do need to fund and create more scientists and others who clearly will create the knowledge, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. but also will be available as peer reviewers. Thank you. Agenda Item: Questions DR. BAILAR: We have some time for questions and comments. First, I would like to thank the speakers for showing us that at near the end of the day, there is still a great deal more to be said. While those of you who want to make comments or ask questions are getting to the microphones, I would like to emphasize something that has come up in various contexts. That is that the peer review of these major documents is necessarily fragmented. We are all used to peer review of original articles submitted to journals, maybe 15 pages, 20 pages double spaces in the fields I know about, maybe up to 30 pages single spaced in contract proposals, grant applications. We are talking here about 500, sometimes 800 pages of densely packed material regarding some major federal action. Furthermore, the kinds of peer review we are used to tend to focus on a single point within a single range of expertise, two, three at the most. Some of the public health problems that I have been involved in have involved NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. several kinds of medical specialties -- biostatistics, epidemiology, delivery of public health services, public education, even such things as law and economics. Nobody can be expert in all of those things. Peer reviewers have to take up a piece at a time for something that large and that complex. This brings me back to the necessarily integrated function of the recipients of this advice, that is in the present context the federal officials who have to make decisions on these immensely important and immensely complicated matters. I must say that as the day has gone on, I have become more and more of a skeptic about whether this approach is appropriate. I am really concerned about the potential for mischief in it. I don't see that the added value is going to be as great as the added cost and the added hassle. I'm quite happy to be convinced otherwise, but at this time on a Thursday afternoon, I'm really pretty doubtful. MS. CASANO: Pat Casano, GE. I have two questions, one for the panel generally, and maybe I really should have asked this of the previous panel, but it didn't occur to me, and then one question for NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Mr. Bailar specifically. It seems to me there have been a lot of comments to the effect of the pool of peer reviewers being small, and questions about getting peer reviewers who have the right expertise and experience basically to pass judgment on agency work products. But there hasn't been any discussion really about the experience and expertise of the people who are writing these reports or projects or whatever in the first instance. I'm curious as to whether your views on the need for peer review, and the amount of peer review would vary depending upon the level of experience and expertise of the people who produce the product to be peer reviewed in the first instance? DR. LOUIS: Just one quick comment. I think for me, the need for it doesn't vary with the expertise of those creating it. I personally want to be peer reviewed on everything that I do, and I hope that I'm a high quality person. I think maybe the type or the context of the peer reviewers may change, but I submit that the decision to peer review shouldn't depend on that. DR. OMENN: At the same time, there is little worse than trying to peer review a really lousy document. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. So, a quality effort from the agency is the best first step to getting a really good, helpful peer review on a final decision; a document that will help the decision-makers make a good decision in the end. John's comment, and Tom's also, I think we owe some kind of alternative if we are not keen about vastly increasing the amount of peer review exercise. And my own response is that ad hoc peer review panels have the most difficulty functioning well, having a sense of the history of the topic, generating a coherent review. And while they may be very useful in some cases -that's why six years ago we recommended judicious use of peer review, I think that reliance on scientific advisory boards and standing advisory committees of the kind that most of the regulatory agencies already have, with appropriate rotation, good balance, peer review of their performance is to be commended and encouraged. I think the agencies have a lot of experience. I know we are not supposed to be shilling for the Academy when we are here, but the studies that are done here are of a great variety, some of which fall in the category of peer review. One of the smaller agencies, the Centers for NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Disease Control, which has been tasked by Congress with many urgent turn around topics on immunization risks and vaccination programs and other kinds of things, CDC has begun to come on a regular basis for peer review of their protocols, and then their findings, and then their recommendations. So, there are alternatives to an ad hocery scheme which I think do work better, have a track record, and could be improved further. DR. WASHINGTON: I wonder if I could just add that the expertise of the staff is critical I think in my experience. I have been on a variety of committees over the years. And of those committees that have had experts who could help in the writing of the report and so forth, have actually functioned a lot better than where the committee had to do almost everything itself. I think that we also need to have people who have kind of been on a committee for a long enough time so the interactions of the committee members can help look at the whole sort of range of issues is very important. So, I favor on these wider groups. And then finally on the turn over, I think that any advisory committee or a set of peer reviewers needs to NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. have a turn over that brings in only the elder scientists and engineers in the field, but also it trains new people with sort of fresh perspectives. And I think that that's critical to sort of dealing with the societal issues that many of these regulatory or scientific issues have to deal with. DR. BAILAR: I think you have raised an important issue, that is the expertise of the staff that write these documents. I think it is worth noting that for most of them it's a full-time job for a considerable period of time, a whole team, whereas the peer reviewers are on the whole, largely occupied with other kinds of things, and cannot give this the intense concentrated, sometimes years long attention that goes into actually writing a document. Because of this, I think the agency staff does tend to build more expertise than outsiders might think. Things are rotated, circulated within the agency. They get broad comment from that source before they go out. I'm not saying there aren't any problems with them. Goodness knows we have seen enough of those. But they do have a lot of input. And that input is in a form that tends to sharpen the technical capabilities of what goes in. MS. CASANO: Thank you. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. My specific question for Mr. Bailar, you indicated that it's a conflict of interest for a scientist who works for the regulated industry that would be affected by a rulemaking to participate in a panel. Scientists from other industries would be okay. But I think you blanketly indicated that a scientist from an environmental group would be all right. What about a scientist who works for an environmental group that is either taken a position on the issue under review, or has used the issue under review to raise funds? DR. OMENN: I guess I was the one who put that slide up. It's exactly what we said in the risk commission report. We said explicitly that even scientists from the regulated industry should be heard from. That's to say, there should be a forum in which they present their views on published literature and any additional data. The problem with additional data has been mentioned several times today. If it hasn't been through difficult publication-related peer review, it is awkward, and it is sometimes presented, and can come from any source. It can come from an environmental group. It can come from the industry. It can come from the agency. Newly NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. discovered data sort of seem compelling in certain circumstances, and get undue attention. And yet, if the new data really are important, you hate to leave it out and be regulating three years ago either way. With regard to the environmental organizations, there are a relatively few scientists who are able and willing to serve on these panels from these organizations. So that I feel it's a voice underrepresented in general, and their biases need to be stated, just like academics or anybody else. Their biases are usually pretty obvious. And it is true, just as academics are sometimes accused of keeping an issue alive because it's a source of grant funding, within environmental organizations people are highly specialized, and if one topic declines in importance compared to other topics in the organization, that resource may be trimmed by the organization. So, people have biases, and that's unavoidable. If the expertise is sufficient, and the biases are known, I think it's useful, and that's the position that the risk commission took. DR. LOUIS: Just a quick comment. In general, I'm quite liberal. I could probably stop there, but I'll say liberal in the context of conflict of interest. I think NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. that sunlight cures most things quite frankly. There are clearly some situations where a person should not be on a review panel, but I think they are relatively few. And I find that if it's a panel especially, and the other panelists know about the person's either sources of support or other biases or statements, and if the panel is balanced, then in fact I don't find it a problem. So, I think that this is document is also too restrictive in terms of the conflict of interest rules. DR. BAILAR: I tend to agree with Dr. Louis. I am not as concerned about conflict of interest as some people, including some here under one specific set of circumstances. That is that the federal officials receiving the advice have sufficient backbone to do the right thing in the face of pressures to do something else. That is not of course always the case, just like journal editors don't always have the sufficient backbone to make the right decisions about papers when they get inappropriate advice from peer reviewers. My experience has been from close to 30 years of very intensive involvement in journal peer review, that reviewers make more mistakes than authors, and the comments just sometimes have to be passed over. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. MR. ANDERSON: I have a comment. I just wanted to endorse the idea that one should consider the alternatives in what this scenario might be without the bulletin. The bulletin could be viewed as an invitation to the science community to contribute independent, external searching review. That can be, under some circumstances, a check, not a blank check, to OMB; a check on, rather than an opportunity. My question is to Gil Omenn. Gil, had you had time to put up your conclusions slide, what did it have on it? DR. OMENN: It was wrapping up everything I said. But I finished a moment ago Fred, when I said we should have a clear statement of what we think is most productive use of peer review. I do believe and I just said that standing advisory boards and committees have special additional value in building understanding of the topics, and having sufficient rotation to be fresh and thoughtful and bring in new people. But to bring people together in a group dynamic that actually helps the agency, and helps the public by interacting in a way across disciplines, just as we routinely said multidisciplinary analysis, multidisciplinary NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. review is essential. I do believe we have certain guidance for operating peer review panels, whether they are standing or ad hoc, which are fairly well captured in the OMB bulletin, and I think well captured in the risk commission report, and congruent with most of what's been said on our panel today. MS. HUSHKA: Hi, Leslie Hushka from Exxon Mobil. And I will open this up to anyone who would like to answer it. In looking at both the guidelines and the bulletin, it appears that OMB has stated the problem is a lower than desired level of quality of information that the federal government is disseminating. And it's perhaps suggesting that peer review is one solution to improve the quality of that information. But there has been a number of concerns certainly raised about how they put forward that solution. And my question is the following. If you believe that peer review is not the solution for this problem, what as scientists would you suggest alternative solutions that OMB explore? And secondly, how would you measure the progress of this solution, of peer review, or any of those alternatives concretely? How would we measure whether we NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. are improving the quality of information? DR. OMENN: I think you can do it stepwise. The first is you need to have really good scientific information. And for a long time, a crucial area of regulatory science was very little investigated, and much based on assumptions, still based largely on assumptions, and that is exposures. So, exposure pathways, exposure levels, variation of exposures. And empirically tested models of exposure have only begun to become more adequate in very recent years. So, that was an example where putting the risk assessment framework together, and realizing this is a crucial element in every risk assessment, generally underinformed, led to new research, a whole new field of exposure analysts, and good progress. Much more needs to be done. Another area where we actually have very little information for most regulatory requirements is variation in host response. The Clean Air Act of 1970 said criteria air pollutant standard, Section 109 standards had to be set such as to protect the most susceptible subgroup in the population. Very few studies were initiated to find out who they were, let alone regulate to protect them. There are a NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. lot of assumptions made, often very large safety factors added, but that's a feasible parameter. So, the underlying science could be better informed. I think the best strategy for that is to really build research agendas across agencies with very important input from the regulators and their advisory boards, looking ahead 5, 10, 15 years of the kinds of issues we already know about that we need to understand better, and the ones we see emerging, which we think need such research agendas. It's what I mentioned this morning that Don Kennedy, when he was at FDA, and others in the regulatory industries over the years have tried to do, both without a whole lot of resources, and without adequate connection with the powerhouse of NSF, NCAR(?), NIH, sufficiently tied to them. The second thing I think is what we talked about before, is the quality of the staff in the agencies. Long before you get to the peer reviewers, having good people in the agencies, and having them feel that the scientific, and the science assessment roles, engineering roles are highly valued, and not just caught up in the political and management side of the agency is essential to a good functioning agency, and a good set of proposals that are NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. respectful of the nation's resources and of the planet we would like to preserve. So, I think only after that we have worry heavily about peer review. DR. BAILAR: I agree with Gil, but would go even one step further back. I would like to see a list of situations in which peer review would have been a substantial help. If nobody can produce such a list, to create such a list, that in itself would be very informative. If we had such a list, we might be able to learn from it what alternatives to peer review might be effective. Any other panel members, Warren, Tom? MR. FITE(?): My name is Richard Fite. I am a peer reviewer for the Department of Agriculture. What bothers me most about the OMB proposal is that I really think it puts the emphasis in the wrong place. Peer review really isn't the problem. The quality of science in our regulatory decision-making isn't the problem. Certainly, there are situations where we could have done better science, I don't deny that. But it really seems to me that the issue that OMB ought to be dealing with is the undue influence of politics in regulatory decision-making. And I'll note that in the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. past several months, probably two or three times I have been told to mute my criticism of agency science, because in essence decisions had already been made. And I just wanted to make that comment, and invite any response. MS. FRIEDMAN: I'm Sharon Friedman from the Forest Service. If the goal of the Data Quality Act is to improve data, and we have talked about peer review as if it's sort of a one size fits all. All you have to do is do this peer review, and everything will be fine. I'm wondering what your thoughts are on any other bullets, non-magic bullets, and I think of course of statistic review. Well, that can be part of peer review. I think of quality assurance, quality control, because I know when we look at studies and try to see if we think they are quality or not, we have a good idea from the members of our peer community about the things that peer reviewers would look at. But we have no idea if their equipment is properly calibrated, if they are taking care of their data adequately. I wonder what you think about these things, and whether there is any other tools in the tool kit that you NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. can think about that might help improve science? MR. TAYLOR: You have stated there is nothing magical. It's very easy to say have a culture of quality and excellence, and have people attracted to and retained in agencies, universities, whatever context you might think of that are rewarded for excellent work, and where the environment itself encourages it. Some of that might be, in fact is peer review. I think for me, at the least the issue that the OMB statements are touching on is more how do you deal with the situation where let's say every molar piece of the science is just excellent. I mean take the counter factual world where good laboratory practices have taken place. Statisticians were around to design studies and help analyze and so on. And the issue is how to put it all together, even if each piece is just perfect? And that's where, at least for me, the big challenge, whether you want to call it peer review or just the appropriate teams for synthesis or whatever, where the regulatory context comes in. Obviously, it's not the world that all the science is just beautiful and perfect. But even it were, much of the challenge that we have been discussed at least tangentially today I think would still exist. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. DR. OMENN: In the red book, and in Ruckelshouse's comments and publications subsequently -- in fact, he provided a very nice quote for our 20th anniversary celebration of the red book, saying that he had really been challenged to get his thoughts together by an invitation to speak in this room 20 years ago. We rely way too much on modeling and assumptions for things that are measurable. Some things are not measurable. It is not measurable what the incidence rate is of a cancer at a putative rate at 1 in 1 million lifetime exposure. But many other things are measurable. And your comment about the quality of the measuring instruments, the locations of monitors. One of the most slightly humorous aspects of the Clean Air Act experience 20 years ago was comparing where different communities placed their monitors. So, in Pittsburgh, where they were trying to show progress cleaning up, they placed it in the highest polluted areas, and showed reductions. In another community, unnamed, where they wanted to detect whether it was getting worse, they put it in relatively pristine areas. And then the same database has both of these sets of numbers comparing these two metropolitan areas when there was no similarity whatsoever NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. in the criteria with which relatively small numbers of monitors had been placed. This is not rocket science. This is not global climate change. This is common sense, and it's not even very costly. So, I think a systematic look at our data sources. Thinking back to the pearls of wizardry, I'm sympathetic to those of you whether you work in a company or an environmental organization or an agency where there is somebody at the end of the chain or the top of the hierarchy who wants you to hew to a certain line. It is good advice to every organization to say avoid the pearls of wizardry. Let the technical people do their thing. There is always enough discretion left within the uncertainty and the safety factors and statutory language to make some political adjustments as permitted in our society. But we shouldn't contaminate the best effort we can make on the scientific, technical side. DR. WASHINGTON: If I could just add a little bit. Obviously, in almost every one of our fields, we have certain scientists who are experts on data quality. People that really take the time to examine on the data, learn how to take the biases out, and learn how to make the corrections in a sound and scientific way. And I think NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. those people are very valuable in our respective fields. Now, one of the issues that we have to face is that we are always going to have incomplete data, and incomplete analysis. And we have to ask if we have reached a certain point where we need to exercise the precautionary principle, because if you are dealing with a problem that has a long time scale, it is better to deal with it early, than to deal with it much later, such as climate change. DR. OMENN: The precautionary principle is actually a hot item in Europe, and somewhat tended to in this country. Some of us in public health and in engineering feel we have been practicing the precautionary principle forever. I mean public health is all about precaution and about intervening to try to prevent exposures, prevent disease, prevent injury. Engineering has a concept which is rather nice it seems to me, in a world where we are consumed by modeling and safety factors which is ALARA, as low as reasonably achievable. And it has been used for decades, fairly reasonably. And the extent which at a reasonable cost, or at least technically feasible mode whatever the cost, but best at a reasonable cost you can make a substantial reduction in exposures or risks of exposures, that is real NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. progress. I'll tell you one of the things we said in the risk commission. We put a figure in that showed the difference between a log scale and a linear scale on risk. And you tell people that you are going to take an action at whatever cost to go from 10 to the minus 3 to 10 to the minus 4 to 10 to the minus 5 risk, most people think those are equal steps. How many people in routine conversation talk about the powers of ten? But when you should them that the first action removed 90 percent of the extrapolated risk, and then next step there is only 10 percent left, and you've got 9 of those, what is there left to spend a fortune on as the mirror curve of costs goes screaming upward? It's a whole different conversation. So, there are styles, facts for risk communication that can engage ordinary stakeholders, and maybe even experts in a productive way. MS. WILHOITE: My name is Sarah Wilhoite. I work for Earthjustice. I was trained as a scientist, and I think I was disillusioned with the scientific community I guess, maybe for not informing the public as much as it might as a NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. community in terms of the types of decisions that we can make based on the knowledge that science has to offer. And I see this peer review problem I guess as a way for the agency possibly to be inviting the scientific community to come in and give some of that knowledge, and inform the regulatory process. And I'm just wondering if you have an answer about how the scientific community might better do that? I guess I heard a lot of comments about how there is not a lot of time among scientists. You are very much focused on your objectives and on being objective in your own work. How is it that that can inform the regulatory process? Because we have a lot of knowledge that we already know, and I guess your comment about kind of the common sense approach. How can we get common sense into the regulation, and build upon the knowledge that the scientific community already has to offer? DR. BAILAR: Well, we don't have time for me to give my platform speech on this. But I think we have major, major problems in the way we train scientists. They are selected, they are trained to do certain kinds of things, and on the whole they do those very well. They are not selected and trained with a view NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. toward the kinds of things you are referring to, and some other important things. You will not find scientific training programs, at least not very many of them, where the degree candidates are given very much special instruction in things like how to affect national policy, how to look at things in a very broad context, how to communicate with the public. Even such things as responsible conduct of science. These things are not included in our standard training programs, possibly because the current generations of the trainers never had this kind of thing, and don't think it's important. I think it is important. We need to start, I think, with our scientific training programs to broaden them to include some kinds of students and some kinds of training that can go on to different kinds of activities. Anyone? DR. LOUIS: Not to disagree with Dr. Bailar, but a slight counterpoint. And that is that scientists are giving a lot of time to whether it be federal agencies or others for review, peer review and otherwise. It is also still the case that for these complicated situations, it takes continuity within the agency to make things happen. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. We are even legalistically have under conflict of commitment rules, for instance at Hopkins it's roughly a day a week. It's the sixth or seventh day, not one of the other days, irrespective of whether you are compensated or not. And so, I think that the cadre of scientists who are sort of able and ready and obviously willing are working at full capacity, and doing a lot. But if it's not your day job, you can't be the one who is carrying the continuity and the integration of the whole thing. So, it's always going to have to be the case of agency or institution, whatever the case may be, core functions, coupled with effective use of consultants, advisory boards, so on and so forth. We do need more people participating in that, but it will never replace the need for critical mass within the agency. DR. BAILAR: I don't see anyone else. I would like to thank the speakers and the audience. [Applause.] DR. MESERVE: At various points during the day there has been reference to some of the legal implications of the bulletin, and our last speaker of the day is going to deal with some of them. He is Sidney Shapiro. He is the associate dean for research and John M. Rounds Professor of NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Law at the University of Kansas School of Law. I know he has written extensively on administrative law. Sidney. Agenda Item: Integration of Peer Review with the Requirements of the Federal Advisory Committee Act and the Administrative Procedure Act - Sidney Shapiro, Associate Dean for Research and John M. Rounds Professor of Law, the University of Kansas School of Law PROF. SHAPIRO: Well, I have noticed the audience has winnowed down a little. So, clearly the administrative law dilettantes have left, and I will speak to the affectionados. I appreciate the invitation to speak. And I would like to talk about two pretty significant legal issues that are likely to arise, particularly in litigation concerning the bulletin. The first is whether the Federal Advisory Committee Act known as FACA or FACA, depending on your preference, whether or not that applies to the peer review committees formed at least to review especially significant information. And second, and perhaps more provocatively, will a court conclude under the Administrative Procedure Act that OMB lacks the legal authority to order agencies to use peer NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. review? In the proposed bulletin OMB suggests to agencies that there are two possible ways of conducting peer review for especially significant information that would avoid the requirements of the Federal Advisory Committee Act. First, to use individual reviewers who do not write a cumulative report, or at least don't meet to write a cumulative report. And the second possibility is to hire a contractor who would organize peer review. As for the strategy of employing individual reviewers, FACA applies to advisory committees. And according to regulations promulgated by the General Services Administration, which is the agency Congress picked to supervise FACA, according to GSA regulations, convening a number of people to obtain the advice of each individual does not establish an advisory committee. So, in fact, if that's the way an agency proceeds, they should be home free in terms of not having to comply with the requirements of FACA. But there are secondary considerations that would come into play, and you have heard others speak about those vis-a-vis the relative importance of having individual reviewers versus having a panel where people can interact, NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. and the peer review becomes the collective product of the panel. So, assuming that's a good idea, that an agency favors having a committee, because that is a better way to do peer review, at least in an individual situation, would that trigger the requirements of the Federal Advisory Committee Act? Now, here is where it gets a little hairy, because there is lots of words used in this statute, and some of them don't have their common meaning, so I will try to lead you through this. The act applies when committees are either established or utilized to obtain advice or recommendations. Therefore, when you hire a consultant to put together a peer review that will be organized as a group, does that trigger the Federal Advisory Committee Act? Well, according to prevailing authority, there would be two issues. Does the agency establish a committee when it hires a contractor to do all the work? And then secondly, if it does that, and that doesn't mean it establishes the committee, does the agency nonetheless, utilize that committee? The Supreme Court is pretty clear that the word "establish" does have a common meaning, and that is that the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. agency establishes, organizes if you will, the committee. So, committees organized by contractors are not established by agencies, and the issue becomes nevertheless, if the agency hires the contractor, is it utilizing that committee? The word "utilized" has not been given its ordinary meaning. According to a Supreme Court case, Public Citizen v. Department of Justice, an agency utilizes an advisory committee when the committee is so closely tied to the agency that it is amenable to strict management by the agency. In a follow-up case on which OMB relies in the proposed bulletin, the DC circuit decided on a two to one split that at least under the facts of this case, which I will give you in a second, when EPA hired a contractor to organize its peer review, that process was not subject to the Federal Advisory Committee Act. In this instance, EPA had reserved to itself, significant powers to control the contractor as to how the contractor carried out its duties. But for the majority, the significant fact was that EPA didn't exercise any of those powers. It just had the contractor go on its merry way. In a dissent, Judge Williams disagreed, and in NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. particular he pointed to the agency's authority to veto any peer reviewers chosen by the contractor. And at least for Judge Williams, the key issue was look, the contractor knows the agency can veto. So the contract is going to do what the agency wants from the get-go, knowing the veto exists, and therefore, at least for the dissenter, this was closely tied to the agency in a sufficient way that he would have applied FACA. Now, let's try to figure out what all this means in the context of the proposed bulletin. The bulletin imposes requirements on agencies for the conduct of peer review for especially significant information. You saw those listed in a couple of the other speakers' slides, so I won't go through those. So, the issue seems to me to be this. Are agencies legally bound to satisfy all those conditions? If the answer is yes, then the agencies cannot contract away their legal duty to satisfy those conditions. They must assert control over the contractor to make sure that their legal duties are satisfied. And if that is the case, then the test of utilization is met, because the agency, even under EPA v. Byrd(?) is exercising actual control over the contractor's process. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. What is at stake? Again, as just mentioned, OMB sets out a set of procedures that an agency must meet when it engages in peer review of especially significant information. Some of those are listed under the OMB column. If the Federal Advisory Committee Act applies, in addition to those procedures, subject some exceptions, the agency's advisory committee, whether it is conducted by a contractor or not, must have open meetings, must keep detailed minutes, and must give public access to its documents, unless they fit one of the exceptions under the Freedom of Information Act, such as for trade secrets or proprietary information. This is the issue to which Fred Anderson referred way back this morning, if any of you can remember this morning, when he wondered whether an agency can release a document so that it becomes public to satisfy say this FACA requirement, in order to have peer review, because after all, Anderson proposes the question, isn't that the dissemination of information? And even if the agency labels it provisional, hasn't been reviewed yet, this is temporary, Fred says well, look, you tell the jury to disregard the statement, but maybe they are not going to disregard it. Fair enough. But like most things in administrative law, there is a balancing NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. process that goes on here. Is it more important to have an open process of the kind that FACA provides, or try to protect against the eventuality to which Fred refers? I happen to think it's more important to have an open process, but others can disagree. What is stake, number two? The Federal Advisory Committee Act requires that membership be fairly balanced as to points of view, and as to functions that are to be performed. At the moment, the proposed bulletin has not similar requirement, although it does have a proposal that if an agency appoints a biased reviewer, they must appoint someone of a contrary bias. So, I think OMB does recognize the importance of balance here. But this seems to me a very awkward way of trying to solve the balance problem. It seems to me the general prophylactic of training to appoint a balanced committee from the get-go is a better way to get at this. So, this is my preference. I think OMB should require open meetings and fair balance. I think if they would do that, the process would have more legitimacy. Incidently, it would avoid the chartering requirements, which can be burdensome, although by the way, the White House controls those. So, if they didn't want to make them NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. burdensome, they wouldn't have to. And most importantly, it would moot the incentive to sue, because if OMB required agencies to use the core of the FACA procedures, then you wouldn't need FACA, you could just use the open government procedures, and that would be that. Let me move to my provocative issue. OMB requires peer review. As you just heard me say, presumably those are legally imposed duties, perhaps not. And if they are not, then agencies can ignore them subject to the political wraith of the White House. But does OMB really have the legal authority to impose peer review on agencies? Did Congress authorize this in general, or for rulemaking? All of these acts are very general, with lots of ambiguous terms, and it may well be any of those terms would justify what OMB is proposing. And because of that, I'm not going to lead you through their arguments. I'm going to lead you through mine on the grounds they are more interesting. Based on the text of the Information Quality Act, one can dispute OMB's claim that it has the authority to require peer review. First, consider the text. The act does not explicitly require or even authorize peer review. NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Moreover, although the act imposes a number of specific duties on OMB, peer review is not one of them. So, I think there are some textual arguments here that peer review may not be required. OMB may not have the authority to require it. But I've got to tell you the lynch pin of my argument would only work for those judges who would look at legislative history. And the legislative history is this. On several occasions Congress failed to pass legislation that would have explicitly mandated peer review of the type OMB is proposing. So, to conclude, that the Information Quality Act authorizes OMB to impose peer review, you would have to conclude that Congress changed its mind in a rider hidden in an appropriations bill that no one knew about in Congress except the representative who sponsored it, and Jim Tauzey who wrote it. So, I think there is a decent argument to be made here that Congress has spoken on the issue of peer review, and it didn't require it. As I say, arguments can be made the other way. That's what we lawyers do. And OMB also has a set of arguments, perhaps equally good, and we'll se. Finally, let me try to do this in just a couple of minutes. I'm not going to use my slides, because I want to NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. do it quickly, and stay within my time limit. Even if OMB has the legal authority to impose peer review, my second argument would be that authority only extends to information which is published in reports, or put on the Web, and not information which is disseminated during rulemaking. And the gist of my argument is this. Congress, in the Information Quality Act said look, we want agencies that are distributing information to establish an administrative mechanism to vet that information. My argument is we already have that mechanism in rulemaking. Goodness knows we have all kinds of procedures in rulemaking to vet information. Indeed, rulemaking provides more procedural protection to vet information than the Information Quality Act. It's pretty clear to me that what Congress was talking about was information on the Web, and information in government reports, because we had no mechanism to do that, none whatsoever. Congress looked around and said, gee, we don't have any mechanism for this other stuff. We ought to get one going. If Congress meant to cover rulemaking as well, it would be superfluous. Congress is saying to rulemaking agencies, oh, by the way, you need to establish a mechanism NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. to vet data. They already have a mechanism to do that. If you look at the debate leading up to the Information Quality Act, it was all about information in reports and on the Web, and I think that's what Congress meant. In any case, I think someone will mount an argument that it doesn't apply to rulemaking at all. Agenda Item: Questions DR. MESERVE: Thank you, Mr. Shapiro, for the very interesting comments. Let's see if there are some questions. PARTICIPANT: Would you comment on the relevance of American Society of Dermatology? And as a second question, isn't there a strong argument that if we adopt peer review for significant regulatory information, we are turning it into an operational function, because all they are doing is evaluating information. They are not making any policy recommendations. PROF. SHAPIRO: Yes, I have heard that argument, and there are sort of two levels of it. Again, there are lots of arguments here, and lots of them are pretty good, so that's why it's a little hard to predict how all of this will come out. One argument is that the Federal Advisory NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Committee Act applies to advice given to agencies. So, are these peer reviewers really advising the agency about something? It would seem to me yes, they are. They are advising the agency about the quality of the study, or the quality of the science being reviewed. So, the next level of question is does it make a difference for purposes of the Federal Advisory Committee Act, that hopefully we'll be able to separate out the policy issues and the science issues. Now, you have heard speakers say it's going to be a little tough to do that, but let's assume for the sake of this question you can separate that out. At one time GSA regulations seemed to suggest that FACA might not apply outside the context of advice about policy. They have issued a new set of regulations last year. That provision seems to have gone away. And then secondly, I'm not quite sure that's an accurate interpretation of the act itself, because you could look through the Federal Advisory Committee Act, and it doesn't say anything about policy. It says advice. So, we are back to my first level. PARTICIPANT: In fairness to OMB, they had also cited this Executive Order 12866 as one of their NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. authorities? What does that do exactly? I understand it coordinates some issues of science across agencies, but is that an authority? Is that anything that they can cite in here? PROF. SHAPIRO: It could well be. That is tied back again to this paperwork act. OMB and OIRA in particular -- OIRA was created in fact to be the agency that implements the paperwork act. And the paperwork act has lots of language about both controlling the amount of information that people have to report back to the government, that's its primary regulatory function. But there is some secondary language in the paperwork act that at least arguably gives OIRA the kind of role they envision for themselves here. So, we lawyers will be arguing about language for which there is no obvious meaning, but they certainly have that set of arguments open to them, absolutely. PARTICIPANT: Earlier this morning OMB went on record saying they did no see public comment as a substitute for peer review. To me, it seems that public comment is the ultimate form of peer review, because it gives everyone a chance to comment simultaneously. Any thoughts on that? NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. PROF. SHAPIRO: Yes, in one sense. It sort of depends what you mean by peer review. So, we are back to Dr. Kennedy, the very first question this morning. There is peer review, and there is peer review, and there is peer review. So, what do we mean here? I was on a panel last week, and one of the speakers pointed out that one of the Australian safety agencies does an Internet peer review, where they put the study up on the Web, and they invite all the scientists who want to, to do individualized peer review. And I said to this person, oh, that's rulemaking. And he said, no it's not, it's peer review. DR. MESERVE: Any other questions? If not, please join me in thanking Prof. Shapiro. [Applause.] Agenda Item: Closing Remarks - Richard A. Meserve, President, Carnegie Institution of Washington DR. MESERVE: The agenda for this session indicates that I am to provide closing remarks. I have to tell you that I don't need peer reviewers to teach me the lesson of the waning attendance that I'm observing in the audience, so I'll be very brief. Our forum for discussion today has obviously been NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. aimed at providing an opportunity for the full and open discussion of the OMB proposal. I hope that all of you found this discussion to be useful. I personally have found it be both rich and very interesting. As I think all of you know, comments on the bulletin are due by the public on December 15, and by agencies on January 16. Although, we of necessity today have focused largely on some of the implications of the proposal, and particularly on some of the challenges it presents, I think that comments will no doubt be most useful to OMB if they not only raise issues, but also propose some solutions. The transcript of today's session will be available on the Academy's Web site in early December. I understand that the Academy staff will be sending an e-mail to all of you who have registered indicating when the transcript is available. And we hope that both this session and the transcript will be useful to people in submitting their comments. Let me just close by thanking the audience for their very thoughtful questions. And I especially want to thank all of the speakers for their very insightful comments. You did a marvelous job at reviewing the NOTE: This is an unedited verbatim transcript of the Peer Review Standards for Regulatory Science and Technical Information Workshop, held on November 18, 2003, prepared by CASET Associates and is not an official report of The National Academies or of the Science, Technology and Law Program. Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. implications of the bulletin. With that, we are adjourned. [Whereupon, the meeting was adjourned at 5:00 pm.]