Document kmeyVVyebkQOYZNjnxynpmzyO

CHAPTER 1 Review Process and Publication Decision ABSTRACT The authors describe the roles of the reviewer within common types of review processes (board review and pool review) used at scholarly journals. In addition to giving background information about the types of review processes used at journals, they discuss how journals identify potential reviewers and select specific reviewers for a manuscript. They explain the types of review forms used by various journals and how reviewers' comments on these forms are used in the editorial decisions made by the journal editors. Finally, the authors describe the role of reviewers' comments in guiding the revision and final editing of manuscripts. Acad. Med. 2001;76:911921. Review Process Judy A. Shea, Addeane S. Caelleigh, Louis Pangaro, and Ann Steinecke The review process starts when a manuscript is received at the editorial office. The first decision focuses on whether the paper should be sent for external review. Journals vary in their practices, but typically an editor reads the paper and makes an early decision for review or no review based on general factors such as the mission of the journal, the match between the audience for the paper and the audience for the journal, the appearance of obvious fatal flaws, and the balance between the content of the paper and other recent publication decisions. Sometimes redundancy is viewed as a benefit, as when a paper provides confirmation of a new idea or method; sometimes it detracts from the perceived contribution because it is seen as being unnecessary repetition of known information. The central reason that a review decision is made at this stage is that reviewers must be conserved. In almost all cases reviewers are uncompensated for significant amounts of time, ranging from a few hours to ``one day'' per manuscript.1 Their expertise is an asset given freely-- ``compensation'' often amounts to annual recognition in a published list of reviewers--and editors try hard not to burden them unduly. If the initial decision is to send the manuscript out for review, reviewers are selected (see the next article in this chapter), and reviews are requested. Journals have different types of review systems. Two basic ones are ``board review'' (all the reviewers are members of a review board or editorial board, which has the responsibility of reviewing the journal's manuscripts) and ``pool review'' (a large number of specialists of many types join the journal's pool of potential reviewers).2,3 Most journals use a mixed form that relies heavily on pool review for the initial reviews but also uses board review to add another review or resolve discrepancies. The exact mechanism by which reviewers are selected also varies from journal to journal. In some cases authors are asked to nominate potential reviewers for their articles, and reviewers are sometimes asked to nominate potential reviewers to increase the reviewer pool. The reviewers may be chosen because of either their content or methodologic expertise, and the mix of reviewers for a paper will depend on the type of review the editor has established as standard for the journal. The pool may very well exist as an electronic list of potential reviewers. In this list, in addition to current contact information and areas of interest and expertise, journals may track other types of data relevant to the review process. For example, it is quite common to track how often a reviewer is asked to review as well as the timeliness and helpfulness of the reviews. Numerous other elements of the review process vary from journal to journal, depending on the journal's history and resources. Journals have developed their own strategies to assure, as much as possible, that the review process is timely and fair for authors and reviewers alike. For example, some journals contact potential reviewers (via phone or fax) ahead of time to ask them if they are able to review a manuscript in a specified amount of time. A few send the review and ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001 911 associated materials without contact, based on prior agreement between the journal and its reviewers. Journals differ in the amounts of time given to reviewers for doing reviews, the reminders they use when reviews are overdue, and the process for giving up on unresponsive reviewers. The numbers of reviews solicited also vary. Many journals request reviews from two or three external reviewers, while others use more. One decision the editor must make is whether to request more reviews than are needed, in the hope that at least a minimum number will be completed in a reasonable time. Alternatively, the policy may be to request a small number and then to follow up with additional reviewers when there is disagreement or unanticipated delay. One element of the review process that is often discussed and that has received research recently is ``blinding,'' also called ``masking.'' Blinding has to do with concealing the identity of reviewers, authors, or both. The traditional and most common form is to conceal the reviewer's identity from the author; the less common is the effort to conceal also the author's identity from the reviewer. Conclusions from studies on the topic are mixed. Originally the thought was that masking the author's identity would lead to fairer reviews,4,5 and the process was endorsed by a small but growing minority of journals. However, other work suggests that masking author identity does not improve the quality of reviews.6 It is often felt, especially by reviewers themselves, that blinding is not uniformly effective because reviewers are often still able to identify the author.7,8 This view is not surprising, since reviewers are often experts who are familiar with current work in the field. Although the quality and the quantity of the evidence are not strong, the research to date seems to show that reviewers are not as accurate as they think they are in guessing the author's identity.9 On the more important point, a growing body of research seems to give little evidence that masking the author's identity affects the publication decision. Still, it is a process that continues to be studied, if for no other reason than that authors consider it more fair to themselves. Currently, most journals do not blind the reviewers to the authors' identities. There is also discussion about the desirability of revealing the reviewers' identities to the authors. Supporters believe that comments will be more fair and balanced when the reviewers publicly identify themselves, and that this openness will help to define the review process as a collegial one from which both the author(s) and reviewer(s) can benefit and enter into professional dialogue.7 One study showed that having reviewers sign their reviews supports the open review of manuscripts.10 Informally, however, it is fair to say that the balance of reviewers' sentiment is against the process. Power imbalances could make it difficult for some reviewers (junior faculty members, for example) to be appropriately critical if the review had to be signed, and it would be too hard to offer critical comments about the work of a colleague or friendly acquaintance. Few if any educational research journals require reviewers to sign their reviews, although some reviewers sign as a matter of principle. There is another form of masking that is very rarely used but worth noting--to blind the editor to the identity of the author, the reviewers, or both until a publication decision is made. This process would be based on the same policy considerations that would lead to blinding reviewers to the identity of authors--to limit positive or negative influence on the decision process. After reviewers are selected, they receive, usually through the mail but now increasingly electronically, a packet that contains the manuscript, review forms, instructions, and sometimes associated materials (e.g., instructions to authors, sample reviews). As noted in the article on review forms later in this chapter, the forms vary in their content and the types of issues that reviewers are asked to address. One commonality is that the reviewer is almost always asked to provide text comments, perhaps separating major from minor issues. Sometimes reviewers are asked to comment sequentially on each major section of the manuscript. Other tasks reviewers may be asked to do include completing rating scales on topics such as clarity, importance, and methodologic soundness. What the reviewer is really doing is assessing whether the different parts of the manuscript meet expectations--this is a measurement task. The measurement task is usually followed by a complementary task--making a recommendation about publication. A recommendation is a judgment based on a reviewer's overall assessment of the worth of the manuscript. (The judgment part of the process is considered more extensively in Chapter 3.) Naturally, this recommendation item will vary depending on the number of categories the journal allows (e.g., accept/reject; accept/ provisional accept/revise/reject; accept/revise/reject). Reviewers are sometimes also given the option of writing confidential comments to the editor. Most journals allow reviewers to suggest that a manuscript be referred for a statistical consultation; specification of other types of needed methodologic or technical expertise is rare. From the reviewer's perspective, several things might be done to make the review process more pleasing after the editorial office's initial efforts to screen out manuscripts that are unlikely to be published and to match the manuscript's content to the reviewer's interests. Giving ample time (e.g., three weeks) is important, and reminders are helpful. Forms should be easy to complete and return. Some journals ask that all materials be returned, while for others it is sufficient to mail or fax the review form and then destroy all other materials. If reviewers must mail back some materials, prestamped and pre-addressed return envelopes should be provided. Some journals are conducting parts or all of the re- 912 A C A D E M I C M E D I C I N E , V O L . 7 6 , N O . 9 / S E P T E M B E R 2 0 0 1 Figure 1. Flow chart of the general review and publication process for a manuscript submitted to a peer-reviewed journal. view process electronically. There are no data yet on reviewers' reactions to electronic reviews. In addition to the process of deciding how (and whether) to solicit reviews for a manuscript, a number of procedures might be considered to support or round out the general review process. For example, the editor and editorial staff must decide whether the reviewers' comments will go to the author, and if they do, whether the editorial staff edit them or send them as received. Two advantages to sending them as they are received is that it reduces the summary and distillation process for the staff, and it can give the author a sense that he or she is getting as much feedback as possible. On the other hand, reviewers' comments often conflict with one another or opinions of the editor, and sometimes reviews contain unhelpful or even inappropriate comments. There is no right or wrong answer--it is simply yet another matter that must be considered. Other issues about which editors (and their editorial boards) make decisions is whether to inform reviewers of the publication decision, either by copying them on the communication to the author or by sending some other kind of summary; whether to involve the same reviewers if revisions are required; and whether to routinely solicit reviewers' opinions about all or parts of the review process. Obviously, there are a multitude of ways these issues can be addressed to create ``the review process'' at any particular journal. (See Figure 1 for a flow chart of the general review and publication process.) Once the reviews are available, the editor must make a publication decision. The complex ways in which that decision is reached are discussed in the article on the publication decision later in this chapter. REFERENCES 1. Yankauer A. Who are the peer reviewers and how much do they review? JAMA. 1990;263:133840. ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001 913 2. Weller AC. Editorial peer review in US medical journals. JAMA. 1990; 263:13447. 3. Hargens LL. Variation in journal peer review systems: possible causes and consequences. JAMA. 1990;263:134852. 4. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. JAMA. 1990;263:13716. 5. Fisher M, Friedman SB, Strauss, B. The effects of blinding on acceptance of research papers by peer review. JAMA. 1994;272:1436. 6. Justice AC, Cho MK, Winker MA, et al. Does masking author identity improve peer review quality? A randomized control trial. JAMA. 1998; 280:2402. 7. Lock S. A Difficult Balance: Editorial Peer Review in Medicine. Philadelphia, PA: ISI Press, 1985:1223. 8. Cho MK, Justice AC, Winker MA, et al. Masking author identity in peer review: what factors influence masking success? JAMA. 1998;280: 2435. 9. Rosenblatt A, Kirk SA. Recognition of authors in blind review of manuscripts. J Soc Service Res. 1980;3:38394. 10. Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998;280:23740. 11. Knoll E. The communities of scientists and journal peer review. JAMA. 1990;263:13302. RESOURCES Armstrong JS. Research on scientific journals: implications for editors and authors. J Forecasting. 1982;1:83104. Bligh, J. What happens to manuscripts submitted to the journal? Med Educ. 1998;32:56770. Chubin DE, Hackett EJ. Peer review and the printed word. In: Chubin DE, Hackett EJ. Peerless Science: Peer Review and U.S. Science Policy. Albany, NY: SUNY Press, 1990. Freedman DX. Editorial processes, safeguards, and remedies. In: Miller DJ, Hersen M (eds). Research Fraud in the Behavioral and Biomedical Sciences. New York: John Wiley & Sons, 1992. Hargens LL. Variation in journal peer review systems: possible causes and consequences. JAMA. 1990;263:134852. Knoll E. The communities of scientists and journal peer review. JAMA. 1990;263:13302. National Research Press. Part 4: Responsibilities. In: Publication Policy. http://www.monographs.nrc.ca/cgi-bin/cisti/journals/rp/rp2 cust e?pub policy. Accessed 6/5/01. Weller AC. Editorial peer review in US medical journals. JAMA. 1990; 263:13447. Selection and Qualities of Reviewers Addeane S. Caelleigh, Judy A. Shea, and Gary Penn The quality and usefulness of a journal rest on the quality of research submitted, its reviewers' critiques, and its editor's judgment. The reviewer is the vital heart of the process, and one of the editor's primary responsibilities is to create and maintain a highquality, productive group of reviewers. To do so, journals are concerned with several areas of reviewers' performances: their knowledge, judgment, constructiveness, ability to write clearly, and willingness to work within the journal's guidelines. In addition, in terms of selecting reviewers for individual papers, editors are responsible for having and enforcing fair policies about how reviewers are selected and for seeing that the natural biases inherent in the review system are countered as much as possible. Journals recruit and organize their reviewers for assignment to manuscripts. For example, editors take into account reviewers' expertise and availability when asking them to review. Also, they take various approaches to monitoring each reviewer's performance, often by rating each review and sometimes by conducting special studies. Journals identify possible reviewers through contacts at professional meetings, personal acquaintances, editorial boards, literature searches, society membership lists, and manuscript bibliographies.1 Some journals encourage authors to suggest reviewers for their manuscripts. Occasionally, potential reviewers seek out journals with unsolicited offers to help. A few journals still require that reviewers be personally known to the editor, but most now use large reviewer pools, and the editor and reviewer may never have met. A factor contributing to the growth of large pools of reviewers is the availability of computer software that now makes it easy to keep detailed files of information about reviewers' expertise, experience, and special interests. Selecting reviewers for an individual manuscript can begin with matching the manuscript topic to a reviewers' area of expertise, but can go farther and ensure that, for example, technical expertise is augmented by particular professional experiences. A reviewer's expertise must be complemented by judgment in order to be useful to a journal. Therefore, the journal is concerned about whether the reviewer can make balanced judgments, keep a sense of proportion when assessing the strengths and weaknesses of a research project, apply appropriate standards, and give definite opinions. To give these opinions, the reviewer must be able to express complex ideas clearly--and, if possible, concisely--for both the editor and the author, whose needs are not always the same. The reviewer serves as adviser to the editor but also as teacher to the author, especially to the less experienced. In 914 A C A D E M I C M E D I C I N E , V O L . 7 6 , N O . 9 / S E P T E M B E R 2 0 0 1 writing critiques, the reviewer should use the criticisms in the review as opportunities to tutor. In reviewing excellent reports by highly experienced researchers, reviewers serve as peer assessors; in contrast, when reviewing weaker reports by novice researchers, they serve very much as teachers. Further, many journals put considerable emphasis on reviewers' giving constructive criticism to authors and expect the reviewers to give their opinions (even, when necessary, stern judgments) in a collegial spirit accompanied by suggestions for improvement. At a minimum, all journals should require civility in reviewers' comments. Journals look for different attributes and qualities in reviewers, depending on the nature of the journal, the role of the reviewer, and the editor's (and sometimes the sponsoring society's) policies.2,3 Unfortunately, there is little empirical evidence to guide a journal in establishing a profile of the good reviewer. Studies at clinical journals to assess the characteristics of peer reviewers who produce good reviews have reported conflicting findings. Although these studies use slightly different definitions of quality, the core categories used in assessing reviews and reviewers have been very similar. Stossel, in the earliest study, found that reviewers with lower academic or professional status produced better critiques of papers than the higher-status reviewers did--and were less likely to refuse to review.4 Several years later, Evans et al. reported that the reviewers who produced the best reviews were less than 40 years old, were known to the editors, and were from highly respected (highly rated) academic institutions.5 A few years later, Black et al. found the only significant factor associated with higher-quality ratings by both editors and authors to be reviewers trained in epidemiology or statistics; reviewers' younger age was an independent predictor of the editors' (but not the authors') ratings, while reviews by members of an editorial board were rated of poorer quality by authors (but not by editors). The ratings increased with the time spent on a review--up to three hours, but not beyond.6 The variety of characteristics, or lack of characteristics, that can be relied on to identify high-quality reviewers lead Fletcher and Fletcher to conclude, in a recent overview, The results of these studies suggest that editors should not have fixed views of what kinds of reviewers might return good reviews. Because the characteristics of good reviews might vary from one setting to another, it seems editors should continue the common practice of grading their own reviewers but recognize that this is an imperfect predictor of their future performance.7 Previous experience with a reviewer can be the most useful guide. Many journals evaluate the reviews they receive, and they refer to the quality ratings for past reviews when considering a reviewer for a manuscript. They also use the ratings, of course, when deciding whether to retain the reviewer. Many editors use simple rating systems that serve primarily as rough triage systems, but some are trying to develop more sophisticated, even standardized instruments.810 Editors study reviewers' effectiveness in specific aspects of review, as well as the overall effectiveness of the review process. Baxt et al. examined how well reviewers at a clinical journal could identify major and minor flaws in a manuscript sent for review.11 Callaham et al. (who are in the same group, at the same journal) then studied the reliability of the editors' subjective ratings of review quality and found them to be moderately reliable and correlated with the reviewers' abilities to report manuscript flaws.12 Journals also are aware of the intrinsic, unconscious biases in the review process, and some research is beginning to be done on this complex topic as well.1315 The biases may be in terms of intellectual positions, preferences for positive outcomes in research, and personal social convictions, as well as the oft-cited biases having to do with ethnicity, nationality, gender, and status. Reviewing consumes a great deal of reviewers' valuable time, yet apart from acknowledgements or thanks from journals it is generally uncompensated. Therefore, it is essential that a journal's reviewer pool be regularly replenished. First, journals need to avoid sending too many papers to the best reviewers, so that these reviewers will not burn out or refuse to review because of the workload. Also, new, qualified reviewers must be added as other reviewers retire or need to take leave from reviewing. Finally, if a journal is successful in attracting more papers, it will need more reviewers. Similarly, if it decides to use more reviewers per paper as a way to improve the review process, it will need more reviewers. Reviewers should be encouraged to involve junior colleagues who would make good reviewers, and most journals have procedures for sharing a reviewing assignment. These colleagues could, when they are ready, be recommended to a journal as full reviewers. Some journals are responding to the obvious need for reviewer training and for more and better feedback to reviewers. The Annals of Internal Medicine has a very supportive program, for example, including an annual report to each reviewer showing where that reviewer's performance falls in the spectrum of the journal's entire pool of reviewers. REFERENCES 1. Weller AC. Editorial peer review in US medical journals. JAMA. 1990; 263:13447. 2. Wilson DG. Ethics for authors and editors. Learned Publishing. 1999; Oct:7. 3. Polak JF. The role of the manuscript reviewer in the peer review process. AJR: Am J Roentgenol. 1995;165:6858. ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001 915 4. Stossel TP. Reviewer status and review quality: experience of the Journal of Clinical Investigation. N Engl J Med. 1985;312:6589. 5. Evans AT, McNutt RA, Fletcher SW, Fletcher RH. The characteristics of peer reviewers who produce good-quality reviews. J Gen Intern Med. 1993;8:4228. 6. Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA. 1998;280:2313. 7. Fletcher RH, Fletcher SW. The effectiveness of editorial peer review. In: Godlee F. Jefferson T (eds). Peer Review in Health Sciences. London, U.K.: BMJ Books, 1999:4556. 8. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review: a randomized trial. JAMA. 1990;263: 13716. 9. Feurer ID, Becker GJ, Picus D, Ramirez E, Darcy MD, Hicks ME. Evaluating peer reviews: pilot testing of a grading instrument. JAMA. 1994; 272:98100. 10. van Rooyen S, Black N, Goodlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol. 1999;52:6259. 11. Baxt WG, Waeckerle JF, Berlin JA, Callaham ML. Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Ann Emerg Med. 1998;32(3 Pt 1):3107. 12. Callaham ML, Baxt WG, Waeckerle JD Wears RL. Reliability of editors' subjective quality ratings of peer reviews of manuscripts. JAMA. 1998;280:22931. 13. Gilbert JR, Williams ES, Lundberg GD. Is there gender bias in JAMA's peer review process? JAMA. 1994;272:13942. 14. Ernst E, Resch KL. Reviewer bias: a blinded experimental study. J Lab Clin Med. 1994;124:17882. 15. Caelleigh AS, Hojat M, Steinecke A, Gonnella JS. Effects of reviewers' gender on assessment of a gender-related standardized manuscript [abstr]. 2001 Peer Review Congress, Barcelona, Spain, to be presented September 1921, 2001. RESOURCES Siegelman SS. Assasins and zealots: variations in peer review. Special report. Radiology. 1991;178:63742. Review Form Ann Steinecke and Judy A. Shea Areview form is the instrument that a reviewer uses when he or she assesses the quality of a manuscript. A review form should make it easy for the reviewer to supply both the editor and the author with detailed information about the strengths and weaknesses of a manuscript. The editor will use the information to help decide whether or not to publish the article; in some cases, an editor relies on nothing other than the information contained in a manuscript's reviews. The author can use the information from a review form to better understand the decision made by the editor and to learn about the strengths and weaknesses of the manuscript and methods for improvement. In this way, the review form is a tool for reviewers to communicate one set of information that can be used for different purposes by the editor and the author. No standard review form is used among the journals in any scientific field; further, there are no studies that demonstrate the best format for review forms. The features of review forms vary widely depending on the type of information each editor seeks from reviewers--some request narrative comments alone,1 while others combine checklists, global ratings, and narrative comments.2 Most are composed to solicit the same general categories of information: the importance of the research question, the quality of the research, the justifiability of the conclusions, and the appropriateness of the manuscript for publication in the journal. In the description and discussion below, the specific examples are taken from research journals in academic medi- cine and the health professions, but they could have been taken from any research field in the biosciences and social sciences. Appendix 3 gives examples of forms from both fields. Different response formats (global ratings, checklists, narrative comments, etc.) offer reviewers different advantages, so they are often used in combination. Global rating scores, such as Teaching and Learning in Medicine's four-point scale (``Should this paper: be published in TLM, be published with minor modifications, be reconsidered only after major changes, be rejected'') are efficient and enable the editor to quickly rank a manuscript. However, global rating scores alone do not assist the editor in making a judgment by providing a nuanced discussion of the manuscript's strengths and weaknesses, nor do they provide the author with detailed comments. For these reasons, global ratings are often not shared with the author. Checklists direct the reviewer to evaluate specific aspects of a study, as in this example from Medical Education. Methods Are these clearly explained? Are the methods appropriate? Is the sample size adequate? Is the sample representative? Are the subjects fully described? Is the statistical method appropriate? Y/N Y/N Y/N Y/N Y/N Y/N 916 A C A D E M I C M E D I C I N E , V O L . 7 6 , N O . 9 / S E P T E M B E R 2 0 0 1 Some journals used a dichotomous (yes/no) scale, with or without comments, while the review forms of others use rating scales for every item (e.g., unacceptable, poor, satisfactory, excellent). For reviewers, the benefits of a checklist include the ready identification of the key elements of a manuscript that should be judged for soundness, particularly those having to do with research method. Further, the reviewer's evaluation of these key elements often provides a source of positive feedback for authors on the review form. Whether the ``direction'' that checklists provide to reviewers in evaluating manuscripts ultimately leads to a more complete evaluation of a manuscript's soundness is open to debate. While one study found that only 25% of reviewers identified methodologic flaws in their narrative comments,3,4 the risk is probably as great that reviewers will not identify methodologic flaws in their checklist responses. However, in combination with an opportunity to discuss a manuscript's strengths and weaknesses in narrative form, checklists can be particularly valuable for identifying significant problems within a manuscript. Most journals ask reviewers to provide some form of narrative comment, through open-ended questions or a section for free response. Narrative comments allow reviewers to respond with the greatest level of detail. The Academic Medicine review form asks reviewers to answer a series of brief open-ended questions such as 2. Please explain why the research design and analysis of the data is or is not appropriate for the research question. and 4. Given the topic and research question, who in the academic medicine community would be interested in reading this paper? Who (if anyone) should read this paper? These questions free reviewers from preformulated responses and allow detailed critique. Alternatively, some journals simply provide reviewers space to add any comments they wish to make about the manuscript. However, free text comments in either format do present several drawbacks for reviewers. They require more time and effort to complete than do global ratings and checklists. Review forms that ask for narrative comments tend to be longer because they need to leave the reviewer with enough space to record comments. Many reviewers respond to questions with hand-written comments, which, with length, can be increasingly difficult for the editor and author to decipher. Alternatively, when reviewers type their comments on separate sheets of paper the added step may discourage timely completion of the review. The British Medical Journal is one journal that has solved these issues by doing away with the review form altogether. Instead, they send reviewers only a copy of the instructions for review and have reviewers return their comments on stationery. Reviewers should be aware of two ways that review forms can inadvertently affect their reviews. First, when completing a review using a form that combines response formats, reviewers may inadvertently contradict what they have indicated on the checklist with what they have written in their comments. For example, a reviewer's global ratings for a manuscript may be positive for publication, whereas narrative comments may elucidate a fatal flaw. When narrative comments are requested, reviewers should be careful to provide the editor and the author with more nuanced and detailed information about the manuscript, explain the severity of the problems they have detected in the manuscript, and estimate the probability of correcting the problem with a revision. The second problem with review forms that reviewers should be mindful of is that studies have revealed reviewers' biases based on the direction, significance, or controversial nature of findings.5,6 Review forms that offer multiple response formats can reduce the risk of a review form's overly directing a reviewer's response. An ill-phrased question can unintentionally lead a reviewer to a biased response. To help diminish bias, reviewers should scrutinize questions for partiality and take advantage of other modes of response to fully explain their responses. There is ongoing debate about whether or not review forms should be tested for inter-rater reliability. Many studies have demonstrated a lack of agreement between reviewers' recommendations, with correlations averaging .3.7 Editors of clinical specialty journals, especially those who rely most heavily on reviewers' recommendations in decision making, value high inter-rater reliability,8,9 and tested review instruments have been described.10 But what is desirable in reviews of clinical manuscripts may not be desirable for medical education manuscripts. Although review forms for these disciplines share many characteristics and aims, particularly for assessing study design and method, medical education editors may prefer forms that allow competing viewpoints, which some suggest strengthens the quality of the decision-making process.11 Finally, in addition to providing reviewers with enough response formats, including a clear statement of the journal's purpose and instructions with the review form can help reviewers understand their responsibilities, the information the editor is seeking, how that information will be used in decision making, and the kinds of responses that are useful to authors. It has been demonstrated that with mailed questionnaires, the attractiveness of the questionnaire's format, the questionnaire's length, and the ease of filling out and returning the questionnaire can affect response rates.5 To encourage reviewers to respond, editors should consider these ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001 917 issues when designing review forms. One recent example of journals' taking convenience for the reviewer into consideration is the increasing use of e-mail- or Web-based review forms to improve response rates and times. By using a standard review form for its research manuscripts and by giving reviewers a choice of response formats (checklists, global ratings, narrative comments for authors and/or editors) a journal can create a review form that can inform both the editor and the author with sufficient and appropriate responses. REFERENCES 1. Bligh J. What happens to manuscripts submitted to the journal? Med Educ. 1998;32:56770. 2. Nyleena M, Riis P, Karlsson Y. Multiple blinded reviews of the same two manuscripts. JAMA. 1994;272:14951. 3. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990;263:13859. 4. Bordage G. Reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports. Acad Med. 2001;76:88996. 5. Armstrong JS, Hubbard R. Does the need for agreement among reviewers inhibit the publication of controversial findings? Behav Brain Sci. 1991;14:1367. 6. Fiske DW, Fogg L. But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. Am Psychologist. 1990;45:5918. 7. Cullen DJ, Macaulay A. Consistency between peer reviewers for a clinical specialty journal. Acad Med. 1992;67:8569. 8. Cicchetti DV. The reliability of peer review for manuscript and grant submissions: a cross-disciplinary investigation. Behav Brain Sci. 1991; 14:11935. 9. Cho MK, Bero LA. Instruments for assessing the quality of drug studies published in the medical literature. JAMA. 1994;272:1014. 10. Eckberg DL. When nonreliability of reviews indicates solid science. Behav Brain Sci. 1991;14:1456. 11. Selitz C, Jahoda M, Deutsch M, Cook SW. Research Methods in Social Relations. New York: Holt, Rinehart & Winston, 1959. RESOURCES For sample review forms from a variety of scholarly journals, see Appendix 3. Publication Decision Judy A. Shea and Addeane S. Caelleigh The publication decision is the last step of the review process. Journals vary in the processes they use for making publication decisions, and each journal should make public how its particular process works. Admittedly, it is neither feasible nor desirable to detail the intricacies and unique responses to individual submissions, so the way many journals make the process public is to publish a description of it, often annually, within a succinct summary of their review process. It is important that every communication of a publication decision include a specific and clearly stated appeal process for authors who are dissatisfied with the outcome. For many journals, the publication of the description of the review process coincides with publication of their submission requirements and annual statistics (e.g., number of manuscripts submitted, acceptance rate). The principal actor in the publication decision process is the editor. While few editors may defer to reviewers' comments for the publication decision--uncharacteristically, one survey of North American medical journal editors did find that almost half of the editors responding based their decisions solely on reviewers' comments1--most journal editors rely on reviewers' comments for advice and reserve the final decision for themselves. This is a major area of consensus among health professions journals and the leading jour- nals of most fields and disciplines. Reviewers make recommendations to editors--they do not have votes. (Journals' instructions to authors will often have statements spelling out the general basis for decisions.) This is a good place to clarify the use of the term ``editor'' as a singular noun. While this applies to the organizational and decision-making structure at some journals, it is important to realize that the exact organization of a journal may vary, and several people may have decision-making responsibilities. For example, each section of a journal may have a separate editor who functions and makes decisions independently. Other journals are structured such that associate editors are assigned their ``own'' manuscripts, and they make publication recommendations that are discussed and decided among the larger group of many associate editors. Most journals have editorial boards, but in general these bodies function as overseers to monitor the activities of the journal from a distance and are seldom involved in daily activities. Clearly, one of the most important ways in which journals vary is in setting the policy that specifies exactly who has decision-making authority. Is it an individual, or a group? The decision about whether or not to publish a manuscript that has been sent out for review is rarely obvious. Occasionally reviewers may agree that a manuscript should not be considered further--that it clearly should not be pub- 918 A C A D E M I C M E D I C I N E , V O L . 7 6 , N O . 9 / S E P T E M B E R 2 0 0 1 lished. Very rarely will reviewers unanimously agree that a manuscript clearly should be published directly as written. Most often, reviewers recommend some middle strategy (e.g., revise) or they offer conflicting recommendations for and against publication. Regardless of the agreement among reviewers, however, the editor must make the publication decision. How does an editor make a decision after reading the reviews--how does he or she balance the recommendations, the pros and cons, from reviewers? The editor must consider each reviewer's opinions, suggestions, and comments and then combine the input from several reviewers before reaching a decision. Are all reviewers' comments given equal weight? Probably not. Those from known experts are weighed more heavily. Remember that the editor often knows the general field well but is not and cannot be expected to be a subject expert for every manuscript. Also, reviews that are constructive, more than a couple of sentences long, and original in the issues that are mentioned receive special attention. In short, an editor is looking for a review that is thoughtful, helpful, and specific. It is important to note, too, that the editor must balance reviewers' suggestions (arguably the most important single part of the process) with many other factors. A manuscript's originality (or its characteristics at the opposite end of the spectrum, redundancy) is one very important consideration. Depending on the manuscript's topic, either feature may be desirable. Other issues that weigh in the decision include length (sometimes short is good; sometimes long is better), a topic's relevance, and what the journal has published recently. These issues are often secondary to the external reviews, which are primarily concerned with quality, but they can be important either to justify a publication decision or to direct future revisions. If an excellent article is rejected because it is too similar to one that was just published, steps can be taken to make sure that the author hears both points: excellence but redundancy. Similarly, an editor can work with an author to shorten or expand a manuscript. In sum, the decision-making process is highly complex, multifactorial, and unique for each paper. It is subjective, but it is neither capricious or uninformed. In many ways, a paper is in competition with all other good papers being processed at the same time. Often the editor must make only a few selections from a large potentially publishable field. This is a very important point that may be overlooked--a manuscript that is generally quite acceptable may be rejected primarily because it is not quite as ``strong'' (e.g., interesting, methodologically sound, different) as are current competitors. Outside of stating how many reviewers are typically requested and making public the review form that they use, there is little about the process that can be succinctly generalized except to say that reviewers' suggestions are taken seriously. Reviewers' comments are almost always the most important ingredient in the process. If the comments agree, they are usually followed. Unfortunately, agreement is rare, and it is in these instances that editors must rely on their judgment, built from experience. So, the editor makes a decision. (In reality, the decision is often not an absolute--some magnitude of revision of the manuscript is required. Those negotiations are discussed in the next article in this chapter.) The final step in the process is that ``the decision'' is conveyed to the author in writing. Several parts of this process are relevant to reviewers and, again, journals differ in how they handle it. Are the reviewers' comments forwarded to the author? If so, are they sent as received or does the editor pick and choose among the comments and compile them in the decision letter? Do the reviewers see a copy of the letter sent to the author? Do they have the benefit of seeing the other reviewers' comments? If particular comments from a reviewer are unusual (e.g., highly technical), will the reviewer be asked to interact directly with the author? Finally, editors make decisions in other ways that are not so obviously tied to the review process. Journals differ in their involvement in editorial work. Some journals publish a paper more or less as it was submitted once the author adequately addresses queries (see ``Presentation and Documentation'' near the end of this chapter). In such a case, poorly written papers about good studies are unlikely to make the cut. Other journals go to great lengths to restructure a manuscript to improve it, highlighting some parts and minimizing or removing the others. Some editors will review each manuscript with an eye towards identifying the usable ``kernel.'' Negotiations with the author determine how much a manuscript can ultimately change, or what part of it can be published. In sum, while journals have well-defined policies and flow charts designed to support efficient and fair publication processes, decisions about any one manuscript have to be viewed within the larger context. Editors are usually more than willing to communicate the important elements in any particular decision, either in writing or verbally, to an author. REFERENCE 1. Wilkes MD, Kravitz RL. Policies, practices, and attitudes of North American medical journal editors. J Gen Intern Med. 1995;10:44350. RESOURCES National Research Press. Part 4: Responsibilities. In: Publication Policy. http://www.monographs.nrc.ca/cgi-bin/cisti/journals/rp/rp2 cust e?pub policy. Accessed 6/5/01. Siegelman SS. Assassins and zealots: variations in peer review. Special report. Radiology. 1991;178:63742. ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001 919 Manuscript Revision and Final Editing Addeane S. Caelleigh and Judy A. Shea Manuscript revisions typically occur at two points: Some journals ask authors to make revisions as a condition of having the manuscript reconsidered or accepted, and some journals require changes as part of final editing before the manuscript is published (and some require both). Some journals have policies--or the resources--to help authors make the revisions, but many do not. And a few do substantive editing, while others do only the most basic copyediting. These processes are, in part, the link between peer reviewers' comments and editors' decisions and also, in part, the journal's contribution to improving good manuscripts. In deciding whether to publish a manuscript, editors may accept manuscripts as submitted, reject them, or, most often, offer authors the option of revising their manuscripts before a decision is made. Editors have a range of reasons for offering that last option and for choosing which revisions to request or require. In making these decisions, editors must maintain a balance between the appropriate use of the reviewers as advisers, the rights and responsibilities of authors, and the rights and responsibilities of editors. Editors' concerns include factors intrinsic to the study described in the manuscript (methodologic standards, for example) and others that are outside the authors' sphere, such as the focus of the journal, the nature of its readership, articles recently published in the editor's and other journals, and the special interests that the editor or journal is emphasizing at a particular time. Reviewers play a role in manuscript revision because they have often provided comments about revisions that are necessary before a manuscript can be accepted for publication. Because it is in everyone's best interest to publish the best possible manuscripts, reviewers are expected to provide constructive comments to authors about improving the manuscript and clear guidelines to the editor about the general nature of revisions that would improve the manuscript.1 The editor uses part or all of these comments in giving instructions about revisions. Editors have the responsibility to be clear and forthright about the revision decision and their instructions. Once the decision is made to offer an author the option to revise a manuscript, the editor should send a first letter that spells out the editor's offer and expectations. The letter should lay out explicitly the nature of the offer being made. For example, is the manuscript being rejected in its present form but with the offer of reconsideration if certain revisions are made? Or, in contrast, is the manuscript accepted for publication contingent upon certain revisions' being made? Also in the letter, the editor should give unambiguous instructions about the revisions, including which are required, which are encouraged but not required, and which are optional. Further, the editor should resolve any conflict between different reviewers' comments, so that the author will know which comments are to be followed in making particular revisions. The letter also should outline clearly what will happen after the revised manuscript is received. For instance, will it go out for further review, or will the next decision be made by the editor alone? Finally, the letter should set out a time line or a deadline for the author's reply. After a manuscript is accepted, final editing is required before publication.1 The depth and extent of this editing vary widely from journal to journal. The editing may be simple copyediting (correcting punctuation and spelling, for example) or may be moderate editing for clarity, or for accuracy in the parts where inaccuracies tend to be found (such as abstracts,2 quotations and references,3,4 and the use of statistics5). Sometimes the editing is deep substantive editing (such as reorganizing structure for internal consistency, for instance). The difference is due mostly to the journal's financial resources but also to its traditions. In some cases, especially in small journals where the editor has no editorial staff, any editing is done by the editor, with the authors sometimes having much and sometimes little input, while the publishing house provides typesetting services only. At many journals, the only editing is done by the publishing house, which provides standardized technical editing to ensure that a manuscript is put into correct grammar, consistent spelling and punctuation, and the journal's format; in this case, the author usually has no input. At a few journals, an editorial staff provides substantive editing. The onus for the revising done during the final editing often falls on the author, because most journals do not have staff editors. Authors should be prepared to seek outside editing help from editorial staff at their universities or from private editors (``author's editors''). 920 A C A D E M I C M E D I C I N E , V O L . 7 6 , N O . 9 / S E P T E M B E R 2 0 0 1 REFERENCES 1. Purcell GP, Donovan SL, Davidoff F. Changes to manuscripts during the editing process: characterizing the evolution of a clinical paper. JAMA. 1998;280:2278. 2. Pitkin RM, Branagan MA, Burnmeister LF. Accuracy of data in abstracts of published research articles. JAMA. 1999;281:112930. 3. Eichorn P, Yankauer A. Do authors check their references? A survey of accuracy of references in three public health journals. Am J Public Health. 1987;77:10112. 4. Evans JT, Nadjari HI, Burchell SA. Quotational and reference accuracy in surgery journals. JAMA. 1990;263:13534. 5. Gardner MJ. An exploratory study of statistical assessment of papers published in the British Medical Journal. JAMA. 1990;263:13557. RESOURCES Browner WS. Choosing a journal and responding to reviewers' comments. In: Publishing and Presenting Clinical Research. Baltimore, MD: Williams & Wilkins, 1999:15363 [esp. 15863]. Day RA. The review process (how to deal with editors). In: How to Write and Publish a Scientific Paper. 5th ed. Phoenix, AZ: Oryx Press, 1998: 98110 [esp. 10510]. Morgan P. How the editor works! In: An Insider's Guide for Medical Authors and Editors. Philadelphia, PA: ISI Press, 1986:6783 [esp. 6773]. Morgan P. Dealing with the editor. In: An Insider's Guide for Medical Authors and Editors. Philadelphia, PA: ISI Press, 1986:8492. Tornquist EM. From Proposal to Publication: An Informal Guide to Writing about Nursing Research. Menlo Park, CA: AddisonWesley, 1986. ACADEMIC MEDICINE, VOL. 76, NO. 9 / SEPTEMBER 2001 921