Reviewing Journal Papers 101

As a graduate student you might not have been invited to review any papers and so chances are that, as a new professor or post-doc, your only experience of this process is what you’ve seen from the receiving end.   Recently a couple of new colleagues asked me to give them some advice on reviewing papers and so I’d thought I’d share with you a bit of our discussion.  

First off – they asked me about the typical format of a review. Having worked as an associate editor for a couple of different journals, I can tell you that there is a wide variety in the scope and depth of the typical review.  The vast majority of reviewers provide only a paragraph or two of vague comments that make you wonder if they have even read the paper.  On the other extreme are those who write a list of comments longer that the manuscript they are reviewing, which makes you wonder if they have anything else to do.  Naturally the appropriate review is somewhere in between; you should plan to spend 1/2 to 1 full day to do a proper review.  Personally, I emulate a formula I learned from my thesis supervisor and which I have also seen in the best reviews I have encountered.  It has three components, which I write in the order  presented below – but that I present to the editor in the opposite order.

Specific Comments

As I read through the paper the first time, I note down specific comments as I go.  These might be as simple as identifying typos or grammar errors if there are not too many; if there are a lot, I will only note the first half dozen or so.  I also note down anything that I think may be incorrect, incomplete or confusing.  In addition, I provided feedback as necessary on the tables and figures.  Here I am looking at whether the figure or table is needed, whether it’s clear and interpretable, and finally whether it’s adequately explained in the text.   Each of these comments is typically no more than one or two sentences long.  I list them in order providing the page and/or line numbers for each one.

General Comments

Once I have completed my first read through the paper, and have a complete list of specific comments, I am ready to write my general comments.  This section can be anywhere from one to a half dozen paragraphs long, depending upon how good or bad the paper is (i.e. it will be shorter for excellent papers). In the first paragraph, I discuss the reasoning behind my opinions on the originality of the paper, and I also comment on the overall quality of the writing (typos, grammar, organization, consistency, etc.).  If I have encountered an excessive number of typing and/or grammar errors, I point out that I have documented only a few of them as examples.  (After all – it’s not the reviewer’s job to provide editorial services to weak authors.)

In each of the subsequent paragraphs I focus on ‘big picture’ issues, things that generally evolve from groupings of specific comments.  I make the case for why they need to be addressed and support this with cited literature if needed.  If there are one or two papers that the authors have missed, then I give them the citations so that they can look them up. If they’ve missed many, I just tell them that.

Overview and Decision

As with most summaries – I write this last, after I have completed the other two components; it’s generally one paragraph long.   I recap in a few sentences what the major thrust of the paper is and I comment on the originality (or lack thereof) of the content. No matter how bad the paper is, I usually try to say something positive about it here. I end this paragraph with my specific recommendations to accept, reject, etc. (using the actual categories employed by that journal) and I mention that general and specific comments follow.

In consideration for the needs (and interests) of the authors and the journal editors, I present these three components in the opposite order.  Specifically, I start with the overview and decision, follow that with the general comments and I put the list of specific comments at the end.

Some other things to keep in mind when doing your review:

  • You are actually writing to the editor not to the authors. Phrase all of your comments with this in mind.  Also, no matter how poor the paper is, keep the discussion polite, respectful, and factual at all times.
  • It is never acceptable to get your grad students to do your reviews for you, nor is it acceptable to give the paper to a colleague to read. If you want to consult with a grad student or colleague about some particular aspect of a paper – write and get permission from the editor first.  If you like the paper enough to want to show it to your own students or colleagues right away, you should relay that request to the authors through the editor. Alternatively, if you do not mind them knowing you were the reviewer, then you could write to the authors and ask them directly once the paper has been accepted.
  • The authors do not need to agree with you for their work to be publishable. If they have presented the facts correctly and have interpreted those facts in a defensible manner, then their opinion is as valid as yours.
  • Our natural tendency is to want to accept every paper – at least I hope that is everyone’s initial feeling about any paper – however, the truth is, many of the papers you review will be poor (needing major revisions) and some will be downright terrible (needing rejection). You are not doing anyone any favours by pussy-footing around these situations.  If the paper needs to be set into one of these two categories, then you need to give the editor the ammunition to back it up.  That doesn’t mean that you can be mean, snide, or sarcastic.  It means that you need to be very specific and thorough in providing the supporting facts.  Here are some of the reasons you might recommend rejection:
    • The work has been done before and absolutely nothing new is contributed by this paper.
    • The content is trivial.  Generally, if the content is equivalent to what I am teaching in my undergraduate courses, I would not consider it publishable.
    • The work is plagiarized.  Either the authors are double publishing their own material or copying others’ material.  Rare – but I have encountered both situations.
    • There are major and/or fundamental technical errors which completely negate the findings.

Finally – a question you might be shy to ask.  Do you have to agree to review every time you are asked?  The answer is ‘no’; however, keep in mind that each of your own papers requires 2 to 3 reviewers.  Therefore, you should be reviewing 2 to 3 papers for every paper you submit (especially if none of your co-authors are reviewing, which will typically be the case if they are all students.) Some editors keep track of these things; one journal I worked for even had a policy of refusing to accept submissions from people who repeatedly refused to review.  Therefore, it’s always good to give a reason if you cannot do a review.  Here are the only reasons I would ever turn down a request for review.

  • It is not in my area of specialization and, therefore, I am not up to date on the relevant literature. It’s just not fair to the authors.
  • It is a journal that I will never, ever publish in and I have already done a few for them in recent months.  (Alternatively, I might just consider sending them a paper if it’s a good journal and other people in my field are starting to publish there.)
  • I am not going to able to meet their deadline and we cannot come to a mutually feasible later deadline.
  • I am doing a couple of reviews for that journal already.

That’s it then – ‘Paper Reviewing 101’. I hope you found it useful.  Please use the comment feature to let me know what you think or to offer your own tips.

If you’re interested in the other side of the equation – how to respond to reviews – keep an eye on my technical writing blog.  I’ll be talking about that in an upcoming post.


4 thoughts on “Reviewing Journal Papers 101

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s