Fixing Peer Review

broken_window

This post is about fixing peer review. dmfant, who some of you might know from synthetic_zero, commented on a recent post on the state of STS scholarship, asking:

have you been following: http://www.cplong.org/2013/11/the-peer-review-coordinator-and-the-collegiality-index/ ?

Which is a discussion about making reviewers and reviews public (rather than granting reviewers anonymity). The idea is that we could improve reviewing scholarly work through a process of making it public and creating, in the process, a “reviewer score” for reviewers that would be known, esteemed, and productive. My response was so long, I will post it here for further consideration.

My response was (also, I write as a faculty member and about faculty issues, so it is a bit myopic in that way):

I’ve been critical of peer review for a long time, probably ever since I was “subjected” to it early on in my career. This new approach is not nearly as much of a solution as I think it is being presented as:

First, one of the few things that I like about PR is exactly that it is NOT a game of hierarchy. You might get a reputation with an editor, but that editor will be replaced eventually, so the influence of “being a good researcher” over “being a good reviewer” was primary.

Second, my guess is that this is just something that will further hurt in-coming scholars, meaning, it is precisely newer scholars that will need to be deemed “good reviewers” and it is senior scholars with tenure who are the most free to not care about being deemed a “good reviewer.”

Third, another problem will be: is this yet another thing I have to do in order to get tenure? Again, the young scholar is held to a higher standard than the senior scholars that will oversee their tenure file, and, again, the senior scholar hold the younger scholar to a higher standard for tenure than they themselves were held to. That, however, depends a lot on how university administrators see such activities: will young scholars be expected to be “good reviewers” as a part of the tenure decision OR will young scholars be rewarded directly for being deemed “good reviewers” in the form of larger raises? I would imagine that rather than being directly awarded for this hard work, it will be just another expectation for young scholars; it will be, in effect, just something that can held against a scholar (i.e., being deemed “bad reviewer”) than something rewarded (i.e., we expect our faculty to be “good reviewers”).

Fourth, the predictable counter-argument would look like this: “oh, silly Nicholas, don’t be so cynical, if we can “fix” the peer review process, then you’ll be able to publish even more research and publish it through a process that is more fair.” We all know that the publication process is unfair. It is a reality. We all come to grips with it one way or another. When I consider the work of my peers, I am able to keep this in mind. However, if we “fix” it, it will still be unfair (i.e., instead of a tyranny of advanced/high-prestige researchers having a strong voice in the review process, we will trade one tyrant for another, the good reviewer, and soon we will be forced to publish what they will accept) and there is no reason to think that “good reviewers” are going to be more fair, promote innovation, or let the author’s voice come through. We will just trade one set of intrenched values for a new set of values that will be intrenched later on.

Fifth, all of this would not work even if, by some magical coincidence it did align university officials, faculty members, and folks that populate promotion and tenure committees … its called “the internet.” Of course, whenever I get a paper, I could determine the author with stunning frequency by just looking-up the title of the paper, which was likely presented in some professional society’s annual meeting that I probably attended, moreover, most of the papers I get to review these days, I know them as soon as I see them because, if you’re an expert in a small area, you know all the main players already. If you want to promote their work, then you can. If you want to thwart their work, then you can. There is no reason to think that a new system should change this. We already know whose papers they are; anonymity for the author seems long dead among specialists. Then comes reviewers: “There seems to be widespread skepticism that peer review without anonymity can be both rigorous and fair” to quote the blog. The idea that anonymity allows one to be “fair” (i.e., fair being defined primarily in terms of being able to say difficult comments plainly) is just BS. There is no reason to believe that in the professional game of science that I (or any other reviewer) cannot be fully honest in a review. Some folks use it to be a jerk. I understand that. I am fine with that. The reason is that some people are jerks. They are jerks in the classroom. They are jerks in their correspondence. I guess I am open to the range of personalities and am not entirely sure that erasing this variation or marginalizing folks that don’t quite have strongly developed social skills or promoting reviewers with glycerine tongues (by rewarding them with higher education’s favorite currency, status) makes much sense. People fail. Editors fail scholars by not taking this into account. This is a human enterprise and because it is I think we need to embrace a level of uncertainty, improve the processes that we have instead of upending a tradition.

In the end, if you want to fix one thing, I would say that we need NOT better reviewers (after all, reviews are just what they are, reviews), but better EDITORS. Show me an editor with a real vision, one capable of making tough decisions, one that is not a total tool of the reviews, an editor that is not constantly looking for ways to reduce their workload and simply embraces the huge responsibility of selecting one paper over another or encouraging the work of a young scholar to develop or reject the sloppy work of a senior scholar whose work is popular among reviews.

NOTE: the picture is from http://kennyjonesradio.com/wp-content/uploads/2009/08/broken_window.jpg

Advertisements

10 thoughts on “Fixing Peer Review

  1. Also, of note: Fabio over at orgtheory.net has claimed that reviewers being so slow is the process: http://orgtheory.wordpress.com/2013/11/22/if-you-arent-part-of-the-journal-review-time-solution-youre-part-of-the-problem/

    And, interestingly, that reviewers that question review processes may be booted from journals!
    http://orgtheory.wordpress.com/2013/11/27/on-getting-booted-from-a-journal/

    Brayden also has a number of good posts on universal review standards: http://orgtheory.wordpress.com/2010/08/09/a-crazy-proposal-universal-peer-review/

    And, related to this discussion, open peer review: http://orgtheory.wordpress.com/2011/04/14/open-peer-review/

    Like

    • To pick up on our exchange below, seems to me that the question (aporia?) of “standards” is the po-mo bugbear in the humanities, do we have anything like ‘objective’ standards that might pull such efforts/judgements above matters of personal taste and other aspects of the gossip-world, I think not…

      Like

      • I am not so sure if standards help. We need a balance. Standards will short-circuit flexibility because they will be set and then forgotten about because we will all be too busy trying to publish rather than actively engage in and shape those standards slowly over time. The balance I mention is this: as scholarship changes and as universities transform their methods of promotion and tenure, our reviewing needs to have the flexibility to do so too. Again, as said above, editors have the power to reject reviews and reviewers. When an editor sees a bad review, they should act on it. For example, I once had a discussion with an editor about this: one of our reviewers contained fewer sentences than months we waited for it. I was not pleased. However, the editor just said “seems like it was not important; your paper is rejected.” This can be handled at that level too. I am not, let me be clear, saying that fixing the editor position is the only solution. I am reflexive enough to know that my personal experiences with poor editors shapes the way I think about these issues. I am, myself, a fast reviewer. Sometimes I will review a paper in the first 24 hours that I get it. I think one of the ways that editors fail their authors is this: reviewers don’t have to defend their reviews.

        Like

      • I think we are talking about the same thing if we are talking about how skilled an editor is (and reviewers are) and not about if they are merely willing to hold folks accountable (I know that academia is often rightly notorious for the failures to get things done in a timely/respectful manner).

        Like

  2. Frankly, I’d like more skilled reviewers and editors AND accountable reviewers and editors; still, I’d take the former over the latter, any day. And yet, managing (wrangling?) academics is a skill in and of itself, so perhaps another solution, instead of standards, is training.

    How do scholars learn to review? I once heard a story from a colleague that went like this: he learned how to review papers when his mentor, after leaving a review on his/her desk for nearly a year, handed it off to my colleague (at the time, a grad student) and told him “review this and get it back to me — that’s your RA assignment for this afternoon.” “When?” my friend asked. “I don’t care,” the faculty mentor responded. WHOA…

    In a similar vein, are there courses on peer review? If so, I’ve never seen one. The same thing is true of editors; managing a journal is a professional skill; however, editors are often selected for their high-rank in a specialty area or because, well, nobody else wants to do it (!) not because of their ability to edit/manage people.

    Like

    • ah I see by “standards” I don’t mean rules I mean know-how/skill/taste/judgement/etc., as rules need applying and around the Wittgensteinian loop we go.
      Indeed researchers are not experts in reviewing and such and I would go further and say not trained experts in teaching either!
      The line for promotion is not so clear on getting ahead because of what you know versus who you know, Rorty&Fish and others working after Kuhn were right that by and large changes in academic theoretical modes/allegiances come about because a new generation comes into power, cliques rule the roost.

      Like

  3. One approach as editor would be to take the “revise and resubmit” approach when dealing with younger scholars seriously enough to give them lengthy, detailed answers. Editors certainly need to serve as gatekeepers, but they can also help contributors know more clearly what they and the reviewers consider quality work. This requires aLOT of work from the editor. It is generally easier to provide such feedback in specialized journals where the community of scholars has more of an investment in bringing new work and young scholars into the the field.

    Like

    • Surely, you are right. The workload itself can be a deterrent, I imagine. It goes back, in my thinking if things to ‘vision’; what’s the editor’s vision for the journal? That could clear-up a lot of issues like where to send, who to review for, and what kind if work to publish. The beauty of saying ‘I want to publish top quality work’ is that you end up publishing whatever you want or whatever reviewers tell you…

      Like

  4. Pingback: What are you supposed do with unfair criticism? | Installing (Social) Order

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s