This post is about fixing peer review. dmfant, who some of you might know from synthetic_zero, commented on a recent post on the state of STS scholarship, asking:

have you been following: http://www.cplong.org/2013/11/the-peer-review-coordinator-and-the-collegiality-index/ ?

Which is a discussion about making reviewers and reviews public (rather than granting reviewers anonymity). The idea is that we could improve reviewing scholarly work through a process of making it public and creating, in the process, a “reviewer score” for reviewers that would be known, esteemed, and productive. My response was so long, I will post it here for further consideration.

My response was (also, I write as a faculty member and about faculty issues, so it is a bit myopic in that way):

I’ve been critical of peer review for a long time, probably ever since I was “subjected” to it early on in my career. This new approach is not nearly as much of a solution as I think it is being presented as:

First, one of the few things that I like about PR is exactly that it is NOT a game of hierarchy. You might get a reputation with an editor, but that editor will be replaced eventually, so the influence of “being a good researcher” over “being a good reviewer” was primary.

Second, my guess is that this is just something that will further hurt in-coming scholars, meaning, it is precisely newer scholars that will need to be deemed “good reviewers” and it is senior scholars with tenure who are the most free to not care about being deemed a “good reviewer.”

Third, another problem will be: is this yet another thing I have to do in order to get tenure? Again, the young scholar is held to a higher standard than the senior scholars that will oversee their tenure file, and, again, the senior scholar hold the younger scholar to a higher standard for tenure than they themselves were held to. That, however, depends a lot on how university administrators see such activities: will young scholars be expected to be “good reviewers” as a part of the tenure decision OR will young scholars be rewarded directly for being deemed “good reviewers” in the form of larger raises? I would imagine that rather than being directly awarded for this hard work, it will be just another expectation for young scholars; it will be, in effect, just something that can held against a scholar (i.e., being deemed “bad reviewer”) than something rewarded (i.e., we expect our faculty to be “good reviewers”).

Fourth, the predictable counter-argument would look like this: “oh, silly Nicholas, don’t be so cynical, if we can “fix” the peer review process, then you’ll be able to publish even more research and publish it through a process that is more fair.” We all know that the publication process is unfair. It is a reality. We all come to grips with it one way or another. When I consider the work of my peers, I am able to keep this in mind. However, if we “fix” it, it will still be unfair (i.e., instead of a tyranny of advanced/high-prestige researchers having a strong voice in the review process, we will trade one tyrant for another, the good reviewer, and soon we will be forced to publish what they will accept) and there is no reason to think that “good reviewers” are going to be more fair, promote innovation, or let the author’s voice come through. We will just trade one set of intrenched values for a new set of values that will be intrenched later on.

Fifth, all of this would not work even if, by some magical coincidence it did align university officials, faculty members, and folks that populate promotion and tenure committees … its called “the internet.” Of course, whenever I get a paper, I could determine the author with stunning frequency by just looking-up the title of the paper, which was likely presented in some professional society’s annual meeting that I probably attended, moreover, most of the papers I get to review these days, I know them as soon as I see them because, if you’re an expert in a small area, you know all the main players already. If you want to promote their work, then you can. If you want to thwart their work, then you can. There is no reason to think that a new system should change this. We already know whose papers they are; anonymity for the author seems long dead among specialists. Then comes reviewers: “There seems to be widespread skepticism that peer review without anonymity can be both rigorous and fair” to quote the blog. The idea that anonymity allows one to be “fair” (i.e., fair being defined primarily in terms of being able to say difficult comments plainly) is just BS. There is no reason to believe that in the professional game of science that I (or any other reviewer) cannot be fully honest in a review. Some folks use it to be a jerk. I understand that. I am fine with that. The reason is that some people are jerks. They are jerks in the classroom. They are jerks in their correspondence. I guess I am open to the range of personalities and am not entirely sure that erasing this variation or marginalizing folks that don’t quite have strongly developed social skills or promoting reviewers with glycerine tongues (by rewarding them with higher education’s favorite currency, status) makes much sense. People fail. Editors fail scholars by not taking this into account. This is a human enterprise and because it is I think we need to embrace a level of uncertainty, improve the processes that we have instead of upending a tradition.

In the end, if you want to fix one thing, I would say that we need NOT better reviewers (after all, reviews are just what they are, reviews), but better EDITORS. Show me an editor with a real vision, one capable of making tough decisions, one that is not a total tool of the reviews, an editor that is not constantly looking for ways to reduce their workload and simply embraces the huge responsibility of selecting one paper over another or encouraging the work of a young scholar to develop or reject the sloppy work of a senior scholar whose work is popular among reviews.

NOTE: the picture is from http://kennyjonesradio.com/wp-content/uploads/2009/08/broken_window.jpg