I recently peeped in a post titled
Three myths about scientific peer review on Michael Nielsen's blog (by way of a
post on Kjerstin's blog). Now; Michael Nielsen is undoubtedly a very smart and knowledgeable guy, and I'm taking nothing away from his professional accomplishments - within his actual profession of theoretical physics and quantum computing (a self-professed "pioneer of quantum computing") - when I say that this particular post failed to impress me in a most spectacular manner. With his post, Nielsen aims to "debunk three widely-believed myths about peer review, myths which can derail sensible discussion of the future of peer review." The three myths are as follows:
Myth number 1: Scientists have always used peer review.While many scientists believe that peer review has been widely employed since early in the history of science, this is false, and worse still, most scientific journals didn't routinely use peer review until the mid 20th century, Nielsen states. He then goes on to mention examples to "illustrate the point" (which point is meant to be illustrated remains somewhat unclear), including the fact that most of the Great Albert Einstein's papers did not pass through peer review. As a matter of fact, it might be that only one of his papers went through a peer -review process, and this review came back negative. Outraged, Einstein wrote a scathing letter to the editor of the journal, and submitted his work elsewhere.
A couple of things with this example. First of all; I don't recall having ever heard anyone use "Oh yeah? Well..scientists have ALWAYS used peer review" as an argument neither for not against peer review. As a matter of fact, I fail to see the merit of using such an argument. The fact that something has been around forever (even though it really hasn't in this case) hardly qualifies as the sole reason for anything.
Second; Albert Einstein was a rare scientific mind, in all likelihood ranking among the most influential scientists in our history. While I'm sure that many scientists would like to think that they are close to his league, the odds are overwhelmingly against them being anywhere near his level. "Peer" means "one that is of equal standing with another". Let me ask you a rhetorical question: Do you expect a peer review system to work particularly well for a singular mind and statistical outlier like Albert Einstein? Personally, I would prefer a system which works for the vast majority of submitted work and has a somewhat lower reliability for values with large deviations from the mean. Pretty much anyone working with "real" data would know that an algorithm which is meant to include extremities and outliers is little more than a model of noise.
Third; the reviewer was correct and Einstein was wrong in this (rare) instance. So; this is actually an example of the peer review process working like a charm - even the work of Albert Einstein could benefit from peer review (not that I'm expecting this to be a robust trend). Using Albert Einstein in an example about peer review....unimpressive.
Myth number 2: Peer review is reliable.Here, Nielsen starts off with the time-tested "every scientist has gotten bad reviews" in a slightly new wrapping:
"Every scientist has a story (or ten) about how they were poorly treated by peer review - the important paper that was unfairly rejected, or the silly editor who ignored their sage advice as a referee. Despite this, many strongly presume that the system works “pretty well”, overall."Ya think? Now; I have personally complained about - what I perceive to be - poor reviews on this blog. Funny thing is; I typically don't make a post every time I get solid, constructive feedback, which actually happens most of the time. I also tend to make blog posts about things which are at least slightly outside of the norm. Moreover, if every scientist has a story about poor treatment, then that represents a very small fraction of the number of publications said scientist ought to have if he or she is anywhere close to productive. And you know what; every student has a tear-dripping story (or ten) about getting an undeservedly poor grade. Every actor or musician can give you a tale involving unfair reviews. Most people feel they pay too much tax, etc.
You can't please everyone, and every system is flawed.
Nielsen goes on to say that there isn't much evidence to suggest that the peer review system "works pretty well", and describes a rather famous example of papers about organic superconductors by German physicist Jan Hendrik Schoen in 2000 and 2001 which were published (after passing peer review) in very prestigious journals. Eventually, the findings appeared too good to be true, and it turned out that much of Schoen's work was fraudulent. Natch, a bunch of his papers was retracted, and there was a big hubbub. I remember this very well; early in 2001 my Ph.D. advisor put me - the newest addition to his research group - on a project aiming to investigate surface-immobilized DNA structures for their possible use as superconductors, based on these papers. For me, that was a waste of time to the tune of two months. To the grad student initially assigned to the project, it was the final straw and exit grad school by way of M.Sc.
Sure; it was really bad that so many manuscripts slipped past the reviewers. Good job it was caught. You know what, though? It wouldn't have caused a big stir unless such a slip-up happened very rarely. Every time some douchebags like Sudbø, the cold fusion people*, or Schoen get caught, there is a big media production, and an outrage in the scientific community. This - to me - is comforting. Does it mean that the peer review system has flaws? Sure - plenty of them. However, this does not amount to much unless you've got an alternative which has been
reliably documented to outperform the existing system. When that time comes, I'm all about the new system. I'm highly susceptible to facts and logic. Bandwagon-jumping - not so much.
Myth number 3: Peer review is the way we determine what's right and wrong in science.I don't really know what to say about this alleged myth, as I've never heard or read about anyone using this one as an argument pro or con peer review either.
There's plenty of room for improvement when it comes to the peer review process. For example, it's pretty obvious trend that the time between submission and acceptance of a manuscript is significantly shorter when big-shot scientists are present on the author list. Any ol' WOS search can confirm this. To a certain extent, this is due to the fact that big-shot scientists are very good at what they do. However, the productivity of some of these prominent scientists, and the rather large number of members in their research groups, highly suggest that their direct involvement with at least some of the papers emanating from their groups is minimal. Still, reviewers might be more likely to give the benefit of the doubt to a paper from a demonstrably excellent scientist from an Ivy-League university than to a work by Dr. Carlos Bandidos from the Dept. of mexican arctic and alien studies, Universidad de mexico. Is it fair? Arguably not. Is this particular bug fixable? You betcha'. A double blind system where the authors don't know the identities of the reviewers and vice versa is used by some journals, and unless it's a very narrow field with few players, I believe this evens the playing ground. When I act as a reviewer, I don't look at the author list until after I've read the manuscript and made up my mind regarding the quality of the work. I must admit that I've been surprised on some occasions by the abysmal quality of manuscripts coming out of some pretty famous labs.
There are plenty of unsatisfactory aspects in the peer review system. However, I have yet to see a system which outperforms it.
*Edit: By "the cold fusion people" I am referring to the famous "work" of Fleischmann and Pons in the late 80's which didn't turn out to be reproducible in any way, shape or form.