Beware the Jury of Peers

 By Traci Watson

taken from a 1997 issue of US NEWS AND WORLD REPORT

"Science is a noble, pure enterprise. At least that's what scientists would have you believe. To support their claim, researchers point to the process called peer review, the scientific version of the impartial jury of peers. The process of peer review is used to decide whether scientists get government grants and to weigh which articles should run in scientific journals.

But last week, a paper published in Nature cast doubt on whether peer review is really the guardian of scientific integrity that it's reputed to be. Two female scientists in Sweden wondered why women applying for fellowships to Sweden's Medical Research Council were accepted less than half as often as men. After studying records of the MRC's peer review committee, the scientists found female applicants had, on average, much lower scores than males--even when the women had published just as many papers in highly respected journals as the men had. Applicants who didn't know anyone on the review panel also got suspiciously low scores. To ward off objections that sexist peer review may be unique to Sweden, the authors note that the United Nations recently declared Sweden the nation with the world's best opportunities for women.

In public, scientists might proclaim shock at these results. In private, many have long grumbled to each other that peer review is rife with cronyism. The only problem is that there doesn't seem to be a more equitable way of making decisions. Harold Varmus, director of the National Institutes of Health, once noted that Churchill's remark about democracy also applies to peer review: "the worst form of government except all the others that have been tried."

For a full account of peer reviewing etc, see http://en.wikipedia.org/wiki/Peer_review

A bit from that:

Criticisms of peer review
One of the most common complaints about the peer review process is that it is slow, and that it typically takes several months or even several years in some fields for a submitted paper to appear in print. In practice, much of the communication about new results in some fields such as astronomy no longer takes place through peer reviewed papers, but rather through preprints submitted onto electronic servers such as arXiv.org.
In addition, some sociologists of science argue that peer review makes the ability to publish susceptible to control by elites and to personal jealousy. The peer review process may suppress dissent against "mainstream'" theories. Reviewers tend to be especially critical of conclusions that contradict their own views, and lenient towards those that accord with them. At the same time, elite scientists are more likely than less established ones to be sought out as referees, particularly by high-prestige journals or publishers. As a result, it has been argued, ideas that harmonize with the elite's are more likely to see print and to appear in premier journals than are iconoclastic or revolutionary ones, which accords with Thomas Kuhn's well-known observations regarding scientific revolutions.
However, others have pointed out that there is a very large number of scientific journals in which one can publish, making control of information difficult. In addition, the decision-making process of peer review, in which each referee gives his opinions separately and without consultation with the other members, is intended to mitigate some of these problems.
While some believe passing the peer-review process is a certification of validity, those who study that process often hold a far more skeptical view. Drummond Rennie, deputy editor of Journal of the American Medical Association is an organizer of the International Congress on Peer Review and Biomedical Publication, which has been held every four years since 1986. [1]. We still don't know how well the peer-review process works, he says, although one thing is clear: "There seems to be no study too fragmented, no hypothesis too trivial, no literature too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print." [2]
In 2006, a group of UK academics launched the online journal Philica, which tries to redress many of the problems of traditional peer review. Unlike in a normal journal, all articles submitted to Philica are published immediately and the review process takes place afterwards. Reviews are still anonymous, but instead of reviewers being chosen by an editor, any researcher who wishes to review an article can do so. Reviews are displayed at the end of each paper, and so are used to give the reader criticism or guidance about the work, rather than to decide whether it is published or not. This means that reviewers cannot suppress ideas if they disagree with them.
Another approach that is similar in spirit to Philica is that of a dynamical peer review site Naboj. Unlike Philica, Naboj is not a full-fledged online journal, but rather it provides an opportunity for users to write peer reviews of preprints at arXiv.org. The review system is modeled on Amazon and users have an opportunity to evaluate the reviews as well as the articles. That way, with a sufficient number of users and reviewers, there will be a convergence towards better quality of the review process.

go to page 1

modern peer reviewer, at NIH