She feared that abandoning peer review would make scientific literature no more reliable than the blogosphere, consisting of an unnavigable mass of articles, most of which were “wrong or misleading”.This is a quote from one of the most interesting articles I read these days: “Slay peer review ‘sacred cow’, says former BMJ chief” by Paul Jump.
I commented previously about replacing peer-review with validation by reproducibility
but now I want to concentrate on this quote, which, according to the author of the article, has been made by “Georgina Mace, professor of biodiversity and ecosystems at University College London”.This is the pro argument in favor of the actual peer review system. Opposed to it, and main subject of the article, is
“Richard Smith, who edited the BMJ between 1991 and 2004, told the Royal Society’s Future of Scholarly Scientific Communication conference on 20 April that there was no evidence that pre-publication peer review improved papers or detected errors or fraud.”
I am very much convinced by this, but let’s think coldly.
Pro peer review is that a majority of peer reviewed articles is formed by correct articles, while a majority of “the blogosphere [is] consisting of an unnavigable mass of articles, most of which were “wrong or misleading””.
Contrary to peer review is that
Referring to John Ioannidis’ famous 2005 paper “Why most published research findings are false”, Dr Smith said “most of what is published in journals is just plain wrong or nonsense”. He added that an experiment carried out during his time at the BMJ had seen eight errors introduced into a 600-word paper that was sent out to 300 reviewers.
“No one found more than five [errors]; the median was two and 20 per cent didn’t spot any,” he said. “If peer review was a drug it would never get on the market because we have lots of evidence of its adverse effects and don’t have evidence of its benefit.””and moreover:
“peer review was too slow, expensive and burdensome on reviewers’ time. It was also biased against innovative papers and was open to abuse by the unscrupulous. He said science would be better off if it abandoned pre-publication peer review entirely and left it to online readers to determine “what matters and what doesn’t”.”
Which I interpret as confidence in the blogosphere-like medium.
Where is the truth? In the middle, as usual.
Here is my opinion, please form yours.
Potentially the new medium comes with new, relatively better means to do research. An important part of the research is communication, and it is clear that the old system is already obsolete. It is kept artificially alive by authority and business interests.
However, it is also true that a majority of productions which are accessible via the new medium are of a very bad quality and unreliable.
a majority of the work of academic painters was good but not brilliant (reliable but not innovative enough), a majority of non academic painters produce crappy cute paintings which average people LOVE. Even now the average people opinion is that anybody can doodle and use random colours to make these new paintings (which are 100 years old btw).That is life.
You can’t accuse a non affiliated painter that he shows his work in the same venue where you find all the cats, kids, wrinkled old people and cute places.
Science side, we live in a sea of crap which is loved by the average people, which the blood of the so called attention economy.
It is difficult for a researcher, who is a particular species of a creator, to find other people willing to spend time not only to share original ideas (which are not liked because strange, by default), but also to invest a work into understanding, into creating.
That is why I believe that:
– there have to be social incentive for these people (and that attention economy thinking is not helping this, being instead a vector of propagation for big budget PR and lolcats and life wisdom quotes)
– and that the creators of new scientific content have to provide as much as possible means for self-validation of their work.