Peer-reviews: soundness vs interest

I am very intrigued by the following idea, which is not new (see further), but I have not seen it discussed in the small world of mathematics.  Peer-reviews have two goals which could be separated, for the benefit of a better communication of research among scholars:

  • to filter submitted articles as a function of the soundness of the research work,
  • to assess  the level of interest of the research work.

The first goal is a must, the second one opens the gate to abuse and subjective bias.

I learned about the idea of separating these two goals from the presentation by Maria Kowalczuk which I shared in this post. Afterwards, I looked on the net to find more. It seems this idea was pioneered by PLoS ONE, with PeerJ being the latest adopter.  The following citation is taken from  “Open and Shut?:UK politicians puzzle over peer review in an open access environment” (2011):

* On splitting traditional peer review into two separate processes: a) assessing a paper’s technical soundness and b) assessing its significance — a model pioneered by open-access publisher PLoS ONE, and now increasingly being adopted by traditional publishers …

Q162 Chair: We have heard that pre-publication peer review in most journals can be split, broadly, into a technical assessment and an impact assessment. Is it important to have both? 

Dr Torkar: … It is fairly straightforward to think about scientific soundness because it should be the fundamental goal of the peer review process that we ensure all the publications are well controlled, that the conclusions are supported and that the study design is appropriate. That is fairly straightforward as a very important aspect which should be addressed as part of the peer review process.

The question of the importance of impact is more difficult. When we think about high impact papers we think about those studies which describe findings that are far reaching and could influence a wide range of scientific communities and inform their next-stage experiments. Therefore, it is quite important to have journals that are selective and reach out to a broad readership, but the assessment of what is important can be quite subjective. That is why it is important, also, to give space to smaller studies that present incremental advances. Collectively, they can actually move fields forward in the long term.

Dr Patterson: … [B]oth these tasks add something to the research communication process. Traditionally, technical assessment and impact assessment are wrapped up in a single process that happens before publication. We think there is an opportunity and, potentially, a lot to be gained from decoupling these two processes into processes best carried out before publication and those better left until after publication.

One way to look at this is as follows. About 1.5 million articles are published every year. Before any of them are published, they are sorted into 25,000 different journals. So the journals are like a massive filtering and sorting process that goes on before publication. The question we have been thinking about is whether that is the right way to organise research. There are benefits to focusing on just the technical assessment before publication and the impact assessment after publication … Online we have the opportunity to rethink, completely, how that works. Both are important, but we think that, potentially, they can be decoupled …

Dr Lawrence: … [I]t is not known immediately how important something is. In fact, it takes quite a while to understand its impact. Also, what is important to some people may not be to others. A small piece of research may be very important if you are working in that key area. Therefore, the impact side of it is very subjective.

Dr Read: … Separating the two is important because of the time scale over which you get your answer. The impact is much longer. I guess the technical peer review is a shorter-term issue.

What intrigues me the most is that, even if the idea comes from the publication of medical research, it rather looks easy to implement in a model of math publication. Indeed, is it not the first purpose of peer-review of a math article  to decide the soundness of mathematical results from within?

In mathematics we have the proof,  which is highly optimized for independent check. True or false, sound or flawed, right? In principle at least, the peer-review in mathematics should serve mainly as a filter for sound results. The reality is different, I think experiences like the ones described in this post (browse through the provided links too,  maybe), are by no means exceptional, but rather common.

As for the interest level, it is well known that in mathematics one never ever knows the long term effect of a mathematical result. It is common in mathematics that results have a big latency, that articles may become suddenly relevant  decades and even centuries after their publication. It should be common-sense in mathematics that one cannot rely on the peer-review assessment of level of interest. But is not and, more often than not, under the umbrella of the relevance for the journal publication lurk darker things, like conflict of interests, over-protection of one field of work against stranger researchers intrusion, or exclusion based on club membership. In few words: the second role of peer-review is used for masking power games.

All that being said, let’s contemplate if the idea of keeping only the first role of peer-reviews is feasible. It certainly is, if it works already for PeerJ and PLoS ONE, why would not work for new models of mathematical publication, where it would be easier to apply? As usual, the most difficult part is to start using it.

Questions: how can be implemented an open, perpetual peer-review  with the main goal of assessment of soundness of mathematical research? Would a system based on comments and manifold contributions from blogs, as the Retraction Watch for example, be part of the the soundness decision, or a part of the level of interest decision? Would, for example, the wikipedia model be better for  the soundness part of peer-reviews and “comments in blogs” dreaded by some mathematicians would be better for assessing the local (in time and space) level of interest?

Your informative or critical comments would be great!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s