Tag Archives: open peer review

Open peer review is something others should do, Open science is something you could do

This post follows Peer review is not independent validation, where it is argued that independent validation is one of the pillars of the scientific method. Peer review is only a part of the editorial process. Of course that peer review is better than nothing, but it is only a social form of validation, much less rigorous than what the scientific method asks.

If the author follows the path of Open science, then the reader has the means to perform an independent validation. This is great news, here is why.

It is much easier to do Open science than to change the legacy publishing system.

Many interesting alternatives to the legacy publishing have been proposed already. There is green OA, there is gold OA (gold is for $), there is arXiv.org. There are many other versions, but the main problem is that research articles are not considered really serious unless they are peer reviewed. Legacy publishing provides this, it is actually the only service they provide. People are used to review for established journals and any alternative publishing system has to be able to compete with that.

So, if you want to make an OA platform, it’s not serious unless you find a way to make other people to peer review the articles. This is hard!

People are slowly understanding that peer review is not what we should aim for. We are so used with the idea that peer review is that great thing which is part of the scientific method. It is not! Independent validation is the thing, peer review is an old, unscientific way (very useful, but not useful enough to allow research finding to pass the validation filter).

The alternative, which is Open science, is that the authors of research findings make open all the data, procedures, programs, etc, everything they have. In this way, any other group of researchers, anybody else willing to try can validate those research findings.

The comparison is striking. The reviewers of the legacy publishing system don’t have magical powers, they just read the article, they browse the data provided by the very limited article format and they make an opinion about the credibility of the research findings. In the legacy system, the reviewer does not have the means to validate the article.

In conclusion, it is much simpler to do Open science than to invent a way to convince people to review your legacy articles. It is enough to make open your data, your programs, etc. It is something that you, the author can do.

You don’t have to wait for the others to do a review for you. Release your data, that’s all.

Advertisements

Peer review is not independent validation

People tend to associate peer review with science. As an example, even today there are still many scientists who believe that an arXiv.org article is not a true article, unless it has been peer reviewed. They can’t trust the article, without reading it first, unless it passed the peer review, as a part of the publishing process.

Just because a researcher puts a latex file in the arXiv.org (I continue with the example), it does not mean that the content of the file has been independently validated, as the scientific method demands.

The part which slips from the attention is that peer review is not independent validation.

Which means that a peer reviewed article is not necessarily one which passes the scientific method filter.

This simple observation is, to me, the key for understanding why so many research results communicated in peer reviewed articles can not be reproduced, or validated, independently. The scale of this peer reviewed article rot is amazing. And well known!

Peer review is a part of the publishing process. By itself, it is only a social validation. Here is why: the reviewers don’t try to validate the results from the article because they don’t have the means to do it in the first place. They do have access only to a story told by the authors. All the reviewers can do is to read the article and to express an opinion about it’s credibility, based on the reviewers experience, competence (and biases).

From the point of view of legacy publishers, peer review makes sense. It is the equivalent of the criteria used by a journalist in order to decide to publish something or not. Not more!

That is why it is very important for science to pass from peer review to validation. This is possible only in an Open Science frame. Once more (in this Open(x) fight) the medical science editors lead. From “Journal Editors To Researchers: Show Everyone Your Clinical Data” by Harlan Krumholz, a quote:

“[…] last Wednesday, the editors of the leading medical journals around the world made a proposal that could change medical science forever. They said that researchers would have to publicly share the data gathered in their clinical studies as a condition of publishing the results in the journals. This idea is now out for public comment.

As it stands now, medical scientists can publish their findings without ever making available the data upon which their conclusions were based.

Only some of the top journals, such as The BMJ, have tried to make data sharing a condition of publication. But authors who didn’t want to comply could just go elsewhere.”

This is much more than simply saying “peer review is bad” (because is not, only that it is not a part of the scientific method, it is a part of the habits of publishers). It is a right step towards Open Science. I repeat here my opinion about OS, in the shortest way I can:

There are 2 parts involved in a research communication:   A (author, creator, the one which has something to disseminate) and R (reader). The legacy publishing process introduces a   B (reviewer).  A puts something in a public place, B expresses a public opinion about this and R uses B’s opinion as a proxy for the value of A’s thing, in order to decide if A’s thing is worthy of R’s attention or not.  Open Access is about the direct interaction of A with R, Open Peer-Review is about transparent interaction of A with B, as seen by R and Validation (as I see it) is improving the format of A’s communication so that R could make a better decision than the social one of counting on B’s opinion.

That’s it! The reader is king and the author should provide everything to the reader, for the reader to be able to independently validate the work. This is the scientific method at work.

 

One of the first articles with means for complete validation by reproducibility

I have not stressed enough this aspect. The article

M. Buliga, Molecular computers

is one of the first articles which comes with complete means of validation by reproducibility.

This means that along with the content of the article, which contains animations and links to demonstrations, comes a github repository with the scripts which can be used to validate (or invalidate, of course) this work.

I can’t show you here how the article looks like, but I can show you a gif created from this  video of a demonstration which appears also in the article (however, with simpler settings, in order to not punish too much the browser).

KebzRb

This is a chemical like computation of the Ackermann(2,2) function.

In itself, is intended to show that if autonomous computing molecules can be created by the means proposed in the article, then impressive feats can be achieved.

This is part of the discussion about peer review and the need to pass to a more evolved way of communicating science.There are several efforts in this direction, like for example PeerJ’s paper-now commented in this post. See also the post Fascinating: micropublications, hypothes.is for more!

Presently one of the most important pieces of this is the peer review, which is the social practice consisting in declarations of one, two, four, etc anonymous professionals that they have checked the work and they consider it valid.

Instead, an ideal should be the article which runs in the browser, i.e. one which comes with means which would allow anybody to validate it up to external resources, like the works by other authors.

(For example, if I write in my article that “According to the work [1]   A is true. Here we prove that B follows from A.” then I should provide means to validate the proof that A implies B, but it would be unrealistical to be ask me to provide means to validate A.)

This is explained in more detail in Reproducibility vs peer review.

Therefore, if you care about evolving the form of the scientific article, then you have a concrete, short example of what can be done in this direction.

Mind that I am stubborn enough to cling to this form of publication, not because I am afraid to submit these beautiful ideas to legacy journals, but because I want to promote new ways of sharing research by using the best content I can make.

_________________________________________

Harsh assessment

I need a hard objective and harsh assessment of the demos, moves pages, all this effort I make. I am looking for funding, I don’t get one presently, so there might be something wrong I do.
Please be as harsh as possible. Thank you!

I am waiting for your comments. If you want to make a private comment then add in your message the following string

pe-240v

and the comment will go to the moderation queue.

If you have not made any comments here, until now, then by default the comment goes to moderation.

So, please mention in the comment if you want to keep it private.

Assessment for what?

or anything about chemlambda.

This is a big project, I see people are interested in more advanced stuff, like distributed computing, but they usually fail to understand the basics.

On the other side, I am a mathematician learning to program. So I’m lousy at that (for the moment), but I hope I make my point about the basics with these demos and help pages.

_________________________________________________________________________

Open notebook science for everyone, done by everyone

I am deeply impressed by the post:

Jean Claude Bradley Memorial Symposium; July 14th; let’s take Open Notebook Science to everyone

Here are some quotes:

Jean-Claude Bradley was one of the most influential open scientists of our time. He was an innovator in all that he did, from Open Education to bleeding edge Open Science; in 2006, he coined the phrase Open Notebook Science. His loss is felt deeply by friends and colleagues around the world.

“Science, and science communication is in crisis. We need bold, simple visions to take us out of this, and Open Notebook Science (ONS) does exactly that. It:

  • is inclusive. Anyone can be involved at any level. You don’t have to be an academic.
  • is honest. Everything that is done is Open, so there is no fraud, no misrepresentation.
  • is immediate. The science is available as it happens. Publication is not an operation, but an attitude of mind
  • is preserved. ONS ensures that the record, and the full record, persists.
  • is repeatable or falsifiable. The full details of what was done are there so the experiment can be challenged or repeated at any time
  • is inexpensive. We waste 100 Billion USD / year of science through bad practice so we save that immediately. But also we get rid of paywalls, lawyers, opportunity costs, nineteenth century publishing practices, etc.”

Every word is true!

This is the future of the research communication. Or at least the beginning of it. ONS has open, perpetual peer review as a subset.

Personal notes.  Look at the left upper corner of this page, it reads:

chorasimilarity | computing with space | open notebook.

Yay! the time  is coming!  the weirdos who write on arXiv, now figshare,  who use open notebooks, all  as a replacement for legacy publication,   will soon be mainstream 🙂

Now, seriously, let’s put some gamification into it, so those who ask “what IS a notebook?”  can play too. They ARE the future. Hope that soon the Game of Research and Review, aka playing  MMORPG  games at the knowledge frontier, will emerge.

There are obvious reasons for that:

  • the smartphone freeds us from staying in one physical place while we surf the virtual world
  • which has as an effect that we rediscover that physical space is important for our interactions, see  Ingress
  • gamification of human activities is replacing the industrial era habits, (pyramidal, static organizations, uniformization, identification of humans with their functions (worker, consumer, customer, student) and with their physical location (this or that country, city, students in the benchs, professors at the chair, payment for working hours ans for staying at the counter, legacy publishing).

See also Notes for “Internet of Things not Internet of Objects”.

 

_________________________________________________

 

 

The tone goes up on the OPEN front

This post has a collection of savory quotes and further comments about the psychological changes which are ongoing, around new ways of dissemination and communication of scientific research.

Aka OPEN …

  • access
  • peer review
  • data
  • notebooks

We are closing to a change, a psychological change, from indifference and disdain from the majority of (more or less established) researchers to a public acknowledgement of the stupidity and immorality of the procedure which is in force, still.

[Rant, jump over if not interested into personal stuff.

Please take into consideration that even if I embrace with full heart these changes, I don’t have any merit or real contribution to these, excepting modest posts here at chorasimilarity, under the tags cost of knowledge and open peer review. More than this, I suffered like probably some of my colleagues by choosing to publish through arXiv mostly and not playing the stupid game, which led to a very damaged career, but unfortunately I did not had the opportunity to create change through participation in teams which now are shaping the future of OPEN whatever. Bravo for them, my best wishes for them, why not sometimes a honest criticism from my small point of view, and thanks for the feeling of revenge which I have, the “I was right” feeling which I hope will grow and grow, because really the research world is damaged to the bones by this incredible stupidity, maybe cupidity and surely lack of competence and care for the future manifested by a majority of leaders.

The second thing I want to mention is that even if I refer to “them”, to a “majority”, all these generalizations have to be nuanced by saying that, as always, as everywhere, the special ones, the creative ones, the salt and pepper of the research world are either excused or completely innocent. They are also everywhere, maybe many of them not in any strong influence position (as in music, for example, the most well known musicians are always never the best, but surely they are among the most hard working), but creating their stuff and possibly not really caring about these social aspects, because they are too deep into the platonic realm. All of them are not the subject or part of any “majority”, they are not “them” in any way.

The third point is that there may be a sloppy use of “young” and “old”. This has nothing to do with physical age. It is true that every old moron was a young moron before. Every old opportunist was a young one some years earlier. Their numbers are continually replenished and we find them everywhere, albeit much more present than the salt and pepper of the research community, and more in the good hard worker, but not really, seriously creative part.  No, young or old refers to the brain quality, not to physical age.

End of rant]

Back to the subject. From timid or rather lonely comments, now we passed to more strong ones.

And the words are harder.

From Causes of the persistence of impact factor mania, by Arturo Casadevall and Ferric C. Fang,

“Science and scientists are currently afflicted by an epidemic of mania manifested by associating the value of research with the journal where the work is published rather than the content of the work itself. The mania is causing profound distortions in the way science is done that are deleterious to the overall scientific enterprise. In this essay, we consider the forces responsible for the persistence of the mania and conclude that it is maintained because it disproportionately benefits elements of the scientific enterprise, including certain well-established scientists, journals, and administrative interests.”

Fully agree with them, besides of this I consider very interesting their explanation that we face a manifestation of the tragedy of the commons.

From Academic self-publishing: a not-so-distant-future, here is a big quote, is too beautiful to crop:

A glimpse into the future
Erin is driving back home from the laboratory with a big smile on her face. After an exciting three-hour brainstorming session discussing the intracranial EEG data from her last experiment, she can’t wait to get her hands back on the manuscript. A new and unexpected interpretation of the findings seems to challenge a popular assumption about the role of sleep in declarative memory consolidation. She had been looking over the figures for more than a month without seeing a clear pattern. But now, thanks to a moment of insight by one of her colleagues, the pieces finally fit together and a new logic is emerging. She realizes it will be hard for the community to accept these new findings, but the methodology is solid and she is now convinced that this is the only reasonable explanation. She is so anxious to see what Axell’s group thinks about new evidence that refutes its theoretical model.

After a week’s hard work, the first draft is ready. All the figures and their long descriptive legends are in place, the literature review is exhaustive, the methodology is clear as a bell, and the conclusions situate the finding in the general context of the role of sleep in memory consolidation. Today, the group had a brief morning meeting to decide which colleagues they will ask to review their draft. Of course, they will ask Axell for his opinion and constructive criticism, but they also agree to invite Barber to confirm that the application of independent component analysis on the data was performed correctly, and Stogiannidis to comment on the modification of the memory consolidation scale. For a review of the general intracranial EEG methodology, the group decides to first approach Favril herself and, if she declines, they will ask Zhang, who recently reviewed the subject for Nature.

After the lunch break, Erin submits the manuscript to the university’s preprint repository that provides a DOI (digital object identifier) and an open attribution licence. When she hits the submit button, she feels a chill running down her spine. More than a year’s hard work is finally freely available to her peers and the public. The next important step is to invite the reviewers. She logs in to her LIBRE profile and inserts the metadata of the manuscript with a hyperlink to the repository version (see LIBRE, 2013). She then clicks the invite reviewer button and writes a quick personal message to Axell, briefly summarizing the main result of the study and why she thinks his opinion is vital for the debate this manuscript will spark. She then invites Stogiannidis to comment on the modification of the memory consolidation scale, and Barber, specifically asking him to check the application of independent component analysis, and also letting him know that all data are freely and openly available at Figshare. After finishing with the formal invitations, Erin tweets the LIBRE link to her followers and sends it as a personal message to specific colleagues from whom she would like to receive general comments. She can now relax. The word is out!

A couple of weeks later, Erin is back at work on the project. Both Favril and Zhang refused to review because of heavy work schedules, but Stogiannidis wrote an excellent report totally approving the modification of her scale. She even suggested a future collaboration to test the new version on a wider sample. Barber also submitted a brief review saying that he doesn’t find any caveats in the analysis and approves the methodology. As Erin expected, Axell didn’t take the new result lightly. He submitted a harsh critique, questioning both the methodology and the interpretation of the main findings. He even mentioned that there is a new paper by his group currently under journal review, reporting on a similar experiment with opposite results. Being pipped to the post and being second to report on this innovative experimental design, he must be really peeved, thinks Erin. She grins. Maybe he will learn the lesson and consider self-publishing next time. Anyway, Erin doesn’t worry too much as there are already two independent colleagues who have marked Axell’s review as biased on LIBRE. Last night, Xiu, Erin’s colleague, finished retouching one of the figures based on a very insightful comment by one of LIBRE’s readers, and today she will upload a new version of the manuscript, inviting some more reviewers.

Two months later, Erin’s paper is now in version number 4.0 and everyone in the group believes it is ready for submission to a journal and further dissemination. The issues raised by seven reviewers have now been adequately addressed, and Axell’s review has received six biased marks and two negative comments. In addition, the paper has attracted a lot of attention in the social media and has been downloaded dozens of times from the institutional repository and has been viewed just over 300 times in LIBRE. The International Journal for the Study of the Role of Sleep in Memory Consolidation has already been in touch with Erin and invited her to submit the paper to them, but everybody in the group thinks the work is of interest to an even wider audience and that it should be submitted to the International Journal for the Study of Memory Consolidation. It charges a little more – 200 euros – but it is slightly more esteemed in the field and well worth the extra outlay. The group is even considering sending the manuscript in parallel to other journals that embrace a broader neuroscience community, now that the group’s copyright and intellectual property rights have been protected. Anyway, what is important (and will count more in the grant proposal Erin plans to submit next year) is that the work has now been openly approved by seven experts in the field. She is also positive that this paper will attract ongoing reviews and that she may even be invited as an expert reviewer herself now that she is more visible in the field. A debate has started in her department about how much the reviewer’s track record should weigh in how future tenure decisions are evaluated, and she has been invited to give a talk on her experience with LIBRE and the versioning of the group’s manuscript, which has now become a dynamic paper (Perakakis et al., 2011).”

I love this, in all details! I consider it among the most well written apology of, particularly, open peer review. [See if you care, also my post Open peer review as a service.]

From Your university is paying too much for journals, by Bjorn Brembs:

“Why are we paying to block public access to research, when we could save billions by allowing access?”

Oh, I’m sure that those in charge with these decisions have their reasons.

From the excellent We have met the enemy: part I, pusillanimous editors, by Mark C. Wilson

“My conclusions, in the absence of further information: senior researchers by and large are too comfortable, too timid, too set in their ways, or too deluded to do what is needed for the good of the research enterprise as a whole. I realize that this may be considered offensive, but what else are the rest of us supposed to think, given everything written above? I have not even touched on the issue of hiring and promotions committees perpetuating myths about impact factors of journals, etc, which is another way in which senior researchers are letting the rest of us down”…

Read also the older, but great We have met the enemy and it is us by Mark Johnston.  I commented about it here.

What is your opinion about all this? It’s getting hotter.

_________________________________________