Gamifying peer-review?

Fact is: there are lots of articles on arXiv and only about a third published traditionally (according to their statistics). Contrary to biology and medical science, where researchers are way more advanced in new publishing models (like PLoS and PeerJ, the second being almost green in flavour), in math and physics we don’t have any other option than  arXiv, which is great, the greatest in fact, the oldest, but … but only if it had a functional peer-review system attached. Then it would be perfect!

It is hard though to come with a model of peer-review for the arXiv. Or for any other green OA publication system, I take the arXiv as example only because I am most fond of. It is hard because there has to be a way to motivate the researchers to do the peer-reviews. For free. This is the main type of psychological argument against having green OA with peer-review. It is a true argument, even if peer-review is made for free in the traditional publishing model.  The difference is that the traditional publishing model is working since the 1960’s and it is now ingrained in the people minds, while any new model of peer-review, for the arXiv or any other green OA publication system, has first to win a significant portion of researchers.

Such a new model does not have to be perfect, only better than the traditional one. For me, a peer-review which is technical, open, pre- and post- “publication” would be perfect. PLoS and PeerJ already have (almost) such a peer-review. Meanwhile, us physicists and mathematicians sit on the greatest database of research articles, greener than green and older than the internet and we have still not found a mean to do the damn peer-review, because nobody has found yet a viral enough solution, despite many proposals and despite brilliant minds.

So, why not gamify the peer-review process? Researchers  like to play as much as children do, it’s part of the mindframe requested for being able to do research. Researchers are driven also by vanity, because they’re smart and highly competitive humans which value playful ideas more than money.

I am thinking about Google Scholar  profiles. I am thinking about vanity surfing. How to add peer-review as a game-like rewarding activity? For building peer communities? Otherwise? Any ideas?

UPDATE:  … suppose that instead of earning points for making comments, asking questions, etc, suppose that based on the google scholar record and on the keywords your articles have, you are automatically assigned a part, one or several research areas (or keywords, whatever). Automatically, you “own” those, or a part, like having shares in a company. But in order to continue to own them, you have to do something concerning peer-reviewing other articles in the area (or from other areas if you are an expansionist Bonaparte). Otherwise your shares slowly decay. Of course, if you have a stem article with loads of citations then you own a big domain and probably you are not willing to loose so much time to manage it. Then, you may delegate others to do this. In this way bonds are created, the others may delegate as well, until the peer-review management process is sustainable. Communities may appear. Say also that the domain you own is like a little country and citations you got from other “countries” are like wealth transfer: if the other country (domain) who cites you is more wealthy then the value of the citation increases. As you see, until now, with the exception of “delegation” everything could be done automatically. From time to time, if you want to increase the wealth of your domain, or to gain shares in it, then you have to do a peer-review for an article where you are competent, according to keywords and citations.

MORE: MMORPGames at the knowledge frontier.

Something like this could be tried and it could be even funny.

About these ads

21 thoughts on “Gamifying peer-review?”

  1. Something like http://mathoverflow.net/, where people try to earn reputation by answering research questions.

    needs:
    – a server
    – install stack exchange (http://meta.stackoverflow.com/questions/42613/install-stackoverflow-like-software-on-my-site)
    – instead of asking questions, add the links and abstracts to arXiv
    – votes are given for the paper, and for how useful is a review
    – can offer bounties to encourage referees, like in any stack exchange

    1. Hi Cristi, I am thinking more about how people happily take care of their own google scholar profile and how to turn vanity of “owning” a research area into a motivation for reviewing, and of course about some reputation system. But more like a kind of Game of life system. It’s nebulous, that’s why I ask, more heads always better statistically.

  2. The Cristi’s StackExchange-like system seems interesting, as it values both paper and reviews.

    However, reviews don’t need to be complete, and there is a lot of redundancy. So, except for “the general picture” (esp. grave flaws), there is a list of issues to be fixed. I propose putting a paper on GitHub (or something like that) as rising specific issues (also with possibility of proposing exact changes). People will be able to comment on that issues and up/down vote them.

    But except for reputation and visibility (like on the SE system), I guess there is little to do. Forcing to much into participation may backfire (i.e. produce a lot of bad reviews, because in some cases “it pays off”).

    1. Right, no forcing. I am still thinking, suppose that instead of earning points for making comments, asking questions, etc, suppose that based on the google scholar record and on the keywords your articles have, you are automatically assigned a part, one or several research areas (or keywords, whatever). Automatically, you “own” those, or a part, like having shares in a company. But in order to continue to own them, you have to do something concerning peer-reviewing other articles in the area (or from other areas if you are an expansionist Bonaparte). Otherwise your shares slowly decay. Of course, if you have a stem article with loads of citations then you own a big domain and probably you are not willing to loose so much time to manage it. Then, you may delegate others to do this. In this way bonds are created, the others may delegate as well, until the peer-review management process is sustainable. Communities may appear. Say also that the domain you own is like a little country and citations you got from other “countries” are like wealth transfer: if the other country (domain) who cites you is more wealthy then the value of the citation increases. As you see, until now, with the exception of “delegation” everything could be done automatically. From time to time, if you want to increase the wealth of your domain, or to gain shares in it, then you have to do a peer-review for an article where you are competent, according to keywords and citations.

      Something like this could be tried and it could be even funny.

  3. I am thinking for some time of a system of review, which may diverge at many points from what you want, but I think it has some common features.

    Briefly, I think that in time the reviews will be more collaborative, and in large part, automated.

    Imagine a paper, which contains a result, and has a part containing the mathematical proof, and a part of consequences and/or applications. I am thinking that the proof part may be, (hopefully more in the future), respect a format which would allow some software to check the proof. For example, calculations that can be done with Mathematica, Maple, etc, inferences that can be checked automatically, and confirm the result, etc. I mentioned this at http://mathoverflow.net/questions/127889/is-rigour-just-a-ritual-that-most-mathematicians-wish-to-get-rid-of-if-they-could/128811#128811
    The point is that an important and tedious part of the review can be automatized. Authors may try to replace the usual format of the proofs with something like this, and ease the work of the referees. So, large part of the peer review of the proof can be automatized. If this will be found useful, software for checking the proofs will be stimulated and develop more.

    Then, it remains to motivate the need for the proof, by providing references that the proof is needed, or by providing applications (which may also involve proofs). Think at an online system. Prof X said that it would be useful if conjecture C is proved, and you have such a proof. If there is a database of open problems, and a format which allows easily to search and compare them to eliminate duplicates, and volunteers to maintain this system, this part can be automatized as well. Volunteers did good jobs with Wikipedia, nLab, etc. So if we maintain a database with open proofs, and proposed solutions, this can be helpful. Now, suppose that the conjecture of Prof X is not present in the database. One can invite him/her, and update the database.

    So a paper can have multiple pieces of peer review, obtained automatically. If the pieces of the puzzle fit perfectly, the paper is reviewed, even if automatically, and not by human peers directly.

    Think at Perelman’s proof of the geometrization conjecture, and how long it took to be confirmed. Consider one or more peer “project manager”, who splits the paper in parts which need to be checked, and then volunteers taking parts of the puzzle to review. This would simplify and fasten the process, by solving subtasks in parallel.

    Normally, people read eprints which are not published, and they have opinions about parts, but not always enough to make a review. But they want to express them. And don’t like to waste time checking calculations and tedious proofs. I think the review process can be improved by partial reviews.

    1. Just to make what I said shorter. The central point is that both sides – the author and the reviewers – have to do something to improve the process. If the authors try to make the paper such that its some parts can be checked automatically, and other parts can be partially refereed, this will ease the reviewing process.

      1. “OK, but how do you motivate people to do that?”

        What is difficult, is to motivate referees to give non-anonymous reviews.

        But if the authors will have means to ease the publication process (whatever “publication” will mean), and if the reviewers will have smaller and more appropriate parts to do, this is enough motivation for both of them.

        If the author understands that there are means to ease the review process, it will apply to these, and the motivation is obvious: to have the paper accepted ASAP.

        If the reviewer is invited to review a piece of a paper, it may find easier to do this. And the reviewer may find easier to sign it, because she will vouch for a part of the paper, not necessarily for all. If the burden of checking proofs and of checking the importance of the result are taken from her shoulders, the reviewer will be more motivated.

        The author checks in many cases the proof anyway, and is interested in solving problems in which others are interested. The author asks advice from peers for some parts anyway, and communicate partial results before having material for a complete paper. These exchanges can also be gathered and reused as partial reviews.

        I see problems in the following.

        1. We will have to have a collaborative system to classify the results, and the open problems. This will be like a formalized wiki or nLab. If this will be seen useful by a critical mass, a “web of math knowledge” will be developed by volunteers, like wikipedia, but fully formalized.

        2. We will need to develop more the symbolic computation and automated proof software.

        In my opinion, these will happen in time anyway. In time, students will be required to learn how to use software to check proofs, will be assigned to put various proofs in the needed format and check them. Will be asked to collaborate to the “web of math knowledge”.

        Such tools help putting order in our knowledge, help knowing better what is already know and what is needed, and consequently will make easier writing papers (you will focus on ideas more than on technical things). Help being refereed easier.

        I think that all motivation exists, and the things are heading there anyway, although it is not visible yet. But blogs like yours will make the things evolve faster.

  4. I like the idea of gamifying a lot. What we’re attempting to do with http://www.peerevaluation.org is to put the whole process of writing, publishing, disseminating, peer reviewing and gathering metrics in the hands of scholars. For those who “get it” and who wish to take control over the dissemination of their work, the process becomes a rewarding and actually playful process. In other fields (music, video art, political activism etc.) this “hands on” approach is very common and truly successful — successful in the sense that independent creators, without any institutional backing are able to build an audience and to have a real impact and presence in their field. For now, not many scholars are using similar strategies. Building a readership and making connections is probably one of the most playful and engaging aspects of social media, extremely rewarding as well — it’s not only about seeing growing numbers of likes and views, it’s also about receiving a lot feedback, criticism, advice (peer review?), making new connections, being considered authoritative on certain topics, being offered collaborations etc.

    1. Thank you for the link, I was not aware of it. It looks excellent. Maybe it could be combined with this open citation graphing and with google scholar (or some open source variant of it?).
      Gamifying could exploit the competitive edge which any researcher has (who invented those role playing games. btw?). Not too much, but not too little also. After all, I am sure that in the research world there are people like Mala, the General Robotwallah from Cory Doctorow’s For the Win (wiki link).

  5. I would like to compare two apparently totally different experiences.

    One is from academia, which is to send the paper to journals, receive peer review, apply changes, send it again etc. Sometimes, this may take years, especially in mathematics.

    The other experience comes from my job as a programmer. You write a program, and ask the compiler to compile it. You get errors, fix them, and compile again. This takes minutes. Moreover, the compiler never tells you stuff like “you should refer to some libraries I wrote on this subject”, “I browsed your code, and I am pretty sure it won’t compile”, etc.

    I think this is a major possibility of improvement of academic peer review, and that’s why I proposed earlier to separate the proof from the motivation/consequences, to make it easier to check.

    Now, I admit that there is human peer review in programming, both in open source, where the contributors can be a huge number, and in small teams working at commercial software. Basically, after the code is compiled and tested, it undergoes peer review from other two programmers. Sometimes, they suggest you to use another design pattern, which they learned recently and are biased toward it, but most of the time the suggestions are useful and practical. But this peer review is usually very brief, due to the fact that the code already passed other tests. Then, other tests follow. One example is to check that the programmer applied the “good practices” from Scott Meyer’s “Effective C++” and “More effective C++”, but there are programs which do this automatically for most of these suggestions.

    In the academic world, it is important that a paper is checked and sent back or published asap, at least for the author, especially when he or she has a dissertation or other reason to be hurry. While these practices are more difficult to organize than in IT, I think we should automatize as much as possible – delegate the boring part to machines, and keep the fun part for us, and in the meantime increase efficiency.

    1. Glad you liked the post. Imagine an academic MMORPG, with kingdoms to rule, extend or defend (research areas), armies to lead (delegating and managing peer-reviews), citations as trade and so on. Based on visual representations (like graphs of interacting research fields, automatically generated), instead of reward points like in mathoverflow. As a side-product, an evolving map of the research world. It would be fun, I guess, many would love it and some would hate it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s