Google AI researchers argue peer review is broken: reviewers work for free while disagreement between them is high, with NeurIPS finding 57% of papers get conflicting verdicts. They propose a seven-point rubric to grade reviews and paying reviewers through sponsorships or fees.
Dear fellow scholars, this is Two Minute Papers with it is time for a position paper. This paper does not have the usual visual fireworks that you see in many of these videos. However, it addresses the cornerstone of scientific publication, which is none other than peer review. When a research group is done with the project, they don't just write up the results and chuck the paper into a repository, but instead they submit it to a scientific venue, for instance a journal or a conference. Then the venue finds several other researchers who are willing to go through the work with a fine tooth comb. In the case of double blind reviews, both the authors and the reviewers remain anonymous to each other. The reviewers now check whether the results are indeed significant, novel, credible and reproducible. If the venue is really good, this process is very tough and thoroughly and this process becomes the scientific version of beating the heck out of someone, but in a constructive manner. If the work is able to withstand serious criticism and ticks the required boxes, it can proceed to get published at this venue. Otherwise it is rejected. So what we heard so far is that the research work is being reviewed. However, scientists at the Google AI Lab raised the issue that the reviewers themselves should also be reviewed. Consider the fact that all scientists are expected to spend a certain percentage of their time to serve the greater good. For instance, throughout my PhD studies, I have reviewed over 30 papers and I am not even done yet. These paper reviews take place without compensation. Let's call this issue number one for now. Issue number two is the explosive growth of the number of submissions over time at the most prestigious machine learning and computer vision conferences. Have a look here. It is of utmost importance that we create a review system that is as fair as possible. After all, thousands of hours spent on research projects are at stake. Add these two issues together and we get a system where the average quality of the reviews will almost certainly decrease over time. Quoting the Authors we believe the key issues here are structural. Reviewers donate their valuable time and expertise anonymously as a service to the community with no compensation or attribute, are increasingly taxed by a rapidly increasing number of submissions, and are held to no. N4 standards in two minute papers. Episode number 84 so more than 200 episodes ago we discussed the Neurips experiment. Leave a Comment if you've been around back then and you enjoyed two Minute papers before, it was cool. But don't worry if this is not the case. This was long ago. So here's a short summary. A large number of papers were secretly disseminated to multiple committees who would review it without knowing about each other and we would have a look whether they would accept or reject the same papers. Re review papers and see if the results are the same, if you will. If we use sophisticated mathematics to create new scientific methods, why not use mathematics to evaluate our own processes? So after doing that, it was found that at a given prescribed acceptance ratio there was a disagreement for 57% of the papers. So is this number good or bad? Let's imagine a completely hypothetical committee that has no idea what they are doing. And as a review they basically toss up a coin and accept or reject the paper based on the result of the coin toss. Let's call them the coin flip committee. The calculations conclude that the coin flip committee would have a disagreement ratio of about 77%. So 57% disagreement, coin flip committee, 77% disagreement. And now to answer whether this is good or bad. This is hardly something to be proud of. The consistency of expert reviewers is significantly closer to a coin flip than to a hypothetical perfect review process. If that is not an indication that we have to do something about this, I am not sure what is. So in this paper, the authors propose two important changes to the system to remedy these issues. Remedy 1 they propose a rubric, a seven point document to evaluate the quality of the reviews. Again, not only the papers are reviewed, but the reviews themselves. It is similar to the ones used in public schools to evaluate student performance, to make sure whether the review was objective, consistent and fair. Remedy 2 reviewers should be incentivized and rewarded for their work. The authors argue that a professional service should be worthy of professional compensation. Now of course this sounds great, but this also requires money. Where should the funds come from? The paper discusses several options. For instance, this could be funded through sponsorships or asking for a reasonable fee when submitting a paper for peer review and introducing a new fee structure for science conferences. This is a short five page paper that is easily readable for everyone, raises excellent points for a very important problem. So needless to say, I highly recommend that you give it a read. As always, the link is in the video description. I hope this video will help raising more awareness to this problem. If we are to create a fair system for evaluating research papers, we better get this right. Thanks for watching and for your generous support and I'll see you next time.
