A short YouTube explainer (Two Minute Papers #84) describing scientific peer review, types of blind review, and the NeurIPS (NIPS) experiment that evaluated consistency by having multiple committees re-review the same submissions. The video reports that NIPS typically has an acceptance rate below 25%; the experiment found 57% disagreement between independent committees (i.e., one committee accepted while the other rejected), compared to ~77% disagreement for a hypothetical random (coin-flip) committee. The narrator notes that the top ~10% of papers are consistently accepted and the bottom ~25% consistently rejected, while papers in the middle are near-random; the video mentions crowdsourcing reviews as a potential avenue for improvement.
Questions this video answers
Below 25% (described as "remarkably low" and typically hovering below 25%).
57% of papers had disagreement (one committee accepted while the other rejected).
About 77% disagreement.
Top ~10% were consistently accepted; bottom ~25% were consistently rejected.
Chapters
Dear fellow scholars, this is two minute papers with we are here to answer a simple what is peer review? Well, in science, making sure that the validity of published results is beyond doubt is of utmost importance. To this end, many scientific journals and conferences exist where researchers can submit their findings in the form of a science paper. As a condition of acceptance, these papers shall undergo extensive scrutiny by typically two to five other scientists. This refereeing process we call peer review. Single blind reviewing means that the names of the reviewers are shrouded in mystery, but the authors of the paper are known to them. In double blind reviews, however, the papers are anonymized and none of the parties know the names of each other. These different kinds of blind reviews were made to eliminate possible people related biases. There's a lot of discussion whether they do a good job at that or not, but this is what they are for. After the review, if the results are found to be correct and the reviews are favorable enough, the paper is accepted and subsequently published in a journal andor presented at a conference. Usually, the higher the prestige of the publication venue is, the higher the likelihood of rejection. Which inevitably raises a big question, how to choose the papers that are to be accepted? As we are scientists, we have to try to ensure that the peer review is a fair and consistent process to measure. If this is the case, the NIPS experiment was born. NIPS is one of the highest quality conferences in machine learning with a remarkably low acceptance ratio which typically hovers below 25%. This is indeed remarkably low considering the fact that many of the best research groups in the world submit their finest works here. So here's the astute idea behind the nips. A large amount of papers would be secretly disseminated to multiple committees. They would review it without knowing about each other and we would have a look whether they would accept or reject the same papers. Re reviewing papers and see if the results are the same, if you will. At the given prescribed acceptance ratio, there was a disagreement for 57% of the papers. This means that one of the committees would accept the paper and the other wouldn't, and vice versa. Now to put this number into perspective, the mathematical model of a random committee was put together. This means that the members of this committee have no idea what they are doing and as a review they basically toss up a coin and accept or reject the paper. Based on the result, the calculations conclude that this random committee would have this disagreement ratio of about 77%. This is hardly something to be proud of. The consistency of expert reviewers is significantly closer to a coin flip than to a hypothetical perfect review process. So 57% disagreement, coin flip committee 77% disagreement. It is not as bad as the coin flip committee. So the question naturally where are the differences? Well, it seems that the top 10% of the papers are clearly accepted by both committees. The bottom 25% of the papers are clearly rejected. This is the good news, and the bad news is that anything in between might as well be decided with a coin toss. If the consistency of peer review is subject to maximization, we clearly have to do something different. Huge respect for the NIPS organizers for doing this laborious experiment for the reviewers who did a ton of extra work, and kudos for the fact that the organizers were willing to release such uncomfortable results. This is very important and is the only way of improving our processes. Hopefully someday we shall have our revenge over the coin flip committee. Can we do something about this? What is a possible solution? Well, of course this is a large and difficult problem for which I don't pretend to have any perfect solutions. But there is a really interesting idea by a renowned professor about crowdsourcing reviews that I found to be spectacular. I leave the blog post in the comment section both for this and the NIPS experiment, and we shall have an entire episode about this soon. Stay tuned. Thanks for watching and for your generous support and I'll see you next time.
