Google DeepMind built SAFE, an AI system that breaks down long-form text into facts and checks them against Google search results. On a test of 16,000 facts, SAFE agreed with human annotators 72% of the time, but when they disagreed, SAFE was right 76% of the time. It costs about 20 times less than human fact-checking.
Google DeepMind has just unveiled a groundbreaking artificial intelligence system that boasts capabilities deemed superhuman in the realm of fact checking. This innovative AI system not only excels in verifying the accuracy of information produced by large language models, but does so with a level of efficiency and cost effectiveness that significantly surpasses human efforts. Michael Nunez, reporting for VentureBeat on March 28, 2024, highlighted this significant advancement, marking a pivotal moment in the ongoing evolution of AI technologies. In an era where the veracity of information is constantly under scrutiny, the introduction of such a system by Google's DeepMind is both timely and imperative. The technology known as the Search Augmented Factuality Evaluator, SAFE employs a sophisticated mechanism that leverages a large language model to dissect and analyze generated text, breaking it down into discrete facts. These facts are then subjected to rigorous verification against Google search results, ensuring an unprecedented level of accuracy in fact checking. DeepMind's innovative approach with SAFE is not just about verifying facts. It's a multifaceted process that involves a comprehensive breakdown of long form responses into individual facts. Each fact undergoes a meticulous evaluation process that incorporates multi step reasoning, including the issuance of search queries to Google search and the subsequent determination of factual accuracy based on the search results. This method was rigorously tested against a DataSet comprising approximately 16,000 facts, with SAFE's assessments aligning with those of human annotators 72% of the time. More impressively, in instances where disagreements arose between SAFE and human raiders, SAFE was found to be correct 76% of the time in a subset analysis of 100 facts. The notion of superhuman performance attributed to SAFE has ignited a debate among experts and observers. Gary Marcus, a renowned AI researcher and critic of hyperbolic claims within the AI community, has voiced concerns over the use of the term superhuman. He argues that surpassing the performance of underpaid crowd workers does not necessarily equate to superhuman capabilities. Marcus contends that a true measure of superhuman performance would require SAFE to be benchmarked against expert human fact checkers who possess a depth of knowledge and expertise far beyond that of average individuals or crowdsourced workers. The cost effectiveness of SAFE stands out as one of its most compelling advantages. Employing this AI system for fact checking purposes is estimated to be approximately 20 times less expensive than relying on human fact checkers. This economic efficiency is particularly significant in in the context of the exponential increase in the volume of content generated by language models. As we continue to navigate through an era of information overload, the need for an affordable, scalable and accurate fact checking solution becomes increasingly critical to further validate the efficacy of safe, the DeepMind team undertook a comprehensive evaluation of the factual accuracy of 13 leading language models across four distinct Gemini, GPT, Claude and Palm2. The evaluation, conducted as part of a new benchmark called Long Fact, revealed a general trend wherein larger models exhibited a reduced propensity for factual inaccuracies. However, it is important to note that even the models that performed the best were not immune to generating false claims, underscoring the inherent risks associated with overreliance on language models that that can articulate information fluently but inaccurately. In this context, the role of automatic fact checking tools like SAFE becomes indispensable, offering a critical safeguard against the dissemination of misinformation. The decision by the DeepMind team to open source the SafeCode and the long fact dataset on GitHub is a commendable move that fosters transparency and facilitates further research and development within the broader academic and scientific community. However, the need for more detailed information regarding the human benchmarks use in the study remains. A deeper understanding of the qualifications, experience and methodologies of the human annotators involved in the comparison with SAFE is essential for a comprehensive assessment of the system's true capabilities and performance. As the development of increasingly sophisticated language models continues at a rapid pace, spearheaded by tech giants and research institutions alike, the capability to automatically verify the accuracy of the outputs generated by these systems assumes paramount importance. Tools such as SAFE represent a significant advancement towards establishing a new standard of trust and accountability in the realm of AI generated content. Nonetheless, the journey towards achieving this goal is contingent upon a transparent, inclusive and rigorous development process. This includes benchmarking against not just any human fact checkers, but against seasoned, experts in the field to accurately gauge the real world impact and effectiveness of automated fact checking mechanisms in combating the pervasive issue of misinformation. Alright, don't forget to hit that subscribe button for more updates. Thanks for tuning in and we'll catch you in the next one.
