Llama 3.1 405B Eliminates Gap Between Open and Closed Source AI

Today on the AI Daily Brief, Meta's Llama 3.1405 B may have closed the gap in the state of the art between open and closed source models. Before that on the brief, OpenAI updates their safety policies without actually updating anything. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show. Notes welcome back to the AI Daily Brief Headlines edition. All the AI Daily news you need in around five minutes. We kick off today with the Latest announcements from OpenAI on their safety strategy. They tweeted very early this morning east coast time Making sure AI can Benefit Everyone starts with building AI that is helpful and safe. We want to share some updates on how we're prioritizing safety in our work. We believe that frontier AI models can greatly benefit society. To help ensure our readiness, our preparedness framework helps evaluate and protect against the risks posed by increasingly powerful models. We won't release a new model if it crosses a medium risk threshold. Until we implement sufficient safety interventions, we are developing levels to help us and stakeholders categorize and track AI progress. This is a work in progress and we'll share more soon. Editors Note this is, I think what we got some information about last week. With their various levels of AI proceeding towards AGI, this tweet thread continues. However, in May our Board of Directors launched a new Safety and Security Committee to evaluate and further develop safety and security recommendations for OpenAI projects and operations. The committee includes leading Cybersecurity expert, retired U.S. army General Paul Nakasone. This review is underway and we'll share more on the steps we'll be taking after it concludes. Our whistleblower policy protects employees rights to make protected disclosures. We also believe rigorous debate about this technology is important and it made changes to our departure process to remove non disparagement terms. Safety has always been central to our work, from aligning model behavior to monitoring for abuse, and we're investing even further as we develop more capable models. Now what's notable to this, at least to me, is that there's nothing new that's actually announced. They frame it as we want to share some updates on how we're prioritizing safety, but there's nothing actually new here. This suggests to me that there was some reason that they felt they wanted to remind everyone of these efforts and put it in a nice easy to point to place, but not that they had something big and new that they wanted to highlight. As I was wondering about what the reasoning was that might be behind this. It started to pop up on X that a group of five senators had sent Sam Altman a note. The note, which was dated yesterday, July 22, reads, Dear Mr. Altman, we write to you regarding recent reports about OpenAI safety and employment practices. OpenAI has announced a guiding commitment to the safe, secure and responsible development of artificial intelligence in the public interest. These reports raise questions about how OpenAI is addressing emerging safety concerns. We seek Additional information from OpenAI about the steps the company is taking to meet its public comments on safety, how the company is internally evaluating its progress on those commitments, and on the company's identification and mitigation of cybersecurity threats. The letter goes on to ask 12 questions requesting that information be shared by August 13th of this year. One of the questions that many on Twitter have commented on is this will OpenAI commit to making its next foundation model available to US government agencies for pre deployment, testing, review, analysis and assessment? Now, it's likely that I'm going to dig deeper into each of these questions in an episode later this week. So for now I will just note that it seems like there is an increasing moment of scrutiny around OpenAI from the US government and that that might be driving some of this increased conversation around safety and preparedness. Next up, Perplexity is back in the news, although this time not for a funding announcement. Amazingly, no. Unfortunately for them, this time it's because Conde Nast has sent a cease and desist letter to the company. The information reports that Conde Nast, which owns publications like the New Yorker, Vogue and Wired, has sent a cease and desist demanding that Perplexity stop using content from these publications in its search results. This follows a letter last month from Forbes, where Forbes accused Perplexity of infringing on its copyright. Conde Nast similarly claims that Perplexity is plagiarizing its content. It seems that the ire around Perplexity isn't around training data, as that's not what Perplexity does, but around the way that Perplexity's AI summarizes news articles. Now, there are a lot of specific technical details in here around whether and where the bots that companies like Perplexity use to allow publishers to block their crawlers actually work. But ultimately this comes down to this new format for search. It feels to me fairly inevitable that the sort of combination of AI generated summary plus links is going to be the default in norm in the future. But the path to get there may be littered with a lot of legal battles proving once again that when it comes right down to it, the only group of people who always win are the lawyers. Moving to some Science A new Google model has, quote, helped make a breakthrough in accurate long range weather and climate predictions, writes the Financial Times. Using a hybrid of machine learning and existing forecasting tools, a model led by Google called Neural GCM successfully harnessed AI to conventional atmospherics physics models to track decades long climate trends and extreme weather events such as cyclones. A recent paper said Neural GCM proved faster, more accurate and used less computing power in tests against a current forecasting model based on atmospheric physics tools called X Shield. In one trial, they say, Neural GCM identified almost the same number of tropical cyclones as conventional extreme weather trackers did, and twice the number of X shields. In another test, based on temperature and humidity, the error rate was between 15 and 50% less one of the things that often gets lost as we debate the big legal and ethical issues of AI is how much it's likely to impact scientific discovery. But that ultimately is going to be the subject for another show that is going to do it for today's AI Daily Brief Headlines edition. Next up, the main episode. Today's episode is brought to you by Super Intelligent, the platform for fun, fast AI learning. Super has a ton of new things going on. We recently announced our partnership with Spotify, through which users of that app can now access Super Intelligent content directly from their mobile apps. We've also just launched the AI Learning Feedback. In addition to seeing the tutorials that we're dropping, there are polls, news items with related lessons and a chance for people to show off the projects and use cases that are making AI come alive for them. We've also just kicked off the Super Summer Challenge, where each week we'll share a new challenge that you can use to discover new AI tools and use cases. Go to BeSuper AI and use code SUPERFUND for 50% off your first two months. That's BeSuper AI. Today's episode is brought to you by Venice. Venice is a private, uncensored, generative AI app. It accesses open source models to enable text, image and code generation without the fear of being spied on or having your data exploited. Discuss anything with Venice without concerns about it being monitored, sold or given to advertisers and governments. Venice is different because your conversations and creations are kept securely within your own browser, never stored or accessible by Venice. Unlike other AI apps, Venice won't tell you what's okay to say or not. Venice won't patronize you. It simply provides direct access to machine intelligence. No topics are off limits. No ideas are taboo. With Venice, you're in control of the AI as you should be. Pro subscriptions are available for $49 a year or $8 per month. Try it for free without an account at Venice AI. Welcome back to the AI Daily Brief. Today we are talking about some leaks around the forthcoming meta llama 3.1405 B model. As Didi from Menlo Ventures puts it, this is potentially the biggest news in AI in several weeks. Llama 3.1405 b leaked day before on 4chan and obliterates GPT4O on most benchmarks, writes Runway Sikichen. For the first time ever, an open source model is state of the art, outperforming OpenAI's GPT4.O and Anthropic Sonnet 3.5 across multiple benchmarks. So today we are going to talk about what we have learned so far about Llama 3.1405 b and what the implications are of the full closure of the gap between open source and closed source. Now to go back we have been waiting for some time for Meta's largest Llama 3 model. About a week and a half ago, we got reports from the Information and others that Meta was planning on releasing the 405 billion parameter version of Llama 3 today on July 23rd. Back in April, we got two smaller models from Llama 3, including the 8 billion and 70 billion parameter models. But this was always going to be the big show. In a post last week about why we should care about this release, Tom's guide wrote, the Llama 3 400B model is particularly exciting as it approaches performance parity with OpenAI's GPT4O model despite using less than half the parameters. Apart from the potential benefits to cost and energy efficiency, there's another significant advantage. One of the most compelling aspects of llama 3 is its open license for research and commercial use. If the 400B model is released under the same open license, it would democratize access to state of the art language capabilities, allowing researchers and developers to leverage this powerful tool for their projects without relying on expensive proprietary APIs. Still, as Tom's Guide pointed out, there had also been some scuttlebutt that Meta was not going to exactly open source this model. Notorious OpenAI leaker Jimmy Apples wrote Meta plans to not open the weights for its 400B model. The hope is that we would quietly not notice and let it slide. Don't let it slide, he followed up saying. Dustin Moskovitz, who is one of the big funders of the EA space and the AI safety movement, is quote, having a loud voice to the ears of lab CEOs behind doors. Then later, however, Jimmy Apple's updated to say apparently at the moment they do plan to open source it. Despite Dustin's objections to Zuck, Open source AI advocate and AI CEO Bindure Reddy made this prediction In a couple of days we will stop talking about politics. Hopefully llama 405B will be the topic of the day. The gap between the closed and open weight models will finally close. And indeed that is what seems to have happened, at least with the information we have so far. Matthew Berman says Suddenly the world has access to an open source model considered state of the art. It beats GPT4O on many benchmarks what a time to be alive. Dataconomy writes in a dramatic turn of events, early benchmarking data for the forthcoming llama 3.1B models, including 8B, 70B and the colossal 405B, were leaked on the local LLAMA subreddit today. The preliminary results suggest that llama 3.1405 b could potentially surpass the performance of the current industry leader, OpenAI's GPT4O across several critical benchmarks. Should that happen, it would represent the first instance of an open source model eclipsing a leading closed source LLM and a couple other things that the community has honed in on hold aside the question of whether it is the state of the art and which of the benchmarks we should care most about if these benchmarks are anywhere close to accurate. The thing that is for sure is that it is a GPT 4.0 class model. In the same way that we can debate around whether GPT4O or Claude 3.5 Sonnet is exactly the better or more performant model. This would add llama 3.405b to that conversation. It's important to note here then, in addition to Llama 3.1405b, we're also getting the 3.1 updates to the 8B and 70B models. Maxime Le Bon writes, The new 70B also looks insane with a significant boost of performance compared to the previous version. And that I think is a part of the story that is actually flying a little bit under the radar comparatively. That is how much better these smaller models have gotten as well. Chris at Hinge Loss also noted that there seemed to be an updated license which removed the Prohibition on using LLAMA 3 to improve other models. He used Diff checker to compare the 3.1 versus 3 llama license. Going back to this idea that the big story once everything settles, might be the smaller models. Kyle Corbett points out that if the leaked benchmarks are correct, llama 317 0B beats GPT4L mini. Aydin McCloud really sums it up when he says, if these llama 3405 benchmarks are real, this will be the world's best model in the hands of everyone to tune cheaper than GPT4O it's hard to overstate how fast everything is changing. Picking up on some of the themes that we were discussing in yesterday's show about the move of competition to the smaller end of the spectrum, Layton Space's Swix called what's happening the under 100B model red wedding. He writes, I do not think that people who criticize OpenAI have sufficiently absorbed the magnitude of disruption that has happened because of 4oh mini. He points out that both Llama 370B and GPT 4oh mini score an 82 on the MMLU, but GPT 4o mini is priced at a sixth of the price of Llama 370B. He points out that many of the models that were state of the art just three months ago are now being dominated by new counterparts. Swix writes, what's the depreciation rate on the flops it took to train them? GPT4 took 500 million to train and it lasted a year. Intelligence Too cheap to meter, but also too ephemeral to Support greater than 5 players doing R and D? Is there an angle here I'm missing? SWIX also compared the new 3.1 benchmarks to Llama 3.0 and came to the conclusion that there was a huge bump for the 8B and that the instruct 70B is mildly better, but that the 405B is still behind flagship models. Now we are slated to actually get this model today and should be able to independently start verifying these benchmarks, and until then we won't know exactly what the situation is. However, if things are close to what they appear to be, we really are, it appears, living in a new paradigm where open source has by and large caught up and closed the gap with closed source. One thing I will be watching is whether that prompts OpenAI and Anthropic to release new models that redefine the state of the art once again. For now though, pretty interesting times. Lots to pay attention to and of course I appreciate you hanging out and listening or watching as always. Until next time peace.