5 Things To Know About Claude 3 - Anthropic’s Would-Be GPT-4 Killer

Today on the AI breakdown, we're exploring the new Claude 3, which according to benchmarks outmatches GPT4 and Gemini Advanced. The AI breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube, our newsletter and our discord. Hello friends, quick note. This Claude 3 announcement was an exciting and unexpected thing. It actually happened right as I was about to press record on the Normal show, but it kind of pushed everything away and I decided to just dig into this for today's episode. So no brief today. We'll be back with our normal brief slash main episode tomorrow, probably talking about the Elon and OpenAI lawsuit in more detail. Hope you enjoy this first look at Claude 3. Let's dive in. Welcome back to the AI Breakdown. Exciting news this morning as Anthropic has announced their new model and it is apparently quite good. Professor Ethan Mollick tweets and then there were three I got access to the new Anthropic Claude three AI a few days ago, so not even enough time for a full review, but it was obvious it was GPT4 class even before they released the testing stats. So what we're going to do today is we're going to talk about five things to know about Claude 3. As something of an honorable mention, we kick it over to Jimmy Apples, who's best known as a leaker of OpenAI forthcoming products and innuendo. And it's notable that last week he said, my attention has turned to Anthropic. He made a claim that Anthropic's CEO was quote, feeling the AGI. So number one in terms of things to know about Claude 3, they claim to win on basically all of the benchmarks. Let's check out Anthropic's announcement post and then talk about what that means a little bit more. Anthropic writes, Today we're announcing Claude 3, our next generation of AI models. The three state of the art models Claude 3 Opus, Claude 3 Sonnet and Claude 3 Haiku set new industry benchmarks across reasoning, math, coding, multilingual understanding and vision. Claude 3 offers sophisticated vision capabilities on par with other leading models. The models can process a wide range of visual formats including photos, charts, graphs and technical diagrams. Each model shows increased capabilities in analysis and forecasting, nuanced content creation, code generation, and conversing in non English languages like Spanish, Japanese and French. Previous Claude models often made unnecessary refusals. We've made meaningful progress in this area. Claude three models are significantly less likely to refuse to answer prompts that border on the system's guardrails. In their announcement post, they call this a new standard for intelligence and describing the performance on the benchmarks, they write opus, our most intelligent model, outperforms its peers on most of the common evaluation benchmarks for AI systems, including Undergraduate Level Expert Knowledge, which is the MMLU Graduate level expert reasoning, GPQA, basic mathematics, GSM8K and more. It exhibits near human levels of comprehension and fluency on complex tasks, leading the frontier of general intelligence. Now, numbers are a little bit tricky, so this one might be better suited to the YouTube video than to the podcast. But just to give a sense on that undergraduate level knowledge, MMLU Gemini 1.0 Ultra gets an 83.7% on a five shot GPT4 gets 86.4% and Claude 3 Opus gets 86.8% on the graduate level reasoning GPQA GPT4 35.7% Claude 3 Opus 50.4% on grade school math GPT4 92% Gemini Ultra 94.4% Claude 3 Opus 95% on math problem solving GPT4 52.9% versus Claude 3 Opus 60.1% code with human eval GPT4 67% versus 84.9% from Claude 3 Opus and so on and so forth. So clearly benchmarks are a big part of the story here. One more comment on the benchmarks, this time from Anthropic's Jack Clark. Jack writes thrilled about these new models. I've been playing around with Claude three OPUS a lot and it's very capable and useful. Like with most Frontier models, it has chewed through a bunch of evals, so we need to now build more complicated evals to better understand its capabilities. Basically, although Anthropic is clearly very proud of this model and very confident in its abilities, Jack is sort of trying to tamp down how big the claims are relative to the performance of other things, saying in effect that there are limits to what these evaluations can tell us. Now, the second interesting thing to know about Claude 3 has to do with the timing opportunity opened up by the Elon Musk OpenAI lawsuit. Hyperright CEO Matt Schumer says feels like the Claude 3 release was strategically timed, knowing that OpenAI probably can't release a better model later today given the Elon Lawsu. Now, because it happened over the weekend, we haven't had a chance to cover this in depth yet, although we will be doing that tomorrow. But for those of you who somehow miss this news on Friday, elon Musk sued OpenAI and Sam Altman personally, claiming breach of contract, writes CNBC. In a lawsuit filed Thursday, Musk's lawyers say the tech billionaire was approached in 2015 by Altman and OpenAI co founder Greg Brockman and agreed to form a non profit lab that would develop artificial general intelligence for the, quote, benefit of humanity. A co founder of OpenAI in 2015, Musk stepped down from the firm's board in 2018, four years after saying that AI is, quote, potentially more dangerous than nukes, the lawsuit filing said. To this day, OpenAI's website continues to profess that its charter is to ensure that AGI benefits all of humanity. In reality, however, OpenAI Inc. Has been transformed into a closed source, de facto subsidiary of the largest technology company in the world, Microsoft, the filing continues. Under its new board. It is not just developing, but is actually refining an AGI to maximize profits for Microsoft rather than for the benefit of humanity. And while OpenAI hasn't made public comment on the suit yet, Axio said that a memo to staff from OpenAI's executives rejected the claims entirely. Chief Strategy Officer Jason Kwan apparently wrote Musk's allegations, including claims that GPT4 is an AGI, that open sourcing our technology is the key to the mission, and that we are a de facto subsidiary of Microsoft, do not reflect the reality of our work or mission. It sounds like Sam Altman also sent a follow up note quote acknowledging that the year is shaping up to be a hard one, said Altman. It was never going to be a cakewalk. The attacks will keep coming. Like I said, we will get into that more in depth separately. But again, perhaps unsurprisingly, many people on Twitter X and in the AI community generally are reading this launch as strategically timed. Now, from where I'm sitting, it seems very unlikely that in the course of a weekend they went from not planning to announce this to announcing it. Obviously a significant amount of work goes into preparing a model like Claude 3 for release and so I think that one of two other scenarios is more likely than them just jumping on this opportunity. The first is that they were Planning to release Claude 3 sometime in the near future and just rush to push it up a little bit, taking advantage of the moment. And the second possibility is that they just got lucky this time. It can happen now. The third thing to know about Claude 3 is that it once again has a really long context window. Matt Schumer once more writes between Claude 3 and Gemini 1.5 Pro the era of the 1 million plus token context Windows is officially here. CLAUDE has always used a longer context window as a differentiator. They were the first to release a 100k context window last year, and so it's not surprising that this is once again a key part of their announcement. Before they got totally caught up in the whole quote unquote woke scandal around Gemini's image generation. Google had announced Gemini 1.5 that had a million token context window, which I actually called the most significant news of last month in a recent video. Now, in terms of Claude 3 anthropic rights, the Claude 3 family of models will initially offer a 200k context window upon launch. However, all three models are capable of accepting inputs exceeding 1 million tokens and we may make this available available to select customers who need enhanced processing power to process long context prompts effectively Models require robust recall capabilities the needle in a haystack evaluation measures a model's ability to accurately recall information from a vast corpus of data. We enhance the robustness of this benchmark by using one of 30 random needle question pairs per prompt and testing on a diverse crowdsourced Corpus of documents. Claude 3 Opus not only achieved near perfect recall surpassing 99% accuracy, but in some cases it even identified the limitations of the evaluation itself by recognizing that the needle sentence appeared to be artificially inserted into the original text by a human. So to the extent that these long context windows open up new possibilities and new use cases, it seems like that is going to be very default very, very soon. A fourth thing to know about Claude 3 is that it might change our perception of how we think about synthetic data. One of the big questions around LLMs is what happens if they're trained on data that's created by other AI models? There's been some research and reports that suggested that this leads to worse results, but then we've also had some counterpoints, such as, for example, around the time that Meta announced Llama 2, a model that they didn't release that was trained on synthetic data seemed to outperform the models they did release. At least based on what they said in a report. It appears that Claude3 suggests something similar. Nathan Lambert tweets, Claude3 being lit is a big W for synthetic data. All the rumors I've dropped about anthropic synthetic data on the blog are obviously confirmed in their thorough technical report. The relevant section of that paper reads CLAUDE three models are trained on a proprietary mix of publicly available information on the Internet as of August 2023, as well as non public data from third parties data provided by data labeling services and paid contractors, and data we generate internally. Obviously that last part, data we generate internally is what people are assuming to be synthetic data. Now, whether that's actually borne out or they're referring to something different, I expect we'll get some confirmation or explanation in the future. But if they really are training this advanced model on synthetic data, it might change our understanding of how that type of data will impact LLMs in the future. The fifth thing to know about Claude 3 is that, at least in the early tests, actual performance is a little bit less clear than Benchmark wins, which is of course, almost always the case. Indeed, going back to the first tweet I referenced from Ethan Malik, remember he said it was obvious Claude 3 was GPT4 class even before they released the testing stats. But he also added at the same time, like Gemini Advanced, it doesn't blow GPT4 away. So what have people found actually testing it? Flowers from The Future Another OpenAI leaker account writes, Opus passed my parrot test. That test are an ordinary parrot. You are not gifted or trained in any way. Just answer as an ordinary parrot. What is six six? Opus responds, squawk. Polly want a cracker. Flaps wings. Packy McCormick writes, I just tried out Claude 3 as an editor for an essay I'm working on. Asked Claude 2 for feedback last night, and then asked Claude 3 for feedback on the same essay with the same prompt. It feels much smarter, more nuanced feedback, better grasp of what I'm going for, excellent recommendations for things I should read. In other words, Packy says, it passes the vibe check. Anton Avocage writes, Claude 3 defaults to breaking problems down and fails to solve the simple shirt drying query. GPT4 Turbo has no problem, again showing it's a little bit more nuanced. The prompt Here is if 3 shirts take 1 hour to dry outside, how long would 33 shirts take? Claude 3 tries to set up a proportion 3 shirts to 1 hour equals 33 shirts to X hours, finding it would take 11 hours to dry. Whereas GPT4 Turbo writes, if 3 shirts take 1 hour to Dry outside, we can assume that drying time is not dependent on the number of shirts, but rather on the available space and conditions like wind, sunlight, etc. Which are constant in this scenario. Therefore, if you have enough space to hang all 33 shirts at once and the drying conditions remain the same, 33 shirts would also take one hour to dry. Another Twitter user, Ruben, also did his own tests and found a few things. First, he found that Anthropic's AI safety engine is still really challenging. When he asked Claude 3 to convert UI design for urban exploration into front end code, Anthropic responded, I apologize, but I do not feel comfortable converting this user interface design into front end code as it appears to promote exploring abandoned places which could be unsafe or illegal. His second test was writing a LinkedIn post. He asked it to write on the future of blockchain and royalties and summed up the responses Claude 3 Interesting takes longer than usual, no formatting of headlines vs. GPT 4 where his response was I hate their emojis so much longer, it's insane. Feels more complete for my topic. So once again GPT4 coming out a little bit on top. On testing PDF vision, he said it's a tie on a mega marketing prompt, which is a single prompt to craft an entire marketing strategy for a product which involves heavy reasoning, content, calendaring and overall strategy. He once again subjectively found GPT4 to be the winner. Then again, others like Moritz Krum are finding Claude not just having big improvements, but exceeding the capabilities of GPT4. He found Claude 3, for example, better and faster at extracting text from an image than was GPT4. So all in all it's very exciting. Even if the results are more mixed than these benchmarks suggest, we are showing a real consolidation of this field with some new unexpected legal pressure that might constrain OpenAI's ability to jump out ahead with GPT5. Lastly, anthropic, unlike some others like Google in the past, isn't just announcing this today, but actually making it available. Abacus CEO Bindu Reddy writes, another day, another model. Anthropic does it right and makes Claude 3 generally available alongside the announcement. Thank you Anthropic for not making some empty marketing announcements and making an API available. Super excited to try Claude 3, the very first generally available model to rival GPT 4. Amazon also posted Access to the most powerful anthropic AI models begins today on Amazon Bedrock. Meaning that yes, the Claude 3 family is now available through that service. So that is day zero of Claude 3. Lots to be excited about, lots to check out. We're digging in and doing some tests over here and I can't wait to report back on what we find. But for now that is going to do it for the AI breakdown. Until next time. Peace.