Is Mysterious GPT2-Chatbot Actually GPT5?

Today on the AI breakdown is GPT2 chatbot, actually GPT5 in the wild. Before that on the Brief, Marques Brownlee calls the rabbit R1 barely reviewable. The AI breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube Discord and our newsletter. Welcome back to the AI Breakdown Brief. All the AI headline news you need in around five minutes. Recently, prominent YouTuber and product reviewer Marques Brownlee got a lot of attention in the AI space when he called the Humane Pin the worst product I've ever reviewed. For now. Basically the gist of that review was that almost everything that was useful about the Humane Pin was better done with just your phone. However, at the same time he saw the glimpse of the future where it could be really nice to not have to be tethered to a screen. This kicked up a whole discussion about the role of reviewers and the nature of what product launches should be. And of course it was a little bit colored by the fact that Humane had been building this product for years and had spent hundreds of millions of dollars on it. But it also brought up this larger question of just whether AI hardware is ready for prime time. Well, that question has come up again with a vengeance now that Marques has dropped his review for the rabbit R1. He sums it up in two Barely reviewable. Interestingly, Marques has not just identified this as a problem in the AI hardware space, but sees it as the apex of a trend that has been growing for a long time. As he puts it, this is the pinnacle of a trend that's been annoying for years, delivering barely finished products to win a race and then continuing to build them after charging full price. Games, phones, cars, now AI in a box. Let's listen to a clip from his review to get the gist of what he's saying.

Like it feels like it used to just be make the thing and then put it on sale. Now it's like put it on sale and then deliver the half baked thing and then iterate and make it better and hopefully with enough updates then it's ready and it's what we promised way back when we first started selling it and then this whole period in the middle is a mess and it's across all kinds of product categories too. It's also happening with cars and vehicles getting announced and then delivering with like a half finished state where you just don't get a lot of the features that you paid for and they're eventually coming soon with a software update. You know smartphones, obviously we've been seeing this for years, but it does seem like now more than ever there's at least one feature, one major feature of every smartphone launch that gets announced, but that's not coming until later in the year. And now these AI based products are at like the apex of this horrible trend where they the thing that you get at the beginning is like borderline non functional compared to all the promises and all the features and all the things that it's supposed to maybe someday be. But you still pay full price at the beginning, which is what makes it so crazy.

Now let's talk about community responses to this. A lot of people agreed. Adam Vestika from the Shortcut writes, this is so true of gaming and it's a horrible trend that started during the last generation. I've got nothing against launching a game in early access, but so many titles are released with the promise of getting better down the line. Darrell Bossenjo writes, the rabbit R1 and humane AI pin are the culmination of a trend of launching unfinished products that are sold at full price, and then users get the promised features only after software updates. I saw many people compare this to treating your early customers like investors, which of course they're not. Dave Snyder writes, this is the byproduct of the product design and management strategy that pushes half baked products for a few data points, an overwrought practice where too many test underdeveloped solutions, groping blindly for signal while ignoring the glaring flaws. In the realm of physical products, this approach is an absolute nightmare, verging on fraud. Maybe, just maybe, we'll get back to shipping what's right any younger joked you An AI hardware CEO says I want to build a fun product that will improve the pace of AI progress. Your investors say ship an MVP and iterate do things that don't scale. Marcus Brownlee says, I about to end this man's whole career. And just as with the Humane Pin, where Marquez was representative of the larger sort of consensus review, take the rabbit R1 is also getting some pretty negative reviews. Even beyond Marques Snazzy Labs writes, the Humane AI Pin is one of the worst products I've ever reviewed and the rabbit R1 is even worse. Video later this week on why good ideas don't translate to good products, and why Apple and Google will clobber all of these AI startups by the end of the year. Now, just to add a little bit of controversial sauce on top of this, Snazzy Labs also added what I would like to reiterate is that the AI pin is leagues better than the R1. Yes, it's overpriced and not very good, but it knows what it wants to be and does a very limited number of things decently well. The same cannot be said of the Rabbit. They're not comparable. But what about the Rabbit team? Ryan Fenwick, who does communications over there, says honest feedback, which is cool. We're in the very early stages of the new AI hardware industry. The important thing is to move fast, continuously update the product and keep improving the experience. For those of you who are on this ride with us, Jesse Liu, the CEO, said, we shall see how fast R1 improves and evolves. We are a tiny team trying to catch the fast pace of AI. The current levels of AI need strong human supervised fine tuning. You can't take your time polishing features without real user testing. We will push the OTA fix as early as tomorrow to address most of the bugs we found. Thanks Marques Brownlee for your very detailed explanation of Lam looking forward for you to do an R1 revisit very soon. So I think that this tweet is important because it's not just a founder trying to justify that they're early, but is, at least in a small level, giving an argument for why this is actually the right process to use. Jesse writes, you can't take your time polishing features without real user testing. Joel Karenin responded to Snazzy Labs with another argument for why the rabbit R1 approach is better than the humane AI pen, saying, I would argue that the rabbit R1 is better because it's substantially cheaper with no subscription. It's meant to be an impulse gadget purchase rather than a lifestyle commitment. Snazzy Labs responded with the snazzy response. I will say you get what you pay for, but I think it's an interesting point. Is it more justified to charge someone $199 once to get that user feedback rather than $6.99 with an ongoing subscription? Now, there is a lot to unpack here. You can make all sorts of arguments from all sorts of angles. I think one of the key questions is consumer expectation versus reality. One of the things that clearly frustrates Marques is the gap between what these devices promise and what they actually deliver. In other words, I think that he would say that these companies are not presenting themselves as an early beta, but presenting themselves as the thing ready to go, ready to be in the world, and then trying to have it both ways, saying, we're just a tiny team and we're trying to grow fast when things don't go that well. Of course, Another line of common commentary, largely from entrepreneurs, was pointing out that every new technology seems barely reviewable at the beginning. Graham Fleming even shared a video of one of Marquez's very early reviews and said, if Marques Brownlee decided his initial videos were barely reviewable instead of posting and iterating, would he be where he is today? Now that's a great score for Twitter points. But of course the difference is that Marques wasn't asking people to pay money for something. Figma engineer Vivek writes, Absolutely love what Marques Brownlee is doing. I don't understand how people can defend dumping trash on consumers and overselling it as the next great thing. Nobody owes anyone a benefit of the doubt, don't want to get reamed by consumers and reviewers. Make something good I think that these questions are really interesting. It seems pretty clear so far that the agentic capacities of AI just aren't really sufficient for the agentic promises of the AI devices that have been announced and released so far. At the same time, that doesn't mean that a those devices won't actually catch up, especially as the agentic capacity of the AI software underneath catches up, and b it doesn't mean that AI hardware couldn't take a different approach focused on a specific type of use case that might actually fit better. For example, neither the Humane AI pin nor the Rabbit R1 are focused on the same use case that some of the next generation of AI devices seem to be of keeping a record of all your conversations and interactions in a way that allows you to go back and better recall that information. Will that be a killer application that doesn't require AI agent capacity? Maybe. Then again, we won't know until they get released. All in all, if nothing else is clear, it's that AI hardware is going to be a very difficult space. And so for now, anyone who buys into these products has to understand that that's what they are getting into. For what it's worth, I think by and large most people do. Now, that was obviously a little long for an AI breakdown brief, but it was a big point of conversation, so I'll just hit a couple more stories before we head on to the main part of the episode. First, Apple is apparently poaching from Google for a secretive AI research lab based in Zurich. Apparently they've hired something like 36 employees from Google over the last few years, much more than any other company they've poached from and Microsoft has announced its latest big global AI investment, with CEO Satya Nadella announcing yesterday that the company would invest $1.7 billion in Indonesia over the next four years. As part of that, they're also committing to help train 2.5 million people in Southeast Asia with AI skills, including about 840,000 in Indonesia itself. Anyways, friends, that is going to do it for today's AI Breakdown brief. Next up, the main AI breakdown all right breakers consensus 2024 marks the 10th gathering of the biggest event that's devoted to all sides of the crypto, BL and Web three ecosystems join pioneering thinkers and builders as they delve into the future of Defi and explore game changing tech from AI to ZK proofs and everything in between. The event is three days of jam packed content, networking and so much more. Some of the speakers at the event include Chris Dixon, the founder and managing partner at a16z crypto Sergey Nazarov, the co founder of Chainlink, Cathie Wood, the CEO of Ark Hester Purse Commissioner, of course from the U.S. sEC, and Tom Emmer, Republican majority whip for the U.S. house of Representatives. Visit consensus2024.coindesk.com to learn more and save 15% on registration with the code breakdown. That is 15% on registration with the code breakdown. Hello Breakers. Quickly, before we get back into the rest of the episode, you guys might have been following along my journey with the AI Breakdown, which is a very similar show to this, but for the artificial intelligence industry. One of the things that I found as I dug into that show is that there was a huge need for better educational resources that were actually practical and useful right away. I've just announced a new company and platform called Super Intelligent that's trying to build exactly that. It's a video platform for learning AI that features fun, fast and immediately useful video tutorials. Each video tutorial is around five minutes and comes with a step by step how to that gets people actually using the tools that we're talking about. We've just gone live with more than 300 tutorials and are adding 30 to 50 more per week. To check it out, go to Be Super AI. That's Be Super AI. Can't wait to see you guys over there. Welcome Back to the AI breakdown. It has been a minute since an OpenAI model was where we were focused. For the last couple weeks everything has been about Llama 3. In fact, as we've discussed on this show, even the small models of llama 3 that have been released so far come sufficiently close to GPT4 level performance that it's made people think very differently about the competitive landscape of LLMs. We're getting serious questions around whether if open source keeps being this close to the state of the art, does it fundamentally change the economics of advanced models? Will people just all opt to build on top of llama 6 instead of paying a premium for GPT6? Anyways, the point is that Zuckerberg has been once again dominating the conversation, but apparently OpenAI was sick of that because for the last day or so everyone has been talking about a model called GPT2 chatbot, which is rumored to secretly be GPT4.5 or even GPT5 out in the wild in advance of an official launch. So where was this model found? Well, it was on the Limsys Chatbot arena. This is an LLM benchmarking site and it appeared as one of the model options which people on the site could test. As Dan Shipper from every put it, limsys.org enables users to chat with various LLMs and rate their output according to different benchmarks without needing to log in. One of the models recently available is GPT2 chatbot. There is no information to be found on that particular model name anywhere on the site or elsewhere. The ratings results generated by Limsys benchmarks are available via their API for all models except for this one. The model name simply appears to be a cover for something else entirely. Yesterday afternoon, Professor Ethan Malik wrote, there is a mysterious new model called GPT4 chatbot accessible from a major LLM benchmarking site. No one knows who made it or what it is, but I have been playing with it a little and it appears to be in the same rough ability level as GPT4. A mysterious GPT4 class model. Neat. One additional thing that Ethan noted was that anonymous testing is a service that limsys.org offers. This came out after Simon Willison responded to Ethan and said, feels to me like a bit of a reputation risk to Limsys. If this is indeed a stealth model launch, there's supposed to be a neutral benchmarking tool, which is not a great look if they're working behind the scenes with model vendors in an opaque manner like this. LIMSYS.org responded to Simon saying, just to clarify, following our policy, we've partnered with several model developers to bring their new models to our platform for community preview testing. These models are strictly for testing and won't be listed on the leaderboard until they go public. In other words, this is a thing that they do now. Of course, it wasn't just the fact that this had a GPT name, but the fact that it appeared really good that got people talking. Everard founder Pietro Scurano A mysterious new model called GPT2 chatbot has appeared on Limsys, and it's really good. Not only does it show incredible reasoning, but it also gets notoriously challenging AI questions right with a much more impressive tone. The example that he shared was a question what weighs more a kilo of feathers or a kilo of lead? This is a historically difficult problem for LLMs, but GPT2 chatbot says a kilogram of feathers and a kilogram of lead weigh the same. They both weigh 1 kg. The key difference between the two is in their volume and density. Kilogram of feathers will take up much more space and appear much larger, while the kilogram of lead will be compact and small. This often leads to the classic trick question, but in terms of weight, a kilo is a kilo. It also apparently is good at math. Andrew gao writes, A GPT2 chatbot just solved an international Math Olympiad problem in one shot. This, in my opinion, is insanely hard. Only the four best math students in the US get to compete. CodeGen founding engineer Chase writes, Can confirm GPT2 chatbot is definitely better at complex code manipulation tasks than Claude Opus or the latest GPT4 did better on all the coding prompts we used to test new models. The vibes are def there. The model is also apparently good at code art. Phil on Twitter writes, GPT2 chatbot is insane at ASCII art, miles ahead of any other model. With Llama 370B, when asked to create an ASCII art unicorn, the output is very janky, whereas with GPT2 chatbot it completely nails it. Cool. Zippity on Twitter also asked about art saying, I asked GPT2 chatbot to generate a simple program for turning doodles into art. Every other model I've tested fails at this. They get a doodle function, but the button does nothing of artistic value. The version that he shares does a much better job. So what are the theories for what this model could actually be? Brian Romley writes, I've been testing GPT2 chatbot for a few days. Today it seems to have gotten much more attention. It surpassed all of our ChatGPT4 benchmarks hypothesis. A few of us concluded it is a form of pre lobotomized ChatGPT4 or heavily trained on it. Runway CEO Siki Chan writes, My best guess GPT4 knowledge plus Q search reasoning equals GPT2 general knowledge seems near identical to GPT4 with much better reasoning and planning capabilities. More expensive inference from tree of thought search would explain relatively slow inference and low rate limit. If true, this is a much bigger deal than it seems. GPT2 is likely to feel pretty similar for most general knowledge queries, but will outperform on reasoning. So I don't think it's an accident that this isn't named GPT 4.5 or GPT 5. It is neither. It's a test bed for Q or whatever you want to call tree of thought plus PRM these days. Q, you'll remember, was a rumored OpenAI reasoning model that got a lot of attention last year. Continuing Siki writes, the next GPT5 will likely continue to ride on the scaling hypothesis plus a reasoning boost from this. Sikhi also writes and no, it isn't GPT2 fine tuned. You are all out of your mind. The knowledge cutoff alone would make that make zero sense. What he's referring to is that another theory is summed up here by Albs, who writes, my guess is this mysterious GPT2 chatbot is literally OpenAI's GPT2 from 2019 fine tuned with modern assistant datasets, in which case that means their original pre training is still amazing and better than everyone else's four years later. Admittedly, most people did not agree. Julian Chamond, the CTO of Hugging Face, writes, my personal guess on GPT2 chatbot given Omar Sanseverio, the chief llama officer at Hugging Face, has been off for the last 10 days at this stage. Strongly suspect it's a side project of his gone viral. Now this was a little tongue in cheek of course, but just speaks to how little information we actually have. What about what GPT2 said about its origins? Andrew Gao again writes, it told me and others that it was made by OpenAI. This is a weak signal though, because of data contamination. A lot of models are trained on OpenAI chats and thus think they were made by OpenAI. When he pulled to ask what do you think GPT 4.5, GPT 5 Grok 2 or another AI company? 58.9% of 2100 voters said GPT4.5 and while that reflects the quality that people are seeing, some have pointed out that if this was the jump between 4 and 4.5, they wouldn't be real happy about that either. Matt schumer from Hyperrite AI says GPT2 chatbot is good, really good. But if this is GPT 4.5. I'm disappointed. Flowers from the future the frequent OpenAI leaker writes, GPT2 isn't better than current GPT4 Turbo, so it's definitely not 4.5 or 5, and not even 4. Either this is a new GPT4 Lite model, or it really is a new GPT2 model with a totally new kind of training or processing. The implications of the latter would be absolutely crazy. Being no help in adding more mystery to the whole thing was Sam Altman, who tweeted I do have a soft spot for GPT2. Funny enough, he had initially written it as GPT2, but then about 8 seconds later edited it to I do have a soft spot for GPT2. No Dash, which of course sent a whole group of people speculating on what that might mean. Even Malik again pointed out the frustration of the crypticness of the industry, saying OpenAI may be one of the most important technology companies in the world today, but they really like to communicate through hints and Oracle whispers what is GPT2? At this rate, we will only know that GPT5 is being launched from an I Love Bees esque alternate reality game. GPT6 will be known by the shapes made by the wheeling of starlings over Palo Alto as well as certain signs in the heavens. And the first letters of every third tweet by Run Smoke Away writes, GPT2 is not the AGI you're looking for, and I think ultimately that's where we're going to land on this. This mystery will likely at some point be solved, or it won't, and it'll just stay mysterious. But the reason that there's so much attention on it is that for as much as we were talking about last week around how a close to GPT4 class open source model could change the game, people are still obsessed with the frontier. They are still obsessed with the true state of the art. Right now. It doesn't seem like that's going to change anytime soon. So for the moment we're just going to have to be content with this mystery. Sure, it's speculative, but it's not the least fun I've ever had in the AI space. Anyways, friends, that is going to do it for today's AI breakdown. Until next time, peace.