Just How Good is Grok-3?

Today on the AI Daily Brief, Grok3 kicks off what appears to be the beginning of model update season. Before that in the headlines, Perplexity launches their own version of Deep Research. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show. Notes. OpenAI's deep research is one of the more exciting products that many people have gotten recently. In fact, if you go on Twitter or X you can find people saying it's the most impressive product they've seen in years. It is, however, behind an extremely expensive paywall. Right now, the only people who have access to Deep Research are those who are paying for OpenAI's $200 Pro tier. And then in comes Perplexity with their own version of Deep Research with the same name, in fact suggesting that they're trying to make this just a category of AI usage like chatbot. And it absolutely obliterates the OpenAI price point. Free users have five queries per day, while pro users get up to 500 daily queries and have access to faster speeds. When asked how the company could offer this tool at this price, CEO Aravan Srinivas said, thankful for open source. We're going to keep making this faster and cheaper. Knowledge should be universally accessible and useful, not kept behind obscenely expensive subscription plans that benefit the corporates, but not in the interests of humanity. So yes, if you're wondering, Sam Altman is now being shanked from below and from above given the aggressiveness of that particular positioning. Perplexity's Deep Research works very similarly to how rival tools work, using a combination of agentic web search and iterative reasoning to generate in depth research reports. They share a bunch of benchmarks, but honestly, I think for this type of product everything is about how it actually performs, and for that you're just going to have to go check it out yourself. Which thankfully you can, given that they offer even free users some number of queries each day. One user asked Perplexity to compare itself to rival Deep Research features, ultimately producing a multi page analysis that summarized perplexity AI excels in speed and accessibility for casual researchers. OpenAI dominates in analytical depth for enterprise applications. Google integrates most seamlessly with existing productivity ecosystems, which honestly seems like a fairly decent write up in summarization. Now if you go cruise around the Internet, you can find people who are saying that Perplexity's version of the tool is every bit as good or even better than OpenAI's, but you also have a lot of sentiment like this one from Siki Chen who writes until you have access to full O3 or Claude 4 or something, you simply are not going to build a better deep research than OpenAI. This is a use case where the raw model reasoning capability matters a lot. Still, from a consumer perspective, obviously more options is a good thing and so glad to see some competition in this space. Next up, an update on Ilya Sutskever, the former OpenAI co founder who is back out fundraising for his new company Safe Superintelligence. Previous reports had Ilya raising about a billion dollars at a 20 billion valuation and it seems like that is now up to a $30 billion plus valuation. Bloomberg reports that Green Oaks Capital Partners will lead the round and plans to invest about half of it. And we still have no idea whether the valuation update from the original $5 billion reflects something new that Ilya has shown investors or is just the premium that the market feels it has to pay for any Ilia product now While the startups like Perplexity race ahead, don't expect the next generation of AI enabled home assistants anytime soon as the big tech companies are struggling, both Alexa and Siri have hit another round of delays. People got excited recently when it was reported that there was an Alexa AI event, but at a last minute go no go meeting last week, apparently Amazon's executives decided that no go was the answer and the Washington Post is now reporting that AI Alexa won't be ready until March 31 or later. The delay is reportedly due to Alexa giving inaccurate answers, which has been the scourge of this development cycle. Apple's AI Siri upgrade is also facing delays after plans were first unveiled all the way back last June at wwdc. Bloomberg reports that the project is facing engineering problems and software bugs, and that while Apple is quote, racing to the finish line, some features planned for an April rollout may be delayed until May or even later. One of the things that this highlights is that the margin of error and consumer forgiveness for AI hallucinations and incorrect answers when it comes to these sort of smart home devices is basically zero. And the risk of finding yourself on the wrong end of some viral clip on social media is really high, making these particular product rollouts a real challenge. Lastly, today, Meta is apparently planning a big investment in humanoid robots. The company will establish a new team within their Reality Labs hardware division, which is the group that has released the Meta Ray Bans and the Meta Quest. The new plan is to develop Meta's hardware for humanoid robots designed to complete household tasks initially focusing on developing sensors to be used by third party startups. In an internal leaked memo, Meta CTO Andrew Bosworth said the core technology we've already invested in and built across Reality Labs and AI are complementary to developing the advancements needed for robotics. We believe that expanding our portfolio to invest in this field will only accrue value to Meta, AI and our mixed and augmented reality programs. I think we're still a little premature, but you are going to see a lot more of the intersection of robotics and AI this year and in the years to come. For now though, that is going to do it for today's AI Daily Brief Headlines edition. Next up, the main episode. Today's episode is brought to you by Vanta Trust isn't just earned, it's demanded. Whether you're a startup founder navigating your first audit or a seasoned security professional scaling your GRC program, proving your commitment to security has never been more critical or more complex. That's where Vanta comes in. Businesses use Vanta to establish trust by automating compliance needs across over 35 frameworks like SoC2 and ISO 27001, centralized security workflows, complete questionnaires up to 5x faster, and proactively manage vendor risk. Vanta can help you start or scale up your security program by connecting you with auditors and experts to conduct your audit and set up your security program quickly. Plus, with automation and AI throughout the platform, Vanta gives you time back so you can focus on building your company. Join over 9,000 global companies like Atlassian, Quora and Factory who use Vanta to manage risk, improve security in real time. For a limited time, this audience gets $1,000 off vanta@vanta.com NLW that's V A N T A.com NLW for $1,000 off. If there is one thing that's clear about AI in 2025, it's that the agents are coming. Vertical agents by industry, Horizontal agent platforms Agents per function if you are running a large enterprise, you will be experimenting with agents next year and given how new this is, all of us are going to be back in pilot mode. That's why Superintelligent is offering a new product for the beginning of this year. It's an agent readiness and opportunity audit. Over the course of a couple quick weeks, we dig in with your team to understand what type of agents make sense for you, to test what type of infrastructure support you need to be ready, and to ultimately come away with a set of actionable recommendations that get you prepared to figure out how agents can transform your business. If you are interested in the Agent Readiness and Opportunity audit, reach out directly to me. NLWSUPER AI Put the word agent in the subject line so I know what you're talking about. And let's have you be a leader in the most dynamic part of the AI market. Hey listeners, want to supercharge your business with AI? In our fast paced world, having a solid AI plan can make all the difference. Enabling organizations to create new value, grow and stay ahead of the competition is what it's all about. KPMG is here to help you create an AI strategy that really works. Don't wait. Now's the time to get ahead. Check out real stories from KPMG of how AI is driving success with its clients at KPMG US AI again, that's www.kpmg us./AI. Now back to the show. Welcome back to the AI Daily Brief. Today we are digging into the always juicy topic of model competition. Specifically, Elon Musk's Xai has released their long awaited flagship model Grok 3. In fact, the launch unveiled a family of models built around the Grok 3 architecture. The flagship model competes against OpenAI's GPT4O. There's also a mini version that's designed for speed. The company will also release reasoning versions of the model in each size shortly. Users, for example, will be able to engage something called Big Brain Mode to add more reasoning time for more difficult queries, and XAI also introduced a mode called Deep Search. Deep Search uses a form of rudimentary agent to search the web in Twitter X post to compile long form reports. Obviously now, in a similar way to how Deep research works with OpenAI, there is also a forthcoming Voice mode which will be rolled out in about a week. According to the announcements, Grok 3 is first available to Premium plus subscribers on X, but M1, Astra and Apple. Insider also claims that XAI will launch a Grok Pro tier at $30 a month or $300 per year. It seems like that subscription might be required to use advanced features like the Deep Search Voice mode and Big Brain mode Now, as these new models come online, Elon announced that Grok 2 would be open sourced in the coming months. He said. Our general approach is that we will open source the last version when the next version is fully out, when Grok 3 is mature and stable, which is probably within a few months, then we'll open source Grok 2. Sam Altman has flagged that he's considered doing the same with OpenAI's older models as well, so maybe that becomes the new norm now. One of the reasons that Grok3 has been highly anticipated is that it's the first model that's trained on a larger scale. Data center Last month Elon claimed that the model was trained using 10 times the compute of GROK2, which was achieved of course with the Colossus Supercluster, the first training cluster capable of networking 100,000 Nvidia H1 hundreds. Grok3 was therefore viewed as the first real test of whether pre training scaling had hit a wall with the last generation of models. Now of course, as is the case with every launch, people are pouring over the benchmarks when it comes to math science encoding benchmarks. Grok 3 mini achieved parity with Gemini 2.0 Pro and Deep Seq V3 and the full size GROK model and of course this is according to XAI itself outperformed on each test by a noticeable margin. Important to note, this was only comparing leading non reasoning models with Grok 3, not putting up the same performance as OpenAI's O3 mini on these tests. For the reasoning models, Both sizes of Grock 3 seem fairly competitive with O1 on low inference settings and and outperform O3 mini on high inference settings. This would imply that the reasoning version of Grok 3 isn't on the same level as the full size O3. Given that we don't have access to either model at this stage, we don't know for sure. Xai noted that Grok3 reasoning is still in beta and will have further post training before its full release. There wasn't a huge boost from scaling pre training, but the gains were there. Professor Ethan Malik writes, based on the early stats, looks like Grok 3 base is going to be a very solid frontier model, suggesting pre training scaling law continues with linear improvements to 10x compute. In essence, Grok 3 doesn't invalidate those scaling laws, but it could also suggest that much, much larger training clusters are needed to see paradigm changing improvements. One benchmark that many people took note of was Chatbot arena, where users vote on which AI output they prefer. While the metric is inherently subjective, it gives a sense of how the models will perform in the market. Investor Gavin Baker writes Grok3 is the first model ever to score over 1400 on Chatbot arena and outperforms the best publicly available reasoning models from OpenAI and GOOG. XAI was founded 13 years after DeepMind and eight years after OpenAI and is now ahead of both the SR71 Blackbird of AI Labs. Baker did of course then note that he is a little biased as an XAI investor AI Breakfast wrote for everyday users, the chatbot arena is the only benchmark that matters. Grok3 is officially the best LLM given the speed at which XAI achieved this, they will only widen the gap over time. A more complete review comes from Andrej Karpathy, and although Karpathy was a co founder at OpenAI, Most people view his take as inherently unbiased given his lack of affiliation today and the general credibility that he has. He wrote a long review on X saying, I was given early access to Grok 3 earlier today, making me, I think, one of the first few who could run a quick vibe check. He goes into a long review, sharing some of his tests around thinking, exploring the deep search feature, trying a bunch of random LLM gotchas, and ultimately here's the conclusion he came to he writes Grok3 + thinking feels somewhere around the state of the art territory of OpenAI's strongest models 01 Pro. At $200 a month and slightly better than deep seq R1 and Gemini 2.0 flash thinking, which is quite incredible considering that the team started from scratch around a year ago. This time scale to state of the art territory is unprecedented. Do also keep in mind the caveats. The models are stochastic and may give slightly different answers each time, and it is very early, so we'll have to wait for a lot more evaluations over a period of the next few days to weeks. The early Elle Marina results look quite encouraging indeed. For now, big congrats to the XAI team. They clearly have huge velocity and momentum now. The larger Context around the Grok 3 launch is the ongoing feud between Elon and Sam Altman and indeed it's very difficult to cover this. Elon especially is more divisive than he's ever been, and it is enormously difficult to find people who can separate whatever they think about Elon in general from their reviews of anything that he touches. Here's how Gary Marcus summed up where this leaves the competition, which I think is fairly reflective of what others think as well. He writes, 1. Sam Altman can breathe easy for now 2. No game changers, no major leap forward here, hallucinations haven't been magically solved, etc. 3. That said, OpenAI's moat keeps diminishing, so price wars will continue and profits will continue to be elusive for everyone except Nvidia. 4. Pure pre training scaling has clearly failed to produce AGI OpenAI leaker Jimmy Apples writes, strong model. The main thing is the speed with which they caught up. I think it lives up to expectations. Strong, offering good dollar value. He then prodded Sam Altman to release 4.5, which we know is coming soon. Earlier in the day when someone had told him to release 4.5 the same day to steal the show, Altman wrote, that wouldn't be very nice Ellipsis. To me, one of the things that really stands out is just how absolutely saturated these benchmarks are and how little I find myself compelled by them when a new model comes out. Ethan Molik again got at this writing. Another thing Grok3 highlights is the urgent need for better batteries of tests and independent testing authorities. Public benchmarks are both meh and saturated, leaving a lot of AI testing to be like food reviews based on taste. If AI is critical to work, we need more, he continues. GPQA diamond and MMLU and ARC AGI look nothing like actual work. He also adds, and this is something I completely agree with, I'm surprised no large IT consulting or even National Standards Agency hasn't stepped in with large scale batteries of private tests, especially given the hundreds of billions of dollars being invested. This is a hugely salient point. Ultimately, it doesn't matter for the vast majority of users how they do on these benchmarks. It matters how they perform in real work settings. And speaking of OpenAI and Elon's fight, the OpenAI board has now formally rejected Elon's $97 billion bid to take over the nonprofit. In a unanimous vote, the board decided that the takeover was, quote, not in the best interest of OpenAI's mission. A statement from chairman Brett Taylor said OpenAI is not for sale, and the board has unanimously rejected Mr. Musk's latest attempt to disrupt his competition. Any potential reorganization of OpenAI will strengthen our nonprofit and its mission to ensure AGI benefits all of humanity. OpenAI lawyers have insisted that Musk's bid doesn't set the price for the nonprofit, which will need to be paid during the conversion to a for profit company. Separately, the Financial Times reports that the company is considering granting special voting rights to the nonprofit board in an attempt to ensure that they aren't a target for a hostile takeover from Musk following the for profit conversion. Meanwhile, XAI itself is heading back to the well for another funding round. Bloomberg reports that the company is seeking to raise $10 billion at a $75 billion valuation, with sources saying that existing investors including Sequoia, Andreessen, Horowitz and Valor Equity Partners all participating in the talks, which are still at an early stage. A significant portion of the new funding seems as though it would pay for upgraded chips at xai's data centers. On Friday, Bloomberg reported that the company was close to closing a $5 billion deal with Dell to provide servers powered by Nvidia's Blackwell GB200 chips. So ultimately friends where we are is that the proof is going to be in the pudding. A lot of folks over the next few weeks will be testing Grok 3 and seeing how it compares to the latest ChatGPT and Claude models. But it also feels to me like this is the beginning of model update season, not the end with both Anthropic and OpenAI promising new models coming soon. So we could have a lot of new developments in the near future which is obviously nothing but good for all of us users. For now though that is going to do it for today's AI Daily brief. Until next time, peace.