Industry Roundup #5: AI Agents Hype vs. Reality, Meta’s $15B stake in Scale AI, and the NBA's AI Ad

Foreign.

Welcome to the show.

Hi, Richie. Thanks for having us.

I mean, I should say welcome back, really. Obviously you've been a guest several times before for our annual Data and AI Trends episodes. But here's your, your first time hosting. So how does it feel to be in the hot seat?

Very privileged. I must say, I feel a little bit like you. It's a, it's a way more comfortable position and needing to want to be given predictions. It's always like, I'm scared for a whole year to see, like, how they play out.

Absolutely. So, yeah, maybe we got a chance to review some of the things you were talking about in January as well today.

Yes, well, maybe I'm not sure how well I'm doing.

All right, great. So for our first story today, we're going be talking about a report from McKinsey's Quantum Black outfit. So that's the AI arm of McKinsey. So they got a report called Seizing the Agentic AI Advantage. Very cool title for how do you actually make use of AI agents or agentic AI in business? And they make a case that there's been a lot of investment in generative AI, but not many returns so far. And the big idea is that most of the use cases at the moment has been very much around what they call horizontal use cases. So these are things that are suitable for a lot of the business or wide swath of your department. So these are things like chatbots, copilots, and they've scaled very well, but they've only given like a small amount of value to each department. And what McKinsey suggests is that what you really want are vertical use cases. This is something that's going to be a large amount of benefit for just one department or just one team or maybe just a few individuals in the organization. So I think they're talking things about like things specifically for hr. So you've got like HR tooling for like better hiring or you've got things for like legal for contract review, all.

This kind of stuff.

But those have not been widely adopted and saying lots of organizational challenges like a lack of skills, the technology is maybe not quite ready yet. And just the challenge of process re engineering is very time consuming. And then suggest maybe AI agents are going to solve all these problems because they can automate stuff and it's going to be brilliant. Now, I'm not sure whether this is just magical thinking. It's like it's the cool new toy that McKinsey want to talk about or whether this is going to be A real thing. So Martin, do you have an opinion on this? Are AI agents going to save us all and make loads of money for businesses?

Well, first of all, I think some businesses are making lots of money out of this device. We're selling the AI tools. So given all the use cases you mentioned, I think there's like an AI startup scale up a little bit for everything. And I recently read some data about like it's my of the fastest growing area of startups in terms of revenue that they bring in. It's also a little bit of a different work, I think world, I think we came out of a world where lots of money was spent on building products and not that much focus on the business side. It seems that people have learned their lessons and actually quite a lot of these AI startups are making a decent amount of money. So the AI cash is going somewhere, like somewhere is sneaky, is money. But I think what you really mean is like, okay, the ones who are buying all these toys are, they are, they see the gains. So I think it's a little bit underwhelming at the moment, but I think it's part of just of the hype curve. I think the benefit will still come. Now recently had with some colleagues some discussions around this and then they're like different takes as well. So, so one of the things that one of my colleagues raised was the, the, the solo paradox or the productivity paradox, like I needed to look it up as well. But basically what it pointed out was that, well, there's actually slow, there was like a slowdown in productivity growth in the US between the 70s and the 80s despite like, okay, like big promises because information technology computers were coming out and that was going to like boost productivity like never seen before. And so despite lots of companies spending a lot of money on computers and technology and expecting much more and better results and people getting faster at doing their jobs, there's like, it is argument that like, okay, actually it's not what we see. If you look at productivity growth, it was a little bit of a long intro to like, okay, well I, I think the, the, the productivity growth will come from AI and that it will start solving actual problems rather than like toy problems. But at the same time like having that productivity paradox in mind, like, okay, am I 100% confident? Like, no, not like that. That's also like not, not the case. But I, that it goes slower than people anticipate, I think is a little bit normal, I would say. Yeah.

So it does seem fairly intuitive that you have to Invest in stuff first and then you reap all the rewards. You get the money, the increased productivity growth, the extra revenue generation, whatever, like, later on. I guess it's a question of how patient you want to be about, like, waiting for that extra cash to come in.

Yes. And like, and also, like, not every investment, like, turns out to be the way that you hope it to be in. So I think there is also still a lot of, like, uncertainty around, like, okay, what are the investments that make sense? What are the investments that don't make sense? Like, AI seems to have product market fit in the marketing field. It seems to have product market fit in the, in the software engineering field. If you think about cursor and like, with. Sorry, cursor and how it's making, like, engineers more productive, I'm more active on the marketing sales side. If I see, like, okay, how much more I can do, like, so there's like, true product market fit. Maybe a little bit on the legal side, but then, okay on the finance side. You see, at least I haven't seen that much yet. So I think we're also trying to figure out, like, okay, where, where. Where does the most value sit? Where does the lowest angle value set? And in that process, well, investments get lost while you try to figure that out.

Absolutely. Yeah. I mean, I think that's fairly natural. Like, not every investment's gonna be a winner, but you need to figure out, like, what's. What the sort of the good bets are gonna be. Now, there's one more thing I want to talk about in this report. So it's perhaps the most controversial bit. So the report is suggesting that the best use cases for agents are going to be just automating business processes. This is a great way of saving money, is you take your existing business processes and then use a bit of software, a bit of AI, to just automate away the bits so humans don't have to do them. Now, a lot of this is really, the AI is not that new or not that involved in it. It's just taking the sort of standard approach that's been around for sort of at least the last decade of just using software to automate business processes. And then maybe there's an LLM in there somewhere. But so there was some pushback from this. So it was Ethan Mollick, so it was a renowned AI professor. He was saying that, well, this is really missing a trick because a lot of the latest models that come out this year, they're really good at reasoning. You can do things that are a Lot more complex than just summarizing bits of text or whatever. So there's a kind of, there's a disagreement about what the best use of AI agents are. Do you want really simple agents or do you want something fancier that's more akin to, like a junior employee? I don't know whether you have an opinion on this.

So in the, in the, in the short term, I agree with the McKinsey report. In the long term, I agree with the Eaton Mollick rebuttal, but for a different reason. So, so I think where McKinsey is right in its report is in the fact that if you do this type of change in a company, and I can only imagine, like, the larger the company, the more difficult it is. Like, there's a certain, like, inertia. And because of that, like, okay, you can come in and you can say, like, hey, we're going to tackle, like, the biggest, most complex process or product that we have and we're going to make it full AI. And probably along the way, like, you lose everyone and your whole investment trying to do, like, falls flat. So from that perspective, like, starting simple, which is like my distillation of like, okay, what they're saying, it makes sense, like, in a pro, in a world where you move more and more towards embedding AI in your organization. So, so that's what I mean, with a different reason. So, so my argument was, okay, why I agree with the McKinsey paper is more from like, okay, organizational change perspective rather than the arguments that, that they make. Because I think in the arguments that they make and the way that Ethan Molik tries to debunk them, like, I'm actually on the side of, of Ethan Mollock, where, okay, he points out, like, the reference models that are in the McKinsey report, they're outdated. They make certain arguments around costs. But, like, what we've been seeing the last three years is that, like, cost has been declining rapidly. They're like, hyper focused. McKinsey's hyper focused on efficiency. But, like, they're then focused on efficiency from, like, yesterday's technology. And like, when I read that, I was thinking about, like, there's this, there's this famous quote by Peter Drucker, like, he's his management consultancy, I think maybe even like one of the founding members of management consultancy, like, if there's a founding member team. But it's like, okay, there's nothing so useless as doing, like, things efficiently that should not be done at all. And like, that part reminded me a little Bit about it. Like, why are we caring about the efficiency of the models that are already obsolete? It's irrelevant. So I think short term, the way that McKinsey is positioning it and approaching it, I think it makes sense, but not necessarily for the reasons that they say it makes sense. And there are more mutinic sides. Like, does that resonate with you or do you feel like. No, it's totally off, Martin. It does.

No. So I like the Peter Drucker quote. And actually I was speaking to one of our customers the other week and so this is someone at a bank and they were saying they were really excited. They were like, we've been building all these agents to automate our fax workflows. And I'm like, I'm kind of conflicted by this because it's like, why he's still using faxes. Like, it's not been a sensible technology at any point in the century. And so it's obviously completely crazy. Like it's just a technology fix over a really stupid process. But obviously this is a smart person. If they had any power to not use faxes, they, they would have done that decades ago.

So they need. Going back to my productivity paradox. Like that's a little bit of the little bit of point. Like they just, they skipped out on the Internet and it still worked out fine for them because I assume they're still a big bank. Like a bank, it's hard to run. So like they have something figured out. So yeah, it goes back to that, that productivity paradox. And like, okay, like what, what is what increases? Will we actually see if any.

Yeah, absolutely. So, yeah, I'm going to be like.

After this, you're going to call me an AI doomer, which I'm not for the sake of the argument. Taking the opposite side here.

Yeah. What is all this stupid AI? Why are we bothering mine? I'll get any productivity out of it. No. So I think this argument, it comes back to strategists versus operationalists. So if you are doing something where it's research, you're not quite sure what the answer is going to be. That's where you need like strong reasoning. You need like very flexible processes. And that's where these higher reasoning models come in and where you really want the fancy agents to help you out. If you are focused on operations, you want really like more rigid processes that are streamlined and do the same thing over and over again at scale. And that's where you care about just having simple automated processes that are repeatable. So yeah, I think it just depends exactly, like, what you're doing. All right, so let's move on to the next story. So before Martin, you mentioned about like, the people making money out of this seem to be like the, the AI startups and scale ups and there's been some big purchases going on in this area. Do you want to tell us about them?

Yeah, sure. So one of the, one of the things that obviously was like big news in the, in the last week was meta slash Facebook for those who are still with the old name. Or like Mark Zuckerberg buying like scale AI. Well, sorry, he didn't buy scale AI. He took a stake of, I believe 49% in scale AI and he, he paid $15 billion for that, which is by any measure a lot of money. Like, okay, for the listeners who don't really know like, what Scale AI does. So basically, basically they solve the data labeling problem. So in short, if you want to have really powerful AI, you're going to need enormous amounts of high quality labeled data. And creating high quality labeled data is something that's incredibly slow, it's very expensive. And what Scale AI did is basically build a factory that produces this type of like high quality labeled data. So if you're like Tesla, you sent like thousands of dollars of video to Scale AI, they go over it frame by frame and they say like, hey, this is a pedestrian and this is a tree and this is like a traffic light and so on and so on. Or like, if you're like OpenAI, you have, you are in need of like real people writing high quality questions and answers. You have a need for people that need to rate AI's responses, responses telling like, okay, is this a good answer? Is this a bad answer? And like, all of that is basically done by scale AI, which makes them enormously valuable because again, like you need this type of like high quality labeled data. And then the question is like, okay, why, why is metas Mark Zuckerberg doing this? Lots of theories out there. One is, well, okay, he feels that the llama models are behind and if you look at the benchmarks, like, that seems to be the case and he feels the need to like, do a catch up. And this type of like, move allows him to do that because on the one hand, well, he kind of becomes exclusive owner of this high quality labeled data. Both Google and I believe, OpenAI, who were former customers of Scale AI already sat like, okay, like, with canceling our contract. So, okay, he now owns this probably very valuable data set and process himself on top. He probably has Now a little bit of insight in how Google and OpenAI were training their models, so that's beneficial. Secondly, there's the CEO of Scale AI is apparently a very talented guy. The what he's built seems to be a good validation that he's a, he's a pretty, pretty, pretty smart cookie and he's gonna lead up Meta's new superintelligence division. So the superintelligent division of Meta is basically, okay, Mark Zuckerberg saying like, okay, we are falling behind. We need a new initiative. The next frontier is super intelligence. So going to put this super talented guy at the top of it. I'm paying $15 billion for being able to do that. And so now I have that. So, okay, that's another theory. Like he needed talent and it's some very expensive AC hire. The first theory was like, okay, yes, he needs high quality data. He has that. Now another theory is like, well, maybe it's a competitive mood move. Like somebody described it as, like, okay, he takes a chess piece of the board move. Like now that OpenAI and Google don't no longer have access to it. Like, okay, is that going to hurt their own model training? And then the final one that. And I'll give it to you, Richie, for some comments. The final one I found interesting was, well, somebody just said, well, it's actually like in Facebook terms, like, it's not a lot of money. It's like 1% of their like market value or something like that. So like, it's a blip for them. So from a risk perspective, it's a, it's a fairly low risk bet to get back into the game. So, yeah, lots of interesting things happening there. Like lots of gossip, intrigues, like, okay, who's going to win this? Will this put them back into the game? But definitely, like a new wordy story in AI World block last week.

Yeah. So it's a good bit of sort of corporate gossip. Absolutely. Like the idea of just investing $15 billion in it being not very much. Very, very foreign to me. But yeah, it's, it's amazing the amount of money involved in the, in the AI game at the moment, actually. So I recently watched the movie Mountainhead. It's kind of like the Succession TV show, but set in Silicon Valley. So I learned a phrase. So to invest a billion dollars, they used the phrase to, to bust a B nut, which is, it sounds kind of rude, but the idea of, yeah, mark Zuckerberg busting 15B nuts on, on scale AI is very interesting. All right, anyway, so yeah, you mentioned Scale AI. We actually had guest from the Scale AI's competitor, Sama AI. So it was Wendy Gonzalez, the CEO, and Duncan de Curtis, one of the senior vice presidents, just talking about this. It's absolutely fascinating the idea that, uh, yeah, data is still perhaps the most important thing in order to get great quality AI. So we talk about AI being the big thing. Mustn't forget about the data because that's what's powering everything. And just like, in terms of trying to get better quality AI, I mean, you basically got a choice of four things. You've got better data that we mentioned, or you can invent new models. But that's slow and it takes time because you've got to research. And then you can either add more compute, power, joint training, but like the latest foundation models, they're costing like upwards of $10 million or like $100 million to create. So we're kind of reaching the limits of like how much you can spend to create new models. And then you can also add like inference, compute, but that makes your models more expensive to run and it makes them slower as well. So then, yeah, that's a terrible user experience. So it looks like better data really is like the, the big solution for getting better AI at the moment.

Yeah, better data and just a hypothesis. But like probably the talent that you have within the company plays a big role as well. And it always plays a big role. Like high quality people always make a difference. There was like this other interview with Sam Altman where he basically said, like, okay, Facebook has been trying to poach some of our most talented people and they're offering them $100 million sign bonus. Which is when you hear the story is like, well, that sounds pretty crazy. But then, well, one of the narratives in this whole acquisition story we just went over is, well, he wanted to have a really good head of his superintelligence division. He identified Alexander Wang, the founder of Scale AI, as being that person. And he wrote a check of 15 billion to own not even half of the company. So like, it's the data, but then apparently like, okay, the, the, the, the people and probably the, the novelty of the ideas that they come up with, like, please. Seems to be playing a big role as well. And maybe Mark is identifying one of the issues. Like, I'd say Mark, I'm definitely not on the first name basis with him. It's maybe another thing is like, maybe one of the key things he, he sees as why Llama is behind is because he does not have a talented enough team at the moment and probably already has a very talented team. Maybe the difference is, is, is, is hard to grasp for us worlds on this topic.

Yeah, maybe. I mean of course because like back in the day, Mark Zuckerberg, he became like the youngest ever like self made billionaire. I think he was like 23 and then it was Alexander Wang from scale who broke that record. He became a billionaire when he was 21. So maybe that's, that's part of the reason he wanted to hire him. Yeah, something like that. All right, so I think you had another story around hiring or around buying things.

Yeah, so another rumor that's going around is that Apple is considering or, well, it's a rumor that they would like to purchase Perplexity and like the number that's been circulating is $20 billion. I'm not sure like okay, to, to what degree that's true, but I think it speaks to the story of the fact that well, Apple seems to be in a bit of a tough spot with its AI plans. So I haven't met a lot of people that are happy with Apple intelligence or the way that Siri is using AI. I think they made quite a lot of promises, none of them fulfilled. I think it's for the first, first time in 10 years. Like so I'm on a Google device, I use Android. So for 10 years I needed to listen about like, okay, why having an iPhone is so much better and why am I not switching to an iPhone and so on and so on. And like now I actually have the OR functionality. They're all locked into their little Apple ecosystem, so maybe I eventually will have the last laugh. But so yeah, but the point being like, okay, so Apple seems to be in a little bit of a, at the moment in the losing spot on everything that has to do with AI. And I think rumors like this are like, I don't know where the origin is. It's like, okay, the question is, are they trying to fix it or not? Do they try to fix it internally? But then internally they have lots of people switching job positions, switching roles, switching functions and so on. So doesn't really look like a stable environment. That said, if you look back at OpenAI in the last 18 months, you could probably make the same argument there. And they kept shipping models. Um, but like from the outside, like it looks all a little bit weird and like carding, I need to buy it in. And then something like Perplexity came around. Now yeah, I think I use Proprexity once in a while, but it's obviously also not the biggest model out there. And Apple has a lot of cash and they have a lot of stock value. So you could even see them making like an even bigger acquisition if they really want to. But it seems that consensus is like, hey, they're going to need to do something because everyone isn't happy with that part of their product. So are they going to buy it in or are they going to develop it themselves? Well, and then maybe a final thing on this is there's like another angle to look at it and is that Google is paying them really a lot of money to have Google be the default search engine on your iPhone. I believe it's also like in these magnitude of $20 billion a year, if you start to integrate AI more and more and you make that a central piece, well, you're losing on $20 million in. It's probably basically pure profit. Yeah, lots of, lots of hard and difficult problems. I think they're definitely like in the corner where, yeah, they're getting hit from all different sites. So it's going to be interesting to see how they get out of this. And if it will be with a, with a big acquisition, there's actually. And then I'll stop and let you react. Richie. There's actually like this. I think the biggest acquisition Apple ever made was like Beats, like, you know, like the Dr. Dre headphones, which is like from a totally different magnitude. I believe it's like a billion something than what's been rumored today. So, yeah, that's like a little bit like. So Apple or Backstreet? Richie, what do you think? Will they. Will they buy or will they build or.

Yeah.

Will they burn and go away?

Yeah, I think Apple disappearing completely, that's more of an outside chance. But certainly they really need to do something. And it's interesting. Is it like Beats is the biggest thing Apple's ever bought? Like, that's crazy because that's a tiny company compared to Apple. When you compare it to something like Salesforce where you know, they bought Slack, they bought Tableau, and it's like constantly acquiring things and then trying to fit them into the business. So, yeah, if Apple does buy something that big, as big as Perplexity or even bigger, then yeah, it doesn't have the reflexes to kind of fold new businesses into itself. So that's going to be a very interesting integration challenge.

Yeah. And I think to the point of scale, AI and the point of talent, maybe that's what Apple needs. Like, okay, they need some true AI talent insights of Apple and okay, what is it that they can buy to bring that talent in scale? AI is of the market now. Like what else is out there to bring that in? I think Microsoft at some point bought one of the companies there could bring in some AI talent. So yeah, let's see what they do.

Yeah, absolutely.

I mean.

Oh, so on that note, there was the Google bought Character AI for it was a few billion dollars and that was basically just to hire back one of their executives had left the company previously.

Maybe that was, was the one that. Yeah, yeah, that's true. Because Character D was founded by one of the people who wrote the original Attention is all you need paper if I remember correctly.

Yeah, all these like hirings for a hundred million dollars. I'm. I'm in the wrong game.

Well, you should write a paper.

All right, cool. Let's move on to the next story. I've got another report for you. This is the EU Generative AI Outlook report. So it's a report to sort of inspire policymakers in the EU just on like what should they do around generative AI? So the big theme is that generative AI ought to be considered a strategic asset for European countries just in order to generate economic growth and increase employment, increase productivity, things like that. So there's a big worry in the report that the EU is lagging behind in compute capacity. So saying the total compute capacity of all these sort of GPU clusters, high performance computing clusters across Europe is less than the size of Xai's Colossus compute cluster. So the Colossus has got, it's like 200,000 Nvidia GPUs in it. So it's a pretty amazing technological feat based in Memphis. But yeah, that's more than the whole EU combined. So the EU is lagging in AI infrastructure. But then beyond that there is the focus of the report is that generative AI should be focused around small and medium enterprises. The focus is on small to medium enterprises because this constitutes 99% of all businesses within the EU. Presumably that's amateur whether it's by headcount or like total number of businesses. Certainly not by, by revenue. But yeah, think every business basically should have some sort of generative AI angle regardless of its size. So there were, there were several sort of perceived challenges there. So one was the infrastructure thing I mentioned, one was just going about process re engineering. The third one was the AI skills gaps. They spent a lot of time talking about well, how do you make sure that people have the right skills? And so the two things that everyone needs to have. So basic AI literacy skills is first of all, just how do you go about using some of these AI tools? But also, do you understand what the implications of using AI are? So basically, what is possible with AI? What is not possible with AI? What should you do with it? Like, what are some sensible use cases? And the other thing on this was that digital skills, including AI skills, and that recognizes basic skills. So this is like a legal term which puts them alongside reading, mathematics and science. So these are things that basically everyone, they think everyone should be taught from school age onwards. This is a big implications just for childhood educational curriculums right through to adult learning. All right, so first of all, Martin, do you think this all sounds like a sensible thing, making AI skills, putting them on the same level as like reading, maths and science?

Yeah. Well, let me first say, because. You definitely triggered me with your opening sentence around like inspiring policymakers to write policies or something along those lines. First of all, I don't think that we should inspire policymakers to write AI rules. For those who watched our talk at the start of the year around our predictions and eu, that I'm not a super big fan of all of this. So this whole push towards like regulation writing policies around this is like, it, it feels a little, it feels, it feels. Well, it feels very European. It feels very like, okay, what we're good at is we're good at writing rules and making policies and doing laws. It reminds me a little bit of that saying, okay, if you want to get started, you need to quit talking and begin doing. I think there's lots of talking, lots of writing, very little doing. Am I a sign of all this? We gonna let policy make us write good policies about AI? Like, not really. Do I, do I agree with what's been said on. Like, okay, yes, we AI probably, like, AI skills is probably going to be like a basic skill like reading and math and science. Like, yes, but does that, like, I think that can, like, if you let, if you let the people free, if you let the citizens of the EU go out themselves, I think I tend to believe that they'll figure this out themselves and like find their own path and then go at their own pace and they'll be supported from the private markets or will be pushed from the private markets, like toward their jobs to do this. Maybe a very quick data point on that probably. I would take the bet that for the past three years, if you would aggregate the data, that the biggest users of Chat GPT were students. So if you think about like, okay, students, students are already adopting this skill. Like, they don't need a policy for, for that. Like, they're already like, they're, they're ahead of the people that write the policy. I would make, I would guess so. So this need for like writing all rules about it. No, I think the market is fixing it themselves. Like our, our children are taking the necessary steps to, to do this. So like, is it a common sense recommendation? Like, yes. Does it need to have everything around it? Like, I don't think so. I think let the people do what they seem best and it seems that if you let them do that then, well, med students as an example. Like, that's already happening.

Okay. Yeah. I want to argue. So yeah, I agree that the market is going to solve many problems in a commercial sense, but I also do think it's important for policymakers have that sort of AI literacy themselves. So having an understanding of like what the point of generative AI is in order that if they do write policies, they're not going to do something stupid. I also think that in non commercial cases it's also like very useful to have some policy there. So for example, if you're designing a school curriculum, you want to know like, what good guidance is.

Well, the best way to avoid doing something soon is stupid. Is not doing it so. Or not doing anything. Like, yes. Should schools think about, okay, how to incorporate AI in their curriculum and make it part of a basic skill? 100%. But I think that's, I think my example of private market was not so. Was not that well chosen because I think, I equally think that like, if you have a motivated educator teacher that they will bring this as well. I'm actually fairly sure that in many classrooms today within Europe, without any policies they're handling, they're playing with AI in a very creative way and thinking about like, okay, how can we, how can we use this? Yeah, maybe I'm a bit too much wrong. Ron Swanson from Life Products and Recreation on this, on this topic. Yeah, I don't like rules.

Yeah, I think it's just different levels of comfort with the bureaucratic engine where they. Okay, all right. So yeah, there's different ways of making it work. All right, let's move on to the next story. You had something about an AI advert.

Yeah. So I don't really follow basketball, but like the, the NBI NBA finals took, took place and there was actually an ad that was fully created by AI and so that was not the first one because like Coca Cola, like did it in the past. And again, like, we referenced that in our. In our startup year podcast last episode. But the, the crazy thing about this ad, it was, by the way, an ad of Kalshi, which is, as far as I sound, like a predictive batting kind of, kind of company. But the crazy thing about this ad was that the, the. The creator, and it was a single creator creator. So one person, like, wrote a blog post describing his workflow flow. And that's what's crazy because again, so this is an ad that goes in prime time during the NBA Finals. And he basically built it with Google, gemini, I think, VO3. He used like a little script, 10 lines, and then edited the clips like in some standard video editing software. And it was dumb. So it took him two days to generate 30 seconds of content. And if you look at the cost estimate, like, if you would exclude, like, his time, it was $600 to generate the content. So I'm probably sure, like, our listeners have heard about the crazy amounts that are paid for advertising during these top spots at the super bowl, at the NBI Finals. And you look at these ads and they just look expenses. If you look at this ad, it actually looks really good. And if you then realize, like, hey, this is created in two days by a single person with a technology that's maybe two, two and a half years, like, old between parentheses, mainstream, that's actually pretty, like, amazing. So, yeah, like, you didn't need to pay a writer. You didn't need, like, any camera persons. Like, no actors needed. Uh, so, yeah, pretty, like, pretty fascinating. Like, yeah. Richie, I don't know. Have you seen the ad?

I have seen the ad, and yeah, I didn't get all the cultural references because again, I'm not a basketball fan from Florida, so it was very odd. Um, I, I like the ad, though. It. It was, it. It was fun. There was some interesting things that I think were caused by technological limitations. So, for example, the.

There's.

It's lots of very, very short sections with quick camera cuts. So it's doing one or two seconds. And this is just because AI can't retain consistency for very long. It also is quirky, and I think that's the only thing you can really get away with at the moment because, like, perfect realism of, like, movements and things like that is not really possible. So having stuff that is weird and fast cutting is kind of the only tone you can go for at the moment.

That's a nice explanation. Like, I was actually, like, my take was like, oh, shit, I'm getting old. Because like this is way too flashy and switches like way too quickly. But I like your take more. Yeah. By the way, it probably makes sense like what you're saying. You clearly see that they need to switch between different scenes very quickly. And it's most likely now I was deliberately paying attention to it like during the first time that I saw the ad and I couldn't spot like any obvious like glitches. Like I, I couldn't see like six fingers or like somebody with three likes or something like that. So at least for me, like maybe if I look at it like a fifth or a sixth time, like I start looking at the glitches. But I don't think that many people look at it like six times in a row. So I did that. It's like, okay, in the first run, I didn't see that much wrong with it. Besides, the gate was very snappy.

Yeah, that's the other thing is if you keep things moving quickly, then it distracts people from the fact that maybe the lighting or the shadows isn't quite right in the background or there's an inconsistency with, I don't know where some background objects are placed. So yeah, keeping things moving quickly seemed to be the secret for things. But I agree like that a lot of the artifacts are disappearing pretty quickly. It is getting much more, much more realistic very fast.

Yeah.

I'm intrigued to see what happens, how this affects like creative industries though, because I mean you mentioned here it was done by one person. So you don't need a cameraman, you don't need actors, you don't need script writers. We had two Hollywood strikes. Was it last year? I can't remember. Yeah, last year. So the screen actor strike and then the Actors Guild striking. Exactly because of this. And now, okay, everything that these people striking, predicted is sort of coming to pass. Real content going out of prime time is being done by AI with no other people needed beyond one person in a prompt. So how do you think this plays out in the real world?

I think in phases and I think the phases are going to follow the length of the content. So going back to the clips point, like, okay, it's deliberate, like little clips basically together. Because like creating a 30 second or a minute ads that's like all in the same, all at the same scene is gonna, it's probably like too, too hard or too glitch prone or anything like that. So I think what we'll see is like, okay, so now we saw like a little ad. Maybe the next thing we See like a more consistent ad. Then we're going to see like a 10 minute standup show by an AI and then we might see like a, a 20 minute sitcom existing out of five episodes. And then like the next thing is maybe like a 70 minute children movie. And I'm like, okay, we're at the two hour like trailer. So I, I think getting from that like 15 second, 10, 15 second clip to like the 1 hour and 30 minutes movie, I, I think that's actually like still a long way to go in art. And I don't think it's because of the creative part. I think people figure out like how to write the prompts, how to work with the camera perspectives. I think it's really going to, I think there is really going to be about the technology without having like any, any insight in this. Like, it's just we've seen massive improvement but like from a blank perspective, it's not that we are, we're still watching 15 seconds and the 15 seconds clips look a lot better, but they're not, they're not like a minute or two minutes long.

Yeah. So I've seen a few sort of videos. So the Door Brothers do quite funny satirical videos and they're kind of a couple of minutes long and that's like, it's amazing. But that's the most I want to concentrate on, on AI generated content at the moment. And part of me wonders like at the moment it's very novel, but is that kind of quirky tone and all the fast cuts, is that going to get tedious quickly, do you think? Or are we going to get past that and have like content that you can concentrate on for a bit longer?

Well, I think it's, I think. Well, I think we're going to need to get past that because I think it's. This is a particular type of content and style that's not going to be like well suited for everyone. So if you want to make this like a really imagine like your Netflix is full of like videos like this ads. Like I wouldn't have a Netflix subscription. It's too, too much, too fast, too loud. My guess is like they will need to fix the setting. Now I'm trying to think like what are good examples? Like maybe. Well, if you look at when Toy Story first came out versus like where we are today. In 3D and computer generated movies, it's also like a very big difference. And that happened. What is it, in the last 20 years or something like that? 20, 25 years. So my guess is we'll get there. But I do think there's still quite some hurdles to, to do. Like this is. There's a story by Seinfeld who's like, okay, talking about like, how do you become a good like stand up comedian? And it's like, okay, doing a five minute bit, like that's how you start like doing like 20 minute. Like it's is, is already difficult during like a one hour. Like it's extremely difficult. And like an hour and 20 minutes, like that's insane. And it's like that's how you, like that's how you separate like the average stand up from the top standup. I think it's gonna, when I talk about the time frames, like that's a little bit the metaphor I have in my, in my mind. Like, yes, you can create a 30 second clip. Like it's, it's hard but doable and it's getting more and more accessible. But now are we going to the 10 minutes and 20 minutes and like when will we be at that? 1 hour 20. And what will it take?

Absolutely. So maybe all these, the screenwriters skilled, the Actors Guild, they've got a bit, a bit more time to, to figure things out before it starts going into like, okay, this is mainstream TV stuff.

Yeah. On the screenwriting, like, I think there's still a pretty good, like somebody needs to write the prompt. So probably good, I would guess.

Yeah, true.

No.

Okay, so actually before we move on, do you have a sense of how much this cost? Like how much?

I have no idea actually, to be honest. Well, we got the benchmark. Yeah. Like as I said, like it seems $600 to generate that video excluding like the cost of the person. But like on the. I don't, I have no idea. On like the token cost. Like specifically, like allegedly this one cost 600.

All right, 600 of clips. So this is like for an individual, it's something you got to think about a bit, but for a business, that's just nothing.

Yeah, indeed.

Okay, all right, cool. So last story, quick one, because we've been talking for a while. So I want to talk about the first benchmark for deep research tools. So these deep research tools are around. You can ask more complicated questions of generative AI. They'll go away and think for 10 minutes and then come back with a report for you. This is, this comes out of research from the University of Science and Technology of China. And they tested so tools from OpenAI, tools from Anthropic, Google, XAI and Perplexity. And so we have a few winners, they split into two categories actually. So the deep research agents are things that are specifically advertised at being able to use tools and are for deep research. So the wins in this category is a Gemini 2.5 Pro, the deep research version of that. And then second place was the OpenAI deep research tool. So those, those are the best ones if you want to try out different deep research tools. The second category, it was just standard issue LLMs with search tool capabilities. So in this case the winner was Claude 3.7 sonnet and second place was perplexity Sonar Reasoning. So most of them had a winner. XAI was the only one which, which didn't have a winner in, in either category. But it seems like in general the tools that are specifically marketed as deep research tools do better at deep. A good thing. Yeah, the Irish are doing something rather than just a bit of branding on the benchmark. The interesting thing was that they took a load of like real world prompts and then they had an LLM to judge whether or not these, these agents were, were working correctly. And to begin with the agents were being too, too nice and they were giving everything a high score, which is stupid because you want to be discriminatory about like what, what's good and what's bad. So in the end they had to get humans who were experts to write reports and then they got the LLMs to compare it to the human report. So there's some human benchmarking involved in this. LLMs can't be trusted sort of by themselves at the moment. The obvious is that whatever the tool is, they're all still really bad at generating citations or at least accurate citations where the content of whatever they're linking to actually says what they, they claim it does. So I think it's a known problem, something to watch out for when you're making use of these tools. Now, Martin, I don't know. Have you, have you used any of the deep research tools? Have you found they work or not?

So I used them a little bit. Like, it's definitely like in the, in, in my AI workflow, it's probably the one that I use the least. I, and I'm not often having that, that type of use case. But let me give you an example. When I, when I did use it, like, okay, I, I use this, I needed to get some information out of like quarterly earnings reports from organizations and actually did a pretty good job there. And what I noticed was like, compared to when I used it in other instances is that because I had like Bad experiences with it also. And it was because the prompt that I provided was not specific, you know, more anecdotal is my own experiences. Like, okay, so what I've learned was that if I give it, like, research me on this earnings report for this particular company and I want to have this type of information and look back in the last like 24 months, like, it gave me actually really good insights when I would say, give me everything of the s and P500 and all the earnings and blah, blah, blah. Like, again, it's too broad. Like, it. I got like some funky behavior. Maybe it's me not writing the right prompts. Maybe it's something on the tool that it's a little bit off putting when you go through that experience. So I actually haven't talked to anyone. And this is all this is on, this is on me. Like, it would actually be good to talk to a couple people who use deep research tools, like on a daily basis and understand how are they using it, what are they, what are they doing? Because I have a good feel of like, okay, I, I use chat cpt, I use Gemini models on a daily basis, but I don't use a deep research. Like, okay, who is the equivalent of me who uses deep research the entire time? And, and yes, you like what, what dating. What they do. So, yeah, yeah.

All right, we have to get Adele back then. I think he's a big fan of the deep research tools, but I agree that I find the workflow incredibly frustrating because the models run for a few minutes. So you have to get the prompt right up front and you write that if you are more specific about what you want, it dramatically increases the chance of you getting things right. But that means that you've then got to do a lot of work up front in thinking, well, what do I actually want? And like, which sites should this thing be looking at in order to pull the information from what do I actually want out of it? And you have to do so much prep upfront to get it right that you don't have to research yourself. So, yeah, in order to get good results, you've really got to be good at that kind of prompt engineering. And you've got to just take the time to think about what you want the answer to be.

Yeah, totally.

Yeah. So the technology's kind of coming together again. It's like, it's the process stuff that it's a little bit frustrating to use at the moment. But yeah, if any of your listeners have been making use of these deep research tools and you found success or you found some challenges, then please do hit us up on social media. Let us know what you found. All right. Wonderful. Well, it's been a pleasure chatting with you, Martin. We have talked for way too long, so let us wrap up here. Have you enjoyed the experience of hosting data frames?

This was nice. Like, it's amazing that you get paid for this. I would do this for free.

Okay. Certainly not taking a salary cut to free.

Okay. No, it was really, really fun. And thanks for having me, Richie. Cheers. Cheers.