The Four Wars of the AI Stack - An Update with Swyx and Alessio of Latent Space

Today on the AI Breakdown, part one of my conversation with Alessio and Swix from Latent Space. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube, our Discord and our newsletter. Hello friends.

Rob.

Right now I am doing a bit of travel with the family and so we are doing interviews this week and today and tomorrow I am bringing you two parts of a great conversation with my friends from Latent space, Alessio and Sean, better known as Swix. You guys might remember a show we did together back six months ago or so and this follows a similar format where we are talking about some of the biggest trends shaping the AI space right now. Hello friends, quick note before we get to the rest of the episode. You have probably heard me talk about the AI Education Beta.

Over the past few months we've had.

A ton of you participate, which has been amazing. And now we're almost ready to announce.

Something big and something new.

If you want to be one of the first to hear about our new approach to learning AI that is hyper practical, hands on, immediately relevant, continuously upgrading and anchored by community. Go to BeSuper AI and sign up to be notified when the project goes live. We're getting there in just a few weeks and I want all of you.

Along for the journey.

Once again, that's Besuper AI. In this half of the show we talk about the four wars of the AI stack, what we've learned about the power of being GPU rich and some.

Of the realigning battles around big tech.

I don't want to waste any more time before we dive in, so let's listen.

All right fellas, welcome back to the AI Breakdown. How are you doing?

I'm good, very good.

Yeah.

With the last, the last time we.

Did the show we were like, oh.

Yeah, let's do check ins like monthly.

About all the things that are going on. And then of course six months later and you know, the world has changed in a thousand way. It's just, it's too busy to even, to even think about podcasting sometimes. But I'm super excited to, to be chatting with you again. I think there's, there's a lot to, to catch up on just that's happened I think in the, you know, the beginning of 2024. And so, you know, we're going to talk today about just kind of a.

Broad sense of where things are in.

Some of the key battles in the AI space and then the, you Know, one of the big things that I, that I'm really excited to have you guys on here for us to talk about where sort of what patterns you're seeing and what people are actually trying to build, you know, where, where developers are spending their, their time and energy and, and, and any sort of, you know, trends there. But maybe let's start, I guess by checking in on, on a framework that you guys actually introduced, which I, I've loved and I've cribbed a couple of times now, which is this sort of four wars of the, of the AI stack. Because first, since I have you here, I'd love, I'd love to hear sort of like where that started gelling and then, and then maybe we can get into, I think, a couple of them that are particularly interesting in light of some recent news.

Yeah, so maybe I'll take this one. So the Four wars is a framework that I came up around trying to recap all of 2023. I tried to write sort of monthly recap pieces and I was trying to figure out what makes one piece of news last longer than another or more significant than another. And I think it's basically always around battlegrounds. Wars are fought around limited resources, and I think probably the most limited resource is talent. But the talent expresses itself in a number of areas. And so I kind of focus on those areas first. So the four wars that we cover are the Data wars, the GPU Rich Poor war, the Multimodal war, and the rag and Ops war. And I think you actually did a dedicated episode to that. So thanks for covering that.

Yeah, yeah. Not only did I do a dedicated episode, I actually used that. I can't remember if I told you guys, I did give you big shout outs, but I used it as a framework for a presentation at Intel's big AI event that they hold each year where they have all their, their folks who are working on AI internally. And it totally resonated. So that's amazing.

That's.

Yeah.

So, so what got me thinking about.

It again is specifically this inflection news that we recently had. This sort of, you know, basically, I can't imagine that anyone who's listening would, wouldn't have thought about it. But, you know, inflection is a.

One of the big contenders.

Right. I think probably most folks would have put them, you know, just a half step behind the anthropics and open AIs of the world in terms of labs. But it's a company that raised 1.3 billion last year, less than a year ago. Reid Hoffman's a co founder, Mustafa Sulaiman who is a co founder of DeepMind, you know, so it's like this is not a, a small startup, let's say, at least in terms of perception. And then we get the news that basically most of the team, it appears, is heading over to Microsoft and they're bringing in a new CEO. And you know, I'm interested in kind of your take on how much that reflects the cold aside, I guess, you know, all the other things that it might be about, how much it reflects this sort of the stark, brutal reality of competing in the frontier model space right now and you know, just the access to compute.

There are a lot of things to say. So first of all, there's always somebody who's more GPU rich than you. So inflection is GPU rich by startup standard, I think about 22,000 H1 hundreds. But obviously that pales compared to Microsoft. The other thing is that this is probably good news maybe for the startups is like being GPU reach is not enough. I think they were building something pretty interesting in PI. Their own model, their own kind of experience. But at the end of the day you're the interface that people consume as end users is really similar to a lot of the others. So. And we'll talk about GPT4 and Cloud 3 and all this stuff. Sometimes when you're a startup you can have a lot of success being GPU poor doing something that the GPU rich are not interested in. You know, we just had our AI center of excellence at thattable and one of the AI leads at one of the big companies was like, oh, we just saved $10 million and we use these models to do translation, you know, and that's it. It's not, it's not AGI, it's just translation. So I think like the inflection part is maybe a calling and awaking to a lot of startups that say, hey, you know, trying to get as much capital as possible, try and get as many GPUs as possible, it's good, but at the end of the day it doesn't build a business, you know, and maybe what inflection? I don't, again, I don't know the reasons behind the inflection choice, but if you say I don't want to build my own company that has 1.3 billion and I want to go do it at Microsoft, it's probably not a resources problem, it's more strategic decisions that you're making as a company. So yeah, that was Kind of my take on it.

Yeah. And I guess on my end, two things actually happened yesterday there was a little bit quieter news, but Stability AI had some pretty major departures as well. And you may not be considering it, but Stability is actually also a GPU rich company in the sense that they were the first new startup in this AI wave to brag about how many GPUs that they have and you should join them. And Imadis is definitely a GPU trader in some sense from his hedge fund days. So Robin Rombach and most of the stable diffusion 3 people left stability yesterday as well. So yesterday was kind of like a big news day for the GPU rich companies. Both inflection and Stability having sort of wind taken out of their sails. I think, yes, it's a data point in the favor of just because you have the GPUs doesn't mean you automatically win. And I think I'll echo what Alessio says there. But in general also, I wonder if this is the start of a major consolidation wave. Just in terms of. I think there was a lot of funding last year and the business models have not been worked out very well. Even inflection couldn't do it. And so I think maybe that's the start of a small consolidation wave. I don't think that's a sign of AI Winter. I keep looking for AI Winter coming. I think this is kind of like a brief cold front.

Yeah, it's super interesting.

So I think.

Bunch of stuff here.

One is, I think to both of.

Your points, in some ways there had already been this very clear demarcation between these two sides where the GPU pours, to use the terminology, like just weren't trying to compete on the same level. Right. You know, the vast majority of people who have started something over the last year, year and a half, call it, we're racing in a different direction. They're trying to find some edge somewhere else. They're trying to build something different. If they're, if they're really trying to innovate, it's in different areas.

And so it's really just this very.

Small handful of companies that are in this like very, you know, it's like the Coherers and Jaspers of the world that like this sort of, you know, that are, that are just sort of a little bit less resourced than, you know, than the other set that I think that this potentially even applies to, you know, everyone else had clearly demarcated into these two, two sides. And there's only a small Handful kind of sitting uncomfortably in the middle perhaps. Let's, let's go back to the idea of, of this sort of AI winter or you know, a cold front or anything like that. So this is something that I spent a lot of time kind of thinking about and noticing. And my perception is that the vast majority of the folks who are trying to call for sort of, you know, a trough of disillusionment or you know, a shifting of the phase to that.

Are people who either a just don't.

Like AI for some other reason. There's plenty of that. You know, people who are saying look.

They'Re doing way worse than they ever thought.

You know, there's a lot of sort of confirmation bias kind of thing going.

On or two media that just needs.

A different narrative, right, because they're sort of sick of you know, know, telling the same story. Same thing happened last summer when every, every outlet jumped on the chat GPT at its first down month story to try to really like kind of hammer this idea that, that, that, that the hype was too much. Meanwhile you have you know, just ridiculous levels of investment from, from enterprises you know, coming in. You have you know, huge, huge volumes of, you know, individual behavior change happening.

But I do think that there's nothing.

Incoherent sort of to your point Swix about that and the consolidation period. Like you know, if you look right now for example, there are, I don't know, probably 25 or 30 credible like build your own chatbot platforms that you know, a lot of which have, you know, raised funding. There's just no universe in which all of those are successful across, you know, even with a, even, even with a total addressable market of every enterprise in the world, you know, you're just inevitably going to see some amount of consolidation. Same with you know, image generators. There are, if you look at a 16Zs top 50 consumer AI apps just based on, you know, web traffic or whatever, there's still like I don't know, a half dozen or ten or something.

Like some ridiculous number of like basically.

Things like midjourney or dall e3 and it just seems impossible that we're going to have that many, you know, ultimately as, as, as sort of, you know, going, going concerns. So I, I don't know, I, I, I think that the, there will be inevitable consolidation because you know, just it's, it's also what kind of like venture rounds are supposed to do. You're not, not everyone who gets a seed round is supposed to get to Series A and not everyone who gets to series A is supposed to get to series B. That's sort of the natural process. I think it, it will be tempting for a lot of people to try to infer from that something about AI not being as sort of big or as, as sort of relevant as, as it was hyped up to be. But I, I kind of think that's the wrong conclusion to come to.

I would say the experimentation surface is a little smaller for image generation. So if you go back maybe six, nine months, most people will tell you why would you build a coding assistant When Copilot and GitHub are just going to win everything? Because they have the data and they have all the stuff. If you fast forward today, a lot of people use cursor. Everybody was excited about the Devin release on Twitter. There are a lot of different ways of attacking the market that are not completion of code, the ID and even cursors. They evolve beyond single line, to chat, to do multi line edits and all that stuff. Image generation I would say yeah, just as, from what I've seen, maybe the product innovation has slowed down at the UX level and people are improving the models. So the race is like how do I make better images? It's not like how do I make the user interact with the generation process better? And that gets tough, you know, it's hard to like really differentiate yourselves. So yeah, that's, that's kind of how I look at it. And when we think about multimodality, maybe the people. Why the reason why people got so excited about Sora is like, oh, this is like a completely, it's not a better image model. This is like a completely different thing. You know and I think the, the creative mind is always looking for something that impacts the viewer in a different way. They really want something different versus the developer mind is like, oh, I have this very annoying thing I want better. I have this very specific use cases that I want to go after. So it's just different. And that's why you see a lot more companies in image generation. But I agree with you that if you fast forward their, there's not going to be 10 of them. It's probably going to be one or two.

Yeah, I mean to me that's why I call it a war. Individually, all these companies can make a story that kind of makes sense, but collectively they cannot all be true. Therefore there is some kind of fight over limited resources here.

Yeah, so it's interesting. We wandered very naturally into sort of another one of these wars which is the Multimodality kind of idea, which is basically a question of whether it's going to be these sort of big everything models that end up winning or whether, you know, you're going to have really specific things, you know, like something, you know, Dall E3 inside of sort of OpenAI's larger models versus, you know, a mid journey or something like that. And at first, you know, I was kind of thinking like for most of the last, call it six months or whatever, it feels pretty definitively both. And in some ways, you know, in that you're seeing just like great innovation on sort of the everything models, but you're also seeing lots and lots happen at sort of the level of kind of individual use cases. But then SORA comes along and just like obliterates what I think anyone thought, you know, where we were when it comes to video generation. So how are you guys thinking about this particular battle or war at the moment?

Yeah, this was definitely a both and story. And Sora tipped things one way for me in terms of scale being all you need and the benefit, I think, of having multiple models being developed under one roof. I think a lot of people aren't aware that SORA was developed in a similar fashion to Dall E3. And Dall E3 had a very interesting paper out where they talked about how they sort of bootstrapped their synthetic data based on GPT4 Vision and GPT4 and it was just all really interesting. If you work on one modality, it enables you to work on other modalities and all that is more beneficial if it's all in the same house. Whereas the individual startups who sort of carve out a single modality and work on that definitely won't have the state of the art stuff helping them out on synthetic data. So I do think the balance is tilted a little bit towards the God model companies, which is challenging for the sort of dedicated modality companies, but everyone's carving out different niches. We just interviewed Suno AI, the sort of music model company, and I don't see OpenAI pursuing music anytime soon.

Yeah, Suno's been phenomenal to play with. Suno has done that rare thing, which I think a number of different AI product categories have done, where people who don't consider themselves particularly interested in doing the thing that the AI enables and find themselves doing a lot more of that thing.

Right.

Like it'd be one thing if just musicians were excited about Suno and using it, but what you're seeing is tons of people who just like Music all of a sudden, like playing around with it and finding themselves kind of down that rabbit hole, which I think is kind of like the highest compliment that you can give one of these startups at the early days of it.

Yeah, I asked them directly in the interview about whether they consider themselves midjourney for music, and he had a more sort of nuanced response there. But I think that probably the business model is going to be very similar because he's focused on the B2C element of that. So, yeah, just to tie back to the question about large multimodality companies versus small dedicated modality companies. Yeah, I highly recommend people to read the Sora blog posts and then read through to the Dall E blog posts because they strongly correlated themselves with the same synthetic data bootstrapping methods as Dall E. And I think once you make those connections, you're like, oh, it is beneficial to have multiple state of the art models in house that all help each other. And that's the one thing that a dedicated modality company cannot do.

Today's podcast is brought to you by Plum. You've probably noticed by now that many of the AI features that are embedded in your favorite products kind of suck. They're cool the first time, but pretty soon you're underwhelmed. That's because truly great AI features require complex pipelines and rigorous testing that most startups simply don't have time or tooling to get right. That's why Plum created a collaborative AI app builder that's purpose built for product teams. Your users deserve better than a glorified GPT wrapper. Blow their minds with plum. Check out useplum.com, that's plum with a B. Send me a note to get early access. So I want to jump.

I want to kind of build off that and move into the sort of like updated GPT4 class landscape. Because that's obviously been another big change over the last couple of months. But for the sake of completeness, is there anything that's worth touching on with sort of the quality quality data or sort of rag ops wars just in terms of anything that's changed, I guess, for you fundamentally in the last couple of months about where those things stand.

So I think we're going to talk about Rag for the Gemini and Clouds discussion later and so maybe briefly discuss the data piece. I think maybe the only new thing was this Reddit deal with Google for like a $60 million deal just ahead of their IPO. Very conveniently turning Reddit into an AI data company also very interestingly, a non exclusive deal, meaning that Reddit can resell that data to someone else. And it probably does become table stakes. A lot of people don't know, but a lot of the web text data set that originally started for GPT 1, 2 and 3 was actually scraped from Reddit, at least the sort of vote scores. And I think that's a very valuable piece of information. So yeah, I think people are figuring out how to pay for data. People are suing each other over data. This war is definitely very much heating up and I don't see it getting any less intense. Next to GPUs, data is going to be the most expensive thing in a model stack company and a lot of people are resorting to synthetic versions of it, which may or may not be kosher based on how far along or how commercially blessed the forms of creating that synthetic data are. I don't know if Alessio, you have any other interactions with data source companies, but that's my two cents.

Yeah, I actually saw Quentin Anthony from Eluderia GTC this week. He's also been working on this. I saw Technium. He's also been working on the data side. I think especially in open source, people are like, okay, if everybody is putting the gates up, so to speak, to the data, we need to make it easier for people that don't have 50 million a year to get access to good data sets. And Jensen at his keynote, he did talk about synthetic data a little bit. So I think that's something that we'll definitely hear more and more of in the enterprise, which never bodes well because then all the people with the data like, oh, the enterprises want to pay now. Let me put a pay here stripe link so that they can give me $50 million. But it worked for Reddit. I think the Stock is up 40% today after opening. So yeah, I don't know if it's all about the Google deal, but it's obviously Reddit as being one of those companies where, hey, you got all this great community, but how are you going to make money? And they try to sell the avatars. I don't know if that it's a great business for them, but the data part sounds as an investor, the data part sounds a lot more interesting than consumer cosmetics.

Yeah, so I think there's more questions around data. I think a lot of people are talking about the interview that Mira Muradi did with the Wall Street Journal where she just basically had no good answer for where they got the data for Sora, I think this is where it's in nobody's interest to be transparent about data. And it's kind of sad for the state of ML and state of AI research, but it is what it is. We have to figure this out as a society just like we did for music and music sharing in sort of the Napster to Spotify transition. And that might take us a decade.

Yeah, I agree. I think, I think that you're right to identify it not just as a sort of technical problem, but as one where society has to have a debate with itself. Because I think that there's, if you sit rationally within it, there's great kind of points on all side. Not to be this sort of, you know, person who sits in the middle constantly. But it's why I think a lot of these legal decisions are going to be really important because, you know, the job of judges is to listen all this stuff and try to come to things and then have other judges disagree and, you know, and have the rest of us all debate at the same time. By the way, as a total aside, I feel like the synthetic data right now is like eggs in the 80s and 90s, like whether they're good for you or bad for you. Like, you know, we get one study that's like synthetic data, you know, there's model collapse and then we have like a hint that llama, you know, two, the most high performant version of it, which was one they didn't release, was trained on synthetic data. So maybe it's good. I just feel like every other week I'm seeing something sort of different about whether it's a good or bad for these models.

Yeah, the branding of this is pretty poor. I would kind of tell people to think about it like cholesterol, there's good cholesterol, bad cholesterol, and you can have good amounts of both. But at this point, it is absolutely without a doubt that most large models from here on out will all be trained at some kind of synthetic data. And that is not a bad thing. There are ways in which you can do it poorly, whether it's commercial in terms of commercial sourcing or in terms of the model performance. But it's without a doubt that good synthetic data is going to help your model. And this is just a question of where to obtain it and what kinds of synthetic data are valuable. Even alpha geometry was a really good example from earlier this year.

If you're using the cholesterol analogy, then my egg thing can't be that far off. Exactly. Let's talk about the sort of the state of the ART and the GPT4 class landscape and how that's changed because obviously, you know, sort of the, the two big things or a couple of the big things that have happened since we last talked were one, you know, Gemini first announcing that a model was coming and then finally it arriving and then very soon after a sort of a different model arriving from Gemini and. And Claude three.

So I guess, you know, I'm not.

Sure exactly where the right place to start with this conversation is, but, you know, maybe very broadly speaking, which of these do you think have made a bigger impact?

Probably the one you can use, right? So Club well, I'm sure Gemini is going to be great once they let me in, but so far I haven't been able to. So I have this small podcaster thing that I built for our podcast which does chapters creation like named entity recognition, summarization and all of that. Cloudtree is better than GPT4. Claude 2 was unusable, so I used GPT4 for everything. And then when Opus came out, I tried them again side by side and I posted it on Twitter as well. Claude is very good. You know, it's much better. It seems to me it's much better than GPT4 at doing writing that is more, you know, I don't know, it just got good vibes, you know, like the GPT4 text, you can tell it's like GPT4, you know, it's like it always uses certain types of words and phrases. And you know, maybe it's just me because I've now done it for 50 podcast episodes. So I've read like 75, 80 generations of these things next to each other. But clutter is really good. I know everybody is freaking out on Twitter about it. My only experience of this is much better has been on the podcast use case. But I know that you know, Quran from, from this research is a very big opus pro opus person. So I think that's also it's great to have people that actually care about other models. I think so far to a lot of people, maybe Anthropic has been the sibling in the corner. It's like cloud releases a new model and then OpenAI releases Sora and there are all these different things. But yeah, the new models are good.

It's interesting. My perception is definitely that just, just observationally, Claude 3 is certainly the first thing that I've seen where lots of people there, no one's debating evals or anything like that. They're talking about the specific use cases that they have that they used to use ChatGPT for every day, you know, day in, day out, that they've now just switched over. And that has, I think, shifted.

A.

Lot of the sort of like, vibe and sentiment in the space too. And I don't necessarily think that it's sort of a, A like full, you know, sort of full knock. Let's put it this way. I think it's less bad for OpenAI than it is good for Anthropic. I think that because GPT5 isn't there, people are not quite willing to sort of like, you know, get overly critical of OpenAI, except insofar as they're wondering where GPT5 is.

But I do think that it makes.

Anthropic look way more credible as a, as a, as a player, as a, you know, as a credible sort of player, you know, as opposed to where they were.

Yeah, and I would say the benchmarks veil is probably getting lifted this year. I think last year people were like, okay, this is better than this on this benchmark, blah, blah, blah, because maybe they did not have a lot of use cases that they did frequently. So it's hard to like compare yourself. So you defer to the benchmarks. I think now as we go into 2024, a lot of people have started to use these models from very sophisticated things that they run in production to some utility that they have on their own. Now they can just run them side by side. And it's like, hey, I don't care that the MMLU score of Opus is slightly lower than GPT4. It just works for me. And I think that's the same way that traditional software as being used by people. Right. You just strive for yourself and which one works best for you. Nobody looks at benchmarks outside of sales white papers. And I think it's great that we're going more in that direction. We have an episode with ADAPT coming out this week and some of their model releases, they specifically say we do not care about benchmarks. So we didn't put them in because we, we don't want to look good on them. We just want the product to work. And I think more and more people will go that way.

Yeah, I would say it does take the win out of the sales for GPT5, which I know we're curious about later on. I think anytime you put out a new state of the art model, you have to break through in some way. And what Claude and Gemini have done is effectively take away any advantage to saying that you have a million token context window. Now everyone's just going to be like, oh, okay, you just matched the other two guys. So that puts an insane amount of pressure on what GPT5 is going to be because it's just going to have the only option it has now, because all the other models are multimodal, all the other models are long context, all the other models have perfect recall. GPT5 has to match everything and do more to not be a flop.

Sam.