Distributed Training, Decentralized AI: Prime Intellect's Master Plan to Make AI Too Cheap to Meter

Execution is cheap. Ideas are worth everything, right? In a world where you can just like, it almost inverts to the current reality. And I think it would just lead to like billions of startups, right? We don't buy like hundreds of billions in compute. So in that sense, like we are not a hotel, we are more like Airbnb or like we are more marketplace sitting on top even of other marketplaces.

Why does this matter? Multiple reasons, right? It's like in the limit, you know, it could create a sort of truly decentralized AI infrastructure that Nobody can control.

We obviously have a lot coming up in terms of improving on that algorithm. I think what we've done so far is just what we've realized, those pseudo gradients we actually send after those hundreds of steps. So it's not the actual gradients of the model. It's the difference between the beginning of the weights and the end state of the weights after all those inner step updates.

In the intelligence age, almost, you want to own a piece of a super intelligent system that is able to generate value where you have actually hard, like, access, like, through, through your ownership in it to the compute, to the intelligence.

Hello and welcome back to The Cognitive Revolution.

Today I'm excited to share my conversation with Vincent Weisser and Johannes Hagemann, founders of Prime Intellect, whose mission is to make intelligence too cheap to meter by building foundational technology to support decentralized, collectively owned AI. Vincent and Johannes stand out for offering a positive vision of a future in which a wide range of AIs empower everyone simultaneously, amplifying each individual's abilities and improving societal resilience, while all actors implicitly check and balance one another's power. At the same time, they've articulated an ambitious master plan and shipped a number of notable milestone projects in pursuit of this goal. Part 1 of their plan is to build an international market for compute, And as of this writing, you can rent an H200 for $1.49 an hour via their website primeintellect.ai. Part 2 is to build software frameworks for distributed training, and in late November they released Intellect-1, proving that distributed training can scale up to at least the 10 billion parameter level. Part 3 is to train high-impact science models, and the MetaGene-1 model, developed in collaboration with the Nucleic Acid Observatory and others, and designed to be useful for pandemic detection, but architecturally incapable of generating new pathogens, is one of the best examples of a defence-favouring AI project that I've seen anywhere. Part 4 is to launch a decentralised protocol for collective ownership of AI models and to collaboratively build towards aligned AGI that benefits all of humanity. While that still remains in front of them to do, given their track record to date, I would not bet against them making a meaningful contribution. We spent much of the first half of this conversation unpacking their vision. To be honest, I'm still not sure how realistic it is to expect that we can maintain a stable societal equilibrium with AI changing everything everywhere all at once. But then again, to be real, this is happening very fast, and I don't think anybody has articulated a credible big picture plan so far. If that's true, and we're mostly just going to keep developing this technology as fast as possible and hope that the resulting AIs end up being mostly harmless by default, I do find a lot to like in their vision for a more decentralized and hopefully resilient balance of power, as opposed to a world dominated by a few major AI players. In the second half of our conversation, we get into the technical details, including both the fundamental challenges and recent progress in distributed training. Johannes walks us through the 3 main parallelization strategies used in model training: data, pipeline, and tensor parallelism. And discusses strategies like DeepMind's DLOCO, which reduces communication overhead by allowing training nodes to process hundreds of steps before needing to aggregate gradients and sync model states. That they've managed to use this and a number of other optimizations to train a 10 billion parameter model across a globally distributed network of compute resources is impressive. And the latest from Google called Streaming DLOCO, which was released just after we'd recorded, suggests that they have not yet hit any fundamental limits. It might still be very difficult to aggregate enough compute to train foundation models from scratch in a distributed fashion, but the recent shift toward reinforcement learning, which is far more inference-heavy and thus friendlier to distributed training approaches, strongly suggests that any number of network groups can probably muster enough compute to train whatever models they might like to reinforce into existence. Or, as Anthropic's Jack Clark put it in a recent edition of Import AI, we might soon live in, quote, a world of models trained continuously in the invisible global compute sea. That world would almost certainly be simultaneously weird and beautiful and scary. But barring extremely draconian measures, the likes of which are well outside the Overton window today, it seems like the sort of thing that technologists can and will create, and that governments will have a very difficult time preventing. Or controlling. Perhaps in the end, we can only hope that DIAC, the strategy of accelerating the differential development of decentralised defences, as exemplified by the Metagene 1 model, will ultimately win out. As always, if you're finding value in the show, we'd appreciate it if you'd share it with friends, write a review on Apple Podcasts or Spotify, or leave us a comment on YouTube. We always welcome your feedback and suggestions via our website, cognitiveevolution.ai, or by DMing me on your favorite social network.

For now, I hope you enjoy this discussion about a positive vision for and the technical underpinnings of decentralized AI development with Vince Advicer and Johannes Hagemann of Prime Intellect. Vince Advicer and Johannes Hagemann, founders of Prime Intellect. Welcome to The Cognitive Revolution.

Thanks for having us.

Thanks for having us.

Yeah, I'm excited for this conversation. For a while, I have been really wondering about distributed training and if it's going to work, what the trade-offs are going to be, and just generally the concept of decentralised AI, I think, has been quite fascinating. And when I recently mentioned on an episode that I'm looking for the right people to talk to about this, your names came to my attention. So I appreciate you being willing to do this and definitely look forward to getting into some of the nitty-gritty details of all the work that you have done to realize this decentralised AI vision. And just before even getting into that though, one of my common refrains is that a positive vision for the future is currently the scarcest resource. And I'm just always amazed by how little of that we hear, including from people leading the frontier AI companies. We get very vague visions of what's going to be good about the future. One thing that has struck me about the two of you as I've explored your work in preparation for this is that I think you do have a positive vision for the future. So I'd love to start off by giving you an opportunity to articulate that positive vision for us a little bit.

So I can kick it off to frame it also in kind of like starting with what we plan to build with Prime and Dedict and then more broadly, like how I see the positive vision of the future with AI. So basically, really, like our goal is to make intelligence and compute too cheap to meter. And I think we're entering that era right now. And really, I think The key goal for us, I think, which would be a key part to make it utopian, is to make it widely accessible and open to everyone. And I think really, like, the goal is to see almost like if we zoom forward, right, a few years or decades ahead, the biggest risk would be if we have this super powerful intelligence and compute, but it's not accessible to everyone, but rather to the select few and kind of like The big states, the big tech giants and individuals. So I think that's really how I see the most positive future. And I think what this enables really is like, I think it empowers like every human and creator to leverage their intelligence, right? And like be able to do more with less, right? It's like basically have agents at their disposal, like create more advanced science, more like have better access to medicine, to education, right? To all of these things. So I think it's actually, I think really like has the potential to usher us into kind of an era of like widely distributed abundance if done right. But if done wrong, it also has the risk that it actually kind of like creates like much harder to access intelligence age, let's say. Right. And I think that's the most important piece. I think that basically large parts of the society is like able to participate in this like upgrade.

Yeah, we're, we're pretty much aligned on that, right? I think it needs to be widely distributed in a sense, right? The most dangerous outcome is that if there's only one superintelligence, right, like there's state versus zero, obviously, but the distributed way is the way we are going forward. So that's how we basically think about the future of open source AGI as well as ASI.

It's interesting that you mentioned AGI and ASI in that, um, in that answer. I think, you know, so many AI worldviews or expectations really diverge on how powerful do people expect the AIs to become. And on what timeline? How would you compare and contrast your own expectations to what we've heard recently from Altmans and Darios who seem to be increasingly confident that we are going to see superhuman AI in the next few years?

Yeah, I think it's basically like most of the views put forward for me are very plausible, maybe even since the last 15 years, since reading The Singularity is Near and Bostrom when it came out. And I think basically we're still on that path, right? I think it's actually surprising how well those predictions held up. I think the most common views of the big lab leaders, like from Demis, Dario, Sam, to folks like Leopold, I think I would put basically high probability mass on their scenarios. And I think they also base it off obviously the research progress that we've seen over the last— I think the biggest question I think is like, how In general, it will be in like the broadest sense, right? And I think a lot of people I talk to also at the labs, like, have high confidence it will get to superhuman coding, superhuman math, maybe superhuman software organizations that run autonomously, right? But that doesn't mean we get superhuman humanoids like tomorrow, right? Like, basically, I think a lot of these things will look probably like autonomous cars or something where it's easy to make radical progress. It's hard to get to 100% accuracy and robustness, right? I think basically an autonomous car that isn't 100% robust, but like 95% robust, right, is a bit useless. And I think we'll see the same with superintelligence where it's like the deployment of it, right, will happen if it has extreme high robustness in its application. So I think it's not enough to have a superhuman investment agent. It can't screw up 1% of the time and lose you all your money. And I think that's how we'll see probably progress shape up, is that actually I think it will move much faster in some areas, like we've also seen, right? Like with the new inference time compute scaling paradigm. And I think we'll see it to move slightly slower than some of the biggest optimists maybe make it out to be in trickier, harder to simulate areas, right? I think really the useful framework, I think it's even folks like Stephen Wolfram on kind of like how computationally irreducible basically different areas are, right? It's more difficult to solve all the diseases.

Yeah.

than to write a bunch of code autonomously. So I think we'll run into the limits of what you can compute, right? And I think that's probably one of the things, like the only maybe bottleneck I see in the next 10 years is like that you can't simulate everything, you can't compute everything, but a surprisingly large amount, right? And you can approximate maybe reality with like pretty good simulations, including for things like biology. So yeah, I'm quite optimistic that like I think most people are not optimistic enough, especially in the face of like all the progress, in the face of like the infrastructure buildout. So I think we're on track for like superintelligence in like probably the next decade.

So can you maybe put a little bit more color on the future that you envision? Like, I hear you on the accessibility part. I definitely think people are underestimating What's coming, but I'm more unsure as to like, you know, is it gonna be good by default, bad by default? You know, what are the hinge points? Obviously a lot of questions there. It's funny you mentioned like Kurzweil too. I've increasingly been using the phrase Kurzweil's revenge recently, 'cause in retrospect, I look back and basically say there was really no other way this was gonna go. You know, it seems like the fact that so many different algorithms seem to work. And the fact that we have the compute at scale and the data at scale, there was really no trajectory where we were going to have these compute and data resources and not have somebody figure out a workable algorithm. So I think in that very fundamental sense of if you have the physical and informational inputs, somebody will figure out how to unlock the lock and we'll get reasonably and potentially very powerful AIs. That seems to have been very well borne out. Do you have a vision for daily life? Am I working a job in your future imagined successful scenario? And is there a stable equilibrium? I really struggle to envision a stable equilibrium. I think the decentralised AI line of thought in general sort of tries to get there, but I would love to have a more concrete, detailed, richer vision of that than I do, to be honest.

Yeah. Yeah. I think like it really depends on different dimensions and axes and also on a timeframe, right? And like personal preferences. I think in many ways, people that are like financially independent, right? They could basically retire, but they could also still work on a startup or like do other things. And I think that's playing out, right? It's like every billionaire in society is working like probably harder, not like less hard on like their philanthropic efforts, right? Like their entrepreneurial efforts and other efforts. And I think we'll see basically more people like behaving like as if they're like financially independent. So it's like, I think in that sense, there's already a glimpse of like, I think what, like how abundance manifests, right? It's like billionaires are, uh, definition living in abundance. But I think like for a lot of people, obviously they have different priorities in life. So I think actually good recent book I read on this was like from Nick Bostrom, like Life and Meaning in a Solved World. And I think some of his points, I think, make a lot of sense where it's like there's so many other aspects to life that people care about, be it having children or doing art, creating things, understanding the world, reading, all of these things that I think people will be able to spend more time in. Even if one sees other, almost like jumps in human progress, like in the industrial age or now, Age to knowledge, like basically in those shifts, I think people were able to basically like get more freedoms and ultimately have an easier time to reach financial independence as one example, but also more broadly, almost like do whatever they want to do. Most people I know could stop working. They don't, right? It's like, and I don't see them to completely stop working once you have AGI. And I think like, what I think is interesting is like even seeing this now. with a lot of people working on this technology. I think to your point, there's the camp of the people that are like too optimistic or too pessimistic and they kind of like give up. So they like resign from their AGI jobs because they're like, oh, time to enjoy the last 2 or 3 years before there's superintelligence. But also on the other side, right? It's like kind of like pessimists who are like, oh, the end is near. And I think that's like a dangerous and kind of like unlikely scenario to play out where it's like, I think the nature of jobs, I think will change. But I think another good line of argument that I've heard, which I agree with, is like, there will be probably like a return to very like human labor. And people might still go to a restaurant and they might still go to a theater. They might not drive an Uber, like that drives itself, right? It's like, so basically I think there will just be like a different sort of labor, right? It's like, it will become even bigger. Like basically everything where you and I would like be even post-AGI, like pay for a human, right? Because it's just more joyful to have a human around than like a steel robot, you know? Like if you want your kids to be taken care of, maybe you don't want your Optimus robot to do it, but like a human you trust and can understand. But I think that's how it will play out. Like I think in many ways it won't actually be absurdly different from today. It's just that I think there will be way less like knowledge work that isn't exceptional, right? Like it will be much more curation.

Yeah.

And even like, I think all of the knowledge workers will just like still work. They'll just work in very amplified ways on different functions. And they'll have an army of people working for them for free, basically, which are they, to an extent, the AIs, right?

The one point on that I want to push on a little bit more is like the billionaire point working today. Because I feel like there's some, maybe you can resolve it, but I feel like there is some contradiction, but the sense that like, Why are they working today? I think in part it's because they feel like they can make a meaningful contribution, right? They want to make a positive difference. Does your vision sort of imply that for the people in that position, that there is not an AI that they can hand that responsibility off to, to do a better job? Because you might imagine a Bill Gates or Dustin Moskovitz or whatever might say, You know, my goal is to eliminate these diseases. I could sit here and run this foundation, but there is a superintelligence that could probably make even better grant evaluation decisions or whatever. And are you imagining that they don't have that or that they just choose not to use it for some reason? That part is always a little weird to me because it feels like if you have a superintelligence, what are even the billionaires doing?

Yeah, totally. I think there was a good interview between Sam Altman and Bill Gates exactly on this. And you could see, I was like, Bill Gates also grappling with it being like, oh, he gets so much meaning out of being so good at like fighting malaria and like, hey, if superintelligence can do it for him, he would lose some meaning because he's so good at it. Which I think is the wrong worldview, right? It's like, hey, if you really care about it, be happy if you can like hand it over to your employee, to your agent or whatever to do it better for you. I think the best almost like founders or like philanthropists or whatever, they build a team that is way better than they are on every dimension. Like basically, maybe they are, they have broad skills, but like ultimately, ideally, they hire people that are like super intelligent compared to them on something very specific. I hope Elon has someone who's way more intelligent than he is on propulsion, on autonomous cars, right? And AI and systems, right? Like on across all those efforts. And I think that's more how it will play out where it's like, like basically execution is cheap. Ideas are worth everything, right? In a world where you can just like, it almost inverts to the current reality.

Yeah.

And I think it would just lead to like billions of startups, right? And billions of movies or much more, right? It's like basically just an explosion of content companies, autonomous creations that ultimately still like a human is either like seeking or creating. I'm kind of extrapolating from like where we are today, right? And like how I see and try to adapt to what I see as like plausible in a year and 5 years and 10 years. Like, I think there's still a lot of things We'll want to do, like, we'll want to work on. And I think to your point, I think like the ultimate, like even this like almost like Maslow pyramid of needs, right? It's like there's things like impact, status, right? And like meaning and fulfillment that are like timeless. And even if you have superintelligence, you might still get like meaning out of specific things, right? That you're like, seek. And I think I see it almost more like humanity will just have like a gigantic workforce of like AIs and agents that like do a lot of the work, but like, it will basically hopefully stay on top of it and like basically manage that and direct it towards the aims like humanity cares about, right? It's like, it's a bit like in some ways our biggest corporations or states are already like superintelligences with a lot of agents in it, and they're directed like through democracy, through like a shell like capitalism or whatever, like into specific directions, right? And I think that actually the biggest risk, right, is that we'll just have like nation states and like for-profit corporations have the superintelligence instead of like everyone to do like everything they want to do.

Hey, we'll continue our interview in a moment after a word from our sponsors. When you said hopefully, the one interjection I was going to ask is, do you maintain a P-doom? Do you entertain that conversation at all?

I think it's not extremely useful as a concept. I think basically, I think a lot of people are way too confident in like, weird sci-fi scenarios and kind of like probabilities stacked on probabilities. But then if you ask them to like unfold them, like they can't really. It's like, I've seen too many people, you know, like they're like, oh yeah, my p-doom is like 20%. And I'm like, why? Oh, because this person also has it. And it's like, basically it usually falls apart. I think it's, I think a mind virus. Like, I think it's a really dangerous concept actually. Like, I wouldn't put strong confidence in anyone putting out random numbers. And I think there's almost like, it's really about like probability math, right? Probability is like between 0 and 100%. And it's like, the universe will go on for billions of years. Like, will something at some point kill it?

Sure.

Like when, how, and what kind of like the order of events is, I think it's like so, so hard to know that I think it's almost not worth pondering. And I think one of the problems like the utilitarian philosophy brought to the space is to have very high confidence and possibly to know future scenarios and events and then basing policy, all their actions. So I think what's more important, and I think the ironic thing is that in this worldview, I think one needs to realize that ultimately on shorter timeframes, the probabilities are stacked in favor of good outcomes. And then even the actions, like, for example, on AI policy, sometimes have the exact opposite consequence where basically it's like someone even with a low p-doom, it's like, oh yeah, that's why we need to regulate it heavily in Europe. Okay, goodbye AI in Europe. Or we need to regulate it heavily in the US. And suddenly things move outside of this. So I think there's basically a cascade of very bad, almost epistemics and reasoning. That follows from people going too deep on PDoMs.

I've certainly seen some of that. At the same time, I do sort of worry that things could get out of control. I answer that question when I'm asked for my own PDoM. I usually say 5 to 95% or like 10 to 90%. And yeah, what I mean by that is basically just nobody has said anything to me that seems compelling enough that I shouldn't worry about. things getting totally out of control, nor that it's so likely to happen that I should check out and spend the rest of my life at the beach or whatever. But I do worry about the possibility of things getting out of control. Do you think that worry is misplaced, or do you share it to some degree? I don't think quantifying it is super useful, but I do think considering it is still useful.

Like, I see it almost like the Industrial Revolution brought, like, it created chaos, like it created havoc, created maybe like civil war in like some ways and like had second, third, all the consequences. Was it that good? Sure. And like, I think, or like the internet, right? It's like created like some like complexities, political consequences and everything else. And I think still, like, ultimately, I think like AI, like almost like benefits like far outweigh risks. And I think also some of the paths to avoid the risks become risks themselves, right? It's like basically like kind of like in a world where like everyone is worried about like the existential risk and then like calls for like one world government. I think probably the one world government is like the bigger risk than actually kind of like sci-fi scenarios that people are worried about. So I think All of these things are very complex systems, right? And ultimately, I think it's hard to know what will be specific risk, right? And I think the biggest risk is actually people losing their autonomy, like putting life on autopilot with AGI, losing their freedoms because they hand it over heavily to the state and to overregulation. So I think really that's my biggest worry is basically that even the most well-intentioned people actually create almost like the worst outcome or scenario unknowingly and unintentionally. And then also in some ways being misused by people that seek power, right? I think like a lot of the people that like were fighting for alignment, safety, and policy have, I think, been co-opted by big labs and big tech that has their own goals and their own intentions. And you can see this really well, right? Like with the big tech giants and the big tech AGI labs. And I know stories about almost all of the AGI labs that are extremely worrying of things happening behind the scenes and how they power seek, even the people that this whole community kind of reveres. So I think that's the thing that I'm really worried about is basically seeing some more details that I think it's like the biggest risk I think is basically full centralization of superintelligence by a few nation states and big tech giants. And basically completely disenfranchising large parts of humanity.

I don't know, like, it seems like it's all in play in a way that it's never before been in my life. But I do hear you very much on the idea that a crazy imbalance of power, or just like, you know, one superintelligence in one organization or one company, or, you know, one even potentially even one individual's hands does seem like a really problematic situation in biology, right? One lesson I've learned over and over again in life is anything that is sufficiently concentrated is dangerous. All the drugs become dangerous when you take them from the leaf that it grew in on the tree. That was fine. But then when you purified it down to 100%, and started huffing that, that's when you really got into problems, right? And that seems like it kind of keeps coming up over and over again. So I am quite sympathetic to the idea that if we can maintain a sort of buffered solution, sort of balance of power, that where everybody is sort of checking everybody else, much like things work today, right? Nobody has the ability to dominate the world unilaterally. If we can maintain that as we bring AI online across all aspects of life, I think that does sound really appealing. Of course, I don't really know what that also could bring. It seems anything is possible there. But I do find a lot to like in that vision of the sort of everybody rising simultaneously, everybody's interests kind of keeping one another in check, no single actor being able to make a mistake that kind of throws everybody else into a huge problem. Let's talk about the company and your kind of master plan for getting there. I love a good master plan. And again, you've got one. So one thing I wasn't able to tell from my research is like, what kind of a company is it? Is it like just normal corporation or something else? So maybe, you know, give us a little bit of the foundational backstory and then take us through your master plan.

So basically, like, almost like our broader goal really, right, is like to make computer intelligence kind of like too cheap to meet. And like, really how we want to do that is like, In multiple stages where basically in the first one, we started with aggregating global compute and creating an efficient compute market, but also the developer interfaces for it, like an API, command line interface, and basically ways for people to just like, no matter what kind of compute they're looking for, like be it 1 H100 or 1,000 H100s, that they can basically make a request and then they find like where they can find the cheapest compute that they're looking for. So that's basically how we started and how we launched 2 or 3 months into the company creation. So basically that really was like also the biggest thing we initially did and which gained a lot of traction. And then really building on top of this and kind of like in conjunction is building basically our decentralized training and other approaches like distributed synthetic data generation and others, which really can then like leverage the global like compute fabric, right? It's like you can envision if there's extremely cheap compute in one place or another and it dynamically shifts. It's extremely beneficial if you can train decentralized in a fault-tolerant way, because then you can basically just save massive amounts, but also really leverage the global compute, especially when there's idle compute available somewhere in the world. So basically it ultimately reduces the cost of intelligence, reduces the cost of compute because you can make more efficient use of this resource. So that's kind of like step number 2. And we also made a lot of progress there and can go into more detail. And connected with it, which is very closely connected, it's like the goal is really to train kind of like leading open models collaboratively that anyone in the world could contribute their idle compute to this network and also get rewarded for it. Basically make the highest possible return on their compute because it gets fully utilized. And then if it's, for example, idle, if it's not utilized, it can contribute this to frontier models, right? to continuous improvement of like all the R1 model from DeepSeek, which is one thing we are looking into. And I think there's basically like, that's almost the third pillar, which really connects to this like more broadly to this like peer-to-peer compute and intelligence, where the goal is anyone in the world could contribute compute, anyone in the world can use that compute. And it's kind of like an extremely efficient market for compute. But on the other side, like I think on a more abstract level, the same can be said about intelligence itself. Basically, be it any AI API, it also becomes sort of a market, right? It's like there's different people hosting LLaMA models, DeepSeek models, diffusion models, and they're also competing. Ultimately, their cost is purely compute, right? So basically, you can lower the cost of those intelligence endpoints, if you like, of agents, of LLaMA, of like other models by having the most efficient compute market. So basically, like an efficient peer-to-peer compute market also enables like efficient peer-to-peer intelligence sitting on top of it. And really, I think the goal is to create like a system that is almost more like a protocol, like parts of the internet that like can basically be maintained like almost like a public good. And people can just like use it trustlessly and permissionlessly. And I think the closest to structures like something like Ethereum, like for most more like smart contracts, which like really the goal, right, is there to like create a technology and make it openly accessible, have a like foundation-like structure to support it. And then over time, like it really can be owned by anyone. Anyone can use it like for the cheapest cost possible. And I think, Derek, that's a broader vision. And I think to your question, how we structure this, kind of like it's a Delaware C-Corp, but really The setup is more akin to Ethereum in the sense that we're creating a foundation and basically giving out grants for people to develop this in the open, right? Like through like open source approach to basically fully like make this accessible to anyone. And anyone could fork the system, improve the system, contribute back to the system, participate in it. And that's really the broader design principles, almost more creating like a public utility to some extent. And also in conjunction with some of those systems like Ethereum. where anyone can also create their own agents and commercialize it, and the agent can make money for you, right? So you're, I think that's also like going back to this utopian vision, like how we see this play out is that like anyone in the world can like contribute to it, own a piece of it, and it basically just like generates like revenues, like for the end users that participate in that system.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Yeah.

A lot of points I wanna follow up on there. Maybe let's start with the compute market and we'll work our way forward in time through the master plan. For starters, how would you describe the compute market today? I think we've had a sort of much hyped cycle of people raising large equity rounds and just plowing all of that proceeds directly into GPUs. In some cases like Nvidia, even just taking equity positions in exchange for GPUs.

CPUs.

It seems like that seems like it's over. Are those the kinds of companies that are now saying, geez, we maybe overbought a little bit and have more, we maybe can't use the cluster we rushed out to buy as much as we thought we could, and now they're contributing to your marketplace? Yeah, I mean, I have a million questions about the compute marketplace, but tell me what you think is interesting that's going on.

I think if you zoom out, GPUs were a tiny, tiny market, right? Like pre-ChatGPT, and they're growing at an exponential rate. And I think there's no slowdown in sight. I think basically it's— and you see this, obviously it's, it's powered a lot in the sense that like, obviously this week was kind of like an ominous, right? Like $100 billion plus in like compute commitments from individual AI companies and labs. And I actually think the trend right now is rather that there's much more demand for compute, including a much longer tail of companies. startups seeking that compute. But I think to your point of like, kind of like how this market works and structures, right? Like how I would describe it, right? It's like a lot of the, like we basically more than a year ago, right? Like we're in a period where like the supply for H100s was extremely constrained. So basically the big guys that paid like extra got more of them and kind of like got them with priority, right? Like the OpenAIs of the world. And then it also shows you kind of like the problem, right? It's like those Big tech giants got priority because they paid more and have like gigantic long orders and clusters. But I think what we've seen shift is basically it's like that there's obviously now more supply. So basically it's like the production started to catch up a bit and obviously also new generation, for example, concretely with Nvidia rolling out. So I think basically a lot of the small startups on the point of like raising capital and deploying it bought long contracts, right? They bought like 1, 2, or 3-year contracts. And they paid hundreds of millions and they had to basically, it's like if you wanted to train a model and you needed 1,000 GPUs, you had to buy a 2-year contract to get it. It wasn't possible to rent them on demand. And then kind of like fast forward, like now, like on platforms like ours, you can rent 1,000 H100s like on demand. And that wasn't possible even 3 to 6 months ago, really. And I think it showcases you that basically from the supply, which the majority was kind of like, for long-term contracts, right? Like, which from a capital perspective are more de-risked and almost more like structured financial products, right? It's like CoreWeave buying like tens of billions in compute, giving them to OpenAI, taking out loans against the GPUs. Like, that model I think is evolving that basically more and more of those compute is actually moving to the on-demand supply. So basically it's like really, I think also what powered all our growth and positioning is that like A year ago or 2, it was impossible to find an H100. And then it was very useful what we offer in the sense that that's also why we started it. Every AI startup and developer we talked to, right, like, had a very hard time to find H100s, but we were able to find them going directly to data centers, talking to some billionaires that bought like tens of thousands of H100s and were looking to like sell them on demand. And, but fast forward, like now there's even more data centers, there's even more clouds that are appearing. And really it's a market that is insanely fragmented. So I think to the outsider, it's not so obvious that basically there's hundreds of clouds, there's thousands of data centers, right? And there will be more of them. And like, there will also be more chips than like Nvidia chips, right? So really you need to aggregate a market that is like by design so fragmented. Like basically Nvidia keeps it fragmented because they don't want to have like an overly powerful buyer, right? Like if OpenAI or Microsoft would be the only buyer, they wouldn't be in a good spot. But then on the other side, Their biggest clients also compete with them, right? It's like Google, Amazon, like all the others, big buyers. They also produce their own chips, right? That compete head-on with Nvidia's chips. So basically they need to be careful and really gave a lot of allocations to modern folks like Lambda Labs, CoreWeave, basically like in some ways smaller clouds than the really big tech clouds because they're more aligned in many ways. So basically that's why supply is kind of like distributed across America, across the world. in hundreds of data centers. And really, as an end user, you kind of want to discover like, hey, I need 200 of them. Where can I find them right now? Like, and it's not so easy and clear because also the, obviously the demand and the supply is shifting in real time, like with all of these platforms. So that's kind of like the broader backdrop. And I think the other piece is like that, I think the, there was obviously some of those like very big and well-known cases of like startups that raised a lot, Then started pre-training models, right? And I think the thing is, that's continuing to be the case, right? It's like, there's like more AGI labs today than a year ago, right? It's like a very obviously well-known case. Out of OpenAI, there's now like 4 AGI labs I can count that weren't around like 1 or 2 years ago. There will also be like a lot of applications that need more and more compute, right? And especially some new use cases like video, like the more heavy reasoning or coding agents. And in general, like agents coming onto the scene this year, they'll need much more compute. So I don't think there will be a slowdown in compute. I think on the other hand, like, will become like the biggest chunk of GDP, probably, like almost endless ramp up to AGI or ASI and probably just like power all the different intelligence use cases.

So would it be fair to say that you are playing a role that's like an aggregator of people that are also selling direct? Like, you kind of like, I don't know, to pick one, like a Kayak type of aggregator where I could go buy directly from the airlines or I could go to Kayak and They give me this sort of menu of all the different airlines, as opposed to like an eBay where it's like, hey, I've got 3 H100s here, you know, can I plug them into your marketplace? It sounds like it's more like the former. Is there— are those good, you know, frameworks from which to think about it?

Yeah, I think like, ultimately, it's not so different from the biggest marketplace in the world, like Amazon or something, right? Where it's like, anyone in the world can like, basically sell compute on our platform. And also, like, anyone that is selling compute, we are in touch with, like, from the hyperscalers to individual data centers, to clouds, to billionaires that have compute, to startups that want— that have too much compute, to startups that want to buy compute and put it back onto the platform when they don't need it, right? Like, to get cost savings out of it. So I think the best analogy is just like a lot of the platforms, right? Like, are kind of like little isolated islands and we try to create the ocean that connects all the islands, right? It's like basically you have some compute in one cloud, you have some in the other, you have some in one big tech, some more in the other. But basically it's very, very hard to orchestrate all of it. So it's like literally a lot of the people we talk to, like even start off with 2 people where we're building their own APIs to like query the availability of all the GPUs. Then they have like different APIs and like that's basically all the work that we are like simplifying for the end user. is that you can just orchestrate this kind of global compute fabric. And I think the other piece is that a lot of them are almost more like a PE. They're basically structured financial products. They borrow a billion to buy a billion of compute and lend money against the compute, right? It's a very high CapEx business. And I think what's very different is we don't buy hundreds of millions, billions in compute. So in that sense, we are not a hotel. We are more like Airbnb or like in that sense, right? We are more marketplace sitting on top even of other marketplaces. So basically some of the marketplaces we talk to that are trying to do something similar, they're not going for the full broad spectrum, right? For example, they don't work with the hyperscalers or with like some of the larger clouds. They just work with like individual data centers. So basically we try to just like aggregate all of them, right? Also over the time and onboard all of them onto one market.

What's the smallest unit of compute that would make practical economic sense for somebody to contribute to the marketplace today?

Yeah, so basically we have every chip that is in demand, right? It's kind of like a market where it's like the majority of the demand falls right now for like H100s, like GH200s, like kind of like A100s still to some extent, or 3090s. But like there's obviously over time, I think reasons to believe that, like, for example, for synthetic data generation, you could even use your, like, MacBook or something. That's not our primary focus right now, but I think it's like, there's compute everywhere, right? And it's not just in the data centers and H100s, even though that's our primary focus. Like, over time, I think the goal is basically that you can literally leverage all the compute out there. Obviously, some of it is like much more useful for the different AI workloads, right? But also it's like, now, for example, that we're shifting to the synthetic data paradigm, it's suddenly some new compute starts to become useful, right? Like now that NVIDIA is shipping their own, like almost like home GPUs, right? Those become potentially interesting over the next few years, like especially if like a lot of people get them. So I think like all, like the answer I think is like all of your compute, but like the compute that is in highest demand, like will be more successful on the market, right? So it's like very market driven in that sense.

Exactly. For users on the platform, obviously all kinds of compute resources would be useful, right? That they Yeah, I mean, for their development purposes and stuff like this. For the more distributed training component of it, right, contributing resources there, it's very much more focused on their high-capacity GPUs, right? Like an H100, an A100 node, a full node, or even full clusters, right? Just because of memory requirements and other components needed to actually do distributed training across those. So there, the focus is more on higher-end GPUs. There's still use cases for like the 4090s or the smaller clusters and stuff like this, right? For large-scale synthetic data generation and other things. For training, you need quite a bit of memory requirements. And that's why it's more focused on A100 and H100. It's also for our distributed training runs.

Part of why I'm interested in sort of the economics or the sort of practicality of contributing to the market is it seems like in the long term, we may be headed for a fight about who can compute under what controls, under what circumstances, with what oversight, whatever. Trillion-dollar data centre is one idea where it's like, it's all super aggregated, it's all super centralised. Obviously, that also becomes, from a geostrategic standpoint, hard to defend, but it is probably relatively easy for a government to control. You can sort of fence the perimeter and know who comes in and out. On the other end, there's not really anything like this for AI yet, but there is Bitcoin and Ethereum. And these are things that are truly like, not something people can shut down. It would be very difficult at this point for even probably— I mean, tell me if you think I'm wrong about this, but it would seem like even the United States government, even if it really wanted to, would have a really hard time shutting down those networks. So I'm kind of wondering, where are we on that spectrum today with these sort of global decentralized compute fabrics? Because if it's— you have to be at a significant scale and it's basically a data centre, it seems like we're obviously not at the extreme of the trillion-dollar data centre, but a lot closer to that where it could be controlled or governed or whatever. Whereas if you really could get down to, I got a Mac Mini on my desk that's idle, and I'll rent that to you for $0.05 an hour, potentially not even for the money, but because I want to contribute to a global network that nobody can shut down, then we sort of are in a very different regime. So where are we today and where do you think we're going?

To maybe touch on the first point you mentioned, the trillion-dollar cluster, I don't think it's going to be one single cluster. Even the Stargate round will be distributed in a way, right? All the big techs are looking into how to train across their distributed training clusters as well. They obviously have a bit more favorable environment than we are in. They have cables between their data centers, 40 gigs of interconnect and at least 100 gigs, which we obviously don't have right now in like a globally distributed setting. But even there, I would argue it's also not that easy to control that you just have like one single location that you have to prevent somebody access to, right? It's, it's gonna be distributed across the globe if you wanna build something like a Stargate cluster because of energy requirements. You can't build a 10-gigawatt cluster right now, but we are gonna be limited to, to multiple like 500-megawatt clusters or something like this over the next couple years.

Yeah. And I think what I would add is Like with this distributed training run, like we trained across Europe, Asia. I think both things can be true at the same time, right? It's like computers extremely distributed. Like it sits in Europe, it sits in mainly America, it sits in Asia already. It sits in other parts of Asia like China. But also there's a ton of compute in India. There's a ton of compute in like other almost like more free like nations, like be it Singapore, Malaysia. And a lot of the compute is shifting to the places. where it is the most free, right? Like where you basically can generate revenues where we are not encumbered by like stringent regulation, right? So like places like Malaysia and Singapore, right? Like seeing a boom in compute. And I think that's one piece. I think the other is like, we'll see these big data centers, but like there's a gigantic long tail of compute, right? Like basically there's a lot of data centers that have like hundreds of thousands or tens of thousands of GPUs. Most don't have actually 100,000 or a million, right? Like those you can count on one hand, right? I'd say they have like 50,000 to 200,000 GPUs. It's more like the xAI, OpenAI, and Microsoft clusters. But there's so much more compute. And I think the other is, to your point, all the lower power compute sitting in people's homes, right? I guess a ton of people with 4090s, 3090s across the world, right? Like in every part of the world. So I think it's not only extremely naive to try to regulate compute, but it's also extremely dangerous. David Deutsch said on a recent interview, it's like regulating code or math, like, it was like, basically, it's like this, on the one side, extremely dangerous, but also extremely naive in the sense that basically it has the exact opposite effect, right? It's like, that leads the compute to shift to nations that are even less easy to regulate and observe. It's like, it shifts to the boundaries and to the actors that don't obey by regulation, right? And to an extent, you could see this even with Bitcoin as Ethereum. And I think the exact same thing will play out. Basically, a lot of it went to the cheapest regions in the world where you could mine, right? A lot of it was in China, and then China banned it, and then a lot of it moved to the US, and a lot of the business associated moved to the US. But energy was cheap in other regions, so it also moved to Iceland or something, or like parts of Africa. And I think like, it's how I see this play out as well, basically, where ultimately, like, you're maybe the best example. Like, we're both from Europe, and I think We both left Europe. And I think one of the reasons is because Europe is known to be extremely diesel and like co-opted by those regulators who ultimately don't have the best intention of the citizens, of the future and of the economy of Europe in mind. But they basically overregulate and kind of like, like Europe didn't even have an AI community right now. It's dead. Like ultimately it was not even really breathing and alive. But like, it's nonexistent now, right? And I think it's the best example, zooming in on a nation that completely fumbled their participation in the new intelligence age by a few co-opted regulators. And I know those people, right, like that regulator, they had the best intentions. They didn't plan to destroy the future of a whole continent, but they did. And I think that's a very cautious lesson. And I think fortunately, like, I'm very excited for America because ultimately America doesn't repeat that mistake. I think it was on track to repeat it under the last administration. And I think fortunately, like, we are on a bright path, like, for optimistic future. Otherwise, the US would've said goodbye to their future and their stake in AGI. And I think that would've been extremely dangerous, would've pushed it even harder to Asia than it already is right now with progress like DeepSeek. So I think basically it's important to think through the second, third order consequences, right? And like what ultimately the goal is. I think There's been a lot of policy people that I know, like some of the also most powerful ones that really even regret some of the policy they kind of like argued for. I know some of the people that indirectly funded some of the biggest ones and are not big fans and they publicly stated or will continue to publicly state it. And I think you would be surprised that like how many of the people that actually enabled the big policy regret having done so and stopped doing so now. And I think that will continue to be the case. I think be more public in the future because now the Overton window shifted.

What policies do you think people are regretting?

I think the EU AI Act will go uniquely into history as being one of the worst ideas for not only Europe, but also having completely zero impact. It didn't really improve anything for anyone, like literally nothing, but it made it worse for small AI startups. Like, obviously it's like a general problem Europe has. It's very proud to regulate. And it's very proud to overregulate industries that are very tiny to nonexistent, right? So I think the EU AI is a very concrete example. I think even the California Act is an example. Even the people that were arguing for it, they said like they should have done it on the US level instead of just like on the level of California. But I think those 2 things stand out for me as the 2 most well-known policy initiatives. And I know some of the key policy people at the big AGI labs that stopped being there. And they regretted it. And I know like some of the by far biggest funders of policy, if you talk to them, they regret that they made those donations. And I think there will be more of that and will be more public in the future. So I think there's a lot basically where people realize the outcomes of policy, like funding superalignment, great, there should be more of that. Ultimately funding policy quite quickly goes wrong, right? Like quite quickly gets co-opted, quite quickly has the unintended consequences. And it can happen that it goes horribly wrong. And I think that was to an extent the case, I think, with a lot of the government, like the policy. But I think on the other side, it's like the majority proposal is compute governance, right? And people like Jay Clark also commented on our work and said like, hey, I think those folks need to update. I think compute governance in the face of decentralized chain working makes zero sense, especially how it's done right now. So I think That's the other piece. Policy proposals were very slow and lacking, but also very naive. And ultimately, I think not actually in the spirit of freedom, not in the spirit of democracy, not in the spirit of basically free markets. And I think that's very problematic. Ultimately, I think it's like, and David Deutsch actually did a really good interview on this, and I think he basically said, it's like, if you're ultimately worried about basically unsolvable problems, it's like the only path forward is to solve them. And it's like ultimately AI can help solve those problems. AI, like we proved with also like one of our first model releases, can help make the world safer with regards to biosafety. So I think maybe the philosophy more broadly underlying this, like, is, I think, well encapsulated by what Vitalik wrote, if you read it on like Diag, of kind of like basically accelerating defensive technologies and differential progress, like democracy and decentralization. That ultimately is the path to probably the brightest future in this direction. It doesn't include governments overregulating. It's rather making sure that ultimately the good actors are supported while the bad actors are stifled. Right. And I think like policy and regulation concretely has the opposite effect.

Right.

It's like the best and most well-intentioned actors, they'll follow it. The bad actors, they don't care. And I think that's the reality to an extent.

Yeah.

With how it's already playing out with like governance.

I feel you on the counterproductive nature of many regulations, for sure. I'm personally worried about that right now with all of our policies that are directed toward China. I'm like, I just can't shake the feeling, no matter how many arguments I hear to the contrary, that it feels like the actions we're taking now are going to make this situation worse. You also mentioned superalignment, and that was one other thought I wanted to kind of pick up on. What do you think is reasonable for society to expect of the frontier labs? I mean, I was a supporter. I went back and forth on SB 1047. Initially I thought, yeah, it seems okay. Then there was a moment where they, like, there was many drafts, of course, right? And then there was one moment in time where it seemed like this sort of panel or committee or commission, whatever it was officially called, was going to be kind of overempowered. And I was like, man, I don't know. It doesn't sound that great to have these, like, 5 people that are appointed by who exactly? The governor. And then they're going to get to make all these decisions. So then I was kind of out on it when that was the version that was being considered. And then they cut that from the final version. And then I sort of would summarise the final version as basically like, if you are doing frontier AI work, you have to have a safety plan. You have to publish that safety plan, and you at least have to be like, open to a certain amount of scrutiny. That to me in the end seemed like pretty reasonable as things go. And I'm very mindful of the risk of unintended or even outright opposite consequences from what policymakers intend. That seemed like a pretty light touch, like not super crazy sort of thing. Even if it's not like law, you could frame it as law, policy, regulation, whatever. Or you could just frame it as like, Normatively, in your opinion, what does it mean for a frontier AI developer to be acting responsibly in today's world? Because it does seem like they're taking sufficiently potentially disruptive steps that they should be proceeding with caution. They should be trying really hard to act responsible. But what does that mean in your mind, or what should we expect of them or demand of them?

Yeah, I think like the, a key aspect is this differentially accelerating defense, right? That's basically that ultimately you and I can't defend against like an AGI if we don't have one ourselves with intent to like whatever, like defend against like cyber, like bio and other things. So I think going back to the frontier labs, I think a lot of the way how it's currently being done, I think it's like very reasonable, right? It's like they do red teaming, they have their own goals to like Publish something that is safe, right? And that basically can't easily get misused, et cetera. I think ultimately the, it's a bit like you can create a technology, right? And like the burden is to an extent on the user, right? It's like a lot of things you can misuse, right? Like be it the internet, a car, or anything. And ultimately most people don't, right? Like most people use it very lawfully and carefully, right? And I think the same we should assume with AGI is that like the majority of humanity by far, like will use it for the best purposes. And I think that's, there's like a very tiny minority that will misuse it. And I think that's really where the focus should be, right? It's like basically stopping misuse of generally like technologies, right? But I think the other is like, it depends, I think, on the flavor of AI research where I think there was like very reasonable steps being done by like all the labs on like safety and alignment and all of these things. And I think the benefit really with like open source is that you have actually more oversight, right? You have more transparency. You have like, just by design, like people are able to jailbreak that and create solutions for the jailbreaks. They're able to basically just very quickly like make those systems much more robust. So basically I think it's like, if I think about a system, right, it's like that I want to hand my healthcare into the education of the kids and like basically A key part of societal functions, I want to make sure that actually we can see the code, we can verify the code, we can run tests against it, right? We can battle test it. So actually, like a black box, like we were served it by the AGI labs, doesn't instill a lot of confidence in me. So I think there's basically, I think, fair demands for people to get more insights into those frontier models than just an API endpoint, right? Basically, Like limited to access the capabilities that actually those models have, right? And I know you, I think also red team, right? Some of the models in the past, like I'm curious on also on your perspective, but I think there's a lot one can like find out, like even using those closed black boxes. But it doesn't mean you can't look into the black box, like you can't understand what's really happening. And I think that's the problem, right? It's like, I think we would be in a way safer situation If all the frontier AGI models would be completely open source and similar to the internet, right? It's like, and a lot of the building blocks of the internet are extremely open, like similar for blockchains, like all of the code, everything is open. They're extremely safe, right? It's like ultimately they're safer for being fully open and having kind of like this almost like adversarial environment where people try to break them, but they can't, right? Like in most cases. And I think that's really like would be the safest path also for this technology. And I think in the face of this, like, ultimately, I think there is some, like, given they have black boxes, I think closed models should have actually more stringent regulation than open source models because ultimately you can't, like, basically, I think they should open their code to like a bunch of people that like look deeply into it, that do tests beyond just having an API endpoint. So I think the jury's really out on like if closed models basically get to the same rigorous testing and transparency that open source models have by nature. But yeah, I'm curious how you think about it also from Red Teaming Lab.

I'm of mixed mind on this. You know, my background has always been pretty libertarian politics, you know, much always skeptical of regulation, generally pro-capitalism, free markets, you know, within a pretty wide range of outcomes. I do wonder how well the sort of open source label really applies to models, because I do think models are not that much less of a black box. They can be somewhat less of a black box when you have the weights, because I'm a very small angel investor in Goodfire, for example, and they're doing interpretability on Llama 3 7DB or 3.1 or 3.3 or whatever it is exactly. So there is some degree to which that black box obviously is starting to get cracked open, but it's still not like we're anywhere close to having the ability to really say we now know that Llama 3 point whatever is fully under control or won't do bad things or anything along those lines. I find myself always looking for some way to square these, to get the best of both worlds or to square the circle where I'm like, I do think more than API access and just generally more access for safety and academic researchers at the big labs seems like it should be a pretty significant priority. What access I had was purely API, very limited, very little information. That's gotten a little bit better, but I would like to see it get better still. But then I also do kind of worry that, and I think the sort of R1 moment is kind of a good example of this, that We don't know exactly what the trajectory of these technologies is going to be. And so there is something kind of risky about open sourcing frontier stuff because you can't really take it back. And if you open sourced Llama 70B and then somebody comes along and does some gain of function research on 70B, as they did with R1, right? I mean, in the R1 paper, they found that they were able to take all these mid-sized open source models to dramatically higher levels of reasoning capability in a way that probably if you'd asked the people that made those models, like, hey, what do you think is the best ever that somebody will be able to do if they fine tune this on these math benchmarks? They'll be like, oh, well, probably not that good, right? And now all of a sudden, whoa, there's like a huge unlock. And it's sort of something that society as a whole is like, that's just now the norm, right? So I do wish there was some way to kind of enable the access, enable the research, enable the understanding, but also give us some way to take certain things back if we do make mistakes. And so far, I think we're mostly just protected by the fact that the AIs that are out there are just not that powerful. And so we haven't had anything where we really needed to take it back. But it certainly seems realistic to me that in the not too distant future, you could have a sequence of events where Something that's a bit more powerful gets open sourced, and it seemed fine at the time, but then there's like another unlock of post-training or whatever that sort of takes it to a whole other level. And then it's like, geez, this now becomes a real problem, and yet there's not really anything that can be done about it. I think you guys actually have one really good example, and maybe I'll just jump to that now because you mentioned the DEAC philosophy. And honestly, one of the best examples of an open-source AI project that I think fits that description is the recent work that you guys put out. And I was really interested to see too that it was in partnership with SecureBio. So you want to tell us about the work that you've done in biology recently that kind of illustrates how we can maybe get the best of both worlds, at least in some domains?

Yes.

Yeah.

So basically for context, we trained or supported basically training of a metagenomic foundation model. For early pandemic detection in wastewater and can also be more broadly extended. And actually, I think 2 or 3 things are interesting. On the one side, it's how almost like little compute it took. I think it was like $20,000, $30,000 of compute to create a state-of-the-art frontier model to detect pandemics earlier. And so we'll continue to train it in a distributed fashion to improve it. Like there's much more data we can train it on. But I think the other interesting fact is that basically there's a lot of defense-favoring open source models that have to be built. And I think a lot of basically very objectively high impact positive models haven't been built yet. It's like the best examples is a lot of different scientific foundation models. It's like if you look at something like AlphaFold, there's very little misuse risk and it's gigantic upside for humanity. So even from almost like a utilitarian perspective, it's like, It's very heavily skewed towards like positive outcomes. Yeah. Like almost like to explain it, how it works, right? It's like basically the data is mainly just a lot of different wastewater data to detect what's in the water. And if you can counter on like, or basically detect like the outbreaks of pandemics early, which was actually the way how COVID was detected in the earliest instantiation was through wastewater detection. And I think it's how you can even like monitor on spread, like the global spread of those risks. So I think that's a good example where it's like, if we put those things in place, right, it's like basically we are in a much safer world, right, in general, right? Like irrespective of AGI, like COVID probably didn't happen due to like intelligent systems. It actually happened due to like, actually like probably at this stage, like quite, quite factually through misuse or like basically it's like mistakes by nation states actors, right? It's like by US and China and like in their involvement in gain-of-function research. And ultimately it lacked the antibodies for society and the defense mechanisms to avoid it. So I think that's really how we also go into this utopian, safe AGI future. It's like if we look very concretely at the risk and biosafety being one of them, and then just solve them. So basically, even if there is a next COVID or pandemic, which surely there will be, that's actually not a problem because we have the vaccines, We have the early detection, we have all the other defenses to basically avoid this spiraling into a problem. But I think, yeah, for the model, basically was like collaboration with UCL and kind of like the nucleic acid observatory and like the SafeDNA team. And they basically did all the research work and we helped them really on the one side with compute, but also on more like the training side, right? Making sure that a good model comes out of this. And I think it's a good example where there's like a lot more things like this, where even like in the millions of compute, you could really transform scientific progress and human progress more broadly. And I think that's really obviously also, I think, going back to the utopian vision, right? It's like, it's intelligent, like solving intelligence to solve everything else. And I think the most important thing to solve really is science. And I think it's like solving all the diseases, solving natural risks, but also Things like climate, right? And energy and a bunch of other pieces that I think will just create an enormous amount of human flourishing and progress. So I think it's actually very key to our broader goal is to help solve science. And really also with a lot of the connections we made in that world, I think a key piece of this is almost like working towards autonomous AI research and autonomous scientists, right? That can safely advance different areas of science. And it's just bottlenecked by almost like the amount of compute you can feed it to do reasoning and solve those problems and simulate them. So I think that's actually a world that we're already starting to move into. It's a world in which AI will massively accelerate scientific progress more broadly.

Yeah, I've been tracking that general phenomenon in a lot of different domains and it is like everything else, it's really ramping up. What I really found interesting about this particular model, and I'll just read a couple of highlights off your blog post. It's called Metagene 1. And then I'd be interested to hear what other sort of ideas you think have similar shape in terms of their defence-favouring properties. But this one I thought was really thoughtfully designed because, of course, we've heard from so many worriers, myself included, from time to time, like, geez, do we really want expert virologists in everybody's pocket. That seems like maybe not a great technology to make freely available in a way we can't take back. This is almost the exact inverse of that, where I think at least what jumps out to me as the key architectural decision is that the model only has a 512-token context length. And so that's obviously a lot shorter than any genome, and it's not going to be able to be used to generate full genomes. There's been a lot of talk. I'm not sure how much exactly research there's been, although I did just see one paper that came out that I want to circle back to around creating AIs that are myopic in general, which is to say we don't want these things necessarily thinking super big long-term plans. We don't want them doing super huge things that we don't understand. We want them to do really well at the thing that we put in front of them that is there for them to do. And this is that, right? 512 tokens, you can only do so much, but that is enough to detect anomalies. And so you've created something there that it does seem overwhelmingly likely that you could put this in everybody's pocket. And really the only thing they could do with it would be monitor their local wastewater for anomalies. And that just seems like— and I love projects like this that are sort of the unilateral provision of a global public good. That's one of my favourite phrases. And this really, I think, fits that bill where you're like, for the people that need this, it's incredible. For the general sort of balance of attack and defence, this seems extremely useful. And yet it's really hard to imagine how anybody could really abuse it. And that is baked into the artefact itself. Now, when you imagine a general purpose scientist, I'm not so sure that trait is always there. What other projects do you have in mind that have that set of properties?

Like, going to this point, I think that's really the philosophy, right? It's like figuring out what could be the biggest contribution also that like almost has a disproportionate impact to the amounts of compute it takes, right? It's like there's other things you could do with hundreds of millions that have less impact, you know, where it's like really about figuring out what's the lowest hanging fruit for human progress, for this differential progress. I think it even like leads in, like I shared with like different people in biosafety about like the work we did. They're very excited to like open up almost like a distributed pathogen detection and monitoring effort, right? Where you could at some point have millions of places analyze those samples, which you already do, but obviously not on a granular individual home basis, but more on a city basis or airports and things like this. But I think more broadly, I think we basically want to make sure that we build the key pieces and the key research Needed to scale progress radically. And I think basically it's almost like figuring out how to create a system with a human in the loop to do science autonomously or AI research autonomously in a safe fashion. I think also the path to things like superalignment, it's at the core of those philosophies being like, realistically, the only way to solve the superintelligence alignment, or the only reasonable path to have autonomous scientific systems solve, for example, pandemics, Is to have the right guardrails and humans in the loop and just the benefits of open source and a lot of participants to drive them in the best direction. So I think we have basically multiple things planned in this direction, including other scientific foundation models that basically just generally just advance progress. For example, a virtual cell foundation model is one idea, or things like a simple autonomous scientist, where you also have some of those mechanisms in place where humans can interact with AI scientists. But I think those are the most promising paths towards just radical positive human progress. And I think it's like one needs to do this iterative deployment, right? Like where one builds out in small steps and then improves, beta tests it and scales it and then makes it accessible to everyone, right? To have basically the biggest potential impact.

Ready to switch gears to distributed training? We've covered a lot of ground, but now I think might be a good time to just dive into the technical details of that. So again, just to remind myself and everybody, why does this matter? I mean, multiple reasons, right? It's like in the limit, it could create a sort of truly decentralised AI infrastructure that nobody can control. It's also just a challenge to build clusters beyond a certain scale. There's also this spare compute notion, major implications for compute governance. So lots of reasons, I think, to care about distributed training. You guys have an excellent blog post, which is 9 months old, but I still found very well worth reading today. And I think that's honestly a rare accomplishment in AI that you could have something that is still worth reading 9 months later. So I commend you for that. I'd love to kind of just walk through that and talk through the details. And then we can go beyond the time of the blog post too, because you guys have actively contributed to the frontier since then. Well, let's just start maybe with what makes distributed training hard. Why was it ever a question as to whether or not it would work?

Yeah, this is mainly about the bandwidth requirements you have for your usual distributed training, right? Even distributed training inside a single cluster is quite hard to do. So You have to make sure that for every kind of like parallelization strategy you use to like parallelize your workloads across all the GPUs you have in one data center, that you do it efficiently in a sense that it works efficiently for the different parallelization strategies, right? And yeah, some of them have more memory or more bandwidth requirements than others, right? So one of the ones that is also used in our Dialoco technique, for example, is normal data parallel training. where you basically do your forward pass, do your backward pass of your whole model, and then you only send the gradients at the end in a very efficient, like, all-reduce manner across all those nodes in the network, right? And there are other techniques that have more memory requirements. So one of those is like tensor model parallelism, where you basically split up the whole weights of your model across different GPUs, and then you have to do communication between different GPUs for every single layer basically in your, in your transformer model, right? And another technique we also mentioned in the blog post, right, is pipeline parallelism where you have different stages. So you only have to communicate between those stages and on all those stages, there are different blocks of layers of your transformer. So you only have to sync the last activation hidden state to the next stage and to your next pipeline stage, which has less memory requirements than your tensor model parallelism. but still a lot more memory, a lot more bandwidth requirements than your data parallelism. And those are the 3 different techniques you can basically go for. There's a couple more other things that people figured out over the last couple of years, but that's the main one. And a lot of people have been going for, for distributed training across like non-co-located clusters now, this is data parallel regime, which in a normal setting is still way too communication intensive to actually do it. Yeah, with interconnect that is slower than 100 gigs for large models. So even in this regime, you have to find techniques to have less communication between the nodes. And yeah, one of the techniques, uh, yeah, we've been working on quite heavily and have been building on top of is this DILoCo technique by Google DeepMind, Distributed Low Communication Training, which is basically a technique where you train on those different islands of devices in a data parallel fashion, but instead of like syncing every step in your training now, You only have to sync every couple hundreds of steps, basically. So you sync only every couple hundred steps, like those pseudo gradients and this mechanism. And then you have an auto optimization loop to merge those results basically from all those different nodes. And yeah, it basically works as efficient as in a centralized cluster with obviously some limitations in a sense on how many actual nodes you can scale to and where it actually works well. Like at the beginning of the training, it doesn't work as well. But in later stages of training, it's pretty much as efficient as your usual data parallel training.

Okay, let me take that back entirely from the top, make sure I understand it. You can then correct me and elaborate on the things that I'm either getting wrong or simplifying. But I do think this is really important for people to understand. So I guess in terms of just the fundamental challenge of a naive setup, right? Data parallelism is the first kind of parallelism. And that just means if I'm going to train a giant model, obviously, if I want to do, for example, 15 trillion tokens, as some famous open source models have done, I'm not going to do one token at a time. The whole game is right. The reason I have these big clusters is I need to parallelise this out. I can't possibly have one model do a forward pass, take the gradient update, and just keep using this one copy of the model. You'd never get there. Now we can just say data parallelism is basically you make a bunch of copies of the model, you train all these different copies, or maybe I should say you run the forward passes and take the backward pass gradients on all these independent copies of the model, which are each processing their own data. And then the aggregation step is the part where a lot of stuff has to fly around. And I think it's underappreciated that the gradient is basically the same footprint as the model itself, right? Because if I've got 100 billion parameters sitting on a server somewhere and I take a backward pass in the limit, the naive approach is I have one adjustment to each one of those 100 billion parameters. So I have 100 billion adjustments that I want to make to the model. And now if I have 10,000 GPUs wide, each having processed its own data and found its gradient in order to learn from that particular data point that it just crunched, now I've got 10,000 times 100 billion numbers that I need to bring to some aggregate form. So then I have this sort of cumulative gradient from this training step. Now I can go apply that to all the copies of the model, and then I can take the next step.

Right.

And then I'm not sure how much we need to add on to that in terms of tensor parallelism or pipeline parallelism to help people at least develop a decent intuition. Maybe there's more you would add, but it seems like what I just described feels to me like the really fundamental issue. And then tensor and pipeline parallelism also is important because your model is too big for a single GPU, then you've got to start to split your model. And so that just adds even more complexity and overhead. But even if you had giant GPUs that could fit all the parameters onto one, you would still have this data parallelism problem where you're just like, I have to run a bunch of copies and I have to figure out some way to aggregate the changes that each instance of the model wants to make. So that I can make like the global change and the overall model can keep getting better. Pause there. Is there anything more that in terms of like fundamental intuition that somebody who's not going to do this themselves, but wants to understand generally what's going on needs to know?

No, I think that's great. It probably makes sense as part of this conversation to stay in this data parallel regime because the other one is even harder to do in a distributed fashion, right? Like there are techniques for this too, obviously, but yeah, as part of this conversation makes sense just to stay in the data parallel regime and understand that use case. But yeah, as you said, all those models have a lot of memory requirements too, right? You can't train your 100 billion parameter model on a 4090 because it takes too much memory. Even the parameters are too big, right? And especially for the training use case, which people don't always realize, right? You have even lots more memory requirements than for inference, right? So in training, you need your model parameters. You have your gradients, which are the same size as your parameters. And then you also have a huge optimizer state, right? To update your, your model parameters, right? And those kinds of things take a lot of memory, which is, yeah, one of the reasons, for example, in our 10 billion parameter run we've done with Intellect-1, yeah, the biggest thing at the time we were able to fit on a node basically was around like a 10 billion parameter model in terms of memory requirements for training, right? And then doing data parallel training across those.

Basically the. Momentum term helps stabilise training overall, right? You could take the gradients without the momentum term, but then you have convergence problems because everything's sort of thrashing around, and the momentum term basically sort of keeps your individual updates from going too far in unusual directions. But again, you have one of those momentum things to keep track of for each parameter, right? So essentially you have a multiple for however many parameters you have, which are numbers, you have a multiple of that number of numbers that you need to keep track of during the training process and then figure out how to aggregate across. So when you go to something like DILoco, which is the distributed training regime, their training scheme, I guess, maybe that DeepMind put out and that you've elaborated on since then, is it as simple as an insight that actually you can just run more forward passes and kind of keep track of things locally and you just simply don't need to aggregate every step and it still works? Or is there more to it than that?

For sure, it's a very empirical result, right? Just showing that it basically works by doing this. It's also not the first work into this direction, obviously, right? There's been a lot of, the whole like federated learning literature has been doing a lot of things in this direction as well as a general algorithm is more called local SGD type of approaches, right? Where you are able to do those local steps and then soon can have an outer step, basically. So it's just been showing to empirically work really well from Google DeepMind on the, I think they scaled in their paper to like a 400 million parameter model, which is, yeah, in retrospect, not that large for large language models these days, right? We scaled it up to the 1 billion parameter size where it was still working, and then to the 10 billion parameter size, right, with the Intellect-1 run. Unfortunately, obviously didn't have a baseline run to compare to, uh, because at that scale, the costs are a bit too large for like a small startup to actually run baseline operations, right, on those kinds of models. But as a general intuition, you can do those local steps and don't have to sync as often. It works a little bit less at the beginning of the training, uh, which is also one of the things that the local paper showed and we showed later on, right? Where at the beginning, it's very clear the gradients have to move in a specific direction, right? And then the merging of those doesn't really help too much, right? But in the later, very flat regime of the loss, data auger is working really well.

Okay, help me understand that a little bit better because I almost had the opposite intuition at first. When you're learning very quickly, it would seem like maybe you need to share those learnings sooner, but it seems like you're saying the opposite. I don't know, I'm confused. Help me explain that a little better.

At the beginning, there's not much signal because they all go into the same direction, right? And then later on in training, there's a lot of signal coming from all those different data parallel ranks. And then that technique just works better in the, in the later stages of training. It's also very much empirical in the sense that we've seen the local converge a little bit slower at the beginning and then pretty much catch up to your baseline experiments with just doing data parallel training. Yeah. Okay.

That's interesting.

So there's not much major trade-off.

There's certainly still things to solve in a sense, right? Like there are 3 things that we obviously work on since we've been doing Intellect-1, right? For the Intellect-2s to run, like around like 3 major components. For the one part, the communication requirements are still too large for like training a 100 billion parameter model with this technique. We would have to have even less communication and have been working on different techniques there to quantize pseudo gradients and other reset directions. So that's one of the things where it has some limitations still, right? Other parts are just memory requirements. So if you take larger nodes of H100s or multiple nodes, obviously it would fit there. So this requirement is gone. But to actually make it possible so everybody could join at least with like some memory requirement of an A100 node, an H100 node. So there's still pretty much a limitation to scale this one. As well as then on the number of workers. So DaLoco currently still has some diminishing returns in terms of How many workers can you actually scale to? And when you, you sync the gradients after like the 500 steps of pseudo gradients, right? The signal gets lost a little bit if you have too many workers. So what we've empirically tested with Intel Lake-1 run, it was scaling well to like 16 workers, for example, right? But to truly make a more distributed training run happen, we have to scale it to hundreds and thousands of workers, right?

Yeah, that's interesting. So, and the. The intuition there is simply that because these things are each kind of overlearning in whatever random directions they happen to be overlearning, and then if you have just too many of those, the whole thing becomes noisy. And because these overlearnings are sort of happening in various directions, they're kind of cancelling each other out. The signal's getting lost.

Yeah, it's not like it doesn't work at all anymore. So we've been scaling to more workers in a sense in our testing, but it's just not as compute efficient anymore as your usual data parallel training. So that's what you want to aim for. You want to, you can take some trade-offs because in a distributed global setting, the FLOPS are maybe cheaper, right? Because we can run on spot instances, we can have people contribute their idle compute resources and so on. So the FLOPS are cheaper, the bandwidth is more extensive. But we still want to be on par in a sense to how efficient it is to train in a centralized cluster.

Yeah, gotcha. So you mentioned quantizing gradients. I assume you've sort of similar, maybe even the same idea as just saying you could sort of round probably a lot of gradients down to zero perhaps, and sort of say anything below a certain size, we just don't have to send across the wire at all, maybe. What other tools are in your toolbox there that are—

Yeah, we obviously have a lot coming up in terms of improving on that algorithm, submitting a paper there soon. I think what we've done so far is just what we've realized, those pseudo gradients we actually send after those hundreds of steps. So it's not the actual gradients of the model. It's a difference between the beginning of the weights and the end state of the weights after all those inner step updates. And what we've realized is that those pseudo-granules are actually pretty easy to, or well to quantize. So what we've done so far for the Intellect-1 run, we haven't been sending those in like 32-bit precision, but we've been sending those in 8-bit precision, which gives us another like 4x reduction in communication requirements. And for the Intellect-1 run, it was enough to basically sync every 100 steps. And so we had a total communication reduction of 400. X, right? Which was enough to still train very efficiently across the whole globe, right? With the interconnect we had there for a 10 billion parameter model. And if you do more inner steps, basically, find more quantization techniques, different things on top of it, we could probably scale that to even larger models.

Yeah, that's interesting. The other thing that really stood out to me was the DeepPaco scheme where, and this I think is interesting for multiple reasons. I mean, I had previously been kind of studying mixture of experts in general and interested to learn that, just like so many other things, the sort of relationship between the experts is generally very opaque. There's not— you should not model the experts as actually being domain subject matter experts in a recognisable area of human pursuit or whatever. On the contrary, just like everything else, it's like, why did that token get routed to that expert or set of experts, like, we don't know. But this Deepako thing starts to segment data by longer sequences, and therefore it's not like you have to juggle things on a token-by-token level. That seems to me both potentially interesting from an efficiency standpoint, but also from an interpretability standpoint, and potentially on some of these gradient sync questions. I wonder if you were to say, what if we sent all of our science information over to this subcluster and we sent all of our literature over to this subcluster? Would that have a chance at sort of allowing these things to do more locally before they need to aggregate? And potentially, would it also create resulting architectures where there actually is like a separation of concerns, more like a traditional software project and less like, you know, a giant spaghetti black box that we are accustomed to. How much promise do you think there is in that sort of, like, let's say semantic data segmentation sort of idea?

Yeah, great question. And yeah, that's, I think what a lot of people are getting wrong, right? In terms of how they think about mixture of experts, right? That they actually have some semantic routing in there that it actually routes to specific experts for specific domains, which is, yeah. Just generally, it doesn't seem to be the case, right? Just seems to be a more efficient way to compute to get a lower loss, right? So there's a usual mix-of-experts way where you basically, which, yeah, it's been behind a lot of the models, right? Like even the newer DeepSeek models, right? Where you route per token level. So for every token you route to a different expert in the MLP. And the DeepARCO way wants you to allow to actually distribute the whole mix of experts. Which is not really possible if you route to a different expert for every token. So you have to do it not in such a granular way, but you can only do it on the sequence level. So for every sequence, you can route to a different expert, which, yeah, as you said, theoretically would allow for like routing to different experts for actual domain experts in a way. Unfortunately, I'm not too bullish on the whole way of like routing it by the domains basically, right? I think it's just a mixture of experts that doesn't work as well as what we've been seeing with the normal token-level mixture of experts. So, um, yeah, would love to see if somebody wants to replicate, uh, DeepARCO in an open way to, yeah, see more research happening, right? Um, and, uh, see if it works as well to actually, um, do the routing on a bit more of a domain level. Um, but yeah, my intuition is, is not to, uh, yeah, that I'm not too bullish on it.

And is that because you just feel like it's too clever and clever things don't work? Or is there— what motivates? Because, I mean, it would really be in some ways a huge win if you could get a semantic segmentation between experts, right? For multiple reasons. But just on the DeepSeek V3 R1 architecture, if I recall correctly, it's 671 billion parameters.

Yeah.

of which I think 37 are active at any given point in time. So it's like 6% or something of the overall parameters in the model are used in any one forward pass. If you could do sufficient level of semantic segmentation across experts, you could have the sort of virologist removed package that I think That's another way I kind of try to think about squaring the circle. If you could localise certain highly sensitive sets of information or knowledge in the model and then distribute a version that just doesn't have the virology knowledge as sort of the base open source case, that might be a great way to be like, look, everybody has everything you really need and you just don't get the virologist. And you probably shouldn't complain too much about that if you're You know, a normal user. But yeah, so can you tell me more about why you think that's ultimately not going to work?

I think it's got to work in a sense, right? You can definitely do it. I'm not as sure about that it works as well as just letting the model learn, right? So adding like this inductive bias on like, hey, we want to route those to those different domains is probably not as efficient to just let the mix of expert learn on their own where they want to route it to. And just empirically has been shown the Microsoft expert is not learning where to actually, that to route it by a domain, right. But rather in a completely different way that is not as interpretable for us, right. Uh, which is unfortunate for interpretability research, obviously, right. It would have learned it if it would have created a better model. So that's, that's my kind of view on it.

Yeah, I think like to add to it, I think there's almost like 2 things that I think are interesting on the one side that like One can build those architectures that are slightly better set up for like a distributed setting. I think on the other side is that we're obviously entering a new scaling paradigm, right? Like of like inference time compute, which scales much better, right? It's like basically you get much more reasoning for much less compute and it's actually perfectly set up for decentralized training, like almost like by coincidence, right? So I think basically DeepArgo, I think, was in many ways interesting. I think looking into the present and future, I think It's more like the R1 scaling paradigm is actually perfect for a decentralized setting on the one side. I think on the other side, it's like basically has very little communication requirements, right? Because a lot of it is scaling through synthetic data and through inference time compute. And I think on the other side, I think to your interpretability point, like the RL chains, right? Like it's good if you can read them, it's good if you can see them, right? Which you can't for like O or O, right? Even for interpretability and safety and whatnot. And even as an end user to understand how it's reasoning. So I think it's actually quite a, in some ways, interpretable paradigm. And I think that's also what the OpenAI folks try to make you believe, right? It's like it shows you how it's reasoning, right? And you can align and kind of edit out specific reasoning. You can add some things into the reasoning, right? Like some safety checks or some—

Yeah.

General checks. I think on the other side though, to be realistic, right, it's still like a Shoggoth way of reasoning, right? It's like basically suddenly starts reasoning in Chinese, like drops some like random bits that like you also can't really understand. And I think it's just like almost like the emergence or like where it's like scaling laws don't like scale reasoning in the exact same way as human reasoning, right? Like they take some like different paths through almost like this multidimensional space, right, of math, like that is kind of like foreign to us, but it's still probably, there's this patterns and structure in those systems, right? Just like maybe not, they don't map like anthropomorphically, like one-on-one to human reasoning and like language. So yeah, I think that's kind of like the paradigm we are in and the paradigm we are like planning to scale is almost more this like inference time compute paradigm in a decentralized fashion.

Yeah, a couple of interesting points there in terms of just very practically on the why is the RL, like the R1 style training more favourable? I don't know if you have better numbers on this because I don't think it was in the R1 paper, but Basically, you're doing a lot more forward passes relative to the number of backward passes. Instead of having to do a backward pass for every document that gets processed through the model, because you want to initially in pretraining, right, you're doing every— I know you guys understand this. This is more for the audience. Every single token you're kind of able to adjust on here. It's like, did you get it right? Did you get it wrong? take the best of n samples and maybe also the worst, and you only have to do the backward pass and look at the gradient for these sort of extremely good or extremely bad generations. And so that just dramatically reduces the overhead in terms of the bandwidth that's required. Right. Anything else that needs to be added there? Is any ratio in the R1 paper? I don't think it said how many generations they're doing and what their sort of best of strategy is, but— You may know from other sources what is kind of the norm there.

Yeah. Like, unfortunately for this time, like, obviously DeepSeek is always pretty transparent in what they're doing, right? Um, for the R1 paper, there were a couple of missing details definitely on the, yeah, how they actually trained that thing, right? How did the infrastructure look like and how long did they train for? What the actual communication requirements were, right? Um, but, uh, in general, like the, the 2 things that are needed for, uh, the DeepSeek R1 model in a sense, right? Like one of the things they've shown that is that supervised fine-tuning on just a bunch of different reasoning chains from, yeah, tasks and verifiers that you've just generated with a different kind of model is helping quite a bit for like actually getting a better R1 at the end with the reinforcement learning approach that is layered on top later, right? And there's 2 components. First of all, the supervised fine-tuning to generate those large-scale synthetic datasets. is very much favorable for a distributed training setting, right? Or distributed network, right? That you can just generate those reasoning chains. Yeah, for the larger models on like H100, H200 nodes. And then, yeah, you can even like use the smaller distilled models for synthetic data generation. And yeah, just have more supervised fine-tuning data to train on. And this improves the performance of the G3 Gamma model.

Yeah.

And then the second stage, right, where you actually do the, uh, GRPO reinforcement learning on top of it. Um, yeah, they don't have the details there, but generally looking at, uh, yeah, what other have, other people have been doing in the reinforcement learning regime, um, on top of language models where you have a bunch of different rollout phases, right? You generate, uh, for 256 questions, uh, different rollouts, right? Um, and then, uh, only do an update step after you've done all those rollouts or do a backward pass only afterwards. Uh, and, uh, yeah, accepting or rejecting a bunch of those, right? Um, we've seen that you have like minutes of, uh, forward passes basically for every backward pass. Um, or it could be hours if you have a lot of rollout phases. So still very early on there, obviously, but, um, yeah, it definitely looks like the whole paradigm seems to be, uh, much more favorable for distributed training. Yeah.

It also probably is worth mentioning that like, there's not going to be a single ratio of forward passes to backward pass, because you're also sort of interacting there with curriculum learning type things, or sort of what problems are you trying and what is your success rate on them? Because you need some signal, right? So if you're doing problems that you only get right 1 in 1,000 times, then you need like 1,000 to get 1 that's right, that allows you to get the reinforcement signal to proceed. So that's part of why the sampling strategy is important. And they have some interesting details on that. And I think in both that and the Kimmy paper, there's a sort of algorithm that's like trying to make sure that the difficulty of the problem is well calibrated so that you're doing hard enough problems that you're still meaningfully learning, but you're doing problems that are in range enough that you don't have to try 1,000 times before you get it right once. So you can learn from that. So I do think that's important to keep in mind too. So what else? I mean, there were a number of other different papers that you run through in the blog post beyond what we've talked about so far. What else do you think is worthy of highlighting or what from your work since that blog post 9 months ago would you say is what people most need to know about?

Yeah, good question. Think back about the blog post. Uh, like one of the other things mentioned in there is, for example, this, uh, swarm parallelism idea. It's a paper by somebody we've been collabing with a lot too, uh, Max Viabini, who's currently at Together AI. And he's done like really tremendous work in the whole distributed training space over the last 6 years and has been doing his PhD in it. And yeah, one of the techniques he's been coming up with there is called swarm parallelism, where you also not only have your data parallel across the world basically, but you can also do, um, like pipeline parallelism. across the world, which at least for like small sequence length, it's actually not that communication intensive, right? Because you only have to sync the last activation state of your pipeline stage. So you only have to sync something which for a transformer model, but also for all the different hybrid variants that are out there right now, is always of the size, like your sequence length times your batch size times your hidden dimension, which at least for like small sequence length is not that big. Or for also small int size and batch size. So that's something you can also potentially do over the internet, but then you also get into other things to be cautious of, right? In a data parallel regime, you, for example, don't have to think about latency, which in one single data center is not an issue. But across the world, latency is a huge issue. And in your data parallel regime, you don't really have to think about it because you have to do the update steps. So So little that if you have like 100 milliseconds of latency, it doesn't add much overhead. But for pipeline parallelism, we have to do that communication quite often with like very small sizes of tensors we have to send there. So that's one of the things to be cautious of there. But yeah, otherwise it's also a great technique to actually scale that also to larger model sizes, which is one of the limitations we have right now in terms of how much, how large of a model can you actually fit on one node.

So does that swarm type thing start to open up the possibility of truly end user, like retail compute being contributed to training projects in the future? Because my laptop can't hold the full DeepSeek V3 or R1, but it might be able to hold one of the experts or one of the layers. How far from that, or how realistic, or how far from that do you think we Are right now.

Yeah, definitely would be a possibility for smaller models, right? You can't still train a large model across a VIT, have too many pipeline stages, right? As well as then the other bottlenecks you run into with latency and so on in like a home GPU type setting, right? But yeah, theoretically it should allow for it. The one thing that makes me a bit bearish on doing this in a whole distributed fashion is how the current research paradigm has been moving, right? Like one of the things we've been doing is we create ever larger sequence lengths because we need those for the R1-style reasoning components, right? So whenever we increase our sequence lengths, it doesn't make a difference for your data parallel training. So your data parallel training, you still have to only send the gradients, which aren't growing with your sequence length, but in pipeline parallelism, your activations, they grow by the sequence length. And if we go for a million sequence lengths to train something like an OSP type model, your activations are going to be way too large to actually send those. So I think there are better techniques. So one of the things we've been rather focused on is offloading a lot of the optimizer states, for example, on a single node, which there are techniques out there, um, with zero offload and a lot of more modern approaches to make that efficient and train like a 100 billion model on a single node of H100s or A100s.

Can you describe offloading a little more? It's essentially moving the momentum terms off of the high bandwidth memory to other storage?

Yeah. To your storage on your node, basically, right? To CPU RAM, right? So that's how offloading works in a sense. And you certainly want to offload the optimizer stage because that takes the most amount of memory, right? Like we touched on it earlier, in your usual, like, mixed precision training, you have your AdamW, right? You have your parameters, you have your gradients, which are stored in mostly 16-bit precision, right? But your optimizer, you have to store your parameters, a copy of it, you have to store in full precision. You have to store your gradients again in full precision, a copy of it. You have the momentum, as you mentioned, and you have the variance also in full precision. So it takes up. like way much more memory than your parameters and gradients. So if you're able to offload that efficiently while still keeping really good training efficiency, that's probably the way to go. And then you can fit like some of the largest models basically on a single node. Obviously you still won't be able to train a DeepSeek R1 with 650 billion parameters on it, but at least to go into the 100 billion parameter regime.

One thing I've been thinking about A decent amount. I'm sure you've seen this blog post from Yi Tei, who was at Google and then left to go start Reca and is now back at Google. I thought the blog post about buying CPU or buying GPUs, I should say, in the wild was a really illuminating one. And basically the long and the short of it was like he was saying, I had no idea how spoiled I was at Google because All the infrastructure there just works. And then you get out into the real world and you're trying to buy from this cluster or that cluster. And this was like a year ago, so you didn't exist yet, maybe. And there was also just a more sort of frothy time in general in trying to buy compute. And also people hadn't worked out problems nearly as much, but it sounded like a real jungle. At what point, if ever, do you expect us to kind of be in a similar place with GPUs as we are with CPUs, by which I mean the hyperscalers years ago now, right, basically figured out how to sort of abstract away the differences in underlying hardware and create consistent containers that they don't really have to worry too much any— there's enough layers of abstraction, they don't really have to worry exactly what physical device is this running on. They just kind of know, the virtual environment that it works on, and that can be consistent, and that's good enough. And on top of that, they have the sort of fault tolerance of, like, one computer craps out or whatever, somebody spills a drink on it, it doesn't matter. They just kind of route around those problems. And that's great, right? That is why we have all the tremendous uptime that we have, so on and so forth. It strikes me that we're not there with GPUs. Is it like, just because software takes time to mature and we're just not there yet and it will get there? Or is there something more fundamental that I may not appreciate? Because then I've heard these, in contrast, I've heard these stories of one GPU goes down and it stops your training run. When I hear that, I'm like, wow, that's really weird. That's a dramatic departure from 10 years ago. Large-scale web products were supported, right, with Facebook 2013 or 2014, 2015, I think was already starting to get pretty far down that path. So what do you make of that? What should I expect in the future?

Yep.

Yeah, as you've mentioned, fault tolerance is a huge part there, right? And it's definitely on the software as well as hardware stack right now that it's not perfect yet in the whole GPU regime, right? If you train across like a, or like do a training run across like 100,000 GPUs and then a single cluster, for example, like the, I don't know, Groq XAI cluster or something like this, right? There will be a node that fails every at least couple hours, right? Maybe less. So right now, a lot of the current training frameworks are not actually made for fault tolerance. So if training run is just going to crash too, there's one node of those thousands of nodes that are going to crash. And yeah, you have to continue your training from the last checkpoint, for example. And yeah, all the big labs, a lot of them have it figured out. The other ones that don't have it figured out are racing to figure it out right now, because otherwise they won't be able to train their models across the largest cluster. And in our case, it's even more crazy in a sense, right? We definitely need fault tolerance. We need even like more than that. We need onboarding of different GPU nodes during the training. And yeah, that's what we've been, we've done with the training framework called Prime. We've also open sourced that, where we have this fault tolerance. So all the Dailoco ranks, all the Data Parallel ranks we have in our training. fault tolerant, whenever somebody drops off, the training is not stopping. You can still continue training. And yeah, that's why I'd like also all the big labs who are interested in like solutions there in a sense, right? How do you actually make this fault tolerance happen? And yeah, over the long term, I think it's going to be fixed obviously on the software side, as well as on the hardware side. We're going to go more into like actually a commodity for compute, which is definitely not the case right now. There's Yeah, you've mentioned the RECA paper where they basically showed there was like a 100x difference in terms of different GPU providers for how reliable their GPUs were, which I don't think, uh, it's definitely, there's probably also always going to be a bit of a better provider, but it's definitely going to turn more on a commodity that you can actually just run on this compute for tolerances being handled and hardware is going to improve as well.

When you do something like the 10 billion parameter model that you have trained in a distributed way. How much of your time is going toward managing that mess and making sure that it works with heterogeneous hardware and things coming online and offline? And do you have an SLA sort of agreement with your providers that they're not just going to yank things out from you? Or are you giving them the freedom to say, no, these are preemptible nodes? Like, We'll take what we can get as we can get it. How much of, you know, so yeah, a little more there and how much of your time and how much of the return comes from that kind of improvement versus like the sort of de loco, like more theoretical information management kind of improvements?

Yeah, I think for the initial like Intellect 1 run, it was a lot of improvements more on the hardcore engineering type work to actually get that done, right? To get fault tolerance into the framework. framework, as well as then covering all the other edge cases you have, having stuff like that you can have like a live recovery where somebody can drop off, but they could also join again or yeah, just join in the middle of training run, right? Which is definitely not supported in any of the other training frameworks right now. So our focus was definitely more on the engineering side. And to be fully transparent, there was still at the beginning a lot of things going wrong. So where we basically had to like fix stuff on the fly where a lot of the edge cases we haven't thought about, right? Which can actually go wrong in such a globally distributed setting had to be fixed on the fly and the training stopped for a couple of hours and then we continued with all the nodes then later on. So that's basically how we've been handling for the Intel Lake 1. And I think at the end of the training run, we, we had a pretty stable solution in a sense, right? That a lot of the components were fixed. I'm sure once we start on Lake 2, there's still gonna be some of the issues where training runs have to be restarted because of edge cases. So you can never be 100% sure. But yeah, that's how we've been handling it.

That narrative strikes me as a decent metaphor for the AI phenomenon as a whole. It's like there's a lot of edge cases we haven't anticipated, and we may have some crashes, and hopefully we'll recover, and hopefully we'll get to a stable equilibrium at some point. I guess let me try to summarize. Where are we on distributed training? There is a mess of hardware management, which it sounds like you've come quite a long way toward taming, but maybe not fully tamed yet. There's also just the fundamentals of information management and the challenges of bandwidth. It seems like the low-hanging fruit has been picked where it was like, actually, you could just do this in a less bandwidth-intensive way and it still works. And now we're getting into things where there's probably more tangible trade-offs where, yeah, you could do this, but then it's not going to converge as fast, or we're not necessarily competitive if we make certain trade-offs. It seems like we're, and you know, in terms of scale, you've got the $10 billion. It seems like you can see a path to $100 billion, but a path to a trillion maybe is not yet clear. Is that a decent summary?

I think there's still a path to almost like AGI and even like superintelligence, like in this paradigm that almost like goes beyond purely just maxing out parameter count, right? Like where I think even the R1 shows us, right? It's like, Scaling that paradigm might get us to AGI, and it might not require trillions of active tokens or parameters. And I think that's, for me, quite a plausible path where it's like there's other pieces of the puzzle that are quite relevant to solve. And I think we're largely on track to solve decentralized training. It will always have like some trade-offs, but that come also with huge benefits, right? It's like, because you can train across the whole world, You're not reliant on like drawing absurd amounts of energy in one place and having like an absurdly large cluster in one place, which you can see play out in real time, right? Like with all the big labs, which is also why they're all extremely interested. Like a bunch of them reach out to us, right? It's like, because it's the key thing that they're trying to solve, right? Like distributed training. So I think in that sense, like it's actually something which even the big labs and everyone basically is tackling. Like, it's not just OASIS-Fringe, like decentralized AI, decentralized training ecosystem. It's OpenAI, it's Gemini, it's all of them, right? And I think that's something which people haven't fully appreciated yet about distributed training. They obviously, as Jonas also mentioned earlier, have like a slightly different paradigm, right? Like they in some cases have very fast interconnect between their clusters. But it is surprising. It's like, even if you look at At OpenAI, it's like, it's not like they're next to each other, right? It's like the plan is like they're actually spread throughout the US, kind of like the data center build out, like that they plan.

You need decentralized training to win over Congress to fund the massive build out because they all need a little, a little of it in each of their respective states and districts.

Yes. And I think on the other side, this also unlocks in just like global communication, right? It's like having fiber like under the Oceans, right? Like connecting the continents, like having Starlink, like all of these things, like opens up theoretically even like compute being in the middle of the ocean, right? It's like if you suddenly can communicate and it's like we had people doing exactly this, like reach out to us because the cost of energy is so much lower if you have like wave energy in the ocean. It's like suddenly basically like a different, almost like even infrastructure paradigm opens up where you don't need the compute necessarily in like the different hubs. But you can have them in extremely rural areas as well. And I think that future is playing out. And I think the question is probably how fast that infrastructure buildout will continue, right? It's like, it's at trillions per dollar. It's like now, I think it's probably gonna get like to the tens of trillions. But Microsoft alone is spending $80 billion a year, right? Like on data center buildout. OpenAI, obviously now like $100 billion plus. And I think There's a lot of others that are underreported, right? It's like the governments have gigantic infrastructure buildouts that are very underreported, including the US government. So I think there's much more compute already being built than even people know about, than is reported. And there's way more being built that it would be hard to even fully have an overview of, right? DeepSeek is the best example, right? It's like there was yesterday a saying by Scale found out that like supposedly they have like 50,000+ H100s, even though they underreport the number because they have export restrictions, right? So there's all of these things in place where it's like, good luck getting the actual number of what the Defense Department is right now building up on the GPU side. I know it's like in the 6-figure, like hundreds plus, I guess probably in the 7 figures already. It's like gigantic clusters like that no one knows, like that are very unknown. And I think that's playing out for China. It's playing out for like a lot of different places where like Basically, there will be specific spots where there will be also an explosion just in compute, which we'll tap into, right? A good example was like TikTok was one of the biggest providers of compute capacity to a lot of these different platforms and ecosystems because they bought so many GPUs, like I think over 600,000. And at times they didn't need them, right? And they gave them to the market basically. And I think those things will continue to play out as basically As like every corporation and every nation state starts ramping up their like data center buildouts.

It's gonna be quite something if seasteading becomes a thing because people wanna compute in the open ocean for like wave energy. You know, hey, we may be headed to the cyberpunk future.

Yeah, I think more what is like the energy there is already so much cheaper, like out of magnitude, that it will kind of like help me even from that angle, probably to some extent. Yeah. So I think it would, like, compute will be to an extent everywhere where it's like allowed and has like geographic advantages, right? Like where it can tap into cheap energy, where it's like easy to maintain.

Yeah.

With Starlink, you could even have a decent, you know, it sounds like you could have a little floating compute. You talk about islands of compute. I mean, you're, you could be really talking true genuine islands in the middle of the Pacific of compute. That would be quite something. to contemplate. What does this mean for Nvidia? I mean, this is not a stock show by any means, but I looked at Nvidia, which of course has blown up, and as of today, $3.6 trillion market cap. And then just for one comp, I looked at AMD, which is actually down a third over the last year or so and is a $200 billion market cap right now. So you've got an 18x ratio of Nvidia to AMD. I wonder what you guys think of Nvidia's prospects for continued dominance of this market. I mean, a sort of naive read of everything that we've talked about would be the sort of abstraction is coming. And if abstraction comes, then that would seem to be almost synonymous with not the kind of moats or profit margins that Nvidia enjoys today.

I think 2 or 3 things can be true at the same time. Right. I think one is that This market is exploding. So I think it was like nonexistent basically 5 years ago. And now it's like in the trillions and we'll go to tens, probably hundreds of trillions, right? Like, I think it's hard to fathom how gigantic this market will be in a sense where it's like everyone in the stack is poised to be like one of the most valuable corporations in history, right? It's like be it Nvidia, ASML, TSMC, AMD, and everyone else. But I think On the other side, it's like, I think it's also very plausible, right? Like that kind of the monopoly really that NVIDIA holds right now, right? It's like they make 100, like over 100% of the profits, right? It's like in the AI industry right now, because everyone is burning through money, right? Like while they stack money, I think their margins and they're like basically having like majority of like AI profits flow to them. I think that will probably over time shift slightly away, right? It's like their margins might compress while their revenue is like 10 to 100x, right? So it's like, They'll probably still be a bigger business, right? Especially in kind of like the optimistic scenarios. But like, the thing is, I think AMD hasn't been the strongest competition, but they're obviously the clear number 2, right? It's like they fumbled it famously on like a lot of different fronts, including on software and also not really building up the ecosystem that I think Nvidia built up. I think my other, like, almost like read of it is that at some point, like, Nvidia has such big free cash flows that they can like reinvest them into like the next generation of chips. And ultimately extend their lead. I think, to be honest, their strongest competition, which is also fumbling it in like a different dimension, is Google with TPUs, which obviously also like has some very like close to their chest internally. Obviously the other big techs like the Amazons and Apples, they won't start adopting Google TPUs, right? But they're also doing their own efforts, right? It's like Amazon has their own chips, Apple has their own chips. So I wouldn't actually underrate almost like the chip efforts from the big tech giants, like more in terms that they can force adoption and everything else. I think basically like AMD will also go from close to 0% market share to slightly higher, but like probably it won't overtake Nvidia anytime soon. And I think there will be, there's other more specialized chip providers, right? There's specialized ASICs just like made for transformers, like the Groqs and Edges of the world. I think they also are basically at close to zero market share right now, but they'll probably gain some market share similar to how like a lot of the other chip providers will. So I think over time, like zooming fast forward, I think Altium will be probably as a class of companies be worth more. Like the margins will be compressed, but the revenues will be way larger. And I think there's hopefully basically more competition to your point on like the software stack, which we are planning to contribute to, right? Like to help commoditize those efforts in a sense where basically the margins ultimately, I think in a perfectly efficient economy, right? It's like, get compressed to the kind of cost of capital with like some risk premium, right? And I think that's hopefully even with like superintelligence in the long run where we're headed, right? It's that they're actually over time, like their margins is opportunity of superintelligence to compress them with like cheap alternatives, right? Over time. So I think that's like, I think the high-level read of it is that they were able obviously to stack those moats, right? From hardware to software, to the interconnects, to like vendor relations and all of these other things. Which is like not easy to break, at least right now, for like, I would say pretty weak competition from like AMD and even weaker from Intel, right? Like they're not even in the race at all.

Yeah. Elon might have something to say about that, I guess, from what I hear recently, but time will tell. Are you familiar with Imad's Intelligent Internet Project? I did an episode with him not too long ago. Of all the other things I've seen in the world, I would say you guys Like vibe match most with him. Are you in touch or are there other things?

We are quite close with him. I think still in, in my chats, like I can't fully wrap my head around yet what exactly he's planning to do, but it's, I think it's also evolving and it hasn't fully launched yet. But I think maybe some differences, right? And, and then some similarities is we are focusing a lot on peer-to-peer compute and intelligence as foundational. pieces of the puzzle. So I think that that's like a difference. I think the other is, as far as I know, ultimately we put a big focus on decentralized training, right, and fixing and making that work. He seems more interested in like slightly different paths or like kind of orthogonal to it. So basically I think it shares maybe like almost like broader vision, but I think the way how we get there is like slightly different for us. But then also I think we really try to obviously compare to not just him, but like a lot of the efforts, just like ship in like small increments, like as quickly as possible, like different pieces of the puzzle and open them up to the open source community. So I think he'll do the same probably soon, like looking at his work before with Stability. So I think he definitely knows and has a track record of building relevant open source models, right? Which isn't the case for others in maybe entering decentralized training or decentralized AI. So like, I do think he'll be able to create some cool open source AI models and create interesting community, which he obviously did quite well with Stability in many ways. So I think that's probably like superpower that he'll continue to play into. And yeah, we're quite collaborative with him and in touch, but I think it's like the same case with a bunch of other efforts in the space where we're just trying to figure out how they can support each other. It's like the different use cases where people can build agents, they can build their own models on top of us, or Emad's network can like, needs compute and like it can draw it from our network. And I think that's kind of the future that I'm envisioning is that similar to what we've seen with something like Ethereum, where it's really the big power and why it ultimately, I think, won in many dimensions is that you have this basically like co-ownership of the whole network and protocol. It's fully open source. You have thousands of people collaborating, having strong incentives to make it successful instead of just like 5 or 10 team members and a few shareholders. Like it's basically a paradigm where ultimately It's kind of almost like the open and decentralized community competing against the closed source labs, I think. And that union basically of players is growing, including some of the foundational open source pieces like PyTorch and the LLaMA community that we're very open and in close touch with. So I think all of those things can stack on each other and build on each other. So like he can leverage our progress on decentralized training and same for everyone else and the other way around. That's really, I think, basically the strong tailwind that space has. It's like we can build on our own and those things. And I think that's really how the most progress will happen in a very short period of time and how I think open source is now officially caught up with closed source since our own. It closed the gap, right? It's like most closed source companies like the XAIs and Anthropic are like now officially behind open source. And I think that's like an update that people, that shifts people's perception. Like, I think a lot of people didn't think it was possible like even a year ago or 2 years ago. And now it's kind of like definitively proven that like OpenAI's lead like melted towards zero, right? It's like their 1 or 2 year lead is now like gone.

Certainly down to months, not years. I would still say, I think, I mean, it is tricky because I have had really good experiences with DeepSeek. It's only been a few days. A lot of people seem to be very enthused about it. I still think if you could only give me one, I would take O1 over R1. And certainly we know O3 is coming too. So there is something there. I would also bet that internally at DeepMind and Anthropic, there is something at least on the R1 level, even if we haven't necessarily seen it. We have seen Gemini flash thinking, and Dario's recent comments about what he's seen internally at Anthropic certainly suggest they've got something cooking that's moving that direction. In terms of the long-term vision or sort of structure of your company, in a way, kind of going back to the beginning, you guys have these compute partners that contribute to the marketplace, but then they also can contribute to these decentralized training runs that you're doing. Are they just contributing in kind? And what is the governance model? Or I'm a little bit fuzzy on what's in it for them. What is the sort of incentive? design that you've done to bring people together and create sustainability on that level?

I think there's different phases to it, right? And I think up till now, like everyone contributed in kind and ultimately was like more like friends and sponsors, right? Like basically like the folks like Hugging Face or Dylan Patel from Seminariuses, but also just like a bunch of like in the internet, right? Like this model and joined it. I think Basically, almost like again, drawing the analogy to something like Ethereum, really the goal of this is to create like a public utility that like anyone that contributes to it can have a piece in and can direct it. And I think we're basically in the process to like finalize the details and work this out. But like it will have like a nonprofit foundation structure that governs kind of basically almost like this provisioning of this public utility. And I think It's in many ways very counter to just like pure startup corporations, just in structure, right? Like it's basically setting up to provision like a utility in a very efficient way, but also in a very open and decentralized way that is permissionless. It's to your point, it's like Vitalik can't stop Ethereum, right? Like even if he wanted to. And I think that's kind of like by design, right? And ultimately, They're not the only one contributing to it. There's hundreds of teams contributing to something like this infrastructure, like everyone using it, everyone like making pull requests to it. And I think that's basically what we are setting up for. And I think that then basically the structure is like that we as a company almost are one of the contributors to that like public infrastructure and utility. But I think it's very, very different from like a traditional corporation in that sense.

So do you envision like a token or a currency of some sort where I contribute compute, and then I get some sort of claim on future governance that I could potentially even resell?

The broader goal, right, is to like create like an alternative system in that sense, like Ethereum, that is like fully permissionless, fully like tokenized and everything. Like, so yeah, in this direction, even though like it's very like very, we are very light and can't also comment too much on like the concrete timeline and everything. Basically, it's a broader question and discussion where it's like almost how economies, right, and even currencies reshape in kind of like this like superintelligence age. And I do think the default outcome, right, is like almost a continuation of like fiat dying, like to an extent, or being like at least Overly inflated with UBI-like measures, right? Which can potentially be funded by the windfall from superintelligence, right? But still means that ultimately people want to flee into less inflating, harder assets. And I do think a currency that is backed by compute is way harder than a currency, like a fiat currency that's backed by nothing. And also then like a cryptocurrency that is backed by even something like Bitcoin. If you really look at it, like, it's not that hard of an asset and currency, right? Like, you don't get much utility out of it. So I think that's kind of like a bit how we're thinking about it. It's like in the intelligence age almost, you want to own a piece of a super intelligent system that is able to generate value where you have actually hard, like, access, like, through your ownership in it to the compute, to the intelligence. And how that exactly can be built, right? Like, and how we are approaching it. We'll like share more about soon, but ultimately I think it basically needs to happen in stages where it ultimately really guarantees also that like anyone like can get involved and has a piece of it, but ultimately where it's also truly permissionless, right? And like in the spirit almost like of the early like cyberpunk internet and not like of the late stage like platform monopoly big techs. And I think that that's really the broader spirit is like making sure that, you know, we lost kind of like the internet to big tech platform monopolies. I want to make sure that we don't lose like superintelligence to them again. I think it's kind of like the default outcome if we don't kind of like veer off in an alternative path.

I think you just brought us to a perfect ending point with that comment. Maybe one little extra bonus would be, what are you guys looking for? You know, that could be Hiring on your team? I know you've got a jobs page with a number of listings. It could be compute contributors or anything else, but what sort of bat signal do you want to put out into the world?

Yeah, appreciate it. Like, I think all of the ones you mentioned, like basically we're actively hiring across like a lot of different roles, like a lot of on the AI research side, on just like general developer side, but even on things like marketing, design, and every other front. So like for anyone that is interested to reach out. Like, feel free to do so. I think similarly, like, with the model collaboration we did, which we didn't touch on, quite actively collaborating with like dozens or hundreds of like different open source AI researchers, like leading scientific institutions, leading like AI labs at universities. So basically we're quite supportive of basically supporting the highest impact initiatives basically with compute, but also with hands-on support to actually build and scale their models. So that's another area. And then, yeah, in general, always happy to connect with anyone interested to like go deep on this and connect or collaborate with us. And there's a lot of surface areas for collaboration, as you also mentioned, for people that have either a computer and want to contribute to the models that they are, or the agents that they're excited about. Like, that's kind of like really what we're planning to enable.

This has been excellent. Vince Weisser and Johannes Hagelman, founders of Prime Intellect. Thank you both for being part of the Cognitive Revolution.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.