The Most Nonconsensus Approach to Training Models | Pluralis & Topology: Mined with CoinFund Ep. 24

I think there's sort of two buckets of decentralized training. You have data parallel decentralized training and you have model parallel decentralized training. And I think one of the things that happened is those two approaches have been conflated under the same term of decentralized training. And my view is that, you know, data parallel decentralized training, this was a known thing. You know, if you, if you look in the federated learning literature, there's people who've been doing this for 10, 20 years. There's a range of methods there about how you can approach communication efficient training in that setting. And it's model parallel. That is the real challenge. That's the thing that I think today I really want to emphasize how non consensus it is. No one thinks that's feasible.

Hello everyone, and welcome back to another episode of Mind with Coin Fund. For those of you tuning in for the first time today, my name is Jake Bruckman and I'm Coin Fund's founder and CEO and managing partner. And we're going to keep the intro short. It's just me hosting today's episode, but this is a conversation I'm really looking forward to having. If you've been a longtime listener of this podcast, then you know that we've been discussing decentralized training in depth. Decentralized training, a non consensus approach to training has sparked a wide debate among AI developers, researchers and investors. But in the past month, with the rise of open source AI app Deepseek, the idea that open source AI was just starting to perform as well as frontier models, or almost as well as frontier models, is starting to become more of a reality. And with that said, we're really excited to have two, you might say, experts in this field on the show today. Please welcome coinfund portfolio company Pluralis research founder Alexander Long, Anthropology Ventures founder and managing partner Casey Caruso to the show today. Alex and Casey, great to have you here.

How's it going?

Good, good.

Thanks for having me on.

Awesome. Do you want to. I know I gave you like a super tourist introduction. Kasey, do you want to give a little intro to yourself so people know you a little bit better?

I'll do 30 seconds. Yeah. My name is Kasey. I'm the founder of Topology Ventures. It's a frontier tech venture firm. And we invest in traditional AI, traditional frontier tech, as well as crypto and the intersection of crypto and AI. So things like Alex.

Awesome. And then, Alex, do you want to say a few more words about your background and how you came to web3ai.

Sure. So my background is all trad AI. Before pluralis, I was working for Amazon as an applied scientist. I was working with a bunch of. Bunch of my colleagues who are now at Pluralis with me. And then before Amazon, I did my PhD on reinforcement learning and non parametric external memory and deep learning. Awesome.

Welcome, guys. All right, so let me just start by getting out of the way the thing that we always ask guests on the show, which is what's that one thing in crypto that nobody is talking about but more people should be paying attention to. Why don't we start with Casey?

I don't know if this is a cop out answer, but it's got to be decentralized training. I mean, that's what we're here to talk about today. And I feel like there really is this belief and I feel that because I was first a non believer and I've been convinced that people are. Most people are where I am right now. We're just saying it's never going to work. And I think it. I think it really might. So. Yeah.

Wow.

Okay, we have to talk more about that, but first let's go to Alex for his answer.

Yeah, that might be shocking that I agree with Kasey on this one. I think training is very interesting.

Surprise.

Okay, guys, so I think we are establishing that decentralized training is a contrarian position. Kasey, can you say more about that? Like, why don't people like this idea or.

Yeah, yeah, people don't like this idea because they think that basically the communication overhead and getting it to work is too high. And what that means is, like, for people. I don't know how technical this audience is, Jake, but basically for people who are less technical, when you're training a model, you need to do something called gradient descent. And what that does is basically minimize the loss function where like weights or parameters are optimized, where the predictions are as close as possible to what they're trying to accurately predict. And so that process is really hard. And gradient design takes a lot of work. And if you decentralize that, the belief is that you can't do it basically efficiently enough for it to be worth it. That's like a very high level overview of what we're talking about today. But I'm curious if you both agree with that distillation of the issue.

What do you think? I can go right into the hot takes here. Right. I think there's sort of two buckets of decentralized training. You have data parallel decentralized Training and you have model parallel decentralized training. And I think one of the things that happened is those two approaches have been conflated under the same term of decentralized training. And my view is that, you know, data parallel decentralized training, this was a known thing. You know, if you, if you look in the federated learning literature, there's people who've been doing this for 10, 20 years. There's a range of methods there about how you can approach communication efficient training in that setting. And it's model parallel that is the real challenge. That's the thing that I think today I really want to emphasize how non consensus it is. No one thinks that's feasible.

Alex, maybe it would be helpful if you could just do one sentence functionally on what is data parallel and what are the limitations of it and what is model parallel?

Yeah, so to step back and set the scene in data parallel, when we're training the model, we're putting copies of the entire weight set on each node. So if you have 10 participants, they're each getting a copy of this model, they're doing a forward pass, they're computing a loss, they're computing a backward pass, and then on some time interval, they're sharing those gradients they've calculated. And there's a synchronization step. That's the data parallel setting. In the model parallel setting, you're splitting the model itself. So if you had 10 participants, you would suddenly have, okay, participant one has layer one, participant two has layer two, sort of sew through the model that way. And the reason that's so much more challenging is if those participants are separated geographically, it's not just a single communication step at the end of training, it's multiple communication steps during training and the backward pass. And that's why the communication efficiency is just so much worse in that setting.

So are you saying in data parallel you need communication efficiency or you don't need communication efficiency?

You still have the problem. The problem is just significantly less pronounced. So you would do a forward step, a backward step, maybe you'll do multiple. And then you would sync the gradients, you know, periodically. And when you sync the gradient, that's when it's expensive. In model parallel, it's all the time.

I think also we should explain who's using data parallelism today and kind of like what the. Today? Yeah, like when we talk about that, this is how the big players are doing it today. And I think we should explain that.

Yeah, there's kind of this rung of parallelism strategies that go up in complexity. And you can put it's probably four major groups. The most simple thing is data parallel. That's what we've already talked about. And then there's the whole other bucket which is the model parallel methods which splits into sub buckets. In model parallel you have pipeline parallel, which is the setting I described before, where you split the thing depthwise. You have another class of things called tensor parallel, where a very rough approximation is thinking about split the model width wise. And you have other approaches, expert parallel, fully shot of data parallel, these kinds of things. And you kind of move up this rung of complexity. And in the centralized case today, all these methods used, they're all combined when you train these large models.

So Alex, it sounds like in data parallel if every node has a copy of the model, it sounds like the limitation is at some point the model gets too big for a given node. Right. Is there such a thing? Can a model get too big to be held in one node?

Yeah, I mean this is the history of standard centralized training as people start with data parallel and the models get bigger. Okay, I've got eight page one hundreds training this model. It still doesn't fit. What do I do? Okay, I have to now split the thing over my devices, so. Exactly right. That's the limitation.

Yeah, Memory, there's not enough memory.

Okay. So when we go from normal data parallel approaches, which is what I guess big tech companies do in their distributed data centers, when we go from that to decentralized data parallel approaches. So we've seen a few things like that on the market, like Distro Diloco, you know, Prime Intellect has launched a few models. Noose is talking about launching a few models.

Right.

What are we gaining or like what's changing as we go from kind of the classical data parallel method to the decentralized.

Yeah, I should say as well, like the centralized case, they're using a lot more than just data parallel. It's data parallel plus all those other things I mentioned. The kind of thing is you really have to, you have what you can do within a node where you have very fast interconnects and then you have what you can do across nodes. That's what changes in decentralized case. You can still have a big node locally where you can do things like FSDP inside the node. You can do things like tensor parallel because you have those devices are physically co located within the node. But the difference is when you go to centralize, the nodes themselves are split geographically rather than. Okay, I have my H1 hundreds in this data center. Right next to it I have another rack of H1 hundreds. It's rack one is in Australia, rack two is in England. And these have to communicate over very, very low bound communication.

So Casey, when you were saying that a lot of people in your position, your colleagues, your investor investment colleagues, when they're not believing in this, in these kind of more decentralized approaches, is that simply because they have never heard of Dyloko and Distro or some of the things that Alex is talking about or they don't think that those things serve the purpose that they think they should serve or how does what's causing them to be skeptical?

A few things. I think the first thing is the utility of it where like right now if you just have co located training, it works really well, it's super efficient, blah blah, blah. So there's like this belief of like what are we gaining even and is it worth it, the research effort? And then I think the other thing is like what Alex is doing is not what everyone else is doing. Like he's not just trying to, you know. Yeah, like leverage either pipeline parallelization or attempted parallelization. It's not just that like the other part is keeping the weights in a way that not every node has all the weights. Like that's a really, really important part of this and that's independent from what we've discussed so far. And that's like not really technically been done before, at least I, at least that I know of. And that is actually like to me the unlock on what happens after or what again the use cases are of this decentralized training. Because for a while the reason I was skeptical is that's where I was stuck. I was like, I don't really believe that this market is going to march towards this because it's not efficient. And I do believe that markets generally march towards efficiency. But if there's a high enough value proposition, then markets change. And so does that make sense Streak?

Absolutely. I mean it makes me think of like just to get back to our roots here a little bit. It makes me think of bitcoin. I remember like a lot of people saying, well like why would you want to use this type of money or this type of asset? It takes up more electricity than the country of Ireland, you know, and it's not backed by a military and it's not actually worth anything and yada, yada, yada. Right. But I think what it buys you is that it buys you a way, buys you a public Good. That implements money or gold or something like that. Right? Like an asset that's not intermediated by any particular person, place, government, corporation and entity. And so then maybe to your point Kasey, the question is, is that worth doing in the case of AI training? Like do we want to have a public good on rails by which people can go and train AI models in the same way that bitcoin is not intermediated in a non intermediated way?

Right.

So what do you guys think is the answer to that question?

Alex? I mean I think it's really obviously clearly yes. And the question is just can we do that?

Right?

If we had a way to actually train these things in collaborative swarms, if we had a way to sort of reduce this dependency on the megacorps releasing these strong base models, I don't think there'd be anyone who would say oh that would be completely pointless or that's not useful to me. All the risk is this, can we do that? That's what has to work today. Otherwise we're stuck in a situation where you basically have, you don't have open source AI today. Today you have AI that is dependent on the megadorps sort of being benevolent and releasing strong weight sets. The question in my head is if there's no llama 4 or if there's no deep seq v4, if these people decide oh we're going to stop releasing these models, what does anyone do? There's no way to collaborate, train a big model. There's no strong guarantee of a strong base model. So I think it's less of a, like is this useful? It's more of a we have to get this to work.

I just wanted to clarify like I completely agree with Alex said but just for people who are less familiar with this, basically what Alex try to say is like right now open source is on the, you know, held up by Mark, by Meta and like in other, in some other players now.

Right.

But like if that stopped, if that support stopped, we'd be kind of screwed. And then you know like these developers don't have the resources to train these models obviously like the amount of capital and Deep seq we'll get into this like Deep Seek is starting to challenge us that the environment is changing really rapidly. But in general, I mean like devs can't just spin this shit up. So we're in a tricky situation. I want to say one more thing too which is that I do think like this conversation isn't just for crypto. I think decentralized training is really relevant for traditional AI too. And a lot of my traditional AI friends, like, this is the only part of crypto AI that they're not like running out of the room about. They actually want to have this conversation because they think it's really interesting. And topology backed, this company called Central Access, and what they're doing is doing software optimization for data centers. And they estimate that 30 to 40% of the cost of running a data center is basically cooling and keeping the infrastructure in place to do what it needs to do. And if we move towards decentralized training, that entire OPEX would be lightened by a order of magnitude. I mean, maybe even go close to zero. And so I guess I do want to talk about the crypto part. And I also think independent of crypto, this could be huge for AI.

Right, nice.

I could go in three different directions right now. But before I do that or do one of those three directions, I have one more question for Alex about model parallel, if that's okay. Alex, is it the right mental model to say that model? Because model parallel is basically like linear scalability for decentralized model training. Like if you want to train a bigger one, you could just add more participants and more and more and more.

Yeah, if we're talking specifically about pipeline parallel, which is one of the sub buckets in the model parallel sort of group, yeah, you can just keep adding devices and you can make the model arbitrarily deep. Okay, nice.

All right, so we Talked about Deep Seq. We talked about how Web3AI might be actually solving real problems for AI in general, which I love that topic. There's a lot to cover there. And we talked a little bit about maybe the social convergence around that from traditional AI. But let's take a minute on deepseek here. I think deepseek basically is showing people that this narrative that has been over and over with high confidence by people like Ilya Sutskever in public settings, by people like Sam Altman in public settings who say it's pointless to compete with us. Proprietary models will always be always ahead of open source. I think that narrative is being smashed. It used to be that there was a trail time of two years before something open would appear. Then it became six months, then it became two months. Now OpenAI launches operator or deep research, and almost instantly there's GitHub repositories that are replicating this functionality. Now they're not perhaps overtaking the quality yet. And the models might not be as powerful yet.

But.

But what deepseek is showing is that an open model is essentially on the cusp of overtaking it. And in another couple of years, hey, maybe like OpenAI is going to have to follow up a week later with something right versus something open. But what are your guys thoughts on that? Like how good is actually deep seek and is it overtaking proprietary stuff? Am I making this up or what do you think?

I mean it's definitely inferior to 01. I think it's really good. The team's really strong. Obviously the things they've done are really good, but it's not better than O. So we're not in this situation where the open source model is better than the closed source quite yet. I think the delta is as narrow as it's ever been. Well, I am not of the view that that delta is going to continue to shrink. My view is that this is probably much more a temporary situation. And I would say immediately right now there are people that are doing R1 style training with 10 times the compute. And if those models end up becoming actually state of the art or they have, you know, genuine, genuine advantage over other closed models, I don't think people would open source those.

Wait, Alex, can you expand on why you think the delta is not going to get smaller? I want to hear more about that.

I think you have two things happening right now. You have this efficiency trend which happens with every new technology where, okay, there's the prototype which was GPT4, was very expensive, it was very inefficient and there's now all these ways where you just can make it better and better and better. And it comes down on this exponentially decaying curve. Like you have these huge big wins initially, but you have another curve going on which is more compute, continues to make these models better on logarithmic curves. And so both these things can be true, right? The models get cheaper, but then people just take the same amount of investment they were spending and just 100% you get a better model. And the other dynamic here is that because the switching costs of the model is so low, the switching cost from the user perspective of going from R1 to O1 to Claude, all that matters is frontier utility, like who has the best model. That's the thing that's going to, in the limit, accumulate most of the use. So all the incentive is, well, I just get slightly better than everyone else. I just get it. I just get better enough that it's a compelling case for people to switch and I get the whole market. And so you have this continual sort of dynamic Here where people are going to keep sinking compute into these models until two things happen. One is marginal increase in intelligence is not beneficial. We're at the point where the models are just good enough for all use cases or scenario. Two more compute stops making the models better. And my view is we're not close to either those things yet.

Wow, I think that's so cogent of you. I think that's so spot on. I think one thing we were looking at earlier is that just to give a concrete data point of how more compute is still making it better and there was this kind of belief that yeah, more compute, more parameters, better model. You know, OpenAI released O3 mini and we were looking through the scorecard of how accurate it is and there's one metric that OpenAI looks at which is like it's called the OpenAI PR. So in, in programming kind of like how good it is at doing OpenAI style PRs it just completely got 0% the mini model. And so I just think it's funny, it's, I mean it's really good at certain things but because these smaller models have less params like they hallucinate more, they're not as good at real world things and overall they just like they can be deployed in production but you just worry because they don't have the context that they need. And so I also am in the belief of, as of right now, I know there was this whole wave of specialized small models last year but I'm just not, I'm just not seeing those come to market leadership. And I agree maybe it will change. I'm not predicting the future but at least where we are today. So I, I'm on your side.

Yeah. And there's another dynamic we should mention here about the small models which is you can always distill the best small model if you have the best big one. So there's, you know, there's no scenario where in my view where a small company goes off and they go well we've made the best seven bill model and no one else can touch us. Like whoever has the best trillion model is going to be able to still the best 7 billion version. So yeah, again, right. It all reinforces like the whole game is the strongest base model. That's the whole thing.

Yeah, quantization works exactly right, Exactly.

Well maybe, maybe this would be a good time to ask about, you know, more inference, time compute because as we're seeing some of these things get put into production, really we start turning to agents and we start turning to Kind of how capable they are and chain of thought. Right. And is it that like the Web3 approach? Alex and Kasey, do you guys think it will be competing on the pre training dimension, on the inference time compute dimension? And maybe we could explain what that is too, more precisely or both. Or maybe there's like a third thing we're not thinking about. Like what do you think?

I mean, I think it's now very clear how the decentralized setting is a lot more feasible in this, in this reasoning model type paradigm. So one of the things I think that's probably going to happen this year is you'll have big collaborative replications of R1 and R1 style runs. And this is just a side effect of those types of models. That type of training is just much more amenable to that setting. So I think you're going to get that. I think you still need strong base models to do that type of post training. So you don't really solve the whole, you don't solve the core problem in my view. But I think very likely that that's going to about sort of start to happen. And you know, you can see how it's so appealing. Right. All you need is this verifiable reward and you can just let these things go and you can see how good they get. And I think we're going to see a lot of that.

Yeah. Jake, do you want to back up and kind of explain imprints time compute or do you want to. Because I could also double.

Yeah, I want you to do it.

How are you?

How you doing? Yeah. So basically before inference time compute, what you would do is you'd run inference, call the model, get back a result. That was it. And now there is this new paradigm where we're pushing more compute to inference time. And it's kind of this belief. It's called chain of thought. It's also called thinking. There's a lot of different words that are using to describe this, but basically simple way to think of it is you're giving the model the ability to think more completely. And so you're calling, you're basically generating a lot of answers and then you're compressing them and then coming up with like the most answer. Like that's the probably the most simple way I could put it without getting too technical. And then what that means though is it shifts. There's a lot of repercussions of this because you're shifting the cost to inference. And so there was kind of this belief that inference networks were going to be commoditized because all the compute was spent on training. But now that's kind of being questioned a little bit because of this time, this inference time, compute. And I also think that it's important to call out that this reasoning model and another word that's kind of used in the same world is reasoning. These new models that OpenAI releasing are reasoning models. And Deep SEQ is a reasoning model. And I think that Alex should explain how Deep seek went from V3 to R1 through what is it called? GRPO, and how that. Because your background's in reinforcement learning. And I'm also like a reinforcement learning pill girl. I think that we should explain why that breakthrough was so important and simple. Like how in the code, it's so simple, but it's so effective. I can explain to you, but I think you should explain it because you're deeper on this.

Yeah, yeah, I can have a pass. I mean, it's one of those things, right, that as soon as it was published, I think everyone in the field was just like, that's just so obvious. Why. Why did. Why did everyone not do that? Which is always the case of really all good things. But okay, stepping back, like, how do you train a normal dense LLM? And then let's explain what does GRPO do different when you train a base model? Normally, you basically, you're training it auto aggressively. You are the thing everyone knows, right? You generate a new token, you check was that the right token? You have some accumulated loss, and then you do a back roll and you're doing this for every sort of token in your training corpus. You're learning to mimic text is kind of the key thing. It's the same thing when you do supervised fine tuning. What you do differently with GRPO and sort of reasoning style reinforcement learning. So you generate a full answer. So you don't have these like intermediate gradients coming in from these intermediate tokens. All you care about is is my final answer correct or not? Is the. Is the destination that this model got to correct? Like yes or no? And you can do it binary or you can do it with some kind of like soft scoring objective. Like how correctly it get sort of depends on the task. And the point is at that point, you're not doing this big backpropagation step. You're actually constructing this GRPO objective. The way you do it is then you then make the model more likely to produce the answer that was right and less likely to produce the answers that were wrong. And you sort of just repeat this process over and over again. The key thing is you've got to have a verifiable reward. To do this, you need to know that that answer was either right or not. And this is one of the big challenges in that setting.

Now I want to touch upon, like, one of the three directions we. We kind of mentioned before. And then I want to go into kind of back into Web3 about how Web3 aids in the. In the process of everything we're talking about. But one of the previous directions, Kasey, that you mentioned was, hey, we're actually solving some key problems here. And in my mind, it's basically like we're helping training happen on commodity or more commodity hardware. We actually see that a little bit with Deepseek already because they did not train on the highest end hardware. They traded on H1 hundreds, reportedly. The question is like, can we get down to like pure commodity hardware? By the way, Google, where Casey, I think you worked in the past, one of the ways that they succeeded is by building data centers from commodity hardware. They were like one of the pioneers of doing that and created a lot of advantages and basically won. So commodity hardware, there's also fault tolerance. I feel like when you're dealing with any decentralized Web3 network, you're going to have machines that go in and out. Now, training processes are very sensitive. Usually if you lose a machine, you might derail the whole thing unless you've taken steps to make those processes robust. So just kind of out of the gate, if you're doing a Web3 network for training, you have to think about that problem. And obviously these very sophisticated kind of like model parallel approaches that Alex is talking about could also be used to train, you know, potentially in centralized contexts as well. So, Alex, did I cover that? Are there other ways that Web3AI is contributing to broader AI?

Yeah, I think. I mean, it's interesting on the Byzantine robustness stuff, it's like all these problems are actually things you deal with in the centralized case, and they kind of just become way more pronounced than the decentralized case. So it's kind of like you're solving the harder version of the problem. Like, for example, you do a standard centralized run, you actually get things like bit flips and devices going offline, just because if you're using 10,000 GPUs, like things break or like a cosmic ray hits something in the communication and stuff goes wrong. So Byzantine robustness is actually studied in that setting. It's actually like deep literature there. And then when you move to the decentralized case, you kind of just go, okay, now people can be actively malicious. So it's now a harder version of this already well studied problem. So as far as advancing the literature side of things, it's less about you're creating whole new fields and it's more about you're dealing with harder cases of these existing problems. Yeah.

On the topic of kind of nodes misbehaving, there's a whole general concept in Web3 Networks, the idea of verifiability, right. So if you're delegating like any task really to a node on a network where you don't know who the operator of that node is and you're tasking them with some kind of computation on your behalf, like how do you know they're doing it correctly? So there's a lot of approaches to solving this and especially like over the years we've had kind of reputation based approaches, we've had approaches based on people staking some economic value, usually in the form of tokens and then getting slashed if they misbehave. More recently there's a trend for a very high degree of verifiability. Like for example, you now can use zero knowledge proofs to actually fully verify computations. Or another way of doing something functionally similar is using a tee which has different security trade offs to ZKPs and different kind of profile. But I would say one company in our portfolio called Giza is actually working on fully ZK verifiable AI inference. Now, because ZK is at this moment in time in 2025, still adds a very big overhead to your computation. Can't really do it on LLMs yet. But what Gizzer is showing is that when you do it on even smaller models, decision trees and things that little agents that optimize certain tests and defi for you, or manage your vaults and move your money to the highest yielding vault or roll your perps and stuff like that when it's time, those things are still very, very useful and they're fully verified and they might be managing millions of dollars. So they should be fully verified. And there's a lot of demand for them too. That's kind of an interesting point. You don't necessarily need to get to LLM intelligence to have comp, you know, AI, AI inference in a decentralized context be very useful.

So Jay, follow up question on that.

Of course.

Okay, so what, yeah, I'm curious, like for the non LLM use cases, just from like a Giza standpoint, what, what is like the demand Right now for that, that you're seeing?

Well, we're Giza launched Arma agent, It's Arma xyz. You guys can go take a look. And it's a very simple thing. It's just kind of like looks at all these yield bearing protocols and then when it notices that you could get a better yield somewhere else, it sort of like flips you to that one, flips your capital to that one automatically. The point is that the agent can interact with DEFI sort of autonomously. And it's kind of funny, but most of the capital right now, like 99.9% of it is being channeled to morpho because that's the one that has the highest yield.

Right.

But the automation is the cool part. And we could see a lot of founders in this market, they talk about what they refer to as defai defi using AI as kind of a key use case for useful agents.

So that's why we're excited about it. Can I say something? Verifiability, I think it's an interesting part of my story, right? So my whole thing was I became convinced the model parallel could work. And as soon as you believe that, then you run into these two problems you mentioned of the verifiability and incentives. And this was actually the thing that made me realize I need to go and engage with crypto land and go and figure this stuff out. Because like you said, all the verifiability is a well studied problem in crypto work. Verifiability is all these work streams of people that are solving it from the game theoretic to the ZK to the Tesla. And when you think it through from first principles, that's why this sort of combination of these two fields actually makes sense. Okay, we can do lifestyle collaborative training, suddenly there's multi parties. How can we. We need to verify the work in some way. This is a well studied thing in crypto. You have to engage, you have to figure this stuff out. And then as soon as you have that, how are you going to incentivize training? You can't, you can't ask people to just donate a huge amount of compute. And this is the other thing that's just extremely well studied in crypto. That's sort of my, my argument for why these misconduct rules make sense. There.

There's actually probably like a super interesting discussion on the incentives part too. If I can go on a little bit of a tangent here for a minute. Because I think like we've, as we've watched Blockchains kind of come up and compete and try to steal each other's customers and projects and stuff. They've always made this case that like, oh well, we'll just launch a new token and then people will be really attracted to the upside of that and then they'll come over and we'll be able to use these tokens as incentives. And actually what we're seeing now is that there's such a saturation of different kinds of blockchains and L1s and L2s and side chains and this kind of chain and that kind of chain and everyone has a token rewards program that now it's not like a no brainer to switch to a new thing now you have to take into consideration the risk that hey, maybe that platform won't make it and maybe that token actually won't automatically go up like it always used to for the last 10 years. So I would say it's become a little bit of a tougher market for incentive programs. Especially in a market like this that we're seeing in Q1 of 25. There's so much volatility short term and like tariffs and interest rates and all this, all these things. So I don't know, it's going to be interesting. Like there's going to be a new sort of demand for a new kind of rewards program that's really, really good in some way that gets people comfortable.

I think that's so spot on, Jake. I think if you look back one way to one analogy is it was just like these customer acquisition like cap programs for a long time, like the cost to acquire a customer and they it has gotten so saturated and it's only I think going to become more saturated with strong because now there's even less fear around launching tokens. So everyone's going to launch tokens. And so the benefit of it really just being a way to build up a treasury, it's just a lot more precarious than it's ever been before. I want to go back to one thing that Alex said or like the two points that Alex made. I think they're correct. I think this idea that some people are bearish verification and they're like, oh, we don't need it. I would love to have one of those people on the podcast because I would just love to hear the rationale because I think you absolutely do. Like how possibly would you do this without verifying? Because just game theory, people are gonna misbehave, people are gonna try to get away with the cheapest thing and so if you're not having some checks and balances verification, it would never lead to a result that would be useful. And so like that, that like core assumption in the dependency chart of this all working has to lead you to crypto. Like, and if you don't believe that. Yeah, again, like, I'd love to chat with you about that.

Oh yeah, people will definitely be, you know, distilling cheaper models that kind of look like the model that you're trying to inference telling you that that's the premium model and saving a lot of money for themselves.

Yeah, not only that, right. Like if I don't like a run, I can go in and just start putting in random gradients. And I, and I dropped a bunch of people's effort and there are people, you know, I don't think anyone has any doubt there's people in the world that would do that for fun.

Yeah, exactly. If, even if not for profit, just for entertainment.

Exactly. Okay, so that's maybe that's a good segue into the next part. Here to talk about monetization opportunities are really like the way that I think about it. Let me paint you guys a picture. The way that I think about it for AI networks is that they're just another form of like web3 public good networks, meaning there's this big giant, almost you might say, monolithic, you know, network of nodes whose job it is to train these models. And the reason it's public is because eventually like you'll be able to join it if you want, you know, like with many Web3 networks today, like if you wanted to, you know, be a video transcoder using your GPU to do that on live peers network, you could just, you know, spin up a node, kind of stake some tokens and join that network and it's all like fully automatic. That's cool. So I see this big giant, you know, open network of nodes and together they can come and train a big model. Now there's an obvious question there, Alex, which is, look, stability, did all these open models and you know, maybe some other companies as well, but what so far everyone has failed to do is to monetize them. So how does it make sense to have an open public network to train stuff when, you know, we already kind of know that open models don't really like make money?

Yeah, excellent question. So pluralis is sort of the two act thing. Act one is model parallel is feasible, but the whole act two is my view is that model parallel gives you this path to monetizing at the model layer and that monetization right now at the base model layer is the thing that's completely missing to make what people would call open source AI actually sustainable. Like if you think about what we have today, right, there's two release paradigms, there's centralized development where you go, you train the model, you incur all the cost, you release it, you can monetize it, very straightforward. Then there's everything that fits onto the bucket of quote unquote open source AI, where the idea is someone at some point incurs a big cost and then gives the result away in one form or another. Either it's you give away the weight set or you give away the weight set plus the training data, or you give away the weight set plus the training data plus the actual competing on boring utility. It's very logical why you would compete on openness. But there is no self sustaining mechanism there. And the big argument for pluralis is I think you can get all the benefits of open source, this permissionless innovation, this guarantee of access, this sort of rich ecosystem of development, while maintaining the ability to monetize a base model. And this is sort of what we're calling this protocol learning framework. It's where development is done in the open models are trained in this collaborative way, but no one actor can obtain or release the full weight set. And so your terminal state is the model is locked to the protocol. And while you have this like rich ecosystem development, people can go fork it, they can train it, you have, you know, it's guaranteed ability for someone to access it. People can train the kind of models they want. You actually also have this way for the trainers to recoup their training cost. And this brings a sustainable economics to this whole domain that to me is completely lacking today. We have to have some kind of third alternative like this or we just retain this dependency on the metas of the world.

So it's an interesting combination because on the one hand it's closed because the weights are closed and that's good because it creates this monetization strategy. And on the other hand it's more open if you're like open sourcing the setup of the thing, right?

Yeah. And my argument would be that is more true to the spirit of open source, where if the data mix is known, if the training high parameters are known, if the whole setup is open, I know more about the behavior of that model than I do if someone gives me a weight set with no other information. You know, Anthropic has a nice paper that it's called the sleeper agents paper. But it's very clear that you can inject behavior into models that it's very hard to detect with only access to the weight set. You can't do that if the training and training setup is not because people can see that you tried to manipulate it.

So basically this creates a situation where I can sort of, I could source some capital, I can create a model, and then I have a business model on that model, which is great. And then someone can come along, they could use my model, or they could source their own capital and sort of do the same thing, or try to innovate on my setup and stuff, and then maybe they'll create something better and then people will use that. And this is a very familiar thing to crypto people in the sense that in the past, you know, you could have bought into a blockchain, a new blockchain, or you could have maybe like started your own. And a lot of people who started their own kind of got a better return until that market was like, very, very saturated, as we just discussed. But in your model, Alex, in Polaris model, wouldn't that mean that, like, a lot of people would come in and like, unnecessarily, like recreate a bunch of the same model and then just try to capture people's attention to use their thing instead of your thing?

I do think when you create new markets like this, you almost always have chaotic behavior at the start, and then it consolidates into sort of rational economic style behavior. So short term, I think that's totally possible. In my view, it's more of a feature than a bug with a protocol. If you have trainers having direct ownership of the model, they're the ones that are directly incentivized to drive use. It's not pluralis's job to go out and say to people, you have to use this model over this model. The key thing is you have this ability to monetize the model level. And once you have that, then you can have these, okay, someone's building on someone else. You can have programmatic value flow. A model is using a data set. You can have programmatic value flow back to the data set. This is, in my view, the missing piece that allows actual value through this whole stack is, is monetizing the model level.

Hasty can centralized or will centralized and decentralized AI, will they converge together or will they be sort of separate for like, right now we have Web two and Web three people, but they're not like, quite converging yet. What do you think the process is for AI.

Well, I want to talk about open source a little bit. Take crypto out of it. Just Open source is fascinating. It's a super strong community. It's been around forever. It's very interesting because it's not driven totally by financialization. It's genuinely a community of people that want to push technology forward. Some of my best friends are just open source devs. That's what they do because they have made enough money and they just want to contribute to open source for the betterment of tech for real. And I know that's hard for crypto people to imagine, but there are people out there that do. But if you think about how Open source works today, like the reality of it is mostly it's open source backed by a bigger player. We already talked about Meta as like a concrete example, but another one that people might be familiar is like Vercell and Next js. Next JS is a super, super well adopted framework. And the only reason, not the only reason, but the reason it's really able to be maintained partially is because it's backed by Vercel. And there's a ton of examples of this in traditional web 2. And the reason I call this out is because what Alex just said is exactly the point, that maybe Next JS is possible by open source developers because they can just create a JavaScript framework and that's not high overhead. But these open source devs can't even contribute to the model layer because of the enormous cost to even start playing around. And so it's like open source is not like that community can't even tap into the lower level yet. And that to me is just, it's crazy. Like, it's just crazy that there's this, these group of people that can't really do it. And if you have money, if you have connections, sure, whatever. Like, you know, there's nuance to everything I'm saying and I don't, I don't want to be like too blanket with this, but at a broad stroke that's how I see it today. And so what we're talking about here is so big in terms of the impact and domino effect it wouldn't just have for crypto, but just in terms of development of software development at whole and unlocking this monetization. It's like, I don't want to go too much on a tirade, but I think the last thing I want to say is that monetizing open source and something I've been thinking about for a long time, like years A lot of time. And because I think it would make it more sustainable. And I think if you could add a financial benefit to this, it would just. The whole community would even further flourish if you did it correctly. And there's a lot of companies that are going after this from a. From a variety of different standpoints. Like, you put the model in a tee and then you have. There's, like, a lot of things going on, but this. This strategy of breaking up the weights so then you can't copy it, but you can still use it in an open way where we're not worried about, you know, our brains being uploaded to Sam Altman's machine and we can still contribute to capitalism and work in a sustainable way. That is so sick.

Love it. Love it. All right, guys, in the last segment here, I want predictions. I want predictions for 2025. What do you think is going to happen in decentralized training or decentralized AI in general? And, yeah, what. What should people be thinking about?

I think we're going to see some stuff coming out soon that is going to probably draw a very large amount of attention to this. So I think the field sort of moves from being still somewhat fringe to sort of very square, sort of in the center of the narrative. I think we hit the academic tipping point. This year is my thing. So if you look at these big conferences, there's these trickles of interest that are showing up. And even just if you look at the ICLR papers that were released that were accepted the other day, now there's about five to six papers which are taking this domain seriously. And these things really happen in tipping points. The word gets out and suddenly everyone is working on this. And especially in an area like this where the sort of potential impact is so large. And also academics are completely locked out from this layer, you can see how it's going to suck a lot of attention. So I think this year we sort of start to see real serious legitimacy in the academic side. I think we're probably going to see some very large models trained in this way that sort of put the whole thing beyond doubt. Right now, it's still me and a few other people saying crazy stuff that no one believes. We just see some. Some just demonstrations that it all works. And I think the. The sort of trickle of interest in this area sort of turns into a flood. That's my prediction. Nice.

What about you, Kasey?

All right. I want to hear yours too, Jake. After this, I think. I think reinforcement learning becomes like a favorite child. I think it's been, I think last year, you know, LLMs were the word. I don't even know how many people know about rl, but RL is such a powerful tool. And I think Deep Seek is the first. Not the first, but Deep Seek. Deep Seek really brought this to the forefront, I would say. And so I think that that continues. And then I think much bigger models are going to be trained. Like I messaged this to Alex. I'm like trillion, trillion perimeter model coming. I just think that's going to happen. And I think that, I think it's going to be wild. I hate putting out numbers because I'm probably going to be wrong. But I'll give you the prediction because you're asking.

Nice, nice. I like that one. Trilly. My prediction is very simple. I have daisummer in my Twitter handle right now and I have had it for a few months. But basically where I'm coming from is this very simple idea that retail investors want to invest in frontier AI, but they can't because they don't have public stocks that give them access to frontier AI and they can't get into the private funding rounds of Anthropic and OpenAI and Perplexity and Mistral and all those companies. And so the most sensible way of getting exposure is through crypto tokens. Now, the crypto AI space has been very, very noisy. It has been very filled with meme coins. It's currently down. It's, you know, a lot of people have, are giving up on it. But as you and Alex are pointing out, there are a number of companies in this space who are solving real hard problems in AI. And once people connect that fact that like crypto AI is actually really important, very real, and the only way that you could invest in it is through crypto tokens, that's when Dai Summer 2025 happens. So that's what I'm here for. That's my prediction, is that we'll be in a, in a very nice bull market in this, at least in this space, but hopefully in all of crypto this summer.

I love that there's still a lot of research to be done, but I love that.

I love it.

All right, Alex Kasey, thank you so much for joining us. We really appreciate having you on the show. That was Topology Ventures founder and managing partner Casey Carosa and Pluralis Research founder Alex Long. For those of you who tuned in, thank you so much for listening. Let's keep the conversation going. Leave us a review. Let us know what you thought. If you want to follow me on Twitter jbrook J B R U K h Make sure to download this podcast where you get your podcasts and we'll be back soon to dig deep on ideas of the blockchain space and decentralized training. Thank you.