AI and Crypto: Countering Centralization | Casey Caruso & Daniel Shorr

So AI is one of the most transformative technologies we've ever made, ever. And I've known that for a long time, since I've been a kid. It's an inherently centralizing technology. And the reason it is, if it's not clear, is because you are more efficient when you have GPUs connected, co located, you can do more from a fundamental hardware level. And so this has this huge ripple effect of basically centralizing the entire thing. As a believer of decentralized technology, I've always felt that one of the biggest threats we have, honestly, not to sound super philosophical, but humanity faces right now is centralizing AI to a point that the innovation doesn't compound.

All right, everyone. So on Empire, you obviously know that we talk a lot about the institutions coming into crypto and that is why we are super excited to share that we are hosting the Digital Asset Summit. We posted this since 2019. It's coming up in London, March 18th to 20th. Don't miss your chance to get ahead of the curve. You can get 20% off with code EMPIRE20. We'll see you in London.

So welcome everyone. I'm really excited to host both Casey and Daniel, which I've known for a bit. The idea was to have a podcast that would cover all things AI and then why it's relevant to crypto. Of course, we've all started to see companies that have been trying to leverage some of the key innovations happening in crypto and figured this would be a great time to have both Kasey and Daniel, who are, in my mind, people that I've learned a lot from in the space. And so, Kasey, Daniel, welcome. I'll let you both introduce yourselves.

Thank you. Yeah, happy to give a quick intro here. So my name's Kasey. I was previously an investment partner at Paradigm and before that was an engineer at Google and studied computer engineering undergrad and then have my master's in AI.

Thanks for having me, Santiago. Casey, always a pleasure. I'm Daniel. I'm one of the co founders of a company called Modulus. We sit at the intersection between AI and crypto. So this should be really fun to be able to dive into this. And before Modulus, I had just wrapped up school at Stanford, also studying CS like everybody else now, but actually with a specific focus from my co founder in AI. So definitely right at the intersection.

Great. So I'd like to spend perhaps the first segment of the pod, really just giving a more 101, 102 kind of foundational knowledge of AI, because I feel like most people don't appreciate the key unlocks and innovation that has happened to lead us to where we are today, where everyone's talking about AI, that it's a hottest thing, that it's going to be hugely transformational, and I certainly believe it. But I think that appreciating why and key innovations that have led us here would be a fantastic setup to then give everyone a good understanding. So I'll leave you two really to have that discussion, so maybe we can get started with that.

So I feel that a lot has changed over the last few years and I can just do a quick history from my perspective empirically, of where AI has been and where we are today. So when I was in school, a lot of what we were doing was not ML, it was really data science and we were doing a lot of statistical models and this whole concept of an autonomous or like a model that would be able to make decisions was a little bit sci fi still. And then I would say like around 2017 when I was at Google, we started to see the power of what now is called a foundation model. And you can think of that as just a big machine learning model that takes a lot of data and is trained on that data. And so something that I think is really underappreciated is that the Internet has basically been this Trojan horse as a way to get us digital data. And our models were historically capped because we didn't have all this information that we could feed as examples. And so yeah, I like to think of that meme of the Trojan horse. And the horse is the Internet and inside the horse is the data. And then on the other side of the wall is machine learning, like modern day machine learning. And so kind of like just going back to 2017, we start to see that, we start to see, oh wow, when we pair transformer based nets with a lot of data with really powerful hardware, we get interesting models and then we kind of kept them at the big tech giants. So behind the closed walls of Google's and Meta, all of those companies had these very powerful models, but they didn't really open source them at the time. And so what's super interesting about now, and one of the reasons I'm expanding my own mandate to invest in AI is we really are at this pinnacle of this revolution where powerful foundation models, pre trained neural nets are accessible to everyone by either running them locally or hitting an API. And so it's a very interesting, we'll look back on this time and say this is a very interesting time in all of history, neural nets have been around since the 60s, but never in this capacity and never to this accuracy. And it's very interesting to be here today because I truly do believe this is the beginning of a totally different transformative age.

Maybe Daniel, or you, Casey, could talk about it. So it sounds like it was a combination of more data, but also hardware becoming cheaper that would allow you to run these models at scale in a cost effective manner.

And the algorithms got a lot better, like reinforcement got a lot better, Transformer based nets got a lot better, the architectures got better. But then I would, I mean, I'm curious, Daniel, your perspective. From my perspective, it's like those three things. It's better data, mainly through the Internet, better models and algorithms, and then also this acceleration of hardware which we're so, so lucky to have now.

Yeah, I mean, absolutely, agreed. I guess maybe of those three, all of which are really, really important, I would say the one that is particularly or has particularly been relevant for the past couple of years at least from my perspective, has actually been the compute cost or the hardware aspects. Right. Because in some sense, I mean, you mentioned this, Casey, but the ideas behind machine learning, these kinds of autonomous models, have been around since the 50s and 60s. And a lot of these algorithms that we are now putting into production, those ideas have also been floating around for 20, 30 years. I think Alexnet was like 2012, 2013 around that period. And the question for me anyways is always kind of what changed between then and now in particular that enabled these kinds of massive models to suddenly become much more effective at the tasks that we've always wanted them to do. And I would make an argument around GPUs and lowering the cost of that silica per unit compute and all those fun metrics. But I think that's a really important part of the story, which has enabled us to bring these algorithmic improvements to our models and then actually digest that data that the Internet has provided. Because I think around the same time, 2017, which is when I started college, we have started, at least in the academic side of things, moving into computer vision and being able to give these models these multimodal expressions. And that's entirely enabled by the fact that COMPUTE was super cheap. Nvidia had to deal with my class so we could just bring up more GPUs and spin up bigger models. And these kinds of seemingly trivial improvements to accessibility, I would say is a huge part of what's responsible for like what AI is today.

Also, just on this point, since you called out cv. I think for listeners that people are less familiar with ML are probably wondering what is working today and what is not working today. And just to concretely answer that question, I think the two things that are working are basically CV computer vision and NLP language models. And so we don't call them that anymore. Those are more like antiquated terms. We call them vision models and language models, but those are the predecessor terms and it's a little bit different and there's nuance in all of this. But just to do this in a broad stroke and so just to say that out loud, I do think that might be helpful. These two categories have really risen the ranks of where we're seeing the biggest gains. And the reason is because of everything Daniel said, both in hardware and with the GPUs and with we have the most data on that. We have the most data of like text and images.

So perhaps if you were to talk to a researcher perhaps five or even 10 years ago, you may not even go that far and say that he just stopped there. Like what would he have told you about the progression of AI in that state versus today? You know, if, if someone in academia just stopped, like was it three years ago? It just felt even maybe two years ago, like how would he have described. He or she would have described AI? Because I feel like that would have been quite different.

Casey. I guess two or three years ago, I don't even think Transformers. It wasn't taught to me in class. We didn't really talk about it very much. It wasn't like the de facto winner of NLP kind of language model discussion at all.

Yeah, I think at Google, yeah, maybe a little different because it was so inbred there, a little more so. But yeah, I think the entire architecture was different. I think we were doing a lot of like LSTMs were still a thing back or yeah, long term, short term memory was a lot of things back then. But the entire ecosystem was so dependent on different architectures that are. We made so much progress in consensus of how to move forward. I think that's the biggest difference is like three years ago I feel like there were different camps of oh, we think this is the right way. Oh, we think this is the right way. And I think now we've really created these stepping stones where we're all sitting here saying, okay, this is the state of art today, let's all work towards this. And then, you know, we're still seeing of course, progression and innovation. But I feel personally like there's a Lot more consensus. I don't know if you disagree with that, Daniel.

No, no, agreed. I think, yeah, it was very, it was much more tribal back even just a couple years ago. Right. Because it was not entirely obvious which techniques would be, would have these like emergent properties that just show up when, when you get to certain compute thresholds. Obviously transformers, diffusion models, like all of these have been, have totally been paradigm shifting but. But yeah, it's just very hard to predict when these architectures will emerge suddenly as. Okay, yeah, like this way of laying out your algorithm seems to make the models really efficient at logic or really efficient at digesting images. So I think even back then a couple of years ago I would have been much more pessimistic about the progress in the field, but clearly it's just gone significantly better.

Yeah, I think. Well, if I remember there's like Google Brain and there was a couple of initiatives obviously like, like there was IBM that like Deep Blue I believe. Like it was. What is it like the chess algorithm that like Watson. Yeah, yeah. And then, then they applied that to like medical stuff. So it's like been percolating. I guess it wouldn't be a podcast about AI. We didn't talk about like ChatGPT and how that has been very accessible for people to start understanding how they could leverage AI in their workflow, in their daily day interactions. What made like OpenAI just rise to become this, what is probably synonymous when most people. I think if we were to survey like a hundred people, like what is AI to you? They're like oh yeah, like OpenAI, like the guy Sam, like chat GPT. But of course it's like there's so much more behind that. But I'm curious like how like some background around that, like were they pioneering particular thing like what made them what they are today from your perspective?

Well, they are the grandfather of the modern day language model. So like let's just start there. They are the first company to really offer a language model that's accessible and offers a step function and utility to the degree that they do. And Daniel, I'm sure you were playing with GPT3 as well, you know, before it really hit mainstream. But it was good, it was okay. But fine tuning was a pain in the butt and for people that are less familiar with fine tuning, just a process to basically specialize a model and it was just a really big pain. As a developer, I remember doing that I think in 2021 and just being a little bit disappointed with the output. And then when GPT 3.5 came out. I mean, I think every developer remembers the day that they tried 3.5 for the first time and it's just like, oh my gosh, this works. And the fine tuning, it's just everything was better to a degree of. In venture, we think a lot about set functions, like where is the bar where the switching costs become clear. And so Santiago, to answer your question, to me it was like when they basically released, I think it was like DaVinci, I forget exactly the specs of it, but there was a model that they released that was just superior and superseded everything else in the market. And then they really use that to crystallize their market leadership by basically making it as easy as possible to use, offering up a UI and interface that anyone could access without needing any developer knowledge. And so from my perspective, that's kind of the progression.

Yeah, I mean, I think this is a weird thing to say now, but like AI companies didn't really exist a few years ago. As in like what does it even mean to have an AI company? Maybe outside of scale AI, you know, around like the data component. It's mostly just products and services that were enhanced with these really hyper specialized models, right. There was no such thing as kind of AI as a product, so to speak. I think it came to or OpenAI's kind of big product level innovation here is to offer that language model in this really accessible form factor and have the model be performant enough where it's starting to get that almost like general intelligence vibe to it, right? Where like my mother could hop on, you know, chat OpenAI, whatever, and actually engage with this product and have like a really, you know, productive experience that's just like never existed before, right? Previously it's like, oh, I have this model which helps me understand risk better in my portfolio or I have a model which can help me pick out faces in a crowd. You know, they're all part of like a bigger product pipeline but with that is the product.

So maybe pulling on that thread a bit more, just going back to the kind of, the more foundational stuff. So the requirements are good, high quality data and the ability to ingest that in a cost efficient manner. How would you describe a language model to your mom, to your dad, to someone that is totally not technical?

I would say that a language model predicts the next word or character by taking examples and using that to predict.

So for instance, yeah, that is literally.

What it's doing for what it's worth, like mechanistically and it's just Crazy that it works that well.

Yeah, that to me was quite eye opening. It's like it's predicting the next word, but not. But somehow the entirety of the text like just is very coherent and it's very accurate. To me that was one of the more mind blowing discoveries around how this actually works. Because I would have thought, I mean, I think there is some relevance, but when I heard that I was like, really? How is this even possible?

It's just wild what you pointed out Santiago around like how does it kind of keep that context had been like the big roadblock to I would say the adoption of transformers for a long time. Right. Like, oh, this idea around attention, this idea around keeping it in memory and navigating context so halfway through a sentence it kind of doesn't just go on a different tangent. If you look at GPT, two GPT, even three. Right. This is the kind of performance issues that you see. And really it was kind of the continued investment in that architecture to figure out exactly how do we get these modules correct or more performant anyways that suddenly it's like, oh my goodness, it's like generating entire conversations with me fully tracking everything.

Yeah, yeah.

And maybe a question in terms of like I'm going to probably say something that may not be correct, but like when it's. When the, when the algorithm is thinking about, okay, probabilistically, it looks at all the data that it has at its disposal to say in this particular word, the next word, to predict the next word. It sort of says in the context of everything that I have, I have in the model and the training is probably this word that is most apt or relevant. And so then it does that very quickly to compose the answer. When you're like doing a prompt for ChatGPT. Right. Can we talk about like the quality of these models, for instance, different languages or the bias in them or when you're trying to apply them for specific verticals and use cases. Are we at a point where they are as relevant as general prompts or are we still in that phase of. There's. There's an opportunity to get way more sophisticated at like specializing these models to be more specific for risk or for bio or whatever.

It absolutely makes sense. It just also covers a lot of ground. Right. So it just depends on which thread you want to I guess, pick at. I mean, I think specialization is as good a place as any. Yeah, I mean, you know, these models are like. I think one of the more surprising things to me anyways is the kind of zero shot like just like the untuned performance of these models are already very, very impressive, right? I mean X instead of Y, right? Like literally just going to any of the GPT prompts that you might have in mind and just inputting it directly for a response. You can always tune that conversation by getting more specific based on the response you're getting from the model to give it more information. But imagine not doing that, just a single query and it getting really accurate results. You ask a question about something medical or something legal and it finds that piece of information or that relevant context. That is again a very surprising feature. The thought for a long time was that these models would have to go through really, really intense fine tuning to be able to even get to that level of performance. And it looks like actually these journalist models are already very good even without that fine tuning. But now you layer on the fine tuning and that performance, that specificity goes to a level that again, you know, three or four years ago I, I would have totally been just flabbergasted by Casey. I don't know if that's been your experience.

Yeah, yeah. So I agree there's a lot in that question. And so I just want to run through a list so the different languages, like this is my question. I think I like to have, you know, anybody can assess is like when you're thinking would an LLM be good at this? You just think is there a lot of data of this on the. And so of course it's really good at every language because there's so much information in every language. And so yes, to answer that question then I feel the question about how good are these models at a base level? And the kind of where are we going next with this specialization? I think of 2023 or I think we'll look back on 2023 and say this was the year of base models. Just like L1s. You'll be like, this was the year that we made like Ethereum and Solana and then it's going to be all right, well what are going to be the L2s on this and what are going to be the apps on this? And again a crude analogy, but bear with me, the way we're going to get there is fine tuning and I just again want to redefine this because I am kind of a little bit hesitant of we're both just hyper technical. So just think about fine tuning as you put in some extra data and you want it to maybe put guardrails on it. That's one way to think about it to behave in a certain way and you want it to prioritize certain information. And so to Santiago's point of these, like Bio being a fascinating now specialized use case for ML, we're absolutely going to see that in 2024. I mean we're going to see so many different protein models coming out in things with like neurotech models, biotech models, all of these different specializations. And we're already seeing this like I think Hervey AI. I'm not an investor. No, no incentive to say this, but I think that's a really fantastic example of a model that's reaching product market fit, I.e. using law as its domain and taking a base model and then training on top of it. And I think they're actually, I don't know too much about them and they're probably going down the direction of maybe building their own foundation model. But the point is that you can take these kind of modular building blocks, very akin to crypto, and start compos them in a way to build more specialized LLMs.

Yeah, on that point, because I do want to transition the discussion so pointing to how are these things being productized and where you see that opportunity and then of course talk about the relevance for crypto. Do you agree or disagree around bias in the data problems and challenges that we may face or is your thinking. No, these models are sophisticated enough that they learn by themselves, they improve by themselves to that kind of true intelligence where the model will just. So for instance, for Law or Bio, like if you're trying to do protein unfolding, like there's just stuff that you could have never looked back on because you are trying to cut new ground into discovering a protein that literally we don't know that exists. Yeah, but maybe it can do a certain relationship. But of course, like Pharma, Pfizer or whatever Moderna might have like data that they're not open sourcing, that they have for themselves that they're not going to share with anyone. And so they might keep that all to themselves. Would you say that like that type of model that has been trained with proprietary data will always be better than a model that is generalized? Like I'm curious if you have a view there.

I think the bias is there because it's in the data. So yes, we have to overcome that. And that's why ChatGPT is not just GPT, the base model. There's a lot of steps post GPT to take that corpus of information, do RLHF on it. Sorry, bring humans in the loop to help them modulate the output. And so, yes, there is bias to answer that concretely. And then the next question of like, will a proprietary data set always outperform a public data set? I think always is a really tough word. I try to stay away from absolutes, first of all. Second of all, I think that, yeah, we're seeing a lot of companies that have these data sets come forward and this will come out when we talk about the value or kind of the intersection of crypto AI. But there's kind of this elephant in the room right now, I feel, in that data is this one of the most important resources in the world. And yet nobody is monetizing it in a way that feels like robust. It's like this thing that you keep and you harvest. And our entire Internet of social Internet has been built on this premise of use our products for free, but we'll secretly, you know, mine your data and then we'll advertise to you. It's just a very opaque, nebulous kind of market structure that we work within right now. And so I hope that answers Santiago. But it's like, yes, there is this kind of belief right now that this private data can be more valuable. And we're seeing that come to fruition time and time again as companies go into AI.

And maybe, Daniel, if you can answer the third part of the question was, in terms of discovery, when you're using antiquated historical data, how does that overlay into, like, how sophisticated is the machine? When you talk about true intelligence, to making breakthroughs like protein, like discovering new proteins or, you know, for law? Well, you're looking at all the precedent that exists out there. But of course, new precedent gets set every day or, you know, every so many years in the Supreme Court. I'm curious, like, in terms of the discovery piece, how much of AI today is just reminding ourselves of what historically has been done with that context of data versus totally new stuff, totally new relationships that we have never conceived as humans before?

Yeah, I think one of the first things that Ryan and I did, Ryan being my co founder when GPT4 became accessible, was we asked it to solve one of the millennium problems or try to give us the proof to Cantor is like, you know, one of the proof based questions that we would get in our math classes, right. Just to see if it could string together enough logic to either fill in things that we've already solved and therefore should be somewhere in its state of bank, although maybe not frequently, or something that we've never solved. Before. Right. Just try to figure out is predicting the next token, the next most likely word in a sentence, enough mechanistic sophistication that it can actually solve these problems that we thought that we could never solve. And the short answer is it gives it a really good try. No, unfortunately we didn't solve one of the millennium problems that day. If we did, that would make headlines, I guess. So there's two, I love two opinions here. One is pattern recognition to the level or the, the sophistication of these language models has never been accessible or has never been this accessible before.

Right.

So certainly I would say there's a whole class of problems. You know, maybe they're, they're not quite discovery or they're borderline on what you would classify as a discovery that will be unlocked because more people can, you know, go at those problems with these new tools. But there are still, I think, problems out there that are absolutely novel.

Right.

That requires a kind of ingenuity that, that we've never exhibited before or have done so very infrequently. And so those kinds of architectural features is very hard for the model to inherit or to pull out that kind of like semantic information. It just requires, I think, a really high degree again of sophistication. With all of that said, I don't know what GPT5 looks like. Gemini looks great. I'm sure the next version is already being developed. It's quite possible that, although so mind boggling to me anyways, that a deep understanding of language itself turns out to be this incredible tool to just solve a ton of problems. I think that would be like the strange, unexpected discovery of like the century. Right. So, yeah, not a very satisfying answer. There's some tension here, even for me, but I do think that I wouldn't be surprised at all if tomorrow it turns out I'm totally wrong and we're solving new math problems or unlocking protein folding or whatever it is with just more sophisticated models.

Yeah. Can I say one more thing on this?

Of course, yeah.

Is that I feel something that you're hitting on is basically asking, is like, look, how smart are these models actually? And I do think another way to juxtapose the past to the present is in the past we used ML to predict things. Probably the most well used or most used application of ML is recommendation algorithms. I think because we use them every single day. And basically what you're doing in a recommendation algorithm is predicting what you'll buy or what you'll see or what you'll Stay on. And that's, yeah, basically one form or one expression of ML. Now, what's interesting about 2023 is it's no longer, yes, it's still predicting in the technology part of it, but the actual use of it is not prediction, it's creation. And that juxtaposition is really, really interesting because you're not using it as a developer to predict what somebody's going to click on. You're using it as an end user to create code, to create images, and to create writing. And I think, for me, like, I think those are three of the most popular use cases right now. And so to Santiago's point of like, can it really, you know, can it really do things that, you know, we can't do or we can't predict? And I agree with Daniel. It's. It's a complex answer, but I just want to call that out because that, to me is one of the most inspiring things about where we are today is it's not. It's kind of broken free of that. And it is in kind of this, like, creativity, innovation, space. And that's just really remarkable, actually, if you just think about it from first.

Principles, yeah, I think that's a great segue into the productization of AI, then eventually going into crypto. But, you know, as an engineer, as an artist, you know, architect, whatever, where are you seeing the most amount of usage and, like, efficiency gains that you talk about? Kasey is like, hey, if you're a developer, you all of a sudden can use AI as a companion kind of tool to just superpower, you know, your skill set. And human plus AI is just remarkable or just purely AI is just better than human. But I'm curious, like, where are you seeing these use cases pop up and what are you mostly interested in?

Sure. I mean, I can start very granular, which is, you know, if you walk into the Modulus Office today In a different WeWork, you would see that everybody has on their monitor pulled up some version of a language model, either copilot, chatgpt, whatever, right? Like, we no longer develop alone, which is kind of odd, right? Like, oh, there's this bug that shows up even with these. Like in our case, you know, cryptography, which is very math heavy, very logic heavy, you know, these areas that you wouldn't expect language models to be a huge resource in, we still use them because they're this incredible, accessible first instinct to check against. Do you have any ideas? I'm seeing this error or how would you solve this problem? And so it's lowering the bar just across the board for all of us on the team too, being more productive and being more creative and kind of solving problems faster. So that's one very specific but just immediately happening example of where LMS are already enhancing our workflow.

Yeah, I don't know if this is common knowledge, but I think software development is one of the most disrupted industries by AI. And I only say this because I was an engineer at Google and I think I can. It's like I think I have the ability to. But we're not quite there yet. But we're almost at a point where AI can replace like an L2 Google engineer. And so, I mean, that's insane when you think about the repercussions of that and just think about how that will change innovation, speed of innovation. And even moreover, when I'm diligencing founders, I will basically not. I don't like hard and fast rules, but I will take a very hard look if they're not using AI to augment their software development process. Because the way I like to think about it is if anyone's played Mario Kart, you know how when you get that star and you're like a super. Yeah, that's what it is like right now if you are a developer with AI, using AI and a lot of people, it's interesting. Not a developer, you're like, oh gosh, the developers, they're really at risk now. They're just going to be automated away. That's actually not true right now. I think that's one of the. Maybe that's one of my contrarian views, is that software engineers are more powerful today than they've ever been. And it's going to be like that for a while where like, even though now we have that, you know, level two engineer being replaced, we're actually just so much better at our jobs. And so we can build what used to take months in weeks. And so I double down what Daniel said.

Instead of, instead of needing 10 engineers as a startup like Modulus, you can, you can have one or two very good engineers leveraging copilot and just becoming like just hugely efficient.

Yes.

With that said, everyone in Modulus, don't worry, you're fine. We need, you know, but definitely, I guess, you know, the economic argument had always been, look, there is this massive demand for developers just across the board, which our education system, our work training programs are not able to meet that. That demand. Not even close. Right. But here comes a tool which 10x is every specific developer, right. Whatever their role is within the org, and I think it will broaden to just beyond developers, who even, you know, executives or, or folks in marketing and all that. It'll look a little different, but concretely today, it's already happening for software engineers.

As an example, we can now, finally, after 30 minutes, I know most people come to Empire to discuss crypto, so maybe we can start talking about crypto and how AI is relevant. And of course, Daniel, in full disclosure, both Casey and I are investors and Daniel's company, Modulus, so you hear really excited about what he's building. Well, there's obviously bias, but anyways, none of this is any advice. So Daniel, maybe you can kick us off in terms of. Give us the story behind your thinking around. Okay, I want to do something in crypto and I want to apply AI to it.

Yeah, for sure. I mean, this is going to sound increasingly, you know, ridiculous. It may be the most dated thing that will be said in this podcast, and that is that about a year ago when we first started Modulus, there was zero AI activity in the sphere of crypto.

Right.

And I think that's just going to get more and more ridiculous again. But it's true. Right? And part of that is because, and I think it's really important to acknowledge this, there is a lot of personality tension between these two technologies. Just like architecturally, what they mean for the world, how they like to be, they're quite different. Right. And the other part is, quite frankly, you know, the crypto industry is a younger one, and never mind the fact that AI is just now really coming on the scene in a big way. But there wasn't a lot of AI competency when it came to crypto as you look across the sector. So when we showed up, and as a reminder at that point, all we knew were the kind of AI training we had received at school, as well as all the crypto kind of white papers and papers we were reading in our free time. We thought, hey, you know, we see this interesting technical inflection in the context of zero knowledge proofs. Don't have to get into that right now, but tldr, this seems like the perfect bridge or the perfect tool to bring this emerging discipline around AI to augment and enhance the crypto world. You know, we were always hearing about, like, UX issues and kind of like, you know, it's a high friction and it seemed like AI features, everything ever represented, from recommendations all the way up to LLMs was the perfect antidote. So that was the origin story, basically the inception of Modulus, Very simple. Just how do we bring these two technologies together so that we could enhance crypto but still honor those two personalities and what they're really, really good at?

Right.

And our solution to that is zk. But there are others and happy to dive into any of that where it's relevant.

Well, I guess maybe, Casey, from, from your perspective, because I always tell people that you've been, as far as I can remember, you were thinking and researching and studying AI before it was cool, before it hit the hype cycle. And McKinsey, once McKenzie starts writing about it, it's sort of game over in my mind. And so like from your standpoint, you've been a crypto investor for years.

And.

In your position, like, how do you think about that and how did modulus become. Why did it become interesting to you?

Yeah, and others.

But we can start there, of course.

So 30,000 square foot view.

Very thing to say.

Yeah, here we go. No, but seriously. So AI is one of the most trans transformative technologies we've ever made, ever. And I've known that for a long time, since I've been a kid. And it's an inherently centralizing technology. And the reason it is, if it's not clear, is because you are more efficient when you have GPUs connected co located, you can do more from a fundamental hardware level. And so this has this huge ripple effect of basically centralizing the entire thing. And as a believer of decentralized technology, I've always felt that one of the biggest threats we have, honestly, not to sound super philosophical, but humanity faces right now is centralizing AI to a point that the innovation doesn't compound. And so that was the scariest thing when I was at Google, is like, holy cow, we have all of this to ourselves kind of situation. And the reason it's so important to bring these two spheres together, to Daniel's point, is you have decentralization, which is a naturally decentralizing tech. You have AI. And that might be the best mechanism or the best method to keeping AI open and accessible. That is one of my biggest goals of what I'm trying to accomplish. And so when Daniel and I originally connected, I mean, modulus has come so so far so fast. But I remember you guys were one of the first ones to put like a small neural net on chain, I think, and I had played with it and you know, whatever you came up from Stanford to visit and just the interests are kind of this meeting of minds, thinking, okay, these technologies need to play nicely. And yes, I know AI people maybe don't love crypto people, but we need to start bridging these worlds because it's going to be one of the most important things to do in our lifetime. And so like I can dive into kind of like how I view the market, but just that's like the substrate to how we've gotten to where we are today or how I feel.

Yeah, let's do that. I mean I think the important thing that I heard there was inherently technology is centralizing force. Crypto is sort of pulling that back and trying to resist that natural gravitational tendency for things to centralize. You can be idealistic, which I think everyone in crypto to some extent fails because you then don't become pragmatic around. Yeah, we can talk all day about how nice it would be for technology not to cluster, not to, you know, become centralized. But the reality is, economically or not that this just may not even we can hope for that, but can it actually work? So maybe this could be a good time to unpack a lot of what you said case and Daniel around, you know, specifically like what is it about zero knowledge proofs that make it relevant, make it actually powerful enough to apply that in others.

Yeah, yeah. I think Daniel, let's start with just like on chain verifiability and what that unlocks and maybe. Yeah, if you want to. That's like one very, very exciting use case of this intersection, like proof of.

Humanity or whatever, you know, I don't know. With your keys.

I don't know. I mean I think just stepping back for a moment, one way to think about Ethereum or you know, take your blockchain network of choice but Ethereum, for the sake of this argument, one way to view it, look at what it is is this incredibly, incredibly robust public record table, right. Where you can just log whether transactions or state or whatever, where everybody can check and take a look and go, okay, yeah, yeah, I can come to a relatively high confidence belief that what's on this record is accurate. Right. And so the capacity for this network to be this great common ground to the world's kind of appetite for high integrity things is a big part of what we try to celebrate at modulus. Right. And I guess I don't know if now it's too early to of take a peek beneath the hood at what zero knowledge cryptography is and why we think it's the right tool for this. But just to give a super, super succinct, I guess, framework for thinking about things, zero knowledge proofs for us is A capacity to make account of things. And the blockchain network Ethereum is just where we log the accounting basically. Right. And all of this is happening in a really trust minimized way. And we think it's an excellent tool to do exactly what Casey mentioned earlier, which is to counteract some of the centralizing influences and tendencies, the nature of something like artificial intelligence. But Casey, maybe you have a different take on public verifiability that we can also work into the framework.

Well, I think something maybe just recent that happened, which is pretty interesting, is I don't know who's read Biden's AI Executive Order, but it's worth reading if you haven't. I also have a summary if that's helpful. But one of the most potent or salient things that I took away from that was the government sees the need for provenance moving forward. And I think verifiability is one of the methods that may achieve that. And I think there's a huge set of challenges with it. But I think it's one of the things where I think if you're playing the pro case of this is like it's not a problem looking for a solution. It has already by the government and by the general public been identified as one of the biggest problems with deepfakes, just model accuracy, model credibility. And so having a system that's accepted to verify them makes total sense moving forward. And I think it's really, really dangerous to predict what's going to happen in AI next. This is one of my more interesting or something that I've been chewing on recently is just that AI has basically been the reverse of what we expected. So like in the beginning of AI we really thought that, or at least like my cohort thought, like reinforcement learning of robots was going to be one of the big first killer acts. We had this kind of revolution with like Boston Dynamics and that whole era. And what ended up happening was robots still are extremely hard to control. And what we one of, one of the first killer use cases of AI is human empathy and relatability and accountability. And those are the things that we thought was going to happen last. And so we're having this like total paradoxical thing with AI where the things we thought would be easy are hard, the things that we thought were hard would be easy. And so I say this as like a caveat of let's have the conversation. And I'm happy to tell you what I think is interesting. I also think it's really dangerous to be prescriptive with such new Technology new in the capabilities.

Yeah. And so the verifiability, the provenance of that, I mean of course the simple lizard brain of mind thinks, okay, blockchains are data rich. To your point, Daniel, the applicability of blockchain is such that anyone can verify. This is what makes blockchains blockchains. If anyone can go in and verify the state and the transition, you know, per block, you know, then, then that makes it really powerful because it keeps the system in check. It's open, it's accessible, it's transparent. Not only that, but over time, if you think about blockchains, they will produce an inordinate amount of data that it could be used for training and other things. And so. And the data is sort of pristine. Like you have these guarantees that it hasn't been tampered with. Of course now most of it has been transactions sending Bitcoin from one address to the other. But it doesn't stop there. I think you can embed a lot of data into these blocks that can make it useful. So back to maybe zero knowledge proofs or how you're actually thinking about modulus as a product. What is the product? What exactly are you selling or building? I guess, yeah.

I mean one thing that I think is packaged within the nature of what machine learning models are in production, we call it these centralizing influences is also the fact that they're like black boxed models for the most part. They're very opaque. It's very hard to predict a specific set of behaviors. It's very hard to point out fraud when this happens, or any type of manipulation, whether, you know, intentional or not, never mind bias and all the rest. Right. These are all like active challenges when it comes to machine learning. And so when you think about applying AI models in production in the real world, those risks just comes bundled in. Now it might not matter so much if it's model trying to pick out a nice outfit for the day or having a conversation with you about the weather or anything like that. But our belief anyways is that it will really matter when the stakes are high, be it financial, legal, medical, some of the contexts that we mentioned before. And one very clear beachhead for these kind of high integrity environments is blockchain based applications. So if you're running a dex, you're trying to balance some pools, anything of that nature here is exactly where that attack vector will be really present. So to answer your question Santiago, what is the product we're working on? Well, were using zero knowledge cryptography to basically prove that an AI model executed the way that it was supposed to. There was no manipulation, there was no tampering. Even if it's a black box model, so to speak, it sits in that box and nobody kind of prodded it to act in a very bizarre manner or in a tampered way. And we make that result something that is both verified on the blockchain. So you know, from now until forever, anybody can verify that that specific AI result acted correctly.

Right.

In accordance to the pre agreement and secondly making that result, you know, native to the blockchain, basically something that can be digested by any DAPP or composed with, you know, with all those properties that we really appreciate about operating on chain. So we call this idea zero knowledge machine learning or zkml. And you know, it's early innings, but we think it's one of the most, it will be one of the most powerful kind of building blocks to not just blockchain services in the future, but how AI is used just society wide.

I think that was so well said. And I think one thing to just put it one step further is once you have this, which gets me really excited, is that it unlocks the rest of the crypto ecosystem. So if you can verify, then there's all these services like Defi for example, that you can plug into. But you can't really do that if you can't verify it. I mean you could, but it's like it's a total unlock and so it's valuable for the output and then it's also valuable for what is on the other side of that. Verifiability.

Exactly. I mean, you know, the naive way of naive mathematically not to make light of anybody's execution, but the simple way to do this is to just write your AI models on the blockchain. But if you're familiar with gas fees and congestion and block space.

All right everyone. So we talk a lot about the institutions coming into crypto on Empire. Santi and I are both headed out to London March 18th to 20th for Blockworks 8th ever Digital Asset Summit. Das. This is an institutional buttoned up conference that we've hosted since 2019. I like to joke that it is probably the last remaining kind of suit and tie event in crypto. People are still wearing suit and tie. It's pretty funny, but you'll actually hear from a lot of the largest institutions in the world coming from Standard charter, fis, JP Morgan framework folks coming out win or mute. Vaneck, Goldman Sachs. There are a couple big themes of this conference. 1, Bitcoin catalyst the having and the spot ETF 2 a view from the buy side 3. RWAs tokenization and stablecoins 4. Global regulatory frameworks 5. Institutional infrastructure including banking and payments and 6 the macro case for crypto. If you have anything to do with the institutional side of crypto, you have to be there. Santi and I got your back. We hooked you up with a 20% off code. It is Empire 20. There is a little competition running internally at Blockworks to see who can drive the most number of tickets. So help Santi and I out, register one with our code and you get 20% off. That is Empire 20.

I was going to ask you just that. You know, we can. As far as I can remember, zero knowledge proves to been this sexy holy grail. But we always bump into how is it cost efficient? Can we do this at scale?

Exactly. Yeah. And actually Casey mentioned this when we first met, we had just put a tiny, tiny, tiny neural network on chain using zero knowledge proofs. So the model is running on a server like anywhere else. But we proved that the model execution was correct and we put the proof in the result on chain. But even with this scaling technology, we were limited to very small models. We had spent a lot of this conversation talking about LLMs and we think there's a lot of applications LLMs to especially high integrity results of LLMs to the blockchain network. But the big question is how do we unlock, you know, access to these massive models, these highly performant, super expressive models with this zero knowledge kind of, you know, framework or zero knowledge path. So that's what we work on full time, basically just trying to figure out what is the right way to make that unlock happen and open up access to these really exciting developments in the AI world. For the on chain audience, especially Kasey.

I mean a lot of what you and I talk about, you know, we've known each other for, for years and, and invested in crypto together. And I think you have a uncanny eye for bullshit as well, you know, because in crypto. No, I'm, I'm saying this with, with, you know, crypto is one of those spaces that is, is so generalized and so exciting. Maybe like AI where it is easy to attract a certain type of builders that, you know, there's a lot of fluff and there's a lot of out there. And look, anyone listening to this might have come with a certain degree of skepticism around. Here we go. Crypto's that industry that anything that is hot crypto people are going to take and just reformat Repurpose and trying to extract money, but not actually build real products. That's just the criticism. If you were to ask most people that are outside of crypto, that's what they would say. How real is this? Are we going to see this in production? Is the opportunity there? And when I say opportunity, it's not the ability to make money because we have cardanos of the world that have not shipped and have, you know, valued in stuff that, you know. Anyways, how real is this and how, like, when you think about the opportunity versus investing in other areas of crypto that are perhaps also exciting, how do you see this as an investor? Yeah.

So my answer would change depending on what angle I take. If it's like the AI first or the crypto first. And I'll say, for every one Daniel I meet, there are 99 people who are just trying to exactly do what you said, which is take crypto and attach AI to it and capitalize on it. And so, like, I think you and.

I have seen collectively, like hundreds of decks.

Yeah. So I see one more decentralized GPU deck. Oh, my God. So let me just say that Daniel is like a very, hey, like, needle in a haystack situation. And we, we live to work with people like Daniel. Right. Who are really here for the right reasons and want to see the change in the world that we want to see. How real is it? Let me just. This is like my dependency chart in my mind. AI must stay open. It must. We have to do that as a society. Crypto in blockchain, decentralization, web3, all of those are very good contenders to help us achieve that. And so how real is it today? There isn't too, too much. I mean. And I'll walk you through, let me walk you through, like, the six things I'm looking at right now. But we're very, very much at the beginning of what this intersection could look like. And you're going to be too early, too early, too early, and then you're going to be too late. And I think that transition is going to happen really quickly. And that's why I'm spending time here, is because it is one of the most interesting asymmetric, I think, opportunities we have in crypto. And I think it's a really good contending technology for AI. And the principles of Web3 are going to be leveraged. It's just cardinal to make sure this technology reaches its full potential.

Potential.

Does that answer it in a. In a way that's satisfying, Santiago?

Well, it could Be satisfying to me. But is it satisfying to the world? Yes, I, I think what you just said is very, I believe it. The two things that you said that stuck for me were crypto is a very, probably the best contender that we have to keep this technology open and to resist that tendency to go and, and centralized. Of course you could do that the other way, which is by enforcement through some sort of act, but that hinders innovation locally but not globally. That could, you know, you could stop GPUs from being sent to China, but still the technology always finds a way. And the second one was, I think, you know, crypto is open source and continues to like. The innovations that we've seen in crypto, like an AI, have felt really slow for 10 years and then very fast. And that is just inherently, I think, the state of technology. And I agree with you, I'm spending time on it, even though I'm not as sophisticated. And so I lean on people like you to help me understand this space and weed through a lot of the crap that's out there.

Well, I, I can personally attest to, I think both of you being extremely high quality operators by my, my impression. So with that, you know, set aside, I, I do want to make an argument here as well though, that I think the end state here results in like a product with a capital P that is just also better. Like period, end of sentence. Right. And if you don't mind me spending like a minute or two on this like mental framework. You know, we spend a lot of time in the world of cryptography and it's worth thinking about what is it that cryptography is doing for us, us in the context of zkml, but also just your knowledge in general. And actually I think it makes sense to visit just what AI models are, even for me anyways, at the extreme, what an AI model ultimately is meant to become is a capacity to scale human decision making just forever, for free. Basically, wherever you want a human to be making a decision, you can bring in a model and have that happen for you, but with even higher performance, which is weird to say about a human being, but nonetheless, this is very exciting, of course, but it's also a little bit problematic, like this idea, because let's say that Casey Santiago, we walk into a court of law and there's a judge giving us a sentence. The three of us committed some terrible, terrible crime, except we didn't. And we want to make sure that the judge is making a fair decision. The way we would do that today is we will look at the record of this judge, we look at all the decisions that they have made before, we look at their credentials, where did they go to school, what law school. We look at their habits, any scandals, whatever. Because human beings have this incredible quality of being really discreet. Like here is a person and there's this web of infrastructure and trust that we can use to evaluate a person based on their past behavior. Compute suffers no such luxury, right? Compute just floats out in the ether, right? It's model V1. It's model V1 1, right. We change a couple parameters, we tune some, you know, some inputs and the output can be radically different. Right? All that to say that for me, anyways, one of the big opportunities here where the convergence of crypto and AI literally makes the output better is where we have models that are now accountable, right, that we can look back on the history of that model, we can test it for robustness, for bias, we can look at all the LLMs, which is consulting this lawyer, all its past recommendations, and make sure that it is fair and it's not make hallucinating case studies and whatever. And so I think taking away all the kind of aspirational ideals we have around decentralization, taking away kind of where we're concerned with, you know, the progress of technology and how it's going to interact with society, just looking at this product now, right, An AI model which is now much more accountable thanks to the work that we're doing in the crypto world, I think it's, you know, for me anyways, it's like, okay, this is a future that I actually think is better. And so that's a big part of why I think, you know, this stuff is worth engaging with and work worth working towards.

Yeah, on the verifiability piece, I totally agree with you. Like, if you can have a high degree of certainty that the model did exactly what you wanted to, and you can trace provenance of what resources it consumed to get to that output, then it becomes highly relevant. Particularly for the example that you described around, you know, you're sentencing someone, there's inherent bias, you know, there's maybe an AI model can help us compare the decision of a human to the one of an AI that may be less biased and then know, contest that outcome. I'm again going back to the, the, the, the challenge of crypto has always been scale and doing these things effectively. You, my understanding is you've been focusing and tinkering on the Ethereum blockchain. Now there are competing L1s that, you know, Are more performant or claim to be more performant? More. You know, I'm. I'm curious in that decision. We don't have to spend a ton of time on it, but I, I'm curious if you have a view on where we may see these applications actually work at scale and be your actual product with like, profitable unit economics, if you will.

Yeah, that's. That's a really good question. I mean, I think, you know, concretely, we spend basically all of our time reducing costs. Right. Increasing performance. Right. We want the, the ZK desk to be as little as possible so that we can bring more performant models to these service. I will say almost all the partners we work with today are on Ethereum. And the reason is actually even less about the performance of the underlying chain and more about the culture. Right. The culture of Ethereum today, although I'm sure this is true as well on many of the other networks, is that of security matters, integrity matters. We want to make sure that we have a stack of smart contracts that compose the foundation of our service, and actually we don't breach that security that's provided by that point, suddenly referencing an AI model that is centralized or closed source. Right. We actually want to bring the ZK in. So we maintain that security, this is something that really, really, really matters to the Ethereum clientele or the Ethereum audience. And so we found a lot of cultural alignment there. That is not to say that that is the extent of our work. Right. One of our other partners is worldcoin. In the context of that collaboration, the proofs aren't even suddenly launching. Actually just trying to convince the World Coin servers that when people attest to their iris scans or attest to their identity, they're not doing that with a manipulated IRS or manipulated identity.

Right.

So, you know, early innings were really just at the beginning. But my expectation is if we can do a good job here, right, by showing that high security, high integrity AI matters and improves DAPPS across the board, we'll see that pop up in Ethereum and Solana and all the other ecosystems.

Casey, would love to get your map of how you think about all the different. You've seen many decks, some of which are, you know, as you said, decentralized kind of compute networks, and something like, how are you thinking about the evolution and the opportunity set in crypto? What's your time horizon? Who are the customers here? Are they crypto native companies like worldcoin or may this be a beach hit into actually pulling in less crypto native people and Maybe them not even thinking or you know, thinking that it's crypto, but just being very focused on, oh, verifiability is something that I can really wrap my head around. And I like, don't tell me if it's crypto, just sell me the product.

Yeah. So I do think that was spot on in that. One of the more exciting things to your question earlier of why spend time on this category? This might be the most successful category to bring non crypto people into crypto. And the reason is because if you look at what are the killer apps of crypto today, in my opinion, like the top two are beyond all ones are NFTs and DeFi. And both of them are inherently crypto native. You know, they weren't from the real world, they were built on chain. And this could be the first subsector to be prominent in a bull run where it's not crypto native. It's actually coordinating things that are outside of our native world. And that is so exciting. But yeah, let me tell you. So I keep a list of the categories of decks I see and kind of the the sectors that people are going after. And here's what I'm seeing the most of and I'll try to keep this brief and I can provide commentary on kind of the pro and con case of each of these. But the first thing as I alluded to was decentralized networks of GPUs. The case here is could we build basically a crowdsourced uber Airbnb for GPUs? And from a technical standpoint it's hard because of what we said where you get so many compounding effects by GPUs being co located. But there's definitely a case that certain use cases that can handle certain latency and just take more time to train. That could work. So that's actually quite interesting. And then the next thing I see, a lot of which I love, is local and peer to peer inference. So if people aren't familiar with inference, inference is basically when you get a result from the model. So you do the pre training and then you run inference which you can either hit an endpoint or which just anything that gives you back the result. What we're seeing with some centralized companies like Ollama, which allows you to run llama2, Mistral, other open source LLMs locally, that's becoming a huge trend is that people say, you know what, I don't want to send my data through this hugging face or replicate endpoint or I don't want to have to rely on WI fi to hit this model. There's so many reasons that they're like, you know what, I just want to run this locally, I just want to run this online device. And that trend of edge computing is not only just happening, but it makes you think, well, what's the next step of edge computing? Well, it might be peer to peer inference. And that's something that a decentralized network would be phenomenal at. And so that's super, super cool. And then the next category I see a lot of is just crypto incentivized behavior. So think about crypto incentivized AI apps where basically we use crypto to incentivize or bootstrap a network. Crypto, incentivized RLHF, ML ops, all of that kind of category which I put, I can do subsectors, but it's basically just taking what one thing we're really good at is promoting certain behaviors and taking that property and mapping it to AI. Then we already talked about verifiability. And then the two final things I see a lot of that I'm excited about are, I would call it self owned models and data and kind of to the point of data being this really lucrative yet unregulated asset. How do we use crypto and blockchain to be able to give power back to the people who actually create the data? And it's not about power, it's about so much more than that. But basically no, no, you go, no.

It'S more like agency, like understanding who's using your data and opting in or opting out and, and actually getting some modernization out of it.

Exactly. And you can imagine a world where just like MetaMask, today we take our MetaMask from crypto app to crypto app to crypto app. You could do the same with your data and models. You could use it on Facebook, you could use it on, you know, whatever the next like stability AI is, whatever it is. And like it would be this portable thing that you take with you. And so that to me kind of makes sense if you squint. There's also a lot of problems there. And the last category I see a lot of is like agentic use cases where basically for people unfamiliar with agents, they're kind of like autonomous ML models that can make decisions and they in a lot of forms are always on, they operate in the background in a lot of ways. And so you can think of how it might make sense for AI agents to use crypto natively as their Payment because they can't really kyc and that might be part of the agent infrastructure that becomes fortified in 2024.

Yeah, the thing that I for me was very exciting was this idea of proof of humanity. You know, you control your keys and you can attest to, you know, hey, I'm human in some way shape or form. And I think you can do that with, you know, cryptographic keys, if you will. Like somehow, I guess the going back to all these examples, while I agree with them, but what would, you know, we can, like what would derail the case for these? Like what are the cons? Are they just purely performance costs? Like if you're a corporation, you might wish and hope that you have more agency or it's more decentralized, but at the end of the day, like if, if a centralized provider is just more cost efficient and performant, I think humans time and time again make these trade offs. Everyone uses social media whether they understand that their data is going to be monetized. You know, it's a free product. So ultimately that wins. For better or for worse. You know, this is an option. But in terms of market size, do you, do you see this actually competing against the centralized solution?

Yeah, but we need, we need more information. We need to see right now a lot of the crypto providers that I talk to in the, for the decentralized compute, for the use case you're bringing up, they're subsidizing costs and that's not sustainable. You need to economically beat the centralized providers or you need to provide another value function which enables the switch. And so I completely agree with what you're saying.

Let's talk about decentralized compute numbers because it is a hot topic, right? You have IO and you have Akash and you have. I've seen like 10 decks in the last two weeks. The hot TLDR is it is extremely hard to get access to like GPUs. It is a hot commodity right now. You cannot find it. And so if you're a startup, you just raised a bunch of venture money and you want to give it to Nvidia, but it's going to take time. You can have this as Airbnb unlock this idle inventory or file coin, right? Like distributed file storage systems is, let's rent it out and use all this unutilized. And crypto is really good at coordination. So you can all of a sudden even if it's not co located you can get the similar performance. And there's a whole subset of people in the market right now that don't even have access to GPUs and are going to now have access to GPUs and they want to pay for that. And so you're just creating a new market opportunity. How real is it? Like, how viable is that? Is it just a short term bottleneck you're fixing or is this actually an enduring product over the next couple of years?

I think it's less real than people think. I'm going to be bold. I think that there's a lot of centralized providers out there, like vast AI that I can hop on and get a GPU. And yeah, maybe H1 hundreds are rented out.

But you may not need the H100. Is this Nvidia like Ferrari?

Right.

No one can get access to analogy. Right, but it's the. If you want the Ferrari for this stuff, it's the H100, right?

Yeah, yeah. A100 is hard to. But yeah, so those are hard to find right now. But there is a lot of GPU marketplaces, there's a lot of centralized providers too that are doing this. And so you can get on like banana.dev or like runpod or I mean, I could list a million of them. And yes, then you have the akashes and the IO.nets and the renders and all of them and, and like Bittensor I think in the future. But the thing that needs to be proven is that you would rather go with a decentralized provider than a centralized marketplace. And there are some early signals of that. There are actually some interesting case studies of technology companies that I won't dox on the podcast, but they're actively choosing to go with decentralized providers for a variety of reasons. But we need to see that sustainability sustained and crystallized because that, that's the most interesting thing I've heard over the last month is getting on the phone with these tech companies that are, you know, based in the Valley. They're just regular AI companies. And they're saying, yeah, I'm going to go with this decentralized GPU provider and I'm going to end my contract with, you know, my other burst provider. And here are the reasons why. I mean, that's like, that's isn't the.

The biggest reason I've heard consistently across these companies is like, it is very localized. So if you want to have access to a GPU that's in Europe, you have to have a different service contract and just the time component becomes faster from a decentralized standpoint because you sort of transcend local.

Yeah, I mean, you don't have, you don't have to do anything. Like, it takes me two seconds to spin up a GPU on a cache. So there is like zero friction in that. And then the other thing is just for people that are scared about the future of AI regulation, this is like more of an ethos thing. But if a CEO is scared about where the US is going with AI reg, they're like much more apt to go and invest in a decentralized architecture because they just want to be safe.

Dan, you've been quiet in the background, but I know you may not, but I want to get your perspective as a builder where you see beyond what you're doing, other opportunities, maybe other founders you've interacted with of real. Maybe the real more interesting question is if you had to pivot today into something else, crypto, AI, what would you be doing?

Yeah, maybe just because we're on the topic of decentralized compute networks and so forth, I would say the challenge actually is not so dissimilar from ours in the sense that there's like a science challenge here, which is that I think Casey is totally right here in that fundamentally, I guess I. This is my sense of things in general anyways. Ultimately it's just about like a better product anyways. Right. Like, if I'm a, you know, if I'm. If I'm running a startup and I need a GPU cluster to train my models on, I'm going to choose a provider which is cheaper, more robust, you know, maybe not more, not as geographically constrained and so forth. Right. And if that happens to be a decentralized network, I don't really care what's on the back of the box. You know, the front of the box is good enough for me. Right. Let me sign this contract and just get rid of this problem. And so insofar as you know, there's another player in the space, for example, called Jensen, right. Which is trying to do this like decentralized GPU training network.

They're more focused on training, right? Not inference.

Yes, they're focused on training and specifically of training large models. And in that case, although the cryptography is complex, it's a very linear challenge. If they can get the cryptography to a low enough cost standpoint or low enough unit economics, so low that it's better than the centralized provider. I just raised a billion dollars to build some cool LLM or foundation model. Why wouldn't I choose this decentralized network? It's 20 to 30% cheaper than going to Amazon or Google. Google or whatever.

Right.

All that to say that. I think that the funny thing is, ultimately the best product wins right. Over a long enough time horizon. And that's what we are about. Right. It's just that the cryptography or the crypto aspects of a lot of these decentralized networks are so early, they're still in their infancy, that we need to oftentimes juice these incentive systems to try and get people to migrate to our networks. But to Casey's earlier point, that's not sustainable and it obfuscates product market fit, which is very, very dangerous. You get addicted to the drip. Right. So to answer your question, Santiago, if I had to pivot and work on something else today.

I don't know what.

I would be doing. I'm absolutely obsessed with our problem statement. But it would also most likely be at the intersection of AI and crypto, just because it's such fertile ground for so much. And I think being focused on the science, those kinds of unlocks are meaningful and moves the whole industry forward.

Yeah, I want to emphasize something you said, which is incentives. And it's something that is. Obviously more people are critical about these days, but I don't think there's anything wrong if you use incentives the right way as a pull mechanism. And once people. Because some, a lot of times these networks become more efficient and so. But you need to kickstart them. And if. And you solve that like that like friction to get people in the network. And then once that's the case, like any platform, you need to solve for supply and demand. So you can do that with incentives. Like Uber was giving away free rides and then subsidized. Like this is the way networks get off the ground to resist the inertia of switching and stuff. So there's nothing wrong with crypto incentives. I feel like somehow that has become taboo and I think people really miss the point. Like that is the most powerful thing that we have is coordination in a more efficient manner. Kasey, I want to perhaps go back to something you said, which is a lot of times, as particularly you said about AI is things will look like they're slow and then all of a sudden you believe that there was going to be a very fast unlock and development around these things. And I wanted to ask both of you this question is, as you think about the next two, three years in crypto or just in general, like, where might we see those huge unlocks, whether it's specific to modulus or other things, how does that Impact crypto overall and specifically like crypto AI market.

Yeah, I think the biggest thing we're seeing in AI is or what we'll see is the productionization. So right now a lot of it. So it's cool. And everyone's having these aha moments this year of ChatGPT and you know, this first agent, but they're not really production level ready. And that's maybe an under appreciated fact. And so I think we're going to see a lot more robustness around us bringing these into enterprises. And so this is like a total, in my opinion a retail driven phenomenon right now. Like AI, of course there's a lot of enterprise applications but in terms of like what people are talking about and excited about, that's retail. And so what needs to happen and like not to get too nerdy here, but what's so different about or what's so different about AI is that the whole thing is non deterministic, meaning that you can't say what the output is going to be. It's, it's a Bayesian situation. And when you just was that with traditional programming, those are rules, these are logic, these are pure functions, you know what's going to come out. And so from a programming standpoint the whole thing in my opinion kind of needs to change. And this will loop back in that the reason things are toys is because we don't know fully how to handle the non deterministic nature. We're figuring this out on the fly. Like the amount of things I have break on a daily basis because I think I have this agent running for me, me and it just runs into this edge case and then I don't know how to handle it. And so once we figure out how to put the guardrails up in a way that we can really trust these systems and say, look, if it does hallucinate, it knows what to do and you know, you can really, really trust this. That is going to unlock the next level of use cases, I think. And like that's just the answer to your question and then how crypto will play into it. I mean again I think it's dangerous to predict here, but it's just like I think that needs to happen before we can even have that conversation. To me that's like the necessary next thing for us to advance. Daniel, I'm so curious what you think about this because. Yeah, I haven't said this out loud too many times.

No, no, it's. Yeah, it's definitely one of those like underappreciated features. Of just AI in production, which is a non determinism. It's a challenge when it comes to Z approving as well. Right. Just explaining that. Right. Like smart contracts, the way they're composed is like, like they're, they're super rigid. That's the whole point. Right. Like super predictable, you know exactly what's going on. And you know, in our world we're entirely obsessed right now with how do we enhance DAPPS with AI.

Right.

And so trying to bring that degree of uncertainty into a regime. Right. Some, some say like these two should never mention mix because of this.

Right.

But try to do this in a way that's responsible, high integrity and again, you know, respects the ethos and the personality of both disciplines is really, really tricky. But I guess to answer San Diego's question in terms of what I'm looking forward to in the short term, it's exactly that, that convergence.

Right.

I think 2024 we're going to see a ton more Dapps with AI features natively just like part of that service stack. Right. Whether it's recommendations, whether it's risk modeling. You know, we just started working with a company called Upshot on, you know, appraising NFT values. Yeah, yeah. And automating these kinds of markets or a project called Ion on digesting like the risk profile, trying to predict slashing with AI models and then you know, issuing stablecoin based on that, that sophisticated AI output. Basically an insurance company on chain. Right. These kinds of products and services are literally impossible without AI, without the ZK attestation that the AI model is high integrity, without manipulation and of course impossible without the rest of the crypto ecosystem to support the inputs and outputs to that service. So 2024 is going to be a very weird year in a good way and maybe sometimes in a bad way. We'll find out.

I see you thinking. Casey, I want to interject here for a minute in terms of. I'm very focused on security. Coming from a defi background, I think that any industry has been handicapped insofar it hasn't figured out security and insurance was instrumental to as far as back as you can think about, any technological revolution required more sophisticated risk management. Like think about explorers, think about, you know, you had to have insurance markets to really. And crypto struggles that even though smart contracts are highly predictable, there are bugs, there are edge cases where a human may not see it, but crypto is a very adversarial environment and these exploits get exploited and things break and they're not fun and so like, I invested in a company called Test Machine and they're very much focused on ingesting all the data that is happening with Defi and all the hacks that have happened and being predictive and being not only reactive, but the hope is that over time you actually become preventative and can, you know, if, if the exploit is happening across a variety of blocks, which is where most exploits happen, then you can at least contain the damage that is being done. And I think this ties into what you were saying, Casey, is this guardrail. Like, if you're, if you're going to make recommendations around the health score of a particular alone on aave, you want to make sure that the AI is not corrupted or it just goes crazy on you. And like, you know, I don't know if that's even possible. I'm not technical enough to understand, like, how you would just have the high degree of certainty that the AI is actually going to be working for you, not against you. If that's a thing, that is a thing.

And Daniel's company will solve this for.

Us in the end.

In the end.

That's the goal. That's the goal, yeah.

Because the verification piece I have not but like super high degree of certainty that this particular model will prevent liquidation for me. They will fix my health score, will fix my loan to the point where I will not be liquidated on chain.

I mean, you can also add. And by the way, this is very, very common with, with AI models in production. You, you literally add guide rails to it. You say, hey, look, if the model predicts, for example, a number below this threshold, just ignore it. You know, let's just keep to this threshold. And these kinds of like post processing features we also bring into the ZK circuit. So, you know, it's part of the commitment we make to the smart contract. And by the way, this is the kind of AI output you're going to be seeing. And when it's not this kind of AI output, we're not going to accept it. Right. That is the base unit of promise that the ZK proof makes to the program that's on chain.

Yeah. Also just on this point. So, Santiago. Yes. I think this is going to be a topic for, well here on out until we really reach more confidence around security of models. But I'm going to give a call out to my friend Josh Payne, who was texting me yesterday a few days ago and was like looking into LLM insurance. And I don't think I've ever heard this before until ham, but this is going to become a. A thing, I think, where you basically have an insurance policy around how your LLMs behave. That's one possible. I don't want to say it's. It's definite, but it. It's possible that that's where the industry goes with security, because it's such a big risk and because we're dealing with non determinism for now, you might end up creating an entire new market around that.

So if a company has a chatbot that you're like a travel agent and you make a certain recommendation on, all of a sudden it creates a whole problem. You get sued because you know it wasn't a human behind that, then you might have insurance towards that. And so that increases the willingness for our customers to adopt AI as a. As a product and have certain guarantees that if things go haywire in this early beta phase, then at least you're covered.

Maybe.

Yeah, that would be. I would be really interested in that, actually. I just love insurance. I think it's one of those things that. Especially in crypto.

Yeah, we can talk offline. I. I like it. An interesting vertical.

Yeah. Yeah. I've been trying to convince insurance companies to insure against crypto risk. It's just still not there, even though I think the opportunity is great. This has been a fascinating discussion. We've been at it for a while. Any last topics that you want to cover that we haven't covered? I mean, we could probably make a second segment of just going very deep into the different types of kind of verticals that we're seeing. But. Yeah, I'm curious if there's anything else that is worth talking about in this pod, at least for now.

I'm actually just curious just to put it back on you, Santiago. Like, you're putting all the spotlight on us, but you also are extremely bright, have been in this industry forever. And I just would love to hear from your perspective what. Yeah. What's keeping you up at night or what's most motivating to you in this intersection? And you alluded to a little bit of it, but I would just. Yeah, I'd love to hear more.

Yeah, I've just been. Perhaps. Well, a lot of things keep me up at night, but as it relates to AI, I'm very excited. I mean, I'm wearing this sweater, like, which is effective accelerationism. This is the only thing I got for Christmas. This is the only thing I wanted. My parents asked me, so I was like, yeah, I want this because for me, I just don't think that we should stop this. I think it's incredibly exciting technology and, and I do think that there are like governments for better, for worse. I don't think have a bad intention. But crypto at least has always been for me like that fourth stool. Like if you think about the three stools of Montesquieu of government like judicial, executive and what's the other one, Legislative. I think crypto is that third, like that fourth stool that really allows civic participation and oversight and like to keep democracies in check because you have an alternative because there's always that transcends local man made rules and limitations. When you have a decentralized network that supersedes that. It's not to say that it blows up the traditional structures, but at least it keeps the system in check. Whether it's just having more flexibility to go and rent out a GPU in Japan. If you're in the US like that, that's just like abundant, that just creates more efficiency, removes friction from the system. So I'm constantly in the lookout of like how can we implement AI, which is delivering a lot of efficiency gains if you're a developer, if you're an artist or whatever. And how can we embed that in crypto, which is sort of like a tale of two cities. It's like crypto is like opening up many different like possibilities but it has always been handicapped by scalability issues and cost and there's friction there as well. So can AI help us like transcend that is is something that I'm focused on. But yeah, like again to what I liked about modules, like the verifiability piece I think is going to be incredibly important like having certainties of like provenance and where the data is coming from. And so I'm more excited about that than just like the next decentralized like you know, compute network to like, you know, rent stuff out. It feels more enduring to figure out like the hard problem is what a modulus type of company is trying to fix. And I'm more excited about meeting those people and figuring that out, even though it may take more years. But I think that's where the real opportunity will be in my mind.

I'm very invested. That being the future of course, obviously. But I guess I don't. Since we're on a topic, I'm curious, Casey, what keeps you up at night? You know, I hope it's not AGI, but you know, maybe it is.

It's closing the technology. It's actually pretty similar to what Santiago is saying. It's just the. It's like we have one life. We're here. Let's see what crazy shit we can do. And in general have. I think that I personally believe in morality and I want to do good in this world. And so I feel that this is one of the biggest hurdles we face right now. And so I am quite scared about regulation. I think there's good intentions in regulation, but I think after reading the executive order, I'm nervous.

Scary. Yeah.

Yeah, I'm definitely nervous. And so that's. I also, I. I want to give a little bit of homework if that's okay, Santiago, or not homework.

We always give homework in this pod.

Yeah, we do. Okay. So I feel that if I'm a really big believer there's. I think there's like two ways to learn generally people can read or people can do. And I'm a big doer or builder and that's how I learn. It's just like playing with things. And so I think if, if anyone listening hasn't done this, there's a few tools in AI and I'll just do this off the top of my head that if you haven't used, just go take the 10 minutes to do it. So everybody's probably done chatgpt and it. Yeah. If you haven't. Yes. First up, go to Mid Journey or go to Discord. Get onto Mid Journey and generate an image. Then go to 11 labs and clone your voice in less than a minute. Okay. Next go to Perplexity and instead of going to Google and ask anything that would require knowledge that isn't within GPT4 and maybe then if you're still interested you could email me and I have more tools for you to use. But just even just those four. Yeah. And. And that will just like make everything we're saying more tactical. I think some people are abstract learners but a linga learning lot more people are more tactical learners. And so I just want to give those as like starting off jumping off points to go into the world.

It's a great idea.

I mean ChatGPT is like I think from a number of users and go to market standpoint it's been the startup that has gathered the most amount of like active users ever or close to being just how many people downloaded this thing are using it on the daily becomes some people have gone as far as saying it will replace Google as your primary search as your primary way to find and acquire knowledge, which is. I could see that version happening. I wanted to end this in maybe a rapid fire But I'm debating maybe we should do that rapid fire. Like a whole set of questions.

Sorry, have the debate or do the rapid fire?

Well, maybe. Maybe not a rapid fire. Because that might be challenging. I may be curious in finding something that you guys don't agree with. That as far as. As far as AI and maybe crypto AI.

Yeah.

Yeah. Between you guys, is there something that in your conversations you have not agreed with? Or maybe not just you. It could be with someone else, maybe within your team, Daniel, or Kasey, with your network of people that are at this intersection. Maybe you. Because, Casey, I think you sit at an interesting vantage point around, you know, core, like hardcore AI that don't care about crypto. And you also know crypto like crazy people like me that you're at that intersection. Like, what are the things that are the most contentious beyond, like, will AI destroy us or not? Because I don't want to get into that discussion. I think no one really knows. But I am curious, as it relates to crypto and AI, what are the things that you disagree with with people the most?

What's your opinion on model marketplaces? Got it.

Maybe. Daniel, I actually think we're gonna be on the same.

But it's a little public.

But they don't work today.

I don't.

Yeah.

Okay.

And no one's talking about it. And it breaks my brain that no one's talking about it. And like, people think they work, but they really, really don't. And the reason they don't work is because. Or at least I'll tell you my thesis. I'm curious. Your thesis. So I'm not going to call any company out, but basically, like go try to work with the model, the incentivized model. I don't even know what you would call that. Basically like marketplaces, where you're incentivized to improve the model. And the problem is that they even perform less than open source models, which should be the floor for them. First of all, it should be the floor. And then you should build off of that with token incentives, but they perform worse. And the reason I think I'm coming to is there's two options and I don't actually know. So maybe, you know, the first is that no one's actually hosting the best open source models. I think that's less likely. What I think is more likely is the incentives or like the reward structure is perverted in a way that it's basically rewarding people to hit benchmarks instead of actual output. So basically, like someone's Getting rewarded for hitting a certain number. And that is not a good proxy for output quality. What we're seeing is models win, that shouldn't win. So I'm curious. Yeah. And I have actually a potential solution that I haven't told anyone yet, but I would be curious. Curious to discuss with you.

Yeah, I'm very curious. I think, unfortunately to Santiago's premise. I think we're pretty aligned here. I think it's very easy to think of models as these commodities that they're uniform surface area, there's little balls and you swap them around until you find the best one. But oftentimes, at least in production, these models, the good ones anyways, are tuned very carefully, they're super specialized and they basically break the moment you give them anything that don't, you know, they're not supposed to anticipate. Right. This is very different than LLMs, right. Which are like the first, like generalists that can take in all kinds of different inputs and kind of still make sense. And so in the context of model marketplaces, the fact that there's all these asymmetries and inconsistencies to their surface area makes them not very composable or very tradable or. Yeah, exactly. You just like overfit to the index or overfit to the benchmarks and then turns out the open source model does better anyways. So yeah, it turns out to be a very high friction marketplace that doesn't work very well. I'm very curious how you'd solve it though.

Okay, so this might be out of bounds for this conversation.

It's all good.

Okay, I'm listening. I think that. So obviously vlms vision language models are kind of this combination of, you know, vision language model working in a better way. They're not mainstream yet, but I think they're going to come out in the next like 2/4. And I think basically you would use that as like a GAN structure to decide instead of using benchmarks at all. Because I think benchmarks are too gamifiable. And so you take out the numbers entirely. You put like a few VLMs that are fighting against each other and then I think that would work.

I love that the solution to non performant AI modeling is more AI models. I think that's absolutely correct.

Yeah.

If there was a solution, it would probably feel something like that.

We'll see.

Yeah, we'll see.

I do think this is an answer because I do think we're contrarian in thinking this. We are aligned. Would argue with us.

Yes. Most People love this idea. Yeah, like, like.

And so I understand it is. And I'm relating it to crypto because a lot of it is through like incentivizing people to do stuff. So you're saying there is a set of companies that are incentivizing people to improve the model. And, and they're setting. It's a problem that you're identifying is it's the benchmarks that are easily gamifiable. And you see this all the time in crypto too, by the way. Yeah. And, and that doesn't. It is sub performant to just a pure open source model.

Yes.

Yeah. I think this is a good place as any to end it. I want to just impart something to me. One of the more interesting things I've heard in this discussion was something you said, Casey, which is in this journey of AI, the things that researchers thought would be easy have turned out to be the hardest. And the things that we researchers thought would be hard to solve have become far easier.

Yes.

And to me that's just mind boggling because it is very humbling. I think this journey of AI has been very humbling to understand what intelligence is and is not. And yeah, it's just like, it's pretty, pretty crazy. I don't. It's just one of those we've talked about at length about how little we understand how the brain works. And this is why it's just a fascinating discussion to have. And so yeah, I really like that.

Well, that's, that's the appeal of good technology I think. Right. You just be repeatedly humbled in, in recognizing how little of the world we actually have, you know, appropriately modeled. But I think it's Santiago, you said, you know, the only way through is just continue to invest in this stuff and solve the hard problems. A la your sweatshirt.

Yeah, I think it's one of those areas where I feel under exposed. I've been looking at funds because I'm not going to do it myself. But I also lean on Casey a ton to find these opportunities and really parse through what is bullshit and what is not. To be fair, a lot of stuff that I thought was bullshit and crypto ended up being not bullshit. And a lot of things that I thought were very real. It's just, that's just the nature of I think Cambrian explosion of interest in things. But we'll see. Kasey and Daniel is fantastic to host you. I hope I didn't interject too much to disrupt the flow of conversation, but we'll definitely, we'll love to have you on to maybe get an update on Modulus and also Casey and all that stuff that you're seeing from where you sit, which I think is hugely fascinating.

Thank you for having us.

Thank you so much. Pleasure.

We'll call this empire crypto AI part 1. There may be 10 parts. So holiday Edition, as are all the Holiday Edition.

Yes. I like that.

All right, everyone, thanks so much for listening. And Casey and Daniel, thanks so much for coming on. I really enjoyed this conversation.

Thank you. Thank you.

Bye, Santiago.

Hey, everyone. Thank you so much for watching today's episode. Really hope you enjoyed it. We wanted to take a second to just remind you about our upcoming Digital Asset Summit in London, March 18th to 20th. Santi and I got your back. Seats are limited and we hooked you up with a 20 off discount code. It is Empire 20, if you heard it earlier in the podcast. There's a little competition running at Blockworks to see who can drive the most number of tickets. So when you register for the Digital Asset Summit, make sure you use our code, Empire 20. See you in London.