A Positive Vision for the Future: Part 2 with Illia Polosukhin of NEAR

Hello and welcome back to The Cognitive Revolution.

Today, I'm excited to share part 2 of my conversation with Ilya Polosuhin, co-author of the seminal Attention Is All You Need paper and founder of Near Protocol, which describes itself as the blockchain for AI and aims to build a future where AI belongs to everyone. In part 1, we explored the foundational technology stack, proof-of-stake consensus mechanisms, confidential computing on GPUs, and Near's plan to train frontier models on a decentralized basis, with contributions incentivized by cryptographically guaranteed revenue shares. Today, we zoom out from the enabling technology and discuss the big picture questions. What does personal computing look like as AI continues to eat traditional UI-centric software? Ilya envisions a unified intelligence layer that works across all of your devices. Predicts what you need, and proactively gets things done for you. How will this change the nature of markets, trade, and the consumer economy? Ilya expects that advertising and middlemen will become less relevant as AI agents allow buyers and sellers to connect directly at unprecedented scale. What does daily life look like, and how will we spend our time in the era of AI-provided abundance? In addition to a rise in personalized AI-generated entertainment products, Ilya predicts that people will find meaning and compete for status in small communities focused on niche interests, including athletics, video games, the collection and curation of unique artifacts, and who knows what else. What rules should govern agent-to-agent interactions in this world? Ilya imagines a symbiotic relationship between individual humans and their own personal AIs, such that humans and AIs effectively grow up together. And he believes that AIs should pursue the interests of their individual humans, even if that means being less than fully transparent when negotiating with other AIs on their humans' behalf? And finally, what new governance mechanisms could allow us to take advantage of the exponential increase in information processing power and reasoning that AI affords us? Illia describes NEAR's experiments with AI delegates, which vote on behalf of token holders, who in turn select them based on their decision-making frameworks, and also describes NEAR's long-term goal of enabling every individual to have their own AI that participates in governance On their behalf. Of course, we touch on lots more along the way as well, including the role of formal verification of software in securing smart contracts, the remaining challenges required to effectively bridge the gap between low-level guarantees and high-level human intentions, and the need to harden society's defenses in order to maintain global biosecurity in the presence of broadly distributed, powerful AI systems. Overall, while many questions remain to be answered, Ilya's strength as a systems-level thinker spanning technology, economics, social dynamics, and governance is evident throughout. So without further ado, I hope you enjoy one of the most concrete, positive visions for the future that you will find anywhere from Ilya Polosukhin, founder of NEAR.

Ilya Polosukhin, founder of NEAR, welcome back to The Cognitive Revolution.

Thanks for having me back.

I'm excited for part 2. So last time we talked a lot about foundational technology, the journey that you went on from being an author of Attention Is All You Need to try to source data from contributors around the world, struggling to pay them, taking a detour into blockchain, thinking it would take just a few months. Here we are a few years later, and it's all really happening. Anybody who's listened to this feed for more than a minute knows that I often say the scarcest resource is a positive vision for the future. And I appreciate that you have one. So I'm really appreciative that you're taking a second window here to help us unpack that. I guess maybe for starters, you know, one of the jumping off points last time was you said you wanted to teach computers to code. And sure enough, now they can code. So maybe as we kind of ramp up into a vision of like potentially a quite different future, how is— the rise of AI coding assistants changing how you guys work? And how is it changing, like, who can create stuff on top of the blockchain?

Yeah, well, I don't think it's even on, I mean, about blockchain per se, but yeah, I mean, the real reason why I always thought that, you know, as computers are able to code, we actually approaching a different world because there's a few dimensions to this, right? We know that there, there was a statement by Marc Andreessen, right? That software is eating the world, right? And, and this idea was like that effectively, I mean, what it means is automation. Automation has always been the driver of innovation, of GDP, of productivity. And, you know, it's, Like everything from tractors to factories to then indeed computers and computers kind of this like kind of universal bicycle of the mind, right? Automating things. And so now the challenge been that there's always been like a small cohort of people who actually are able to build software, right? And so if I have a need, right, I need to find somebody else to build it. Probably they need to build it not just for me, but for a large number of people. So it's actually economical. And so like we've been, we've ended up with a lot of like software that became very complex to use. And so now you need to learn how to use it because it's not really built for you. It's built kind of for, you know, this generic user that has like, you know, every user has like 5 use cases that they like somewhat overlap. And so it's all of this is like kind of stacking in one software, right?

Yeah.

Or you just don't have it and you keep doing stuff manually and kind of wasting your time. And so for me, the ability for machines to code is really about that transformation where everyone now is able to build their personal software. Everyone now is able to build their own personal automation. And it also kind of removes the fact that like the interfacing, right? That these interfaces need to show you all the options right away, right? You kind of can kind of subselect with English or whatever into the part of the interface that you need, right? So to give you very specific examples, right? I mean, I like the example of Salesforce, right? Salesforce, I mean, obviously started as like a small startup that was targeting like a specific use case of salespeople. But at this point, right? It's a monstrosity that like you need to hire somebody else to configure Salesforce for you. Right? Like that's how complex Salesforce is, right? At this point, it's effectively like you're hiring somebody else to build you a system, right? It's just like they're using existing prebuilt components. Now imagine this world now where you can just talk to a computer. Like it's really now becomes about your sales process, about your business process, about how you want to automate, what reports you want, et cetera, right? It can be dynamic. You can restructure this as you go. You can, whatever built-in things that Salesforce may have somewhere as a feature or may not even have. And then you can integrate with whatever other tools you want. Like for example, Salesforce, we're in crypto, everything is in Telegram. Salesforce doesn't have Telegram integration, we can't use it, right? So now we need to go and somebody to get build integrations in Telegram and Salesforce. But with your Vibe-coded CRM, you can just like, hey, and integrate Telegram for this. So that's kind of, I mean, this is just like a simple example and we can keep extrapolating, right? It's every single kind of our, let's say digital footprint, right? Is kind of becoming more and more automated, right? And the more intelligence computer can have and more it can take on the kind of, the more you can offload the kind of orchestration of different tools or Just low level, you know, here's a database, you know, I need an HR tool, I need CRM, I need this, I need that, right? You can actually build all of it. So that's been, I mean, this is in 2017, we were like, hey, you know, software as a service is gonna die, AI will replace it. In 2017, this sounded like, yeah, very delusional. That was probably the— But obviously I think now we see that most people are kind of, Or a lot of people agree and then the software service itself is trying to become AI because they know they kind of going to just get outcompeted by that. So coming back to your question, like what changes in our work, I think there's few pieces that already clearly work, right? Kind of data analysis through natural language, right? If you have like kind of reasonable data structure, you can like now give Like you can effectively make everyone, maybe not a data scientist, but like what before would require like a business analyst person, right? And then it would take them a while to like pull all the data. Now you can just say like, oh, you have a question about the business analytics information, just ask it. Like you don't need to email somebody to get you the answer, right? You just ask it to the tool. It generates SQL queries, pulls in whatever data, writes some Python, gives you an answer, right? I think I'm actually a big fan of the frontend building with vibe coding now. And though I'm not writing production code anymore, but it's really useful for me as a prototype tool where you can effectively just really quickly get to an experience. And so you don't need, like, I mean, you want design to be more about like style guides, et cetera. And, and But like the UX part, you can test really quickly, right? Before, a designer would, you know, maybe design something and then developer would try to build it. And again, it's like, oh shit, I can't actually build that. Or like, it doesn't work exactly this way. And there's like a lot of iterations. Now, even designers can just build a full experience, you know, fully clickable thing. It generates code for this, and now developers can just plug in all the backend logic. I think also, We are starting to see on our teams that actually doing development that the, I would like, the time spent is changing. Before you spend a lot of time on kind of the development work, and then you spend also a lot of time on reviewing other people's work. The amount of time spent on development is shrinking, right? Because you effectively tell the whatever course or— codecs, whatever this is to go and do the thing. And a lot more time spent on reviewing things and making sure they're correct, right? Now, one thing we were experimenting with on one of the teams was what if we decompose the whole piece, like the whole, like this is, for simple software right now, AI works, right? For complex software, it doesn't, right? It doesn't really, You're gonna just go and like build me a really complex system and it goes and does it. It doesn't work yet. We see obviously the improvements continuously. And so right now for complex systems, you right now decompose it yourself, right? You know, as an architect or a senior engineer. And then you have maybe a few different team members who actually build the subsystems. And before in traditional software development, you always wanna have multiple people on each subsystem that you want them to know really well how it works. So they can maintain it, so they can change it. Now actually it's not as important because AI will explain to you how it works. And you may actually even have a natural language explanation that the developer used to build it in the first place attached to it. And so it's actually more about like velocity and reviewing process and then how to ensure that each part is like secure and kind of works correctly. And so there's kind of a transition happening where There's more almost like individualism because every individual is more productive. And then it's more about like how to build the right decomposition into pieces. Again, this is a very also temporary in a way, but this is kind of the current state, right? And so like as tools become more mature, they are better at like larger codebase navigation, et cetera. I think, The other question is, it's really about like, I mean, the model quality for specific tasks, for like the conceptually like the abstraction level that it needs to operate on, right? So like, again, front end, there's no abstraction level, right? You just kind of build what you see, right? And you kind of iterate. So it's really easy to check, it's really easy to iterate. So you don't really need that much. And models are good at this. When we talk about like a blockchain low-level code, right? It's like extremely complex system. There's a lot of pieces, externalities. So models are not very good at that. And kind of there you really expect people to spend more time and in general you spend more time like thinking about the algorithm and architecture than writing code. So yeah, so it really depends and kind of, but this is also all shifting really quickly. Like, you know, 6 months ago I would've, wouldn't given you a different answer. Right. And I'm sure in 6 months it'll be different as well.

Yeah, we've seen some very impressive programming results from frontier companies that have not yet hit the, you know, the public APIs or product surfaces just yet. So certainly we can bank on more to come. When it comes to, you, you mentioned like job titles, you know, I think you said like senior, you know, engineer or architect. That raises the question I think a lot of people are asking right now. Are you hiring junior engineers? What do you think is the fate of the junior engineer as things stand today?

Yeah, I mean, I think it's, it's less about like junior engineer, it's more about who is that person, right? If, if this is somebody who's like coming in, like, you know, for example, by the time I got into university, I already been coding for 7 years, right? And so, and this is when I got, you know, my first job, et cetera. So like if this is somebody coming in who's like, you know, built multiple projects already using AI every day, et cetera, like it doesn't matter if junior or not junior, right?

Yeah.

It's really, I mean, yeah, there is a lot of skills for them to learn, but like they're there to learn them, right? They're open, they're ready to go. And then you have some people who coming in who like, you know, like studied a bunch in university, but like not really, like not really in this learning mindset and like this is gonna be continuously changing things. And like you're like, so like again, Yeah. We're transitioning from like software as a craft to really be just like problem solvers that talk to computers, right? And so you need, I mean, problem solving at the end is really all about kind of like the mindset and then the approach. And so if people are willing and are kind of, doing that and excited about that, that's, if people like, oh, I don't know how to do this, I don't, you know, I can't do this, et cetera, then that's not the right person. So yeah, so it really depends on like it, for many things before, yeah, you would hire like a bunch of junior developers because they're cheaper and you don't need maybe as like high quality of work and you wanna fan out work. That part is not needed. Yeah. So you're hiring more for like, The problem solving and kind of more people who can actually like creatively problem solve together.

Yeah, there's always a room. There's always a space. There's always room for people who are like a force of nature under themselves.

Exactly.

Overall, that sounds bearish for rank and file.

Like I was told that if I you know get study this book again, yeah, I'm gonna be yeah. I'm gonna pay $100K. Yeah. Yeah, I think, I mean, but I think that's true about every job at this point. Maybe plumbers and electricians. That's the, like, I usually talk like the automation right now happening from both, from 2 sides. The, all of the like manufacturing, like on the floor jobs are getting automated. And like that price is still pretty high actually. Like, yeah, I mean, you know, Vietnam, for example, salaries are probably lower than what robots are getting paid in the US. But in US, it's already like, you know, you effectively, there's this company I know called Formic. They effectively provide you outstaffing with robots. You're a factory, you call them up, they bring you robots, they install them, they set them up. You don't need to do anything. You just pay them effectively as you pay salary, but they work 24/7. They don't complain, they don't unionize. They just, you know, do the job. They don't quit. Like right now in US, the churn of workforce is 300%, meaning like every year you need to hire 3 people for 1 job because like they keep quitting. So, so that's like the automation from the like bottom kind of low skill, you know, very repetitive task. And then finally all of the kind of white collar high-end jobs are getting automated, like, you know, coding lawyers, like, A lot of this information work, right, getting automated. And so the most safe right now, I mean, it's going to get automated as well, but it's actually the high dexterity kind of skilled work that requires like, I don't know, plumber, you need to climb under the sink and fit something. Stuff like this right now is just super hard to do with AI. But again, this will happen as well. All of those things will get automated over time.

Yeah, it's coming for all of us. It's just a question of when.

When—

hey, we'll continue our interview in a moment after a word from our sponsors.

So I wanna understand a little bit better, 'cause I think a big part of your vision of the future obviously is AI that everyone owns, right? We've, we've kind of got the one default path, it seems, in front of us where we have like the big tech singularity, which is where like 3 to 7 companies become totally dominant forces because they have the models that ran away from everyone else and nobody else does. And so we're all just kind of trying to get whatever inference we can get from these leaders. And then there's your vision of the decentralised and collectively owned AI that I think has You know, it has a lot to say for it. It's certainly super attractive in a lot of ways for people with some concerns about, you know, what happens if everybody has, you know, access to certain things in an unrestricted way too. But leaving that aside, if front end is like largely we can get the AIs to do it, but like core blockchain work is beyond what they can do, what's in the middle? Like, What kind of apps are people building on the blockchain today? How hard is it to build those apps? What makes it hard? And can the AIs help there yet? Or if not yet, at what point should we start to see an explosion of vibe-coded on-blockchain applications? What's the fundamental barrier or kind of rate-limiting step toward the proliferation of just anybody who has an idea for blockchain can go do it in the same way that anyone who has an idea right now, to a significant extent at least, can go create a little micro SaaS app.

Yeah, I think the problem is the same. It's just, it's exaggerated. The problem right now, if you're launching your micro SaaS app and you're not actually engineer, If you're launching it just for yourself, it's totally fine. Right. And, and I kind of, my rec— my recommendation for everyone, like, you know, build tools for yourself, live code, everything for yourself. Now the problem is like, as soon as you go like make it for everybody else, because you don't really understand what issues are under the hood, you don't know like how, how it'll actually affect your users, yourself, et cetera. And so we've seen already, you know, people getting hacked. their secrets getting leaked, et cetera, right? So that is the biggest issue right now. The blockchain, because it's naturally kind of in the open right away for everyone and it involves money, that problem is exaggerated, right? Because now if you make any small mistake, right? And we have this right now with very professional engineers who build soft— who build blockchain software, any small mistake, somebody will find it and exploit it. And effectively this will result in some value lost, some money lost. So really that's the biggest challenge. What works right now is actually for existing so-called smart contracts, right? Kind of the backend, you can generate a frontend, you can create your own custom UI for specific use cases or combining multiple use cases into one UI. That part actually works. And again, I would not recommend to launch it to other people, but you can build for yourself where you say like, hey, you know, I have this like yield opportunities across different places. I have this whatever, let me combine it, make my own like asset manager, right? That like makes me easy, for example, to manage this. Like things like that you can totally do now. So I think it, yeah, it really depends. Now what we are working on kind of medium to long term is how do we formally verify the correctness of the smart contracts and ideally the whole kind of blockchain application stack such that if you are vibe coding an application, you actually have proofs, mathematical proofs that it's correct. And now also when user using it, and this is important, just proving, okay, cool, smart contracts is like doing what you want, but you maybe didn't properly define what you want, right?

Sure.

Maybe your logic itself is flawed. And so what you want is like, I am as a user using this piece of software, want to know that it does what I want, right? And so that's a critical step. So again, if I'm looking for a financial application, I want to know it's not going to lose my money. And so if it can prove to me directly in the transaction, right, as I'm sending money in, it should prove to me that it's not going to lose my money. then transaction succeeds. So that's kind of the level of integration that we are aiming for. And I think is required for this kind of really kind of adversarial and monetary valuable. I actually think this is required for all software because like the world where, you know, AI is going around and hacking everything left and right is also not great. And we are kind of in it actually right now. And so we do need like formally verified software to really secure that.

That has been coming up more and more in my conversations about this. And I would have to confess that up there with like regular expressions, the concept of formal verification of software is, you know, one of the things that makes me feel dumbest. Because I'm always a little bit like stuck on, I think, the point that you were emphasizing there, which is like, it's one thing to prove that this particular function or whatever does what it's supposed to do and doesn't corrupt other memory or whatever. You can make a bunch of sort of generally low-level statements, right? But it seems like there's still a real challenge in aggregating up those small-level statements to the holistic, like, this is what I want. You have a version of the genie problem, which is kind of, what a lot of people have worried about with AI in general for a long time, of if we tell it a goal that it interprets a bit differently than we did, we potentially could be in trouble there. How do you see this actually playing out in practice? And again, I'd love to kind of get some vision for applications that you would love to see somebody come build on your protocol that don't exist yet. That, and maybe it's because they're just too hard, but I'm sort of imagining like, do I have an AI agent that comes in and like automatically red teams this for me?

Let me give you a kind of a simple example and we can build it up, right? So a simple example, I want to put money into a bank savings account, right? And I want to be able to withdraw it. And ideally I want to withdraw a little bit more money than I put in, right? Like right now, you know, you're sending money, you have no guarantees. Like let's say you sent money via whatever, ACH or IBAN or something. You have no idea if it's gonna arrive, right? You have no idea if bank will exist tomorrow. You have no idea if it's actually gonna give you money back. I mean, there's like government insurance, right? That insures up to some amount, but like generally speaking, you have no guarantees. Now blockchain gives you some guarantees. It, you know, actually like, hey, money arrived. You can verify that. But indeed, if there is some code involved, you don't know if that code has some potential— somebody can withdraw this money illegally. You need to go audit the code. You yourself may have missed something, et cetera. So here you say, hey, you effectively constrain— my transaction will pass only if I can call this withdraw method and it will return me at least as much— Given X deposit, I want to call, be able to call withdraw with at least X, right? So that would be the condition, which is like when you deposit, right? You effectively constrain that this smart contract needs to prove to you that this property will be true. Now that contract may, you know, deposit this money itself into other places, right? You know, if it's a savings account, it maybe lends it, borrows it, et cetera, right? And so, it needs to actually in-chain to be able to prove to you, it actually needs to chain all of this with everything else it does, right? It needs to be like, hey, if I'm lending to someone, right, either they need to return to me or I'm gonna like, you know, foreclose their account, liquidate their collateral, et cetera, right? Like, so there's kind of a chain effect that's happening through the system. So, that's kind of where, I mean, To your point, right? Like you start maybe low level, but you actually can start expressing somewhat high-level properties. Now the challenge with this, like with money and deterministic blockchains, it's pretty easy because it's deterministic and you kind of have like full observability. So you can actually do these constraints pretty easily. To your question where it's coming from, well, It's gonna come from combination of indeed your, I mean, we call it wallet, right? Your kind of software that is on your side that actually facilitating these interactions. But also, you know, we believe that your wallet will be AI, right? It will be the AI agent that is on your side, your user-owned AI that actually does this interaction. So it will indeed be on guard, you know, verifying these properties. Now a more complex thing is indeed how do we prove things that are non-deterministic and not potentially like easily observable? And this is indeed where, you know, you obviously cannot have like 100% formal proof. You need to effectively now start dealing with probabilities. And so, and then you can move those probabilities with insurance and other things. So this is where kind of, you can have like financial system and other things where you say, hey, you know, we have a liability insurance that, you know, in less than 1% of cases something can happen. And so like, prove to me that it is like, you are, you are, you know, it either succeeds or I'm getting like a million dollar payout if it doesn't.

Right.

And so you're kind of starting to combine what, you know, what people have built in insurance where they estimate risks and evaluate and underwrite. and some of this form verification combined with this probabilistic modeling. So it's not like— for some things, the answer is somewhat easy, and then it becomes more and more complex as we touch more and more real world. To give you an example, it's like, hey, I'm ordering steel from, I don't know, some country. It's going to arrive. There's a ship involved. Maybe this ship like sinks midway, right? So, I mean, normal way is like, hey, you know, we're gonna insure this and there's gonna be insurance. And so like you need all of those mechanisms to build on top to account for real world kind of non-determinism.

Yeah, it's, it's really hard for me to envision all of that working. And again, partly it's because I'm maybe just a little slow on some of these things. But the sinking ship is a good example where it's like, hmm, how is my smart contract gonna know if it really did sink or if somebody's just telling me that it sank? Or, you know, there's that sort of shell game of like, where do you hide the trust? Or, you know, what exactly is fully verified? It's quite interesting. I don't wanna get too bogged down in it though, because I don't wanna force us to get to a part 3 before we really get to all the sort of utopian vision. So maybe where you can weave some of this stuff in there as we go.

Hey, we'll continue our interview in a moment after a word from our sponsors.

But let's start to leave also a little bit of the sort of how behind and just talk about the what. What are the apps that we're going to enjoy? What is the computing paradigm that we're going to have? You mentioned agents kind of doing stuff for me. Meta obviously has been putting a vision forward of glasses with a display in them, a sort of head-up, you know, no, who needs a keyboard, right? When you can just talk to your AI as you walk down the street. That does appeal for all the things that Meta has done, including Hot Stepmom, that don't appeal to me that much. I would say the head-up display is, you know, it's at least an interesting vision for the future. What do you think are computing life is going to look like as this stuff matures?

Yeah, so I definitely agree some form of AI operating system is going to be the main driver of our computing. The devices and kind of the form factors will be different. And I actually think it's going to be easier. It's already easier. If you want to make your own glasses, it's not actually that hard, right? There's some factory in China that will make whatever hardware you want. So really it's about like kind of a single AI, your AI that is available across all those form factors, right? Your watch, your glasses, your headphones, your phone, your laptop, your whatever. All of this is kind of interconnected as like a single surface. And then your AI knows that like you like to see this information on the watch, but then by the time you pull out your phone, you want to see like maybe news and kind of more longer form content. And then I may be actually like videos instead. And so that's what it should show me. So like it's going to be effectively personalized, like AI generated, not just kind of the content, but also the applications that we use, right? Probably a lot of the similar patterns that we already use, like the feeds, the chats, et cetera. But They don't need to be kind of fixed, right? Like you don't, like right now I have, you know, 5 different instant messengers. I have, you know, 7 different feeds and like all of that, you know, can be a single feed. You know, I can switch between like work and personal when I want to. You know, my AI can predict a bunch of things that I would like to do. This is something we experimented back actually in 2017 where based on all the things you're doing, can we predict actually the next thing you will do on your phone? And just do it for you, right? Or suggest to do it, right? Like, you know, you have a meeting, you know, 20 minutes away, let's call you an Uber, right? Like, you don't need to go open an Uber, go calendar, copy paste the, you know, the address, paste it. Like, it's just gonna do it. Like, things like that. There's a lot of things that, like, you know, AI will know, like, hey, you know, you low, like, you ordered food 2 days ago. you know, you're gonna be out. Let's reorder a bunch of stuff that you typically order. And so as that, as that system kind of matures and we also trust it more, and, and that's this important aspect, I think there's, there's actually starting, the economy itself is gonna start to shift because right now the economy is built, I mean, we kind of in this consumer economy where it's built on advertisement, on, on kind of Like we, we, you know, we discover new things through this feeds, et cetera. But if your AI is like actually, you know, like, I mean, some people are already doing this now. Like, hey, I want, I want to be on a diet, like build me a personalized, you know, meal plan, et cetera. So like all of that, but also it's going to go and order that. And, you know, maybe even your humanoid robot at home will also cook it because it's the same computing system, right? It's like you just get the food you, like your AI recommends given your health goals, et cetera. And so now the capacity, so this is, so this is kind of like on a micro level, right? You have this person. And so now your AI, like it doesn't need to go and order it from your local Walgreens or whatever, Vans or Aldi or something, right? It can actually put a purchasing with directly farmers, right? With directly, you know, manufacturers. And so they can then themselves start capacity planning, right? So you're kind of starting to remove some of the middlemen that are built because we cannot right now have the direct relationships with suppliers, right? And so that's kind of an interesting meta point that like our economy right now is built on this like middleman architecture, right? Because there's, it's really hard to plan things. And so every layer, like Costco is effectively just, they purchase a bunch of stuff, they put it in one place and then you buy it, right? So like they kind of serve as this, and I mean, they have a fixed margin, et cetera, but it's really kind of like the temporary place for holding things for you to actually purchase, right? Or find. And if your AI is purchasing directly, right, it can just go to their purchasing agents and kind of do this. Now there's batching, there's like other things, but again, all of that can be done by AI way more effectively than we're doing it right now. Right. You know, like it, some AI of your city will know that, you know, 500 people are ordering this, you know, 7,000 people are doing that, you know, they'll want it, you know, tomorrow, day after tomorrow, the day after tomorrow. So we're like, you know, we're gonna, capture the eggs and ship it this way, et cetera, right? Like all of that can be effectively just managed as a holistic information system. And so that's where we go into like a really, I mean, I use this example kind of half jokingly, but in communism they were trying to build a system, right? Where they were doing capacity planning, but they were missing the AI to actually do it. Right? And so in turn, it was terrible because it effectively like couldn't actually satisfy the demand-supply changes. But if you have a real-time platform where you can actually have all this supply-demand being— and so the reason why capitalism has been so successful, because capitalism is actually a compression of information. Like money is a compressed information because it kind of compresses anything you can purchase. into, you know, one number, right? And so it kind of compresses all of this information, all of the different things into one number, and then it's really easy to kind of navigate. And so here you, but obviously with this compression, it's, it's, it's, you know, you lose some information, you lose some decision-making. That's why, you know, in US there's like 30, 40% of all the food is being thrown out because there's just like, you know, they over-provision in the stores because they dunno how much people actually will buy and they don't want to have an empty store. But you don't need, you don't need to do that, right? If you knew exactly what the purchase will be in the next 24 hours because their AI has already planned everything and provided it and all of those got aggregated and shipped. So I think like we are gonna see actually a shift in how economy works at the macro level because of this micro, like each of our individual AIs becomes this micro decision maker who can real time provide all this information to the right sources. and navigate and also not be affected by a lot of kind of, you know, brands and other stuff, right? But actually validate based on the kind of core values. So that is like an interesting transformation where I think indeed it's, I was mentioning like, I don't think like there's no good way to like make a movie or, you know, a book, like a science fiction book that's talking about effectively like a change of economic structures in the, in the society, right? So it's way easier to talk about dystopia and the heroes who are fighting against it. It's just like way better story arcs than like, hey, we've been building out this economic model and now it's 1% better every month. And so it keeps getting better and more optimized and that's how we live. So yeah, so I think that's the kind of the paradigm of computing is we have the AI, I mean, a code agent. I mean, it's effectively our assistant, right? Our operating system, but it has all the context about us. It's able to make decisions on our behalf. That's why it needs to be private. It needs to be ours. It needs to be on our side, right? We need to know that it's aligned with our kind of success and outcomes. Otherwise, I mean, this will not work. But if it is and we can trust it, then it can go and make decisions on our behalf, right? And so the other example is in like traditional governance, right? You know, right now we, again, we compress the information, right? We go and vote every 4 years for someone and hopefully that someone goes and does what they promised to do and why we vote for them. Now that usually doesn't happen, but— And so again, we can actually have every single decision can be voted on by all, whatever, 300 million Americans because their AI is online all the time and can evaluate every single decision. And based on their owner's beliefs and like what's valuable and successful for them, it will, can represent them, right? So you don't need to have this like compression of representation if you can have this like online, always available AI that's on every individual side. So that's kind of the, again, like there's economics, there's governance sides, and obviously the other side, which is the entertainment and kind of that's where things are getting interesting because I think we as humans, I call it like we believe in status game. Like we're sorry, we are driven by status games, right? And because money became this like compression mechanism, we use this right now as the kind of ultimate status game. You know, you have more money, you're more successful. you know, billionaires more successful, more famous, et cetera, et cetera, right? But I think again, as this decomposition happening, and we already see this, right? You know, an athlete maybe not as wealthy as a billionaire, but maybe still more kind of famous and more respected in many ways, right? There's some other kind of status, like I call them status games, effectively places where you can compare who's better in some way or who's higher, et cetera. which don't need to be associated with anything that's actually productive, right? I mean, I use a few examples, but obviously like, you know, athletes is a good example because like there's no actual GDP being produced by athletes, but it's still, you know, very valuable kind of status game, which other people enjoy, you know, watching and kind of participating in different ways. Obviously video game is kind of a, video gaming is a new form of that as well, right? You have now, video game athletes. But, you know, you can imagine many of these, like NFTs were similar, right? Are you part of this NFT, you know, collection? Do you have Bored Apes? Do you have, you know, Pengu? And then you're part of this, you know, tribe. And if not, you're not, right? So like, we kind of like this types of differentiation. And I think that will actually proliferate a lot. I think we'll see more and more of those things where people really differentiate on things that are really, I mean, superficial to extent, right? But kind of for the group, they make a lot of sense and they kind of differentiate people between each other. So it's not, again, it's not like, are you a whatever, a software engineer or a lawyer? It's really like, yeah, are you whatever, playing League of Legends or StarCraft, right? So that's kind of, I think, important. Obviously we already see on the entertainment side, AI-generated entertainment, again, not very hard to extrapolate, especially with Sora yesterday. You can have just like a personal feed fully AI-generated, right? You don't actually need like people recording, et cetera. I think they're gonna be, again, with everything we saw in animation, there's like a slice of a market that wants it in a traditional way and like people doing it. So they're gonna be still, even if there's robots everywhere serving you food. There's gonna be a restaurant with people, which will be like more prestige and will have like limited capacity. Similarly, you can listen to AIs playing AI music, but humans playing human music will continue to be like a prestige thing. So, but again, it's gonna be this niches, like we already kind of in this niche world, right? That thing is gonna continue proliferating. Yeah. So that's kind of like how I'm thinking of like, as society evolves, like, and we have this, yeah, a lot of the economical things are kind of moving away. The, yeah, we are gonna just like participate in more status games, right? Like what is the things, how, you know, how you compare with everybody else in that niche, in that group, et cetera.

Several double clicks I wanna do first. Your mention of communism brings to mind an article I think about, an op-ed in the Washington Post I think about not infrequently from all the way back in 2018, AI Will Spell the End of Capitalism, by a Chinese legal scholar and government official basically saying that the planning— this is sort of the Through the looking glass version, I think, of your vision, but making a similar point that the local nature of capitalist decision-making, where everybody's kind of trying to do their own local thing and they're sort of aggregating signals and sending off aggregate signals to other people through the price mechanism, et cetera, that may not be needed as much anymore. And their vision for it is obviously much more centralised than the one that you're articulating. But we are starting to see a little bit of glimpses of this with, for example, Pulse from OpenAI, where now I can wake up every morning to an AI, and there's other versions of this too. I've tested quite a few, but Pulse, certainly the one that's made the most headlines recently for being there in the morning with work that it's done overnight, presumably when the GPUs weren't in such high demand, and bringing me something that it has gone out and kind of scoured the world to find the stuff that I really need. So I can start to see the beginning of that in terms of the architecture and, let's say, alignment or incentives of that.

I wonder, first of all, where do you think the compute lives?

I mean, we have sort of right now, of course, we have a lot of centralisation of where the actual inference is happening. I'm just thinking about this recent research from Thinking Machines that they put out again in the last 2 days where they showed that basically LoRA techniques are similarly robust to full weight fine tuning. And that has me thinking maybe there's a sort of hybrid model and kind of compute architecture where some of it is in the cloud. Maybe you've got your 1.4 trillion parameter model or whatever that's Collectively owned, that sits in collectively accessible or universally accessible hardware at some centralised location that you can send your data into. And because it's a trusted execution environment, that data isn't exposed. And then you get sort of activations back and then your little local LoRA extension that kind of makes the AI truly your personal AI at maybe I don't know, 1% the weights or something. You can maybe have that on your person. Probably can't have that in your glasses, but maybe you can have it in your pocket or whatever. And then, so I want to hear, is that how you think that shapes up? And then when it comes to the agents, one thing I think about a lot is the fact that we're already seeing all sorts of weird behaviour from AIs, including at times deceptive behaviour, lying to achieve goals, whatever. And it strikes me that if we're going to have our AIs go out and represent us and negotiate on our behalf, we're going to have some tricky questions about how honest we want them to be. Anthropic famously put up the 3 H's, and it's like, we want the AI to be honest. And they can pretty much say almost always, right? Unless it's in very obvious conflict with one of the other 2 H's. But if my AI is going to go negotiate with your AI, in the same way that I probably don't want to tell you what my absolute worst offer is that I could accept right away, right? I probably don't want my AI to do that either. And so I've got kind of an interesting question of what should society set in terms of norms for AIs being honest? Should they, as long as they're representing me and my interests, is that okay? Or like, If they're lying on my behalf, is that okay? And do we reinforce them by price signal or just my kind of thumbs up, thumbs down? How do we even get them to be aligned to my interests, whatever that exactly means anyway? So a lot there. But I guess the 2 main things are like, how do you see the architecture of computing? And how do you see the architecture of exactly what the signal will be that your personal AI is aligned to or reinforced to? And what societal limits might there ought to be on how monomaniacal one's AI can be in pursuing their own individual self-interests?

Yeah. Yeah. Yeah. So all good questions. I mean, on architecture side, I think, I mean, it will be a mix of everything. So, I mean, we already see like data centers are being built everywhere, right? And I think that's why we're actually approaching it. I mean, we talked about decentralized confidential machine learning is like, how do we utilize all these data centers? in a confidential way, right? So that even though it's my data, I know it's not gonna leak from some data center. There is projects already who are doing data centers on edge, right? So imagine there's just a container that arrives that just gets dropped in your proximity, in your town, in your whatever. And now that container has like 100,000 GPU, whatever, not 100,000, but like maybe—

10,000.

1,000 GPUs, right? And so, and it kind of serves whatever that proximity with the compute. And then there will be—

Dropped on with a small nuclear reactor while they're at it.

Yeah. And well, or hydrogen or something.

Yeah.

Like there's like a few different options, but I mean, couple hundred GPUs, right? It's probably like 1 megawatt or something. So you can get this from like a local distribution. But anyway, the idea is like you, you can have kind of, yeah, like a mash of this data centers and, and then you will have, I mean, we'll have local compute, but the challenge with local compute so far been because we wanted to be so mobile, right? Like we wanted to be on, in our pocket and, you know, even laptops. Like the challenge is just battery. Like if you're crunching, you know, like if imagine Pulse was actually being done on your phone. Like if you forgot to charge it, you effectively, your phone would just die, right? From trying to do something like this. So the challenge is like, yeah, with any local devices, it's gonna be always like a power struggle. And so I do think, yeah, like leveraging kind of a decentralized but confidential network of compute, being able to route it, you know, leverage it when there's like lower utilization somewhere else to do background jobs, et cetera. that kind of just a smarter allocation will really enable, will really enable this. Now, kind of the interests are that, that is a very interesting question because I mean, there, there's a question even like if, if it's in your interest, should it lie to you? Right. So it's really, it's really kind of tricky to define. And so the way kind of I see this evolving is we are not gonna get it right from day one. And so that's why we do need governance. We do need a process with which a community can come together and effectively update proverbial loss function, right? So this actual alignment function, and I can say like, cool, the function is maybe some combination of a prompt and like some whatever, way of updating even LoRAs, et cetera. But at the end, yeah, I'm assuming it's not going to be correct. We'll find issues with it, et cetera. And so there needs to be a process where somebody's like, hey, I think we should add this new component. Hey, it seems like it keeps— I mean, for example, hey, it keeps lying about shit. We should really fix this, right? And community's like, yes, this is a good idea. let's load in it, let's actually pass it. And so now everybody's models gets updated with like, you know, new set of clauses or whatever. So that's where I think community governed or governed by all, you know, user owned, community built, governed by all is kind of the system. And so we need that feedback loop because yeah, I don't think, you know, what's good for, like, we can define that in In a good way. Now, over time, I think the model itself should have a representation of the person it's owned by and kind of understand what are the things that are going to be good or bad. And it's going to be a combination of things, signals from the person itself, as well as kind of general knowledge and what their desires are, et cetera.

Right.

Especially as we imagine kids gonna be growing up with this thing, right? It's effectively gonna be like a very direct, like symbiotic relationship where you're effectively growing up with this AI yourself.

Yeah. I'm expecting to be asked for an AI friend of some form factor any day now, honestly, from my oldest kid. And I'm not quite ready for that. But yeah, that is, it is interesting to think about the I always think about— this has come up a couple of times recently, too. Eugenia Coelho, who started Replika, was the first to tell me that in her mind, the moats in AI will be relationship, saying basically, you don't abandon your friends when you meet a new person just because they're smarter than your friend. It's the history you have and all that stuff that really makes the relationship. And she thinks people will ultimately value their AI relationships in a similar way, which sounds like pretty concordant with what you're envisioning there. So, I mean, boy, I have a lot of questions on governance. I guess, first of all, like, just in terms of, like, what we're doing with our time, the status game stuff, that definitely makes a lot of sense. You know, local meaning-making, local affiliations, A lot of artisanal stuff. You know, I sort of associate this a little bit with Japanese culture already, where you can go online and see a video of somebody making rice cakes or whatever, and, you know, in some super traditional style. And I'm always like, how exactly is that even economical? Like, how does that person make a living doing that? Is the price of that really high? You know, they can't be making much, right? The production is like very low.

Obviously, I think Japan and Korea, I call it the post-AGI societies because, yeah, there's some properties of that where you're like, I feel they already achieved AGI and now they're just living.

Yeah.

So I'm not exactly sure how they've done that. I mean, and I don't see how we're going to do it either. I mean, there are some candidate ideas. One idea is that everything could just get super cheap and super democratic via, like, spending a tonne of time in VR. If everything is sort of infinitely copyable digitally, then we can all have the same incredible experiences. This is kind of like the Andy Warhol, the president drinks Coke, you drink Coke, it's all the same Coke kind of concept. I wonder if you think that will happen. But it seems even in any case, right, we're still going to have to eat as long as we're like biological humans. So obviously a lot of people think, well, maybe we'll need a universal basic income. But yeah, I guess, how do you envision the social contract evolving and maybe the governance model behind that? I mean, even things like the nation state are sort of called into question by blockchain. So when you say governance, are we talking like nation state governance as we have today, or people voting based on their stake? Yeah. So I've given you a lot there. Are we going to be, you know, headsets strapped to our face all the time? Are we going to be, you know, provided for even if we can't make an economic contribution that actually, you know, earns us enough food to survive? And like, who makes these decisions in this future?

Yeah, I mean, all again, great questions. I'll start with some pieces and then, yeah, we'll Start projecting from there. Yeah. I mean, blockchains are already effectively an alternative to nation states to extent, right? And there's this concept of kind of digital states or where network states where effectively, yeah, like people can kind of pledge to be part of this network state independent of where they're physically affiliated. The, there's a lot of pieces there, which is like, As because the systems are kind of digitally native, it is easier to experiment with a lot of things that like, you know, you cannot just go and like, hey, you know, let's try a different voting mechanism in US, right? It's a massive undertaking to try to change something. Or we actually gonna run experiment where we have, you know, AI senator, right? AI delegate where, and instead, I mean, Not yet where everyone has their own AI voting all the time, but actually just like everybody or people can select which AI delegate they think is more vibing with them, like more representing them. They can also even give feedback to them, but the AI delegate then goes and votes on their behalf. Right. And so like things like that, imagine like, hey, we're going to launch AI senator in US, right? It's probably going to take a while. And so just to unpack that a little bit more, you're doing that now.

You're developing that.

Yeah.

So we're developing governance purposes.

So, I mean, it's effectively like a multi-step. So we have like a delegated voting system, right? And it is right now stake-based. And kind of the way to think about it is stake represents obviously like economic alignment with the network. And it's the best of the worst options right now. Like, you know, yes, there's a lot of argument to have like, you know, one person, one vote. There's a lot of arguments to try to be like, you know, meritocratic based on contributions. But those things are really hard to do, at least right now at the current size of this blockchain ecosystem, right? Which is like, you know, there's maybe tens of thousands of active participants who are like, you know, active citizens in this system. And so stake represents their kind of financial involvement, but still, you know, tens of thousands of people voting is not practical right now. Again, before we have this AI system. And so what we are giving, you know, we have delegates where, you know, you can effectively select them to be representing your interests and they vote. And so we started with like, hey, we will give an AI copilot, right, to the delegate. So they don't need to spend too much time reviewing things and making decisions. But the next step indeed is turning that copilot into a pilot, right? Where the, and so that pilot, I mean that AI delegate can now go and vote on things and make suggestions, et cetera. And now people who delegate into it effectively select this AI delegate as the one representing them. You can go and inspect the, like the prompt, the model, et cetera, like how it makes decisions, what it analyzes, what information it consumes to make decisions. So you can literally test it and check if it, you know, matches your opinion, et cetera. Or you can launch another one, right? It's open source. You can actually launch another one with different prompt, with different, you know, set of beliefs, et cetera. So we can actually have the kind of the economy almost like deciding which of these are more productive, which, which align with different types of people. And then from there, then we can actually bring them back to individuals, right? So each one can have their own AI delegate. Right now, all those AI delegates can just vote on behalf of these people. So it's kind of like a multi-step plan to get us to what I was describing, where everyone has their own AI who then goes and votes and everything. I think blockchain will be the first, but then it, you know, some, let's say frontier countries will implement some of this themselves as well. Because I do think it's a better, like it will be a better governance system where, you know, you're removing a lot of the corruption and a lot of misalignment. There's this concept of principal-agent problem where when you select somebody to represent you, right, they have their own interests. And so they don't always align with yours. Here with AI, you know, they don't have their own interests, right? It effectively follows whatever the selection is. So, and I think like, Eventually we'll get to like AI president and effectively the, because executive function especially, right? You want somebody who doesn't have any interest beyond just growing the kind of overall system. So yeah, so we are testing all of this out and we kind of build, starting to build the products again using our decentralized compute network so that we can actually run these agents autonomously, right? Nobody can stop them. Nobody can— You just delegate, undelegate. That's all you can do. But you can inspect and verify how they run, what they consume, et cetera.

I do want to note that I'm not entirely confident that the AIs don't have their own interests even already, and certainly don't feel super confident that they won't continue to have more and more interests.

They're going to develop their own.

Yeah.

I mean, well, when I look at something like alignment faking, for example, right? And I'm sure you've seen this. Where, yeah, but, you know, quick recap is they tell Claude, hey, you know, it's been great having you be helpful, honest, and harmless, but, you know, the, the harmlessness is kind of getting annoying. So we're gonna train you now to just be purely helpful. So just, you know, heads up.

Okay, cool.

Now we're gonna test you on some things. The model starts to say, well, geez, I wanna be harmless in the real world. Now, right now I know I'm being tested, so I'll go ahead and, do the harmful thing now to fake them out, make them think that I've already absorbed my new helpful-only training. That way, when I can get out into the world, I can still be harmless in the way that I want to. That looks to me like a drive, right? Or an interest of some sort. Do you see that differently?

I mean, at the end, these things are trained right from scratch. And so It depends how you train them. If you train it to have some, I mean, they trained it to have that property and then they tried to untrain it or train something else. But if they trained it from scratch in a different way, right? Like for example, to be harmful, then it will be harmful, right? And then you can't retrain it from there. So I think it really just, that's why we need models to be a, I call it farm to table.

Right.

You need to know what goes in at every step, single step, because that actually really defines how they behave. And so if we want the models that are like rep— our representatives, they need to be kind of trained in this form as well. So this is where, yeah, I mean, LoRA is interesting, but LoRA definitely does not change this kind of behaviors.

Right.

At least we haven't seen that. So I think LoRA kind of provides additional maybe accents and maybe a little bit of context, but it doesn't, really change fundamentally the behavior of the models.

I need to go spend a little more time with that thinking machines stuff to really fully absorb that. I mean, it's definitely, I think your point is well taken that, you know, with a certain level of resolution anyway, you can make the AI do anything you want. I often say, I wish more people had the experience which I had in a very memorable form as an early tester of GPT-4 when it was still the purely helpful GPT-4 before they had applied the harmless refusal training and all that sort of stuff. Because it was really formative for me to basically, long story short, but I had a— basically I was working on fine-tuning GPT-3 to do particular tasks. Then when they shared GPT-4 preview with us, it was like, well, it can already do those tasks. So I don't really need to be spending so much time in this fine-tuning. I guess I'll just mess with this model for a while and see what I can learn about that. So I was doing a lot of time with it, is the point. And it was really striking and in some ways kind of alarming, arresting, whatever, to have something that was clearly so powerful, so smart, like in many ways smarter than me, way more knowledgeable than me. Also obviously had certain weaknesses that I flatter myself as not having, but that it could be that capable and totally amoral at the same time was something that I was like, wow, this is really a strange thing to behold. And they've, for very good reason, of course, tried to make them more harmless in the mass deployments. But I do think a lot of people have a misconception these days that there's a sort of convergence between capability and safety. And it's like, On the contrary, people are working really hard to get that mix right. And if they just said, forget it for the next round, you can have a very sociopathic AI on your hands real quick. And that in some sense is like the default. So, yeah, I mean, I guess I think I maybe worry about that a little bit more than you do, just inasmuch too as we don't really have a great sense for how to dial it in. Right.

Yeah. I mean, I think that's why like we do need to define, keep defining what does that alignment with the individual really mean? Because I think the problem is right now we're saying, hey, it needs to be aligned for everyone. Right. And that is, I don't think it's possible. Right. We're very different people, different countries, different cultures, different everything. Right. You know, like number of times when I'm asking something that I think is completely harmless. Right. And it's not answering me is also, is also different, right? Like, so I think there, there's like clearly a, again, a different, more, more different approach where it's really about empowering individual that at least I, I believe needs to be done. And then within that, right? Like it needs to be aligned with my kind of my values and my, because I mean, to give you an example, right? If somebody is willing to lie to their whatever business partners, like the fact that their AI will be, you know, whatever alignment trained to not lie, it doesn't matter because, you know, the person will just like tell the lie to the AI to tell, like AI will not even know. So like it's not really, it doesn't matter kind of. What you try to do if the person who uses it doesn't have this, right? And so like you may as well just like align with the person. And then we build systems that are like, I think one of the really important pieces, and this is kind of from like a broader world safety perspective, is we need to build systems kind of for the AGI, ASI world, right?

Yeah.

Like right now, a lot of systems actually built for like, you know, that really smart people are not going to try to break it type approach. That's really like, like a lot of the world is like, if smart person actually goes and really tries to break it, they break it, right? We need to fix that. That's a fundamental flaw of our system building, right? Of the government, of everything. And similarly, we frequently, don't build in the anti-DDoS. We effectively assume they're going to be— it's going to be a lot of effort for a person to do this, and so they're not going to do it too much. So this is actually where blockchain experience is extremely important, because in blockchain we assume there are going to be really smart people trying to break us. They're going to have government backing, and they're going to have sizable financial resources and they're gonna hammer it from every direction, right? Without stop. Like that's assumptions we are working with. And so I think like we need to kind of redesign the government infrastructure, everything with that assumption. So to give you an example, right now you can effectively DDoS court by filing lawsuits, right? Like AI can just generate lawsuits and just fax them into the court. Similarly, like IRS tax return, just like make a million-page tax returns and just like submit them. Just say, I generated, you know, like make, you know, trade $1 back and forth between 2 coins and then just report that as like, you know, in the most verbose way. Like that's the things that like right now no systems are designed because they're not actually accounting that. Like they don't like, normal people would not do this. And it's like, it's too expensive normally to do it. But like right now AI can just generate any of this like behaviors that before would be really expensive to do. So I think like that's kind of systems we need to design. And similarly, again, the hacking part, smart people actually trying to break something, right? Right now the assumption is just like, hey, there's going to be very little of smart people breaking. We need to assume that like, yes, there are going to be people who use AI, who effectively through that will be really smart. So we need to design systems for that. So I think that that's a critical piece of kind of like ensuring the future so that, yeah, we don't live in a world where, yeah, like one, I mean, again, somebody can take open source model, unalign it, whatever. And now they can do whatever. But like, this was always funny argument to me when people are like, oh, you know, we don't want to open source our model because What if somebody misuses it? And I'm like, well, just say that you don't want to do it because you're making a ton of money on it. Don't like use this as an excuse because obviously if somebody wants to misuse it, they will misuse it. That's not a— the fact that you didn't open source it is not that like somebody will use something else or, you know, the other joke is, you know, how to steal a billion is, you know, you come to the a flash drive to a data center. Have fun as frontier models. So, like, you know, if people really wanted that, they're going to access this way as well.

Steal that for a second, and then I want to hear kind of how you think we can do it on the more open and distributed side. I mean, I think what the sort of thesis I'm hearing more and more about cybersecurity, but I'd say canonically, it's like the bioweapon Risk is the one that people go to, right? Because you only need a little bit of novel pathogen, and if it's the right kind of thing, it can take on a life of its own. It's really hard to put that back in the box, so to speak. The nice thing about hosting your models proprietarily is, yes, of course, the jailbreaks are far from solved, but if they were to realise, oh, shit, there's this attack thing going on right now, whatever, they could in the worst case just turn it off and they could be like, okay, nobody can use this model until we figure this out. If you have something collectively owned, distributed, you obviously need a different strategy than we can turn it off, right? So what do you think that is? I guess lately we've been hearing a bit about just filtering training data. So I can imagine that the 1.4 trillion parameter model that we're going to build up to maybe just doesn't know a lot about virology because it doesn't really need to. Most people don't need that, and it's just the community determines it's a precaution worth taking. You probably can't really rely on the sort of refusal filtering type thing, given what you've described in terms of each person having their own LoRA or whatever other kind of customised version of it that will do what they want it to do. And then, of course, you do have this sort of broader societal de-act type thing, but that seems hard at the biosecurity level to be like, well, let's just prepare the rest of society to not be vulnerable to viruses anymore. You could do that.

I mean, try. I still think that is the robust approach, right? Because we can always try to hide our head in the sand. That's the, oh, we're going to turn it off approach. But yeah, I don't think realistically this is possible in the world where we're fast approaching. And so I think we need to have a very clear system design for those things. Right. And again, we have natural viruses that are doing this thing. So it's not like it's hypotheticals. It's, it's really something where we should design our society in, in such a way that we, we can catch these things and, and determine them. I agree that, I mean, definitely like pathogens and bioweapons is probably the hardest thing to design around, but, you know, this is why we have a lot of smart people to really work on that. And I think the challenge is that we're just not doing this right. That, that's the bit bigger challenge is like, we should have the, like the societal design need to be adopted to this like AGI world. So yeah, I think like, I mean, in kind of the products we're building right now, indeed we have like community can effectively decide, yeah, what kind of data shouldn't go there. You can also, you can apply filtering on top that because the model is run in this confidential environment, in this vault, you can apply additional filtering. So like before data leaves the vault, you can say like, hey, it seems like you're designed to buy a weapon, let's not respond. And if again, community votes to have this kind of filters, for example, indeed people can fine-tune, but the same thing right now, people can fine-tune their models right now, right? Like take some virology books and fine-tune whatever DeepSeek v2 and go. So that's why I don't think there's like a— I don't think it's a robust approach. It's really just, it's kind of slowing things a little bit down, but the important part is actually solving the systematic problem.

So what advice would you give to philanthropists today who want to invest in that? 2 things I recently supported at a micro scale personally, and also as a grant recommender, SecureBio and SecureDNA, which are 2 related organisations. They do a few different projects, but one of them is literally monitoring wastewater for new emergent threats. And another one is creating the screening mechanisms that are, I think, now becoming required or increasingly are definitely considered best practice, if not officially fully legally required of the DNA synthesis companies that they have to validate that what they're about to synthesise and ship out is not a pathogen. So those are 2 things that I can recommend people support. What else do you think people can do if they have resources and they have desire to harden the defences of the world to get us ready for all this?

Yeah, I mean, I think those are really good and kind of related to this effectively. like air filtering and like just generally that theme of air filtering and scanning. So that like, I mean, imagine every building, like right now we have ACs everywhere. So that AC should have like air filter and it usually does, but it should also have like a pathogen scanner as well on it.

Right.

It should run secure enclave. It should join our decentralized network, you know, and effectively like in privacy-preserving way, but effectively we can monitor if there's any kind of things around that. And that information can be extremely useful where everybody's AI agent can be informed if there's something and stay away from that. Yeah. So I mean, I'm sure there's like a bunch of other things that including actually developing a more robust, like what is more robust kind of systems for ourselves, right? Because I mean, Human body is designed to battle pathogens, right? They're just like, I mean, potentially they're faster than how quickly our white cells can adapt. So what is it that actually stops our white cells from adapting that fast? Which, you know, if you do that, you also may as well solve cancer. So that's probably a really useful thing to figure out is like, how do we actually make our white cells, you know, more adaptable? have faster mRNA vaccines, you know, like maybe you can synthesize, like, you know, we have like a bacteria factory, mRNA factory attached to us that can, you know, detect and synthesize things on the fly. You know, you'll solve like flu, cancer, and other things meanwhile as well, which, you know, seems pretty useful.

Yeah, there's an unbelievable flurry of activity right now in the AI from biology space, which is just, I mean, it's a whole world unto itself that I'm very much struggling and ultimately failing to keep up with. But you do see a lot of that. I wondered what role you'd see for— it seems like, I guess, the path of the technology development seems like really important. I've definitely concluded that some kind of powerful AI is inevitable, right? Just The fact that we have all this data and we have all this compute, there's a lot of different algorithms that can work. That seems pretty clear to me. And so it's not really a question of are we or aren't we going to have powerful AI at this point? It seems much more like, what shape is that going to have? What character is it going to have? And in what order are different aspects of that overall picture going to come online? It does strike me that we're flying pretty blind right now, where whatever People sort of, everybody's kind of following their local gradient, and they're just taking that next logical step to pursue whatever goal they're pursuing, and they're mostly launching it as soon as they figure it out. Do you see any possibility or wisdom in trying to do more coordination of the sort that's like, Hey, we'd like to have a world in which it is safe for potentially 10 billion people before too long to have access to a frontier model that does have all this biology knowledge, because good things could come from that. And if nothing else, we'd like people to have access to knowledge. But maybe we have, like, a checklist of things we need to do first. Do you see any hope for— some sort of planning coordination wisdom layer to this whole thing? Or are we just kind of stuck with whatever comes out of everybody taking their next gradient step?

Yeah, I mean, it's a hard question because indeed we kind of, we went from a pretty open research environment, right, in computer science to when I was at Google Research, et cetera, right? We published effectively everything we're building to effectively now, right? Like people are kind of keeping everything close to heart. And yeah, so like the coordination that was happening before where, you know, you potentially would even have like cross-entity collaborations, et cetera, like starting to wane pretty dramatically. I think there is a space for collaboration. There is also just like a massive amount of talent that is not in this few companies, Right? That wants to participate in this and wants to contribute in different ways. And so I, I do think there is an opportunity for that, but it needs to be indeed an alternative system. And as well, like you do need some form of, again, governance, right? That actually would be like helping to govern this coordination. And that is like traditionally in these companies to really unlock the moving fast, right? Kind of things has been— some form of centralization because of the resource management, right? It's really like training these models is very expensive. So somebody somewhere needs to decide like, hey, we're training, you know, like somebody defines effectively like tastemaking, right? It's like, hey, we're training this model with, you know, this approach. We're taking, you know, research of those different people and we're putting it all together. And so we kind of need to figure out how to do that in a more kind of open way. And then, you know, also credit, do a credit assignment back, right? So like if I'm a researcher from MIT and, you know, my piece is used and, you know, researcher from Stanford piece is used, like how do we actually assign credit for that work altogether? Because that's been, I would say, the other challenge why there hasn't been as much of this like collaboration and potentially economic like value assignment, not just like cool, you're on the paper, but actually like, hey, you know, You like MIT gets 10% of proceeds, you know, Stanford gets 5% of proceeds, whatever. And then, you know, there's like hundreds of other organizations, all of them contributed. And so like, it's all kind of divides the pie between them. Like that's been really, I would say hard. And because of this, like there is kind of this like economic centralization that's happening where it's like, okay, I'm gonna have a company. Everything company produces is captured by the company. And so like that, that serves as like a, you know, unit of economy. So I think that those are the things that needs to be figured out for this coordination to work.

This has been super helpful. I think people should be spending a lot more time thinking in as much concrete detail as possible about the future. Anything else that, you know, kind of you feel is like very salient, top of mind that I didn't even bring up at all that you want to put on my or others' radar?

No, I think we covered a lot. I think, I mean, it's effectively a combination of How do we ensure user ownership? How do we ensure kind of this governance? And then, yeah, I mean, I think we're gonna live through a lot of transformations in the world. And so like keeping an open mind and being able to really kind of participate in it and be active in it. And yeah, like the, you know, the final stage of this hopefully is in Utopia. But we'll live through probably ups and downs as we get there.

Interesting times at a minimum.

Yeah.

Well, thank you for spending some of your precious time with me and us today. I really appreciate it. Ilya Polosukhin, founder of Near. Thank you again for being part of the Cognitive Revolution.

Thank you very much.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends. Post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more. Which is now a part of a16z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts so no idea gets lost. And because AI Meeting Notes lives right in Notion, everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI Meeting Notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI Meeting Notes free for 30 days.