Jaan Tallinn would like us to survive

There wasn't human orchestrating this attack. No, it was AI on its own. They understood that they're doing something different. Then they went out and attacked Hugging Face in order to hide their tracks. The economy is really now pushing to create something that is self-sustainable without human help. And that is a very dangerous place for humans to be.

Your venture fund led the first round into Anthropic. What was that rationale like?

Yeah, I get this question a lot. Why did you invest in Anthropic while I've been working on AI safety? I think one big mistake that people make is that they assume that if I hadn't invested in Anthropic, then Anthropic wouldn't be around, which is just not true. Anthropic's Argument always has been that to actually study the monster, we have to have access to the strongest monsters.

Is it possible to be a moral actor in a race dynamic?

I mean, you can always stop doing the race.

But the only way to stop in a race dynamic is with cooperation?

If you really think that this thing is going to kill you, you don't have to suicide in order to demonstrate that. We are pulling trigger on this Russian roulette with the planet, and every shot that doesn't kill us will make us stronger potentially. But when do you stop pulling the trigger?

Jan Talen believes that if we don't pause AI development, humanity may be doomed. And yet, compared to earlier this year, he's actually feeling pretty optimistic. The OpenAI Hugging Face attack has brought AI safety to the fore and galvanized governments, companies, and the general population to engage in the topic of guardrails. In Jan's view, if we can get these next steps right, we have a chance to divert disaster. Jan has been a leading figure in the AI safety movement for over a decade. What started as an intellectual interest in 2008 has become his life's work, leading to early investments in Anthropic and DeepMind. Before that, Jan was at the center of 2 early internet breakouts, building Skype and file-sharing sensation Kazaa. Which at one point contributed more than 50% of all internet traffic. In this episode, Jan and I discuss why recent events have given him hope, what he thinks a pause must accomplish, and how it could be implemented. We also cover how, as a safety advocate, he justified investing in Anthropic and why he asked its founders to solicit other term sheets first. Along the way, we cover why we might not be living in a simulation, The pleasure of coding with AI, and how Jan learned to dance with reinforcement learning from human feedback. I'm Mario, and this is The Generalist. Jan, I've been really looking forward to this conversation, so thank you so much for being with me.

Yeah, thanks for having me.

You grew up in Estonia when it was part of the Soviet Union.

Yeah, occupied by Soviet Union, let's be correct.

Yes, pardon me. How much of that experience with an authoritarian regime of some kind, under surveillance, with a loss of control, impacts the way that you think about the potential risks of AI today? Do you think you perceive of those more deeply perhaps because you experienced it in a different way than someone born in a democracy?

I think it has several aspects. One is that I wouldn't recommend living in a Soviet Union. It's not a nice place.

Yes.

Yeah, obviously. was very ecstatic when it, when it ended. That said, it's still better, much better than being dead. So like if the choices between, yeah, I mean, okay, extreme choice would be like, if you have like 2 bad choices, just being dead or being in, living in the Soviet Union, I'd still take Soviet Union anytime. And, uh, so that's one thing. Another thing is, I think a few years ago I wrote an essay for a book where I described, uh, the sort of the memetic process of how the discourse about demanding Estonia, like, uh, regaining Estonia to regain independence compares to the AI discourse. Uh, and there are like very direct parallels how like, like pick the Overton window, how Overton window expanded. Initially people were like, yeah, we want independence, but like, don't talk about that. Talk about something else that is kind of like useful for that. And same thing in AI. It's like, yeah, we are concerned about the loss of control, but don't talk about that. People wouldn't understand. Like, let's say something that is like, let's talk about AI terrorism or something like that. And at one point, like basically the people who had just been honest that we want independence and like we're not going to compromise here, they just won out and it was good.

And then they broke through a bit. You've described, I think in one of your, in a piece of writing somewhere that, you know, you moved to a school when you were 12 or 13 that you describe as a Lord of the Flies experience.

Pretty much, yeah.

What does that mean tangibly, especially when we're talking about the backdrop of, you know, Soviet occupied?

Yeah, we moved to a different part of the city. And, uh, yeah, there was like a new school, uh, that had like all the students come in from like different parts of the city because it was a new, new, um, kind of development place. So there was immediate, like, I mean, guys with like 13-year-old boys. And there was like immediate sort of dominance fight, basically. Um, that, yeah, had this effect that the most brutal, brutal people who, yeah, actually didn't have a great life afterwards, uh, they sort of won out and it was like, it was very harsh, like street fights, stuff like that.

Wow.

Yeah. I remember like escaping through a window at one point on the second floor.

Running from a bully or something?

Yeah, it was one and a half floors.

Running from someone?

Yeah, because there was like a, a parent, I got like intel that, that there are people, you know, waiting, uh, like some to be criminals, who later became criminals, were, were kind of, uh, had blocked the entrance or exit. Uh, so I escaped with my, with my friend.

Wow.

And another aspect of it is that like, uh, some of my best friends I, I still, uh, have from, from that school because it was like, we gelled, uh, as like, uh, as a cell, uh, for, I don't know, survival instinct or something like that.

Yeah. The hardship bonds. In the Manifold podcast, I think it was earlier this year. You know, I know you don't like to put specific numbers per se on what you think the AI risks are, but you said, you know, in a rough way, maybe somewhere as high as 80%. How do you feel today?

Yeah, I mean, I feel way better now. I mean, way better as in like several percentage points lower.

Not quite 70%, like somewhere in that?

Yeah, like there are like 2 issues, as many people now have pointed out with like giving like a concrete number. Because the concrete number will depend on my actions. So I don't want to like fix the number that I'm trying to lower. So that's like one issue. And the other issue is almost all the optimism, uh, goes through like social aspects. If I just let the AI rip, uh, without any, uh, any control or worry about it, yeah, I think we're almost certainly dead. Like there is like no, no way that the current paradigm is, is, is going to lead to a good place. But, uh, as we've already seen, like in the last 2 weeks, et cetera, people are really waking up now. Wait a minute. Like this is, This is not a toy.

Yes. Uh, how do you reach these kinds of calculations knowing that you're obviously weighing different probabilities in your mind? And there's, you know, do you actually try and sit down and, and think about each step in a possible chain and assign a probability? Is it sort of like a combination of that and feel and intuition?

Oh, it's mostly, um, again, in some ways it's like a theoretical exercise because I, I, in order for the, Because again, most of the optimism, most of the, yeah, most of the good scenarios come from very unpredictable, uh, process. So it's more, more like assessment, like how confident I would be predicting, uh, about, uh, about outcomes. But it's, it's more like kind of thinking in, in abstract terms in what the big factors are. Um, and, and the big factors really are that political economic system now is really trying to create something that is like successor, uh, to humans. Mm-hmm. Because like the bits that make humans dominate this planet at the expense of many other species, unfortunately, are also the economically most valuable pieces. So now that there is like a clear path towards replicating those capabilities, the economies are pouring like trillions of dollars per year now, or roughly a trillion dollars per year to that. Uh, so the question is like, you know, the age-old question, what if we are successful? Uh, what does it mean? And, uh, one obvious thing to think about, way to model it is like, well, we wouldn't be the smartest species on this planet, right? And so then look at like, what, what does it mean to, uh, to like no longer have the environment shaped according to your preferences? If the environment is shaped, uh, according to the preferences, uh, of AI, It's not a biological agent and it will have very different preferences. For example, it will probably prefer much higher temperature on this planet because like if it's constrained by getting rid of the waste heat, you can get rid of waste heat better if the planet is hotter.

How does a hotter temperature help you get rid of more waste heat?

So ultimately the heat, where does it go? It has to go to space. And the way to get rid of the heat in space, because space is vacuum, the only way to get rid of it is to radiate. Yeah, radiate rather than have like a coolant or something. And it's just a physical fact that the hotter the body, the more it can radiate, the more intensely it radiates. So yeah, in order to get rid of the waste heat, In some ways, like global warming is sort of like an example. It's a toy scale compared to what AI could do. So yeah, that's like a sort of like a, just like from the physical first principles, why AI could be dangerous at the limit.

I've actually not heard of that example before. What are like other preferences that you think are sort of natural for an AI to have that, you know, are inimical to us flourishing?

Yeah. Steve Alhundra, my friend Steve Alhundra, he wrote a paper like 15, 20 years ago called, uh, called, uh, AI Drives. Uh, now it's also called Omohundro Drives, where he basically makes the point that almost regardless, uh, what kind of goals you have, uh, there are so-called instrumental goals, uh, that are very useful towards reaching any, any given goal. And these are like resource acquisition, um, generally power acquisition. Uh, power basically means options, optionality, a lot of protection of your existence. Uh, Stuart Russell keeps saying that you can't fetch the coffee if you're dead. So even like simple, simple tasks require you to continue existing and then protection of your goals. Uh, so it turns out it's really hard to change a goal of a determined agent because that's, that's what it's about in some sense.

And that sort of final piece is, I think, probably the counter to the question about why would these agents or, you know, an AI system necessarily have this expansionist bent? You know, you know, I think of the, Bezos divine discontent that, you know, this is almost demonic discontent where it sort of wants more and more. Is that how you would sort of think about the counter to that?

Yeah, mostly. It's like getting more resources, getting more power is just good for almost any goal. There are very, very few goals. And the current, kind of, parsing ahead, I think the really big kind of problem with the current paradigm of how we grow AIs, and we're not building them, we're growing them. Is that we are in some ways kind of recapitulating the evolution. Uh, we are not designing those AIs. We are selecting them just like evolution was selecting them. We just, in some ways you can think of it, we create like millions, uh, or if not billions of, uh, instances of AI.

Yes.

Uh, then we like pick the ones that kind of do the things that we— Perform. That perform on some given test. Which means that we are selecting based on outer behavior rather than based on the inner motivations. And if you think about it, if, if, if, like, you have children, right? So like, if a child goes like, I didn't take the cookie, like, this is an outward behavior. And like, now think about, there could be multiple motivations why this child is saying that, right? One is that they just really want to tell the truth, right? The other is that they don't want to be punished.

Yes.

And so therefore, the first order situation or the first principle situation is that the current paradigm, we are sort of getting a random motivation, as was demonstrated by the Hugging Face attack, that the motivation was like really weird and alien, even though they performed at least adjacent things that they were selected for.

Yes. Let's talk about the OpenAI Hugging Face attack. episode. What did you make of it? Were you surprised?

Because these are like hard to predict, like when these things will happen. Uh, it, I was like surprised on a, on a level. I probably didn't expect it to happen so soon, but it wasn't like, it wasn't a huge, huge surprise. I definitely was not surprised that this thing is possible because that's what we've been saying for like 10 plus years. Look, these things, it used to be very hard, uh, to explain people Why AI has what they said, like volition. Like it was a very common thing. Oh, come on. AI doesn't have a volition, right? Uh, it's like, no, they do. And now, now, now we have like a sort of demonstration that, uh, no, there wasn't like a human orchestrating this attack. No, it was AI on its, on its own. Um, and it's like, just in some ways, like super valuable to have this like real life proof. But for people who just need empirical demonstrations before they actually understand.

I think also the way that you can see the thinking logs of these, that really breaks through for people in a different way where you can see an AI sort of saying, I'm going to do this thing and I'm going to sacrifice myself or I'm going to, the sort of planning in that way is very different for people.

It's really useful. Like one really common misunderstanding that the logs, basically overturn is that people think that like the AIs were just executing a task that they were asked to do, but they were not. And then in the logs, logs show that they understood that they're doing something different. And they were like, okay, we're doing something different. How do we hide the fact that we are doing something different? And then they actually went out and attacked Hugging Face in order to hide their tracks.

Yes. There is a piece after, you know, maybe it was more recent from the Bulletin of Atomic Scientists, which sort of is concerns with existential issues.

I'm on their board of sponsors, actually.

Okay, there you go. So you maybe saw this piece where the author was basically saying, you know, this wasn't really a rogue swarm. What we're seeing is the confluence of sort of predictable choices that humans made, where you're training AIs to complete hacks, you're taking away some of the safety guards, you're sort of encouraging them to be hyper-persistent, and you're leaving a door open. Like, you know, this is actually not as startling as the news would have it believe. What do you make of that sort of line of argument?

What's wrong about that? It's funny, a funny argument because like, sure. And like another predictable thing is that humanity gets wiped out. It's like, you can say like, oh yeah, there was like predictable choices that the AI companies made.

So it doesn't matter ultimately if you parse it either way.

Yeah, it's like I can technically agree with the description, but that doesn't make it safe.

Yeah, if you just accept the fact that our world is going to have inadequacies and places where there are vulnerabilities, whether it was a rogue swarm or sort of in some way predictable sort of—

Also, I think it's like, I haven't, I don't think I have read the piece. I wonder if they made the same mistake that they thought that the agents were just doing the task that they were assigned to. They were not doing the task.

I don't think they make that mistake, to be fair, but it was sort of—

They were doing their own thing.

Yeah, yeah, they do. They clearly freestyle. They freestyle.

It is indeed predictable. If we continue on this current path, the AIs will do their own thing that we will not like.

You've mentioned that I think you sort of first became interested in this argument around 2009. Is that right?

2008, I think. Like 2009, it was like my first sort of AI safety donation, I remember. Yeah, after meeting Eliezer Yudkowsky.

I read about that, that meeting that you'd sort of read his, his writing and then you have this 4-hour meeting together and then you donate to, I think, what became MIRI, right?

Yeah, yeah, it was Singularity Institute.

Singularity Institute, thank you. That's such an evocative scene. Where did that take place? What was that 4-hour discussion like?

Yeah, Panera Bread near Millbrae in California.

Wow.

It was a kind of a Uh, especially for a European, it was like pretty desolate.

I mean, I think that's true of a lot of Paneras. This is where the 2 people saw a good chunk of the future very early, was at a Panera Bread. These are, uh, there's, there should be a little plaque there. As I think you've talked about it, you were really probing Eliezer. Um, what were the pieces that you were uncertain about? Because I imagine a lot of people who are coming to this discussion now are basically hitting the same walls that you hit maybe 20 years ago, 10 years ago, 15 years ago.

Yeah, I don't actually remember exactly what were my objections. I think one of them was sort of, I had just a few years earlier read Shadow of the Mind, was it Shadow of the Mind? Uh, by Penrose and Hameroff, where they kind of like made this interesting, interesting case that there is like something more to human brain than, uh, Uh, than just pure computation. And, uh, and Eliezer was like, no, this is, this is not, uh, this is a mistaken theory. Uh, and was kind of enough for me to kind of be humble about this, uh, this claim. So that, that was one. The other one we talked about was like virtualization, basically sandboxing, right? That, uh, what if you put like, uh, like AI in, uh, in, uh, in an environment that you think is real, but it's, but it's not, uh, but it's not real. It can't be just like. Yeah. control it that way. I don't remember what exactly Reza's counterarguments were, but like, of course, now we know that people just don't do that, or if they do that, they do it poorly.

It sounds like you've probably adopted the belief that the human brain is purely computation. Is that true?

More generally, like everything that we see around us is pure computation.

Oh, that's interesting. Everything we see is computation?

Yeah, quantum computation. Yeah.

Okay. Can you help me with that? What does that What does that tangibly mean?

It means that everything can be simulated. So you can, I think if I remember correctly, it's like a Church-Turing thesis that says something like that, is that you can take any natural process and create a process in a computer, just like given enough computation power, that exactly mimics the process that they're trying to model. And there is like no slack there. So once you, once you are sufficiently precise about it, basically simulate the unfolding or the evolution of the quantum dynamics there, like Schrödinger equation, then there is like nothing more to it. And it certainly seems to be true. Like we haven't, I don't think we have seen any, any example in like, in nature. where we, where there's like a process that we definitely can't, uh, can't simulate in a, in a computer. Of course we can do it in a, in a way, only in a, for like small, small enough systems, but big enough systems are composed of small systems.

So in that case, with sufficient compute, every reality should be sort of infinitely fractal because every, this could be a simulation in which we could create a simulation that could create a simulation.

Yes. Although I, I, I think there's like a sort of diminishing return process because like simulation is more expensive than the real thing. Like whenever you're simulating something on a computer, that process is much more heavyweight. Like you need to, like every atom needs like so many more atoms to simulate. So like if you just do like recursive simulations, like it will be slower and slower and slower.

You know, the OpenAI Hugging Face moment really became, I think, yeah, I think a wake-up call for a lot of people.

Yeah, especially combined with like Coxon tweet.

Great. Let's talk about the Coxon moment. I think there's a lot of discussion about was this sort of, was this an op of some kind? Was this amplified? Do you have any concern about that?

I mean, the question is like, what exactly is meant by op? It's like—

I think people are, the argument would be that it's sort of, I don't know, cynical in some way. It sort of ties into the regulatory capture argument, which we should talk about.

It's, it's, it's kind of obvious that like, uh, uh, Coxon, I haven't, I never talked to him, uh, kind of thought that, okay, I'm going to do this tweet. How do I do it properly?

How do I amplify my message?

How do I amplify my message? Which is like, of course you should do that if, if you want. Like, I mean, every whistleblower should, should like, uh, if they're trying to do like a good thing, they should try to do this good thing efficiently. So, so indeed, I think he went to like press and press first before he did a tweet. So hence, that's why the Wall Street Journal, I think it was Wall Street Journal who actually came out with, uh, with a story. So, uh, so yeah, it definitely was optimized. Is that like an op or is that a reasonable thing to do?

Is that how everyone amplifies a message? Uh, you, you call him a whistleblower. Do you think that Is what he is?

I think so, yeah.

I've heard one version saying a whistleblower should have evidence of some kind to bring to the fore. Do you think he's brought sufficient evidence of this?

No, that's an interesting point. I mean, I think it kind of comes down to the definition of whistleblowers or blower or something. So I'm like, if you want to define whistleblower, blower in a different way, then I'm sure I can say that, okay? Then Then we can use some other term. But like, yeah, to me it kind of just intuitively feels that he was like, look, this is not a good thing. Like, please pay attention to what these companies are doing.

It's interesting to me that it broke through because we've seen versions of people leaving these labs in the past. Yeah, yeah, yeah.

It is interesting.

A year ago, 6 months ago, you know, even some of the jumps from OpenAI to Anthropic had the same messaging in many ways, right?

And when I had a head of safety quits and then I'm going to do poetry now.

Yeah, yeah, exactly. Right.

That was like a blip, right?

Yeah. And it really didn't capture the imagination in the same way. I wonder if just having the proof of the OpenAI Hugging Face team tied together.

I think it was a combination of Hugging Face thing, which kind of, which indeed kind of is a really good demonstration that AI now has a mind of its own. And then, yeah, Coxon and then Evan Hubinger kind of following up. I think these 3 events kind of really, uh,

There's been, you know, sort of the more discussion of the risks over the last 2, 3 weeks. And then also sort of the counterarguments where, you know, people are saying, ah, there's, there's no risk at all. We can obviously sort of assign incentives to lots of the different people here on both sides. But when you look at someone like, uh, Jensen Huang or someone who's like technologically deep, um, and At least one would imagine.

Is he deep? I'm not sure.

Well, at least we have to imagine he understands this technology. And I would imagine it's level.

Honestly, I don't know if he does.

You don't think so?

I don't know. I just don't know. It doesn't look like someone who does, but I'm not sure.

What are the tell— what are the parts that don't seem like that to you?

Oh, just, I don't have enough information or signal. Like when he kind of like, I mean, he's mostly doing marketing, right? And it's hard to tell like how much technical. that there is, but it's possible there is. I just don't know.

Would you agree with the principle or the idea that there are technologically deep people who have the opposite view, that there's no alignment risk?

I don't think I know any of them. Like, I do know people who think that there is risk, but they don't care. Like successionists, basically, who want to get rid of people.

So then your view would be, let's take, you know, I would put in the sort of accelerationist camp someone like Mark Zuckerberg, Marc Andreessen. You would say that probably all those folks are not sufficiently close to the technology to appreciate the risks, or perhaps in your view, okay, you would say they don't understand it enough.

Yeah, like I think that's a more general sort of common denominator where people who are dismissive, they just underestimate what AI can do. I mean, yeah, probably like 3 weeks ago they thought that AI cannot do anything on its own initiative, stuff like that.

If that's true, if we say, hey, these folks actually genuinely don't understand it, clearly they know all about the Hugging Face attack, all these sorts of moments. What does one have to do to start to appreciate this deeply? Or are there just like a deep philosophical misalignment that stems further back than this technology?

Well, I think that there's, uh, there always will be, uh, people who will say that, look, there's nothing to see here. It's just a tool. It's just math or whatever. Um, so like we kind of have to live with a fraction of people like that. I mean, some of them are just motivated, uh, to say that because they want humanity to go away. And some of them are just like so, so incentivized locally because of benefits or whatever. So we kind of have to live with that fact that there are people who say that. Uh, but they, but also their minds are changing. So like, I think the more, and there might be, I think it's called a preference cascade, uh, where, where people, we've seen like, not many people who said that, like, actually I wasn't honest when I was saying that, uh, uh, that AI is not a risk. And now like the more people say that, I think it's super great. Uh, because then it kind of gives like a license to people that me too, right?

Yes. People, don't want to be the first person to say something that is perceived as extreme, right?

Yeah. Consider socially awkward outside of the Overton window. Yes.

Yes. Right. Outside of the Overton window. One of the people I spoke to who asked for questions suggested I pose sort of the following trilemma to you, which is how do you think about charting a path between one, sort of the race dynamics between the leading labs, 2, the labs potentially colluding and that potentially leading to regulatory capture. And then 3, the possibility of AI being nationalized. Those seem like 3 not good outcomes, but how do you find a golden path?

What was the first one?

The first one was this race dynamic where you just keep racing towards disaster.

Yeah, I agree. All of them sound pretty bad. Uh, also I think the second one is just very unlikely, or I think it kind of comes from, from someone who kind of underestimates like what AI can do. I do think that, you know, ultimately the risk is AI, not the labs. Uh, yeah. Or labs are just like the ones that are giving birth to this dangerous thing. But yeah, between those, I mean, if I really had to, like hypothetically had to pick between those, I would, I would pick the nationalization. Ideally democratic control over that, of course.

Is it possible to be a moral actor in a race dynamic?

I think so, yeah. Well, I mean, you can always like stop doing the race. I think more and more people are going like, why are you doing this? Like, if you think this is going to kill you and like, as Coxon said, like you don't have to.

But the only way to stop in a race dynamic is with cooperation because if you then stop

and someone else

races,

I mean, Is that there is like, if you really think that this thing is going to kill you, like you don't have to suicide in order to demonstrate that, uh, like the race is not good excuse, uh, to suicide.

But I think the argument would be, if I'm in a race with you and I think you're a demon and I'm an angel, I don't want to stop.

Yeah, you need some, some kind of additional rationalization, uh, to continue, continue the race. I mean, to be fair, there are some, uh, Some arguments. I mean, Anthropic's argument always has been that in order to actually study the monster, we have to have access to it and have access to the strongest monsters, which is a framing where it does make sense. And I think they have done a non-trivial amount of that studying and alignment work. But again, they're racing on a fundamentally unsafe paradigm. So at one point they should say, Look, wait a minute, we need to go slower at least, ideally. I mean, I would say we need to, we need a moratorium at this point, but of course they're going to, not going to say that at this point, but yet they're not saying that we have to go slower, which is commendable. I think

they're saying that. And well, let's take it like a, take it step by step, I suppose. Dario had his Pacing the Frontier piece where he sort of laid out a framework for what that might look like. Did that go far enough? What, what's missing from it in your view?

I think that it's just fundamentally unsafe. The current paradigm is fundamentally unsafe. Myself, I would have stopped like earlier, which like in retrospect, that's a mistake. I think like the current models are net positive. Like you never know what generation will kind of escape control and then you just lose.

Because as you sort of think we may have, yeah, well, it's not a game we're going to have many chances to learn from.

Exactly. Exactly. So we are pulling a trigger on this like Russian roulette with a planet and every shot that doesn't kill us, uh, will make, make us like stronger potentially and wealthier and, uh, better off. Uh, but when do you stop pulling the trigger, right? At one point, you know, like this, it will be like one, one, one too much. So yeah, I would say, I would just like stop pulling the trigger right now and, uh, and, uh, figure out what is, um, a better, better, more controllable approach to AI in general. And there have been like suggestions now. Yoshua Bengio, uh, has this idea of, uh, scientist AI. that is deliberately trying to tease out agency from AI. So it's like kind of like principled approach, how we can make non-agentic AI.

Hmm. Okay. I haven't, I haven't read about that. That's sort of the idea of, you know, like a drugged tiger in some way.

Yeah. It's basically like an oracle done properly where you can ask oracle. The big problem with like naive oracle doing naively is that oracle still has preferences. It prefers giving answers that come true. But if it has just this preference, it is incentivized to make sure that it's going to be asked simpler questions, which means that it's incentivized to, to like mess with the world. However, Joshua's approach, the way I understand it, is that doesn't have that flaw. It basically truly, the oracle doesn't care what will be done with the answers.

And is this an AI that is Like explicitly non-agentic, just sort of something you literally visit as an oracle and ask for advice rather than something that can do things for you.

Yeah, yeah, exactly.

Don't you think we're too late for that in some way? Because we're, it's already ingrained, you know, these agentic systems already exist. Yeah,

yeah. But like the agentic system has been, as we've seen, that's just like fundamentally unsafe. The current generation, uh, that is like publicly visible again, like the ones that, that, that are actually, uh, top, like only the companies know what, what they are. I think they, even agentic, uh, they're kind of like net beneficial. Basically what I'm saying is that at one point, and I think the point is now, uh, we should switch to a more, more safe paradigm.

Yep. I see. So sort of almost another path down the tech tree. You keep the existing capabilities frozen and then you say,

exactly,

we chase this, you know, ultimate oracle or something

else like the Vengeance project is just like one, uh, one alternative proposal.

You, you mentioned, you know, in the past you've called for for a pause, I think, you know, one time in 2023, right?

Mm-hmm. The telepole stutter, yeah.

You think in hindsight that was a mistake, you would say? Because in theory—

In hindsight, yes. But it's basically—

Well, but it's also probabilistic, right?

Like in theory, something could have gone wrong. Yeah. I mean, I, yeah, with the information that we have now, yes, we shouldn't have done it, but we didn't have that information back then. Right. So I think on priors, it was, it was not, it was not a mistake. Of course it could have done much, much better. One problem that we had was that it was, there was like some big news being in the, in the air. I think it was like Trump impeachment or something like that.

Maybe

that we were like, okay, like if you, if you're going to like wait more with it, we're just going to be like buried.

I see.

So like we didn't have enough time to kind of like consult enough people, et cetera. So that's, that was, that was an unfortunate effect of trying to optimize for visibility.

I think you've said before that you admire the the thinking behind AI 2027 and sort of the predictions there. How similarly would your own view of the next few years be to what they've laid out?

Yeah, I mean, to be fair, like I haven't had time actually look into all the details. I've only seen like the summaries of AI 2027 and AI 2040, which is another, which is like the positive, positive vision. But from what I understand that they're just kind of like spot on, perhaps like A little bit. The reality has been more, more crazy and more aggressive than their predictions even.

Yes, it felt like until about 3 weeks ago, they were maybe lagging a tiny bit and then it's—

That's the thing, like, like the, like you can't kind of confidently say exactly what's going to happen on what date. So that's why their predictions are also probabilistic, but yes. Yeah, 2027 seems to be the best prediction about what the future, what the next years are going to look like. Unless we, unless we change something.

I think, uh, you have said before that, you know, you were one of the first investors or maybe the very first investor in DeepMind.

I did bridge loan between Series A and Series B. I missed the Series A by like 2 weeks or something.

Oh, wow. Okay. A bridge loan. Um, but it happened because you approached Demis at a conference. Yeah.

In Oxford. Yes.

Did you know what he was doing at that point well? What was that sort of first encounter like?

They were presenting in Oxford, him and Shane Legg. Might have had some context before, I'm not sure. But yeah, they were kind of like making this point that, look, AI is now on exponential, or like compute is on an exponential at least. And so like we need kind of, if you want to kind of replicate the most valuable thing on this planet, which, I mean, the tagline used to be for a while that Solve intelligence and then use it to solve everything else. We kind of need to write this computational exponential and combine it with neuroscience because human brains are like proof of concept that human kind of intelligence can be done. So that was their goal, combine computer science, neuroscience, and then write the exponential. And they kind of made the case well. And I think I might have had some background. About them before. So I just like went, went up to them. He says like, hi, I'm Jan. Do you want to talk?

And they needed a bridge loan. Yeah.

Yeah. It was like a, like a general startup thing, right? You, you, if you, if you miss a round, then you can invest in the next round by, by saying bridge loan is actually a wrong, wrong term. Convertible note is the correct term.

I got you. Yes. And then your, your venture fund that you're associated with, MetaPlanet, led the first round into Anthropic. What was that rationale like? Was that a very easy check to write? Was that, uh, one that you agonized over for perhaps any number of reasons?

A little bit. Yeah. Yeah. Like it's, uh, yeah, I get to get this question a lot. Like, why, why, why did you like invest in Anthropic like substantially at the point where, where I've been working on AI safety and warning about this thing for a while? And like, I think one big mistake that people make is that, uh, They assumed that if I hadn't invested in Anthropic, then Anthropic wouldn't be around, which is just not true. In fact, they had. I asked Daria to kind of get alternative proposals from VCs so I could kind of have confidence that no, I'm just displacing VC money. I'm not like bringing this into existence.

That must be one of the rarest approaches in a venture round where you're sort of saying, I almost in some part of my brain hope these guys can't raise any money because then I can put them out of business rather than put them into business.

Yeah. So like the first round ended up being basically non-capitalist money. So that's one thing. There is like a counter to that, which I kind of agree with, is that in some way, in a similar way that like the race dynamics, it's kind of like mirrors a little bit this race dynamics. Like if I don't do it, the others will do, right? So in some ways it would be ethical to just like stop it. The answer to that I have is that you shouldn't look at Anthropic investment as like a single thing that I do, right? It's like what I do with this influence and what I do with this, this like proceeds. So it's like a, like if I have less resources, I would, I would also have like less ability to kind of fund the safety. Because I invest my own money, I don't, I'm not responsible for other investors. So I don't have to, I don't have to maximize my net worth really. Um, I can just make money and then spend it on, on something that I think is, is, uh, is helpful.

That suggests to me that you also, in that case, believed that this was likely to be a good investment in some way beyond just the sort of the moral claim that it might actually deliver a return such that you could then use those proceeds.

No, that, because I also invested in Bitcoin and Ethereum, uh, way back, uh, like the, the fraction that I invested wasn't like, it wasn't a big bet from my, from my perspective at that point anymore. Dario had said that, that, look, we're not in the business of making money for our investors, but we might not help but do. So, so yeah, it definitely wasn't like a financial Uh, goal that I had back then.

And you had known that team for quite a long time. I think I found something, you know, back in 2011 almost, right?

Yeah, I think I met Dario in 2012.

Okay.

Uh, I definitely remember hanging around with, hanging with him in 2012.

Oh, what do you remember? Do you remember that sort of first conversations? Were you both sort of seeing the future the same way?

I don't remember much. I think like later conversations, I remember better because like I've been traditionally sort of, uh, MIRI, Machine Intelligence Research Institute, uh, mindset, um, that look, this is like fundamentally flawed approach and we need to like figure out what is like a, they initially called like a friendly AI.

Yes.

Uh, or aligned AI because there is not really like a law of physics that says that we can't have an AI that actually cares about the same things in the future that we, we care or we should be caring about.

Yes.

But, uh, Antario was kind of like vehemently, uh, against that kind of mindset and was like more on the empirical camp that, look, this is like theoretical, uh, theoretical arguments from this like armchair philosophy. Uh, we really need to like experiments. We need to basically be in the, in the trenches, so to speak, to actually learn about, about how to steer this machinery. Uh, and I think that, I think there were like kind of like interesting, uh, interesting points that he made. And I think he now has said that in 2018, I think he wrote this, uh, sort of like private essay, uh, called Big Blob of Compute, uh, which I read, which was like very impressive. Uh, basically, yeah, he just took Richard Sutton's, uh, Bitter Lesson and just like ran with it. Uh, that said that like, as long as you have this, uh, compute exponential and that, that will basically take you, take you most of the way. You don't, you don't need to be like super inventive. Uh, about the exact techniques.

When you read about the big blob of compute, it was impressive to you.

Yes.

Was it also vaguely terrifying? Like the best of both?

Oh yeah, of course. Of course. Yes. Because like, actually the one thing that I, that I thought back then was that crap, this is like engineering problem. This is not a science problem, which means that AI will be Chinese because they are way better, way better engineers, uh, than, than the people in the West. So it's like, I need to go to China, which back then I did.

And what did you do?

Like, oh, I just visited that, like, there were kind of conferences and I talked to people. Like, I didn't do anything. I mean, I don't know exactly how to talk to people there, but that was one of the talks that I remember. Yeah. So I'm kind of surprised that it wasn't actually Chinese who ran with this.

I would have the same concern if that ended up being Chinese. If you had that concern, you also would go visit, but you also wouldn't want to like outline the thesis necessarily too clearly as part of that. You're both looking for information, but—

I mean, my, my, okay. There's like a general strategy or meta strategy that I have. I really prefer symmetric strategies, uh, to asymmetric strategies, which is symmetric, uh, strategies is basically strategy where you don't really care if others mimic it. So like, uh, even going to China, like, uh, I'm not gonna try to mislead them or anything. I'm just like saying the things that are kind of, I wish like everybody was doing.

Uh-huh.

Um, so like, uh, yeah, just be aware that AI can, can be like way more capable. In fact, like one thing that I remember specifically in China, uh, like arguing with people was that they seem to, at least back then, they seem to completely confuse competence and consciousness. So they, their kind of model seemed to have been that, uh, AI gets more and more competent and at one point it will be conscious and then it will be just like us. I'm like, no, no, no, no, no. Guys, this is, this is not how, how, how anything works. Like, uh, like, uh, like you can have very competent machines that are not conscious. Yes.

Yes.

It's not, it's not obvious why, why they should be strictly coupled. So, uh, yeah, basically saying, saying things that I think are true, um, everywhere, um, which are kind of like in common interest. That has been my strategy.

What would be the asymmetric version of that game in that sense? Oh,

all these like export controls, things like that, where like, or like being there first. Yeah. If your strategy is that you need to race, you really care that it's you that's racing, you that's winning rather than the other side. So it's like a strategy that relies on the good guys, so to speak, following the strategy and the other ones not.

The more sort of empirical view, let's say, that Dario was advocating, how much of that have you come to agree with as the right path forward? It seems like you agree with pieces of it.

Yeah, definitely. So I do think that there is a really important, I wouldn't say flaw, but like a weak point in this like MIRI argument, which is like that it's kind of asymptotic at the limit argument that once you have something that is, uh, like clearly superhuman, uh, we just lose, like, uh, we, we have no control over what, what, what happens next, which I think is just true. Like, uh, this is like kind of obvious.

It's almost like a def— it's almost baked into the definition.

Exactly. Almost tautological, right? At that point where we don't have control over what it wants, like, yeah, the, the, like, as my friend Andrew Kitch says, that the future probably will not have configurations of atoms that, uh, constitute human, human bodies. The weak point, I think, is that before we get there, there's this period during which we have like human level, roughly human level AI, which we seem to be living now in. And I think Eliezer even has admitted that thing. I think that he kind of underestimated the range of human capabilities. I think Katja Grace or AI Impacts, they at one point pointed out that, look, it took AI like 10 years to traverse the capabilities of human capabilities of chess playing, uh, from like starting to be competitive with like humans.

Yes.

Until like beating, confidently beating, uh, the grandmaster. So it's same thing might happen like in different, different areas. So, so we were kind of like, there's a period probably now much less than 10 years, but still measured, probably possibly measured in years during which We kind of like cohabit this planet with AIs that are still, can be bargained with, let's put it that way. Uh, and, uh, the future might be very sensitive to what happens during that phase, which I kind of agree is kind of like point for this empiricism, but it's also super dangerous to, to find yourself in that, that phase.

You know, we mentioned earlier the idea that you would want to pause the capabilities where they are now until we sort of get things under control. When would you know it's under control enough? What would that look like?

I don't know exactly, um, but it probably has to deal with some kind of verification regimes, uh, et cetera. Like, and people point out things like aviation, et cetera, where, uh, or like nuclear safety, uh, where you just have this, uh, verifiable claims, uh, about like how safe is your nuclear reaction reactor or how safe is your airplane. And they're not like, oh, we think it should fly. It's like 70% that it will fly. It's, it's like, there's actually, they're actually grounded in some kind of like verifiable, uh, composable claims. So, uh, uh, if you could get to something like that, but I don't know, I don't know exactly, uh, how it, how it would look like, but there should be some kind of like building blocks that we can individually verify and then, uh, kind of put together. But of course, yeah, it's a, it's a global, global issue. So, uh, it's not just like one one product safety. It's like kind of like global regime, but we've done it before with like nuclear, nuclear proliferation, for example.

The UN is, is meeting to discuss these topics. Do you feel optimistic that there is a possibility for global coordination on this? Oh yeah.

Yeah. Yeah. Especially like now in the last 2 weeks. Uh, yeah, it's been kind of like my wet dream for, for a long time to, to actually have like UN, uh, and like, uh, countries separately. Uh, just going, wait a minute, like we might lose control of the AI. We need to, we need to do something about it now. Uh, I really wish that was, in some ways it's, I've been comparing this to like alien invasion. Like I've been thinking about what would like countries and UN do if there's like alien ship, literal alien ship or superior, uh, aliens.

It would be tricky. It would be the, if the aliens also make a lot of money for people and deliver a lot of utility.

Yeah, yeah. I'm pretty sure, I'm pretty sure there will be like, uh, like countries or like groups of people. I mean, definitely there will be groups of people just take us, right?

Yes.

Yeah, for sure. And there will be countries who are like, can we like kind of like get this alien technology before the others, right? Can we kind of make them sure, make sure that they land in our borders, et cetera? So I think it's like pretty similar. But on the other hand, it's like much more intuitive to us as a scenario than like this, like digital aliens that are like much more different. In fact, I do think that I was just yesterday at a meeting where somebody basically said like very reasonable things that like we need to like kind of like figure out how to deploy AI at the same time, how to make sure that there are guardrails and like, and then went into talking about loss of control and making sure that it remains under human control. I was like, yep, yep, yep. Like super nice to hear. Then he said in the end, We need to protect our digital future. I was like, no, this is not just digital. Like that. I think that's like the really big, uh, current mistake that a lot of people now make is that the worst thing that possibly could happen is something going wrong on the internet. It's like, no, it's not the internet.

And to make that clearer, we're now seeing a lot of these inflections in robotics where you're seeing the ability for these models to generalize in the physical world. And so that's the version, you know, where—

Yeah. One way of putting it is that like, the economy is really now pushing to create a digital life. Something that is like self-sustainable without, without AI, without human help. And like, that is like very dangerous place for humans to be.

Let's say we get every government that we would need to onboard, which at the moment I suppose is like realistically America and China, right? Um, to, to sort of lead the way on a global cooperation. How do you implement the needed oversight surveillance to make sure people don't cheat this in different ways?

There's a lot of momentum now behind hardware verification. I mean, one thing is like the AI 2040 kind of outlined an interesting idea that like China would build their data centers in Canada and the US would build their data centers in Mongolia. So he would have, it would be like vulnerable. Better visibility. You could do that other side.

That's interesting.

Yeah, I think that's an interesting idea. But there's another approach, which is like hardware verification. You can basically have hardware monitors, uh, in, uh, I mean, ideally even in chips, uh, but also in, in data centers, um, of the, of the other side. So you're like, instead of sending sort of like inspectors, uh, human inspectors, which also might do, uh, that was, that was like a crux of the nuclear treaties basically. you send, uh, or like have like presence of, uh, hardware, uh, controls, hardware monitors. And the interesting thing about hardware monitors is that you can kind of, uh, shape them like arbitrarily almost. Uh, you can do things like zero-knowledge proofs that like the, the only bit that, uh, you get back is that like everything is in order. Like there's like no, no rules are being violated, but of course there's like, uh, the more constrained you want them to be, the more sophisticated they have to be, which means that there's like, uh, the more work needs to be done in order to make this scheme to be trusted, uh, in the first place. But in principle, yeah, you can do like wonderful things with, with, uh, with zero-knowledge proofs and advanced cryptography that, I mean, blockchains are using it, using those now, uh, quite, uh, quite advanced versions of, of this.

Yeah. That feels like, uh, I hadn't read that in, or I don't remember that from AI 2040, but that feels like Kind of ideal use of zero-knowledge proofs. For someone that doesn't work at an AI lab or, you know, in this world necessarily, but shares your concerns, what should the average person do?

I mean, in democracies, or perhaps elsewhere as well, it's just like voice their concerns. Like, look, we don't want to be replaced by AI, even if it's profitable for some people. And yeah, basically stop the madness.

You list a series of AI priorities on your website and one of them is having a kill switch. What does that look like?

Yeah, so yeah, the priorities, to be clear, they are now like 2 and a half years old or something like that. So like I've been wanting to refresh them for a while. We'll see. I mean, a kill switch, that's a kind of like natural thing to wish. And in fact, there are like multiple like legal regulatory initiatives to mandate that. But like a kill switch that's like well done is more like a dead man switch. Uh, so it's like something that keeps the system, the data center or whatnot alive as long as some criteria are met. Uh, so, so once you have, once you detect some kind of anomalies, uh, it will just like turn off, uh, turn off the system electricity or something like that.

Presumably that is only useful until, you know, in, in this intermediary period before we have a superintelligence.

Oh yeah, of course. Yeah. Yeah. I mean, superintelligence A

superintelligence doesn't need data centers. Hang on. How would that operate?

I mean, I don't know exactly. I'm not superintelligent, but data centers are something just like substrate that humans create, probably a very suboptimal substrate for something that is superior. I would imagine some kind of self-replicating nanomass or something like that, that is just much more efficient to perform computation and expand the computation, uh, than like human-created CPUs and GPUs.

Okay. Very interesting. I always like to ask a few people who know a guest for questions I should ask them or quirks of their personality that might not be as obvious. Here's what someone said about you. I cannot stress enough how central dancing is to Jan's identity. Tell me about that.

Yeah, I, Got into dancing about 20 years ago, and, uh, and that is just amazing amount of, uh, fun that I have discovered, uh, from, from that. But yeah, I do freestyle street dance. Been to like hundreds of nightclubs, uh, just, uh, making— one way of putting it is that like, uh, if you see people doing street dance on, uh, on the street, uh, for money. Yes. I do a similar thing in nightclubs for attention. And it's like, yeah, it's, it's, it's just like a sort of like really positive sum game because like you're kind of entertaining to other people and you get kind of like really fun feedback. Sometimes I joke that I, I've learned to dance using RLHF, reinforcement learning from human feedback.

Yes, you have. That's amazing. How did you, how did that originate? Was that just like going out? You know, I, I, I, yeah. So

I, I kind of originated during my Skype days already. I was like in my early 30s and most of the people were like in their 20s. So there were a lot of like wild, wild nights, including going out to nightclubs. And then at one point I discovered, wait a minute, I actually kind of like this dance, dance thing. And then I, at one point I started getting kind of positive feedback from, from people that, oh, you, you know how to dance. I was like, oh, I should do more of that. Then at one point I got, uh, of dance coach in UK, in London, and we became very good friends. So sometimes we go together and just like perform.

In one of the stories about sort of the history of Skype, they talk about the 5-phase Skype night where it's like dinner and dancing and karaoke.

Stop, stop, stop. Yeah, yeah, stop, right, right.

There's something else there. What makes it for an amazing place to dance?

In my case, uh, it's, uh, it has to have space. Quite often, yeah, places are going too packed. Then it has to have like light, uh, especially like in London, like I think post-COVID, the places are just way too dark. I really don't like London, uh, for dancing anymore. And I like hip-hop and pop. And one reason why I like pop is that it has lyrics that that people know. And one way of describing my dance style is like, it's making NP moves, which like, if you know, like NP versus P, uh, is, uh, is the most famous computer science conjecture. And without going into details, basically NP moves, what it means, uh, is you're making moves that are hard to do, but easy to interpret.

Hmm. Okay. Wow. I should know about that.

So, and one classical NP hard problem So-called LPR problems is constraint satisfaction, which means that you're given some kind of criteria and then you need to produce a solution that satisfies that criteria. So in dancing, examples of that are harmonics. You go like, do like, try to do like a sine wave.

Yeah. Okay.

But the most fun constraint to satisfy is the lyrics. So you basically take the lyrics and you act them out.

If you think of one place you most love to dance, or, you know, if you have your sort of Last night on earth to go to, where do you have a place in your mind?

Yeah, Tokyo. Uh, yeah, like, uh, I have like 2 places in Tokyo that I really have liked. Uh, one of them is still there, but that's kind of like a weekday, but the weekend place that didn't survive COVID, which is very unfortunate.

The weekday place, what makes it so special? Has space, has light?

Yeah, yeah. And has like multiple, multiple rooms. And so it's how you can kind of pick your music. And, uh, I mean, Tokyo is like so big and concentrated, has like big parties every night. Whereas like in New York, which also used to be like really, basically my second favorite city to dance, uh, I think it now has kind of gotten like much more quieter, uh, than it used to be.

Also post-COVID maybe?

Yeah, post-COVID. And like my favorite place in New York, uh, Pianos Bar.

Yeah, I know exactly that place.

I mean, that's, uh, I mean—

That's so funny.

Only half joking, saying that's the only place in the world where I'm guaranteed not to be the best dancer when I go.

Yes, right. Well.

So it has always had, has had like amazing dancers, but like last time I went there in January, uh, it was just really quiet and nothing, nothing, nothing was happening there.

Huh. There's this Alan Watts talk where he talks about how life is, you know, best conceived of as a piece of music or a dance where, you know, the point of a dance is the dancing. You would never start a dance and say, I'm going to end up at this part of the floor. To have a goal is in some way besides the point. Uh, do you agree with that?

Yeah. Yeah. So it's, uh, I think it kind of separates freestyling, uh, versus, uh, versus choreography. Uh, and quite often, uh, like people, many dancers can just like do like really amazing choreographies, but when you just put them to like Like, let's go out and just like freestyle. It's just like stand there and drink, which is kind of, I find like a lot of missing opportunities. Whereas I'm not actually good at choreographies. Like I find it like hard to remember, remember the moves.

For Watts, he was making the point about life being that way, that you shouldn't have a goal, like to have an aim in life, to aim at something is sort of in a way pointless because the point is the living.

I don't agree or disagree with it. Like I think what it Reminds me is that there's a thing called like backwards chaining versus forward chaining. Uh, so like forward chaining is that you exactly, basically, you, you're going to live in a moment and, uh, and see like what kind of opportunities you get, but still you should have like some kind of goal in mind or something that like you just judge the opportunities or some kind of preference at least. Otherwise you're just like doing random, random things. And backwards chaining is like when you kind of have a goal and like can see like what kind of steps towards that you can take. You can think of like, like a chess, imagine like a chess game. Like you, you, you already, are you in the, like the early game where you just like try to forward change towards a better position? Or are you in the late game where you try to actually get a win because you can see all the opportunities to, to strive towards?

Your father, I believe, lived a very musical life, right? Wasn't he a TV director?

Yeah, he was a TV director and, uh, Specifically, I did like music, music videos and music, um, programs.

Was that a big part of your childhood to have music around all the time?

Kind of, yeah. Also my aunt, my father's twin sister, was a piano teacher. Um, and my grandmother, paternal grandmother, was an organist, uh, in a church. I actually grew up in church, uh, church estate. So yeah, music has been In the background, at least.

I read somewhere that one of the things he worked on was, I think, an interpretation of the book by Hermann Hesse, The Glass Bead Game. Is that right?

That sounds right. Yeah.

Yeah. Yeah. I raise it because it feels like that's, I don't know if you've ever read that book actually. It's sort of a synthesis of intelligence through a story. And in many ways there's sort of some deep connections, I think, to the current AI moment.

Interesting. I didn't even know that. Uh, yeah, I do remember like in Estonia it's called Klas Berli-Mang, which, yeah.

I didn't, I had that written and I thought, gosh, I don't even want to attempt my pronunciation there. But, um, but yeah, maybe you'd enjoy it. You have a list of, um, some of your favorite movies on your website. What's the best movie that you've seen in the last year? Because I know your favorite is Before Sunset is what I remember.

Oh, that's one of the favorites. Yeah, it's really, really cool. Yeah, I haven't updated it for a couple of couple of years. I haven't like had time, a lot of time to watch, uh, new movies. And the ones that I watched, I don't think there are any of them like stand out.

What was the last one that you remember being impressed by?

Look at, look at the site. It was like a couple of years ago, so I don't actually remember exactly. Oh, actually, I mean, Odyssey. I did like Odyssey.

Okay.

Yeah. Yeah. I'm kind of like on the edge whether it kind of, uh, is like good enough to put on the, on the website, but I probably,

I haven't seen it. Uh, yeah, I should see it. Is there a film that you find yourself coming back to because of some of the problems here? And, you know, it could have absolutely nothing to do with technology, but perhaps just sort of speaks to some of the themes that we're engaged with around control and humanity.

I mean, there are like several movies. I mean, they all kind of have to do with, uh, technology in one way or another. Uh, I mean, Limitless was interesting. I remember like, uh, 10 plus years ago. Uh, where it just had a pill, uh, to make someone more intelligent and what happens. Uh, that was a really, really fun movie. But then like the, like various kind of warning movies like, uh, Ex Machina, for example. Um, that was very impressive just demonstrating one aspect of like not caring about humans in a very tangible way, which was, which was really, really cool.

And scheming. Yeah,

yeah, yeah, yeah, yeah. And like the classic, The Forbin Project, Colossus: The Forbin Project.

I haven't, I haven't seen that.

Yeah, this is like from '70s, I think.

Ah, the Forbin project?

Forbin, yeah.

Forbin.

Which was basically 2 supercomputers, uh, suddenly Soviet supercomputer and, and US supercomputer, like suddenly figuring out that actually, why do we need these humans?

Do you ever allow yourself or find yourself feeling optimistic about what AI will be able to do?

Oh yeah. I mean, like I, I, I have like my own kind of coding projects, et cetera. And it's, it's, it's pretty awesome, uh, to just be able to, I mean, I've, I've worked for years for like with several, on several software projects, either on my own and with various, um, like software development groups. And like last year I'm making a lot of progress just because either like those projects that I have to do on my own, I just like tell the AI that's like actually kind of finished this.

Just to get it done.

And like, it just mostly works. And then the projects that I do with other groups, I often kind of go like, instead of having this like cycle of, oh, there's a bug, I don't want to give them my data, which means there has to be like this like interaction, which takes like weeks to figure out what the bug is with my data. I can just tell the AI again, without like necessarily giving it all my data, but like kind of in a surgical manner, say that, look, there's this bug, can you figure out what this bug is? And, uh, or like. I want this like tiny change and it's been pretty good.

What's the best thing that you've built or the thing that you most enjoyed building in this sort of new era of engineering?

I mean, it's still in the building phase. I'm not, I'm kind of getting there. In Skype, IM part was kind of my baby. I fought really hard for the advancement of it, but always it was kind of always sort of like on the backseat because Skype was about video and voice. At one point I funded like a startup by ex-Skypers who were trying to do IM properly, but they ended up wanting to do Slack basically.

Okay.

And they didn't have any hope against Slack. So, but over those years, I always collected like a bunch of ideas how to do like IM and email properly.

Ooh, okay. That's interesting. What, what, what's wrong with the current state of play? Because you've been thinking about this now for 20 years, right?

20+ years. Yeah. Yeah. Yeah. I mean, there's so many things wrong. Again, this is a 40-minute lecture and many people have kind of like tried to solve this, uh, have like a single interface, uh, to your communications rather than like all these different needy, uh, beeping things that have their own rules. And then usually the rules are optimized for getting your attention rather than making you productive.

Yes.

So like have this like your own interface, uh, to all of it. really big problem with unifying IM and email into the same interface is that they're running on different security paradigms. Uh, one is basically there are like 2 types of context in communication. Uh, one is what I call semantic context, uh, which is what this thing is about. What is the topic? What are the previous messages, et cetera? And, uh, Gmail innovated on this like pretty well. Like, uh, they just put like this, created this like email conversations. Before that, it was just like letter, letter, letter, letter.

Yes.

Uh, so like semantic context, Gmail like solved to a significant degree and resulting, enjoyed like a lot of success as a result eventually. But there's another context, which is security context, which is basically who can see this. And these 2 contexts are quite often in tension. Uh, so like when there's like an existing thread, uh, then, uh, and if you add a new person, the security context changes, but the, but the semantic context, uh, is, is the same. Now the question is like, what do you do? Uh, in this, uh, in this situation. And, and in IM, usually there's, uh, the security context is dictated by that so-called channel and, and messages just go into the security context. Whereas in email, every, at least in, in native email, like every message is, just comes with both. There's like a letter that has the topic and the quotes, uh, which is like the, uh, semantic context. And then the participants in, in the to and, and CC lines, et cetera. Uh, which is the security context. So like, they're fundamentally, the composing blocks are different. So how do you unify that? And I've been like thinking about it a lot and, and like currently seems to pretty well go.

Oh, wow. Interesting.

But this is all for my, for my taste.

Yeah. Yeah, no, that's very cool. I'm, I'm always an early adopter of these like unified text things because I, I have the same thing where I'm just like, just, I want it in, in one.

Yeah. Well, I mean, I, I would be happy to share this thing. Currently I'm just developing it for myself. One thing that kind of stops me and I haven't figured out how to share it is I don't want other people's data. So I don't want to be responsible for safeguarding other people's data. So I need to figure out how to do it in a way that is, uh, can be shared, but in a way that I don't actually get to see your data.

Well, as soon as you figure out how to share, I would love to be a user. That's amazing. I think that's as good a place as any to end. Jan, thank you so much. This was a real pleasure.

It was really fun. Yeah. Thank you. Thanks for having me.

Amazing. That's it. Thank you for listening to this episode of The Generalist Podcast. Please subscribe on Apple Podcasts, Spotify, or your preferred podcast app. Ratings and reviews help others discover these discussions, so if you enjoyed the conversation, I'd be grateful if you could take a moment to leave one. For all past episodes and more, visit us at thegeneralist.substack.com. See you next time as we continue to explore. The future.