Futurist Gerd Leonhard - with Futurist David Wood - AGI By 2030? Think Again!

Humans can do all kinds of things. A machine can never go close to. You know, we can think abstractly, we can be creative, we use our emotions to think, we use our body to think. Of course, that's human intelligence. It's biological, organic, non binary, multinary intelligence. And then we have machine intelligence, which is much more about patterns, recognition, information, database system structure and so on. Completely different. And that's why I think we should not bother to think much about machines and can be like humans. That's not really the issue and that's not really what we're talking about. Artificial general intelligence will not be like a superhuman, it'll be an alien, a different kind of intelligence, a completely different intelligence. Very important to remember this. The way things are going in the current discussions going Back to the 50s, this was the original symbol of intelligent machines, of computing, of robotics, of course. And the first thing I want to nail down is what our AI actually is. And it's Demis Hassabis from Deep Mind. Now Google, of course, saying that it's about computer systems that turn data and information into knowledge. Important definition, computers that turn information into knowledge. Now here's a clip from Mustafa Sulaiman, who is now the CEO of AI for Microsoft.

The economics of information are about to radically change because we're going to reduce the cost of production of knowledge to zero marginal cost. And this is just a very difficult thing for people to intuit. But in 15 or 20 years time, we will be producing new scientific, cultural knowledge at almost zero marginal cost. It will be widely open, sourced and available to everybody. And I think that is going to be a true inflection point in the history of our species. Because what are we collectively as an organism of humans, other than a knowledge, an intellectual production engine? We produce knowledge, our science makes us better. And so what we really want in the world, in my opinion, are new engines that can turbocharge discovery and invention.

That's Mustafa Suleiman. I'm not sure I would want a machine to be turbocharging my inventions or creativity, but certainly very fiery speech. And this was a great interview. You ought to watch the whole thing. It's available of course on YouTube, but powerful message. We're going to have a machine that has knowledge, a machine that invents things, a machine that organizes, a machine that reasons. And the question is, do we want that and could we control it? But bottom line for artificial general intelligence is a machine that surpasses human capabilities and in the majority of tasks, so a machine that can surpass humans in intellectual tasks, because the machine doesn't have the body, of course, but initially at least, will be mostly about cognitive task and knowledge work. And that's important. Definition is like the meteor coming down from above, closing our days of knowledge and being the superior species with knowledge. So very important. Stat here you can see how all of the things that kind of make us more intelligent are being sort of closed in on by AI. So this is like understanding of images, language recognition and code generation and grade school math and all of these things that we're seeing around us now. Machines can do most of those. But here's important to remember what the difference is. Machine intelligence is not the same than human intelligence. Machine intelligence is, is a simulation of that intelligence. So if a machine can read or speak, it simulates us reading or speaking. That's really not the same thing, but it's a different level of intelligence. But suffice to say, machines will be capable of doing those things, like writing code, like a basic reasoning, like reading, translating. And this is getting closer and closer, as you can see on this chart, about 95%. And here's Jan Lecun, who runs Meta Intelligence. So this is Facebook, of course, a long time ago. And he talks about the difficulty of comparing intelligences. Intelligence is not a linear thing that.

You can measure with a scalar, with a single number. Can you say that humans are smarter than orangutans? In some ways, yes.

But in some ways, orangutans are smarter than humans in a lot of domains.

So intelligence is a collection of skills and an ability to acquire new skills efficiently. Right.

And the collection of skills that a.

Particular intelligent entity possess or is capable of learning quickly is different from the collection of skills of another one.

And because it's a multidimensional thing, the.

Set of skills is high dimensional space. You can't measure, you can compare, you.

Cannot compare two things as to whether.

One is more intelligent than the other. It's multidimensional.

Important message here. This is of course, on Lex Friedman's podcast. We cannot compare human intelligence and machine intelligence. It's a different category. It will be superior to human processing and brain space and our intelligence. Yes, but this is an important question. How do we actually measure this? And which way exactly are we going with this? And you know, a human has a body, as you can see here in this chart. Many psychologists will tell you that basically we don't think with the brain, we think with the body. The brain is just part of the body. It's a holistic view of intelligence. And that is really way beyond machines. Of course, you can give machines a body, a fake body, and that's already happening. But will it be like our body? Is it going to be a holistic view? I think this is one of the very, very big distinctions of human intelligence, is to use a body, to have a body, to be able to do those things. One of the leaders in this stuff is Joshua Bengio. He is at University of Montreal. He is one of the touring award winners from the couple years ago on artificial intelligence, of course. And he has something really interesting to say about this. I figured I should give him an AI voice.

I also find statements like AIs cannot have true intelligence or the AIs just predict the next word unconvincing. I agree that if one defines true intelligence as the way humans are, intelligent AIs don't have true intelligence. Their way of processing information and reasoning is different from ours. In a conversation about potential catastrophic AI risks, this is a distraction. What matters for such a conversation is what can the AI achieve? How good is it at problem solving? That's how I think of AGI and asi. A level of AI capabilities at which an AI is as good as or better than a human expert at solving basically any problem, excluding problems that require physical actions. There is also the current level of AI ability with a very high level of mastery of language and visual material and more and more capabilities in a broader variety of cognitive tasks. Finally, there is no scientific reason to believe that humanity is at the pinnacle of intelligence. In fact, in many specialized cognitive tasks, computers already surpass humans. Hence, even ASI is plausible. And unless one relies on arguments based on personal beliefs rather than science, the possibility of AGI and ASI cannot be ruled out. Joshua Bengio University of Montreal Thanks, Josh.

So that's a pretty sobering message. It is possible to create an intelligence that is superior to humans in solving pretty much any problem. And that should be, technically speaking, possible. I think we need to think about this as we go forward about the discussion about who's going to be in charge of that effort of building AI like this. We already had Stephen Hawking, the scientist, argue that super intelligence is physically possible because there's no real physical law precluding the particles from being organized in new ways that perform even better than they are with humans and in human brains. So the question really is, should we build such a machine and who's going to be in charge of such a machine? And so I have over the years come up with this graphic that I use a Lot in the distinction between I think, what's okay and what's good and what's not. And the first part of its IA I call intelligent assistance. That's essentially software, smart software, intelligent software, cognitive computing machines that can think parentheses in patterns, binary thinking, and then sort of more broader AI machines can have some basic reasoning and understanding and then general intelligence. And then we can think about how we keep one apart from the other. How do we use intelligent assistance and AI to solve our global problems of climate change and, and food and water and energy and all of these things which it will be very good for. And how can we make a barrier for creating a general intelligence that will surpass all of us and leave us in the dust, make us the second most intelligent species? And maybe we need sort of a temporary fill before we get to that point, you know, to safeguard against surprises. Many researchers have said that we have four stages of AI. The first one is about classifying data and information. That's a simple system that looks at data and classifies and sees patterns. The second stage is to generate stuff. And we are now at that point with ChatGPT, generative AI, conversational AI is to generate content, data, relationships between objects, to translate, to make sense out of things. Not common sense, but basic computer sense, generating content and data. And I think we're getting that point very quickly now. The third stage that AI could reach would be then some sort of limited reasoning, understanding, making conjectures, thinking about what might be possible, thinking ahead. And that's really what OpenAI is setting out to do. The fourth stage is to be recursive, to reinvent itself, to use one AI to build the next AI essentially and to create what's called an intelligence explosion. I'll have some quotes on that later. The last two stages really worry me. Giving into a machine, reasoning and making decisions on our behalf and sort of deauthorizing us, dehumanizing us and de skilling us. I think that is a dangerous path and we need to think about how far do we want to go with this and why is it just going to be about making more money. So there are intelligent assistance project like you see Thompson and Reuters here. This is the co counsel. That's essentially fancy software to make a lawyer more capable, faster to respond for document generation and so on. That is truly intelligent assistance because it won't do much without you. Going back to Mustafa Sulaiman for a minute, he talks about the fact that people will have intelligent assistance for everything. For banking, financial affairs, real estate, dating, of course and graphic design and writing things basically that we all have our own personal digital assistant.

Everybody is going to have an intelligent assistant, a personal intelligence that knows you, that is super smart, that understands your personal history and can actually hold state. It can preserve things in its working memory. So it will be able to reason over your day, help you prioritize your time, help you invent, be much more creative. It'll be a research assistant, but it'll also be a coach and a companion companion. And so it's going to feel like having intelligence as a commodity, cheap, widely available, making us making everybody smarter and more productive.

So everybody's going to be smarter and more productive and we have intelligent assistance. We'll be completely dependent on them, that's for sure. Right. Just like we're dependent on the iPhone now. So OpenAI is emerging as a major force of this driving artificial general intelligence, I would say the primary force really. And it reminds me a lot of the sort of Tyrell Corporation from the Blade runner movies from 1982, where that becomes sort of the Monolith, the company that runs everything. And Dr. Tyrell would be Sam Altman, I suppose. And so they've come up with this really interesting map. And this was leaked the other day, I think just two days ago to Bloomberg. And so they came up with a summary about what the goal is of OpenAI. And so they said this is the order. First the chatbots and AI with conversational language skills and then what they call the reasoner, an AI that can reason, that can understand, that can make sense out of this, that has common sense. And then the third level, that already scares me quite a bit, right. There is an AI that can be an agent act on my behalf. And I think that would be fantastic, of course, if it worked. And then AI could be innovating and, and helping us to do research and looking at a trillion facts at the same time. Finally, AI could also start organization, run companies. And I'll show you a clip on this later where I'm thinking like, okay, so there's nothing else left for us to do but to chill, I suppose, or to watch the AIs at work. So here's Sam again, Sam Altman, and he talks about the possibilities of AI building companies and running the world on our behalf. Good luck.

If you let yourself like think out sci fi fiction dream world really far where you can be like, okay, AGI, start a new trillion dollar company.

Yeah.

And it can like be the CEO, it can figure out how to go raise capital, it can do all of the engineering, it can do the marketing, it can do the distribution or whatever. And you type this like one thing into a prompt and push to enter.

This is a really interesting clip, but there's a great cartoon that I saw the other day that reminded me a lot of this scene, right? It says, and so while the end of the world will be terrible and kill us all and so on, in the meantime we can start some great companies and make lots of money. Sam Altman also said the other day, the coming change of AI will be about the most important human capabilities. The ability to think, create, understand and reason. And again, I would say, wow, the ability to think, create and reason. Isn't that something that really should stick with us? Yes, AI can create basic things. Yes, it can understand basic facts. But to reason, the human ability to put all those things together, is that a good goal to have, or is it just because that goal will kind of turn us into superhumans or gods, if you wish? Is that the sort of new religion that we're pursuing? Great quote here from Stuart Russell. He says intelligence means having the power to shape the world in your interest. And if we can shape the world in our interest, why would we want to give more intelligence to a machine so they can shape the world in their interest? Even if their interest was our interest, could we be sure that it would stay to be their interest? But autonomous intelligence, not something that we should pursue, I don't think that's something that strikes me as a suicide mission, because obviously an autonomous intelligence would not have our interest to pursue the world in our liking and to shape the world in the way that we want. And OpenAI just said a few days ago they're reviving the Q project, was it recalled years ago now to be the Project Strawberry. And if we have superintelligence like this that is omnipotent, knows every fact, knows everybody, knows every quote, knows every language, knows everything about everything. And we're getting close to that. It's like the N is A times, you know, 1 million. Nick Bostrom defines super intelligence as any intellect that greatly exceeds the cognitive performance of humans in all, all domains, including creativity, planning, social skills. He says it's totally possible because there's no physical restriction for that to happen. And I tend to agree on that assessment. It is possible to build this. So would containment be possible? Could we lock this super intelligent being, a digital entity, into a glass bubble and threaten to shut off the power if it didn't function right? Is that actually feasible? Many people say it's not, and I tend to agree on that. Here's Stuart Roswell coming in again from UC Berkeley, talking about what that means and how we can build intelligent machines. And what would be the result if we actually achieved super intelligence. This is an important question because we always talk about making it and building it. But what if we actually achieved it?

What we try to do in AI is to make machines intelligent and eventually at least as intelligent as humans in every dimension. And so what would happen if we actually succeeded in that goal? So once I started thinking about that, I realized we didn't have an answer. Do we have an answer today? No. With the current generation of AI systems like ChatGPT, for any question you might care about, like, is it going to replicate itself? Is it going to break into another computer system? Is it going to advise terrorists on how to build biological weapons? Is to it going, Is it pursuing its own goals that we don't know about? The answer is we haven't the faintest idea, but we think we should be able to put it on the market anyway. We're probably spending more on that than we are on all of the rest of science put together. And yet we have no real idea what happens if we succeed.

Should we really value intelligence over humanity? That's really what it comes down to. It's not the only thing that matters. So let's talk about if AGI is possible, and then I'll give you a quick run through Dystopia. There's a great job I just found a few days ago about predictions. And we had Elon Musk predicting self driving cars in 2015. And then Geoffrey Hinton was predicting radiologists to be obsolete in 2016. Now the last scene is radiologists in regular gas cars driving back to work. And that's our reality. So here's the bottom line of this. As we know, of course, we overestimate the effect of technology in the short run and underestimate the effect in the long run. Amara's Law, Bill Gates law. And this is what's happening with AI, and we totally underestimate the effect in five or ten years. In fact, I believe that we're going to see AGI by 2030. This is not far away. It's not like a hypothetical thing happening.

Artificial intelligence is nothing but digital brains inside large computers. That's what artificial intelligence is. Every single interest in AI that you've seen is based on this idea.

And as you go up the chart, it represents exponential growth. Nvidia just came out with a chip which is half A trillion calculations per second. So this little chart represents a growth of 75 quadrillion fold increase. I was asked to make a prediction of when would we see AGI, artificial general intelligence. And so I figured that this chart would continue, which it has, and I figured we'd need about a trillion calculations per second to do AGI, so I estimated 2029. Musk says it's going to happen in two years. It's not an unreasonable position, other people saying three or four years. I'm sticking with five years, but it could happen soon. Very good summary here. That was of course Ray Kurzweil, who gave birth to the very term of singularity and who has a new book out called the Singularity Is Nearer. Interesting book. I read the last one, so I recommend you take a look at it. The philosopher David Chalmers argues that the brain is a mechanical system and therefore it can be emulated. I tend to agree on this. Yeah, it can be emulated, but an emulation, of course is not the same than the original. It's a simulation and, and nevertheless we can emulate such a thing. And an emulator will sometimes be closed, sometimes not. But yes, it can be copied, so to speak. I agree five years to AGI, that's kind of the bottom line. I know that's shocking for many of us when we think about the potential of what that means. I mean, AGI, general intelligence of machines that can do any cognitive process better than us is the end of work. But is our society prepared for this? And who's leading this? Five years to AGI, that's 2030. We are going back to Geoffrey Hinton in this talk show here at Amtech Digital, where he talks very plainly about what we're expecting with artificial general intelligence and what the outcome could be.

The worst case scenario that you think is conceivable? Oh, I think it's quite conceivable that humanity is just a passing phase in the evolutionary intelligence. So one of the ways in which these systems might escape control is by.

Writing their own computer code to modify themselves.

They will be able to manipulate people. And these will be very good at convincing people because they'll have learned from all the novels that were ever written, all the books by Machiavelli, all the political connivances.

They'll know all that stuff.

They'll know how to do it.

I think we're moving into a period when for the first time ever, we.

May have things more intelligent than us.

Geoffrey Hinton, this is what he says the worst case is this could be intelligent entities that can modify and replicate themselves because they are smart enough to organize that without our consent. We can expect AI that's very good at manipulating people. So good that we wouldn't know that it's an AI. And it could also be sort of manufacturing a synthetic reality. He says we may have things more intelligent than us for the first time in history. Nick Bostrom again says the reason is that we can have a brain, a digital brain, much bigger than human, because we don't have the restrictions of the neurons.

What we do know is that the ultimate limits to information processing in machine substrate lie far outside the limits in biological tissue. This comes down to physics. A biological neuron fires maybe at 200 hertz, 200 times a second. But even a present day transistor operates at a gigahertz. Neurons propagate Slowly in Axon 100 meters per second tops. But in computers, signals can travel at the speed of light. There are also size limitations, like a human brain has to fit inside a cranium. But a computer can be the size of a warehouse or larger.

So that's the difference. It's possible to go far beyond humans and that takes us into the darkness. So here's our friend Sam Altman again talking about what AGI really is. And that's already scary by itself, even though of course he's acting like it's reason for a party.

The term AGI is thrown around a lot. How would you define AGI and how do you think you'll know? I think there's a lot of valid definitions to this, but for me, AGI is basically the, the equivalent of a median human that you could hire as a co worker and then they could say do anything that you'd be happy with a remote coworker doing just behind a computer, which includes, you know, learning how to go be a doctor, learning how to go be a very competent coder. There's a lot of stuff that a median human is capable of getting good at. And I think one of the skills of an AGI is not any particular milestone, but a meta skill of learning to figure things out and that it can go decide to get good at whatever you need.

So we're going to fabricate medium humans all around us and they'll be serving us and so we don't have to do anything anymore. We can just have the permanent chill. A more powerful message even is Yuval Harari that you may know as a famous author and an altogether really important critic of many things around us. And this is what he says this.

Is the end of human history. The end of human dominated history. History will continue with somebody else in control. In five years, there'll be a technology that can make decisions independently and that can create new ideas independently.

So he's saying that that's kind of the final phase of being human as we know it. Here's our friend Elon Musk.

AI doesn't have to be evil to destroy humanity. If AI has a goal and humanity just happens to be in the way, it will destroy humanity as a matter.

Of course, without even thinking about it.

No hard feelings. It's just like if we're building a road and an ant hill happens to.

Be in the way.

We don't hate ants. We're just building a road. And so, goodbye, anthill.

So the message here, right, no second chance. Once it's out there in charge, it will not give us a second chance.

Well, unless it's limited, narrow superintelligence, I think you mostly don't get to observe it because you are dead. Unfortunately. What I think that the actual catastrophe scenario for AI looks like. Big advancements in a research lab may be driven by them getting a giant venture capital investment and being able to spend 10 times as much on GPUs as they did before. Maybe driven by a new algorithmic advance like transformers, maybe driven by hammering out some tweaks in last year's algorithmic advance that gets a thing to finally work efficiently. And the AI there goes over a critical threshold which, you know, like, most obviously could be like, can write the next AI. The problem is that we do not get 50 years to try and try again and observe that we were wrong and come up with a different theory and realize that the entire thing is going to be like, way more difficult than realized at the start. Because the first time you fail at aligning something much smarter than you are, you. You die. And you do not get to try again.

You don't get to try again. I want to share these kind of dark moments with you because I think they're important to realize as we look at the challenges of artificial intelligence and the challenges of general intelligence. So, Josh Abegio, so one is scientific.

Like, how do we solve the control and alignment problem of AI? Is there a way that we could build the AI that it's going to be safe if we follow some protocols that would be the method to deflect the asteroid away from the Earth. The second thing we need to solve is political. It is connected to the question of democracy. It is, well, how do we make sure that powerful AI systems, even if we knew how to control them, are not controlled by bad people, bad actors, who will will grab power, destroy democracy, or even just make mistakes and not follow the protocols properly. And we all lose two issues.

We have the control problem and making it aligned to human needs. And the second one is the collaboration problem, the governance problem. Both of those are serious issues right now. And I do think that we should think about how far you want to go while those issues are unsolved. There's also people who says that basically we shouldn't worry about this right now because the existential risk isn't our problem, it's the other risk that our problem. So it's best to not think about whether an AI is like an alien or whether it's like a human. But he says this.

Regulators should not prioritize existential risk posed by superintelligent AI. Instead they should address the problems which are in front of them, making models safer and their operations more predictable in line with human needs and norms. Regulations should focus on preventing inappropriate deployment of AI. Blaise Aguera y Arcas via the Economist not his actual voice.

So preventing inappropriate development, that's what he thinks we should focus on. Again, something that we discussed, if that would probably have to do both, I think that's really what it comes down to. And then we have the black box problem, you know, which is another one of those big challenges. Once an AI is powerful enough, if you're a judge, a lawyer, a policeman doing medical work, then how do you know how to rely on it? And how do you know which decisions it makes that you follow? And Yuval again has a great piece on this.

And increasingly it's algorithms making all these decisions for us or about us.

Is that possible?

It's already happening increasingly, you know, you apply to a bank to get a loan in many places, it's no longer a human banker who is making this decision about you, whether to give you a loan, whether to give you a mortgage. It's in the algorithm.

Yeah, you can see the point about the black box quite clearly. And this already goes back to Bill Joy, the founder of Sun Microsystems. In the year 2000, in Wired magazine, he said this very much the same thing. That's 25, 24 years ago, a long time.

The human race might easily permit itself to drift into a position of such dependence on the machines that it would have no practical choice but to accept all of the machines decisions as society and the problems that face it become more and more complex. And machines become more and more intelligent. People will let machines make more of their decisions for them, simply because machine made decisions will bring better results than man made ones. Bill Joy 2020.

That was in 2020, pretty much the same message. And we're here now, right? We may have machines that make decisions. We know that they're probably better decisions. So we just kind of retire and we abdicate and we kind of let things go and we stop thinking. And that will be the end of, I think, of our independence. And then we have things like ChatGPT that already can pretend to be completely human, as you can see here.

Hey, how's it going? Hey there. It's going great. How about you? I see you're rocking an open AI hoodie. Nice choice. What's up with that ceiling though? Are you in a cool industrial style office or something? Well, can you take a guess at what it might be doing based on what I'm showing you here? Hmm. From what I can see, it looks like you're in some kind of recording.

Well, you get the point. You know, this is Scarlett Johansson, clearly emulated voice and chatgpt conversation. How do we know what's real or not? How do we know what's okay or not? How do we know who they are, who we are, and how does it all fit together? And this will be a very, very powerful change. It will lead to what's called the coordination problem to figure out what we want and who is in charge of actually making that happen. So the politics of AI are a big deal as again, I think to Joshua Bengio says if the coordination worked, then we could sit down together and we could say, well, if we have issues, we can talk about it and we can use what's called the precautionary principle to solve them, to work it out together. But if they don't work, and we have Americans thinking about the market and we have Europeans thinking about humanity, and then we have Chinese people thinking about the state and the power of the state, you know, we're unlikely to find a conclusion in this. So the dynamics of these self interests are huge. And Joshua Bengio kind of nails it. You know, we need to figure out how to work together so that we can solve these problems together. The European AI act is a step in that right direction. There's four levels which I will not bother you with, but basically looking at what is okay, what is kind of okay, what's forbidden, and what is completely out for, for example, like social credit scoring. So let's talk about what that all has to do with money and economics. I wonder how we can create a future fit economic logic that is not self destructive as we've done in the past. And I wonder if we have to go first to solve this issue of having to pause capitalism, so to speak, rather than pause AI as some people have proposed in the past. And, and so I go back to this trailer a lot when I think about artificial intelligence, how I started thinking about it 1982.

Commerce is our goal here at Tyrell. More human than human is our motto.

More human than human is our motto. Commerce is the goal. And whoever is in charge of AGI and underneath AI kind of rules the world. That's been said for a long time, but now this is becoming a national security issue, it's becoming a political issue, it's becoming a global geopolitical issue. And if AI is used mostly for corporate gains and corporate purposes, like we can see on this chart, what would AGI do? Would it do something different? We can see here the biggest companies in the world reducing headcount, bringing up profits, and this is happening because of technology. Using technology to reduce the number of people working for you and to bring in more money. I think that's of course legitimate to some degree, but not to the utmost degree. By saying we're going to have digital entities that are 90% of our staff, I can't see that playing out very well. And so we have this concept again laid out brilliantly by Josh Abengio. The problem comes when safety and security are falling victim to monetization, when we're looking at money as the primary factor, when profit maximization is above everything, when it's not people, planet, purpose, prosperity, but just plain prosperity that will kill us with AGI. That's my key message. And we need to think about that and what that means and how we can reverse that. Great show here by Jon Stewart on artificial intelligence. Just a few weeks ago on his daily show.

The way of life that you are accustomed to is no match for the.

Promise of more profits and new markets.

Which sounds brutal if you're a human, but at least those other disruptions took place over a century or decades.

AI is going to be ready to.

Take over by Thursday.

And once that happens, what the is.

There left for the rest of us to do?

Time is not a terrible thing, AI.

Freeing us up to think about things at a higher level.

We'll be able to express ourselves in new creative ways.

It's not joblessness, it's self actualizing me time.

This is really a Great trial. You gotta watch the whole thing. It's 14 minutes. Right. But I think in many ways there's a great quote from Jameson saying it's easier to imagine the end of the world than the end of capitalism. And this is a very big topic when we think about artificial general intelligence. We're doing this because it will make trillions. In a nutshell. Here's my message. We need to marry proaction with precaution. We need to go forward and embrace technology. But we should not become technology. We should use precaution to be careful as to what exactly we're doing and why and who is in charge and what are the arguments. You know, we need to mix those two things. Yes. Intelligent assistance, that seems like an obvious thing to me. That will be beneficial to most of us. Artificial intelligence, to some degree. The graduation of that would be the gold price. If we can do that safely. But artificial intelligence, I think that's a big no unless we can figure out how to do it together and safely for the progress of humanity. But not in an arms race and not in a capitalist race where we're looking to gain trillions while everybody else loses. Not if we're going to do like we did climate change and global warming, the fossil fuel industry, which is self destructive with a spiral that ends in global warming of 5 degrees if we're not careful. So really important. We are also stuck in this sort of King Midas paradigm. We wish that everything we touch turns into gold now. We wish that everything we touches will turn into intelligence. We should be careful about this. Not everything needs to be intelligent. Not everything should be intelligent. We have to be careful what we wish for. The Trump campaign has announced yesterday they're going to draft a proposal for Manhattan Project for artificial intelligence to create a domination scheme of American AI. I think this is just utterly ludicrous and will lead to an arms race that will go exactly where the other things went of the Manhattan Project, which is the bomb from Oppenheimer to AI. Great article in the New York Times the other day. It ends here, nowhere else. That is something we need to keep in mind. If we're going to build something like this, shouldn't we take charge and collaborate to make it come out the other way? Shouldn't we think about what we're doing with our money and with our inventions? Josh Abengio again. We need to make sure that no single human, no single corporation, no single country government can abuse the power of AGI, can at the expense of the common good, do something that May be the end of all of us. We need to figure out an act, a regulation, a convent, a non proliferation treaty that takes us to the benefit of AI but not to the disbenefit of, of AGI. We need a technocratic oath for companies that says, yes, we're doing this, but we're going to prioritize a positive outcome for humanity and not just talk about it, but also actually do it. Right now we have the opposite. We have companies that used to be a nonprofit like OpenAI. Clearly no longer. Now it's all for profit. Flat out pedal to the metal and the future of artificial intelligence. That doesn't bode well. We need to think about the precautionary principle, the principle that says, okay, if it causes really big damage. We need to think about whether that can be remedied before we do it. Whether proaction and precaution can go hand in hand because it may prove disastrous otherwise. We need to collaborate on a global level. We may need to relinquish to not do it, the possibility of, of AGI. People argue, you're dreaming. We're not going to do that because it's so powerful. We must have it. Yeah, that's the Midas, the king Midas problem. Can we relinquish something again? Bill Joy says he proposed relinquishment, limiting the pursuit of something that might be very bad for us as a potential of humanity's future. Mitigating the risk of attention from AI should be a global priority alongside other societal risks. In 2023, we had this agreement of hundreds of scientists and I signed this as well, even though I'm not a scientist. But I did sign it. I think it was number 6,000 that we should pause AI. I think we should not necessarily pause AI or IA in developing the business tools, but we should pause the development of general intelligence until we figure out who's in charge and how we can safely proceed and maybe even relinquish the idea of building a machine that is smarter than all humans together. Because here's the question. Who is mission control for humanity? Is it OpenAI? Nvidia? IBM? Microsoft? Baidu, Alibaba? Are they mission control together or individually? I mean, OpenAI is certainly acting like mission control.

What.

Where is that going? Or is it here, down in the 27th floor of a large building, the United nations, who are also still there, but inhabiting a small part of this high rise? Who is in charge of this? Is it the public? Is it us? Is it the un? Is it companies? Right now it very much looks like companies are in charge of the future of humanity. And to me, it's basically like this. The wolf you feed is the wolf that wins. We need to feed a different wolf to get a better outcome, to get a scenario where we win, where AGI can become the part of our lives that will definitely bring flourishing and human benefit.