Futurists Calum Chace and Gerd Leonhard: AI, the Singularity, the Future of Humans and Machines

Hello everyone, this is Gerd Leonhardt, futurist, CEO of the Futures Agency and author of Technology versus Humanity in Zurich, Switzerland. And I'm talking with Callum Chase tonight who's also an author. I read several of his books. I'm quite a fan in fact, and we had a really interesting debate the other day in London about technology, humanity, AI and the future. So Callum, welcome and please just come of give us a quick rundown what you do.

Thanks, Gert. Well, it's a great pleasure to be here. As you say, I write books and give talks about artificial intelligence. I've been thinking about this for a long time, about the same sort of length of time that you have, because I read too much science fiction as a boy and always assumed that one day we would create a super intelligence, but didn't think it would happen for, you know, many lifetimes after mine until in 1999 I read Ray Kurtz file and he blew my mind somewhat, making me think that it could happen in my lifetime. So in 2012 I became a full time writer. Before that I spent 30 years in business and in journalism and I've written three books on AI, one tech thriller and a couple of non fiction books, Surviving AI and the Economic Singularity. And I've just got a new book coming out called the Two Singularities.

Okay.

Which is published by an academic publisher later this month.

So can you give us a quick explanation of the singularity? Lots of people are confused about it. I use the word all the time and it's sometimes hard to agree on the definition. Definition, just like intelligence. But can you give us a brief sort of what do you think is the singularity?

Sure. According to one account, there are 17 different definitions. What I use it to mean is what I think John von Neumann used it to mean. And then 1950 time in a process where the, where a variable becomes infinite. So a black hole, at the center of black hole, the gravitational field becomes infinite. The result is that the rules of physics stop working. And it was applied to human affairs, sociology, technology, by John von neumann in the 50s. And he simply used it to mean a time of enormous change. More than a. More than disruption, more than a revolution. Everything changes. And I think we've got two of those sorts of events coming up and we can get into that. So I just use the word singularity to mean the biggest possible kind of change that can happen.

Yeah, I use a definition that, you know, it's very close to this one where I say that it's basically the point where technology becomes becomes infinitely and recursively powerful. So, you know, basically going beyond what we can imagine, beyond what we can fathom. And that is the point where I think there is a potential convergence of man and machine at that point. You know, Ray talks about, I don't know, 2045 now or so. Right. And the other important part, I think, before we go into the discussion on AI and these things, is to define intelligence. I think we talked about that in London as well. Defining intelligence, to me is really important because I think a lot of people don't really understand the difference when we talk about artificial intelligence, which to me is more like artificial smartness, you know, as Kevin Kelly says, because intelligence to me is something innately very human. And what is your view on intelligence? What defines it and why is it important?

So there we come across our first point of disagreement. Gert. I don't think there's anything necessarily human about intelligence. I think even chickens and certainly dogs and dolphins have a form of intelligence. It's not as developed as human intelligence, we believe, and lots of good reasons to believe that, but, you know, they clearly are intelligent intelligence. The best definition I know of it is that it is the ability to solve problems in a range of different environments. And that's a bit vague, but it doesn't get much less vague. You know, if you. If you dive into human intelligence, you find the ability to solve a range of different types of problems. Other intelligences, you know, alien. Alien intelligences or artificial intelligences may be able to solve all sorts of different problems that we can't even imagine. So, you know, the space of possible intelligences, I think, is very, very big.

Yeah, it's interesting. You know, we just made a new movie that's coming out in two weeks on artificial intelligence, and I was looking at this topic and saying, well, that's an interesting definition, but the other one I really liked is the capacity to enhance your capacity, essentially. Right? The capacity. The capacity to change your capacity, which is a very human thing. And this is the capacity that we have for change. And it's very unique to humans, you know, And I think that if you're looking at the more traditional definitions of intelligence, you know, Gardner and others, there's like eight different types of human intelligences. Kinesthetic, the body, social, emotional, intellectual and musical. And others. Right. And as I'm looking at the world of machines around me, I can. I can safely say, I think that the computing capacity of machines will surpass us very quickly. And that's quite obvious right now. That's not really the case, but we're pretty close. But will it actually have the capacity for this kind of intelligence? It's not binary and I wonder about that. And so what's your take on this?

I think it probably will, but nobody knows for sure yet. I think there's no reason to believe that there's anything uniquely brilliant about the particular forms of matter that are arranged inside our skulls. You know, we have 86 billion neurons inside our skulls and each of those neurons is attached to another thousand or so neurons, sometimes 10,000. And each of those neurons actually, turns out, is a little computer in itself. And it's a very powerful, very smart machine. But I don't think there's any reason to think that another machine couldn't be more intelligent than us, couldn't solve much bigger problems that we can and solve all the problems that we can.

But what about the issue of the body? I mean, this is a minor issue, of course, too many technology oriented futurists. But you know, since we have a body, we are, as Siligman and others have said, you know, we think with the body, not with the brain. So we, we have neurons, but we also have biomes. Right. And this is all impacting our capacity of thinking and making it very difficult to match, in my view.

Yeah, the central nervous system runs down and has communication structures that go right down the spine and right down into the body. And obviously our nerves throughout our body communicate with our brains. That's the way we're configured. Although actually if you are paralyzed, if you have the great misfortune to be paralyzed from the neck down, and all that is functioning is your, is your brain, you know, all of your other nerves are lost to your brain. You don't disappear. You know, the personality, the personhood, the sense of self does not disappear. And I don't see why in theory you couldn't have a brain and a vat, so it would be different kind of existence.

But even if you are paralyzed, you still have a body.

Well, you don't really have the use of it. It's cut off.

You don't have the use of it, but you still have it. I mean, this is the key question I think that is important to investigate. Do you believe that organisms are algorithms? As I think many others have argued, including, I think IBM's chief keeps saying that we are in the end, mathematic, we are algorithms.

I think we could be emulated by structures using algorithms and data in very different ways than our brain. I don't think it makes much sense to say we are algorithms. I mean, algorithms are kind of loosely, using language, very loosely. They're sorts of programs. They have to operate on something, they have to have a substrate and they have to have data to work with. So. But I do think that we are computing machines. And I also believe that our, our minds as distinct from our brains, you know, the mind being the thing that the brain produces. I think that could be, could be instantiated in a different substrate. It could. In other words, you. I think in principle we don't know whether it'll ever be possible to do it technologically or so when. But in principle, I think you could replicate a human mind inside another kind of machine.

Yeah, I think we agree on that in principle. This doesn't seem unfeasible considering the exponential change of technology. Right. And quantum computing and others. I'm just saying I believe that we are not machines in that sense. And if we are, I think we're far away from being discovered as how that machine actually works. So to me, this hypothetical idea of saying that we may be able to sort of copy what we are and create artificial humans and sentient machines or whatever you want, you know, artificial spirits, if you want. You know, I think that is, I think technically speaking, that may not be unfeasible. I'm just saying it's probably not desirable. Or we're getting to the point to where we would merge with technology, would lose some of the things that we have as humans. For example, I find many things that make us human to be anti algorithmic, like mistakes, serendipity, discovery. And I think that the idea of saying this is all programmable, kind of. I find both very depressing. I'm sure people can relate to that, but also kind of highly optimistic and in many ways dehumanizing. The effort of actually finding out what makes us human.

Yeah, I think this is absolutely fascinating. This goes to the heart of where you and I, I think, disagree. I don't think the most important thing about us is the fact that we're human. If you think about what does it mean to be human, it really means that you are able to create another human in the traditional way. Once you stop being able to do that, you know, because you've changed genetically, then you're no longer human. And that's when, you know, when one species evolves into another species by. You define whether they've crossed that, that boundary by whether they can still reproduce with the, with the previous species. Now, I don't think that's the important thing about us at all. I think the important thing about you and me is our mind. It's the ability to have conversations like this. And I don't think we need to be in human bodies with human evolutionary designed brains in order to have our minds. And I think it would be a glorious future for us to be able to transcend our human limitations. Humans have to die, and that's not something I'm keen on doing. Humans can't, can't live in space very easily, and I would love to be able to jump to the moon, but.

You know, the transcending limitations. An interesting angle. You know, the, the old, you know, Marshall McLuhan philosophy, you know, from his first book think was 1975 or something, where he talked about basically whenever we extend our humanity with tools, you know, with technology, we also amputate some of the other things that we do.

And I don't think that's a bad thing. You know, I. My ancestors, going back quite a few generations, would have known how to light a fire from scratch without a box of matches, would have known how to trap and kill a deer, would have known how to skin it and make clothes out of it. If I was put on a desert island without all the tools of modern living, I don't. I don't think I'd last more than a week. I don't have those skills. I don't need those skills. Skills, they're not relevant to me living in 20th, 21st century Western Europe, and I don't want those skills.

Yeah, but that's. Well, that's. This is not a black or white issue, right? I mean, some skills like navigating a map or so we can possibly do without, even though it's a shame that we can't. But other skills like building a relationship, learning an instrument, falling in love, you know, I mean, those are all kind of like, I don't think we want to do without that just because it's possible, like giving birth outside of your body in this exogenesis, you know, because it would be more convenient and it would be obviously a huge business if you didn't want to bother with that yourself. And this is particular angle that is really not making me very happy about the technology debate is that it's always this intertwining with lots and lots of money. Like longevity is a $6 trillion business. And this is kind of like, I understand that it could make money that it would be convenient, but I find that very dehumanizing in principle, just by design, not really a nice thing. To contemplate longevity. Yes, but you know, at what price and what will it actually achieve for us? And will it be equal for everyone?

Longevity, if we succeed in achieving it, will. What it will provide for us is the ability not to die, which strikes me as being a very good thing. And as for whether it will available to everybody, you know, the experience of technology over and over again is the first time a new technology is unveiled. It's ludicrously expensive. Only very, very rich people are able to afford it and it doesn't work very well. We use rich people as the guinea pigs for our technology that doesn't work. And by the time it works, people then figure out how to make it very cheap. So, you know, you and I are old enough to remember when the first mobile phones came out, they were the size of a briefcase. They were very heavy, they were ludus, short batteries, life. They didn't work very well. And you could only phone about three people because nobody else had them. That's not true because you could phone landlines. And now literally half the world has a smartphone. People in Nigeria and Bangladesh have smartphones. That is what will happen with longevity. There will probably be a huge row when it becomes available first, because initially it will be very expensive and it will take time to disseminate. But smartphones have gone around the world in 10 years, and I think the same thing will happen to powerful longevity technologies.

Yeah, I think that's true in these cases. But I know you believe in the free market. I don't believe the free market will regulate the things that will squash down the possible revenues that are being generated from that free market. Just like Facebook isn't going to be regulated, even though all the things that we know about it. And he got away by the senators in the Congress not knowing which questions to ask. So I think the free market will not regulate that because it is ultimately something where, I mean, if you're looking at genome editing, you know, if it's going to be invented, roughly the estimations are about 50 trillion euros it would cost to actually achieve this. It is extremely unlikely that it would be equally distributed and causing more inequality like we have now. You know, the current world of technology has not increased equality in general. It has decreased poverty sometimes. But, you know, we don't live in a more equal world because we have technology.

No, we live in a more equal world because we have. Sorry. We do live in a more equal world because we have technology and also because we have capitalism. It is completely undeniable that globally inequality has improved hugely in the last 25 years. And it isn't just because China has evolved so that instead of 90% of their people being in poverty now, something like 1% are. You know, that that's an enormous change. Well done, China. But it's. The rest of the developing world is going the same way as well, and. But even in the west, you know, people think that we are in a more unequal state than we were. It's actually not true. The US and the UK are probably the more unequal of the Western societies. But even there, inequality got worse in the 80s and it stayed fairly steady since then. Now, there's two separate things going on here, though. I'm not a. I'm not. I don't advocate complete free markets with no regulation. Absolutely not. You know, I've studied enough history, and in particular the early stages of the Industrial Revolution, to know that that is a path to disaster. What you need is a free market, but with regulation and with welfare safety nets now, there's no better way that we know of to reduce the cost of goods and services than free market. And it is the free market which will get the cost of longevity down to a point where it becomes available to everybody. And the reason for that is very simple. You make a lot more money by selling a gadget to 5 billion people at a price that they can all afford than you do by selling diamond encrusted one to ten people who can each afford to pay you £100 million for it.

Well, that's all true, but of course, you know, these aren't gadgets. These are reprogramming of the human body and the.

Yeah, but the principle still applies, I think.

Well, I think they. To a very large degree, people can choose to use gadgets or cars or whatever, but this would be kind of like you wouldn't make a choice not to use it. Right?

I mean, I would, but I think you might, you might fear it would make you less human.

Right, But I mean, are you obviously a believer in capitalism and in technology? And I am also a believer in a free market and in technology, but only to a certain point. I don't quite see the logic. I mean, first of all, inequality in the sense of poverty has decreased. We have less poverty. But that's not about inequality. The really rich and the sort of, the people that used to be in the mid level of society, they're actually descending down lower in most countries.

I mean, it really, that really isn't true. Gerd, I really encourage you to go and look at a website called Our World in Data.

Yeah, I know the site. Yeah, Max's site.

Yeah, exactly. And you know that, you know, Max is carrying on the work of Hans Rosling and Steven Pink has written a very important book on this called Enlightenment Now. And he's completely wrong about artificial intelligence, but he's right about everything else. And the world is not getting less equal. This is. This is a myth. But I want to say one other thing because I don't want to give the wrong impression. I think that free markets are very good at getting prices down, but there are things that they do which can be damaging. You know, there are externalities, and that's why you need regulation. You need regulation firstly to make sure there genuinely is a free market, and you need regulation to make sure that some of the unwanted side effects are not dangerous. And, you know, arguably, and I'm not sure this is true, but arguably the failure of Facebook at the moment, the fact that it's allowing people to spread fake news so freely isn't because of the sort of the price problem with free markets, it's with other issues which may need to be regulated. So, like I say, you need free markets with regulation, with welfare, safety nets.

I think that the issue of externality is the crucial issue, right? When we have technology that can change our entire lives, like, you know, in the early days, industrialization and the gas engine and so on, and now technology, exponential technology, AI and so on, then we have increasingly a huge number of externalities that are being generated by a business model that's kind of creating a new world without really bringing the old world along, right? So the externality of social media is the usual media, you know, public media, private media, you know, that's not social. And they have taken the damage from this. So what we have now is an environment where social media is generating all kinds of manipulations and bubbles and filter bubbles and of course, echo chambers and so on, and other media is suffering as a result. So we have. These externalities, I think are a huge deal. For example, when it's about AI, what is the externality of AI? Well, if I use smart machines all around myself all the time, I forget myself. I forget how to do things. I become utterly dependent, just like people are utterly dependent on Facebook. That's why they don't quit, which, by the way, I just did quit three weeks ago, regardless of all of those things. But, you know, that really worries me that I'm going to be dependent in an externality kind of way. I can't Make a move. I can't think, I can't work. I can't. Because technology is a constant companion, you know, for. And then I'm really at that point to where I feel like I'm becoming a machine.

So I don't think you need to worry about that until and unless we get to artificial general intelligence. And then quickly that would become super intelligence. Because until that point, your mind is going to be able to do really, really important things that no AI can do for you. It's going to be able to make strategic decisions about what you. What you want to achieve and how to achieve it. It's going to inform your, you know, it's going to give you your emotions, and the AI won't give you those. It won't help you decide what to do.

But it may squash it. You know, it may. I mean, in a trivial sense, we have that now. When I use Google Maps, I kind of forget where I am. Right. Because I just go from A to B and I listen to Google and that's it. Right. And that's not much damage. You know, it's mostly funny. But when I use machines then to provide other things for me, for example, dating advice or medical advice or whatever, I may eventually just kind of outsource and deskill myself, you know, pretty much across the board.

Well, I think you will, but I don't think those are skills that you really need to have at that point, like just. I think it's just like knowing how to make fire with a couple of pieces of damp wood. That's not a skill that you need in the 21st century and in the 22nd century, if we, if we get there as unaugmented humans, which is unlikely, but if we did, you wouldn't need the skill of, you know, knowing how to figure out where you are or where you want to go next, because your AI will. Will be better at that than you could be. So.

Well, it seems all very risky to me. My paradigm is to say, let the machines do what they're good at, which is a routine for humans. You know, a mindless monkey work routine, you know, financial transactions, filing, bookkeeping that nobody wants to do, really. And that is not fundamental. But, you know, transcending humanity would also mean things like, you know, I'm going to stop worry about the process of achieving something like learning an instrument, because I could instantly download a program to play that instrument. Yeah. So I haven't gone through the process. And it's a process that counts for the results. Like dating is Already a great example. In many major cities around the world, a woman can't get a date because all the guys are doing Tinder.

Why does that mean a woman can't get a date?

Well, that means a woman that wants to have a date has to use whatever the guys are using to get a date. Right. And the guys have gotten so lazy. This is primarily between our 25 to 40 sort of range. But there's been lots of studies on this that guys are essentially not willing to go through the traditional process of dating because, you know, they're using an app. Right.

So then a smart woman would surely use the app. I mean, you've got to fish where the fish are, right?

Absolutely. But you know, the result you get from this is so markedly different. I'm not saying it's wrong, it's just different because the primary purpose of Tinder is not to generate relationships, it's to get to the point quickly. Right. I'm not saying that's a bad thing. I'm just saying that obviously changes people behavior and it's kind of like, like, like the Viagra syndrome. You know, once you, once you figure out that you may need this to be more fortified, maybe that becomes a new normal.

Yeah. I mean, speaking as a bloke, I don't think it's true that blokes are not interested in long term relationships.

No, I'm not saying that. I'm saying that this is a shortcut, right. To get to the end result, which is to have sex. Right. And for that shortcut, it's working. But maybe the shortcut becomes the new purpose. And I think that's a bad thing because that, that relates to a sort of a subversion of the original thought, which it doesn't have to be a relationship, but dating is different than swiping.

That assumes that all that men want is sex and that isn't all that men want.

Well, I think that has been the result of these kind of platforms. I'm not saying that that is the case. I'm just saying it seems like too many shortcuts and too many easy ways of doing things will leave us unskilled and unwilling and lazy.

I bet when the first, when people first invented balls and dances where young men and young women could meet, I bet the previous generation was saying, well, you know, this is, this is not good. This is undermining our humanity. You know, it's too easy for you to meet.

That's right. I hear the argument all the time. But you know, just because it wasn't true back then. It doesn't mean that something that's amplified by 10,000 in terms of magnitude is actually not true either. Right. Because, I mean, you know, so that. That's apples and oranges, right? I mean, for example, to take a cholesterol pill is bad enough, and I don't like them, so I stopped taking them. But, you know, to then fortify my body with technology so that I can keep on existing and actually, you know, get new organs on demand and have nanobots in my bloodstream and that sort of thing, that to me, is an entirely different level than taking a pill.

Even though would you not want to do that? If somebody said, you've got cancer, we can put these nanobots in you and they will cure the cancer, would you refuse to take them?

Well, I think, again, that's a different case of cancer versus saying a higher blood pressure or so. Right. But. But to engineer myself in a larger way, I would be concerned about this. You know, if I want. If I had cancer and I wanted to live and that was available, of course, you know, the choice would be that you would want this. But the question is, would I then also use it to change things are not diseases?

Okay, well, imagine say you have a bad habit, like, I don't know, you chew your fingernails. That's not a good example because it's not bad enough. But you smoke. You have a bad habit in that you smoke. And somebody said, we can insert a probe into your brain and remove that habit for you and it won't change anything else about your mind or about your brain, would you accept that probe then?

Then I would worry about exactly the same thing now that I would worry about the cholesterol pills. Right? Because allegedly, it's not changing anything else, but it does change a lot of other things. So it's actually very bad for you.

So if they tested on lots of people, lots of people who are less skittish than you, and no other side effects were observed in a million control tests. Would you take. Would you take that probe if you, you know, if you were smoking?

I think in this particular case, I wouldn't because I think I could find another way to quit smoking. But I realize where you're getting at. You know, this, this is obviously mean. It's obviously not a black or white scenario. But I think making the decision about what is ethically wrong for yourself and for your feeling about yourself and then for society. Right. Those are very large issues.

They are. And we're going to have really interesting and very fierce debates about it because some people are going to want to change everything about their bodies if they can afford it and if the technology is available and I'll be one of them.

And of course, you know, people were.

Saying that's wrong and you shouldn't do it. And then you got a debate about, you know, you know, should I be able to do whatever I want to do with my body if it doesn't hurt anybody else, then then of course.

The issue would be under the arms race, you know, the kind of like would an UN augmented human have a chance in this world? And there have been many science fiction books about that one. Right. But I think ultimately we have to agree on what makes us human, even if we can't define ultimately what that means in 200 years. But on a gradual level we'll have to define. For example, one thing I think we can all agree on is that automatic weapons that kill without human supervision, like drones and others, are technically feasible but probably not desirable and should be banned.

Well, that's a big debate as well, but might take too long to get into that.

Yeah, but I mean these are issues that are practical today. And what I'm wondering in my book, you know, and also my speeches is whether we can find a bottom line ethic, you know, that will keep us from basically having a free for all. And the worst thing I'm worried about is the, the arms race of AI and genetic engineering and of course geoengineering. Right. So I agree with you on that, that it may not be a bad thing for humans and a society to investigate this. But then what happens with a bad society, a bad country, a bad actor, a bad human to use that same powerful technology. And so there's got to be a process of just like the nuclear non proliferation treaties which have worked, you know, more or less until now, you know, how we're going to find a governance of these issues. And that will not be an easy task.

No, that's the sort of thing we're going to be discussing very heatedly for decades to come.

So let's, let's do one more question, then we'll wrap up the conversation. Hopefully we'll get to do this on stage one of these days. I really like talking to you. I think we, we have always a good way to connect without hitting the sort of, you know, the walls of paradigms here.

Yeah, it'd be fun.

So, artificial intelligence, threat or not threat? You know, what's your take on this?

Both when Nick Bostrom is book superintelligence came out shortly after that. We had what I call the three wise men moment when Stephen Hawking, then Elon Musk and then Bill Gates all said the same thing. They said that strong AI, artificial general intelligence, and then superintelligence is almost certainly coming. And when it arrives, it will either be the best thing or the worst thing ever to happen to humanity. And I think that is exactly correct. Before then, it's probably going to give us technological unemployment. I think that will happen in about a generation. It isn't going to happen in the next five years, or probably not 10, but in a generation I think it will. Artificial intelligence is incredibly powerful and it can give us enormous benefits. It already is giving us quite significant benefits. But those benefits will be utterly transformational. They will be, they will call singularities. And they can have very positive effects or very negative effects, and it's up to us to make sure we get the positive ones.

Okay, yeah, I think I agree on that. Most of that for me, it's, you know, what I like to say basically, is that I think intelligent assistance, you know, ia, that's really what we have today, right? It's not really thinking machines. It's. It's assisting machines. Right? So IA, in my view, is 98% positive, apart from the unemployment, which, which are natural result of automation and stuff that we have to deal with. But to let those machines, to let these machines go into a situation to where they can increase their own capacity or become beyond this part of actually being an assistant. In other words, generally intelligent. That, to me is a threshold that I think we have to be very careful about, and that needs to be discussed. And I think, as you agree.

This.

Is really about governance and regulation and also finding a way to contain the worst possible scenarios. And this is really what we have to work on. So I'm, you know, part of what I'm talking about always is my, you know, digital ethics manifesto or some sort of, you know, I call this sometimes jokingly, the EPA for humanity, you know, the protection agency for humanity, you know, and so that is an open debate that we're having. And I agree with you that the benefits of AI or IA rather, will be fundamentally reshaping our society, but we will also have to distribute them. And that really hasn't happened, even though the inequality issue is kind of open, but not just looking at the Gini index and those kind of things. But I think to a large degree, the benefits of technology today haven't gone to everyone, and the externalities, the damages have gone to everyone. Like media.

Yeah, well we'll have to agree to disagree there.

Yeah, yeah, we have to leave that up for another debate. But in general, you know, I am an optimist as well. I think that we're not at that point to where I see that threat as imminent. And most AI is still as dumb as a toaster and just does the job, but does a very good job there. Like driving a car, right? Yeah, but it is not intelligent like we are. And no matter how we define this, I think it will be some time. But we have to debate on how far we want to take this and what the overview of man machine are and do we, do we have carve outs and do we only allow things in case of disease or do we make this a public property like a genome treatment and things like that. So I hope to debate that with you. Can you tell people that are listening on to this podcast or this conversation where they can find more about you?

Sure. So my books are at Amazon. Surviving AI and the Economic Singularity and shortly the Two Singularities. And I tweet like crazy cccallum and also have a blog which is www.pandora's brain.com.

Wow, fancy. Okay, so while people can find me very easily. So I have a futurist, gert erg like futuristgert.com and then I have tech versus human tech v as human for my book and 500 hours of films and videos on YouTube. You find it there on Gertube. That's the shortcut. G E R D tube. Com and G Leonhardt on Twitter. Of course. I'm also heavily into Twitter. So one of these days we'll do this live on stage. And thanks very much for tuning in everyone. And thanks Colm for your time. My pleasure and hope to talk to you down the road.

Bye bye.