#350 How to Make Hard Choices in AI with Atay Kozlovski, Researcher at the University of Zurich

Generative AI is transforming industries at an unprecedented pace. But as AI changes how you or your team work, one thing is clear: your skills also need to evolve. At DataCamp, we offer everything you or your team need to adapt and thrive with AI. Whether it's business users looking to get the most out of ChatGPT and Copilot, or developers and data scientists looking to fine-tune models, you can learn the entire AI skill spectrum on DataCamp. Power your AI transformation today. Start learning at datacamp.com.

We're creating an autonomous or highly automated system, and yet we want to control them. Those two concepts seem to be contradictory, right? It's sort of a tug of war, and we're in a transition phase where it's the most dangerous and volatile because we don't know what to expect.

Building and using AI are rife with ethical issues. Today we're gonna think about what can go wrong with AI and then ask two big questions to try and tease out how to do the right thing. Firstly, as AI systems gain more power and autonomy, how can humans exert control over them? Secondly, is it possible to ethically create deepfakes?

But responsibility is something that can only be attributed to a moral agent. A system cannot be deemed morally responsible for, for anything. Hype is something that should be left for marketing, right? And our job, either as consumers or as project managers, is to fight the hype.

Joining us to discuss these ethical challenges is Atay Kozlowski. He's a philosophy researcher with a focus on how to make hard choices and ethics in AI. Ete is a research fellow at Delft University of Technology.

The system was providing hiring recommendations on the basis of the retinal movement of the candidates. Depending on how quickly your eyes move or how many times you blink, that correlated into the system to whether you will be a good candidate or not for that position. The mantra that we're hearing coming out of Silicon Valley, I believe them when, you know, democracy's at stake, that's a bad way to go.

Let's pick our way through some thorny ethical problems with AI. Hi, Atai, welcome to the show.

Hi, thanks for having me. Great to be here.

Great to have you here. Uh, to begin, what's the biggest ethical AI disaster you've seen so far?

Yeah, you know, I'm, uh, an academic philosopher, so words for me are very important. So the biggest AI disaster, uh, I don't know if I could exactly pinpoint that, but, uh, Over the past 3 years, I've been working with a lot of different kinds of systems from military technology to civilian uses to individual private uses. And in every single sector, we're seeing risks and dangers. Just look at the news coming out every day, right? We're seeing cases in which chatbots are promoting or motivating people to commit suicide. We're seeing misuses by governments of these technologies. So it's. It's a bit overwhelming to keep up. There's a lot of good, a lot of bad. Hopefully we'll be able to, you know, break some of these things down as we talk about it.

Absolutely. And there's definitely a lot of good happening with AI, but I agree there are also more problems and there are new kinds of problems as well. What do you think the most common failure modes are with AI? So why do we keep having these AI ethical disasters?

If we take a step back, we have a new technology. That has arrived and it's, we like to call it disruptive technology, right? It's reshaping all these different industries and sectors that it's touching on. And we're in a transition phase where it's the most dangerous and volatile because we don't know what to expect. We don't know how to use these systems properly. The development pace is incredible at times, again, for good and for bad, right? And all this uncertainty is causing a lot of problems and it's causing a lot of misuse or overuse over lines. As we look at specific systems, we notice some kind of typical types of mistakes that users tend to do, or users or developers tend to do. And we can sort of break those down if we like and go through them one at a time. But some of these will be very familiar to the listeners, some perhaps not. But these include overuse of these systems. So what's come to be known as automation bias. This is this idea that once we have a system, we tend to sort of go with whatever it tells us to do. To do or recommends us, even to the point where we sort of discount our own intuition or our own thoughts about this. That's one type of mistake we're seeing quite often. Algorithmic bias is another, right, where these really, really complex systems rely on huge amounts of data and we're not exactly certain why it's making certain mistakes, or it leans towards certain ways that become almost invisible to us as users. We'll have to break these down into actual case studies in order to sort of see where these come about, but those are sort of typical types of cases that we see come up quite often.

Yeah. So that first one on automation bias, I had a case of this recently. So I was taking a flight and going through passport control and there's a machine that reads my passport and it takes a photo of me, compares, am I the same as my passport photo? And the machine said no. And the person, the guard, he looked at me and he told me, this isn't you. I'm like, yes it is. So I'm obviously me. And we stood there for about 2 minutes. He had no process. So he's, I've no way of dealing with this. And so there's a queue building up behind me and it got very awkward. And eventually he said, okay, go on. And it was a really stupid process. And the problem was that he didn't know what to do. When the AI was wrong. So I agree automation bias is a problem, but can you tell me how you go about resolving this sort of thing? So what do you do when the AI is wrong, but you know, there's a mistake?

Well, yeah, I think those are two sort of different aspects to it. One is when you're, you're aware, right? Like in your case, you know, oh, this is obviously something wrong. Um, then you have to sort of make sure that you have especially if you're a company and you have a lot of different employees in this part of your workflow, right? So you have to make sure that you have built-in protocols in order to deal with those situations, that people don't come across this for the first time and they're completely stumped. Go prove that you are you now, like, you know, the typical comedy that comes up or tragedy, right, that arises in literature. There was a book I read last year called I Am Stiller, and it's— I think there's a movie coming out about it. And it's basically the opposite case, right? It's about a person who decides to shed his identity and everyone in society refuses to accept what he's doing, right? No, you are Stiller. We know that, right? And he sort of rejects this completely, but he stumbles constantly about proving that he is not who he is or that he is who he is. So these types of identity issues come up in weird ways. When it comes to technology. So we said that's one side, right? When we know there's an error and that's a mistake. The more dangerous type, of course, is when we don't know there's an error and we sort of get into that tendency of just following the instructions blindly. This can be really benign sometimes, right? Imagine you're using ChatGPT to tell you to go for lunch, right? Where should I eat lunch today? And it recommends you a restaurant that doesn't exist. We've all come across these hallucinations, right? And yet you go to that place, you arrive there and you're like, oh, okay, well, you know, the stakes are low. So you go to the restaurant next door. Fair enough. If you're at the airport and then, you know, you're being suddenly said that no, this is not who you are and now you need to go to immigration and you're being held in jail cell for 2 days until you can, I don't know, bring your wife or your in-laws to attest that you are who you are. So it could be far worse. Maybe I could give an example of a system that I worked closely with.

Yeah, absolutely. I'd love to hear an example.

So I, a few, about 2 years ago now, oh, time flies, worked on analyzing a system that was used by the Israeli Defense Forces in the war in Gaza. And this was a recommendation system that they had created in order to enhance their intelligence capabilities. The idea was, you know, we have vast amounts of data that we are collecting and we have a huge population. We need to pinpoint exact enemy operatives within that vast population. 2.5 million people, we want to find 30,000. Quite difficult. So they created this system known as Lavender. And what Lavender did was basically provide a risk assessment score to every single civilian or person living in the Gaza Strip. 2.3 million people, you get an assessment on each one of them and a score. How likely are they operatives of a terrorist organization? And then these lists would be transmitted to the Air Force or to the ground force in order to, to act on them and to attack these targets. Essentially what the, what the system did was create a kill list for the military in order to, to know where to attack. At its height, we have records showing that the system created 37,000 recommendations of the kill list. And this went out and sort of helped the military accelerate its attack in Gaza. And that's what sort of allowed it to solve that bottleneck that every intelligence agency knows. Validating targets takes time. It's very difficult. Um, and sort of they went from a process where they can validate 50 to 100 targets with it per week to a situation where you can do 1,000 targets per day. So it's, it's a order of magnitude higher, right? Where with a click of a button, you can get 30,000 recommendations. Now what happened in this process as they implemented it, we were talking about Automation Bias, right? So this was a new system that went into force and reports are coming out that the traditional way in which they would evaluate these targets was sort of abandoned. An analyst would get a target and he or she would have about 20 seconds to approve that target because of the scale and magnitude of the amount of recommendations coming up. Within that 20 seconds, that's the due diligence, they would have to approve the target. So basically what they could validate is this a man or a woman, right? Because they would have audio recordings, they might have visual recording coming in. If it was a man, probably the system's okay, let's approve it, stamp it out, send it out. And essentially what we know from internal testings is that the system would produce about a 10% false positive rate, right? So let's say you had a list of about 40,000 people. You know for a fact that 4,000 of them are false positives. And yet these go on the kill list. And it's, it's, it's mind-boggling to think about that. So you have this idea, you want to improve your intelligence system. You want to create a system that helps defend your country and security, and you're using it in a way that creates egregious human rights violations. It's just senseless almost. So really that was an extreme, extreme example of how when you implement this type of system and you don't do it correctly, it has huge implications.

My jaw just dropped. That's absolutely awful. I've got some background in machine learning, so I know you're always gonna get false positives, and in this case, consequences of getting A false positive, absolutely horrendous. So targeting an innocent person to be killed because of a problem in the model and then not having a process to deal with this mistake, it's just a horrendous systemic failure. It makes me angry. How do you go about stopping this sort of thing happening in the future?

A lot of my research has focused on trying to understand this notion of what type of control do we need to maintain over an algorithmic system in order to ensure that it doesn't create, you know, in the extreme cases, these horrible outcomes and less extreme cases, maybe irritating outcomes or bad outcomes. So our focus has been on this notion of control. And we've worked with a theory called meaningful human control. And this sounds very plausible, right? We want to be in control of our system. But there's something of an oxymoron here, right? So we're creating an autonomous or highly automated system. And yet we want to control them. Those two concepts seem to be contradictory, right? It's sort of a tug of war. The more autonomy you give, the less control you have. So in order to sort of try and make sense of this paradox, what we started doing is started thinking, okay, you can't have operational control over the system, right? The system is running at a speed that we can't handle using data that we can't analyze or understand. We can't physically control the operations. Instead, what we want to do is we want to have an indirect type of control by designing the system to meet certain types of standards. This is obvious. It sounds plausible and intuitive. And what we were working within that framework is trying to define or to clarify, well, what does that actually mean in practice? What kind of conditions would you need in order to say you have a good type of indirect control over the system? So I can go through 3 steps that we took in that, in developing this theory. The first is to shift our focus from only algorithmic analysis to what's known as a social technical analysis, right? You have to expand your scope of analysis beyond just the algorithm or the software that you're using. You have to consider constantly the bigger picture. There are humans involved, there's a pipeline, there are multiple actors where the algorithm is only one part of that. So whenever I say a system, I'm talking about that holistic social technical perspective, right? Expanding not just out. That's why. You wanna comment on that or should I go on to 2 and 3?

Talk me through what are these socio technical aspects?

Yeah, so consider for instance the norms of practice in your, in wherever you're implementing this tool. So if you're in a military facility, you might have a very hierarchical system where people follow orders very, very clearly and you can sort of rely on that in order to ensure that the system's functioning the way you want it to. On the other hand, if you're working at a call center like how I look with these headphones, you might say discipline is not such a strong, uh, uh, so we have to account for that when we're creating our algorithm, right? So what I'm saying is we have external sort of variables that indicate what risks we need to account for. So some of them can be normative, the behavior of the people involved. Others could be how many different actors you need to communicate between, right? And a third might be sort of the history of that domain of practice. Maybe later I'll tell you about a case far less egregious than the military one, but I worked on a project at a hospital where we were looking at developing, using LLM tools in order to reduce workload for doctors. And one of the most curious findings that I had during that process was that, you know, there's a sort of culture that exists in a hospital where the doctors have a certain level of expectation to be responsible for their patients. And it's something that, that it, you can't expect this, for instance, if you go to a tech company, it's a different type of culture. And if you account for that culture when you're designing the tool, then you can take something like shortcuts and sometimes the other way around, right? You know what you need to implement. So for instance, in that case, I'll go into more details later, but we knew, for instance, that the doctors know that the buck starts with them. Them, right? I'm in charge for my patient, regardless of what happens around, I'm gonna hold, be held responsible for whatever happens. And that sort of allowed us more flexibility when we were trying to think which functions can we or can we not delegate. So that's what I mean by socio-technical, accounting for the surrounding, thinking beyond just the code and the algorithm itself.

I like the idea. So you gotta think about the people who are involved, you gotta think about the organizational structures, 'cause otherwise having the technology in isolation is not going to make sense.

Right. And that's a risk of also when we do like off-the-shelf products, right? So a product can be designed with one process in mind, and then you implement it in a different context, and it suddenly doesn't work or doesn't work as well as you would want, right? You always have to make these adjustments. And also when you design these tools from zero, you bring that in and you adjust from the very beginning, from the get-go.

So you were talking about doctors and you said, the doctors know that they are accountable. So is that something you always want? Like, do you always want a human who's gonna be accountable for whatever process you're running? Or would you ever want a case where the AI is in charge and makes the final decision?

This is good. This is interesting. Maybe this also leads to the second point that we talk about in the theory, which is called tracing. So Tracing is the second condition which we think about, which deals exactly with that question. At the end of the day, who is responsible for whatever outcomes come from the system? Um, so I'm of the opinion that a system might be able to make the end decision in certain contexts. That's okay. But responsibility is something that can only be attributed to a moral agent. A system cannot be deemed morally responsible for anything. It's not a moral subject of our— it's not appropriate to say that the system is responsible in a meaningful sense. So responsibility is something that we only attribute to human beings in that sense. Now, we could break that down a bit. So responsibility, first, you can distinguish between legal and moral, right? So legal is to a certain degree arbitrary, right? Whatever the law says and whatever jurisdiction you're in, that's the definition, right? So you're liable in one jurisdiction and another suddenly not. So I'm setting aside the legal issue of responsibility, whereas moral responsibility allows us to do more abstract analysis. And when we come as philosophers to evaluate this, we usually focus on 3 types of accounts of responsibility. The first is accountability, which is who is to blame for something happening. The second is answerability, which is the question of who can provide me an explanation for what happened. And the third is attributability. To who do I attribute the decision or action that was made? And what does that reflect on that person? So let me give you an example. Let's take, you know, let's say you're using, you have a hiring process at your company, right? And you have Herbert, the HR executive, and he's always done all the hiring and vetting and everything. And let's say Herbert is a bit of a racist and he decides to exclude a person on the basis of their nationality. And that person later complains and a board of inquiry says, well, Herbert, what were you doing? You're obviously responsible, right? You're accountable, you're answerable, and you're attributable, right? We can attribute all three of these to you. You are to blame. You can explain to us why you were discriminatory, and this reflects your character of being a person who is a bit racist. Okay. Now, when we incorporate an AI system, suddenly we have problems which are often referred to in the literature as responsibility gaps. It means that one of these three accounts suddenly cannot be clearly identified. Now imagine that Herbert's a really good guy, but the system that he's working with has a bit of an algorithmic bias, which makes it racist. Okay, system can't be racist, right? Obviously we're talking hyperbolically, but the system tends to be discriminatory against minorities. Herbert receives the recommendation. And he says, okay, this candidate, exclude this candidate. Shortlist. All right, move on. Now, now again, we have the same problem. Someone was excluded. They complain. What can the board actually tell Herbert at this point? Is Herbert accountable, answerable, attributable? Well, it depends very much how you designed the system, how you implemented it. And when we come and look at different types of systems, we say you need to think of these three aspects. In advance when you're designing it. And you have to create the system, not just the algorithm, the entire system such that you'll be able to answer these 3 questions down the line when something goes wrong. That's the goal. We wanna be able to identify a human actor who can be accountable, answerable, attributable. And if not, then we need to be aware that this is something that exists. For instance, if we're working with a black box system, right? No one's gonna be able to explain to us why the system acted. Okay, if we know that in advance, perhaps we can work that in. But the point of the analysis is to sort of take these into account in advance and design for them, essentially.

Okay, seems very important to have all three versions of responsibility built in then. So is this something you need to have designed right from the start, or is it something you can retrofit into existing systems?

Yeah, I mean, anything's possible. And remember, I'm not the technical expert, right? So you're not going to get any deep technical answers from me. I'm the philosopher. I give the, the normative analysis. Perhaps it's possible to make adjustments down the line. I think personally that the best way to achieve this is to come with the, the right mindset from the beginning. And again, this is from an intuition of mine, right? If you have a system that's already out there and working, changing it drastically is going to be way harder than if you built it in a certain way from the beginning. That seems Pretty obvious, right? So sometimes we can fix a broken system and sometimes not, obviously. But that also leads us to the third point that we— that I mentioned within the Meaningful Human Control framework, which is called tracking. Now, tracking is the idea that if we're going to delegate a task, any task whatsoever, to a system, we need to be sure that the system can reason properly, just like we would essentially. In that case, of course, the system might take into account way more information than we could. It might be able to come up with much faster answers and better explanations, perhaps, right? Nevertheless, from a moral standpoint, we want the system to be able to arrive at the right conclusion for the right reasons also. So this comes across sometimes very funny. I came across this paper that was cataloging the weird biases and correlations that recommendation systems were making for hiring decisions. And one of them that I was intrigued with was that a system was providing hiring recommendations on the basis of the retinal movement of a candidate. So depending on how quickly your eyes move or how many times you blink in a certain number of time, that correlated into the system to whether you will be a good candidate or not for that position. This seems to be absolutely absurd, right? And even if, you know, the almighty being came down and told me this is the correct answer, it still seems absurd to me because I can't, I can't grasp the basic idea of why those two correlate to each other. So sort of, we don't just want the right answer, we want also the right answer for the right reason. In many cases, not always,, but in many, many cases, right? Especially in normative cases, in those cases where values, morals, and ethics are involved, like in the case of discrimination, for instance. Another case that was discovered was that dog owners were deemed to be less suitable for certain jobs. Now, again, maybe in, in a huge corpus of data, such correlations do exist. Does that sound reasonable to to exclude someone from a job because they own a dog? I would say no, this sounds like a repugnant way to hire people, right? Something that should not happen. So these are the types of issues. So when we talk about tracking, what we are talking about is the reasoning process so that the system can identify the relevant reasons and identify how relevant they are, their weights, right? So that we don't suddenly have all the relevant reasons, but the weighting is skewed, right? So I only hire this type of person even if that would be a relevant reason. Okay, so those are sort of the 3 issues that we look at. And what we say sort of is that when these 3 issues are met— now it's not a yes or no, it's always to a certain degree— if they are sufficiently met, then we could say that even though we don't have any operational control of the system, so even in theory if the system is fully autonomous, But it meets tracing, meets tracking, and we've done this over a social technical perspective, then we would say we have meaningful control over that fully autonomous system. So this paradox is sort of solved by introducing these aspects into it.

Yeah, I can see how there's a lot of really silly features that can be introduced into models that have a significant effect when you're training them, but then you start to think about them.

Yeah.

Does this actually makes sense, and it probably doesn't. I mean, like dog ownership. So unless you're working at a vet or maybe like a zoo, then it's not gonna have a strong effect on the person's performance. So it seems like you're working towards a framework for determining whether you can have control of the AI system or not. And I know there's, there are a lot of AI ethics frameworks out there. So do any of these frameworks cover AI control?

Well, no, it's not a high-level principled framework. The idea is that you would take these concepts, right, or these categories, and you would systematically apply them to your use context. It has to be done from the beginning for every use context. And the idea is that this will sort of broaden the design process. And if you account for these issues, because for every context, we have different issues that come up, right? I'm not a fan of these documents of AI values or AI principles or constitutions for AI systems. I just think there's too much versatility. It's too much context-dependent and situationally dependent. So not just on the use case, but also on the specific location and timing of it, that it really requires sort of an analysis, a particular analysis of that specific case. Nevertheless, we can learn from other cases, right? So as we gain more and more experience, like with everything, we sort of come to expect certain types of errors. Let me give one example. We know that these systems often are designed with one idea in mind, but when they go out to the wild, so to speak, people find interesting and weird ways to use these tools. So an example I came across which I thought was really funny at this hospital project, what they were developing there was a tool to help delegate the writing of discharge letters. So the idea is instead that the doctor would have to go through your entire file and summarize everything and then give you that discharge letter when you leave the hospital. The AI tool will do that for the doctor. Basically an LLM that feeds into the, that we feed to it the data, your medical records and the notes of the doctor, and it summarizes it. Okay, now they did a lot of really good steps to make sure that this is safe. They limited the types of interactions between the doctor and the system. Them. They excluded certain types of sensitive information that they wanted the doctor to manually insert. So, so they did it really carefully to avoid these problems. And here comes the twist, right? When we started talking with the doctors and interviewing them to see how they're using the tool, we learned on the very first interview that they found a really cool use for this as they were starting to use the tool. If they were gone for a few days from the ward, when they came back Instead of looking at the files of the patient, they would just look at the summary that the AI produced to see what happened with my patient over the last 3 days that I wasn't here. And that basically bypasses all the guardrails that they built in because everything is supposed to be protecting it on the day that it goes out, that a supervisor goes over the document, the doctor has to approve it first, a lot of really safeguards, and they just bypassed all of them completely. Yeah, so that's what I mean by context dependency, right? Sometimes you can't anticipate that, and when it's out there, people will find strange and weird ways to use it.

Oh yeah, whatever you're planning as a project manager, once it gets into the hands of users, it's gonna be chaos, because they'll find new ways of using the feature that you hadn't thought of. I guess you have to have some kind of feedback loop to find out if these are good new use cases or if they're problems, and so you want to feed that information back into the updates. So talk me through, what do you need to do to fix these emergent problems?

As a philosopher and as a person who sees himself as a critic, not an anti-AI person, but a person who's trying to engage with this critically to understand the limitations and the risks that are coming up, that's the advantage of not being at a tech company. I don't need to sell anything, so it's fun. I think one of the things we see often is people don't understand the systems they're working with, right? And they go too quickly from the idea of what this is going to do to actually putting it out there in the field and then learning retrospectively, oh no, it's not exactly working as we anticipated. We're seeing this with employees being fired and replaced by AI and 3 weeks later being rehired because the tool can't do the job, right? With journalists, with call centers, with different kinds of place. So what do we learn from that? We learn that hype is something that should be left for marketing, right? And our job, either as consumers or as project managers, is to fight the hype, right? We have to be really clear-minded when we're working with these tools. And it's hard sometimes. I mean, there's extreme pressure to become AI literate, right? And to be at the edge of the adopter, right? To be first adopters. I get that. But I mean, if you do that, you have to expect risks, right? And, and punishment. So, okay. Being critical of the system, trying to combat the hype. And I think as you're using the tool, like with any other tools, right? Create discourse. One of the egregious uses that I came across was with a system that was deployed at hospitals and it was meant to help Doctors and nurses identify cases of sepsis. So very, very high-stake issue. They wanted high, better sensors to be able to alert them in advance. What ended up happening is that these sensors would go off and would create false positives. And the hospital had implemented sort of punishments for nurses if they failed to act on these alarms. Alarms that would go off. And this caused invasive procedures for patients unnecessarily, high costs of tests that had to be imposed because the system went off. So sort of there were conflicts between the expert on the ground and the system and sort of the hospital had built these protocols, go with the system always, right? So that's a lesson we need to learn.

So that sounds like we're back to that automation bias problem we talked about before, where management trusts the system and then tells all the employees to trust the system. And then we're back to employees not having control over things. And a lot of the examples you talked about, it sounds like, uh, you went through healthcare examples, hiring, warfare. So these are all high-risk situations where there are big consequences of things going wrong. So, Are there any other areas where you think there's consistently gonna be a high risk of ethical issues with AI?

Yeah, I mean, the list is endless, unfortunately, right? Let's take a few actual cases that we're seeing recently, right? In the States, we're seeing the alliance between the tech industry and governance and government, specifically with ICE and immigration policies. We're seeing the use of these facial recognition technology decision support systems and recommender systems, social scoring systems, all these systems that have shown signs that they are brittle, that they tend to make mistakes, that they need clear oversight, are being used, to my mind, in the most, uh, the least risk-averse ways possible. They're just being deployed from, uh, the creator's palantír to the officer in the field, right, without anything in tween, sort of. And we're seeing the consequences. We're seeing these social scoring systems creating discriminatory action. We're seeing people being persecuted without any reason. We're seeing people lose their privacy, even if that— some might consider that to be a low-stakes issue. We know the risks of privacy loss. So this is a huge problem when military-grade technology trickles towards civilian uses and policing. These are what are known as dual-use And that's, I mean, maybe it's appropriate in warfare to use these to disregard civil rights and human rights. Of course, it's not appropriate in other circumstances. So we need to be aware of that. Another example might be that we've seen many errors, unfortunately, is the use in welfare systems. I don't know if you're familiar with these cases, but in Europe and in the US, we've seen a number of cases where they've tried to develop these tools to detect fraud, welfare fraud. And because these tools were implemented in typically bad ways, this caused a lot of discrimination and misuse of these tools where people lost their jobs, where people's families were broken up because they were accused of crimes they didn't commit. They had to repay funds. That they were actually eligible to receive. And even we've seen cases of people committing suicide after being accused of these. So the consequences are enormous, right? And we see the case in Denmark, in France, in the UK. Amnesty International provides detailed reports about these cases. They do a really good job. So if anyone's interested to read up on that, I recommend their website. They have great reports that really dig deep into the use of these novel technologies. And sort of the risks that we're taking on ourselves as we implement these. What's common across all these use cases is that those who are most vulnerable will be hurt the most. And I mean, that's typical and unfortunate and horrible. And that's the reality, right? So it's the immigrants, it's those who the state helps fund, it's those who are anyway without recourse to legal defense. And so the vulnerable. People remain vulnerable, unfortunately. So that's another example. Finally, maybe last examples, we're seeing interpersonal relationships, right? We're seeing the use of these chatbots in mental healthcare and in relation to this loneliness epidemic that we are living through, all of us. And people are gradually relying on these systems more and more to find intimacy, to find sort of a shoulder to lean on metaphorically, obviously. And some of the results are good. A lot of the results are just boring, and a small percentage are very, very bad. And we're seeing cases in which some people are suffering very much from these. We've heard about cases of teenagers going down this rabbit hole and falling in love with these systems, and finally it somehow motivating them to commit suicide. Terrible and horrible. We're seeing other cases of people abandoning relationships in the real world in favor of these romantic AI partners. Again, I'm not trying to be judgmental here. I'm just saying that we're seeing a disruption to how society has sort of been functioning over the last 20, 30, 40 years of the advent of these personal software use. And this AI revolution now of LLMs is extremely disruptive. In every single domain that we can see.

Oh man, that's just a big list of terrible things with AI. One thing that stands out to me from that is governmental issues, particularly, um, you talked about policing there. So policing just seemed like a very high-risk area where things can go horribly wrong. Um, so you've got high consequences of making a mistake, either a criminal goes free or you're putting someone innocent in jail. You've got invasions of privacy as well. There's just lots of things that can go badly wrong there. And then beyond that, you mentioned chatbots. So that feels like a more recent problem. Like there've been problems with AI and technology in governmental departments for more than a decade now, but chatbots is a newer thing. And I guess the next stage of that is around deepfakes. So I know you've done a lot of research on deepfakes as well. So, Talk me through your research in this area.

Yeah, gladly. That's sort of the second stream of my research. So one is about this MHC stuff, this meaningful human control. And the second is really about not specifically deepfakes. Deepfakes is one of them, but the field of trying to simulate human beings using AI. So there are two sort of issues we need to distinguish there. One is these AI companions that we come across. And these are basically just generic LLMs. They're not sort of trying to simulate or to represent a specific person. That's the ChatGPTs, Claude, whatever. Yeah, that's sort of an area that I don't do research on. What I'm interested in is on the other half where we take these LLMs and then we train them on personalized data in order to enable them to create simulations of specific people. Of myself, of you, of anyone we want, right? And that's sort of where I do a lot of my research. We can go through a few examples. I'm sure these are familiar to most people. What I did was create a kind of taxonomy to try and make sense of this phenomena. Most people are familiar with deepfakes. Most people know, you know, there's this problem of deepfakes that recently came in the news, this idea of nudify apps and Grok, and that was creating deepfake pornography and these really horrible use cases where mostly winning 98% is female abuse, right? So they're the ones suffering from this. And that is excellent that we're dealing with. I don't find it very interesting to talk about that from my perspective because I think our intuitions are clear, right? This is a horrible use case that shouldn't be out there. There should be legal protection for these people. This is just abuse. And sort of from a philosophical standpoint, there's— it seems like there's nothing much more to say about that. So my research doesn't go into that side, but rather onto all those edge cases where we might say, hmm, that's interesting, maybe that's okay. So for instance, um, we can think about two distinctions here. Are we representing someone who is still alive, or are we creating a representation of someone who has passed away already. On the alive front, we might use these types of systems. I might want to create a deepfake of myself in order to delegate certain tasks that I don't like to do. For instance, today I could have sent my digital duplicate to do this podcast, but of course, why would I want to do that? Because this is the fun, to come here and talk. But sort of recently, the CEO of Zoom, Eric Yuan, came out and said that Soon every one of us will be sending our digital duplicates to Zoom meetings so that we can go and do what's important in life, which is go to the beach. So that's sort of the priorities there. Um, but that's sort of what we're thinking about. We're seeing professors creating these digital duplicates to assist students in answering questions about the course material. You can go 24 hours a day, you can go onto their website and chat with the professor, which is data curated to answer specific questions on course that you're taking. Really interesting use case, right? It seems very positive. A lot of people will benefit from that. Other cases that are starting to get a bit more borderline, we're seeing influencers creating digital versions of themselves so fans can interact with them and have sort of quasi-parasocial relations with them. So one famous case was the influencer Karen Marjorie. She created Marjorie AI, sorry, Karen AI. And for the low price of $1 a minute, you could talk to Karen AI and chat about it. And of course, these talks very, very quickly deteriorated to sexting and to, you know, all the dark fantasies that everyone has. And she actually decided to cancel that and to shut it off after a while. But now I think there's a reboot going on. But in general, that's a trend that we're seeing. People try to create in order to market themselves in different kinds of ways, use these types of tools. And we might say there are advantages and disadvantages of that. What's your intuition? What do you think?

I really like this because a lot of the discussion around fake versions of people, deepfakes, is about the negative side of things where it's celebrities saying stupid things or it's nude versions of people or it's some kind of dross. So I love that there are positive use cases as well. Recently I was playing around with a platform called Delphi. So, um, they've got an AI Arnold Schwarzenegger. I had a great conversation with him about his movies, him, it, and yeah, it's a lot of fun just chatting with pretend celebrities. So that's a novel use case. I also like the professor example. So having AI teachers is a huge thing, something we are working on a lot at DataCamp. Uh, so talk me through how you make these positive use cases. Are there any philosophical criteria for determining whether you're going to have a positive use case of a fake person?

Surely, yeah. So the category I was talking about now, for instance, we focused on consent, right? Does the person know that this is happening? Is this done with their consent? And that sort of establishes prima facie reasoning to think that this is okay, right? It makes it morally acceptable if I agree to this happening to myself. So that's sort of a way we can think about it. On the other hand, we know that there are a lot of cases that are done without consent. And again, they don't have to be as terrible as the deepfake pornography or nudes. Some colleagues of mine and I have worked on a principle which names 5 different conditions that you need to think about when you're coming to create these systems. They talk about consent, Transparency, authenticity. As I'm naming these titles, I notice that I'm going back to doing what I said that I don't like, which is listing principles, right? Yeah, but in my defense, the colleagues made the list and then I did case studies evaluating the list. So I stuck to my guns on that part. But yeah, it would be basically these same principles that we were trying to think in different contexts. Maybe even without consent. Let me just give one example about that, because sometimes people think, you know, consent is a basic must. We've actually come across interesting cases where we think consent might not be necessary, at least plausibly. So one example that colleagues of mine have worked on is called the P4, the preference— the patient preference predictor. And the idea is we have unfortunately a lot of people who have car crashes or who are sick and they go into comas and they're unable to tell us their preference for medical treatment. And then we have to rely on surrogates. So we contact the next of kin or we contact someone who's related to this person. We tell them, tell us what John would want to happen, right? Or Atai, yeah. Would a Thai want us to do the surgery or not? You have to decide. Now, there's a lot of data showing that these surrogates suffer from PTSD symptoms where they suffer mental health problems not knowing did they make the right choice, right? And so my colleagues, what they tried to do is create a system that would assist surrogates in making those decisions by training a system on, on large amounts of personal data of the person in order to derive from that what would the preference be of that person in this medical situation, and then use this as a recommender for the surrogate to help them sort of make a decision in that case. Now, first of all, there might not be consent in that case, right? If the person is in a coma, they cannot consent. Maybe we could get that before, a prerequisite. Secondly, the system might make mistakes. But I use this as a borderline example where we can see the, the, the help that that might offer, right? Especially if people are suffering in those cases. We could see where this might go wrong. I don't know. This brings me back to one of the lessons that I unfortunately learned in the hospital when I was working there, is that sometimes you have to ask the weird question of, is this better than what we have, right? Is this good enough? And this is an annoying question, especially for philosophers. Philosopher because we deal in the abstract and the ideal. But I mean, if you look at what doctors are working with sometimes in the hospital, they have really old infrastructure, they have terrible working hours, they're under extreme amount of stress. And if we can alleviate that even a bit, maybe in some circumstances that's worth certain risks that we take. I'm just saying that these are the types of trade-offs that we have to make as we use these systems. Okay, but I, I, I've Yeah, gone on a tangent a bit here. Um, maybe going back to the, the digital recreations, these deepfakes, uh, the opposite side of this would be, again with the consent issue, is when we start using this technology to simulate people who have passed away already. Now again, here we have a mixed bag of good and bad. Um, so would that be cool if you're learning, uh, in a classroom, uh, you're studying about the, the Roman Empire or the Roman Republic, right? And Julius Caesar comes to class and starts talking with you, or he's helping you do homework about the culture. Seems great. Why not? That's really fun. It might be engaging for students. That's cool. Can we have the consent of Julius Caesar? It's almost a silly question, right? Obviously not. So again, we see situations where we say, well, we need to adjust the criteria here. On the other hand, let me give some more actual maybe examples. Examples of cases where we're seeing the simulation of deceased people. We're seeing two main case studies. Well, three, but I'll focus on two. One is grief-related recreations. A lot of people who have lost a loved one want to reconnect with that person or want to maintain some kind of bond, and they find that by creating these simulations, they can talk again to that person. Talk with quotation air quotes, right? And we're seeing this is becoming— it's slowly moving from a niche to an industry. A whole industry is being developed around recreating in simulation our loved ones in order to help us in some sense cope with our grief. We can go down that if you're interested, but I'll just explain the second example. The second one is for certain kinds of projects or political ends. And there what we're seeing is the recreation of individual people in order to promote a certain type of end. For example, there was a tragic shooting, the Parkland shooting that happened where a lot of students were killed by a school shooting. And the parents of one of these victims, his name was Joaquin Oliver, they created a simulation of him after his death in order for that simulation to help advocate for gun control legislation. Now, again, we can ask, is this okay or not? That's why it's interesting for a philosopher to work on this. But that's the type of use case. The parents are using the visual image of their son. You can talk with that simulation, you can interact with it, and it will tell you why we need gun legislation in the US, why is that important. So those are the two types of cases, and happy to go into either of them if you want.

For each of these, I think the tricky thing is going to be how much of this is going to be authentic in that person's voice and and how much of it are you adding on to them? So like, if it's dead granny speaking to their descendants, then you're gonna want to have them talk as they would've when they're alive. Like, these are relatives, you know what granny sounds like. So it's gonna be really easy to get into that uncanny valley where granny doesn't sound quite right and it gets creepy. So is that the main issue here? Or other ethical issues around bringing dead people back to life?

I think that's one of the issues, right? How accurate do we want it? I'm working now on a paper trying to catalog the different types of inaccuracies that come up and what are their implications. Sometimes we want very accurate recreation, visually, audio. Sometimes actually not. Sometimes we think that our goal can be met by the anthropomorphic morphizing. An interesting example, there was an exhibition that I saw in Germany commemorating the life of Willi Graf. Willi was a member of a resistance group that fought against the Nazis, and he was arrested and executed during the Nazi regime. And this— and he's a hero for his fight and struggle against this oppressive, horrible regime. And this commemoration exhibition used a digital simulation of Willi Graf where you could interact with it. And it's based on the biography written about his life. So that's sort of the data that was used for it. But when you're interacting with it, you see a stick figure. So it's purposefully meant to highlight what is missing, right? So it's meant to create that contrast. This is not Willi Graf. This is just a representation of certain aspects. Now, sometimes that's actually what we want. With grandma, maybe we want a perfect recreation of her. And we're seeing a lot of companies that deal with hyper-realistic anthropomorphic avatars that the goal is precision in terms of audio, precision in terms of visuals. My research focuses on gaps that I believe cannot be recreated. So here's an example. I think that data can capture only certain elements of our personality and our identity. And I think that there will always be gaps. And when you're creating these simulations, the whole point is for them to create novel speech acts. We don't want it to quote me specifically. Probably won't have those sources. And even if we did, we want it to be interactive and engaging. You can't just use quotes all the time, but when it's using a novel speech act, what am I supposed to think? Is that what Attai would've said? Is that what Attai does think? And I argue in some of my papers that there's always a gap that we need to be extremely aware of, that this is not what Attai is saying or would've said, it's what he might have said. And there's a huge gap there, a huge gap. And to emphasize this, and this is also a risk, and to emphasize this, I often use a mythical folklore creature from Jewish folklore, and this is called the dybbuk. The dybbuk is a type of malevolent spirit that inhabits a body and uses that body as its puppet for its own ends. It speaks to the the puppet's voice, using the puppet's voice, it speaks through it. And when I'm working on this technology, I always tell people, keep the deep book in mind and be careful not to become the deep book. And unfortunately, I think that often this technology, we are speaking through the mouths of those simulations that we are creating. And people can do whatever they want, right? I'm not trying to tell people what to do. Talk to your dead relatives if that makes you feel good. Fair enough. Be aware that this is not them and be aware that this is highly curated and what you're hearing, just like with sycophantic chatbots, is often meant for your engagement, not for anything else. And if you have that in mind, okay, then we might see still some positive things, but maybe not. And my concerns are the lack of awareness, right, that people might You know that this is what's happening. Absolutely.

And now I'm just thinking, what would an AI recreation of me after I die sound like? And so most of the recording to me is data-framed episodes. So dead Richie comes back to life as AI. I'd be great about talking about data, talking about AI, and probably gonna bore my descendants to tears. But yeah, there's a gap about what AI me could talk about. Without it being improvised. So the other thing you talked about was the idea of political advocacy. So for example, people who are killed in a school shooting, and then you recreate them for a political end. So there, they're definitely not talking as themselves. So it's gonna be someone feeding words to them, and that feels like a different ethical issue to me.

Oh, definitely. Here's a case that was was discussed quite a lot in the news. I don't know, maybe I'm so much in my epistemic bubble that I see it often. Maybe others have never heard of this. The case is about a man named Christopher Pelkey. Now, Christopher was murdered during a road rage incident, and then his accuser was convicted and found guilty. And the family petitioned the judge during the victim statement segment before sentencing to allow them to present a deepfake video of Christopher speaking to the judge and to whoever else was there, giving his own victim statement. The judge allowed it, and we have the recording. It's online. Anyone can find it if they want, of Christopher Pelkey providing his victim statement posthumously, right? And that raises the exact issues that you mentioned, right? These are obviously not his words. The family felt it was his sister who wrote the speech, and they used a hyper-realistic avatar of Christopher to speak these words. But at the end of the day, these are not Christopher's words. And I found that to be egregious in that the avatar constantly uses the first-person singular pronouns, I, I. And so it's speaking as if it is Christopher. And so it doesn't create that kind of distance that I think would be more acceptable and suitable. It sort of tries to make all the other influences invisible, right? It tries to eliminate them that we don't see. At the end of the day, what we see is Christopher talking and his words, but that were again made— he was turned into a puppet in this sense. And he says certain things that are really crazy, like he speaks to the murderer and says, in another lifetime we could have been friends. That's what sort of the avatar says. And then it thanks the judge for certain. So you ask yourself, how manipulative is this? How, even if this is somehow faithful to what Christopher would have wanted to say, how does this impact now the judicial system? Should we be having these type of influences coming into courtroom? A lot of difficult questions that need need to be addressed beyond the sort of philosophy of whether this accurately represents Christopher or not. Absolutely.

I mean, to me, it seems like that's just a ploy to manipulate a jury rather than it being good evidence to put in a trial. All right, so we've talked about a lot of different ethical issues. If you are an AI practitioner, what are the most important philosophical skills ethical skills that you need?

Yeah, I don't know. That's a hard question to ask. I can speak more to myself and to my strategies, how I approach issues, and then maybe I'll share about a project that I'm working on because that might sort of demonstrate some of this. So as you heard throughout our conversation, I've mentioned Jewish issues several times. That's sort of strong in my identity and background. I've come from a family of— I'm descended from Holocaust survivors, and that's always been an issue that has been of great interest and importance to me. And when I came across this technology a few years ago, one of the first ideas that I had was, wow, maybe I can use this to help my grandfather, who had passed away in 2006, share his survivor story, his Holocaust survival story. And it sort of got into that whole field of survival testimony of Holocaust survivors. And there's a huge problem there because, uh, there are very few survivors remaining, right? They're very old, the last ones that are remaining, and they were very young during the war. So we're losing that firsthand sort of interaction with survivors. And educators in this field are trying to think, how will we adapt to a world without that? And sort of the questions that I ask myself Well, can we use this technology to help? And sort of, I started working on that. And as I worked on it, I started coming across all these philosophical deep questions and ethical conundrums and principles. So to abstract from that sort of idea, this is not very suitable to companies that are trying to market products, right? But I think slow and steady is the way to go. I think we have to be very reflective. When we are using delicate technology in a field like Holocaust remembrance. You know, it's a very sensitive issue for many people. It deals with a lot of difficult topics, obviously. I think you need to be very, very conservative in how you do this. So I like to say sometimes that I think you need to wear two hats. As a philosopher, whenever I teach to my students, I tell them, ask the most outrageous questions you can think of. You know, push things to the extreme. What if Christopher Pelkey, we couldn't just use him in the courtroom but then continue living with him? Ask those questions, engage with them, think about that. That's when you're wearing the philosophy hat. When you're a policymaker or a company that's creating a product, I think you have to switch that hat and be extremely risk-averse. You need to be very, very careful. And if you come with that mindset of risk aversion, and as a philosopher, you bring sort of this ethical background with you and you read about these topics, then I think you can anticipate a lot of the inappropriate or bad things that come out. I can give an example from my project. So I'm working now with the son of a Holocaust survivor called Eva Kor. Eva lived in Indiana. And she became an advocate for Holocaust remembrance throughout her life. She died in 2019. And her son has sort of continued her work of education. They go once a year to Auschwitz with tours with students and athletes, and they do wonderful work. They have a museum that they created there, the Candles Museum. And what we were working on is creating a digital version of Eva in these AI models. What I thought, sometimes these things are very clunky. You can't really interact with them well. And I went and I created a version of Eva together with Alex and worked alongside him a bit in order to get good data sources and to make sure that they're agreeing to this. And as I was developing it, I started coming across difficult questions. So one silly example, it was replying to me with a style of, you know, a ChatGPT style and ending in its sentences. It would like ask me back a question every time it answered something, just like you would get when you interact with Claude or Gemini or whatever. And I wanted to edit that out to make sure that it doesn't respond in this way anymore. And as I was doing that, I started getting these chills of, whoa, wait a minute, I'm making this Eva simulation speak in certain ways I'm, I'm literally becoming the puppet master at this point. And that was really hard for me, even though I felt like I was doing something very well-intended. And yeah, so having worked on this for 2 years, I'm still not certain that it's good to do this or that it will be beneficial at the end of the day. And obviously companies can't adopt this very, very slow attitude of, you know, really testing, really making sure you stand behind this product or Do you stand behind what you're doing? But I think that's the way to approach this, you know, be fully committed to what you do and endorse what you're making and be aware of what are the risks and what are the trade-offs that you're doing. So full awareness of that.

Okay, a lot of the principles here around critical thinking of what could go wrong. So you need to be able to predict ethical disasters before you release them to the public. You don't want to deal consequences. So you wanna deal with them upfront in the design.

So I'm working on a project now, which is trying to consider, can we use, um, uh, these digital avatar simulations in order to promote political participation? We want more people to be involved. We want people to be more aware of what's going on. And sort of what we're working on is we're partnering with politicians to create digital versions of themselves. And we're doing this in Switzerland where I'm based. And Switzerland has a direct democracy system, right? So there are a lot of referendas that you, you as a voter, need to vote on a lot of issues that you don't know much about. And what you usually get is a packet of information saying the, the position of the different parties, the stakes involved, and then what do you think. And it could be about weird things that you've never thought about in your life, right? And so one of the projects we're doing is to have have the possibility for these voters to interact with a digital version of the politicians on this specific referendum that you need to vote. Obviously, you can't engage with the politicians directly, right? Just because of the numbers of modern democracies, not possible. The question we're sort of researching is, can we use these tools in order to create that type of intimate conversation between citizen and their representative. Now, we're testing this only internally, right? We're doing experiments with this because there are high stakes here. Is it— how will this basically manipulate voters, right? How will this— will it hallucinate? Will it go off track and suddenly spout the opposite opinion that we don't want it to do? So we're sort of playing around with that. Now, we see the potential benefit, but we see also the downside. So it's just as you said, right? Critical thinking, approaching this with an open-minded sort of perspective, but well aware that this, you know, might at the end of the day not work out and not be successful, rather than moving fast and breaking things, uh, the mantra that we're hearing coming out of Silicon Valley. Um, I believe that when, you know, democracy is at stake, that's a bad way to go. So, um, I'm hoping that this type of project might offer insights into one way to use this technology that might promote democratic values. And part of the problem is that I'm very concerned that the opposite will happen. I think rather than increasing knowledge and access to information, what would more likely happen is that we would automate the voting process. We would say, well, I don't really need people to I can just use the big AI to predict what you're going to vote, and that's good enough. So I see that as the extreme opposite on this spectrum of democratic AI systems. So I'm trying to combat that in that sense. But I think that gives sort of some insight in how to approach these types of projects. Now, in reality, can everyone do that? Of course not, right? We sort of have this privilege of, again, being in academia, not having to sell anything, just doing research. And it's very scary that a lot of these companies today— Google, Meta, Amazon— they're more lucrative for researchers, right? Researchers can get their huge salaries and get all the toys to play with, and then they sort of abandon these ethical research principles in favor of going to these big companies. I can understand that on a personal level, but as a phenomenon, I think it's extremely dangerous. And we're seeing the trend of publications coming out from these companies. In my opinion, these— there's a little asterisk next to it always. Corporate research needs to be taken with a grain of salt, right? I mean, how else can you think about it? Absolutely.

I mean, I love the idea of using AI to educate people, to increase civic engagement, having more informed voters. I mean, that's just an unambiguously wonderful thing. I guess if it works, of course, and I guess that's the key, right? You need to make sure that you test things properly, you need to think carefully about the consequences before it goes live, and you've just broken democracy. But I should say, I mean, I work for a tech company, so I have sympathy for tech research groups, but I have also worked in university in I agree. It's a very different experience for how you approach building things.

And let's be honest, universities aren't perfect either, right? I mean, we've seen since the new Trump administration, the amount of clashes that have come between government and universities and whether these are legitimate or not, I'm not going into that. It just highlights that universities are not neutral, right? They have positions. So I've sort of painted it as if, oh, this is the good guys and, but it's obviously not like that. And I have a lot of colleagues working in these companies, so I think some of them do excellent stuff. Yeah, I'm just being a bit, uh, of a critic here, a naysayer for a second.

Yeah, you're a little bit salty. Yeah, uh, I mean, of course universities are a great place for fundamental research. Uh, this is a necessary component of a research ecosystem.

Yeah, I have a colleague who teaches at Texas A&M, and unfortunately he was recently told what he's allowed— that he's not allowed to teach any more Plato texts in his philosophy lecture because this conflicts with the new, what is it called, the new regulations about diversity, equity, and inclusion, right? So the anti-regulations that came out. So yeah, censorship exists also in universities, obviously, and I mean, people who are more radical than I am say that I'm working for the man.

So, yep, get your information from different sources. And again, we're back to critical thinking being an important skill. All right, to wrap up, I always want more people to learn from. So whose work are you most excited about at the moment?

I try to read a lot, so, you know, that's my job also. But I think getting multiple sources, I try to get out of my own bubble that I think we are all sort of being algorithmically contained in that it can be very difficult sometimes because you think you're seeing the objective reality and then suddenly it pops and you see, whoa, what's going on? There's so many other perspectives. So I think, you know, showing interest, trying to actively seek other types of opinions is important in order to think of things. I think that, you know, in my sort of domain, I have a colleague, Professor Sven Nyholm, who does excellent work, and I follow him and read all his new articles. There's some media channels that I follow. One is 404 Media. I think they do a lot of interesting tech-related exposés. There's a center at Cambridge called the Center for Future of Intelligence, which I like very much their work., and I follow several of the scholars working there. So I think those are interesting places to look.

There's just a wealth of reading around ethical issues in AI. I mean, it's just such an exciting topic. So lots of ideas worth looking out for.

Wonderful. I think that maybe I'll just add to that, that I think another issue that's important is humility. And I try to do that as often as I can to recognize that I don't know everything and try and be, you know, self-critical as much as I can. These issues. So this is a really complex topic. A lot of the stuff goes over my head, even though I'm supposedly supposed to be an expert in it and working on this day to day. So I think we need to acknowledge that. And I think asking questions and saying, I don't understand, or doing more research is crucial regardless of sort of the topic we're working on. And yeah, so engaging like in these types of conversations, I find to be wonderful because I get other perspectives. Coming at me from outside the philosophy bubble. So I really appreciate that you invited me here and that I had a chance to talk here today. Let me just say that I love when people contact me, so feel free to do that. I'm mostly on LinkedIn, it's the only social platform that I use, or my email. So if you hated what I said or you liked what I said, just let me know. It's all good. I love discussing this stuff and I'm always open to do that. So Yeah, so really, Rich, thank you so much for having me here.

And I really appreciate that. Okay, thank you for your time, Artai.