Staving off disaster through AI safety research

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by Linode, our cloud server of choice and we're excited to share. They've recently launched dedicated CPU instances. If you have build boxes, cicd, video encoding, machine learning, game servers, databases, data mining or application servers that be full duty 100 cpu all day every day, then check out Linode's dedicated CPU instances. These instances are fully dedicated and shared with no one else so there's no CPU steal or competing for these resources with other Linodes. Pricing is very competitive and starts out at 30 bucks a month. Learn more and get started at linode.com changelog again lino.com changelog Foreign. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community follow us on Twitter. We're at Practical AIFM and now onto the show.

Welcome to the Practical AI Podcast. This is Chris Benson, your co host as well as the Chief AI Strategist at Lockheed Martin RMS APA Innovations this week you're going to hear one of a series of episodes recorded in late January 2019 at the Applied Machine Learning Days Conference in Lausanne, Switzerland. My co host, Daniel Whitenack, was going to join me, but had to cancel for personal reasons shortly before the conference. Please forgive the noise of the conference in the background. I recorded right in the midst of the flurry of conference activities, separately from the podcast. Daniel successfully managed the AI for Good track at Applied Machine Learning Days from America and I was one of his speakers. Now, without further delay, I hope you enjoy the interview. My guest today is Elmadi El Mhamdi and He is a PhD student who's just finishing up here at EPFL in Switzerland and he has been focusing on technical AI safety and robustness in biological systems. Welcome to the show. And did I actually say your name correctly?

That was good.

And if you could start us off, we've talked a little bit before we started recording. You have a fascinating background. Will you share a bit of that as we start this off with the listeners?

I've been trained as a physicist, so I did math and physics as a bachelor's in Morocco, then moved to France Switzerland and Germany. But I've been trained as a physicist. I even worked in physics research. I've been a research engineer in physics, the physics of condensed matter, like semiconductors for things like photovoltaics, solar cells. But then I drifted a bit for about five years before coming back for a PhD. So I started like I did research in physics, but then at the same time with some friends we co founded a media platform in Morocco called memphokinch which was some sort of like a news aggregate during the 2011 events. That's some, some people called the Arab Spring. And during that period I was more and more convinced that the web was enabling through those platforms tools to help people circumvent usual platform like sorry, usual intermediate bodies like electoral political parties, established news organizations to self organize. But at the same time there was a harmful effect which we will start being more aware of five years later during the last events in the US for example.

And would that be misrepresentation of events like fake news and that kind of thing? Are you referring to true?

And so back then 2011, 2012, there was another thing that caught my attention which was that whenever we put a lot of effort, me and my colleagues at Mafia Kinch would put a lot of effort in doing a deep investigative work on some very relevant public issue and then publish it. The readership would be very low compared to a three minute video by some activists who just like self record himself or herself with a basic camera and start speaking in a very simple word and it will take off. And back then 2012 say I kind of stopped being very involved in Mfaquinch. I was still working in physics by the way, but I thought that the video platforms will play an even increasing role as the bandwidth and access to heavy content like video will be democratized. And I said yeah, okay. So the video sharing seemed to be more powerful than text sharing on the web. I think this can help a lot into this. Like something that also I care about, which is science education. And as much as, as much as politically, like as much as videos on political issues would have more spread than text. I thought that videos on science would have for like the general audience for kids to make kids motivated about science or like just like to tutor people. I was not aware of the Khan Academy back then. Someone like someone showed me a Khan Academy after I did my first videos. And so I said good, okay, I will start a video project that would just do tutoring in physics and math, something I was kind of good at doing, which is tutor people in maths and physics. So I started a tutoring project in maths and physics while working in condensed matter physics back then. But then a professor here in computer science, Rashid Gharawi, convinced me to join efforts and to do that full time with him here in Lausanne. So we got initial funding from Google and from the faculty. Then the faculty of course, like the faculty took over the funding and we got enough funding to do it full time. And they started learning about computer science as a fundamental science and was realizing how epistemologically relevant computability and concepts like decidability were to understand.

And about when was that? Just for the timeline. I know from 2011, 2012, this would have been moved on another year.

This transition would happen in 2013. So in 2013 I left my job as a physicist engineer, came to Lausanne. By the mid June, I think June 2013, I left my job as a research engineer in physics and came to Lausanne to fully start this tutoring project that became an official tutoring platform of epfl. So bachelor students who had a very good reception. It's not the kind of YouTube channel that would go popular because it's on very specific and technical topics and it's in French. Most of the content is in French because EPFL is a French speaking university. So the audience was not huge. It's a small sized audience, but there was a high quality. For example, we had a high retention rate compared to say MOOCs, like MOOC platforms. They had a 8% or 7% retention rate. We had something close to 70% retention rate because it was a tutoring. So we were addressing questions bachelor students of EPFL would struggle on before an exam, like how to compute this third derivative of this physicist stuff, whatever object.

So that really does sound quite a lot like Khan Academy in what you were doing. Obviously you're doing it in French and doing it for the students here. So we can kind of think of it in that kind of context and where you're going. And so where did that lead you?

It led me because I was funded by the computer science department. It led me to learn more about computing. So back then when I was trained as a physicist, I viewed computer science as this engineering thing where you debug Java and C code. I didn't like really, but I was not aware and I was not educated on this fundamental science of computing. And little by little I started educating myself. I started learning learning theory. That's so meta. So I started reading about learning theory. The work of Lesley Valiant, for example, the work of Vapnik, Schervonenkis and also the fundamental CS parts. Turing and I kind of buy into a few calls, for example from Net Slovania, to make computing a natural science. I think it's a very powerful epistemological tool to understand natural phenomena. In terms of. I'd like to call computing as the science of the feasible. Like what can be done? What can be done? Like complexity theory, what can be done in an amount of time with an amount of resource. And I would like to view learning theory as the science of the learnable. What can be learned given an amount of time and an amount of data points and amount of samples. And I love that. So I wrote a proposal to start a PhD trying to understand biological processes with compatibility tools, not as a computer scientists collaborating with biologists and coding stuff for them, but not bringing the engineering part of CS, bringing the epistemological part of CS that view complex systems through complexity theory, resource, etc. And the main guiding line was robustness. So could we explain robustness in biological processes with computational tools. So could we explain, for example, why an ant colony is robust to randomly killing some of the ants up to certain level without having a central authority allocating tasks and telling ants, oh, by the way, we had a certain amount of foragers that died. Yeah, those of you who were doing, I don't know, nursing should switch to foraging. And we know, like myrmecologists, biologists who study ants know that there is no central authority doing that. It's self organized and it's robust, it's fault tolerant. The brain also is a very good example of a robust structure where there's no central authority telling neurons what to do. Or to a certain extent, it's very distributed and robust and it can tolerate the loss of some of the nodes. So that was the starting line. Let's understand the fault tolerance of biological processes with tools from algorithmic theory and those kind of tools and in the distributed computing group. So that was like a very physicsy. So that was something that could bring the physicist in me again to liking, like doing research five years after I left my master's. But little by little I was just my awareness on more applied aspects of machine learning will grow. So I told her I was trying to understand fault tolerance in neural networks. So how does error propagate in a neural network when some of the neurons are removed? Today this is not a practical problem because neural networks, like neurons in neural networks, do not fail. A neural network is simulated in a machine. So the unit of Failure is a whole machine, not a single neuron. This will become a problem when we will have neuromorphic hardware. If you heard about this, could you.

Define what that is? Specifically?

Neuromorphic hardware is a class of hardware that is itself built as a neural net. So the hardware itself contains pieces that behave like a neuron and pieces that behave like a synapse. While today we just simulate neural networks as a software.

So would it be fair to say then that because you are implementing hardware in the form of a neural network that you can have just like any other machine out there, you can have parts of the machine fail. And therefore, unlike today where it's just software and you either have it's all working or it's not, you can have parts of that hardware in the form of a neural network fail. And therefore it's a new problem for us to solve. Which is why you were saying it's not practical. Am I understanding you correctly?

It's not a really new problem. It was a very popular problem in the 90s and 80s before the last AI winter, because people were expecting neuromorphic hardware to arise next day. So people like you find a lot of papers in the 80s 90s about fault tolerance in neural nets and they will talk about VLSI circuits, very large. But then neuromorphic hardware didn't happen and we simulate neural nets on machines and people stopped caring about this problem. But yeah, I find it a very good physics, I find it a very good problem for someone who thinks like a physicist, like me. So I cared about it, even though there's no normal regard where room in the air today, but little by little people who are relevant in machine learning would tell me, look, we don't care about this yet though if you could understand how error impacts learning in distributed frameworks, like when we train machine learning systems over a set of machines, that might be relevant today. So I switched a bit of interest. I published a paper on like, I approved some bounds on error propagation in neural nets. The mathematical modeling I did there was also useful to study biomolecular networks with some friends from the John Hopkins Medical School, because it turns out that biomolecular networks are just weighted graphs of nonlinear nodes, just like neural nets.

Oh no, that's pretty cool. I had never thought of it that way. Okay, so that's the. That. So that I know that you, I was going to ask you that. Was that you had talked about that factness that you were dealing with the robustness and biological systems with the technical AI safety and is that the crossover there. Are we getting to that or am I. Am I jumping in?

Not yet, but the glue is already there. The glue is fault tolerance. So there are like two hemispheres. In my PhD, one hemisphere was due in robustness in biological systems and one hemisphere was due in technical AI safety. They don't seem to be related, but they are actually true fault tolerance. So I cared about fault tolerance. What happens in a complex system when some nodes are knocked out or are misbehaving or are lying to the group.

Oh, okay, there. So you've gotten to the crux of it. I know that as we were talking when we first met and you started talking about that, I can't wait to hear how this goes in because it's fascinating how you've pulled together multiple fields that may not be obviously related upfront, but through fault tolerance. And then you were making comments earlier about how this affects things like fake news and falsified information that goes forward. So take us there.

Let's go to more the technical AI safety part of my research. I like to tell this when I say that I work in, when I say to my friends, oh no, for the past two years I switched a bit interest caring about technical AI safety. They would go like this. Oh yeah, isn't this about killer robots and rogue self driving cars and things we'll have in the far future? And I think partly because the media was always showing those kind of motivation when they talk about AI safety. I always like to tell them that there are killer robots already about us. They're very dumb and primitive and doing very basic machine learning and they are called recommender systems.

No, that's great. But you'll have to kind of explain what you mean by that because that's a little bit of a shocker when you hear that.

So imagine a young couple of parents who just had a kid and then they go to a search engine and type medical advice on vaccines for young kids. And then they got an initial piece of content that tells them that this is harmful, it can cause autism, and then their kids can die. And this is really a conspiracy by Big Pharma to make us just buy their products and then the platform recommend them another video telling them similar stuff and another one and another one and another one. And actually that could also happen to people who didn't even search for that. Just like they were just like looking for medical advice on some random topic for herpes and then they end up on a video telling them, oh, there's this Big Pharma conspiracy Don't take your kids for vaccine.

So it's funny that you say that because I actually have friends and even extended family members that that exact use case has applied for them. And we have gotten into debates on the benefit of vaccine. And so I love the fact that you know, you started in kind of from that academic perspective but you're, you're now touching on something that affects lives every day by millions of people out there. And it's a very common misconception. So I love the fact that where you're going, keep going. Sorry about that.

Now today, just this year, I think for the first time in maybe several years, I don't know how much but it's like for the first time in at least the past five years or so, the World Health Organization listed vaccine hesitancy as a public health issue. It is listed. I'll give you the reference after the it is listed so you can give the link to the audience.

Yeah, we'll include that in the show notes.

So the World Health Organization listed vaccine hesitancy in its 2019 report on the measure like it's now in the rank of HIV and Ebola because there is a surge of anti vaccine resentment and there is a surge of vaccine preventable diseases. Some estimations I remember from that report you can maybe check if like I might miss some details but for example, there is a surge of 30% in measles in developed countries. I'm not talking about, I'm talking about like countries that solved, that's used to solve that measles 30%. There are some other reports who speak about about 1,600 deaths in the US per year from vaccine preventable diseases. That's three per day. That's more than terrorism. So this is a less reliable report but the World Health Organization One is talking about a 30% surge of measles. That's a vaccine preventable disease. And the resentment is growing. There were also studies on the opposite people's opinion on vaccines in France today and 10 years ago and they consistently show a growth of this resentment. So this is clearly a public health issue and we can say with confidence that yeah, poisoned machine learning already kills people. Think about killer robots. I'd like to tell them let's just first care for, let's care about poisoned recommender systems and probably what we will do to solve that might probably help in preventing something in the long term. People tend to think about killer robots in the long term and far future stuff we shouldn't worry about too much. I'd like to reply. I'd always like to reply that no, no, we should care about killer recommender systems that are pushing parents into not vaccinating their kids. There are surges of cases like measles, not only in the us, in Switzerland here there was a case last year, there was a, an outbreak in the primary school, I think, or a kindergarten. So in March, I think you can search for that in this region, Lausanne region. And this is a serious problem that is literally already killing some people. I think new generations who didn't witness the past. My generation didn't see. What does a non vaccination past look like? I'm from Morocco, I grew up in Morocco until I was 21. My aunt had polio, she was handicapped for life. She was born in the 50s and she was not vaccinated back then. So I could see what an unvaccinated past looked like. I think it was even uglier than what I could see because I just saw the survivors. And I think that my generation in the west is not aware of how lucky we are today. And recommender systems today as they maximize watch time. So yeah, the problem is that when we maximize for some metric, we tend to screw stuff in other metrics. And maybe maximizing watch time is now leading to what we do today. So how could we turn that into formalizable scientific questions? If you look at machine learning today, if you look at how it is done, you would find that fundamentally there is an averaging mechanism. So when you do gradient descent, that's just a protocol to update parameters. That's just a protocol to update parameters. Okay, you do it thanks to some data points. So you leverage some data points, you compute gradients using those data points and then you aggregate those gradients. And how it is done today, it's mostly with averaging those gradients or variance of averaging. If you ask a sociologist about averaging, would you do averaging to do socioeconomics of religion? Any reasonable sociologist would tell you, please don't take the average as a funny illustration. Maybe it's not really funny, but it's a bit sad. In my talks, I ask people who thinks that the GDP per capita in Finland, Denmark and Sweden is higher than the GDP per capita in the U.S. most people in the room raise their hand because they think that the GDP per capita in Sweden, Finland and Denmark is higher. Actually, that's the opposite. It's slightly higher in the us So.

I know that I was one of those people you're referring to. I would have said the other way around. It's Interesting. I did not realize that.

Yeah, you have even more striking cases like you can take the GDP per capita in Germany and the US or something like that and you will find that the one in the US is way higher, I think. But for sure Denmark, Finland and Sweden have GDP per capita according to the last OECD or CIA reports, slightly lower than the one in the us. But no one is full enough to say that the typical Swedish citizen has has a poorest life or a comparable life to a typical US citizen. Unfortunately, the typical US citizen tends to have less access to public education, healthcare, etc. Why? Because averaging is not robust. If you take the average and you have a bunch of over rich billionaire and several homeless people, yeah, the average might be good. I come from a country where this is also, I think when I meet a Moroccan and Algerian. So we have a neighboring country called Algeria. And if you ask any educated Moroccan or Algerian in Europe, where do you think the median now this time the median income, access to healthcare or whatever is higher, they would tend to say Morocco. Because there's this big outlying cities like Rabat and Casablanca where you see fancy constructions and very good cars on the road and think that yeah, this country seems to be a bit richer than Algeria. But it turns out that's not the case. The median Algerian has a better life than the median Moroccan. But Morocco has a bunch of outliers that think of themselves as a middle class while they are not. So sociologists, short story. Sociologists were aware of the weakness of averaging from at least the 19th century. If you read Emile Durkheim or if you read the Weber, the first data, scientists who were probably sociologists and they were aware of this problem and they will tell you, yeah, like take the salaries, rank them, take the one that splits the distribution in two halves. That could be a better way to evaluate a country than taking the average. So if you do that, I see.

What you're saying on that and I was going to ask you how that the weakness of averages were kind of tying back in to the use case that you're addressing there.

So now, yeah, a naive idea is, yeah, let's port that into machine learning. Let's take median gradients instead of average gradients. So people behave on a social network, their behavior create gradients. What's happening today is that the social network will use the average gradients to update the model. If there is a minority of hyperactive, hyper motivated extremists, they might screw the recommender system.

So to tie this back in, I mean this is exactly what we're seeing day in and day out with the impact of social media in a negative way on our lives. So it's fascinating as you've kind of come in through this academic path that you've taken, but you've landed squarely in the middle of a gigantic problem that we're facing around the world. I know as a US citizen, we are having a lot of a political conversation right now around exactly this. So what are the implications of this?

The implications might be, for example, what happened last year with the crisis acts for conspiracy. So I don't know if you remember, there was this very sad shooting in Florida and the parkland at that high school. And a few survivors of that shooting, David Hogg, Emma Gonzalez and others, they raised to prominence with their campaign promoting more safety measures and gun control measures that would protect high schools from shootings. And there was a video claiming that those kids are not real survivors from the shooting. They were crisis actors. Crisis actors used to promote gun control on television. And this video went on the front page of YouTube.

So basically you're talking about an instance of pure fake news in terms of you're having a bad actor that is creating a fiction to serve therein, just to serve their end. It had no basis in reality.

It doesn't end on the video being featured on, say if you went to YouTube.com that day, you would find this video in the US that was the featured video on the front page. But it didn't end there. Those kids received death threats because people believed the video. The video spread, it became very popular and the spread was done even though YouTube apologized. So YouTube apologized of course later and they fixed the problem, but it was too late. Like the harm was done. The kids received death threats. And imagine you are surviving a shooting and then you receive death threats because people massively saw a video saying that you are a crisis actor going to the television to promote a political ideology of gun control.

So is your research into robustness and stuff? How is your research, how can it be applied to these real life situations that we're all trying to figure out right now? How would you, what are your solutions?

Of course, real life solutions are very complex. I'm not claiming that we have fully, fully full bullet, sorry, bulletproof solutions to complex real life problems, but we could at least fix the obvious real life problems. And the obvious real life problems is that recommender systems should stop averaging gradients, for example. Now, I'm not claiming that this is pure poisoning. What's happened to YouTube? I don't know. I don't know what happened exactly to YouTube, but I would say a first fix would to stop taking the average. Maybe if YouTube already fixed that, or maybe YouTube, maybe that's another problem that I was not aware of. But let's say there is a situation where you average people's behavior and the first fix would to stop averaging because you would be vulnerable to extremists groups.

So, so would it be fair? Earlier you mentioned median. Would that be a better selection? So fundamentally the, the approach that we're taking in machine learning in terms of the, the choices we're making as we're putting our algorithms together for a given use case or solution, in some cases maybe we're making, we're, we're kind of following the herd and we're doing what other people have done on other projects. But in the case that we're talking about, it's, it's not serving us well because you can have extreme ends of that distribution that are able to take advantage of it.

Most importantly, spotting those extreme ends today is becoming harder and harder. I talk to bankers and insurance companies, they're very good at doing fraud detection and they typically would do it with tools like pca. I don't know how much details I should go into this podcast, but this is a method that detects big tendencies in a data set. The problem with that, so it's very good to spot outliers, but the cost of doing it grows quadratically as the data set is big. So it prevents you from leveraging high dimensional big data, as we like to say today. So it's, it narrows down the scope of your tool to simple linear regression, logistic regression. You can't do it. You can't do those kind of fraud detection mechanism on something as massive as a video platform. So we need something that scales at most linearly with the dimension of the model of the data. And finding something that behaves like a median in high dimension is a hard problem. So the technical solution we've been working on, me and my colleagues since I jumped on this problem two years ago or so, I took a break from the biological robustness track. I'm getting back to it now, but I took a break for two years and I fully worked on this poisoning, resilience and another AI safety question called safe interoptability with some friends. But yeah, on the poisoning side we've been trying to find alternatives to the median because in high dimensions you can't, as I said, you can't rank like you rank salaries and then you spot the salaries that split the salaries into two halves. Half the population earns less than 3,000, half the population earns more than 3,000. 3,000 is the median. Fine. How do you do that for vectors? So for multidimensional data, you can't rank vectors. You can't say, oh, this is smaller than this one. Imagine like you have a million spreadsheets, each spreadsheet containing a million cell. You can't rank them, so you want to find the median spreadsheets. That's more or less what we're trying to do in a practical manner so fast. And so that's what we've been doing. We've derived the series of algorithms that behave like a median and that provides guarantees that it is bounded in between a majority of points, et cetera. And we proved. So we've been also promoting the fact that security measures always have a rigorous proven. Whenever we found a bug we have to go back and modify. But it's not, it's very good to. Security measures should not be supported only by empirical evidence because you can never simulate all the possible attacks. So we always tried to prove that this protocol called gradient descent will always converge despite the existence of a fraction of poisoners. So we had the first paper on that in NeurIPS 2017. I'll give references and if you want to.

Yeah, we'll definitely include those in the show notes, I guess. Is it fair to say these higher order algorithms that you're talking about, is this a way of kind of maybe evolving gradient descent or maybe replacing it in such a way that we start having real tools to deal with poisoning and with fake news instances and such as that.

Yeah. So talking about tools, I've been. So my work has been more on. I was the guy who would find an algorithm and prove that this algorithm satisfies this requirement. But then I've been trying also to work with my colleagues and co authors who are more on the engineering side, to port this on tools as soon as possible. So I said, yeah, we had this first paper in NORIPS, then we published follow ups in ICML, two follow ups in ICYMET 2018, one in asynchronous settings and one in very high dimensional settings. But now we have a fourth work where we took TensorFlow, like this famous Google framework, to do machine learning. We took TensorFlow and we replaced every averaging in the gradient aggregation parts of it with how the algorithms have been promoting for the past two years. And my friend, my colleagues Sebastian Sani and George they made it work on TensorFlow. And not only that, as also as a side bonus, they also made TensorFlow work. Communicating with UDP. So now not only TensorFlow, the version of TensorFlow we will publish on GitHub this week, is Byzantine resilient, so it tolerates poison ingredients up to a certain fraction. But it also can communicate over udp, which is an unreliable communication protocol, instead of the previous one which required TCP IP because you cannot afford losing packages, etc. So as a bonus, now you can communicate over a faster but less reliable communication channel. That's not really like, it doesn't have to do only with the medium stuff. They also did some technical changes.

So if you were an engineer out there and you'd listen to this and wanted to, to take advantage of that, because I had a sense that that's where you were going in terms of the research. You've now kind of have your own approach to gradient descent. Do you foresee that ever being included with TensorFlow or do you think is the usage of what the output of the work, these tools that you've created? Do you think it'll be common enough for dealing with things like poisoning and with dealing with bad actors trying to take advantage of the data set? Do you think we're going to gradually evolve into using these types of updated algorithms to replace the average based stuff, or do you think it's always going to be a little bit more a specialized thing?

I don't know if you know Stuart Russell, this famous professor at Berkeley. Stuart Russell, yeah. So Stuart Russell is one of the pioneers of modern AI. He wrote that textbook AI A Modern Approach with Peter Norvig. And I like one of his arguments. We met in a conference weeks ago in Puerto Rico in this beneficial AI conference in Puerto Rico by the Future of Life Institute. And I like one of his arguments for AI safety where he said if you talk to civil engineering people, you will never find someone talking about bridges and someone else talking about safe bridges, which are bridges that do not fall apart after three hours. So not falling apart after three hours of deployment is part of the definition of a bridge. I think that the feeling I had from talking to attendants of applied machine learning days is we are going slowly towards this good direction where most of the people involved in machine learning research are more and more aware that not falling apart after a few hours of production is part of the definition of a bridge. And I think we will stop talking about safe AI and AI, it should become part of the definition.

Yeah, so it sounds like It's a foundational thing that we probably should have been thinking about ahead of time. But it will become the de facto standard. It's essentially going to the success of safety. AI essentially eclipses itself. It just becomes AI and the tools we use.

And then now, coming back to your question, is poisoning? Maybe I'm rephrasing exactly. Not exactly what you say. That is poisoning really solvable like that? The bad news. There is always a bad news in computing. People tend to forget that computer science was founded by an impossibility theorem. Turing, before proving what algorithms could do, he started by proving what algorithms could never do. The halting problem. You could never find an algorithm that audits algorithms and says whether this algorithm will terminate or not. Okay, so algorithmic science, computer science started out of an impossibility result. We have to really remember that. And we are a field of science. I like that. We are a field of science where impossibility results are foundational because they narrow down the scope of what you can do. You cannot do this. So you can only do what is within this scope. On the left, good distributed computing. So the field I'm part of partially also has strong impossibility results. You can't solve consensus. You can't agree if a fraction of the nodes is malicious and exceeding a certain fraction. So for example, if we want to agree on a common decision and more 51% of the group are malicious, we will not agree on the safest choice. This is trivial. There are similar theorems in game theory, by the way, like the Ro Gibbard theorem, like the impossibility theorems for democracy and social choice. We also have impossibility results for distributed machine learning. Or you can just think of it like machine learning, gradient based machine learning that are not new. We were just like. I'm not claiming that we were behind that. We just renewed the interest in them. They were proven in particular in 85 by a Belgian guy called Peter Roussev, mathematician and the community of robust statistics. You could actually prove that if you have a group of estimators. So a group of random variables. And so sorry, not a group of estimators. A group of random variables following some distribution. And an estimator could not guess the mean of those random variables if more than half of them are adversarial. And then he coined this thing called the breakdown point. We call it Byzantine fault tolerance in distributed computing because it has to do with a thought experiment called the Byzantine generals problem, that we don't really need to go there. It's just an agreement problem between three generals surrounding a city. So if one of them is corrupt, they can't agree whether to attack or not. So if you have n generals surrounding a city and the city only needs to corrupt a third, it doesn't need to corrupt everyone. If it corrupts only a third of the generals, the generals could not agree on the common decision. And the same you cannot make gradient descent work if a certain fraction is not reliable. So if most people are promoting anti vaccine, of course no solution will work. I'm not claiming that we have a.

Fluid proof, so there's limitations. In other words, there's success to be had, but there's also some limitations that if certain circumstances like that many are working against you, you won't be able to overcome that.

But then people, people on those big platforms I think are smart enough to realize that and they are realizing that. I saw a very good press release from YouTube last week where they said that they will actively now try to work to prevent phony medical advice to be recommended on YouTube. So this is not about censorship, it's just about not recommending. So they are actively looking at the problem and I believe they have enough smart people to think about that. And what I'm working on now as a follow up of what I've mentioned before are situation where you don't have a majority of reliable nodes but you have a minority of experts. It's some sort of epistocracy. So you give the power to those who know. So imagine you have the John Hopkins Medical School YouTube account, the Pasteur Institute in French medical account and YouTube account. And then you have the Hospital of Lausanne, etc. And they're producing content on say vaccine. But then you have a majority of poisoners of anti vaxxers and you might want to do something in the PageRank style. So some sort of like a page rank gradient descent where you follow the experts got you.

So you're basically taking, you want to take advantage of their expertise, which is a way of countering the fact that you have a majority of poisoners in there. So it sounds like you're almost taking a couple of tools and making a composite out of it. As we start to finish up is how can practitioners out there start to take advantage of these results that you found and the research that you've done to help better this situation we find ourselves in now where we have so much poisoning going on with so many people trying to. In search of an answer, what are some practical tips that you can offer?

I would say start by reading the literature. There's like literature on poisoning has been there before I even start doing machine learning. There was like people who started looking at that since at least 2004 and people who had made significant progress in 2012. 13 and so yeah, there is a good literature to be read. They could also like we will release a GitHub repo with the code based on the algorithms I've been promoting before. So like my colleagues will release that on GitHub so they could take it, play with it, find bugs, potential bugs in it, find new vulnerabilities we didn't see. So the space of vulnerabilities is technically limited. So you can always find new vulnerabilities or a new threat model for which our. Because you always make a threat model and maybe we overlooked another threat model and they can make progress on that. I would also advise taking data sets that might give you a sense of what a recommender system does and try to poison it just to understand how easy it is. And maybe you might find the vulnerability because now we enter in an era where you don't need to be like a classical hacker, you don't need to penetrate, you don't need to do a penetration in the servers and the system to poison a recommender system. You just need to behave like. Comment Dislike post So maybe there are still much more vulnerabilities that could be allowing people to just behave and look legit and poison, I don't know, make a movie platform, recommend the suicidal content to a depressed user. So this is something we don't want to have and I would bet that those things do not need hacking inside the servers and I don't know, finding a zero day and switching the code. I think there are like because of high dimensionality we have a paper called the Hidden Vulnerability of Distributed Learning in Byzantium. And the hidden vulnerability is basically high dimension. Today as we are making machine learning powerful, we are learning more and more high dimensional models. And these high dimensionalities give a lot of leeway, a lot of margin to attackers. So the bad news is that as machine learning is going to be high dimensional and powerful, it is also becoming very wide in the amount of leeway it gives to attackers. So I think yeah, a good starting point would like try to play with those algorithms and find eventual vulnerabilities we overlooked. And yeah, if you are a practitioner and you don't care much about the. Yeah, if you're a theoretician, I would also be very happy to hear about what we might have missed in the theoretical analysis. So maybe there's a bug in our proof and I'll be happy to learn that and work on fixing that. But if you are a practitioner and you don't care much about the theory, I would say download the GitHub repo of my colleagues and try to improve it and try to apply it on public data sets that are more relevant for recommender systems and maybe for other stuff, not only recommender systems. So I just like yeah for something like to conclude on I've been overusing recommender systems here because I think this is the most pressing example of killer robots we have today. People are not killed by, are not massively killed by self driving cars. They're more like killed with hate speech and anti vaccine. But of course poisoning will become a problem also for self driving cars. If you poison the traffic sign and then you make self driving cars learn an irrelevant model, you might start leading them into unsafe behavior. But the idea of poisoning resilience is very broad. So it doesn't apply only to recommender systems. You can think of your own problem and your own motivation and try to improve on that.

That's fantastic and we'll certainly include the GitHub repo in the show notes. But I'll tell you what, you concluded that was a strong conclusion. I mean if there's anything that makes me realize how relevant what you're talking about is even beyond social media is the fact that we have all these, we have now cars and trucks and other vehicles and, and other IoT devices that may be mobile that could be poisoned along the way and that that, that itself can present a physical danger separate from that. So it's, it's amazing how relevant what you're working on is going to be to our future. Thank you very much for coming on the show and I really appreciate you taking the time late in the conference to do this.

Thank you. You're welcome.

All right, thank you for tuning into this episode of Practically I. If you enjoyed this show, do us a favor, go on itunes, give us a us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend. Whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of rollbar. Check them out@roar.com changelog and we're hosted on Linode cloud servers. Head to lino.com changelog check them out Support this show this episode is hosted by D. Daniel Whitenack and Chris Benson. Editing is done by Tim Smith, the music is by Breakmaster Cylinder and you can find more shows just like this@changelow.com when you go there. Pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

Sa.