Exploring deep reinforcement learning

For me, AI will change everything, but it's really important to consider that it will not solve everything. For instance, we all talk about the question that AI will help a lot in healthcare, as I mentioned, but the problem is that if people can't afford to access these tools, the problem is not solved. So I think what's important to consider is that we need to improve AI, but AI must be really accessible to everyone to be a solution to human problems.

Big thanks to our partners Linode, Fastly and launchdarkly. We love Lino. They keep it fast and simple. Check them out@linode.com changelog Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by LaunchDarkly.

Get a demo at launchdarkly.com hey Jared, here. One of the things we can count on in the software industry is change. The state of the art changes so fast, in fact, that keeping up can feel like a whole other job on top of your actual job. That's why we created Changelog Weekly. It's our totally free newsletter that we drop in your inbox each and every Sunday. We link to the latest news, the best articles, and the most interesting projects that you should be aware of. We also add a little commentary from us saying why something's important, pointing you to other instances of a trend, or just making a dorky joke to keep it lively. So if you haven't yet, I recommend subscribing to Changelog Weekly and help us help you keep up with the Latest. Head to changelog.com weekly and sign up today. Again, it's totally free and we never spam you.

Yuck.

One last time. That's changelog.com weekly. Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changeog.com community and follow us on Twitter. We're Racticalai FM.

Welcome to another episode of the Practical AI Podcast. We are the podcast that tries to make artificial intelligence practical, productive and accessible to everyone. My name is Chris Benson. I am one of your co hosts and Daniel is actually out this week. He has told me I'm allowed to tell you he's down with COVID and so we are hoping Daniel feels better. Much. Very fast. But for those of you who joined us last week, I have a treat for you. I have an excellent co host. A proven co host. You already know her. Her Name is Natalie Pistunovich. Sorry Natalie. And thank you for agreeing to come back in while Daniel is struggling through Covid and co hosting with me. Welcome back.

Thanks Chris. It's always fun to be here.

Thanks. Fantastic. And we to dive right into the show here we have a guest who is from one of the leading organizations in the AI world. It's one that we always love to talk about. One of our favorite organizations and this particular guest has been doing some pretty cool stuff I know from the pre show conversation. So I would like to introduce Thomas Simonini, if I got the last name correct. Thomas.

That's correct.

Welcome. You are a developer advocate at Hugging Face, which is the. Which is the organization I was talking about working on deep Reinforcement learning. And I know you're also the founder of a deep reinforcement learning course, but I guess if you other than that little tiny tidbit that I've offered the audience already, if you would tell us a little bit about your background, how you arrived at where you're at doing deep reinforcement learning and how you got there. And a lot of people I know ask us out of the episodes kind of how did you get there? Because they want to learn from you. So if you can kind of share people share with people how you got to where you're at. Sure.

So first thing first, thanks for inviting me. It's a vast question. Initially I have a bachelor degree of law and political sciences and what happened is that I was working in a startup incubator and I was helping people who want to make startup to choose the technology behind their website because I was a full stack web developer in addition to my studies. And what happened is that I met a lot of people who were working on machine learning and I didn't know about that and I discovered this domain and after my bachelor degree I quit my studies and I started to self study deep learning initially and in 2018 I discovered deep reinforcement learning and I fell in love with this domain and at the same time I started to write articles about deep reinforcement learning and it became a course. It was not supposed it was just one article but it works really great and it became a free and open source course. And after that in 2019 I started to work at Data IQ for one year and a half where I work on implementing deep reinforcement learning agents for real world applications. Dynamic pricing for instance. Dynamic pricing is when you need to find a price. For instance if you have an airline ticket and the goal is that you find the best price to make the most benefits. And after that I quit to work in the video game industry and finally I'm working on Hugging Face to work on implementing deep reinforcement learning in the ecosystem of Hugging Face.

So I'm wondering just because you mentioned it several times at the front and I know I mentioned at the beginning as well, there are a few people out there that may have heard of deep reinforcement learning, but they don't have the background. They haven't had a chance to dive into it themselves. Could you give us a quick kind of, you know, tell us a little bit about what it is and what's involved in it, you know, especially if to differentiate it from some of the other forms of deep learning.

So to say it simply deep reinforcement learning is to learn from action. So the idea is that you have an agent that we learn by interacting with his environment. If we take a simple example, you can make an agent that we learn to play Super Mario Bros. So to play video games. And how he does that, it just try some action in the environment so going left or right and based on the rewards he gets. So for instance, if you die, it's negative, it will improve its behavior to win the game.

And Hugging Face, where you're the developer advocate now. So this is an AI community, right? And it's all open source. And so what is your mission? How do you do things?

Why so in Hugging Face, what we want to do is to democratize AI and what we call good AI through, as you mentioned, open source models and open source library. The idea, if you want, it's the GitHub for the AI models. It means it's a place where you can access a lot of trained models, very powerful and you can use it for your own problems you have. And for instance, you can use a model to translate from English to French, etc. And our goal is really to democratize the access of AI by providing a place where you can share or upload models. But it's much than that. There is the Transformer library that allows you to create Transformers. So Transformer is models, natural language processing models. You have dataset library to create rapidly dataset or to share data set and so much more. So I think, yeah.

So I have a follow up question to that and that is in my day job, because believe it or not, running the podcast isn't the primary thing that I do. This is fun, this is passion. But in my day job, I was talking to my boss about an hour ago and I told him I was interviewing you and that you're with Hugging Face as a developer advocate and he, not for the first time either, said, I just love Hugging Face. He said, that's the library I want to use every time I'm doing nlp. And he went on for several minutes about you and so about the company and stuff. But it begs the question for those who aren't intimately familiar and haven't used it, what is it, in your opinion, as a developer advocate about Hugging Face that you're doing right? So many people really love using the libraries out of Hugging Face, and it's kind of a darling in the industry. What is it the Hugging Face is doing right that maybe others. I'm asking you to give away the Secret Sauce, I guess that others would do well to emulate.

Well, I think I will take my experience because I discovered Hugging Face, I think it was eight months ago. I think it's really the fact that you can rapidly, even though you're not a specialist in nlp, you can rapidly grab a model and use it for your own needs. In two lines of code, we have something called Pipeline. It's the first thing I discover is in two lines you can, for instance, translate, you can have a model that generates conversation, et cetera. For me, the Secret Sauce is really the place where you have so much trained model and so much powerful model that you can use in two lines of code. And I will also add that it's also community because we are a very, very strong community. It means that when you have questions, people are really willing to reply. Even people outside of Hugging Face, on the forum, on the Discord and on. It really helped me when I started to train my first model to have people who are very involved and that do not work at Hugging Face to help us?

And would you say that the community of users of Hugging Face, is it more developers? Is it more people who are just general enthusiasts about AI or more data scientists? Who do you see active there?

I think it's both of them because we have a lot of researchers, but we have also a lot of enthusiasts. And for instance, there is a lot of people who worked on computer vision models, on natural language processing models, and there is a lot of people who just started in deep learning or in deep reinforcement learning would directly try models. So it's very diverse. But I think the majority of people who use Hugging Face are researcher and data scientists. But there is also a growing part of really beginners who just want to learn and to try some stuff.

So one of the things that you know and I know that Hugging Face covers a lot of different kind of Approaches in the libraries. We're hearing the last year or two especially a lot about kind of the rise of deep reinforcement learning. It's been around for a while, obviously, but it's really come into its own. We're seeing it in more and more places. And what is it about that particular approach compared to some of the other approaches that it has replaced over time that is helping that become such a kind of one of the top tier, first things you want to go to approaches these days. What's caused the rise of deep reinforcement learning?

Well, I think it's because deep reinforcement learning is able to learn. We call that a policy. So a behavior, it means choosing an action given a state is able to learn hidden behavior that you couldn't train with classical models. So I think it's the first thing. And the second thing is also because deep reinforcement learning is becoming more and more efficient. It was a big problem for a long time in deep reinforcement learning. It takes a lot and a lot of time to train compared to other models. And we are seeing that nowadays we have new models that are more and more powerful and require less and less data to train.

So one of the things I'm also wondering about is are there limitations to deep reinforcement learning that you're seeing? What are the limits? We've seen some pretty amazing capabilities from all of these different deep learning capabilities over the last few years, these different algorithms. But what are the limitations? I know that I've heard people make some kind of crazy suggestions in various areas and you're kind of like we're not there yet or something. What are some of the limitations that you might caution people and say, maybe not, or maybe you're not looking at the right solution or maybe we're just not there yet. Any thoughts around that?

Yeah, there is. For people who are doing deep reinforcement learning. They all know. Article by Irpan Alex called Deep reinforcement learning does not work yet. I think it was published in 2018. Nobody wants to talk about these articles, but it really presented every limitation of deep reinforcement learning. Limitations are, as I mentioned, sampling inefficient, but it's really changing. So sampling inefficient means that you need a lot and a lot and a lot of experience from your agent before you know it's becoming good. And the second one is the most important is the frame of transfer learning. What we simply means is that we are not able for now, when you train an agent to transfer to another agent. And the problem with that is also the problem of generalization. For instance, last Week I tried to train an agent to play Super Mario Bros. So it's a famous game and when I try to change the background, the background color of the game, the agent become very bad. So there is a big problem is really the transfer learning and generalization error. What we simply means is that for now we can consider that a lot of deep reinforcement learning agents just overfit their environment so they can't generalize. Which is why I think it's a big problem in the deep reinforcement learning community.

The Changelog is deep discussions in and around the world of software and it's been going for over a decade. We interview hackers like Chris Anderson from 3D Robotics.

At the time, drones were like predators and global hawks and military industrial and they were classified and super, you know.

10 billion dollar things. And we had just built a drone.

With Lego pieces around the dining room.

Table programmed by a nine year old and it's like, okay, that should not be possible. You know, when a nine year old can do something that is classified, that.

Literally export control as munition with Lego with toy pieces. Something important in this world has changed.

Leaders like Devin Zugel from GitHub in.

The like 10 to 15 year range or 20 year range.

What I would really like is for.

If you have like three 12 year.

Olds hanging out and one of them's.

Like, I want to be a firefighter. Another one's like I want to be a lawyer. I want one of them to say.

That I want to be an open.

Source developer and innovators like Amel Hussain.

I've yet to kind of see applications.

At scale that don't use multiple languages, that don't have just arcane stories behind why this weirdo thing exists.

You know, like, all right, when you open this file, you're going to have.

To turn around three times and tap your nose once. Like it's just the most hilarious stories, you know. But applications are living, breathing, they have craft, it's normal. So I want to normalize weirdness because that's just how applications evolve over time.

Welcome to the Changelog. Please listen to an episode from our catalog that interests you and subscribe today. We'd love to have you with us.

So the deep reinforcement learning course that you've been developing, how long have you been working on that?

It started in 2018 and the last update was during the first lockdown. So it was in 2020. And initially it was just supposed to be two articles and it became 10 articles and eight videos. I stopped updating it because I just Updated the first articles because they were the oldest, but now there is so much more educational resources. That theory is still good, but the implementation are not relevant. You can still use it. There are still people who use it, but I really advise people to look at the theory and use other imp. There is, for instance, Costa Yong, who made very good implementation with what we call the CleanRL library that you can use if you want to learn to implement from scratch, which is always a good idea to learn to implement from scratch, even though there is very good libraries. The best way to understand each architecture in deep reinforcement learning is really to implement by yourself.

Gotcha. So would the target audience be someone, it sounds like someone who probably already is comfortable writing some code and probably has a little bit of familiarity with some of the other architectures out there, or do you think it's a good one for someone who's just getting into it? How would you position somebody that you're recommending for your course?

The course is really from beginner to expert. You just need to know to program in Python. But even though you don't know About PyTorch or TensorFlow, you can learn with the course. And I think that's why the course works great, is that you don't need to be a mathematician to follow this course. Because in each chapter, what we do is that we really explain each part of the formulas. Because I think the big problem in deep reinforcement learning and in AI in general, is that most of the people who start if they don't have a big, strong mathematical background, they can be very scared when they read a paper and there is big formulas. So the idea with this course is really to be very technical, but to go it step by step, not just to start with mathematics first.

Do you think that deep reinforcement learning is a good first step into the deep learning world? It came along after Natalie and I had our first exposures and stuff to the larger framework of deep learning. But if somebody was just coming in today, maybe they're in university and they're studying, is this a good place to start? Or would you say no? No, go back to convolutional neural network or go back to other NLP things historically, start with Transformers. How would you position deep reinforcement learning compared to that array of possible starting points?

Well, I think it's better first to start with deep learning, because in deep reinforcement learning, we use neural network and also to work a little bit on convolutional neural network before starting deep reinforcement learning, because it's skills that you need to have before being able to start to learn deep reinforcement learning.

And you mentioned about you that you're building the next generation AI in games. So tell us a little bit about that. That sounds really interesting.

Yeah. So before joining Hugging Face I really wanted to work in the video game industry. And what I started to do was initially to train reinforcement learning model for small games, casual games. But what I discovered is that in this part game studio use what we call behavior tree which is much easier. It's AI made by hand where the developer defined each action to take for an agent. And then after that I started to see some demos about people who have conversations with AI, especially using OpenAI GPT3. And I was fascinating with this part because I was like, you know there is anakipnis which works a lot on the video game industry. That explains that in video games you had a lot of improvement among graphics. But in terms of interaction we still at the dawn of age. So the idea was how we can use natural language processing models to create better interaction with what we call non playable characters. So non playable characters are in games, the AI you have conversation with. And so I start to make some demo on that. We made a demo where you directly speak with NPC and they reply using OpenAI GPT3 and an AI called Replika, which is an AI that generates voices and at Hugging Face. We are currently working on a project, we don't have the title yet, it's called Murder on the Lighthouse where you need to there is a murder and you have three characters and you need to investigate who's the killer by, you know, speak with them and find contradiction in their testimony.

So that sounds good. So now that we're like into gaming, I gotta ask as you talk about it I want you to expand just a little bit on it. Just I'm taking a slight detour here but because I can. This is one of those moments I will say as a parent when I have a 9 year old daughter interested in this and she's going to want to hear this part. And so I'm kind of taking a detour to ask on behalf of my daughter. How should as you're doing, you're using deep reinforcement learning in gaming. How do you kind of think about doing that? So there are a ton of kids I know because I'm listening to their calls. You know, we're in the pandemic, they're all doing online calls. So I will hear a whole bunch of young ladies that are deeply interested in gaming talking about it. How can they Think about what they need to be doing going forward to get into things like deep reinforcement learning. They want to do it in the context of gaming. Do you have some advice, since I have hijacked the conversation so that those young ladies can start thinking about how they do this thing that that's what they really want to do. You're doing the thing that they're aspiring to.

Well, if they want to use deep reinforcement learning in games, I think so the first advice is to start to learn deep reinforcement learning. But there is also a lot of library that and I think the best is to use Unity. They have something called ML Agent that allows you to create your game and then to train your agent on the game you created. For now, in video games industry, what can be done with deep reinforcement learning is mostly testing the game. So it means having an agent that will try every part of the environment. But I'm pretty sure that in the future we'll have AI in games that will be made with deep reinforcement learning. For now it's not the case. We don't have published games who use it. But I really think that in the next year it will be the case. But there is still some drawbacks to find solutions.

So as a bit of a tease for those who are curious to try more of the course, for example, I saw that on the website, there is of course Doom run Doom on everything. So what are you doing in that course? For example, with Doom and with the games that we know.

So during the course we use a different environment. We use Sonic, we use Mario, we use Doom. And in Doom what you do there is two levels. One, it's a simple level where you are in poison place. So you lose health every time you walk. And so you need to grab the health pack to survive. So there is health pack spawn randomly. And the second one is you are in a corridor and there is some monster and you need to shoot them before going at the end of the corridor, otherwise they will kill you. And after that you have tons of other environments that you can use. Because always what we do in the chapter is that we always start with simple environment and then we show you how you can iterate a more complex one. And in thisdoom there is deathmatch environment that you can use and you can train it. But it takes much more time than these smaller environments.

So I guess at this point with a little bit of the background on kind of how to get into the gaming and how to do that, where if someone goes through and they've done your course and they've started and they've kind of, they're learning their way into it. It's their first steps. When they get to the end of the course, what kinds of things do you expect that they'll be able to do and be comfortable with? And then also in addition to that kind of where would you think they might want to go next? So they've kind of come through your course. It's the first one. They've now not only dipped their toes into this field but but have gotten some good hands on experience. What comes next, what can they do and where can they go from there?

So after the course they normally have the skill until the state of the art. So we go until proximal policy optimization, which is one of the state of the art architecture and deep reinforcement learning. So they have the skills in terms of theory and in terms of practice. The next step will depend on what they want to do. If they want to work video games, I would advise to learn to use Unity ML agent and in Unit ML Agent they have free courses on how to use this library. And on the other hand, if they want to work in research, I will advise to use spinning up reinforcement learning. So it's a course and a library made by OpenAI or to use clean reinforcement learning, the one I talk about, to dive deeper on more complex implementation.

So what are some surprising or unexpected uses that you see? Deep reinforcement learning was being used in the community.

I think there was multiple. The first was DeepMind who use reinforcement learning for their cooler system in their server bay. The other thing I think we work, when I was at the Tyco, we work on what we call dynamic pricing. And for video games. Yeah, video games, there is a lot of funny things. For instance, you can train your agent on what we call Mujoko and you can train two feet, but you train to learn to walk with only two feet. You train to make what they call the half cheetah, to run, etc. So there is a lot of funny things around it.

So you say that the thing that was in my head as you were describing that was it seems like you really have to break things down to address things in very precise ways. So as we are moving forward and this is in the deep reinforcement learning toolkit, for lack of a better word, is now part of creating games and all sorts of other things as well, obviously. Is that really what you kind of have to do is break down all of those things and think about how do I train just the feet, just the walking Might there Be a time that you foresee where you can combine those and it it's able to do very complex things or are we still a long way from that point? Are you still going to have to do everything broken down in its individual constituents for a while?

No, you can. There is for instance, environments in ML agent where you train a complete humanoid to walk, et cetera. But there is a technique that we call curriculum learning. The idea is that you start to train your agents on very simple environments and then you increase the complexity. So for instance, if you have an ensemble fighter environment, you start first to train your agent to learn to shoot snowball and then you add the enemy. This way the agent will learn step by step. So it helps the training to have that kind of techniques.

You bring some interesting perspective into those different things that are or the different usages for the deep reinforcement learning in your community. And in general, do you see that your original degree, which was not technical and was a bit of a different background, did it contribute to something? Did it help you in the way? Can you motivate people who have, who are positioned in a similar starting point as you were when you started? And how can this be a benefit?

Yeah, I think it's very nice because I think the premise that when I started I thought, you know, that it was, it was a negative thing to not have a technical background and to come from humanities background. But I think it's really important to consider that it's something positive when you start because you have other way of thinking of other way of working than people who are more technical. It's something to keep in mind and I think that it's something that is is feasible because nowadays you have access to so much free educational resources online. For instance, if you want to learn mathematics, you have MIT OpenCourseWare. If you want to learn deep learning, you have Udacity and you have a lot of MOOC online. So my advice for people who come from humanities or less technical thing is just try start and you'll see that it's really funny to study. So and the second thing is really don't feel like you don't belong here because you're not technical. You need to continue to work on and just don't think about the fact that you don't belong here because you belong here. If you do stuff, you belong here, whatever your background.

I'm so glad Natalie asked me that because your answer really got me thinking about something here and that is, I'd like to expand that a little bit, extend the line of thought that you're on. We've been historically kind of in a place where there was quite a, quite a lot of barriers to entry for people moving into deep learning. And early on, the university systems around the world are changing now to accommodate this. Most of us in the early days were coming from someplace else. It was hard for some people, more so than others. I was just a software developer and I didn't have a lot of the mathematics on that. And so I had a steep learning curve myself. But we've seen this trend over the years of more tooling. Hugging Face has been a huge part of that, that of the tooling becoming better and better. And it's kind of democratized the entire space and it's allowed a lot more people to come into it without having to jump through quite as difficult of a set of hurdles as maybe the beginning days when there wasn't so much in the way of tooling and learning resources. So do you envision a day where you can be productive along these things? Where the things like doing deep reinforcement learning for gaming and stuff becomes a low enough bar to where somebody with some level of aptitude, some interest in that can kind of dive into it, even if it's not their primary thing to where they, you know, where it's. Do you envision that it becomes that level of democratized, or do you think there'll always be a certain level that you've got to really get to, that's meaning. And when I say that meaning a bit of a challenge to get to as an entry point into it. How do you see that changing over time?

Well, I think it's clearly possible because at Hugging Face, we already have something called AutoNLP that allows you to train and to create models without coding. And I really believe that the next step in deep reinforcement learning is really that type of thing being able to just directly train the model without having to know everything about the model or how it works, etc. But I think, yeah, there is still work before that, but in the future it's clearly possible because AI will become something that we'll use as we use software. So we will not need to know how it works, we will not need to know behind the hood how it works. We just need to start the training and having some trendymos results.

And in addition to the human part of this, what are the fields that you see in the future that will be revolutionized by deep learning?

I think everything, especially the industry with the robotics, we'll see more and more robotics And I think on the one hand is a good thing because it will remove a lot of work that are very hard and that really underpaid. So it's a good thing. But also I think in health, we already see application in health with cancer detection. We see also with dmine, who works on the question of protein detection. So for me, AI will change everything, but it's really important to consider that it will not solve everything. For instance, we all talk about the question that AI will help a lot in healthcare, as I mentioned, but the problem is that if people can't afford to access these tools, the problem is not solved. So I think what's important to consider is that we need to improve AI, but AI must be really accessible to everyone to be a solution to human problems.

That's a great point you make there. And we find ourselves in different aspects talking about pieces of that all the time in terms of who has access to the technology. And some of us who are privileged to be in this field, maybe we get stuck in our bubble at times and there's, there's robots are starting to come around here and it's right around the corner and we're thinking about this. But there's kind of huge, huge segment of the world out there who hasn't really had access to this yet. So given that, you know, even with the best work from people like you and Hugging Face and others out there, we definitely have challenges of equity in terms of people having access to good solutions for their life. Do you have any thoughts on and things that we can do? Because I know, I think it needs to be forefront of our mind, things that we can do that are in this field to try to steer these solutions toward those places and people that might need a Morse than we do in a lot of ways that don't have all the benefits that some of us are enjoying right now. Any, any thoughts on how we find that path? It seems really hard to me.

Well, I think there is a lot of positing things that happening, especially because of open education. You know, the majority of people who look at MOOC are people from developing countries. So the new generation, especially in countries, non developing countries, are more and more educated. So I think if we want that AI will be accessible for everyone, we need to make sure that our models are open source, that everyone can participate in the documentation, participate in the project. And the second thing is more, I think a politician thing because we will need laws to force companies to make their AI accessible, to make their AI that we can analyze what's inside the AI to avoid bias and all that problems. So I think it's too part for us. So for people who work in AI, we really need to work on the question of making educational content free or at least very cheap and working on open source projects and try to share as maximum as possible. And on the oversight, we need to push our politicians to work on legislation to make sure that AI will be here for good and not for negative stuff.

Yep, that makes a lot of sense. I absolutely agree. Octoberfest is also time of the year where everybody who wants to start with open source, I always like recommending contributing to the docs. Like you say this is a wonderful way to participate and and it has less of a threatening entry threshold. So I could absolutely not agree more. So the future scientist who is listening to this episode today, what is the first step that they should take in this field?

I think the first step will be to just start with an introduction to deep learning and then to try different projects just to see what you prefer. If you want to dive more on nlp, so natural language processing, on deep reinforcement learning or on computer vision, audio, et cetera. I think the idea is not really to overthink this part because most of people that I met want to start always get stuck because there is a wide range of possibilities and the only thing is just type on Google what you want to start once and start to learn deep learning. Take the first link, see how it wails, how it goes and then iterate from that. But yeah, don't overthink because most of people overthink and just say, oh, this is maybe not the best resources, this is not maybe what I want to do, et cetera. Just try and you'll see.

So that future scientist has taken that first step. They are diving into this and doing exactly what you just said along the way. They're taking your course and they are learning these skills and they're in a world that is increasingly, you know, having this, this just amazing gamified experience that is now becoming available widely and not just, you know, in the context of video games, but but across all of life. And simultaneously, deep reinforcement learning is driving robotics, which are becoming ever more common as these new scientists are growing up. I'm thinking obviously of my daughter in this context, as I'm sure you figured out. So, so what, as they grow up and they are adults now and they've come through this process, what do you think the world looks like? What do you think maybe some opportunities are challenges that they might face and along the way, I would love your opinion about what you'd like to see as opposed to what you think will be there if you have any preferences about what you're aspiring for right now as a data scientist doing this kind of work.

Well, I think yeah, in the future, clearly we will have much more robots than nowadays. They will do a lot of tasks, especially social tasks, in terms of helping older people with the question of autonomy. We will have autonomous cars, all of that question. What I really don't want to see especially is the question of using these technologies for military things because we see more and more the question of especially deep reinforcement like learning. We see more and more people talking about using deep reinforcement learning for robots during battles, on that kind of stuff. And I think this is the worst thing to do. I think it's something that's unfortunately going to happen because if someone does it, all the other country wants to do it to protect themselves. But it's really something bad because it's really something we don't want to do and we shouldn't do. But unfortunately, I think we don't have a lot of power on that. The thing is, I want to be positive is when you work in this industry, we always need to think about the consequences of what we do. Sometimes it's very small consequences, but when we work on some models, we can induce bias that can be negative for a group of people and it can be much worse than that if we speak about robots in army. But it's always important when we work on something to think on the consequences because sometimes we are too focused on the project and we don't think about the outcomes of what we are doing.

Well, Thomas, thank you very, very much for coming onto the show today. Natalie, thank you very much for co hosting with me. This has been a lot of fun. It's been a great conversation and really appreciate you. I hope you will come back sometimes time and tell us kind of what you've done next in deep reinforcement, learning and other things.

Sure. Thanks.

Thanks, Thomas.

All right, that's Practical AI for this week. Thanks for listening. If this is your first time with us, subscribe now at PracticalAI FM or simply search for Practical AI in your favorite podcast app. We're in there. And if you're a longtime listener, do us a solid by recommending the show to a friend. Word of mouth is still the number one way people find new podcasts they love. Special thanks to our partners for supporting our work fastly. Launchdarkly and linode. We appreciate it and to the mysterious Brake Master Cylinder for cranking out new beats for us all the time.

That's all for now.

We'll talk to you next week.

Sam.