Robot hands solving Rubik's cubes

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by Digital Ocean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive starting at 40 bucks a month. Learn more and get started for free with a 100 credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're PracticalAI FM and now onto the show.

Welcome to another fully connected episode of Practical AI. In these fully connected episodes, my co host Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss some of the latest AI news and we'll dig into some learning resources to help you level up your machine learning and AI game. I'm joined today as always by my co host Chris Benson, who is a principal AI strategist at Lockheed Martin. And I'm Daniel Whitenack, a data scientist at SIL International. How you doing Chris?

I'm doing great. How's it going today Daniel?

It is going great. It is a nice fall day here and enjoying the like two weeks of perfect weather before it's bitterly cold in the Midwest. Yeah, trying to enjoy that while I can.

See when it gets bitterly cold you can come down here. I think we're actually going to see each other in Chattanooga.

Yeah, I'm excited about that.

Yeah.

We'll both be at the Project Voice Conference which is happening in Chattanooga in January. So come by and say hi to us if you're around to talk to you about AI and voice and all of those things and we can talk.

About our Nice mild Southern winters here.

Sounds good. Now on to, I guess some AI news, as this is a fully connected episode. I don't know about you, Chris, but for like the past week or however long it's been, about half of my Twitter feed has been like giphyse and videos of robot hands manipulating Rubik's cubes. Has it been the same for you?

It has. It's been kind of the big topic as we're recording this over the last couple of weeks. It's been a really interesting thing. And diving into it, as we have both done in anticipation of this talk, has been a lot of fun. So looking forward to today.

Yeah, well, maybe to start out with, we should probably get a baseline. How good are you at solving Rubik's cubes, Chris?

So not so good by myself. So if I was going to do it, like for my daughter, I would cheat and look at one of those algorithms. You can see these manual algorithms.

Your daughter might beat you.

Yeah, she probably would, actually. I would have to. Daddy would have to go and cheat and look up. Because there are solutions out there, you know.

Right. Methods and such.

Yeah, but, but no, no, for real. I tried it when I was younger. I haven't tried it recently and utterly failed. So pathetic.

I'm not good at solving Rubik's cubes. I will admit that I've had some friends try to teach me various things. I remember one in particular in college. He was one of those just brilliant, naturally brilliant guys. It was one of my friends in, in the physics program. And when I would ride with him from his house to school, he would always try to solve. So he would drive with one hand and solve a Rubik's cube with the other hand, one handed while he drove, which I'm not recommending at all.

Oh, doesn't sound like a good idea.

Yeah, it was about like a five minute drive. And he could solve the Rubik's cube with, with his one hand, which kind of disgusted me a little bit and was probably not safe in any way, but.

So my question to you is, had he looked at one of those, you know, algorithms that you can manually do it? Because I know there's one that I have seen before, I have not memorized it or anything, but you can solve any Rubik's cube in a certain number of very small steps. I mean, it's less than 20 that you would do.

Yeah, I mean, it must have been something like that. Eventually he went on to like the five by five. So there's like three by threes and then he went on to like the five by fives, which I can't even comprehend working on. But yeah, that's a little bit too much for me. But maybe not so for OpenAI's Rubik's Cube solving robot hand, which is going to be what we're going to talk about today, they solved the three by three, maybe they'll go on to the five by five. But yeah, just to clarify, you know what we were talking about in terms of the robot hand solving the Rubik's Cube. If you weren't aware, OpenAI, a research team there just published a blog post and a paper, at least a pre print of a paper about using a robotic hand to solve a Rubik's cube and it's pretty impressive. What were your initial reactions, Chris?

Correct me if I'm wrong. If I recall, they did use one of the standard algorithms for the Rubik's Cube itself, I think. But I believe that most of the training was focused on the articulation of the hand itself. Am I right or am I wrong?

Yeah, so there was definitely some pushback against the results in particular maybe as related to that and we can get into that in a bit. But yeah, I think the main point was like if you just imagine manipulating a Rubik's cube with one of your hands and solving it, even for a human like that, manipulation is not trivial. And so the idea that they could get a robotic hand to manipulate this Rubik's cube towards the solution, even if they are using this sort of other algorithm for the solution, is pretty astounding because it's a really hard manipulation problem, I guess, is the idea.

I agree. And I think, I mean it's almost whether or not you're solving the Rubik's Cube is almost an aside in this. I think the fascinating thing for me at least was seeing this robotic hand and there's just one. So you can't, you know, you, you don't have both hands to put it in, but the robot is both manipulating the cube in the appropriate directions and using the robotic fingers to pivot the cube. And the dexterity required to do that is something that I certainly would be challenged myself to try to recreate with my human hand at all. So it was just the ability for the robot to be able to act with that kind of dexterity was just impressive to watch the video that they had on the, on the site, not.

Only that they kind of solved it once, but they did a whole bunch of different interesting things with perturbations. Too. So this is kind of similar maybe to those Boston Dynamics videos and like the memes that are also, like, associated with these Boston Dynamics videos, where they kind of like hit robots with hockey sticks and things, but they hit the robotic hand with like a stuffed giraffe and pushed it with a pen and threw a blanket over the hand and other things. And we're testing the adaptation of the hand to be able to continue towards the solution and continue manipulating the Rubik's Cube in light of these things that weren't part of the training data, but were perturbations that it had not been exposed to before. And I think that that also is part of the big story here, is that the adaptability of the solution, I guess.

I agree they added a lot of variability, which I know we're about to talk toward in this conversation. And there's an entire algorithm that's designed around that, which we'll discuss in a moment. But it was just, it was an impressive bit of engineering, both from a data science standpoint, but also, as I watched that video, I just kept thinking of all the applications and not even necessarily robotic hands, but if you start thinking of other manipulation tools that you might have on a robot and applying the same principles, I'm pretty excited about it.

So congrats to the OpenAI team. If any of you are listening. I hope maybe one or two of you will come across this podcast. Congrats and great work. So maybe what we could do now, Chris, I think would be good for me, and probably the listeners too, is to just kind of break down how OpenAI went about this solution, what they used in terms of neural networks and setting up the solution and training and simulation and all of those things to just kind of break down the main parts of this study. Does that sound okay?

That sounds like a good idea.

So I went to the OpenAI blog post about this research, which we'll of course link in our show notes and kind of the first sentence that they have there is, we've trained a pair of neural networks to solve the Rubik's Cube with a human like robot hand. The neural networks are trained entirely in simulation, using the same reinforcement learning code as OpenAI5, paired with a new technique called automatic domain randomization. The system can handle situations it never saw during training, such as being prodded by a stuffed giraffe. That summarization there is just like packed full of all sorts of things that I would love to break down. Maybe not so much the stuff draft, although I was intrigued by that as well. And later on down, they also have another good summarization. They say we train neural networks to solve the Rubik's Cube and simulation using reinforcement learning and co Siemba's algorithm. I'm probably butchering that name. I'm sorry, that's how I would have said it. And then they also talk about domain randomization. So some of these things I'm more familiar with than others. And there's some new things here as well. So maybe we can start by just digging into these pair of neural networks that they're talking about. Were you able to kind of deduce from their blog post or their paper anything about this pair of neural networks? What pair of neural networks are these?

Yeah, for the computer vision side, they used three cameras to address the aspects of the Rubik's Cube. And then for the manipulation, they had a reinforcement learning policy. And so they had the control policy network that was addressing that. And so both of those networks had to be trained to be able to work together to be able to manipulate the Rubik's Cube and understand where they were statewise at any given moment.

So the two networks, I guess then would be this control policy network and then the convolutional neural network, which has to do with estimating the position of the cube in the hand. So the control policy network in terms of input and output, what I was getting from the paper is basically the input of this control policy network would be the positionings, kind of noisy positionings of the Rubik's Cube and the robot hand. So they were actually either from the convolutional neural network or from sensors in the cube and the hand. They had these positionings of the cube and the robot hand. So you have these positionings, basically the state of how the cube is in the hand as input. And then the output would be, what action does the robot hand need to take in particular, that brings the state of those things closer to solving the cube. So in comes these what they're calling observations, out comes actions. Now, I guess for those observations, they need some way to estimate the positioning of the cube. And is that where the convolutional neural network comes in?

Yeah, I think they were using the convolutional neural network for the three cameras that were observing from different positions, the various aspects of the Rubik's Cube. And then I do recall also that they mentioned that the robot's hand was also tied to a 3D positioning capability that they had, so they'd understand where the robot was at Any given moment in relation to how the cube was to combine those two so that it could do the policy based manipulation.

Yeah, it was a combination of, you know, the detected position of the cube, the, the tracked position of the robot hand, fingers, or hand in general. And then they also were able to kind of track the faces of the cube by instrumenting the cube as well. So there's definitely a lot of instrumentation going on here, but essentially all of that is considered observations which is passed into this control policy network to produce what action should the hand do. And my understanding is that these actions are essentially the sort of desired angles that they want to manipulate the cube into or the faces into. I'm not sure what the dimensionality of that is because there's a lot of these faces of the Rubik's Cube. So all of the observations come in and the outcomes, the actions you mentioned that like the control policy had to do with reinforcement learning, the convolutional neural network for the position detection was separate. As a reminder, kind of what's the difference in training the control policy via reinforcement learning versus the convolutional neural network in another way?

So the convolutional neural network is when you're training it, you're trying to recognize certain patterns. And so obviously the patterns here are the various faces of the Rubik's Cube and the different colored faces on each of those cubes and how they're working together. And so recognizing what that is at any given moment. So for cube state is what the cnn, the convolutional neural network, would have been doing. Reinforcement learning, though, is the other side, the control policy side, where the network has to figure out with this articulated hand. And so it has many different joints and motors that are driving it at different angles. And they have to work in unison to be able to, to work with what it's determined to be the state of the Rubik's Cube at any moment in time. So it's not just one motion, it's a whole collection of motions. And it does that through essentially trial and error that the algorithm puts it through to learn the process. And so you could think of the CNN as perception oriented, and you could think of the reinforcement learning as what do I do next with my articulated hand to achieve the next state that I'm trying to get to? So it takes them both, if I also recall, just to throw out one extra thing. They had a type of recurrent neural network called an lstm Long Short Term memory, which takes into account time the time sequence between the different observations and the policy based movements doing. You have to have a sequence for that to make any sense because obviously a robotic hand that's trying to do a task has a series of moves it has to do. So it actually takes all three of those to work together to be able to apply itself to this problem.

The reinforcement learning, we won't do a huge deep dive into that because we actually have a couple of other episodes on that already. I would recommend first off taking a look at episode 14 from Wojciech Zaremba actually at OpenAI. So he was on the team that helped do this research or did this research. And he goes a little bit more into reinforcement learning in in that episode and kind of compares reinforcement learning to like semi supervision. So giving the the network a sort of treat, like giving a dog a treat or a reward as it performs actions that get closer to a solution versus the convolutional neural network which is just supervised. So you have input images that have a labeled position and you're trying to predict that position in a sort of supervised manner. So we'll link our other episodes about reinforcement learning in the show. Notes.

Correct. I guess to summarize that up, the algorithm has what's called an agent which takes actions to act upon the environment it's in and is rewarded or not depending on the outcome of that. And then adjustments are made based on what happened and it goes around and around in that process.

What is up Practically I listeners, we're working with Infinite Red to promote their free AI mini course. It's called AI Demystified. Learn more enroll at learn AI.infinite.red. this free five day mini course is a great introduction to the most important concepts, types and business applications for AI and machine learning. Each day of the course includes a lesson, a quiz and an assignment to submit your learning. And after you've completed the course, you'll also get a certificate of completion for your LinkedIn profile or for your portfolio. If you've been feeling lost in the world of AI and hearing lots of buzzwords, then by the end of this mini course you'll be able to speak intelligently about AI and machine learning and their practical business applications. Again, this course is completely free. Learn more and enroll at learnai.infinite.red Again, learn AI.infinite.red.

All right, so they show some great pictures of their experimental setup, like in the physical world and also in simulation and particularly in that summarization that we talked about initially when we got into this, they talk about training in simulation. And I know that you've had some experience in the robotics world before and have run across different scenarios. Why is simulation important, particularly in robotics?

Because probably the biggest reason, and it's really, really, really common in robotics to start in simulation to the point where it's probably. That's the case in almost all cases. So in simulation, you can essentially create the circumstances that you're operating in. You're essentially able to simulate the data that applies to the situation and you can have the environment created. And so if you don't have a whole bunch of the ability to apply real world data upfront, that may not have been created yet, and you don't have an environment that lends itself to testing. You know, just as a kind of an obvious example, in my own job at Lockheed Martin, one of the things we build are airplanes. And so if you're trying to figure out how an airplane will fly around, around, and you haven't yet built the airplane, it's a real problem because, you know, you have to do it in a simulated world where you can try different things out. And that lends itself to the engineering. And so if you kind of take that example and you push it out in the world, you can apply that in a lot of different ways. So it's a much more inexpensive way to start your training out. And depending on your situation, you may be able to do almost all of your training in simulation, which it sounds like they did in this case.

Yeah, in your example, like the airplane example, like it's inexpensive, I guess, because you don't have to manufacture 2000 aircraft with different variations and then see, you know, which 10 of them don't crash. And then you utilize those as what you design off of.

Right.

You're able to kind of go through a lot of different scenarios, a lot of different configurations in a simulated sense. In the same way, I think OpenAI like they don't need a thousand robot hands to be manipulating cubes and getting the data off of those to create a kind of vast training data set. What they do is, you know, they create the simulated world using, I think, a lot of technology from like gaming and image rendering and all of that. So like Unity 3D. So they create this virtual hand, virtual cube and position it in various ways to recreate as real of imagery and as real as position data as they can without actually having to position a real robot hand into those positions and gather the data off of the hand, which I imagine would take an enormous amount of time and effort. So Yeah, I mean, you were kind of getting at the engineering side of things. And even though they aren't engineering a bunch of different hands to collect this data, it seems like this simulation task in itself is actually a pretty big engineering burden.

It is, but simulation has been around for a long time. And I know we talked about it, we had our episode where we discussed high performance computing, hpc and we talked a bit about, in that conversation about the simulation world and the requirements and, and it's been around for decades. And so gaming, as you pointed out, also does that. There's a lot of industries that utilize it, automotive. And so trying to get the physics of what you're working on and all the environmental considerations is at least something. It is complex, but it's something that is somewhat known. And it's much less expensive to do it in that simulated world than to do what you just described. The kind of the ludicrous. Create a whole bunch of versions of the same thing and figure out which one works and which one doesn't. Which would be pretty prohibitively expensive.

Right. And could be dangerous, depending on what you're trying to simulate, I guess. Absolutely, in the airplane case. But going back to the Rubik's Cube, I'm thinking back to my friend in my physics class who was brilliant and I was in a mechanics class with him where we had to do these simulations of real life mechanics. So classical mechanics like pendulums and all this stuff and, and more complicated things that are fairly chaotic, like, you know, multi weight pendulums with bends in them and all of these things. And I remember that that was extremely challenging. And one of the things that they note in the paper is that it is challenging. So you have to in some sense use simulated data for these types of problems in robotics. But it is challenging to create simulated environments that capture all of the physics of the real world. And this is where this sort of domain randomization stuff comes in. So they bring up this idea of domain randomization, which is actually a previously introduced idea where in a created environment for a simulation, you introduce some amount of randomness in the renderings of that simulator, or some variability in the simulator based on randomness that you're basically trying to simulate, that there's going to be variations in the environment that it sees when you transfer the model that you're training from the simulated world into the real world. Because that transfer can be rather difficult. But OpenAI took this a step further, that's my understanding.

Yeah, they created this approach which they called automatic domain randomization, where they systematically created that randomization as part of their training process. And, you know, it was done in simulation, as we've been discussing. And it was interesting in that it was a technique that could increase the ability for the control policy to be able to generalize to the environment that it's in. If they had not done that, for instance, and the articulated hand, robotic hand, had been maneuvering the Rubik's Cube around, and any kind of interference was introduced, you know, going back to your stuffed giraffe comment a little while ago, you know, that could completely throw it off. But if as part of the training process, you are constantly introducing different types of interference in all sorts of different ways, and as part of its reinforcement learning process for its control policy, it has to learn to cope with each of those forms of interference, then it is better able to generalize once you've completed learning down the road. I was fascinated reading through their white paper on how they approach that. I think it's a great next step.

Yeah. And one of the things I think that they were interested in is this idea of handling situations that weren't seen during training. So the idea is, okay, how do we extend domain randomization in a way that creates an even more robust model that will be able to adapt quickly and return to a sort of baseline performance even when things happen that we don't anticipate? And so one of the key pieces of how they extended the domain randomization was that they didn't just randomize the simulation, they randomized it in increasingly difficult ways. So one of the things that they mentioned in their blog post is like, they would increase the size or decrease the size of the simulated Rubik's Cube more and more or less and less as the training went on. So whenever the neural network achieved a certain level of performance, then they would throw something at it even harder. Like they would change the size of the Rubik's Cube even more and kind of continually push it to have to adapt more and more to harder and harder things. And in this way, it kind of. They created what they're kind of calling or what they're theorizing is something that leads to what they're calling emergent meta learning. So if you remember, we had this episode with Cheryl Chen from Google where we talked about AutoML and meta learning, which are kind of related. Maybe those were in. We talked about them in a slightly different context with. With Cheryl, but here they're talking about emergent meta learning, which sounds like this really weird term to me, and it's almost Like a term that doesn't mean anything. It's like emergent and it's meta.

Very new age sounding there.

Yeah. What does that even mean? I'm not sure. So what do you get, if anything, from. From that?

Well, I actually drew an analogy between what they were doing with that and kind we as humans do, in the sense of as they kept cranking up the difficulty by changing the parameters into something more difficult. It reminded me, as I read that about, for instance, teaching my daughter to ride a bike and first just learning how to sit on it and pedal with training wheels on and start steering it, and then as she got comfortable with that and going over curbs and then taking the training wheels off and having to learn how to do balance and all that, or even myself, I would like to learn how to juggle just for fun. And recently I picked up some balls and I started with just one. And as I started mastering that, I ramped up the difficulty into two. And I'm not to three yet, so. But I think their approach really reminded me of how humans learn along the way. And I thought that was striking since that doesn't always. As much as we like to draw analogy between different AI approaches and humans and brain and such, that doesn't always reflect the reality.

It's really interesting. First off, little piece of trivia. I was the unpopular kid in high school who was a juggling troop. Next time we're together, we'll have to experiment with that. But I do like the analogy that that gives, because in juggling or some activity like that that a human learns, each time you add, like, let's say you add an element that you're juggling, like a new ring or ball or club or whatever it is, it doesn't necessarily take you an exponentially longer time to adapt to that. That new situation, but you've kind of learned how to. You've learned how to learn a new juggling trick, and so you kind of just kind of bolt it onto your juggling toolkit. It doesn't always carry through like that. But here with meta learning, you know, meta learning itself just means kind of learning to learn, right? And so what they're saying is that by introducing this element of increasingly difficult domain randomization into the training, then what's emerging from that process is a network that is able to learn, to learn, as new perturbations come in, like this stuffed giraffe or. Or whatever it is. So I like your illustration very much. The last thing that they mentioned, we can just mention really quickly in their paper that maybe needs some definition is this Cosiemba algorithm, as you mentioned, the solving of the Rubik's Cube. In other words, how do I need to move the faces of the cube to move towards a solution of the cube that was actually not a neural network that was deciding those operations. So, like, the raw operations, like move this block to this position, move this face, rotate it one direction. That was not a neural net that was deciding that. Right?

Correct. Yeah, that was. They're using an algorithm, and this is one of several that I believe exist where you follow a known sequence of moves and rotations and it will eventually get you where you're trying to get to. So, kind of going back to what we said in the very beginning, they were really focused on understanding the state that they were in and articulating the robotic hand to achieve the next state, which this algorithm, which was an already known thing based on this algorithm.

Well, talking about the fact that they use the AI models to manipulate the robot hand, but not actually, quote, unquote, solve the Rubik's Cube, that and a few other things created a bit of pushback in terms of the community's reception to that and various people saying strong things. I saw a couple of tweets in this regard about, you know, how like, this algorithm that they're using to solve the Rubik's Cube, it's been around for 17 years and it's symbolic, it's not like a neural network work.

So they were missing the point, in my view.

Yeah. And they, they also talked about, oh, well, like, the Rubik's Cube is all instrumented with LEDs and sensors and all of these things. It's not just like I threw, for example, I couldn't go into the store and buy a Rubik's Cube and put it in the hand of this robot and then have it do the manipulation, because their cube was a special cube or these sorts of things. So this was kind of the pushback. Was that in their blog post? So if I, if I go back to the blog post here, their kind of headline is Solving Rubik's Cube with a Robotic hand. And they say, trained a pair of neural networks to solve the Rubik's Cube. So the pushback was like, well, did they really do that?

So in my view, OpenAI's mistake here was not in what they actually did. It was in the marketing of the blog post and how they titled it. I think what they did was awesome and a lot more important. If you used a neural network to solve A Rubik's Cube, that might be a cool little project to do, but it's very, very specific and it's not. You know, maybe I'm wrong, but I don't see the applicability off the top of my head to generalize on that. And so what they were focusing on was how do you create the control policy for that articulated hand and learning how to do that better? Because in my view, that is much, much more valuable because you can use that in so many different use cases out there. And if OpenAI's purpose is to kind of help the rest of us get to the next level, I think that they did exactly the right thing. I think they focused on the right thing. So they should have worded the blog post better, probably. But other than that, I personally am very happy to see them do this work.

And I think that even the strongest critics, at least in looking through Twitter, the Twitterverse and interwebs on various things that I found even the strongest critics, I don't think characterize what they did as not impressive, but maybe that it was mischaracterized, as you're saying. So, yeah, I think that there is. And that kind of got me going down a train of thought. You know, why did people react in such a way, like this sort of way to this article with such vigor? I started thinking like, you know, back a few years ago or whenever it would be, would people have reacted the same way? And why are people so sensitive to AI hype? Do you have any thoughts on. On that?

Well, I think. I think we're all subjected to quite a lot of AI hype. And I know we've discussed that on numerous episodes in the past, which is one of the reasons, as an aside, that you and I decided to make this practical AI practical, productive and accessible to everyone, because we were, to some extent, even at the very start, recognizing the overwhelming amount of hype that was then and continues to this day and probably will for a long time to come. So I get that people that the message needs to be more specific to the actual amazing work that was done. And so they need to be careful how they're putting their work out there in terms of what the actual impressive thing is that we should focus on. But I get that people feel a little bit burned by the hype, but also sometimes recognize we're all human, we all make mistakes and focus on where the good is on this. I would ask people to take a glass half full approach.

Yeah.

And you know, let's look at this and go wow. And now this is something that we can all use going forward and not focus on whether somebody could have done a better job on a title.

Yeah, it is kind of the situation where we're in, I guess. You know, this probably relates to some of the discussion around like cancel culture and all of that stuff. And I don't, I don't really want to get into the politics and views on that in this episode, but we, I think, have been burned quite a bit by people just, you know, saying all this stuff about AI or blockchain or quantum this or whatever when it hasn't really proved true. And I think we do need to be careful about this in particular as related to certain things. There's a lot of talk about bias now in AI, and I think that's probably a of part point that, you know, is not being hyped enough, that we're not concerned about that enough. At least that's my personal opinion. I saw some tweets saying that Eric Schmidt of Google, the chairman, was saying some things about, oh, we don't need to yell that much about bias now in AI, and I think, you know, we need to be practical in how we approach these things. But there's a lot to sift through out there when you start looking up terms like AI and quantum and other things. Even just this last week, I don't know if you were following, Google made the announcement about finally achieving quantum supremacy.

Oh, yeah, that made me cringe.

Yeah. They said this thing would have taken a classical computer 10,000 years to solve. And then IBM kind of worked on it a bit and solved it in two and a half days with a classical computer. So in some sense, I don't know, this is part of the scientific process in the sense that if you claim something, people should be able to, you know, test it and see if you're really, what you're claiming is true or false. It's maybe the difference now is that that testing is like, it's so much more public now. Used to that would happen in like private peer review and all of that. Now it's kind of happening on Twitter and all over the Internet. So I'm not sure. I'm still kind of parsing through whether parts of that are all good or bad.

I think the fundamental issue there is with the acceleration process that you just described and the fact that you don't have something, you don't produce a result that gets peer reviewed over a period of weeks or months and then kind of eventually goes out to the larger world. At this point, AI and these other fields that we're talking about are happening so fast that you kind of have to get your stuff out there as an organization before somebody is likely to steal your thunder and maybe reveal their results that were very similar to yours, but before you get there. And so, especially with so much of this work being done by industry, I think what I'm about to say holds true even for academia. There's a marketing desire to say, hey, look at this great work we did, because people are looking for customers, people are looking for funding. And so there is definitely a marketing consideration in doing this advanced scientific work. But the marketing and the science, you need to find the best balance you can and not let your marketing overreach your science, or you're going to get a lot of backlash on that. So I think it's tough to be in an organization these days where you're trying to do amazing work and have it validated, but also get the benefit of that work in other domains, such as continued funding and continued customers.

Yeah. So I guess one of the best responses to some of the criticism that I saw on the OpenAI stuff was like, hey, look, we shouldn't be solving Rubik's Cubes with neural networks, because we don't have to. And we've had the solution for a long time and it works fine, but we don't have the solution to this sort of robotic manipulation, which is actually maybe easier for humans, but really hard for computers and AI systems. And so what they tackled was the harder of the problems. And there's certainly other cases of this that, you know, are present in AI. Like, you know, detecting and working with sarcasm in text is extremely difficult. Like, or even just sentiment is still a hard problem, just because it's easy for humans to maybe detect that. And like, even, like, combine text with facial or voice with facial expressions and all of that to detect sarcasm or volume or tone or all of those things. And that's really, really tough for a computer to solve.

If you think about not having facial expression, the way we type out emails or texts, for instance, we humans also have a hard time detecting sarcasm when it's there, when it's not. If we don't have other sources other than the text alone.

Yeah. Which is why I think a lot of people in that space are talking about this sort of multimodal learning where you have maybe text and video or imagery and that sort of thing too, where these people solving those types of problems, maybe sentiment analysis is something where we're like, oh, well, humans can do that easily. Why do we need to work on computer algorithms to solve this sort of thing? It's part of this effort to really tackle the hard problems that are hard for computers and advance AI more generally versus just kind of solving already solved things just quicker or faster. That sort of train.

If you have an option that is less expensive to do it, then take that option. Don't take the AI route. It's, you know, apply AI where it makes sense to apply AI.

Amen. All right, well, on this front, we always like to end these fully connected episodes with some learning resources. And in particular, if you're wanting to learn more about reinforcement learning. That was a confusing statement. Very meta learning. Learning about reinforcement Learning. Again, episodes 40 and 14 of this podcast you might want to take a listen to, but OpenAI uses this OpenAI gem to train their reinforcement learning algorithm, and that is something that you can get your hands on. There's a tutorial that we'll link in our show notes that kind of shows how to use this OpenAI gem. So if that's something you're interested in exploring, then we'll certainly link it in the notes. And There's PyTorch and TensorFlow tutorials as well that deal with reinforcement learning. So there's a bunch of ways that you can get hands on with this sort of thing, even if you don't have a robotic hand handy. There's my pun.

Your hand handy. Yeah, and I don't.

I don't either. Cool. Well, thanks for going through this discussion with me, Chris. It was a lot of fun.

It was interesting article.

Very interesting in all sorts of different ways, technically, culturally and reception wise and all of those things. So, yeah, definitely go take a look at the blog post and the paper if you haven't yet and let us know what you think you can reach out to us and in our Slack channel, which you can find if you go to changelog.com community or reach out to us on our LinkedIn page, we're happy to get good feedback, whether that's positive or negative things, but also to hear what your thoughts are on potentially controversial and interesting topics like this. So reach out and. Yeah, I think that's all we've got for today. It's great to talk to you, Chris.

It was a good one. Talk to you soon, Daniel.

All right, thank you for tuning in to this episode of Practically I. If you enjoyed this show, do us a favor, go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network. Share a link with a friend. Whatever you gotta do. Share the show with a friend if you enjoyed it and bandwidth for Changelog is provided by Fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of Rollbar. Check them out@robot.com changelog and we're hosted on Linode Cloud Servers. Head to linode.com changelog check them out. Support this show this Episode episode is hosted by Daniel White Knack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@change law.com when you go there, Pop in your email address. Get our weekly email keeping you up to date with the news and podcast for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.