AutoML and AI at Google

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very Competitive starting at 40 bucks a month. Learn more and get started for free with a 100 of credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practicalai FM. And now onto the show.

Welcome to Practical AI. I'm Daniel Whitenack, a data scientist with SIL International and I'm here with my co host Chris Benson who is a Principal AI strategist at Lockheed Martin. How you doing Chris?

Doing great. How's it going Daniel?

It's going really good. I spent some vacation today, time doing a little bit of backpacking in in Minnesota. So just about the opposite from my day to day life sitting in front of my computer screen. So it was, it was a really nice disconnected time. So happy to, happy to have that time. Do you, you have any, any good times over Labor Day?

Yeah, I had a good time with the family over Labor Day. We went to a local lake in the Atlanta area and did a little kind of a beach thing which was nice. And I too am looking forward to a couple of weeks of vacation coming up. I'll be in the uk. My wife is British so we spend a lot of time over there and other than that, lots and lots of business travel, conferences, things like that.

Yeah, definitely you're keeping busy for sure. Well today we have Cheryl Chen on the show. She is an ML developer at Google. I met her recently over the summer. She's doing some really awesome things and has done some really quite a bit of different things while she's been at Google. But really happy to have her on the show today to talk about some of the things that she's doing with, with Google Cloud and some. Some other AI things. So welcome Cheryl.

Hey, nice to meet you.

Yeah. Well could you just give us maybe just a little bit of background about, you know, how you got into computers and eventually AI and programming and ended up at Google and doing machine learning and all of these things.

Yeah, of course. So yeah. So currently I work at Google like you said and my primary area of focus is auto machine learning, which has been this up and coming practice in machine learning where you're basically given the label data and at the end of it you get a predictive model without having to really do very much machine learning in the process. And what brought me here, I guess like that story starts with my first computer which I would actually say was the Nintendo, which kind of dates me, I guess.

Nice. Yeah, it's like the original Nintendo.

The original Nintendo was like my first, you know, gateway into technology. And I just remember I have a lot of memories of just playing hours upon hours of games and being captivated by these virtual worlds, these pixel, very pixelated virtual worlds. And not only was it just wondering how it was possible that we could have the input output, all that feedback coming out of my tv, but thinking and dreaming of how I could make it better. And so naturally I went to a high school that happened to have computer science. And I remember asking, what is computer science? I was in a classroom eventually of four or five students that happened to have a particular interest. It wasn't a very widely popular subject area. Then of course, naturally you go to college. And I think I'd forgotten that I loved video games at some point or that that was kind of my motivator. And so at some point I think it was like, and I was late to playing this game, but Final Fantasy vii.

Nice.

Really got me back into thinking about these worlds and the stories that they told. So then I signed up for some summer research and I did a research project looking at artificial intelligence and storytelling which then led me to UC Santa Cruz where I studied under Dr. Michael Matias. He was a leading expert in interactive intelligent dramas, he would call them. So they're like basically these. He created this game called Facade, which is this very lifelike, you know, reactive, natural language driven interaction or game with within this story of this couple that you would go visit them and talk to Them. And at the time it was like, that was like 10 years ago. It was like state of the art. You know, there hadn't been anything like that before. And that really, really motivated me to go and pursue studies under him. So I went to do grad school there. And while I was in grad school, I thought I would be like an academic for the rest of my life. Thought I wanted to be a professor. And in some ways I still do want to be a professor. But I was in Silicon Valley and I took an internship with Google because somebody asked me why not? And it totally changed the way I saw the world. I thought of impact in a totally different way. And so I started, like you said, at Google. I did a couple different roles. So I was at YouTube, working on YouTube search as a software engineer, really looking at search algorithms and the ways that like little changes here and there could impact how people use the product. Then I wanted to be more publicly engaged. So I was doing developer advocacy at Google Cloud, working with the machine learning and AI tools. And that's really kind of where that was also around the time the ML boom was happening. One of the big like, oh my gosh, look at all this progress, look at all of the abilities that weren't possible before enabled with machine learning. And that was probably like 2016. Yeah, it was around 2016. And then I ended up doing a rotation with Google Brain, which is one of the premier groups working in AI today, working on Project Magento, which does music and machine learning creativity. And then as I'm like digging deeper into these products, that led me to AutoML, which I explained at the beginning. But yeah, that's basically my journey from video games to working in ML at Google.

That's amazing. Yeah, I'm interested in like one of the things you said was like it kind of, you know, going to Google and experiencing that kind of reset. Your ideas about impact, do you mean like in academia you can have an impact because you can research new things and kind of release them into the world in some sense? Is that the kind of impact you're thinking of or how do you view impact these days in terms of like the AI work that Google's doing?

Yeah, I mean, I think you're right in that like what we do in academia is that we, we write papers and these papers hopefully are cited by future papers and that people receive them well and it helps continue to advance the state of the art. And to me, that was very romanticized in my mind when I went to undergrad. I thought this is where everyone's just talking about ideas and challenging one another to do better. And I found that in grad school more so. But then it was kind of all I knew. Right. It's like when you're looking up to your professors, you're like, oh, I'm going to become that someday. You think, in addition to all that, think about how you're going to be a teacher and how you're going to mentor people and how you're going to grow a group that way. And so, you know, I'm almost embarrassed to tell this story, but I honestly didn't know. Like I didn't go into computer science thinking I'm going to be a software engineer. I'm going to, you know, like I wasn't influenced by the dot com stuff. I really had no idea. I wanted to make video games. Like I would call up Nintendo's hotline, like I think it's like 1-800. I had it like memorized, you know, like 2553700. I don't even know if that's right. But it was something like that. And I would ask them like, how do I make video games? That was how detached I was to how kind of the non academic world functioned. So when I went to Google I was really excited because it was a whole new world that I didn't understand a whole lot about. One of the questions they asked me was, this is really embarrassing. How many users did I think YouTube had? And I was like, okay, what's a big number? For me in research, having 100 users was like, that was a lot of users. Too many users. I remember the statistician telling me like 100 users. It's like too much data. You collected too much.

Yeah, I don't think what I coded in grad school ever had 100 users. For sure.

Yeah, yeah, yeah. So the number I gave and sometimes I'll even like multiply my guess by a factor of 10 because it's so embarrassing. I said, I was like oh like 10,000. I didn't even think about it. I said 10,000 and the manager like laughed and then said nevermind and, and then moved on.

There's at least 10,000.

That's a good story.

Yeah, no, this was, and this was like just that was like how new I was to it. That's how little I knew about like the, the product side of things, like the real world applications. Yeah. And sometimes when I tell the story I'll say 100,000 because it's a little bit like, okay, like, you know, I could see that. But yeah, no, I literally said 10,000.

Yeah. So, like, the scale of the impact of YouTube and like, anything you would integrate in YouTube, including, like, whatever it is, like, you know, whether it's AI related or not, is going to have an impact on a scale that's just totally outside or just totally different than what it is in academia, basically.

Absolutely. Yeah.

So I gotta. Before I ask a question, I'm actually gonna throw back all the way to the beginning when you were talking about your childhood in high school. And I think I might be slightly older than you because my first computer game was the original Atari system.

Wow. Yeah.

And then when I learned how to. When. When in high school, they were still teaching typing on typewriters, that was just coming to an end there. I know, I know. So. And we didn't have a computer in the school at the time, so I'm slightly older than you, but just barely.

Yeah. I work with a guy. I just think this is cool. I work with a guy who met his wife messaging back and forth on the. I think it was the University of Michigan, like, mainframe. So it was like there was only, like, there was no Internet. Right. So they like, let students access, like, the University of Michigan mainframe, and they were like two of the students messaging on there, and they eventually, you know, talked and got married. I thought that was pretty great.

That might be the only mainframe romance story I've ever heard of in my life.

So.

Okay, so I want to actually get a little clarification because, you know, we are all always hearing about everything AI from Google, and there's all these different things. Google Brain, Magenta, TensorFlow, Google AI, Google Cloud, you name it. And so how do these different parts of Google that work in AI areas relate to each other? And are some of those the same as Google Brain and Google AI the same or not? And how does TensorFlow as a tool fit into that ecosystem? And can you just give us a sense of where all these pieces of AI fit in at Google and how they relate?

Yeah, I would say that that's a really good question, and I definitely have a pretty good insight, having worked with a lot of these teams. So like any big company, there's always like, there's like the research side and there's the product side. Right. And so when you're. If you're. If you're looking to work in technology and you want to be a research scientist, then you're looking at being the leading expert in your area. So you're good at doing image recognition and you're like one of you have all these publications. If you're looking to be like on the product side, oftentimes they're looking for people who are like straight up builders. So engineers that are looking at people who have business sense, they're looking at people who understand users, things like that. So in some ways their objectives are very different. One is looking at being published at Neurips or iClare, these top machine learning conferences. And some are looking on the other side of things. You're looking at people who want to create a tool that has a lot of users. So at Google it's a big company, like close to 100,000 employees. It's in the ballpark, I think and it's all over the world. And so a lot of these projects are really like, it's really kind of an opportunity for people to see what really sticks, what's actually going to show traction. So you see a lot of like organic emergence of different applications, technology, whether it's VR, AR, whether it's like, you know, home systems like the Google Assistant, whether it is something like AI and ML. So like what is the landscape? So like I've heard people say like you don't want the outside facing representation of your, of what you do to look like your organizational chart. So it doesn't. You don't want the team structure to be directly reflecting of like what it is that you're providing for people in the, you know, the public to use and to see and to experience. I would say that the way that they're shaping that and a lot of this is like, I think, you know, just being good storytellers or being good marketers to kind of put the tools in places where people who need them will be able to get them the easiest. So we have like, you know, Google Cloud, right? That's where they're moving towards providing business solutions, they're providing enterprise technologies, things like that. So things that will help your business. Then there's what we call Google AI. A lot of Google AI. Within Google it's called Research and Machine Intelligence. It's basically a lot of the advanced looking at how many users we have. But more looking at is the work that's coming out of here relevant to the state of the art or advancing the state of the art. Those are two of the bigger organizations that we could dissect even further. But then outside of that there's one more group which is, I would say more of a collection of groups where you have Google Assistant Google Photos. A lot of these Google products use machine learning and AI to create better features, to create better user experiences and things like that. So Google Cloud, I think, is pretty straightforward. If you go to the cloud website, you can kind of look at all the different offerings and things like that. I think Cloud, AI and Google. Google Research is probably where it's kind of like, okay, where does TensorFlow fit? And where's the AI side of things? So I could talk about it from. I would say it was like, it was Google I O 2017. And that was when we moved to being an AI first company. Yep. I remember, like, Sundar was like, AI first. I had just joined Google Brain at that time, and we were in a building that was like, Google's based in Mountain View. We have the Googleplex. It was in the main area, but not in the Googleplex, which is a set of four. Let's see, 40, 41, 42, 43, four buildings. And so a lot of the Google Brain team, it was actually across the street. So I joined Sundar Announces. We're AI first, and all of a sudden they're getting moved to the center of campus. And so I remember my desk was like, maybe like 30ft away from, like, Sundar's desk because of that move. And like, of course, I never really saw Sundar. And there was like a wall that separated us, and you couldn't badge into that side of the building. So, like, I think we were. I'm just like, looking at, like, the blueprints of the building. I was kind of like, oh, I'm like, pretty close to this. We're probably breathing the same air or something. But I didn't get to see. Anyways, that's just to kind of give you a of topologically how the feel of it is. So TensorFlow. TensorFlow basically dwells. It lives. It's being developed and improved within the Google Brain team. Google Brain Team was at the time run by this superstar engineer, Jeff Dean, that he's a celebrity inside and outside of Google. I believe Andrew Ng and some other people also were responsible for starting this, you know, Google Brain effort. And when we became AI first, it kind of got moved into, you know, a more central location physically, but also as far as focus goes. So then Brain isn't the only thing in Cloud AI though. Right? Because we have. We have groups like Machine Perception and Descartes, which, you know, have. You guys know Raker as well?

Yes, absolutely. Yeah. Legend.

Okay. Absolutely. Yeah. Okay. You guys would. So like, Ray Kurzweil has A team, right? And that team isn't a part of brain, but he's also doing incredible work. So what? One of my favorite works that came out of Ray Kurzweil's group is the Talk to books. And so Ray Kurzweil has these amazing, fantastical aspirations for technology. And so his thought is, what if you needed to know something in a book? What if you could just ask the book and say, hey book, tell me what's the meaning of life? And the book can just spit out an answer. And so that was the kind of every research lead would like, kind of bring in their own, their own flavor.

You know, ironically, that's kind of how the rest of us think about just Google Search in a lot of ways, you know, as hey, we're just going to go and find out what the answer is. So yeah, just funny to hear it coming out of your mouth there.

No, totally. And it's like, and so like in a team like that, like they're thinking, you know, what is the next evolution of search? Right now it's just, it's been kind of like this like keyword, like pattern matching, where you look for the exact pattern of characters that you, you've typed in. But what if there was more of a semantic reasoning behind search and results that arise from those queries? So when you talk to your Google assistant, you have the ability to kind of, it's kind of trying to reason more about your intent behind what you're trying to look for. And so they've created this idea of rather than keyword search, what about semantic search? And they created this game, it's called Symantris, where you have a list of objects and then you're trying to semantically match the meanings and you can knock out different words if you can get a direct match or there's a 1 degree separation between meanings. Anyways, anecdotally that's something that I found really, really delightful. But okay, so then Google Brain, run by Jeff Dean and Jeff Dean, now JG was originally the man in charge for research, but he has left and Jeff Dean now is running RMI and doing a lot of the research leadership. Google Brain though at the time, and now focuses really on how machine learning can make the world better. And so there's a huge medical side of it. So there are a lot of people looking at medical advancements and medical use cases for machine learning. There's Project Magenta, which is under kind of the Generative models group where they're looking at how we can generate not only just music and art, but text. And one of the cool use cases of that is generating Wikipedia pages because you do have a large, well curated data set from Wikipedia and so how can you use that to generate text in intuitive ways? And yeah, robotics is another group. So then there's all these different topics use cases. And I think the reason why cloud AI came about was that as you can tell, it's kind of there's a lot going on under the hood and cloud AI is kind of this landing spot, this portal where you can get the cloud AI stuff, where you can get into what Magenta is doing, where you can see what kind of research is coming out of the group in a more user facing side of things, rather than if you're Jeff Dean overseeing the research army and limbs.

Hello there, this is Jared Santo, managing editor here at Changelog. The fact that you're listening to this means you are actively investing in your future in this industry. Things move fast and keeping up is hard work. Help us help you stay relevant by subscribing to Changelog weekly. We track, log and contextualize what's happening in software throughout the week and deliver it directly to your inbox on Sunday mornings. Head to changelog.com weekly to browse the archives, subscribe and push the easy button on your continuing education. That's all from me. Once again, that's changelog.com weekly.

So you've mentioned AutoML a couple times or auto machine learning, and I think it would be great if we like jump into a little bit of detail on that because it's definitely come up a few times on the show, but as we were kind of discussing prior to starting the recording, we haven't really talked about it in any sort of detail. So I'm wondering if you could just kind of describe AutoML generally, how it's different than how people are used to doing ML. Maybe because some people might think ML is kind of automated in some ways already in the sense that, oh, I can get a CSV, right. And then I can go use Random Forest from Scikit, Learn and choose the column. That's the labels. Right. And it seems like it kind of just happens for me. Right. So how is, like, how is automl kind of different than that? And maybe also like, why is it, why is it needed in certain scenarios?

Yeah, So I love like, you know, practical AI. Right. Let's be practical. Let's think about how that would be.

Great.

Yeah, yeah, perfect. Yeah, I want to be. Now let's take this down. The practical route. So we have machine learning, we have AI. How does that impact the world? How does it affect the everyday person? A lot of people in the us they use smartphones and so now they're able to, now they're able to. If they're traveling abroad, they could take a picture of a menu and get that translated on the spot, or they don't even have to take a picture just within the computer screen. They can translate from Spanish to English. A lot of this image recognition or language understanding comes from advancements and machine learning. On the user side, there's already a lot of people using it. What they found is that a lot of these tools are useful. If you want to be able to use voice commands, you don't need to necessarily retrain. I don't even know how much data they used, but all the data that they required to train that voice model, that voice, speech to text model, they don't need to necessarily have to go through like every, every company that needs, that doesn't need to go through and do that again and again, right? So Google, having worked on this technology and using it in its own products, makes this available for people. So you can come in, you have an app that you want to build, you want to use like you know, you want it to be voice activated or you want to be able to take images and figure out if they are kid friendly images. Like there are these models that Google has pre trained and that they work, and Google is ensuring that these models work well. So without any data, with just the desire to have this feature, you can already plug and play certain products that Google have in machine learning, right? Anything with a lot of stuff requires anything that requires vision or audio processing or natural language. I think these are very common use cases. And then on the other end of things, right, like you were mentioning, you could set up your own Python environment, you could be coding in TensorFlow, you could train your own model, you could feature engineer on your own, you can hyperparameter on your own. You can look at the models and the precision and recall and assess how good these models are doing and then you can do it all over again. And as data, as, you know, maybe you have something running in production, you can kind of have all of that, all the control over every aspect of that. That would be the other end. And of course, enterprise cloud technologies, they provide the environments. You have Docker containers, you have virtual machines that can also satisfy that need. What automl does is it focuses on the middle piece or at least trying to figure out how you Go from one end to the other. So on one end you have pre trained APIs where you have basically no control over how that model is making decisions. And then you have like build it from scratch, like do it all yourself. For a lot of businesses it's kind of hard to go sometimes from research directly to production. And so trying to find that middle point is what people working in auto machine learning are trying to figure out. Things like hyper parameter tuning, right. Are we tuning these weights around to kind of get a feel? And people before have said this is a highly intuitive thing where you kind of get a sense of whether this number should be bigger or lower. Auto machine learning, one of the aspects is try to figure out how to just simply do that with the tools that we have now. It could just be a matter of thinking you might be able to figure out. Safe search or kid friendly images is something that pretty universal for the most part, but maybe not for every culture. You might want to label your own data as this image isn't safe. Maybe like it's not enough to know which pictures have cats in them. Like you can go to your Google Photos right now and do that, right? You can go into your Google Photos, type in cat pictures, get all the cat pictures. But what if you want to know what type of cat or what if you want to know like you know, different colors of cats? There's specificity that these pre trained, more general use cases may not provide. And so automl in the most like general use case is basically giving you the opportunity to label your own data and then feed that in. So you're getting customization without having to build everything up from scratch.

Yeah. So am I correct in saying that kind of automl encompasses what could be a lot of things? So there's also all this other sort of jargon that we're running across, right? Like meta learning, like learning to learn. And then there's like the transfer learning side of things which is taking.

Pre.

Trained models and either fine tuning on additional small amount of data or maybe adding layers to a network or something like that. So is automl kind of an encompassing term that encompasses a lot of those things or is it more specific or separate from those things?

Yeah, as we know in AIML jargon is like, like often it's very controversial sometimes. No, I would say so. Like the people who I like, you know, learning to learn, like these are all groups that for which AutoML is the product. Like they draw upon the findings and the answer isn't like how do we make machine Learning better, which is a research question. What it's really doing is it's trying to figure out how do we deliver the best prediction model. And so maybe that prediction model is like heavily wrong, rules based. And they basically figured out in like 2 if statements you can figure out the best prediction for a certain problem. Right. I think automl, the goal really is to make sure that you're providing the most accurate results given the data and the kind of outcomes that you're looking for.

Yeah. So it's kind of a general term that I guess could encompass many different things. So it could. Like you were mentioning hyperparameters. Right. Like there could be this sort of automated way of determining hyperparameters. I know there's like maybe more complicated things like figuring out like the right types of, you know, nodes or architecture of your neural network and that sort of thing. So I guess all of that would fit into AutoML. Would that be appropriate to say?

I guess absolutely. Like one of my, there's this researcher on Google Brain that works like who I'm friends with. And so we joke a lot and he's always like, we joke that he uses up so much of the computing for the whole company. There's a leaderboard for who uses the most computing throughout all of Google. And one thing that I always joke about, I'm like, oh yeah, how are those hyperparameters doing? Because it's kind of a. I'm kind of minimalizing the kind of work that he's doing by joking about that. But I think it's hyperparameters tuning is just such a fundamental something that people who get their introduction to ML know about and they know that this is this thing that's been highly human intuited practice. And so it's kind of this go to how would you automate machine learning type deal. But no, whenever I joke about it, he's just like, I don't mean to, I only do it in jest to minimalize it a little bit. But it's a very common example to use. But yeah, like you said, it's way more than that. It's the kind of what kind of algorithm are you being, what kind of model are you training, what kind of approach are you. Even within RNNs, there's already so many different ways of applying and using and deploying RNNs, things like that.

And that almost takes right into what I was about to. I was kind of had at the top of my mind anyway. And that is how is AutoML being used in production and what makes it important for the future of AI and ML. So if people are using it in some way out in the real world, where is that going to lead for them? Where is that going to take them?

No, that's a good question. I was actually talking to one of the gaming partners. So like a gaming company. And I was. It's interesting because on one hand you think, you know, automl's trying to take a lot of that in between the statistical work out of it. Right. So I was talking to data science teams and I ultimately feel like automl purpose should really be able to be able to extend the data scientists abilities and intuitions. It should be give them a sense of which features are the most relevant, things like that. Sometimes I feel like a lot of this, there's ML engineers as data scientists, they work together and sometimes they're working separate. But I do think that ultimately it's the data scientist that's serving that AutoML role, especially for specifically structured data. So we're talking about tabular spreadsheet, like data. I feel like a lot of that work, human intuition, has been coming from data scientists at large. And the question is, does AutoML meet the standard of the data scientist? Does it extend the capability of the data scientists? Does it give the data scientist who has been this expert stewarding this area, does it give the data scientists ability to do more or to get a better sense of what direction should we go in in making these decisions? So yeah, so I think that when we talk about these problems, we think like if you're a gaming company and you're looking at use cases for whether a player is going to continue, whether a player is kind of plateauing and really the game is just not engaging anymore and at what point that is, and you're trying to make a prediction that after these actions this next thing's happening, how do you correct course so that you make something that is more satisfying to the user? Things like that. Where I think a data scientist would sit and kind of go through, here are the signals and here are the signs. I think automl would be an additional tool to be able to help arrive at those conclusions in a more efficient way.

So there is I guess some level of expertise that's needed to, to use AutoML. Like I was looking at one of the, I'm not sure where I was seeing this. I think this is one of the examples when I was looking at AutoML on the Google Cloud site. But my life right now at least is all about AI and Natural language. And I think I was seeing one of the examples or use cases they were talking about was like, oh, you have this machine translation model maybe that's pre trained for some language pair like English to German or whatever. But really you need like some very specific types of vocabulary translated. Like maybe it's like legal vocabulary or maybe it's like some domain that's not covered well by the data that the model was trained on. So you need to kind of get that model, you know, fine tuned a bit to that, to that, you know, vocabulary that's not sort of general vocabulary. So I'm assuming that still to structure that data, to get it in the language pairs that you need it in to understand and evaluate if your model is doing well, even if you're not setting all of the hyperparameters or you're not coding all of the neural network architecture, there's still a level of expertise that's required to set up that experiment and be able to know if it's working. Is that right?

So I would say that it really. We were talking about tabular data earlier, right? I can kind of walk through a little bit of like how that would look. But I would say to answer your question, that you really need to, I think the expertise really comes in once the model is trained. Because from that you kind of, you need to have kind of a sense of whether the model is doing well enough for you. So every, for every, you know, in production, every, every feature, every use case is going to be different. And sometimes, you know, you know, if you're 99% accurate, that's good. So for some of them that's not good enough. Right. But really I would say the expertise comes in with, with also once if say the model needs to be better or say some change needs to be made to be able to understand like how to engineer or shape the data in a way that's going to give you a better outcome. So there's two different ends of it. Right? You could just trial and error it, give it data, run the prediction or train it and then run the predictions and then see how well it does and if it's good enough. From your sense, I would say that that's kind of the bare minimum. But I think to hold it to kind of higher standards and to kind of oversee what I've said earlier, the state of the art is doing. I do think it's important to have somebody with data science or statistical understanding or machine learning understanding to be able to see whether to kind of just not Let it run in the wild. And this is something that people talk about, automl, this is true democratization because you're really just giving. All you have to do is just have the label data, show it lots of examples and then it's going to make the prediction. I've talked to the product manager, the product side of TensorFlow, going back to the research side, what they would sometimes say is, is that truly democratization? Because at the end of the day you're relying on Google Cloud automail tables to make this decision and you're controlling these two ends. You get to decide what data goes in and you get to decide whether this prediction model goes into production. But arguably TensorFlow, this open source tool for machine learning, is ultimately giving you all the control and all the power. It's just the learning curve is much higher. There's the cost benefit of that. But yeah, I would say that I could teach a 13 year old how to use with no machine learning understanding, somebody who understands how to point and click and load in files, how to train their own model and give them kind of an intuition of how that should evaluate and how to test that out.

So one of the things I'm wondering, I'm pretty interested. I know in a few minutes I'm going to ask you to kind of take us through kind of an example, if you would. But before we get to there, what are some of the disadvantages maybe for automl, things like would interpretability be one of those and are there, are there certain kinds of domains for which it might not be the most suitable thing? And you know, and I guess since I'm talking about, you know, what is it, you know, potentially not best suited for what are some of the biggest research areas, the biggest open questions that you guys are taking AutoML into, you know, so what's it, what's it challenged on and where do you, where do you see it going in the future in terms of applicability?

Yeah, yeah, I guess in the, in the spirit of AI, it's like I'll throw out more buzzwords like, like humans in the loop and mixed initiative, co.

Creative systems, differential privacy.

So I think the question I get asked most when I'm working with data scientists or showing teams, customers, partners, people who are already or want to use machine learning, when I show them these tools, you get kind of a range of questions. If somebody is more on the data science, machine learning side, they'll say things like, what if I want to be able to get in there and make some changes? I mean it gives you this tool. And honestly, this tool changed. It launched AutoML tables for tabular data launched in April of this year. So it's changed a lot already since then. It's not been that long. So it's constantly changing. But right now, what is available to you is once the model is trained, you can call a REST PI REST API, you can export the model as a TensorFlow package and download it onto your own machine and run it from there. There's things like that that you can do, right? But what we get asked often is like, well, can you give us more information? We want more analytics. We want to see really how well the features are doing in more detail. And that is totally under the hood. You don't have access to that. You can't, um, you know, say if you wanted to, you couldn't really tune the hyper parameters in this case. It's really just giving you the endpoints of machine learning from training to, to inference. Um, yeah, so, so I, I would say that that's probably one of the drawbacks is that you're ultimately not getting the amount of control that you would if you were building your own model from scratch. And that question of control, or at least intuition, and you said interpretability and understanding why it's making these decisions, it's even more abstract. It's like, like the, it's almost like by design is trying to like reduce complexity by not showing you these things. But again, with like, you know, this concept of humans in the loop or understanding what kind of controls do we want as we, like, you know, the whole point of machine learning, and it is almost like seen in research, is the more humans are involved, the more they're going to mess it up. But like, like, as we, as we reintroduce like, like working in the team. Magenta, right, it was focused on. Project Magenta was focused on creating generative tools like extensions of humans. And so this concept of where does the human, what kind of control the human wants is very important. The problem is that machine learning can get very complicated very quickly. So either people are just overwhelmed with the burden of choice that they have and not really understanding how it all works together, or it's like we're kind of taking that away so that they can still make decisions without kind of taking for granted all the stuff that happens in between, if that makes sense.

Hey, guess what? Brain Science is officially launched. Episode number one is on the feed right now, so head to changelow.com brain science to listen, to subscribe and to join us on this journey of exploring the human mind once again. Changeloud.com brain science or search for brain science in your favorite podcast app.

Okay, so let's say. So I've never used AutoML other than just kind of browsing around the website and looking at some examples and that sort of thing in very general detail. So I was wondering if you could kind of just walk, walk us through like if I'm, if I'm going to go implement my, my first AutoML application or workflow, what are kind of the steps I go through at least with you know, Google's AutoML tools.

Yeah, yeah, of course. And yeah, you know, Google's not the only company that's offering these, these capabilities. But, but yeah, it definitely, definitely starts with what you've been saying earlier about how you know, you have certain language like say you're a German doctor and you need something, you need a tool that recognizes that, that context or you're you know, you know, searching for a cat and food pictures is general, but maybe you're an automotive manufacturer and you need to, you have all this label data of car parts that isn't going to show up in the more you know, general use cases of the pre trained models. So what you need to do is you need to, you have this data set and you know this data set is good and accurate. What you would do in this case, let's use like talked a lot about tabular data. We're using spreadsheet data spreadsheet is something that almost everybody uses today. But in this case, what feature we're looking to establish within a spreadsheet is the ability to say, select a column in a spreadsheet and say given all these other columns, can you predict what the value of this column is going to be? Let's say it could be categorical, it could be numerical. Like the example that I've been using a lot is there's one of the open data sets that we have is like a banking data set that shows different types of banking customers and what they like things from how often they deposit to what their income is to what kind of jobs they have and then trying to infer which banking customer is most likely to have a, to turn on direct deposit. So they're going to have their paychecks directly go to the bank. What you would do is you would find first of all whether it's, we talked about CSV files, of course this is Google. So you could have your data in BigQuery and you just give it the BigQuery ID, you go to automl tables and you would load in that data set, you would import it and given all the different. So it'll be labeled, it'll be either like categories or numbers or text. And from there you have the opportunity to kind of look at the different column names. From the column names, you can then nullify or select your features. In this case they have things like age, education, balance, housing loans, things like that. And from there you look at each of the column types and you select one to be the target. In this case we're going to select direct deposit. So this is like a true false. Like does this person use direct deposit? Do they not? Then there's other analytics you can look at, right? So it gives you opportunity to see kind of the spread or whether there's any features that, or any columns that have missing values or invalid values, things like that, where you, you can kind of remove. And this is where a data scientist said that these kinds of tools actually are very useful because they often have data, they have lines and lines and lines and lines of data for which some of the data might not even be valid. So being able to detect that automatically would be very useful for them. From that point on, you would just then train the model and the selection you'd make there is how long you feel like training for. So you could say I want to train usually like one hour, honestly, for maybe these are approximations and it changes case by case. But like, you know, an example, right? For this case, I would say for 4,000 lines of input, you could train for an hour and you'll get a decent result. You'll get like a, you know, 95% accuracy for predicting whether or not someone has direct deposit. Once it's trained, you select that threshold if it finishes earlier. And of course it's going to detect things like over overfitting when it's like just kind of reaching diminishing returns. So it will terminate early as well if it needs to. But you get like, at the end of that you get like an evaluation. So you look at, you know, there's the recall, there's precision, there's looking at the true positive rates and the false positive rates. There's the accuracy. So in this case, this model that I'm just going to pull up one right now, this model has accuracy of 91%, which is pretty good. Again, case by case, it's like what is your use case? What is your threshold in production? And it's going to differ. It also show you feature importance, how the labels are doing. And you can then, basically you could deploy it right then and there. You can just deploy it within AutoML and then you can run online predictions, you can run batch predictions, or you can export the model. So you can, like I said earlier, export it. It'll be a TensorFlow package. You can run the model in a Docker container, you could take that Docker container, you can download it onto your local machine, or you can just not do any of that and just use REST API and be able to make predictions that way. That gives you the end to end. Basically. Once you load it in and you, you click train and the training completes, you're basically ready to end. You have to deploy it as well, but you're ready for online predictions after those few steps.

Just to clarify, to make sure, I know Daniel and I have come at things from more of a framework perspective and using TensorFlow or PyTorch or one of the other frameworks over time, in this case that you're talking about, you're really not doing that at all. AutoML is its own process and you're selecting the data from BigQuery or elsewhere, pulling it in and essentially producing a product at the end in the form of a model that you can go use. Is that fair that it is a parallel path versus an integrated path with whatever framework you're using?

Absolutely, yeah. You're loading the data, you click a button, it says train, and then an hour later you have this model which you kind of look at the stats for it and then you're online after it's deployed. So it's like a few clicks. I mean, getting the data is non trivial, I would say. And then making sure that the model is actually performing the way you want it to is also non trivial. But yeah, it is like a few clicks away. And then you have what you would normally do if you had like, you know, you had like maybe Even like a TensorFlow notebook. Right. You would still have to kind of, you still have the ability to like, you know, choose the type of approach you want to use and then change the values around a little bit. But in this case, you don't really have that. You get to select the target of which you want to be what you're predicting for, but that's about it.

Yeah, I definitely see what you're saying about the expertise that's needed. Because if you're looking at a screen and you're getting first of all a package out or a Docker container out, you need to understand how to work with those things or like REST API. So there's like engineering expertise there, of course, but even if you're presented with the screen and it says like, oh, here's precision and recall and these different things, like, you know, there's. There has to be a level of understanding of those things to be able to understand, you know, if your model is really doing what you want it to do. Like you said, it seems like there's like, if I'm thinking about my AI workflow or machine learning workflow, there's of course always the data side of things, which is still going to be in some cases hard, like you're saying here, because, like, you know, the data quality is going to influence things. There's the training process, which, you know, if you do it manually, might involve, like you were saying, hyperparameter tuning, like trying out a bunch of different architectures or models and that sort of thing to where you get something that works, then you have to evaluate that and then put it into production. It seems like here you're really greasing the wheels around that training process side of things where you don't have to think as much about the architecture, the specific model, the hyperparameters. And then you're also helping a bit in terms of course productionizing that in the sense of making a REST API available and that sort of thing. So you can spend a lot more time thinking about the data and the type of model that you want out, rather than those sorts of architectures and other things.

Yeah, that's totally it. That's the idea is to make that more accessible, to give people the opportunity to kind of use it at that level of abstraction. But I think in some point we're trying to look at the full sense of how much control and what kind of control we want to have as we create these models as well.

Cool.

Yeah. So to kind of wrap up here, I think this has been great to. I mean, I've learned a lot going through this.

Me too.

And definitely want to try out some of these things. So I guess ending, you know, to kind of end up with some useful things. I was wondering if you could just kind of recommend, like, so thinking about someone that's maybe wanting to try out these sorts of automl techniques, what the kind of easiest way for them to try out those things are. Maybe there's tutorials or like, you know, notebooks or. I don't know what the interface is to try those things. And then also if people are interested in kind of learning generally about, like, maybe the methods behind AutoML and like working on those, you know, like learning to learn things and hyper parameters, automated hyperparameter tuning if there's any kind of specific places that you could point them to.

So yeah, for people who want to know about Automl in general, I mean the Google AI, the portal they link to. One of the groups, the researcher that, one of the researchers that I hear a lot working in this area is Kwok Le. He's a Google brain research scientist that has a team of people working in auto machine learning. So anything under him, any of his publications I think would give you kind of an understanding of the state of the art within that area for people who want to try it out. So if you go to cloud.google.com automl It'll kind of bring you to all things Automl that is available as tools for people to use. It gives you kind of a list of, of the different applications or verticals. There's like you know, site where you have AutoML vision, AutoML video intelligence. So this is again you're providing your own label data. There's language so Automl natural language, Automl translation and then structured data which is what we talked about today, Automl tables, there's a few like customer testimonies where you can see how it's being used and a little bit of a description. And then you'll find if you go to structured data there's a, there's a video narrated by me, I pulled together this video for explaining kind of the basics of it. But if you want to try it again you would choose from those categories. And so if you choose AutoML Tables there'll be the opportunity, it'll kind of pull up the UI and the cloud console. So of course you'll have to have like a Google Cloud platform account but you do get some. You could train for free for up to a certain amount to kind of try it out for yourself and then actually the cost is very much. I'm not on the sales side so I actually don't know the exact numbers but every time I've talked to a solutions engineer and heard them quote, it's like very low amounts of money once you actually want to get it going. But yeah, I do know that there's amount that you get to use for free. So if you just want to check it out, try it out yourself, it's pretty accessible. And even if not that, if you want to even try some of the pre trained APIs like you want to upload a cat picture and see if it knows your Cat is a cat. You can just do that within the web. So if you just want to see how well these pre trained ones work, they have online demos within the website as well.

Awesome. Well thanks for pointing those things out. We'll definitely share those in the show links, so make sure and check out the show links. We'll put those links there and you can can try out Automl and definitely consider joining our community somehow. So you can go to changelog.com community and we have a Slack team where you can chat about Automl and practical AI and all those things. We also are active on Twitter and LinkedIn so make sure you engage with us. Tell us what you're using Automl for or where you found it useful or what research you're interested in. But thank you so much Cheryl for, for joining us. Really great conversation. It was really great to hear your story, hear about some of the things you're doing and and the things Google is doing in in automl. So thank you for taking time.

Thanks a lot.

Yeah, thank you.

All right, thank you for tuning into this episode of Practical AI. If you enjoyed this show, do us a favor. Favorite Go on itunes, give us a rating. Go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend. Whatever you gotta do, share the show with a friend. If you enjoyed it. And bandwidth for Changelog is provided by fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of Rollbar. Check them out@robot.com changelog and we're hosted on Linode Cloud Servers. Head to linode.com changelog check them out. Support this show show this episode is hosted by Daniel White Knack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@change law.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcast for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.