Pie & AI: TensorFlow Specialization Launch @ Google HQ

All right, so there's been a lot of press and excitement around AI. Can one of you, both of you give us your thoughts on the state of AI now and where it's going next? 5, 10, maybe 20 years and even further, what do you think will be the most eye opening advancement?

So I've been saying for four years now AI is a new electricity. And I think over the last several years we've made tremendous progress. The number of jobs created, number of applications, amounts of revenue, companies are building, many trends are up and to the right and I think that momentum continues. But if you look at what we've done so far, I think we've transformed the software industry. Companies like Google and other tech companies that you see around this area are very good at using AI. And now it's beyond just handful of the largest companies, many software Internet companies, including recent IPO stars, our industry is very good at using AI. One of the things I'm most excited about is for us to now go and transform the rest of those other industries, everything from manufacturing to agriculture to healthcare to transportation logistics. We live in a little bit of a bubble here in Silicon Valley, but most of the economy is not the software Internet industry. And having kind of helped the software Internet industry do this, there's still more work to be done, still a lot more great projects at Google and companies like Google that should be done. But I'd love to see the whole economy, even more of the world transform with AI so that we can use this and make really a lot of humanity more powerful, more effective. And as part of doing that, with the last wealth of creation, really the rise of the Internet, I think we created a lot of wealth, which is great, but we also contributed to wealth and equality. And I hope that this next time round, with this next wave of wealth creation that we will do using AI, we can take everyone along with us. And I think part of that is the democratization of AI knowledge and AI tools, which TensorFlow has been a huge contributor. You.

Yeah, and totally agree. And for me, part of my passion, as Alina mentioned earlier on, is really software developers. And I think in order to be able to build the world that Andrew was envisioning, you know, software developers are going to be the front line in doing that. So where I would like to see AI in 5, 10, 15 years or whatever would be that. How many of us, by the way, identify as a software developer here, just out of interest? Okay. Most of the room. Cool. So that those of us who are software developers would have AI as A normal part of our toolkit in the same way as anything else is the same way as understanding SQL to access databases or understanding presentation layer versus coding layers like in web and understanding high level languages like Java or C or Swift or those type of things that it would just be something that's much more normalized as part of our toolkit. And I think when that's something that becomes normalized in our toolkit, that's when we'll be able to build the things that can take advantage of AI very quickly. And it's amazing when you start thinking about the scenarios of problems that we all solve on a daily basis, how much we can improve those with AI.

Techniques just to build on something. Lauren said Geoff Hinton once mentioned to me that today we have a lot of software engineers, we write programs to computers what to do. Jeff mentioned that in the future maybe we need equal numbers of people programming computers as showing computers or using machine learning. And I don't know if in the further future, 10 years, if the ratio will be 50, 50 or some other ratio but Stanford's PhD program, over 50% of the applicants want to work on AI. And I don't think it's unrealistic if in the relatively further future we have equal numbers of people using programming types of techniques as machine learning techniques. And it would be an exciting world if a large fraction, maybe 50%, I don't know of software engineering was machine learning based.

It was actually interesting that what Andrew was sharing as well is that when we started off building the course that we're working on, I shared a statistic as to why I wanted to us to think about shaping the course the way that we did. And there are various surveys out there by people like Stack Overflow and EDC that talk about the number of developers in the world. And it's usually like in the high 20 million, 27, 28, 29, 30 million. And then there was a survey done by Baidu in China about AI practitioners and there were 300,000. And so it's like we were saying like if we can help a significant percentage, like say 10% of those 30 million software developers become trained in AI and ML, we will 10x the number of AI practitioners that are out there and I find that's like to me that's particularly inspirational. So if you end up in a few years time, 2, 3 years time with 3 million AI practitioners as opposed to 300,000, think about all the things that they're going to be able to do. Think about all the new solutions that we'll have.

So you jump directly into my next question. There's a huge shortage of AI professionals right now. I know that both of you are teaching courses and putting content out there. Can you talk a little bit about what this will basically do to actually improve that ratio that you just mentioned?

Online courses, online videos, Lauren's, a lot of YouTube videos as well. I think this knowledge is very helpful. Maybe I just mentioned one thing that's less widely known, which is the importance of enterprises in driving this type of education. So many of you I know have been signing up for the tensile specialization for deep learning, special zoom, things like that. And that's great. One of the trends I'm seeing strongly is that more and more companies are facilitating having huge numbers, sometimes hundreds, sometimes thousands of people learn these things sort of at the same time. A few weeks ago I was talking to this group from a company that about 2,000 employees and they were getting every single employee, starting to get every single employee to take AI for everyone, which is really interesting to do. And I think when a community of people like that in a company signs up to learn TensorFlow or learn about these tools, you can dramatically accelerate the rate at which a company is able to embrace these tools. And that's one exciting trend, actually. Since I'm back at Google, I remember when very early days of Google Brain, first thing I did was actually teach a class in Google on deep learning. And so we met once per week and I think something like 50 engineers showed up every week. We had them watch some videos that they created at Stanford and then they would listen to me. And sometimes others, sometimes Jeff Dean, sometimes others talk about machine learning. But we could do that because Google had good machine learning people or a few of us were already there. One of the ways the world has changed is that even if a company does not currently have Lawrence as an employee, well, they can get Lawrence in digital formats. And I think that is a wonderful change that will really accelerate a lot of companies adoption of AI.

Yeah. And it's things such as like what Andrew was saying and going back to the question you were asking about empowering people in learning AI, to me, one of the great advantages of it is. Is anybody familiar with the Gartner hype cycle? Have you seen? It's like. So it's a curve. I see a lot of hands. It's a curve that starts with a technology trigger and then it goes up to something called the peak of inflated expectations. And then it falls into the trough of disillusionment. Before it rises again into the plateau of productivity. And like with AI right now, it's like there's so much of this hype and this peak of inflated expectations that what I love, like when you're saying for AI, for everybody going to everybody in a company, and not just the engineers, it helps those in the company break through those inflated expectations that they can understand what AI is, what it does, what machine learning is, what machine learning does, and set realistic expectations for it. And I often joke that my job at Google is to help people tunnel through that curve, to go straight to the trough of disillusionment. So I'm a professional disillusioner. But. But it's once you reach that point that productivity really begins. And there's a famous saying that those who don't learn from history are condemned to repeat it. And I always like to think about if you remember when the smartphone first came out, when Steve Jobs went on stage and said one more thing and then he introduced the iPhone, the world hit this peak of inflated expectations. Everybody was going to throw away their laptop, everybody was going to throw away their desktop. All you'd ever need is a phone. All your compute could be done on that. And then people got the devices and they realized their restrictions. They were amazing powerful devices, but the restrictions of the smaller screen, of a touch interface, of only being able to run one app at a time, the world fell into the trough of disillusionment. But then out of that productivity road and new scenarios came out of it. One that I always like to give an example of companies like Uber and Lyft. Once upon a time you would tell people that you don't talk to strangers on the Internet and you don't get into strangers cars. Now we literally summon strangers from the Internet and get into their car. But it's become a really, really convenient thing. That app scenario became really possible because of that trough of disillusionment rising into the plateau of productivity. And I think as more and more AI gets seated in the enterprise and things like Andrew's AI for everybody becomes something for everybody. And business decision makers, as well as technical decision makers, the human resources people, the CEOs of the company are learning more about what it is, then they can start really becoming realistic about what they can do with it. And that's when, to me, the exciting things begin to happen.

So since you both mentioned education, how people basically need to learn to move into these amazing fields and move the industry along, can you both talk about a new course and your favorite part of the course maybe content wise.

Okay, can I start this one? My favorite part of the course is that before every lesson, Andrew and I have a conversation about it where we get to kind of geek out about what it is that we're talking about, be it convolutions or natural language processing and all that kind of stuff. And that's something that, for me, has really helped shape how the rest of the world sees AI. And, like, Andrew just has this magnificent vision for AI for the world. And, like, I suffer from massive imposter syndrome during those conversations, but I always come away from them feeling a little bit smarter. So they're my favorite part of the course. And he gave me a mug as well. And anybody who gives me a mug is the.

Yeah, you're too kind. You know, one of my most memorable moments, I'll tell a story and then tell the generalizable lesson, was when one day out of the blue, Lawrence slacked me, sent me a message with a Irish poem generated by an LSTM that started, you know, Andrew sang a saddle song, and then it went into Irish lyrics.

Hey, diddly diddleys and nonny knows.

Yeah, And I think it was cool. You know, it was like a LSTM generated. Lawrence had trained it on, what, 80 Irish songs, and then use an LSTM to generate very weird Irish song lyrics. And I thought that was cool. Well, for two reasons. I had never imagined that one of the benefits of working on AI is, you know, it's like, don't do AI just to work on exciting projects or to have an impact in the world. Do it. Because you can also get customized Irish poetry. But one thing I've learned about AI is keep the fun in it as well. Recently I met an engineer that works for a large, that works for a travel agency, and she had written. Actually, a small team had written a piece of software using machine learning that generated customer greetings for yes, I'm simplifying some detail to preserve. And the managers told them, this doesn't generate. This was one of their company's first forays into machine learning. It was like a fun hackathon project. And their managers actually went to them and said, look, this doesn't generate any revenue. Stop doing this. And I really disagree with the approach of that particular management team because it's true that AI offers lots of great opportunities. If you have an idea for doing something huge, go and do that. But until you are ready to have that huge idea, much better to do something rather than nothing. And I see a lot of people Start off by doing fun hackathon projects. Lawrence has a great rock, paper, scissors example in the course as well. And you're not going to. Maybe you change the world by doing AI for rock paper scissors, maybe not. But it's fine, do it. It's fun, you enjoy it. And it's often by doing those fun projects you gain the skills that allows you to practice to then. And the goal is not for you to be building rock, paper, scissors level apps for the next 10 years, it's for you to do that homework exercise, learn to do the small thing and actually that exercise runs in a web browser. It's actually really cool. And then over time learn to do bigger and bigger projects so that if in two years you then see an opportunity to do something that helps 10 million people or something, that you've now practiced the skills of data collection, data cleaning, modeling, deployment, debugging after post deployment, that lets you then do those big opportunities later. Just encourage you do the fun stuff in machine learning too. Do the live serious projects if you can, but until then give respect to the people doing the fun hackathon projects too, because that's how we grow as a community.

I actually have a very specific question about the course. Why does it focus on keras and not TensorFlow?

Okay, that's a great question. Can I take that one, Andrew? So there's a very specific reason. I wouldn't exactly say it focuses on one over the other. It's more Keras is Now part of TensorFlow. And with TensorFlow 2, one of the things that we really wanted to do was democratize AI, make it a lot easier and particularly for developers to learn how to build machine learning and deep learning applications. Earlier I mentioned the 300,000 number and the 30 million number. The 300,000 number are people who are very familiar with graphs and very familiar with low level ops. The 30 million number are not. And so in order to give them a smooth on ramp to AI machine learning, deep learning, do things like convolutional neural networks and write Irish songs with LSTMs. Keras is really, really powerful for the that. So the early courses focus on using the high level APIs in TensorFlow2 that are Keras based. It's not a case of it's a choice of one over the other. It's just the case of as you do the early stuff and as you on ramp to this, we find it's a lot easier for developers to be able to build things relatively quickly and build powerful and fun things relatively quickly using Keras as we create more modules in the course, we've got a very long plan for this, for a lot of stuff. Then later on we're going to be going into the deeper APIs, into the graph based stuff, into the more research oriented stuff. But I do want to allay that it's not a case of we're choosing Keras over TensorFlow, it's just really Keras is part of TensorFlow and it's a tool that I think has helped make TensorFlow a lot easier to get started with.

How would you recommend that somebody completely new to AI get started and get going?

Can I start with this one, Andrew? Because I started by doing his course and so when we first met to talk about this course, I told him that I think I embarrassed him a little. So Andrew has some terrific courses online. There's the Stanford. Is it CS231B? CS231A229. I never remember the number, Sorry. But so that course was terrific. For me to understand and demystify a lot of the concepts in machine learning, you know, it's. Well, I started reading a lot of papers and I started reading a lot of tutorials and they start talking about things like, hey, you need a really good loss function or you really need a really good optimizer. And as a coder I was like, why? And I didn't understand and I didn't understand how these worked and there'd be things like all those acronym, soup, cnn, rnn, lstm. It's like I just didn't understand all of that. So for me that's actually how I got started and inspired on the journey that I'm on. Funny story is I actually did AI like 30 years ago, was in college and AI back then was prologue and Lisp. Anybody ever study those languages? Anybody ever wear out the bracket keys on their keyboard doing Lisp code? If you've ever programmed in Lisp, it's literally bracket, bracket, bracket, bracket, bracket, bracket, bracket, piece of code, bracket, bracket, bracket. It's really strange. Anyway, I digress. So I moved away from AI and then in the last few years I've moved back into AI. And that was the first thing that got me started, to really see how things have advanced and how powerful they are. And I'd strongly recommend that to anybody. And then what we wanted to do with this course then was to help people who are developers to get started by having something that really meshes with that, that's oriented towards coders. So Andrew's content, the Deeplearning, AI specialization, all that kind of stuff to really help you understand what machine learning is on the inside and then something that complements that for coders to say, okay, now you want to build a convolutional neural network. You know what convolutions are, you know how they work. Here's the code for convolutional neural network and here's some practical examples on classifying rock paper scissors or classifying fashion items and that kind of thing with the hope that when these two mesh together that they will really help people become experts in it. That's my recommendation for getting started. Do both.

I feel like the causes that we're creating DeepMind AI Lawrence some of the earlier things we did at Stanford, think of them like Pokemon. You really want to collect them all.

Who's going to be first to tweet that?

But I think that actually some of the non people of other technical backgrounds that were bad for everyone. The machine learning course out of Stanford is still a common on ramp. It's actually the, I think over 2.5 million people have enrolled in that. I think it's probably the largest on ramp machine learning in the world still today. The deep learning specialization, the new TensorFlow specialization. I think there are complementary skills that will teach you different mixes of applied versus deeper theoretical knowledge. And I think all of these things hopefully will contribute to your being a successful machine learning engineer.

That's great. So I want to switch gears a little bit and ask both of you about what recent advancements in AI are you most excited about?

Oh, I wasn't expecting that question. I'll mention one that maybe isn't widely known but that I'm excited about which is self supervised learning. So you know about supervised learning, learning with labeled data. You may have heard of unsupervised learning algorithms like K means some autoencoders, gans learning from unlabeled data. There's this weird thing in between self supervised learning which is when you take unlabeled data and make up labels and feed that into supervised learning algorithm. So word embeddings is one example of supervised, of a self supervised learning where you take unlabeled English corpora, hide a word and have the algorithm predict the missing word. And this is great for learning word embeddings. But in computer vision there are now sort of made up tasks such as I can take a picture, cut it into a three by three grid. This is called the jigsaw task. Take a picture, cut into nine pieces, shuffle it so nine factorial permutations and have the algorithm try to figure out what was the original permutation of these nine pieces. And so called the jigsaw task. This is one of many, many examples. Another one, I'm going to take a picture, randomly rotate it so I can do this with unlabeled data and I ask you to predict what was the original rotation of this image. So. Oh, sorry people, maybe I should use a handheld. I tried to speak up. So these are all very clever examples of ways to make up labeled tasks for supervised learning. And it turns out when you do these things, both for NLP and computer vision and a little bit increasingly other domains as well, this allows you to learn the hidden representation that can be transferred or whatever to many other tasks. I think it's always a little bit hard to predict what's coming down the pipe. I think supervised learning is not going to run on Steam for a long time, frankly. If we just use supervised learning, there's so many applications that we could build that would be very valuable that have not yet been built. But if you look at technologies that are a little bit further down, I think I'm very excited. And ICML International Office for Machine Learning. I spent a whole day just listening in on the self supervised learning workshop. But that's. This is actually one technology that some people are hearing about that has not yet hit the limits of scale. So a lot of research groups are just trying to find more data, faster computers to do this and I'm excited about the potential to let us use more unlabeled data. This is just one weird thing I happen to know about. Seems too early to call. Maybe RNNs had a little bit of hype cycle that Lawrence mentioned and it's not as hyped anymore. But I think if you want to capture long term dependencies, I think they're still a wonderful tool.

Yeah, and, and attention is not going to fill that.

I think attention, I feel like attention was a brilliant mechanism, was a brilliant idea. I think that, yeah, I think that ideas like attention, which was actually built on top of RNN in many of the early instantiations or transformanets done at scale still seem like very promising process of tool. One of the hardest things to do in today's world is still scoping good AI projects. Right. So when, whether you're startup or whether you're a team in a big company to identify the valuable use cases, that, that has to meet two criteria simultaneously. One, the engineers have to really build it because of the hype about AI, sometimes they over elevate the expectations, like oh, maybe we'll have, you know, chatbots, they can have fully general purpose conversations and talk about anything. Well, we just don't know how to do that today. But so realistic expectations about what you can build, do you have enough data? What's the engineering timeline to execute that on? That's one. And second is a business validation. If you build this learning algorithm, will this actually help a manufacturing plant improve yield and improve efficiency? Or will this actually help agricultural machinery so that when the farmer dries it, more crop comes out of the back? Or, or will this actually help radiologists diagnose X ray faster and at comparable higher accuracy but do that business validation. So when my teams are brainstorming projects, we will often brainstorm multiple ideas. Actually when we work with large companies we may brainstorm say half a dozen ideas and then for each of the ideas we subject it to, sometimes a small number of weeks of technical diligence where we validate the idea is technically feasible as well as business diligence where we built conviction, where we try to figure out if this idea really is valuable before we execute the idea. And I think that's true both for startups as well as for large projects that one might want to do in a company. Yes, I think Knowledge Draft is one of those AI technologies that is widely used, very powerful within large companies like Google and a few others, but that the PR it gets or the amount of academic interest it gets seems significantly smaller than its actual utility. And I think, you know, frankly, despite all the excitement about deep learning, I love deep learning. Not everything should be done with deep learning. And if you don't have a lot of data, you need to cut up more prior knowledge. And I think the best AI teams don't use deep learning for everything. I think the best AI teams have a portfolio of techniques ranging from, you know, deep learning. But sometimes, you know, sometimes you use XG boost, right? And sometimes you use. Actually a couple weeks ago my teams and I were sitting down and debugging some things out of pca, right? And we're looking at the residuals from PCA and trying to figure out what's going on to prophecy, graphical models, to knowledge graph, to these other tools. And so I think the best AI teams actually have a portfolio of techniques and then choose the right tool for the right job. Um, I think actually one of the things, I think, I think Lawrence mentioned deep learning specialization in addition to TensorFlow specialization. One of the things that the deep Learning specialization does is teach the strategy of machine learning, including, for example, when should you use an end to end deep learning algorithm? Sometimes it works great, but sometimes you should not use that. So one of the very practical things that deep learning specialization teaches is strategies for designing neural network architectures so that in addition to the coding, which the TensorFlow specialization gets you into very quickly, you have the conceptual framework to understand the algorithms at a deeper level and then also make these strategic choices. I've seen a lot of machine learning teams make the wrong strategic decision, just pursue the wrong approach, such as use end to end deep learning when they should be building a knowledge graph or whatever. And when you make the wrong strategic decision, it's very easy for a team to burn six months and get nowhere. And then sometimes someone you know, and then sometimes you know, someone like Lawrence or me goes to a team and says, hey, why don't you, you know, just build a knowledge draft there, use PCA there and they dramatically accelerate their progress. So I think it's important to have those strategies right in addition to the coding. So machine learning journey, the deep learning specialization teach you those things in addition to the mechanics of coding that you get from, you also need to be good at coding.

I think it's an attribute of the hype cycle where it's like, hey, machine learning fixes everything, can cure everything. Let's apply machine learning. And managers who don't know anything about AI machine learning say we'll make our product better by putting machine learning in it and that kind of stuff. I think once we bust through that hype cycle and there are the skills out there to be able to determine when it should be used, when it shouldn't be used, I think things will get a lot better. We'll be at the trough of disillusionment, heading upwards in the plateau of productivity. But I actually really love what you were just saying about the fact that deep learning algorithms or models that you build are just classifying images or labeling images. For a lot of people, that's the end product. But it's a case of when you start looking at innovative solutions like you're talking about that that feeds a knowledge graph, that's when the real fun begins. I think like the example I spoke about earlier on, like instead of a device that just says this is a butterfly and tells a kid that it's a butterfly, the kid already knows it's a butterfly, you know, that kind of thing, but can actually craft that into an overall product that helps them tell a story later on. That's when it begins to get exciting. And just going back to there was a question earlier about like 5, 10, 15 years from now. To me, it's like that when machine learning and deep learning become part of the programmer's regular toolbox instead of a magical solution in and of its own, right. That those kind of really cool things happen. So I'm curious to learn more about what you're doing, because that sounds really exciting.

I'll mention one weird thing I noticed a couple of years ago, I used to go into rooms. You said, machine learning is not magic. A few years ago when I was speaking to CEO audiences, I'd go into the room and sometimes say, hey everyone, nice to meet you. Then I would actually say, just so you know, machine learning is not magic. I would say those words. And the surprising thing was I saw everyone around the table relax a little bit and we somehow hyped it up so much that that's even worth saying. So you could try that, I guess, with your csu.

I'm going to plug my YouTube channel. If you go to TensorFlow, sorry, YouTube.com tensorflow There was a talk from Google I O this year about federated learning that I'd strongly recommend that you look at for the rest of us, just in summary, the idea behind federated learning is that devices like your Android device are not really suited for training models on but they are very good at generating data that models could be trained from. And often it's a collection of people who have data in a collected environment, like maybe the garbage cans environment I mentioned earlier on, that can generate data that can dial home to a central server that can retrain and then can download models to those devices in a connected scenario, is really the big idea behind federated learning. And we've realized that it's something that very important for us to continue working on. I would just check out that I.O. talk for more details on it.

If I'm not mistaken, there's a really good blog post.

Oh yeah, we got one on this.

Okay.

That's a cartoon that lays everything out.

Okay, blog.tensorflow.org is where it is. There was another part of the question, differential.

Yeah, let me just even. Let me distract you. One of the exciting trends that is coming through machine learning, which is that edge devices have been getting more and more compute at a relatively fast rate. Cloud has been growing compute as well at a different Moore's Law type of rate, and the network bandwidth is also getting faster, but at yet another different Moore's Law and beyond Agent cloud. There's also starting to be FOG computing or these other servers like compute in your, in your local network interchange or compute in your WI FI router. But we're starting to see these multiple tiers of compute devices from the cloud to fog to edge. And because these Moore's Law are improving at different rates both in compute and in networking, it feels like every year sometimes it feels like every year we're rebalancing what workload to do where based on the trade offs of, you know, latency versus aggregating data from multiple users to get better performance or whatever. So it's actually very complicated and very dynamic and the dynamism is driven because these different models are advancing it at different speeds. So that'll be something exciting to watch out for. And differential privacy, I think, you know, I think I give, I guess Apple deserves a lot of credit for kicking off that conversation, even if at that time the technology was relatively mature. But, but it is actually starting to work. So for example, I think was it Don Song at Berkeley and Uber working together did some work showing that you could train language models from data where if you add noise in the right way, you can train very good language models that do not make it as easy for someone to recover like a Social Security number that was in the text that was used to train the language model. So I think differential privacy starts it off with a little bit of, you know, hype. But, but I do see road traction and I think so I think it's encouraging that tools like differential privacy are starting to work well enough that when companies want to put in these things as safeguards for privacy that they're starting to be more options. So it turns out that there are many different skills that successful machine learning practices should have and it depends on the company as well as on the role. I think there are some to give to give you an example of portfolio of skills. I think that a lot of attention is on the modeling skill, how to choose the right network architecture, choose the right model, how to code up on TensorFlow or code up in whatever framework that's important. I think that general software engineering skills are also important and sometimes a little bit underappreciated by some, some of the machine learning community. After you build a model in Jupyter Notebook, there's still a huge gap between what runs on your laptop in Jupyter notebook versus something you can deploy. And so the software engineering skills as well as the practical engineering deployment skills are important. I think that for very large data sets you need strong data engineering skills as well. Depending on exactly where you want to deploy, the ability to do programming at the edge, even, even real time embedded is important. And then if you end up doing more data science work where the goal is less to have a running AI system but to, you know, produce a set of business insights, then statistics analysis skills are also important. So one of the AI, it actually encompasses a lot of things and I think people with many things in this portfolio skills tend to do better. Oh, and also within AI, you know, don't just learn deep learning, learn also, you know, other things, other things in machine learning or things like knowledge graphs we talked about probably graphical models, specialized skills in computer vision, specialized skills in nlp, maybe even specialized skills in robotics or other sectors. So depending on there's actually a lot to learn. And your portfolio of skills will make you more or less qualified for different specific roles. Because there may be a job in a company that needs more computer vision expertise, but less NLP expertise or needs more big data engineering because it is as massive versus other things.

And I think I'd add to that there's the traditional software development skills of understanding where your stuff gets deployed. So like being able to understand to scale out to the enterprise, you know, when you build your model or when you build the stuff that wraps your model, how do you scale that out? How do you build that so that you can serve a billion users instead of 1000 test users, that type of thing. But there's also one really important one that tends to be overlooked and that in any software engineering, not just deep learning and AI, and that's being good at requirements analysis and understanding the requirements of your end user and understanding the requirements of your customer. And I think in some ways that also gets expanded somewhat in AI because AI and machine learned stuff are very data driven and it's very easy to overfit to the data that you have and not just be able to, and not be able to deliver what your end user needs or what your end user want. And there's the B word. Can I use the B word? Bias. There's a lot of discussion around bias in AI and I think it's a really, really good skill for a software engineer or an AI engineer or an AI practitioner to be able to spot that before it goes out. And it's like that's one of the, to me, one of the newfound skills that's out there now. I'd like to tell a funny story about that, but who has done the convolutions part of Our course and has done the rock, paper, scissors exercise. A few. Okay. Not many. Okay. So for the rock, paper, scissors exercise, I generated a data set of 840 rock, 840 paper, and 840 scissors. But let me just do a quick survey. How would you do scissors? Just hold your hands up and do scissors. Okay. I see one person who does it like me. So somebody does that. That's a first over there. I haven't never seen it like that before, but I grew up in Ireland and in the uk, where this is a rude gesture. So I do scissors like this, like you did. Are you from the U.K. yeah. And so it's like, you know, so I do scissors like this, but almost everybody does it with the thumb in. But if you take a look at the data set that I created for that, it was one person created the. The data set. And I brought my implicit bias along with me, because every image in the data set with scissors is like this. And most of you did scissors with the thumb in. And so it's a silly example, and it's a trivial example, but it also shows how easy bias can creep in. And when you're building something that's data driven in the same way as this is, having the skill to be able to spot biases like that, I think is a very, very important skill that will help you downstream because you don't want to deliver a model that will eventually cause offense or cause problems by delivering a model because you delivered it with your implicit bias along with it. In my case, it's relatively harmless. I did a talk at Google I O where I did the rock, paper, scissors example. And then afterwards, like, you know, somebody kind of was in the crowd, and I was answering some questions, and they just went, who does scissors like that? And that's how I learned about the problem. Nobody told me. And so I think, you know, being able to spot bias and being able to, like, understand the needs of your customers and to avoid bias and problems that can come as a result of bias is also a super, super important skill to learn.

You know, I think that academia and corporations are both great places to advance machine learning, and I've done both. Well, one of the nice things is that what used to be accessible only to the giant tech companies is now much more widely accessible, in fact. So let's see again. All right, since I'm here at Google, one of the first projects I led was the slightly infamous Google Cat, where we use 16,000, you know, Google's Cloud CPUs to learn to recognize a cat from YouTube. And it was a big deal at the time, right? Made the front page, the New York Times website. And so it was a big deal at the time. Now you could log into a cloud service, swipe your credit card, pay about $2,000 and replicate that experiment. So what used to be a big deal, even for large tech companies is now accessible, even it's not like $2,000 is cheap. That's the real money. But there's much more widely accessible. And second, I think that while scale is one way to drive progress, Bert, XLNET, GPT2, all of these were great examples of, you know, using many ideas that were, you know, published. Right. And somewhat known and scaling it much bigger and showing dramatic improvements in results just by scaling up data and compute. I think it's encouraging that we've not yet exhausted that particular engine for growth, but I think that there are plenty of paths for algorithmic innovation. I think a lot of research on small data really doesn't need that much scale. So I think the things you could do in academia and the things you can do in the large company are different. And maybe one example, one nice thing about great universities is that you could do almost, you know, faculty can do almost anything, right. We feel like doing within, you know, with very wide latitude. And so when, for example, when a Stanford is a fly autonomous helicopters, I just did that. Didn't have to worry about revenue model. Decided to work on AI for healthcare. Great partners on the same campus. Sometimes we can just talk to, to form those interdisciplinary collaborations or decide to do AI for climate change and just very quickly move into these different sectors. And so I think the academic freedom of universities and the freedom to work on a lot of different problems, even if you don't necessarily have the data and compute resources. I've been on both sides and love both sides and I think both have unique strengths.

All right, I think it's time for pizza. Yeah.

So pine AI. You can't do pine AI without pie.

We've got pizza pie and we've got sweet pie.

Thank you all.