Deep Learning State of the Art (2020)

Welcome to 2020 and welcome to the Deep Learning Lecture series. Let's start it off today to take a quick whirlwind tour of all the exciting things that happened in 17, 18 and 19 especially, and the amazing things we're going to see in this year, in 2020. Also as part of this series, it's going to be a few talks from some of the top people in learning in artificial intelligence. After today, of course, start at the broad the celebrations from the Turing Award to the limitations and the debates and the exciting growth. First and first, of course, let's step back to the quote I've used before. I love it. I'll keep reusing it. AI began not with Alan Turing a McCarthy, but with the ancient wish to forge the gods, a quote from Pamela McCordick. And machines who think that visualization there is just 3% of the neurons in our brain of the thalamocortical system, that magical thing between our ears that allows us all to see and hear and think and reason and hope and dream and fear our eventual mortality. All of that is the thing we wish to understand. That's the dream of artificial intelligence and recreate versions of it, echoes of it in engineering of our intelligence systems. That's the dream we should never forget. In the details, I'll talk the exciting stuff I'll talk about today. That's sort of the. The reason why this is exciting, this mystery. That's our mind, the modern human brain, the modern human as we know them today, know and love them today. It's just about 300,000 years ago, and the industrial revolution is about 300 years ago. That's 0.1% of the development since the early modern human being is when we've seen a lot of the machinery. The machine was born not in stories, but in actuality. The machine was engineered since the Industrial Revolution and the steam engine and the mechanized factory system and the machining tools, that's just 0.1% in the history and that's the 300 years. Now we zoom in to the 60, 70 years since the founder, the father arguably, of artificial intelligence, Alan Turing and the dreams. You know that there's always been the dance in artificial intelligence between the dreams, the mathematical foundations, and when the dreams meet, the engineering, the practice, the reality. So Alan Turing has spoken many times that by the year 2000 that he would be sure that the Turing Test, natural language, would be passed. It seems probably you said that, that once the machine thinking method is started, it would not take long to outstrip our feeble Powers they would be able to converse with each other, to sharpen their wits. At some stage, therefore, we should have to expect the machines to take control. A little shout out to self play there. So that's the dream. Both the father of the mathematical foundation of artificial intelligence and the father of dreams in artificial intelligence. And that dream again in the early days was taking reality. The practice met with the Perceptron, often thought of as a single layer neural network. But actually what's not as much known as Frank Rosenblatt was also the developer of the multi layer Perceptron. And that history zooming through has amazed our civilization. To me, one of the most inspiring things in the world of games. First with the great Garry kasparov losing to IBM G Blue in 1997, then Lee Sedol losing to AlphaGo in 2016. Seminal moments and captivating the world through the engineering of actual real world systems. Robots on four wheels as we'll talk about today. From Waymo to Tesla, to all of the autonomous vehicle companies working in the space. Robots on two legs, captivating the world of what actuation, what kind of manipulation can be achieved. The history of deep learning from 1943, the initial models from neuroscience thinking about neural networks, how to model neural networks mathematically, to the creation, as I said, of the single layer and the multilayer perceptron by Frank Rosenblatt and Frank 57 and 62 to the ideas of back propagation and recurrent neural nets in the 70s and 80s, to convolutional neural networks and LSTMs and bidirectional RNNs in the 80s and 90s, to the birth of the deep learning term and the new wave of the revolution in 2006, to the ImageNet and AlexNet, the seminal moment that captivated the possibility to the imagination of the AI community of what neural networks can do in the image and natural language space. Closely following years after to the development of the popularization of Gans generative adversarial networks with AlphaGo and AlphaZero in 2016 and 7 and as we'll talk about language models of transformers in 17, 18 and 19, those has been. The last few years have been dominated by the ideas of of deep learning in the space of natural language processing. Okay, celebrations. This year the Turing Award was given for deep learning. This is like deep learning has grown up. We can finally start giving awards. Yann Lecun, Jeffrey Hinton, Yosha Benja received the Turing Award for the conceptual engineering breakthroughs that have made deep neural networks a critical component of computing. I would also like to add that perhaps the popularization in the face of skepticism, and for those a little bit older have known the skepticism that neural networks have received throughout the 90s in the face of that skepticism, continuing pushing, believing and working in this field and popularizing it through in the face of that skepticism, I think is part of the reason these three folks have received the award. But of course, the community that contributed to deep learning is bigger, much bigger than those three, many of whom might be here today at mit, broadly, in academia, in industry. Looking at the early key figures, Walter Pitts and Warren McCulloch, as I mentioned, for the computational models of the neural nets, these ideas of thinking that the kind of neural networks, biological neural networks can have in our brain can be modeled mathematically and then the engineering of those models into actual physical and conceptual mathematical Systems by Frank Rosenblatt 57 again, single layer, multi layer in 1962 you could say Frank Rosenblatt is the father of deep learning. The first person to really in 62 mention the idea of multiple hidden layers in neural networks. As far as I know, somebody was correct me. But in 1965, shout out to the Soviet Union and Ukraine. The person who is considered to be the father of deep learning, Alexei Evokinenko and V.G. lapa, co author of that work, is the first learning algorithms on multi layer perceptrons, multiple hidden layers. The work on backpropagation, automatic differentiation, 1970, 1979 convolution Neural networks were first introduced. And John Hopfield looking at recurrent neural networks, what are now called Hopfield networks, a special kind of recurrent neural networks. That's the early birth of deep learning. I want to mention this because it's been a contentious space. Now that we can celebrate the incredible accomplishments of deep learning. Much like in reinforcement learning in academia, credit assignment is a big problem. And the embodiment of that almost a point of meme is the great Jurgen Schmidhuber. I encourage for people who are interested in the amazing contribution of the different people in the deep learning field to read his work on deep learning and neural networks. It's an overview of all the various people who have contributed besides Yann Lecun, Geoffrey Hinton and Yoshua Bengio. It's a big beautiful community, so full of great ideas and full of great people. My hope for this community, given the tension, as some of you might have seen around this kind of credit assignment problem, is that we have more not on this slide, but love there can never be enough love in the world, but general respect, open mindedness and collaboration and credit sharing in the community. Less derision, jealousy and stubbornness and silos, academic silos within institutions, within disciplines. Also, 2019 was the first time it became cool to highlight the limits of deep learning. This is the interesting moment in time. Several books, several papers have come out in the past couple of years highlighting that deep learning is not able to do the kind of the broad spectrum of tasks that we can think of the artificial intelligence system being able to do, like common sense reasoning, like building knowledge bases and so on. Rodney Brooks said, by 2020 the popular press starts having stories that the era of deep learning is over. And certainly there has been echoes of that through the press, through the Twitter sphere and all that kind of world. And I'd like to say that a little skepticism, a little criticism is really good, always for the community, but not too much. It's like a little spice in the soup of progress. Aside from that kind of skepticism, the growth of CVPR, ICLEAR, NeurIPS, all these conference submission papers has grown year over year. There's been a lot of exciting research, some of which I'd like to cover today. My hope in this space of deep learning growth celebrations and limitations for 2020 is that there's less both less hype and less anti hype, less tweets on how there's too much hype in AI and more solid research, less criticism and more doing. But again, a little criticism, a little spice is always good for the recipe. Hybrid research, less contentious counterproductive debates and more open minded interdisciplinary collaboration across neuroscience, cognitive science, computer science, robotics, mathematics, physics across all these disciplines working together. And the research topics that I would love to see more contributions to as we'll briefly talk about in some domains is reasoning, common sense reasoning, integrating that into the learning architecture, active learning and lifelong learning, multimodal, multitask learning, open domain conversation. So expanding the success of natural language to dialogue to open domain dialogue and conversation and then applications, the two most exciting, one of which we'll talk about is medical and autonomous vehicles, then algorithmic ethics in all of its forms, fairness, privacy, bias. There's been a lot of exciting research there. I hope that continues taking responsibility for the flaws in our data and the flaws in our human ethics and then robotics in terms of deep learning application, robotics. I'd love to see a lot of development, continued development, deep reinforcement learning, application and robotics and robot manipulation. By the way, there might be a little bit of time for questions at the end if you have a really pressing question. You can ask it along the way too. Questions so far, thank God. Okay, so first the practical the deep learning and deep RL frameworks. This has really been a year where the frameworks have really matured and converged towards two popular deep learning frameworks that people have used as TensorFlow and PyTorch or TensorFlow 2.0 and PyTorch 1.3 is the most recent version and they've converged towards each other, taking the best features, removing the weaknesses from each other. So that competition has been really fruitful in some sense for the development of the community. So on the TensorFlow side, eager execution so imperative programming how you would program in Python has become the default has been first integrated, made easy to use and become the default. And on the Pytorch side, Tor script allowed for now graph representation. So do what you're used to be able to do and what used to be the default default Mode of operation, TensorFlow allow you to have this intermediate representation that's in graph form. On the TensorFlow side, just the deep Keras integration and promotion as the primary citizen, the default citizen of the API of the way you interact with TensorFlow allowing complete beginners, just anybody outside of machine Learning to use TensorFlow with just a few lines of code to to train and do inference with the model. That's really exciting. They cleaned up the API, the documentation and so on and of course maturing the JavaScript and the browser implementation of TensorFlow TensorFlow Lite being able to run TensorFlow on phones mobile and serving. Apparently this is something industry cares a lot about of course is being able to efficiently use models in the cloud and Pytorch catching up with TPU support and experimental versions of Pytorch mobile. So being able to run a smartphone on their side this tense, exciting competition. Oh, and I almost forgot to mention we have to say goodbye to our favorite Python 2. This is the year that support finally in the January 1, 2020 support for Python 2 and Tensorflows and PyTorch support for Python 2 has ended. So goodbye print, goodbye cruel world. Okay, on the reinforcement learning front, we're kind of in the same space as JavaScript libraries are in. There's no clear winners coming out if if you're a beginner in the space. The one I recommend is a fork of open app baselines is stable baselines, but there's a lot of exciting ones. Some of them are really closely built on TensorFlow, some are built on Pytorch of course from Google From Facebook, from DeepMind, Dopamine, TF Agents, Tensorforce, most of these I've used. If you have specific questions, I can answer them. So stable baselines is the OpenAI baselines fork. As I said, this implements a lot of the basic deep RL algorithms, PPO, i2C, everything good documentation and just allows very simple, minimal few lines of code implementation of the basic the matching of the basic algorithms of the OpenAI gym environments. That's the one I recommend. Okay. For the framework world, my hope for 2020 is framework agnostic research. So one of the things I didn't mention is Pytorch has really become almost overtaking TensorFlow in popularity in the research world. We what I'd love to see is being able to develop an architecture in TensorFlow or develop it in Pytorch, which you currently can. And then Trent, once you train the model to be able to easily transfer it to the other From PyTorch to TensorFlow and from TensorFlow to PyTorch. Currently it takes 3, 4, 5 hours if you know what you're doing in both languages to do that. It'd be nice if there was a very easy way to do that transfer. Then the maturing of the Deep RL frameworks. I'd love it to see OpenAI step up DeepMind to step up and really take some of these frameworks to maturity that we can all agree on. Much like OpenAI gym for the Environment world has done and continued work that Keras has started and many other wrappers around TensorFlow started of Greater and greater abstractions allowing machine learning to be used by people outside of the machine learning field. I think the powerful thing about supervised, sort of basic, vanilla supervised learning is that people in biology, chemistry, in neuroscience, in physics, in astronomy can deal with a huge amount of data that they're working with and without needing to learn any of the details of even Python. So that that I would love to see greater and greater abstractions which empower scientists outside the field. Okay. Natural Language Processing 2017, 2018 was when the transformer was developed and its power was demonstrated most especially by bert, achieving a lot of state of the art results on a lot of language benchmarks, from sentence classification to tagging, question answering and so on. There's hundreds of data sets and benchmarks that emerged, most of which BERT has dominated in 2018. 2019 was sort of the year that the transformer really exploded in terms of all the different variations again starting from BERT xlnet. It's very cool to use BERT in The name of your new derivative of a Transformer. Roberta distilled Bert from hugging face, Salesforce, OpenAI's GPT2 of course, Albert and Megatron from Nvidia. Huge Transformer. A few tools have emerged. So one on Hugging Face is a company and also a repository that has implemented in Both PyTorch and TensorFlow a lot of these Transformer based natural language models. So that's really exciting. So most people here can just use it easily. So those are already pre trained models. And the other exciting stuff is Sebastian Ruder, great researcher in the field of natural language processing, has put together NLP Progress, which is all the different benchmarks for all the different natural language tasks, tracking who sort of leaderboards of who's winning where. Okay, I'll mention a few models that stand out the work from this year. Megatron LM from Nvidia is basically taking, I believe the GPT2 Transformer model and just putting it on steroids. Right? 8.3 versus 1.5 billion parameters. And a lot of interesting stuff there, as you would expect from Nvidia. Of course there's always brilliant research, but also interesting aspects about how to train in a parallel way. Model and data parallelism in the training. The first breakthrough results in terms of performance. The model that replaced BERT as king of Transformers is xlnet from CMU at Google Research. They combine the bi directionality from BERT and the recurrence aspect of Transformer xl, the relative positional embeddings and the recurrence mechanism of Transformer xl. So taking the bidirectionality and the recurrence, combining it to achieve state of the art performance on 20 tasks. Albert is a recent addition from Google Research and it reduces significantly the amount of parameters versus BERT by doing parameter sharing across the layers. And it has achieved state of the art results on 12 NLP tasks, including the difficult Stanford question answering benchmark of Squad 2. And they provide open source TensorFlow implementation including a number of ready to use pre trained language models. Okay, Another way for people who are completely new to this field, a bunch of apps right with Transformer is one of them from Hugging Face popped up that allows you to explore the capabilities of these language models. And I think they're quite fascinating from a philosophical point of view. And this has actually been at the core of a lot of the tension of how much do these transformers actually understand. Basically memorizing the statistics of the language in a self supervised way by reading a lot of text, is that really understanding? A lot of people say no until it Impresses and then everybody will say it's obvious. But write with Transformer is a really powerful way to generate text to reveal to you how much these models really learn. Before this yesterday actually just came up with a bunch of prompts. So on the left is a prompt. You give it the meaning of life. Here, for example, is not what I think it is. It's what I do to make it. And you can do a lot of prompts of this nature. It's very profound and some of them will be just absurd. You'll make sense of it statistically, but it'll be absurd and reveal that the model really doesn't understand the fundamentals of the prompt that's being provided. But at the same time it's incredible what kind of text it's able to generate. The limits of deep learning I'm just having fun with this at this point are still in the process of being figured out. Very true. Had to type this most important person in the history of deep learning is probably Andrew Eng. I have to agree. So this model knows what it's doing and I tried to get it to say something nice about me and that took a lot of attempts. So this is kind of funny is finally it did one. I said Lex Freeman's best quality is that he's smart. Said finally but never nothing but ever happens. But I think he gets more attention every Twitter comment ever. That's very true. Okay. A nice way to sort of reveal through this that the models are not able to do any kind of understanding of language is just to do prompts that show understanding of concepts and being able to reason with those concepts. Common sense reasoning trivia. One is doing two plus two is a three, five is a six, seven. The result is a simple equation. Four and two plus three is like he got it right and then it changed its mind. Okay, two minus two is seven. So on. You can reveal any kind of reasoning, you can do it blocks, you can ask it about gravity, all those kinds of things. It shows that it doesn't understand the fundamentals of the concepts that are being reasoned about. And I'll mention of work that takes it beyond towards that reasoning world in the next few slides. But I should also mention with this GPT2 model, if you remember about a year ago there was a lot of thinking about this 1.5 billion parameter model from OpenAI. The thought was it might be so powerful that it would be dangerous. And so the idea from OpenAI is when you have an AI system that you're about to release that might turn out to be dangerous in this case used probably by Russians fake news for misinformation. That's the kind of thinking is how do we release it? And I think while it turned out that the GPT2 model is not quite so dangerous, that humans are in fact more dangerous than AI currently that thought experiment is very interesting. They released a report on release strategies and the social impacts of language models that almost didn't get as much attention as I think it should. And it was a little bit disappointing to me how little people are worried about this kind of situation. It was more of an eye roll about oh, these language models aren't as smart as we thought they might be. But the reality is once they are, it's very interesting thought experiment of how should the process go of companies and experts communicating with each other during that release? This report think thinks through some of those details. My takeaway from just reading the report from this whole year of that event is that conversation on this topic are difficult because we as the public seem to penalize anybody trying to have that conversation. And the model of sharing privately confidentially between ML machine learning organizations and experts is not there. There's no incentive or a model or a history or a culture of sharing. Okay, Best paper from ACL the main conference for languages was on the difficult task of so we talked about language models. Now there's the task taking it a step further of dialogue. Multi domain task oriented dialogue. That's sort of like the next challenge for dialogue systems. And they've had a few ideas on how to perform dialog state tracking across domains, achieving state of the art performance on multiwas, which is a five domain challenging, very difficult five domain human to human dialogue data set. There's a few ideas there. I should probably hurry up and start skipping stuff on the common sense reasoning, which is really interesting. Is this one of the open questions for the deep learning community? AI community in general is how can we have hybrid systems of whether it's symbolic AI, deep learning or generally common sense reasoning with learning systems. And there's been a few papers in this space. One of my favorite systems from Salesforce on building a data set where we can start to do question answering and figuring out the concepts that are being explored in the question and answering. Here the question while eating a hamburger with friends, what are people trying to do? Multiple choice, have fun, tasty indigestion. The idea that needs to be generated there, and that's where the language model would come in, is that usually a hamburger with friends indicates a good Time. So you basically take the question, generate the common sense concept and from that be able to determine the multiple choice. What's happening? What's the state of affairs in this particular question? Okay, Alexa Prize again hasn't received nearly enough attention that I think it should have, perhaps because there hasn't been major breakthroughs. But it's open domain conversations that all of us, anybody who owns an Alexa, can participate in as a provider of data. But there's been a lot of amazing work from universities across the world on the Alexa Prize in the last couple of years and there's been a lot of interesting lessons summarized in papers and blog posts. A few lessons from Alquist team that I particularly like, and this is kind of echoes the work in IBM Watson with the Jeopardy Challenge is that one of the big ones is that machine learning is not an essential tool for effective conversation yet. So machine learning is useful for general chitchat when you fail at deep, meaningful conversation or actually understanding what the topic we're talking about. So throwing in chit chat and classification, sort of classifying intent, finding the entities, detecting the sentiment of the sentences, that's sort of an assistive tool. But the fundamentals of the conversation are the following. So first you have to break it apart sort of conversation is a, you can think of it as a long dance. And the way you have fun dancing is you break it up into a set of moves and turns and so on and focus on that sort of live in the moment kind of thing. So focus on small parts of the conversation taken at a time. Then also have a graph. Sort of conversation is also all about tangents. So have a graph of topics and be ready to jump context from one context to the other and back. If you look at some of these natural language conversations that they publish, it's just all over the place in terms of topics. You jump back and forth. And that's the beauty, the humor, the wit, the fun of conversations. You jump around from topic to topic and opinions. One of the things that natural language systems don't seem to have much is opinions. If I learned anything, one of the simplest way to convey intelligence is to be very opinionated about something and confident. And that's a really interesting concept about. And in general, there's just a lot of lessons. Oh, and finally, of course, maximize entertainment, not information. This is true for autonomous vehicles, this is true for natural language. Conversation is fun, should be part of the objective function. Okay, lots of lessons to learn there. This is really the Lobner Prize, the touring test of our generation That's I'm excited to see if there's anybody's able to solve the Alexa prize again. Alexa prize is you're tasked with talking to a bot and the measure of quality is the same as in a lot of enterprises just measuring how good was that conversation. But also the task is to try to continue the conversation for 20 minutes. If you try to talk to a bot today and you have a choice to talk to a bot or go do something else, watch Netflix, you last probably less than 10 seconds, you'll be bored. The point is to continue trapping you in the conversation because you're enjoying it so much. And the 20 minutes is. That's a really nice benchmark for passing the spirit of what the Turing test stood for. Examples here from the Alexa prize from the Alquist bottom so the difference in two kinds of conversations. So Alco says, have you been in Brazil? The user says, what is the population of Brazil? Alco says it is about 20 million. User says, well okay, this is what happens a lot with like I mentioned, multi domain conversation is once you jump to a new domain you stay there. Once you switch context, you stay there. The reality is you want to jump back back and continue jumping around like in the second most more successful conversation. Have you been in Brazil? What is the population of Brazil? It is around 20 million. Anyway, I was saying have you been in Brazil? So jumping back in context, that's how conversation goes tangent to tangent and back quickly. There's been a lot of sequence to sequence kind of work using natural language to summarize a lot of applications. One for make iclear that I wanted to highlight from Technion that I find particularly interesting is the abstract syntax tree based summarization of code. So modeling computer code in this case sadly Java and C in trees in syntax trees and then operating on those trees to then do the summarization in text. Here an example of a basic power of two function on the bottom right in Java the code to sec summarization says get power of two. That's an exciting possibility of automated documentation of source code. I thought it was particularly interesting. And the future there is bright. Okay. Hopes for 2020 for natural language processing is reasoning. Common sense reasoning becomes greater and greater part of the transformer type language model work that we see in the deep learning world. Extending the context from hundreds or thousands of words to tens of thousands of words. Being able to read entire stories and maintain the context which transformers again with XL net transformer Excel is starting to be able to do. But we're still far away from that long term lifelong maintenance of context dialogue, open domain dialogue forever. Since Alan Turing to today is the dream of artificial intelligence being able to pass the Turing test and the dream of sort of natural language model. Transformers are self supervised learning and the dream of Yann Lecun is.