State of AI Report 2020 (review)

I'm so, so excited. You know why? Partially because it's Halloween. So if we go here and check out the date, 31st of October, 2000 and— oh, is my face in the way there? Doesn't really matter. But anyway, it's Halloween. Little piece of advice. If you are getting dressed up, I'm not really dressed up. If you are getting dressed up and going to some sort of party for Halloween, make sure that everyone else is getting dressed up too. Because if you go there and you're dressed as like a goblin and everyone else is in plain clothes, you're going to stand out. Now that might be depending on how inclined you are, that might be a good thing. Anyway, that's not what this video is about. Happy Halloween, by the way. We are going to go through, I've been waiting till the end of the month to go through this, the State of AI report for 2020. Woo! How exciting. So this actually came out at the start of the month, um, October 1st, 2020. Again, we're at October 31st, 2020. Now what I figured, uh, be fun to do is just go through it for probably an hour or so and then actually let's get a stopwatch out. Stopwatch. 6:00. There we go. Let's go through it for the next hour or so. Um, of course if you don't want to listen to me talk over this, you can go read it through yourself. Www.stateofai.com. I mean, stateof.ai. Big shout out to Nathan Benh for creating this and Ian Hogarth. So without any further ado, let's jump into it. By the way, I'm just offering my commentary on here. I'm not going to claim that I'm some sort of expert, know everything, the insides and outs of what's in this report. I actually don't know what's in this report. So yeah, take anything I say with a grain of salt and be sure to do your own research because that's what I'm excited to do. That's what I use these things for. I use them as information and then I go research it myself to figure out more. But let's get started. Thank you. Thank you, Nathan Benach, Benach and Ian Hogarth. Just make sure my head is in there. Where we got here? Anyway, I'll change that as we go through. So introduction. So we've got some definitions about AI, multiple multidisciplinary field of science and engineering whose goal is to create intelligent machines. Yes, yes, yes. These are the key dimensions in the report. So I did actually go through last year's report and it was phenomenal. So I am really excited to go through this year's. So it's broken down into research, talent, industry, politics, and predictions. How many slides do we have here? Holy crap. Okay, Daniel, you're probably going to have to be quick. No rush. We've got some contributions. Jack Clark, if you haven't signed up to his newsletter yet, make sure you do. Jeff Ding, Chip Hewin. Chip Hewin also has an amazing blog. Check her blog out. Andrej Karpathy. Some names that I'm not going to pretend that I can pronounce. Some definitions. I'm not going to read all them out. I'll let you read them out. Algorithm, model, supervised learning, unsupervised learning, transfer learning, natural language processing. An executive summary. Oh, this might be worth going through. Let me just— I'm going to shrink myself down real small. That way it's a focus on the content and not just my face. All right. So what do we have here? Research: A new generation of transfer language models are unlocking NLP use cases. So yeah, I think that's, that's a big thing, right? Transformers are kind of the architecture of choice for NLP at the moment. But that being said, I did watch a Weights and Biases podcast with someone who's working in the NLP field, and they found that Transformers were, were too big to use in practice for their, for their application. So again, it's about tailoring the tools that you have to the problem that you have. Huge models, large companies, and massive training costs dominate the hottest area of AI today. NLP. Yep. GPT-3. I think that costs like $11 million to train. Biology is experimenting its AI moment from medical images to genetics, proteomics, chemistry to drug discovery. AI is mostly closed source. Only 15% of papers publish their code, which harms accountability and reproducibility in AI. 15%. We need to step those numbers up, team. Come on. Talent, industry, politics. Let's get into the slides. Scorecard. Reviewing the predictions from 2019. New natural language processing companies raise $100 million in 12 months. So what happens with these reports is at the end they make predictions for the next year. I probably might make my own prediction. Why don't we do that? Um, did they get this correct? Yes, they did. So a fair few companies are getting funding there in the NLP space. No autonomous vehicle companies drive above 15 million miles. Waymo, 1.5 million. Cruise, nearly a million. Baidu, 108K miles. I wonder, is Tesla a self-driving car company? Could you include them in there? Cause they've done a fair few. I don't know how many. Privacy preserving ML is adopted by an F2000 company. Yup. Machine learning. Well, that kind of makes sense. Like it's a privacy, I think that's like the trend of 2020 is privacy. I mean, that's where Apple's differentiating itself from things like Facebook, Google, Amazon and whatnot. I mean, Google and Facebook are sort of taking a step to go, you know what, we can make your data private if you want. Apple sort of front and foremost, everything's private from the get-go. But it kind of makes sense just going forward. Like if you could do ML privately, why not just do it? Unis build de novo undergrad AI degrees. So this is like a 'If a uni has, a university has a dedicated AI specialization,' which I mean, it's kind of the buzzword, like you could use AI, like I mean, at the moment you could just dedicate it towards like a degree that teaches mathematics and coding and data processing. So they got that right. 'Google has a major breakthrough in quantum computing.' They sort of got that right. And 'Governance of AI becomes a key issue and one major AI company makes substantial governance model change.' Nope. Business as usual. Research. Let's do it. AI is less open than you would think. Only 15% of papers publish their code. See, that's, that's pretty astonishing to me. Oh, and this is, this is a source from Papers with Code. Um, great website by the way. You need to check that one out. So research code paper implementations are important for accountability, reproducibility, and driving progress in AI. I totally agree. So there's been times where I've read a paper and I'm like, this is amazing results. And then gone, you know what, I'm going to go look at their code. And the code wasn't available. I'm like, oh, you know what, I'll spend a week trying to reproduce these results. I actually did this in the past 2 weeks. And I thought I did as close as possible to the paper itself. And I was getting like 2 percentage points of F1 score below what they said they got. And I thought I'd got everything correct. So I emailed the authors. I'm still waiting back on a reply. Notable organizations that don't publish all of their code are OpenAI and DeepMind. So this is, yeah, OpenAI, you kind of got it in your name, OpenAI. But if you're keeping all your code and research private, well then is that really open? Again, I'm not the one making the decisions to put these things out there. But in terms of the space itself, I know a lot of these companies are sort of private entities as well. So a lot of their work in AI is proprietary. It's how they kind of make their money. So if another company was to leverage that, they would become competition. But again, it seems going forward, if people like you or I could start working on the things that large companies are working on right now, just in a room like this, I mean, chances are you just increase the opportunity for new discoveries. Papers with Code tracks openly published code and benchmarks model performance. Yeah, Papers with Code is a great website. I love that website. So it's tracking state of the art. Facebook's PyTorch is fast outpacing Google's TensorFlow in research papers. Oh, which tends to be a leading indicator of production use down the line. Wow, that's pretty cool. So where's percent of PyTorch papers of total TensorFlow, PyTorch papers. Holy crap, 75% above 50. So 55— oh, there we go, 55% of them have switched to PyTorch. You know what, the more and more I hear about PyTorch— so I'm, I'm more versed in TensorFlow than I am PyTorch. I mean, I've been a long advocate for like, you should, you should know a little bit of both. But the more I'm seeing this is PyTorch is, PyTorch is coming up, you know. Well, this is clear. I mean, it's more than coming up, it's taking over. So I think I've got some learning to do over the summer, but that's pretty cool. PyTorch is also more popular than TensorFlow in paper implementations on GitHub. 47% of these implementations are based on PyTorch versus 18% for TensorFlow. Wow. That's, that's pretty full-on. I mean, JAX is another framework, is a Google framework that is more math-friendly and favored for work outside of convolutional models and transformers. MXNet, Caffe2. Caffe2 is like a, uh, I think it's in pure C++, maybe. Only one repo. Um, so yeah, that's, that's very interesting to see, is that PyTorch is becoming more, more popular. Well, it is more popular than TensorFlow. But that being said, does the tool you're using get the job done? You don't necessarily have to use a tool because it's more popular. Language models. Welcome to the billion parameter club. We're talking billies now, baby. Who's going to be the first language model company or the first company to build a language model with a trillion? Microsoft? OpenAI? DeepMind? Google, who's gonna hit the trilly first? I wanna see this. Trilly. So first models, large companies and massive training costs dominates the hottest area of NLP today. Look at that, GPT-3, 175 billion parameters. Where's BERT? Is BERT on here? BERT Large. So BERT Large, I mean, that was the state of the art. I mean, that kind of cascaded the whole transformer revolution, right? BERT, 340 million parameters. And that came out at the end of 2018. So 2018 to 2020 onwards, look at that. We have quite a large step change, an order of magnitude or more. 10x, 100x, nearly, nearly 2 orders of magnitude here. Bigger models, datasets, and compute budgets clearly drive performance. Empirical scaling laws of neural language models show smooth power law relationships, which means that as model performance increases, the model size and amount of computation has to increase more rapidly. So this is, this is very interesting research. And it's kind of clear, like what you've seen over the past few months in the NLP world is basically if you have the ability to build a bigger model and the ability to run ridiculous amounts of compute, you're going to win. Is that the, is that like the way to go forward? Like if we, if you think about Moore's Law of compute, it's always going to increase. Some would argue that it's dead. I don't know on that. Enough. But if you just imagine compute power is continually going to increase, um, is that how we get to, to more and more state-of-the-art NLP models? We just— as compute power increase, the models get better. Not entirely sure. Tuning billions of model parameters costs millions of dollars. Yeah, here we go. This is— so this is where it's like limited to— like, if you— unless you're— here we go— OpenAI's 175 billion parameter GPT-3. Could have cost tens of millions to train. Tens of millions. That's a pretty costly experiment. Like, I mean, imagine being the engineer at OpenAI who like hit the enter key to like kick off that experiment. It's like, all right, have we got all the code lined up? We're going to train this full big dog, 175 billion parameter model, and it's going to cost 10 to 20 mil. So did you get— did you make sure you've tuned all your hyperparameters? Did you make sure you're saving the model results? Budget was $10 million. So yeah, that's like what we said. So Google's T5 costs well above $1.3 million for a single run. I mean, yeah, this is like, these are the kind of experiments that only large-scale companies can run. So what I'm excited for, like going forward, is research like this, but making it like super, super efficient. That's what I'm really excited for. Like, because I mean, of course I'm excited to see like large companies throw millions of dollars at training runs, but I'd like to be able to see like someone, you know what, we got 95% of the results of GPT-3 on a single GPU. Is it possible? I don't know. Um, to achieve the needed quality improvements in machine translation, Google's final model trained for the equivalent of 22 TPU v3 core years, or 5 days with 2048 cores nonstop. Whoa, quality gain. Okay, that's what you want. So you have to train for a ridiculous amount of time. Well, in terms of computation cost to get— oh, where's the improvement from? What's that? An 8, 2048 divided by 128, 16. So 16 times, you need 16 times more compute to get double the performance gain. To me, that's, that doesn't really add up, you know, 16 times more compute, but only double the performance gain. We're rapidly approaching outrageous computational, economic, and environmental costs to gain incrementally smaller improvements in model performance. Yeah, this is what I'm saying, right? It's like if we're, Without major new research breakthroughs, dropping the ImageNet error rate from 11.5% to 1% would require over $100 billion billion. Oh my God. $100 billion billion. Where we got? Today the target, today the error rate is 11.5%. So right now to train state-of-the-art on ImageNet, it looks like it costs, can you see that? $10. 10 to the power of 6, that's a million. Wow, I can't zoom in here. You're just gonna have to check out this slide. So yeah, that's what I'm saying, where, where although massive, massive models are like exciting to like read about and see or whatever, um, they're just getting like outlandishly expensive and costly on the environment to train. Like if they're releasing so much there, I mean, you could argue that yeah, Google's data centers are all, uh renewable energy. I'm not entirely sure if they are, but I think they're, they're moving that way. Um, but still massive increase in costs for not that great an increase in performance. A larger model needs less data than a smaller peer to achieve the same performance. Yeah, this is very interesting research as well. So this has implications for problems where training data samples are expensive to generate, which likely confers an advantage to large companies entering new domains with supervised learning-based models. Yeah, so this graph here, what it's showing, we've got test loss here, tokens processed. If you have 10— so a billion parameters, 10 to the power of 9, your, your test loss is lower. But if you only have 1,000 parameters, your test loss is higher. So larger models require fewer samples to reach the same performance. The optimal model size grows smoothly with the loss target and compute budget. So what do we got here? Compute, PF days, line color indicates number of parameters. This is actually beautiful graphs once you sort of dig in and figure out what they're saying. So the larger— this is the large model here. Compute efficient training stops far short of convergence. So the larger the model, the few fewer samples it requires. And as the model size increase, the amount of compute increases. That makes sense. Low resource languages with limited training data are a beneficiary of large models. So Google made use of their large language models to deliver high quality translations for languages with limited amounts of training data. For example, Hanza and Uzbek. This highlights the benefits of transfer learning. Yeah, so this is a big thing. Like, it's very— and this is my own like cognitive bias, like in the field of natural language processing and just just in life in general, is you kind of get narrowed down. Like if I'm fluent in English and I read English and I write English, so I kind of, if I'm not interacting with languages like Hanza and Uzbek, I don't have experience with what it's like to work in those sort of, if I was working on a natural language processing problem with a language like that. And so I really, really do like, even though I personally don't work with many foreign languages at all. I really do like seeing research where it's including languages with, with limited training data or that aren't as popular as English. So although this is not my field of work, I love seeing this stuff. Even as deep learning consumes more data, it continues to get more efficient. Okay, so this is, this is good to hear. Since 2012, the amount of compute needed to train a neural network to the same performance on ImageNet classification has been decreasing by a factor of 2 every 16 months. Wow. Okay. Two distinct areas of compute in training AI systems. So the first era was prior. I think this is about 2000 and— let's see, 2012, this line here. And the modern era. So I'm not sure what this axis is, but it just looks like the modern networks are increasing compute, but the training efficiency factor. So you got over here, I don't know if you can see that, but that's EfficientNet-B0, which is a computer vision model, is getting— has a— I'm not sure how this efficiency factor is measured, but it's right up the top there compared to an architecture like MobileNet V2, which was state of the art like 2 years ago. So this is good to see. Yet for some cases like dialogue, Small data-efficient models can trump large models. Yeah, so this is what I was saying. I am— PolyAI, I'm not too familiar with, with their company. Let's, uh, let's get a PolyAI. Let's see what they do. Enterprise-ready voice assistance for customer service. Okay, so you could call up— oh wow, that's cool. Accept human level conversations. All right. Wonderful. So they're doing— oh wow, we're only up to slide 23 and we're like 20 minutes in. Faster, Daniel, faster. Okay. So PolyAI is crushing it in how good their conversational AI is. So intent accuracy. Oh, they're outdoing BERT. And amount of data. So on low data, they're still getting about 80% accuracy. Yeah, this is the stuff that I mentioned before I really like to see is that like, hey, we got sometimes like, especially when I was working at my, my previous role, if a really efficient model, so say it took 100x less compute but would get 95 or 97, 98% of the results of a model that took 100 times more compute, we would usually go with the more efficient model even though its results weren't as good. Because on the long run, even though its results weren't as good, it was easier to deploy, it actually ran in production, yada yada ya. So I like seeing these things, like getting good performance with not as much data. So what have they done? PolyAI, a London-based conversational AI company, open-sourced their ConvRT model, a pre-trained contextual re-ranker based on transformers. Transformers coming at it again. Their model outperforms Google's BERT model in conversational applications, especially in low-data regimes, suggesting BERT is far from silver bullet for all NLP tasks. BERT, although is an amazing model. Oh, there we go. Model size, BERT 1.3 gigabytes. Yes, that's the exact problem we ran into. We tried to use BERT, it was too big. And now ConvertRT, Convert, ConvRT is only 59 megabytes. That's what's up. Like an order of magnitude, two, nearly two orders of, order of magnitude and a half times smaller. A new generation of transformer language models are unlocking new NLP use cases. I mean, GPT-3 is like— you've obviously seen their, um, oh, here we go, code generation, gpt3examples.com. Here's a sentence describing what Google's homepage should look like, and here's GPT generating the code for it nearly perfectly. Okay, so describe a layout. 2 light gray buttons that say search Google and I'm feeling lucky with padding in between them. And it recreated Google's website. That's wild. Absolutely wild. You know, someone was asking, like someone was arguing. I heard that GPT-3, okay, it's— it can take like instructions like this, but it can't tell you why it produced like the output that it did. And I would argue as well, if you think back to humans, if you give someone— humans like an instruction like Walk up this hill. And then the person walks up that hill. It's like, why did you walk on the left side of that hill? And so you don't really know. Anyway, computer, please convert my code into another programming language. This is really cool. So I don't know C++. I know Python. But even if I look at that, I would have to, um, I would have to go through it. That would take me a significant amount of time to like reverse engineer and figure out, you know. But apparently an unsupervised machine translation model trained on GitHub projects with 1,000 parallel functions can translate 90% of these functions from C++ to Java and 57— so 90% of C++ to Java and 57% of Python functions into C++ and successfully pass unit test. What? That's wild. And see, this is what excites me about machine learning, right, is is if you can frame your problem really, really well, like what inputs do you have? Okay, a bunch of code on GitHub. And what outputs do you want? In other words, translate this code into this code language. If you can frame that really well and then, you know, sort of like the basic premise of unsupervised learning, supervised learning, what type of problem you have, you can very quickly or not very quickly, but you can start to bridge the gap between their inputs and the outputs you desire using machine learning. It just requires you to frame your problem very well. Once you get that, I mean, you're seeing examples of this all over the place, is turning code into a translation problem, which essentially is what it is, right? If I was a Python expert and a C++ expert, and I wanted to convert this function here, sum of k subarray, into C++, I could go through this and translate it just like I would translate something like English to French. Computer, can you automatically repair my buggy programs too? Oh, here we go. We're all done. Computers are now starting to repair code. Asota is set on DeepFix, which is a program repair benchmark for correcting Correct intro programming assignments in C. Wow. Yeah. So unless you can write bug-free code, you're going to get overtaken by deep learning algorithms. No, I don't think so. I think we're still a little while off of a— maybe, who knows, GPT-4 might be able to write fluent Python code. NLP benchmarks take a beating. Over a dozen teams outrank human GLUE baseline. That's wild. It was only 12 months ago that the GLUE benchmark was beat by 1 point. Now SuperGLUE is in sight. So I believe GLUE is like a way to evaluate your NLP model. GLUE NLP. So GLUE benchmark. Here we go. GLUE, the General Language Understanding Evaluation. GLUE benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems. I'll let you read about that. Now, oh, and there's also super glue. There we go. I'm imagining super glue is just a harder version of glue. Um, and 12 months ago, one company or one research group broke the human baseline and see, that's what happens in a cascade is usually when someone sort of makes a little breakthrough, you're going to watch a whole bunch of other people do it as well because it's like that kind of a follow the leader effect, you know? So that's really cool. NLP is just taking over. Let's step this up. Let's, uh, we're only up to slide 28. What's next after SuperGLUE? Let's, let's do a few of these a bit faster. What's next after SuperGLUE? More challenging NLP benchmarks zero in on knowledge. Yeah. So again, if we're trying to evaluate these, these super crazy big models, we need harder, harder ways to evaluate them and harder tests for them to if they're just blazing through the old tests. I mean, future models are just, well, if you're getting all these models that are just crushing them, well, is a test really helpful? Transformers' ability to generalize is remarkable. It can be thought of as a new layer type that is more powerful than convolutions, whew, because it can process sets of inputs and fuse information more globally. Yeah, so this is wild. GPT-2, which was originally a natural language model, was used for image completion. So it was fed, say, half an image and then it completed it to this and then this. That is wild. Actually, I want to show you a paper that I found. What's it called? Images Transformers for Computer Vision. Am I going to see it? Oh, this is one. End-to-end object detection with Transformers. Go check that out. But I believe I saved it to my pocket. There we go. Bring this over here. An image is worth 16 by 16 words. So Transformers for image recognition. What does it say here? When pre-trained on large amounts of data and transferred to multiple recognition benchmarks, Vision Transformer attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. That is a major key right there. Could Transformers be the architecture to solve them all? Who knows? And it's actually, it's actually worth diving into because the way I view Transformers is just a whole bunch of attention layers stacked on top of each other. Um, very primitive description there, but go check it out. Biology is experimenting— experiencing its AI moment. Moment. Over 21,000 papers in 2020 alone. This is what I'm, I'm really excited for, the crossover of AI and health. Um, we've got a bunch more papers coming. Who knows what's going to come into it, but that's, it's exciting to see the research grow from physical object recognition to cell painting, decoding biology through images. So now we have a large image database of Cells treated with very various chemical agents. So instead of getting great results on ImageNet, you can now run your computer vision models on different images of cells and find some insights there. Deep learning on cellular microscopy accelerates biological discovery with drug screens. This is wild. Use deep learning to just to discover new drugs. Oh my goodness. Hard word. Ophthalmology advances as a sandbox for deep learning applied to medical imaging. Let me, I don't know what this word is. How do we spell that? Ophthalmology. What does this do? What's an ophthalmologist do? An ophthalmologist is a specialist doctor who diagnoses and manages eye conditions and disorders of the visual system. Hmm. I thought that was an optometrist. Maybe I'm getting that wrong. Anyway, AI-based screening mammography reduces false positives and false negatives in two large clinical, clinically representative datasets from US and UK. Yeah, I think I saw this, this research. So Google Health and DeepMind, um, the AI system and its ensemble of three deep learning models operating on individual lesions, individual breasts, and the full case was trained to produce a cancer risk score between 0 and 1 for the entire mammography case. The system outperformed human radiologists and could generalize to US data. That is really cool. When trained on UK data only. So this is, I mean, this kind of goes to show is not only are we all like human, is that if, if one, like, country has some sort of breakthrough, so, so on mammography for US data, we can bring that to the UK, or vice versa. This is what happened here. The, the model could generalize to US people when it was only trained on UK data. So that's the exciting thing about biology, is that although we are all different, this, this kind of shows that it can a use case developed in one area of the world can be adjusted for another area of the world. Causal inference, taking ML beyond correlation. Most ML applications utilize statistical techniques to explore correlations between variables. This requires that experimental conditions remain the same and that trained ML system is applied to the same kind of data as the training data. Yeah, this is a tough one, right? So it's like if you train a machine learning algorithm on a certain training set and you deploy that into production and the data that you're, you're getting in your, your production system is not from the same distribution as your, your training dataset, well then your model is not going to perform as well. But again, this ignores a major component of how humans learn by reasoning about cause and effect. So if we learn something right, if you go If you learn how to ride a bike and it's a small bike and then you go to try and ride a bigger bike at the start, it might not be that great, but then you can, you can leverage the skills you learned on the smaller bike to adjust to the bigger bike. So if you imagine like an ML system trained on one set of training data and then it finds out that it's in its, it's in test production on the test set, it's not performing as well. How can it leverage what it's learned from the training data to improve its performance on the test data. So many pioneers in the field, including Judea Pearl, pictured, and Yoshua Bengio, believe that this will be a powerful new way to enable ML systems to generalize better, be more robust, and contribute more to decision-making. Causal reasoning is a vital missing ingredient for applying AI to medical diagnosis. Yeah, that's really important, right? So you want causal reasoning when you're making some sort of medical diagnosis. Model explainability is an important area of AI safety. A new approach aims to incorporate causal structure between input features into model explanations. Okay, so this is okay. So it says Shapley values have a flaw. Asymmetric Shapley values are proposed to, to incorporate this causal information. So if we look up SHAP values. I think SHAP values— I've only had a little bit of experience. SHAP, here we go. SHAP is a game theoretic approach to explain the output of any machine learning model. It connects optimal credit allocation with local explanations using the classic Shapley values from game theory and their related extensions. So yeah, if you want to explain the outputs of your machine learning model, you're probably going to to look into SHAP, and apparently there's a better version now, an improved version, asymmetric Shapley values. Reinforcement learning helps ensure that molecules you discover in silico can actually be synthesized in the lab. This helps chemists avoid dead end, dead ends during drug discovery. Yeah, there'll be nothing worse than finding a promising drug and then figuring out, you know what, we can't actually make that in the lab. Have you, have you, have your desired molecule? ML will generate a synthesis plan faster than you can. Wow. So this is, yeah, where, well, this is, I mean, this is more so chemistry than biology, isn't it? Again, repurposing the transformer architecture. I need to get, I need to get my head fully around the transformer architecture. That may be a future project for me going forward is using transformers for just, just figuring out how to apply them to almost anything. Graph neural networks. I've heard of these. I haven't looked into them much, but I hear they're making some great insights, particularly on 3D data. So i.e., non-Euclidean space. So yeah, most deep learning methods focus on learning from 2D input data. See, I get confused when I read that because I imagine tensors to be multidimensional. But if you think about it, yeah, an image is only length by width. A sequence is like one dimension. Hmm, I need to look into more into graph neural networks. Graph neural networks learn to guide antibiotic drug screening, leading to new drugs in vivo. Oh, I always get these mixed up. In vivo. Studies that are in vivo are those which the effects of various biological entities are tested on whole— okay, in vivo is on actual living organisms. Enhancing chemical property prediction using graph neural networks. Whoa. So this is, yeah, this is definitely something that I need to, um, so here's what I do when I see something that I don't know. I go graph neural networks. It's amazing what you can learn if you Google it. A gentle introduction to graph neural network basics. Here we go. I'm going to bookmark this and read that later. AI sifts through chemical space using DNA-encoded small molecule libraries. Dell. Again, graph neural networks trained on Dell. So I imagine there's some— so this is again, this is, this is the formulating your problem. What are your inputs? So in the case of graph neural networks, I'm assuming the data is, is not suitable for traditional deep learning neural networks. So they had to adjust the bridge between the desired inputs and outputs of an ML system. Language models show promise in learning to predict protein properties from amino acid sequences alone. I remember when I first started learning about language models, I thought that Well, if we imagine DNA is just a sequence of A's, C's, G's, and T's, why can't we just train a language model on that and then figure out sequences in there? Of course, way easier said than done, obviously. But this is showing some promise. Well, as it says, shows promise in learning to predict protein properties from amino acid sequences alone. So if you have ACGTTTAGCTTA, can you predict what kind of protein is going to come from that sequence? COVID-19 analyzing symptoms from over 4 million contributors detects novel disease symptoms ahead of public health community and could inform diagnosis without test. Wow. So loss of smell is the most predictive symptom of COVID-19. If you've lost your smell, go and get a COVID test. Oh, Zoe, I love that company. What is it? Joinzoe.com. I actually applied for a job here. I got denied. So I believe they're working on personalized nutrition. Yeah, here we go. Learn the groundbreaking science of predicting nutrition. That's, that's what I'm slowly moving towards. You know, I'm not in a rush, not in a rush. The trajectory is going that way. We come here, might have to start my own company. Drug discovery goes open source to tackle COVID-19. This is a rare example of where AI is used, actively used on a clearly defined problem. It's part of the COVID-19 response. I think this is, this is, yeah, one of the cool things that like open sourcing AI techniques can be used for. So we, at the start of the video, we talked about how— what is stopwatch? We talked about how only 15% of research papers' codes are available. But if we have like a global scale problem, like something like COVID-19, then in that case it makes sense to just open source everything and get all hands on deck basically. You never know, some teenager in a room who's just an absolute prodigy, just spending 8 hours a day working on some sort of problem might stumble upon some sort of breakthrough. Of course rare, but that's, if you look back in history, that's, that's how a lot of discoveries have been found. And if you ever say, oh, I'm too young, that's a, that's a lame excuse. Whatever age you are, you can, you can get into these things. I mean, it's, it's code and it's math. Math is the language of nature. Missed out on strawberries and cream this year. A controllable synthetic video version of Wimbledon tennis matches. What? Combining a model of player and tennis ball trajectories, pose estimation, and unpaired image-to-image translation to create a realistic controllable tennis match. Hold on. A realistically controllable tennis match video between any players you wish. Vid2Player. I gotta check this one out. Vid2Player Stanford. Here we go. What's this going to show me? That can't be. Is that fake? What? You're kidding me. This is generated. Okay, that's wild. This is where— what do they use for that? I'm gonna have to read that. I'll save that. This is what I do. I save things and I tag it with ML to pocket. Great app. Um, we're only 47 through. Let's do a burner. Attention turns to computer vision. Boom. Transformers for computer vision and object detection. There we go. DETR, detection transformer. DETR uses 2D images, features from a CNN, flattens them into a sequence and then uses transformers to model pairwise interactions between the features. Holy crap. That's so cool. Computer vision predicts where an agent can walk behind what is seen. I think, yeah, computer— there's— so NLP is having its moment, but there's still a lot like computer vision is just getting wild lately. The things you can do with one camera. I mean, look at like the new iPhones have like that depth thing and whatever. Will that be helpful? Who knows? That's again, if you can define the inputs and outputs of your system, you can work out the bridge between. Computer vision learns stereo from single images. So that's depth from single images. Enabling the use of consumer-grade 360 cameras in construction using deep learning. Learning dynamic behaviors through latent imitation, imagination. Wow. So Dreamer is an RL agent that solves long horizon tasks. So things that are in the future. or tries to predict steps that are in the future. That's how I think of long horizon. from images purely through an imagined world. This reminds me of World Models, great paper a few years ago. Dreamer predicts both actions and state values by training purely in an imagined— oh my God— an imagined latent space. I think this is where we're also going to see a big step in the coming years is latent space. So synthetic a massive improvement to synthetic data learning. Learning to drive by predicting and reasoning about the future. Predicting how a given driving situation will unfold, ranging from what the driver will do and the behavior of dynamic agents in the scene, can help autonomous agents learn how to drive from videos. Yeah, so This is what I think. Yeah, it can be. If you imagine self-driving cars, to me the data labeling problem, if you're collecting data for self-driving cars, you've got so many different variables. You've got lights, cars, roads, lanes, et cetera, et cetera. A human, of course, when I drive, I'm like registering those things. But one way that I heard that I think Comma AI, especially George Hotz, is talking about training it is to, take all the training data and the labels are what a human driver does in a given situation rather than labeling every single car. And I think that's a really cool approach to it. So I think this is kind of what, what this is talking about, learning to drive by predicting and reasoning about the future, what the driver will do. Visual question answering about everyday images. Again, this is really cool. This is helpful. This is from Laura is an approach that reads text in an image and jointly reasons about the image and text content to answer a question from a fixed or by selecting one of the OCR strings derived from the image. So this is, yeah, this is really cool. It's like, can you take a photo of something and not only grab the image details, but grab the text details that are in there? Because that's so much of what we— if you imagine a food package. What's the image of the food and what is the information on there? Can you capture that in, in one go? Learning a multi-purpose generative model from a single natural image. Wow. On-device computer vision models that won't drain your battery. Yeah, this is what— this, this is what gets me really excited is EfficientDet D7. So EfficientDet is a object detection algorithm achieves state-of-the-art on COCO object detection tasks with 4 to 9 times fewer model parameters than the best-in-class and can run 2 to 4 times faster on GPUs and 5 to 11 times faster on CPUs than other detectors. So this is, this is the stuff that I like to see, um, is getting incredible results, way more efficient. I mean, who— let's be real, who doesn't like to see that? Evolving Entire Algorithms from Basic Mathematical Operations Alone with AutoML Zero. That was a wild paper. So it starts off AutoML Zero from how I understand it was, let's look this up. It starts with basic mathematical operations like addition, plus, minus, AutoML Zero. Go read this paper or the blog post and then What I usually do is I read the blog post and then I read the paper. So the blog post is typically, uh, worded in not as a technical way as a paper. So that kind of gives me the ground base of understanding what's going on. And then I read the paper and sometimes a lot of the time I read a paper the first time and it's like, wow, this is, this is like a foreign language to me. But then I read back through it again and I go, okay, someone's put a lot of work in here to describe their work. Um, where we go? Yeah, search for— rather than just, uh, searching for, um, the best neural network architecture, AutoML Zero searches for an entire algorithm from scratch. Crazy. But it did take a, a large amount of compute power to do. Kicked off by Google in 2016, federated learning research is now booming. Federated learning is— so imagine I have my computer and you have your computer and you have an Xbox and they're not getting used. Rather than use a whole bunch of, of deep learning or GPU chips on a server, why don't we harness the power of computers that aren't in use? So instead of training on 10 supercomputers in Google's server, why don't we train on a million Android phones that people aren't using while it's on charge overnight. So you imagine your, your phone could run a very small part of a model while you're not using it and it's plugged into charge. So it's taking advantage of compute power that exists but isn't being used, which to me makes a lot of sense. Good to see that's growing. OpenMined, the leading open source community for privacy preserving preserving ML. Yeah. If you want privacy-preserving ML, OpenMind, founded, I believe, by Andrew Trask, who is someone you should definitely follow on Twitter. I am Trask. Here we go. So he posts, he posts some great stuff. Where we go here. Yeah. Lead OpenMind, senior research scientist at DeepMind, PhD student. Teacher. So just phenomenal person. And, but yeah, he posts some great tweets about like getting started in ML basically. Oh, there we go. Federated learning. Cavemen had supercomputers in their brain, running the world's most powerful machine learning algorithm for thousands of years. What held them back for so long? Access to training data beyond their family and village. Books, libraries, internet were completely Got federated learning. There we go. Speaking of federated learning. So yeah, go give, uh, Andrew Trask a follow. Um, yeah, big shout out to Andrew Trask. Thank you for the work that you do. Far out. We need to really bust through here. Prospective testing begins for privacy-preserving AI applied to medical imaging. Again, that just makes sense. Privacy-preserving AI for medical stuff. Gaussian processes strike back. I don't know enough about Gaussian processes, but quantified uncertainty and faster training speed. So this is, yes, this is what we want uncertainty in. So if your model makes a prediction, how certain of it is of that prediction? Is it of that prediction? So in a system you would, you would like to know, okay, maybe, um, you, so again, a problem that I worked on. We had a system where our model was 99.8% accurate on a binary classification task. So really good. But it was kind of like, if it gets something wrong, well, that's not good. So we wanted to know rather than, we wanted to know something more than just how confident a model was on a prediction. We wanted to know how much uncertainty was around that. And the way we did that was through Monte Carlo dropout. Monte Carlo dropout. And what is Monte Carlo dropout, etc., etc. And I believe the reason why we couldn't use Gaussian processes is because they weren't fast enough. And now we can get quantified uncertainty. Here we go. GP training time is reduced from 15 minutes to 40 seconds. Beautiful. I'm going to look that up. I'm going to save that again. Second Mind. AI collision process. Boom. I'll save that for later. Here we go. 2019 prediction outcome, Google quantum supremacy. Section 2, talent. The great brain drain. AI professors depart US universities for technology companies. To me, this is kind of like, okay, university's paying you $100 grand a year. If you come work at Google, we'll give you $1 million a year. And you can still research the stuff that you're passionate about. That's how, that's how I see that. I mean, a lot of people who, who wouldn't make that decision almost. Tech companies endow AI professorships in return for poaching, but is it really enough? New professorships may free the ladder for young academic talents to rise. Meanwhile, some companies, including Facebook, champion the dual academic-industry affiliation as a solution. Some academics don't buy it. This model assumes people can slice their time and attention like a computer, but people can't do this. Okay, so they want them to spend, spend some of your time, 80% in industry and then 20% of your time at the university. No thank you. See, humans don't operate very well when we half-ass things. That's like number one rule in life. Don't, don't half-ass shit. Always full-ass. The loss of AI professors seems to matter. Departures correlates with reduced graduate entrepreneurship across 69 universities. Wow. This is interesting. So if the professor leaves, the people don't start businesses. I'm not sure how that— well, it's correlation, right? So this is what I want to— I would like to see is, of course, more businesses. Can €100 million buy you 50 professors for a new AI institute? Uh, the Eindhoven Artificial Intelligence Systems Institute in the Netherlands plans to recruit 50 professors. So drop in $100 mil if you're an AI professor and you want to go work in the Netherlands. They've got a big cash stack. $100 mil donation from Silver Lake. 29 prediction outcome: Abu Dhabi opens the world's first AI university. Wow. The beautiful thing— well, I— to me, the internet is the AI university, the code university, the tech university. All the— imagine this, right? Imagine you have all the learning resources that you need to, to learn a subject. Oh wait, you do. You have Google. Like there's no shortage of learning resources. There's a shortage of, uh, uh, like the ability to dedicate oneself to learn. That's, that's what the shortage is. Chinese-educated researchers make increasingly significant contributions at Neureps. Yeah, this is amazing. China are just, just pumping out AI papers. After leaving university in China, 54% of graduates go on to publish at NeurIPS to move to the USA. Again, I can't really comment on this. I've never lived in China or the US. The US is an incredibly strong talent retainer post-PhD. Actually, speaking of the US, I did have the idea when I was studying my AI master's degree online, um, I couldn't think I could find a job in Australia and then I decided to think about going to the US and lo and behold, before I did that, I got a job in Australia. So, hmm, not sure where I was going with that. Oh yeah. Um, I was attracted to the US cause I think you see all the large tech companies, you see all the research, a fair, fairly large chunk, if not the majority comes out of the US. So it makes sense to want to go there. 3 of the biggest questions that you can ask yourself in life is what to do, who to be with, and where to live. It's worth spending a fairly long time on those 3 questions. Foreign national graduates of USAID PhD programs are most likely to end up in large companies. Um, that makes sense because I think large companies can just pay you more. Foreign nationals are 2 times more likely to join large companies in part due to their H-1B sponsoring power. We only have 4 minutes left. The UK and China, the biggest beneficiaries of American-educated AI PhDs. UK. Oh, the UK and China. So people from the US go to the UK and China. The majority of top AI researchers working in the US were not trained in America. Oh wow. So 27% come from China. Holy, that's very interesting. Given how dependent America's AI industry is on immigrants, there's being a strong backlash to Trump's proclamation to suspend H-1B visas. I don't know enough about that, but to me it kind of, it makes sense if you're, if you're allowing smart people into your country to work on and help businesses. Makes sense. Again, I don't make policy around that. American institutions and corporations continue to dominate NeurIPS 2019 papers. Google, Stanford, CMU, MIT, and Microsoft Research own the top 5. Look at that. Google is just crushing it. But then again, it's, uh, quantity or quality. The same is true at ICML, which is another massive machine learning conference. American organizations cement their leadership position. Google doing a lot. Stanford, MIT, Berkeley, CMU, Microsoft, Facebook. Leading universities continue to expand AI course enrollment. Stanford now teaches 10 times the students per year during 1999 to 2004 and twice as many as 2012 to 2014. Well, you know what? All of these Stanford courses, I'm pretty sure all of them are online. So if we look at this CS224N, boom. If you want to do Stanford's Natural Language Processing with Deep Learning, you can, you can go through that. You've got the coursework here, you've got the lectures on YouTube. Create your own assignment from it. Share your work on GitHub, share your work on your own website, and then start applying to companies using natural language processing. Demand outstrips supply for AI talent. Here we go. Analysis of US data shows that 3 times more job postings than job views for AI-related roles. Job postings grew 12 times faster than job viewings in the last, in the last couple of years from late 2016 to late 2018. I'm losing my ability to say 2000 and something. So yeah, this is saying that 3 times more job postings, so postings per million, searches per million. It's a lot more— there's a lot of demand out there for AI, AI talent. While hot, the AI talent market is not immune to COVID-19 during the pandemic. Here we go. Oh, this is a cool little, um, graph. So TensorFlow and Keras has more job postings on LinkedIn than PyTorch, but PyTorch is more common in— as we saw at the start of the video, PyTorch is more common in in research. Oh, this is from François Chalet, who is the founder of TF/Keras, so might be a little biased there. I mean, founder of— maker of Keras, which is now part of TensorFlow, which is kind of confusing, but I digress. Industry. We're only up to section 3. We've been going for almost a full hour. Um, the first phase 1 clinical trial of An AI-designed drug begins in Japan to treat OCD patients. Wow. AI-designed drugs are now coming to, to the real world. That's awesome. Emergent evidence is that large pharma is validating AI-first therapeutic discovery outputs. AI drug discovery. 2019 prediction outcome. Large pharma and startups ally around privacy-preserving machine learning for drug discovery. Yeah, well, that makes sense, right? As we've said, privacy-preserving ML for medical stuff. If it can be done, it probably should be done. Deep learning models interpret protein biology to find new therapeutics. This is what we were saying before, is combining ML with carefully designed experiments has enabled LabGenius to increase the number of potential drug candidates by up to 100,000 unfold. So that's improving how many people can take a certain type of drug. There we go. That 1-hour timer is done. You know what? I think what I might do is because we have a whole bunch of slides to go, I have to go to, uh, jiu-jitsu training right now, but I might come back later today and finish this off. We might go for another half an hour or so. Why not? This is fun. And again, if you want to just read this thing yourself, look for the top link in the description. So I will be back, uh, in a couple of hours for me, but it'll only be a second for you. Well, well, well, fancy seeing you here. Training's done. Massive session. Let's finish off this. Where were we? We were on slide number 88. I just have to tick something off. Give me a second. Today's goal, snap necks and cash checks. Yep. I tick that off. Jiu-Jitsu training, did snap a few necks. And here we go. Let's get, let's get 30 minutes back on the clock. Boom. And we're going to finish off the rest of these slides using genetic, metabolic, and metagenomic and meal contest information from 1,100 study participants to predict individuals' metabolic response to food at scale. Now this is what I'm really interested in 'cause my like formal education is in nutrition. So I went to university, I studied science and nutrition for, well, food science and nutrition for a few years. So I have a deep interest in that and I love seeing this. So Zoe, I did apply for a job there once, but then I realized, you know what, I prefer to work on my own for the time being. You know what? I might really see, see what they have. But a big shout out to Zoe. I love the work that you're doing. ML predictions of glucose triglyceride response 2 hours after meal consumption correlate 70% of the time with the actual measured responses. Zoe's commercial AI test-driven kit launched in the US in August 2020. So only recently. So I've got to go check out their paper. Papers or recent findings. In 2019, the FDA acknowledged that the traditional paradigm of medical device regulation was not designed for AI-first software, which improves over time. I totally agree. So this is what I'm thinking, like, just in terms of regulatory systems that have been set up, if things are like 20 years old, like, a lot has changed in the last 20 years. Uh, we didn't have smartphones, everyone didn't have a supercomputer in their pocket, we We didn't have, the internet wasn't as prevalent as it was. And I'm not saying all regulatory systems are set up like 20 years ago. Who knows when they were set up? Maybe longer, maybe shorter. But we're starting to move into a world where the interconnectedness is just extreme. So old laws may not apply to such a severe amount of interconnectedness. So I think going forward, there has to be some sort of updated system to, to regulate good machine learning practices. New international guidelines are drafted for clinical trial protocols, SPIRIT-AI, and reports, CONSORT-AI, that involve AI systems in a bid to improve both quality and transparency. I'm not sure what that is. AI-based medical imaging studies have a major problem. 20,000 recent studies in the field found that less than 1% of these studies had high-quality design and reporting. Yeah, so Hmm. This is another big thing. Like you could do a study, um, and think that you find valuable insights, but if it wasn't structured in the correct way, like say it wasn't built up with a, with a science foundation, uh, all of your findings could be completely wrong. And you know, the worst thing than discovering something that doesn't work is finding something that does work but for the wrong thing. So that's a big, big problem. So if you're designing a study, Make sure you do your research and make sure the system is set up or the study that you're doing is correct so that you don't get to the end and think, oh, oops, the, the flaw in my methodology has rendered all of our results useless. The first reimbursement approval for deep learning-based medical imaging product has been guaranteed by the Centers for Medicine and Medicaid Services, CMS, in the US. Wow. That's really cool. So Vis AI was granted a new technology add-on payment of up to $1,040 per use in patients with suspected strokes. So I believe it might be an app for some sort. Winning reimbursement from the CMS is a critical step towards any new system becoming implemented in clinical medicine because it creates a needed financial incentive to drive use. Yeah, so this is one of the things. It's one thing to be able to code like an app, like you could, you could build a prototype of something in a day. The hard part will be scaling it up, of course. And then the even harder part will be implementing it into an actual system. So getting it into people's hands. If you're building a medical device, getting it into people's hands. I mean, in theory it could work really well, but then in practice, totally different thing. So really cool to see Vis AI getting something working in practice. Um, US states continue to legislate autonomous vehicle policies. Uh, this is just making sense. Um, as autonomous vehicles start to arise, we're going to need to figure out how they interact with the rest of the world. Even so, driverless cars are not so driverless. Only 3 of 66 companies with automated vehicle testing permits in California are allowed to test without safety drivers since 2018. Hmm, so Waymo, Nuro, and AutoX— I haven't even heard of AutoX, to be honest— are the only 3 companies who can have their cars driving without safety drivers. Well, that's, I suppose, if you want full autonomous vehicles, right? I think we're gonna— a better thing is before full autonomous vehicles, we're going to go Level 3. So with, like, I'm not sure what the exact levels are, but I'm not sure we're gonna ever, who knows, ever is a long time, ever move towards full autonomy because there's just so, so many different variables. I mean, highway full autonomy, that kind of makes sense. But when you're coming into like suburbia, different scenarios, who knows, maybe there'll be zones, right? So like you can only fly your drone in a certain area, you can only use your autonomous vehicle in a certain area. That, that kind of makes sense. Um, anyway, self-driving mileage in California remains microscopic compared to human driving. So we got here— wow. But this is— it's a slow, it's a slow burn, right? Like, of course self-driving is going to be smaller than human driving, but then you imagine if, if autonomy does become a thing, the curve of the amount of kilometers driven by, or miles driven by autonomous vehicles is going to exponentially increase, uh, compared to human mileage. Because if you have a lot of autonomous vehicles out on the road, you can just scale them up. All of a sudden, this 2,000— 2,874,000 miles could easily turn into 20 million miles. Sketchy metrics. Tracking AV progress is complicated by the industry's focus on miles per disengagement, which is hard to benchmark and is not reported across all US states. That's a good, that's a good, uh, a good statement, you know. It's like, how do you judge, um, how good a self-driving car is at self-driving? So disengagement means, uh, how many miles did the car drive before it had to go, you know what, I can't handle this scenario, human take over. And Baidu looks like it's, it's had a mammoth improvement over the last Wow. Over the last year, nearly +9,000%. But again, use your gut, trust your gut. If improvements on year on year of +9,000%, if they sound too good to be true. Self-driving. When even a billion dollars isn't enough. So Amazon acquired Zoox, which is a self-driving car company. Um, I think Amazon want to get into the self, the autonomous ride-hailing, uh, they want a piece of the pie, you know. If autonomous vehicles are going to be a thing, this is basically Amazon's model. It's like, if, if an industry is making money and we can do it better, we're going to come into that. So I think they acquired some sort of self-driving car company, Zoox, but that apparently this car company was burning $30 million per month in early 2020. See, this is the challenge, right? Autonomous vehicles, yes, they have a massive payoff, but you also need a ridiculous amount of capital to start off. So $30 million per month and 900 full-time employees with no like deliver— like they haven't— of course they deliver prototypes, but they haven't delivered like a working product. This is why you've got to hand it to people like Tesla and Comma AI who are, uh, I'm not sure if Tesla's profitable, but they're definitely delivering vehicles and they're they at least have massive amounts of revenue, and Comma AI deliver shippable intermediates and are also profitable. And so they're sort of, they're making profit while building self-driving cars. That's the, that's if you can ever, that's to me, that's getting paid to learn. A really good scenario in life is getting paid to learn. If you can set that up for yourself, do it. The main self-driving contenders raised almost $7 billion in private rounds since July 2019. Holy crap. Look at all these self-driving. I didn't know there was this many self-driving car companies. Voyage, AutoX, Pony.ai. That's a good name. Waymo, 5AI, AI.Motive, Didi. Another self-driving group, Didi. So Didi is like a Uber in China. We've got Didi in Australia too. I'm not sure if you have Didi where you live. But you can order an Uber or a DiDi, they basically offer the same thing. But again, it's very, they're very interested in self-driving cars because if they could replace human drivers, they don't have to pay the human driver. Spins off from its parent and raises $500 million. DiDi self-driving unit raised $500 million from SoftBank Vision Fund. Grew its team from 200 to 400 last year and launched its ride-hailing service to consumers in Shanghai in late, July 2020. Wow. Capital is used vertically integrate and deepen technology modes, e.g., in-house LiDAR. So a lot of companies, self-driving car companies are using LiDAR, which is to me, it's like it's depth sensing. So it's rather than just a camera, it's sensing depth from the environment. But again, to me, this is just adding more pieces to the puzzle. Of course, I'm not an expert in self-driving cars, but the more, the way I think about it is the more pieces you add to the puzzle, the more complex the problem becomes. So if you remember from before, there was some computer vision research that was generating depth from, from a single image. So I was listening to a conversation from George Hotz, who's from Comma AI, was saying that if we can't push a single camera to its limits, so Comma AI use a single camera, then adding multiple cameras or adding LiDAR and all that sort of stuff is not going to to help us. So I'm a real big fan of embracing constraints. Meanwhile, LiDAR incumbent Velodyne and challenger Luminar both go public on the NASDAQ via reverse mergers, SPAC, to compete with hardware and ADAS software. So there's some LiDAR companies going public. Supervised learning and cost of edge cases, new technology approaches are needed. Yeah. I think this is, um, this is what we've been discussing, right? Improvements in supervised ML systems over time expectation. So rather than following this exponential, this power curve, we're following an S-curve here. So we've seen a massive amount of AI performance, but now we're sort of tapering off. So rather than seeing exponential improvements in the quality of AI performance, Moore's Law, we're instead seeing exponential increases in the cost to improve AI systems. Supervised ML seems to follow an S-curve. Yeah. I think this is what we're going to see going forward is more, far more unsupervised and semi-supervised methods just, just taking over. Of course, we're going to squeeze supervised for as much as it's worth, but going forward over the next couple of years, that, that to me is where, where I'm seeing the trends that are heading. Leading companies crowdsource ideas from open source using data they've generated. I like this approach too, right? This is leveraging the power of the internet. A real good business model is to invert the internet. It's what Amazon have done. So what they did is they were creating all these data servers, uh, for their own store when Amazon was just an online bookstore. Then they realized, hold on, a lot of other companies have problems creating data servers all over the world. So why don't we just create some data servers and then rent them out to other companies? And so Amazon Web Services was born. So these companies here have open sourced their datasets and now they've made things like Kaggle competitions or got people like you and me to see what kind of work they can do on these datasets and then leverage the things that the people around the world figure out for them. The next step: new models and a shift in focus from perception to motion prediction. Hmm. Use of ML and self-driving is still mostly limited perception with large parts of stack, stack Large parts of the stack hand-engineered. Yeah, this is what we were talking about before. Much of ML self-driving focus on what's going around the vehicle, but think about it when you drive. What the self-driving car should do is mostly hand-engineered, making development difficult and slow. Yeah, this is what I was thinking, right? If, if I'm a self-driving car company and I'm hand-engineering features like this is what a car looks like, um, this is what the car should do if you see a car in front of you, that's like using a lot of software 1.0, right? Like, so a lot of rule-based stuff. So of course you're using computer vision and machine learning to, to perceive what's around, but then you have to do a lot of rule-based stuff to go, you know what? Okay. Yep. You can see that red light. Stop if you see a red light. So a bunch of if statements to go along with your machine learning. Where I think going forward is, and it makes sense to me, is something like the MuZero approach. So MuZero, here we go. So this is reinforcement learning sort of style. Now, constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Where's MuZero? However, in real-world problems, the dynamics governing the environment are often complex and unknown. So this is, this is like a self-driving environment. If we're in— if we're driving along the road, the dynamics governing that environment are often complex and unknown. So you never know what's going to happen in a road-based scenario. Of course, there are some things that are going to be quite similar depending on where you're driving. But what happens if a car comes out of nowhere? These are the edge cases that are going to be really hard for supervised ML to tackle. Because a machine— a supervised machine learning algorithm doesn't do too well on things that are outside of the distribution of its training set. Now if we go here, MuZero learns a model that when applied iteratively predicts the quantities most directly relevant to planning: the reward, the action selection policy, and the value function. When evaluated on 57 Atari games, the canonical video game environment for testing AI techniques in which model-based planning approaches have historically struggled, our new algorithm achieved a new state of the art. When evaluated on Go, Chess, and Shogi without any knowledge of the game rules, MuZero matched the superhuman performance of the AlphaZero algorithm that was supplied with game rules. Boom. So look at that without any knowledge of the game rules. So what if we tell, what if we treated, again, I'm not an expert, not an expert. This is just me thinking out loud. What if we treated self-driving and this is where things like Tesla's Dojo, Tesla, Dojo, like a simulated way. Yeah. So simulating driving. What if we treated driving like a game, right? And applied an algorithm like MuZero instead of hand labeling all these features of what to do. Just your, your labels could be what the human does in a certain scenario. So it comes up to a red light, stops. You don't have to handcraft the, the feature of a red light, like a bounding box around a red light, just to have the whole scenario as the game world. Again, my thinking here is just out loud. If you want to see more on this sort of thinking, I'd go watch all the videos of George Hotz talking about self-driving cars. He knows a lot more than I do. The new frontier of self-driving development is machine learning for planning. Ah, here we go. "New algorithms working akin to AlphaGo and trained on large amount of human driving demonstrations are being developed." Yeah, okay. That's where I see the future of self-driving cars. "The consumer-first approach to self-driving. Tesla has hundreds of thousands of Autopilot-enabled cars in the wild and consumers help it towards, inch towards full self-driving." Yeah, so this is Tesla's moat. They have the whole full stack. They have the hardware of the car, They have the software and they have the data. Oh, there we go. comma.ai. So these are my two favorite companies in the self-driving world. I think the other ones like Lyft, Waymo, and Uber are just like, to me they're just burning cash, like VC cash. It's hard to me. It's hard for a problem, uh, to be like fully involved in a problem when you're just burning someone else's cash. But when you're Tesla, or Comma AI and making you have profits off your customers, your actual customers who are using your product, then you kind of got like skin in the game. If the ship goes down, you go down. But if you're at one of these funded companies, if it goes down, well, it doesn't really matter to you because it was someone else's money. AI problems like self-driving thrive on compute. New providers of specialized AI compute platforms are already onto their generation 2 products. Well, this is exciting, all right, because NVIDIA need— NVIDIA need competition because they're, they're just— they own the basically the AI chip market. What— AMD have great GPUs. I'm pretty sure they just released some new ones. I haven't checked them out yet, but, um, a lot of the tooling isn't there for them. So I have an AMD GPU in my MacBook Pro here, but I can't use libraries like TensorFlow and PyTorch without a whole bunch of hacking away, or a framework like PlateML, which is, which is, truth be told, I checked out the GitHub, it's coming along. Whereas with a NVIDIA GPU, I just run one, one command basically, and I can start writing deep learning code straight away. So this is where I see if they can get the tooling right, not only just the hardware but the software to run the deep learning libraries, well then I'm all for that. See, look at this, there we go. They're already— NVIDIA Watch out for Graphcore, they're coming for you. What's that? So that's about a 12x reduction in cost for the same amount of compute. 16 times faster training time for image classification model EfficientNet. See, this is what I'm talking about. Using more efficient, effective, dedicated compute. Google's new TPU v4 delivers up to a 3.7x training speedup over TPU v3. Sure, I haven't found like, Have you ever tried to use a TPU? Sure, you can connect it to Colab, but if you want to use a TPU on a large scale, it takes quite a bit of engineering, special— a little bit of finesse. NVIDIA will not rest either, up to 2.5x training speedups with the new A100 GPU versus V100. Wow, so 4x performance gains on MLPerf in 1.5 years. So this is where we're like it's inevitable, right? Like to, well, who knows? Who knows what we can predict? But compute, you can just, if you just imagine compute is consistently gonna go up in somehow, because even if one like section of the computing, so like fabrication of an AI chip doesn't sort of improve, maybe the algorithm improves. And so then it'll look like even though it's running off of hardware, the algorithm will be able to go faster. The rise of MLOps, DevOps for ML. Signals an industry shift from technology R&D, how to build models to operations, how to run models. Yeah. So this is what I'm seeing too, is, so the model building stage of machine learning pipelines has, it's getting pretty mature. So we know like, okay, computer vision, use convolutional, architecture. You can use transfer learning for almost any problem you can imagine these days. Now it's how do we get that model that we're building pretty succinctly into the hands of others. And so yeah, this is where a lot of my time is going to be going in the future, is upskilling myself on MLOps. So if you'd like to see a video on that, let me know. As AI adoption grows, regulators, regulators give developers more time, more to think about. Again, making sense. Any kind of new technology should not be slowed down, but it also should not be sort of— I don't think it's a good thing to have just the wild, wild west of just letting anything happen without sort of thinking about how this is going to affect people. Because it can be too easy for nerds like us, like engineers, just sitting away tapping and writing code and not thinking about what happens to an actual person on the other end of that code. Enterprises report that AI drives revenue in sales and marketing while reducing costs in supply chain management. Makes sense. RPA and computer vision are the most common deployed techniques in the enterprise. RPA is robotic process automation. I don't have much experience there. AI dialogue assistants are live handling calls from UK customers today. PolyAI, we've seen them throughout this So it's rolled out its voice assistant for hospitality in the UK. Wow, that's pretty cool. So I think like things that we see for like just booking in like appointments or just calling a place and just getting some basic frequently asked questions, like that's where we wanna just use like a chatbot or an AI voice assistant to just be able to give someone information about a business. Computer vision unlocks faster accident and disaster recovery intervention. Oh wow. That's really cool. Tractable AI captures and processes images, imagery of the damage to automatically predict its repair costs. Wow. So we had a potential proof of concept on that at one stage. Can we capture images of— it looks like they've done that, capturing images of damaged cars, uploading it to some system, and then using computer vision to estimate how much it'll cost. 30 days to 1 week. Oh, I really like that. That's a great use case. No-code ML automation, a universal prediction API for 360 customer data. I think 360 might be like a, what's it called? Like a black book for customers. I can't remember, a database of some sort for your different customers. NLP is used to automate quantification of a company's environmental, social, and governance ESG perception using the world's news. Oh wow. So understanding NLP can derive ESG perception scores by assessing the relationships and sentiments of products and companies with respect to client-specific ESG reputation pillars. Oh wow. So just, just by analysing, analysing the news about different companies, you can tell how much pollution they have, what's their sustainability. Wow, that's a really cool way to sort of monitor if companies are staying true to their word, right? If they're reporting something but the algorithm disagrees, it's like, why is there such a discrepancy here? Machine learning protects humans from email spear phishing attacks. This is just spam detection on 2020, you know? Computer vision detects subtle evidence of tampered identity documents. Yep, that makes sense. Use computer vision to see if you got a fake ID, yo. AI is the key to web-scale content analysis for money laundering and terrorist financing. I think NLP again. Article collection and classification as well as entity recognition and disambiguation to support downstream risk classification of people and organizations. Wow. Uses deep learning techniques to cover up to 85% of the risk data in all key geographies. Wow. Hmm, I'm not sure what kind of like websites they'd be analyzing, but I'm guessing it's, yeah, it's fraud detection and seeing if like if a certain website is, uh, is funding a terrorist organization. That's a great use case. Machine translation unlocks financial crime classification globally. Machine translation is used to generate multilingual training data for financial crime classification. Wow. So they use— so if they don't have any data for a certain type of what is it, financial crime on a certain language, they can use translation to translate, I'm guessing English to another language and use that data to train on. That's phenomenal. BERT language model goes mainstream, upgrading Google and Microsoft's Bing search query understanding. Okay, so this makes sense. Open source publication. So BERT came out at the end of 2018. And now it's used in large-scale production within 12 months. Unless Google already had it in production before they publicly said that they did. I think Microsoft are using it as well. Robotic installations are achieving millions of robotic picks per month. So robots going into factories. Well, that makes sense, right? It's like, if you're working in a factory, is that the most fulfilling work that you can do? Get the robots to do the least fulfilling work, get the humans to do the creative stuff. Manufacturing. CNC machine programming starts to be automated. So this is another massive thing, right? Is, uh, I don't think about these things because it's like we like write code and we don't— well, you might make things, but for me personally, I write code on a machine that's already been made for me. I rarely think about how the machine got made. And from what I've heard, we all just think that, yeah, laptops just come out of a factory. But how do you design the machine to build the laptop? So the machine that builds the machine— and I've heard Elon say, like, from, from car manufacturing— is the machine that builds a machine is way harder to design than the machine itself. So the machine that builds a car is harder to design than the car itself. Open source model and dataset sharing is driving NLP's Cambrian explosion. Big shout out to Hugging Face here. Hugging Face's Transformers library in production, 5 million pip installs, 2,500 community transformer models trained in over 164 languages. How good is that? By 430 contributors. So Hugging Face Transformers, look them up if you want to use transformers for your NLP task. They could, if transformers get repurposed for computer vision, Hugging Face's transformers for NLP could get It could have a separate arm, could be forked. Now we've got Hugging Face Transformers for NLP, Hugging Face Transformers for computer vision. Hugging Face could become the new actual OpenAI since OpenAI are basically closed AI now. Open source conversational AI expands its footprint across industry. Rasa's libraries and tools have clocked up 2 million downloads and have open source, have open source 400+ contributors. So I believe Rasa And we've only got 1 minute left on the timer. How many slides are we through? 128. Oh, let's just go to the end. You know what? Well, we're almost there. We'll just keep going. Rasa. Rasa open source conversational AI. This is cool. Build contextual assistants that really help customers. Wow. Okay, so you can build your own chatbot, compare plans. How much is this? Free and open source. Wow. If you're enterprise, you pay big. I like that model. Let's go here. Well done, Rasa. Private $15 mil funding rounds or above $15 mil for AI-first companies remain strong despite, in spite of COVID-19. That's good to hear. If you're building an AI company, COVID-19 isn't really impacting how your opportunity to get funding. Politics. Now again, this is probably my least experience of expertise in this whole report. So yeah, I'm probably going to burn through this one fairly fast. If you want a more in-depth, of course you can check out the report yourself. Ethical risk. A group of researchers have spent years helping to frame the ethical risks of deploying ML in certain sensitive contexts. This year, the issues went mainstream. Ah, the Perpetual lineup. Facial recognition is remarkably common around the world. 50% of the world currently allows the use of facial recognition. Holy crap. That's more than I thought. Again, well, I mean, we all— the iPhone uses facial recognition now, right? Potential risk to wrongful arrest. Yeah, here's where it can go bad. When facial recognition recognizes the wrong person. Two known examples of wrongful arrests, May '19 and January 2020. Likely just the tip of the iceberg. Yeah, see, imagine if you got I mean, it's going to happen almost in any system, right? It's going to be a human, even if a human recognized someone, they're going to have some, some percentage of error. Same goes for a facial recognition system. It's not going to be 100% perfect. You might be able to reduce the error rate, but a perfect system? Facial recognition, Facebook settles class action lawsuit for $650 mil. My goodness, who sued them? It went against Illinois' biometric privacy law. Ah, I see. Maximum exposure via the lawsuit was $47 billion. In the end, the suit is likely to net each affected user $200 to $400. So this is like kind of Facebook's motto is just, uh, uh, just do things and ask for permission later, even if they're illegal. Clearview exposes what is now technically possible with facial recognition. A search engine for face. For faces. Oh, great. So Clearview, I think, are a facial recognition system, like at an airport. Clearview. Yeah, I think if you go to an airport, instead of like scanning in, you can like, um, scan your face and go through quicker or something like that. Anyway, you can check that out. But how they got their data set was we'll just scrape the internet of everyone's faces and now we'll just resell that, those images back to law enforcement agencies. Scraped photos from Facebook, YouTube, Venmo, and millions of other websites. Now I mean, hmm, do people know that when they're putting their stuff online, like I'm making this video now, my face is probably in all of these, these databases everywhere, but do people know that their faces are now in this database? I don't think many people would know that. Facial recognition, more thoughtful approaches gather steam. Large technology companies are taking a more careful path. Yep. Microsoft deleted 10 million faces. Amazon announced a 1-year pause on letting police use its facial recognition system. Okay. Apple is asked by New York's MTA to enable Face ID for passengers while they wear a mask. Yep. A lot of people wearing masks during COVID More thoughtful approaches. Reducing bias in datasets. Yep, totally agree. That should definitely happen. We should do our best to do that. Even though the argument is, even though datasets are biased, that's just because human nature is biased. Well, think about this, is, is that human nature? We can also adapt to scenarios that we have to adapt to. So if removing bias is a good thing, then should we not do that? Facial recognition, a new legal precedent in the UK emphasizes that facial recognition Tools cannot move fast and break things. Hahaha. If you're doing facial recognition, do not use the Facebook motto. Um, Washington State passes new law with active support from Microsoft. Legal challenge in China. Oh, deepfakes. Increased awareness of deepfakes causes a rush of activity by China and California. California passed law AB 730 aimed at deepfakes, which criminalizes distributing audio or video that gives a false damaging impression of a politician's words or actions. Yeah, you want these laws to happen? You want these laws for deepfakes? Just find a prominent politician, create a deepfake of them, and then just send it to them and make sure they see it. Because once they see it, they'll be like, oh crap, this stuff is pretty full on. And then lo and behold, shock, the law of passing legislation against deepfakes probably be sped up. Algorithm decision-making, regulatory pressure builds. New Zealand's Prime Minister says they're the first in the world to produce a set of standards for how public agencies should use algorithms to make decisions. I like New Zealand's Prime Minister. What's her name? I think she's doing a great job. New Zealand Prime Minister. And look, I don't agree with all of her decisions, of course. It's rare for anyone to agree with anyone's. Jacinda Ardern. But I think she seems to be the most thoughtful world leader at the moment, put it that way, in my view. And I don't really pay too much attention to politics. GPT-3, like GPT-2, still outputs biased predictions when prompted with topics of religion. Ah, okay, again, this bias is— you could argue, the researchers could argue GPT-3 was trained on the internet, so the bias is from the internet. Well, again, we can engineer ways to remove this bias. From DeepMind to US Army research lab, AI research agendas start to overlap. Holy crap! 3 months after DeepMind's StarCraft 2 breakthrough, the US Army publishes interesting StarCraft results. So now DeepMind's research— closed mind— is, uh, going into the US Army. That is incredible, incredible. While it is notable that cutting-edge research ideas are migrating from academic and corporate research labs to military labs, wow, US Army continues to make major investments into implement military AI systems. Of course, this makes sense if we're using AI for a whole bunch of different things, especially like computer vision. I'm imagining a drone flying over of where it needs to fly, uh, rather than have a human in that, just use computer vision to understand the landscape. Startups at intersection of AI and defense raise large financing rounds. Yes, I think I saw this drone before, Enduro. It took, uh, took a shot at DJI. America now builds the best drones. Because before, um, what's it called? PJ Lucky? Is it PJ Lucky? And Dural? Yes. So there's a blog post somewhere which talks about this drone, which apparently before this drone came out, the Ghost 4, DJI built the best drones in the world. And of course, they're China. And so China and US, as much as I'd love to believe they're friends, they still have some sort of military sort of angst between each other. So if DJI are building the best drones and drones become a thing in military combat, which I'm sure they will, it'd be a good thing to know that the US have great military drones as well. Is US-China competition weakening the missile technology control regime? AlphaGo and AlphaStar, AlphaDogfight. Big dogs gotta eat. But this is, looks like it's deep reinforcement learning for fighter pilots. Holy crap. The top AI developed by Heron Systems beat a human pilot 5-nil. What? That's insane. The winning AI used hyper-aggressive tactics of flying very close to its opponent whilst continually firing with lower regard for the survival of its own plane. Ah, the anonymous pilot said the standard things that we do as fighter pilots aren't working. Hmm. Well, the winning AI, it could use hyper-aggressive tactics of flying very close to its opponent because it doesn't have consideration for its own, like, well-being. Whereas a pilot, a fighter pilot, doesn't want to fly too close to someone else because if they do, they'll, they'll get dusted and they'll be gone. Whereas an AI is just like, I don't care, I'm just going to fly as close to you as I want and just try to destroy you. The US Secretary of Defense targets 2024 for real-life AI versus human dogfight. Holy crap, 2024. The AI agent's resounding victory demonstrated the ability of advanced algorithms to outperform humans in virtual dogfights. These simulations will culminate in a real-world competition involving full-scale tactical aircraft in 2024. We're going to have AI fighter pilots versus human fighter pilots in 2024. That's mental to me. Many actors attempt to define principles for responsible use of AI. Totally agree. It's very hard problem, right? Two of the leading AI conferences adopt new ethics codes. NeurIPS and ICLR both propose new ethical principles and expectations of researchers, but no mandatory code and data sharing. Hmm. Again, I kind of understand this. I would like to see, of course, code and data sharing. But again, some of the people and authors and companies that go there, if they do share their code and data, is that affecting the business of the company they represent? So if Google and Facebook are sponsoring their researchers, authors are required to provide an explicit disclosure of funding and competing interests. So yeah, we built this computer vision model that's the best in the world at understanding faces, but we're also paid by Facebook, and so that improves Facebook's business. Google is leaning into fairness, interpretability, privacy, and security of AI models. People AI Guidebook. That's a good resource actually. I checked that one out. So it tells you explainability and trust, data collection and valuation, user needs and defining success. White House extends its ban on Chinese companies with ties to surveillance in Xinjiang. I'm pronouncing that wrong for sure. I'm not— I'm again, I don't know too much about this. Huawei is increasing dominant player in the smartphones and investing heavily in machine learning technology. Wow, Huawei is taking over Samsung, but I'm pretty sure Huawei's banned from, um, the US now, isn't it? President of Huawei's consumer unit declared no chips and no supply. Okay, foreign companies that use the US chipmaking equipment would be required to obtain a US license before supplying certain chips to Huawei. Okay, semiconductors amplify the geopolitical significance of Taiwan and particularly TSMC. So TSMC make like the majority of the world's like smartphone chips. I'm pretty sure. Yeah. The US technology industry and TSMC are significantly codependent with 60% of TSMC sales coming from the US. TSMC said it would spend $12 billion to create a chip fab in Arizona. See, to me this makes sense. If you're going to be creating something fundamental for your business, it'd be good if you could create that close to your business. So if something like what happens in the world, like what happens in 2020, COVID, like borders shut down and stuff like that, and you don't have access to your supply chain, it'd be good that if you didn't necessarily need to cross so much borders. So if you're in the US and you're Apple and you're making everything in China and you want your business to keep running, it'd be good if some of what you made in China was also made at home. Taiwan's TSMC remains dominant in R&D expenditure and semi, semiconductor manufacturing. So it's the only fabricator with a 5 nanometer manufacturing process and is now working on 3 nanometers. Holy crap. Chinese government sets up an additional $29 billion state-backed fund to reduce its dependency on American semiconductor technology. Yeah, see, this is what I'm saying. A lot of this is what's happened in the world over the last year is that a lot of like companies and businesses and entities have realized, hold on, how much do we rely on external parties to drive our fundamental business? And so if the Chinese government is relying too much on America and something goes wrong in their relationship, of course I don't want that to happen, but they probably want chips to be made in their own country. China hires over 100 TSMC engineers in push to close the gap in semiconductor capabilities. So yeah, this is what I'm— I don't know enough, but from what I know, TSMC is like the one of the best chip manufacturers in the world. U.S. Senate proposes the Chips for America Act. Makes sense. Half the world's advanced chips are designed in America, but only 12 are manufactured there. Hmm, sounds like that could be improved. Given mounting concerns over chips, cross-border MMA remains highly politicized. The vast majority of acquisitions have been blocked. So I believe MMA is merge and acquisition. So these ones have been blocked, all these companies trying to merge and acquire. But April 2020, China allows NVIDIA USA $6.9 billion acquisition of Mellanox. Israel. Okay, so Nvidia, the potential acquire acquisition of Arm by Nvidia will be a major test of where things stand. So didn't— I'm pretty sure they bought them, didn't they? Nvidia buys Arm. Okay, there we go. Is this legit? Nvidia buys Arm from SoftBank for $40 billion. There we go. Well done, NVIDIA. NVIDIA said on Sunday that it would acquire the British chip designer Arm from SoftBank. Okay, now again, I'm not sure— that's what's at New York Times. Hopefully that's trustworthy. Who knows what you can trust these days. AI nationalism. Governments increasingly plan to scrutinize acquisitions of AI companies. The likely state of Arm is to NVIDIA is questioned by many, including its founder, uh, Hermann Hauser. A leading founder and investor argues it would be bad for the UK if Arm is acquired by NVIDIA. Well, I think they just did, according to the news. AI nationalism in the US. AI budgets continue to expand. Federal budget for AI, for non-defense AI R&D $1.5 billion in 2021. A major new bipartisan act is proposed. AI nationalism, China decentralizing policy experimentation to cities. Another wave of countries declare national AI strategies. Yep, this is making sense. As AI becomes more prevalent, countries need to know where they stand on it. US tax code incentivize replacing humans with robots. Hmm, I don't know what I think on that. Jobs at risk of automation in the EU and in 19 countries. Yeah, see, I see these things and a lot of this looks to me just like I'd have to spend a fair bit of time trying to interpret this, these graphs. And so what I'm thinking here is of course a lot of jobs are at, uh, at risk of AI automation, but again people can adapt, right? There's, there's— people have the ability to adapt. That's the most important point to remember, is that people have the ability to adapt to new scenarios. They have to believe it first themselves, of course. Bengio, Hassabis, and other AI research leaders united at NeurIPS 2019 in a call to action for climate change. A position paper and workshop explored various high-leverage problems where ML methods can be applied. Tackling climate change with machine learning. Oh, so there's a paper on that. I think these are scenarios where ML is high leverage. Automatic monitoring with remote sensing, scientific discovery, optimized systems, new battery materials, carbon capture, reducing food waste. Oh, I like that one. I wonder how we could do that. Accelerate physical simulations, climate models, and energy scheduling. The authors note that ML is a part of the solution. It is a tool that enables other tools across fields. This is why ML interests me so much. Again, what we talked about before, the information theory problem setup. If you have inputs and you have some sort of desired output, ML is a part of your toolset that can be the bridge between those two. Predictions, my favorite. 8 predictions for the next 12 months. The race to build larger language models continues and we see the first 10 trillion parameter model. Who's it gonna be? Who's it gonna be? Microsoft, Facebook, Google? Who do you think? Hugging Face maybe. Attention-based neural net or OpenAI or DeepMind, closed AI. Attention-based neural networks move from NLP to computer vision in achieving state-of-the-art results. Yes, I would agree with that. Transformer models. I don't doubt that we'll see in the next sort of 12 months Transformer models take over state-of-the-art of CNNs because that's just where the trends are going, I'm pretty sure. A major corporate AI lab shuts down as its parent company changes strategy. Oh, interesting. In response to US Department of Defense activity and investment in US-based military AI startups, a wave of Chinese and European defense-based AI startups. So yeah, okay, other countries see that US is investing heaps in the AI military space They're like, yeah, we need some AI in our military space. One of the leading AI-first drug discovery startups either IPOs or is acquired for over $1 billion. So this is gonna be massive. Like Big Pharma is huge, obviously. And so if AI-first drug discovery startups are doing good shit, the Big Pharma companies will be like, yep, we'll take over. DeepMind makes a major breakthrough in structural biology and drug discovery beyond AlphaFold. So AlphaFold is currently their model for looking at protein data. Facebook makes a major breakthrough in augmented and virtual reality with 3D computer vision. I think this is where Facebook have to head, right? Because to me, Facebook as a platform is just, I don't even use it. Like it's, I like the, Facebook could have like groups service and events service and just get rid of the timeline and that's all you have. You have groups and you have events and I would pay for that. Don't let me, don't, Don't make me watch heaps of ads. Just allow me to create a private group and private events and I can invite people through a link. Nice and simple. But yeah, I think that's where they're going. They acquired Oculus. They have a whole bunch of image and whatever data with, um, uh, Instagram and Facebook and whatnot. So that kind of makes sense if they've got a lot of smart people in the right places. Nvidia does not end up completing its acquisition of Arm. Hmm. Well, that's interesting to see. I mean, this was made on October 1st, and that article we saw of NVIDIA acquiring Arm was September 14th or something like that. Conclusion: Congratulations on making it to the end of the State of the AI report. Thanks for reading. Thank you for watching this video. If you followed along the whole way through, we're probably at 2 hours or something now. Whoo! I talked for a long time. Of course, yes, it is a snapshot of progress. Last year's issue was published on June 26th, 2019. Of course, this is not capturing everything, but it is a good snapshot to sort of just tell how the field is progressing. I mean, it might seem the things that you do day to day might not be making large exponential gains, but year on year, that's, that's a good time period to evaluate what's happening. Big shout out to the authors. Thank you to Nathan Benach, I'm probably saying that incorrectly, and Ian Hogarth, who are both deep into the field of AI and investing and company creating and whatnot. So thank you very much. We have one more thing. I will be back in a second. Ah, you thought we'd talk about the state of AI for 2020 without an without Future Dan showing up. Well, I'm here. I'm always here. Let's be real. I've got some predictions. Future Dan's 2020, or 2021 more so, AI predictions. Now, the transformer overflows traditional CNNs as the best architecture for computer vision. Now that, that aligned with the State of AI report 2020. PyTorch and TensorFlow have a baby and call it PyFlow. Or Tensor Torch. Let's be real, it could happen. NVIDIA faces serious competition, like actual competition, from an AI chip company. It could be Graphcore. Graphcore, I'm rooting for you. And NVIDIA, I'm rooting for you as well. Competition is a good thing. Semi-supervised and unsupervised methods like SimCLR2 take over supervised for state-of-the-art, less data. SimCLR2 is a framework for semi-supervised computer vision, so check that out. A self-driving car company shows state-of-the-art results with MuZero-like setup. So again, rather than all these handcrafted features, they use AlphaGo type. $2 billion plus invested into health plus AI startups. I think this is where, this is the year that sort of AI and data starts really moving into healthcare. And full-stack ML. So not just building models, which is a mature, which is to me becoming a mature science, becomes the norm as model architectures design. So full-stack ML, getting your stuff out to the world, that's where I'm going to be spending a lot of my— oh, actually I'm Future Dan, so I've already spent a lot of my time there. So that's what I can tell you for what the future's like. Where's, uh, let's go back to actual Dan. Whoa, um, you didn't see that. It is Halloween anyway. But that's it. The State of AI Report 2020 review. I'll probably do another one next year. Leave a comment below of what your favorite thing was. What are your predictions for, for upcoming in 2021? What are you most excited about? And as always, don't forget to check it out. Big shout out to Nathan and Ian for creating it. And keep learning, keep creating. I'll see you next time.