You Won't BELIEVE What AI Can Do Now! (NEW 2024 A.I REPORT Reveals All)

So Stanford University have released their Measuring Trends in AI report for 2024. And this is their most comprehensive up to date analysis. And it arrives at the important moment when AI's influence on society has never been more pronounced. And essentially this AI index report tracks and visualizes data related to AI. And they try to provide unbiased, rigorously, vetted, broadly sourced data in order for policymakers, research researchers, executives, journalists and the general public to develop a more thorough and nuanced understanding of the complex field of AI. So this report should show you where the general trends in artificial intelligence are and will likely help you inform you about whatever it is that you may be thinking for the future of AI. So there's going to be 10 sections of these and I will leave timestamps in the bottom of the video. So if you want to look around and skip around, that will be there. So sort of the first things that we have is that industry continues to dominate from frontier AI research. We can see here that if we look at 2023 and leading onwards, and actually the trends, we can see that industry produced 51 notable machine learning models, while academia contributed to only 15. And there were 21 notable models resulting from industry academia collaborations in 2023 with a new high. But of course, overall, we can see that this trend is continuing to increase in the number of machine learning models that we're going to see. And one thing that I would like to see is I would like to see the government start to get involved in some kind of projects because I think that, you know, that kind of thing is important because, you know, with private companies leading the way, I'm wondering if there's going to be some crazy power imbalance in the future. Now, in addition, in chapter one, they talk about how, you know, there were 149 foundation models that were released, which is double the amount in 2022. And of these newly released models, 65% were open source, compared to only 44% in 2022 and 33% in 2021. So once again, of course we can see this trend of AI is continuing to increase. And it's pretty fascinating because I think one of the things that we love to look at here is that we can see that on the open tab right here, we can see that we have such a smaller percentage, almost even 32 to 28. And then we can see here that it's 98 open source models released in 2023. And open source is far and outpacing the ones that are limited. But of course, state of the art. Limited models are still reigning supreme areas like GPT4 and of course successors, which are probably going to be Claude 4, GPT 5 and Gemini 2. Those models are of course all closed source. But I think it's really interesting to see the trend that open source is just increasingly, increasingly going to dominate the space. And it's pretty, pretty interesting to see how that trend is increasing. And I'm wondering, considering how Elon Musk has been contributing to that, if that trend is going to increase because recently there were some regulations. So this is going to be a trend that I will look back on to see if open source continues because there are some strict regulations about systems. And the reason I state that is because right now open source systems are getting really, really good. But I'm wondering what happens when those open source systems get to a level that is pretty powerful. So right here this is pretty fascinating because what they actually did was they released the estimated training costs of the models from, from 2017 to 2023. And we can see that of course there is a huge amount of funding going into these models. And GPT4, surprisingly, they estimated it according to the AI index. They estimated that GPT4's training course was actually $78 million, which is a little bit cheaper than I initially thought because I thought that it would be, you know, around that 200 million dollar mark. But it seems that Gemini Ultra is at 191 million dollars. So I'm guessing that this shows us that, you know, Open Eyes GPT4, which was trained on the Nvidia chips, is definitely far more efficient and more cost effective. Because Gemini Ultra doesn't surpass GPT4. I mean, technically it does in the benchmarks, but in terms of the leaderboards and the human eval, in terms of the human evaluations, not the human eval benchmarks, I'm talking about the leaderboard benchmarks in the playground. It seems that, you know, GPT4 seems to be a little bit more efficient, but we didn't get, of course, Claude 3's release because that was just really recent. So I would have loved to know how much they spent on training Claude, but I guess that there isn't any estimate on that just yet. Now, of course this is pretty interesting because this is where we take a look at some of the countries that are leading machine learning. And we can see here that in terms of the top countries, we literally have the United States leading the rest of the world by a far far margin. I mean, you can see that China is following Close behind. France is of course close behind as well, and Germany is, is of course still there. But the point is, is that the United States is, you know, far, far superior in terms of, you know, putting things out. And of course, you know, a lot of people are going to say, well, the United States is a lot bigger. So of course. But I think the United States culture, you know, the San Francisco area, the competitiveness, all of those factors, especially with their tech industry, just leads them to, you know, far outpace the rest of the world. And like I said before, I wonder if this trend will change in the future. Will China manage to release more and more models? Are they going to be restrictive on certain things? Will France catch up with open source? Will the UK finally get into the mix and start releasing a lot more things? I mean, it's going to be very interesting as well, considering. Also, you know, we did get some information that the UAE was going to potentially be an area where Sam Altman was going to be releasing certain models and experimenting with certain AI technologies. He talked about it being a regulatory playground. And we can see here, this is where Sam Altman says the UAE could become a regulatory sandbox for AI. And that's why I'm stating that, you know, maybe the UAE is going to be something that far surpasses things in the future if Sam Altman is potentially investing in there. So that's just something that's quite interesting on the countries. But I do think that, you know, the better model is far important than how many models you do produce. Now, something interesting as well, something that I was looking at quite a little bit, but I didn't manage to find too much, was that there was a lot more AI patents going out. You can see that there have been a huge increase, 62% increase in the amount of AI patents being granted around the world. And this is pretty fascinating because it goes to show that a lot of people are inventing things with AI that may come in the future that are going to be really, really game changing. And I do wonder if many people in the future. And yeah, I'm wondering if when superintelligence gets here, what this scale kind of looks like. If it just looks, you know, goes vertical. Because superintelligence, you know, theoretically speaking, should be able to generate ideas in a way which we haven't thought of yet. So that would be pretty interesting. And what was also interesting, if we're looking at patents, it seems that China is leading the global AI patents origin with 61.1% significantly outpacing the United States, which accounted for 20.9% of AI patent origins, which is quite an interesting trend. So seems that China is trying to lock down a lot of the things that they are innovating with. And of course, open source AI research has been absolutely exploding. We can actually see that there is literally a sharp increase right here. Not sure if you can see this, but right here we can see that the graph, I wouldn't say it goes vertical, but it does increase substantially right there. That is a pretty incredible jump. And you see it says, Since 2011, the number of AI related projects on GitHub has seen a consistent increase, growing from 845 in 2011 to approximately 1.8 million just in 2023. And there was a sharp 59.3% rise in the total number of GitHub projects alone. Is of course related to the release of ChatGPT in late 2022. And of course it's more than tripling from 4 million in 2022 to 12.2 million, which is absolutely insane. And I can't imagine how many projects are going on right now. So if you don't use GitHub, you should probably use GitHub because it's pretty cool. And of course, the number of AI publications in the world continues to increase, so the number of research papers continues to substantially go up. And this is a good trend because overall we can see that the number of, you know, research being done is continuing to increase, and it's good that research is being shared worldwide. Now this is where we have the technical part of the section. So chapter two is essentially where we're looking at the performance of these models. And this is essentially where we take a look at what these models are capable of compared to humans and uncertain. So we can see here, this is chapter two, and this is where we are talking about the AI performance in terms of a variety of different benchmarks compared to humans. Now, this chart was actually going viral on Twitter because it shows an increasingly incredible trend, which is that, you know, these AI systems across different technical benchmarks shows us that AI systems are increasingly becoming better than humans and performing above the human baseline. You can see that AI has surpassed human performance on several benchmarks, including some in image classification, some in visual reasoning and English understanding. Yet it trails behind on more complex tasks like competition level mathematics, visual common sense reasoning and planning. And we can see here that the trend across the board, as all of these have started, didn't mean to do that. But as all of these have started, we can see that as we get into this area right here, which is 2023 and above, you know, it seems that the human baseline which is right here, it seems to be getting overtaken and we can see, know, image classification, natural language understanding, most of these. And of course, you know, some of these, even the ones that are lagging behind, we can see, like with the mathematics and the basic level reading comprehension, we can see some of these that are, you know, quite behind here. And the mmlu, which is a multitask language understanding, whilst they have been behind and they are lagging behind, they are quickly and sharply coming up. So it seems that this trend is going to continue into 2024, where probably by the end of 2024 we're probably going to have AI that is pretty much near 100% of the human baseline. And maybe this is what Elon Musk was talking about when he said that humans are going to be smarter than AI in pretty much everything by next year. And I think if we look at the trend where things are going all the way until 2023, we are literally so close to that benchmark, with several already being above that. So I wouldn't be surprised if that did happen because it doesn't look like we're far away from that. I mean, GPT5 could just take things to a completely new level. So it will be interesting as well. And remember, this graph can go above, some of them are like 110%. I'm pretty sure it could get to 120% above the human baseline, which would be rather fascinating. So what we need to take a look at as well is here comes multimodal AI. So essentially they said traditionally, AI systems have been limited in scope, with language models excelling in text comprehension, but faltering in image processing and vice versa. However, recent advancements have led to the development of strong multimodal models such as Google's Gemini and OpenAI's GPT4. These models demonstrate flexibility and are capable of handling images and text and even some instances can process audio. And that is of course, Google's Gemini 1.5 Pro. So what we have here, as we can see that the baseline has continued to increase in terms of multimodal AIs capability all the way to 90.04% where the human baseline is 89.8. So it now matches what humans can do. I'm pretty sure that this baseline is from Gemini 1.5 Pro, but I could be mistaken because GPT4 is a little bit multimodal, but it doesn't have Audio and video like Gemini 1.5 Pro does now. So we can see here that we look at the MMMU and this is different from the mmlu. And we can see that the AI models have reached performance saturation on established benchmarks such as imixnet Squad and Superglue, prompting researchers to develop more challenging ones. In 2023, several challenging new benchmarks have emerged, including the SWE bench for coding. And that one has been recently under some very interesting controversy, which I'll get to later because some people are alleging that the demo wasn't real. But there are still many open source projects that are doing well on the SWE bench anyways. HEIM for image generation, MMMU for general reasoning and Mocha for moral reasoning, and the Agent bench for agent based behavior and the HALU EVAL for hallucinations. And we can see here that, you know, the reasoning is pretty much very, very hard for AI systems. It's not like we've seen any crazy, crazy jumps here. Now I could be eating my words. As GPT5 gets released and as you know, other models get released because they are developing interesting tactics in order to reason in a better way. We can see that, that it seems to not be absolutely incredible because humans are still better in that area at the 82%. So I do think that GPT4 probably might touch human area because with some trends, what we've seen and the way how Sam Altman has spoken about GPT4, he even referred to GPT4 as being stupid. And as most people right now would believe that GPT4 is like some kind of genius in your pocket. So I'm guessing that that jump could be pretty crazy on the human expert level. So as these harder benchmarks emerge, it'll be interesting to see which AI manages to take that frontier level. One of the things I really wanted to talk about as well was the human evaluation. And this is the LMSYS chatbot arena for LLMs which have an Elo rating. And essentially what this is, this is where you have A and B tested. You're basically blind testing two AI systems and then you rate which one is better. So for example, I could ask an AI system to generate a script for a video about AI trends in 2020 and then two different models will select and plan a response, will give me a response and then I select if I think the one on the right is better or the one on the left is better. If the one on the right is better, I give that a rating and I say this one is better. And then it bumps up the ELO rating for that model. And we can see that this is a very, very useful benchmarks because it's not just based on what's in a test, it's based on how the user feels overall. And over time we can see an increasing trend that models are getting better and better at this. And right now it seems that GPT4 Turbo is still in the lead, even after Claude 3 was just released. So some people do argue that this is the only benchmarks that matter, because some of the benchmarks that we've previously had, of course, may have even faced some contamination and some errors that have even been in those tests before. So it will be interesting to see if this thing is used more, and I definitely think that it should be. Something that we also got to see here was that thanks to the LLMs, robots have become more flexible. So it says the fusion of language modeling with robotics has given rise to more flexible robotic systems like Palme and RT2. Beyond their improved robotic capabilities, these models can ask questions, which marks a significant step towards robots that can interact more effectively with the real world. And we can see, so yeah, you can see here that the evolution of these models is increasing in capabilities and robotics is of course something that is quite harder than traditional AI and but I do think that there are going to be some other breakthroughs that do complement each other that will lead to a more effective robot in the future. And we're already seeing that with robots like figure one, which have been remarkably impressive in their capabilities to perform certain tasks which they weren't teleoperated on. They were 100% done through a neural network. Which goes to show that we are improving at a really, really rapid pace. And generally this has been one of my favorite robot demos because I just love how fluid and seamless this robot manages to move. We can also see that on agent bench across eight environments, the overall score is increasing. So it says creating AI agent systems capable of autonomous operation in specific environments has long challenged computer scientists. However, emerging research suggests that the performance of autonomous AI agents is improving. Current agents can now master complex games like Minecraft and effectively tackle real world tasks such as shopping and research assistance. And this is pretty true. We've seen quite a lot of things in terms of AI agents. One of the things the paper actually does talk about was Voyager, which was done by Nvidia. And this was something that was really, really fascinating because Voyager's performance improvements over prior state of the art in Minecraft shows that it was a really really key moment using GPT4 to increase the reasoning abilities of the agent and to be able to do something in Minecraft that could, you know, learn, explore and plan in open ended worlds. And it seems that this is going to be pretty crazy in the future as well. We get more and more powerful systems that can be utilized. So something else that this paper also highlights was musicgen and Music lm. And it actually talks about the evaluation of music gen and baseline models or music caps. And these are just the benchmarks on how good the music is. And it shows that, you know, music is getting a lot better. But it doesn't show the recent YU and of course the recent sudo v3 that was recently released. And this is something that I think is going to be really interesting because music is reaching the inflection point where the general public is probably going to release, realize that it's really good and arguably on the level of the standard of music that we do get produced by various artists. So here we actually talk about the Closed LLM significantly outperforming the open ones. And this is just basically GPT4 compared to Mixtral. Of course that's not the only two closed and open source models that do exist, but I'm just talking about the more notable ones that do exist. And essentially what they're stating is that the difference between the closed versus open source percentage difference is you can see that the agent bench is 317% in terms of the difference of abilities and these other ones there is a clear distinct difference. But I'm wondering if that difference will remain the same because this chart doesn't show an actual like, you know, increase in terms of, you know, from year on year, I'm wondering if that difference will be slowed down and if that gap will eventually get closed. So it'll be interesting to see as well how that does change. So this is where they actually start to talk about responsible AI. And this is something that we do need to talk about because whilst yes, the system's abilities increasing is quite good for the consumer, there are increasingly a number of people who actually do use technology maliciously. Now we're not the people who fall into that category, but they do exist, which means we do actually have to acknowledge it. And this is where they talk about political deepfakes are easy to generate and difficult to detect. Now I would argue that one thing that I've noticed as a trend online is that people more and more are just anything they see online, they just don't believe it anymore. So whilst yes, you do have to, you know, believe certain things. I think we might be moving into the age where nothing is believed anymore. But this right here, they talk about how political deepfakes are pretty easy. So an AI selects content to counter. So an AI can scrape the Internet for content, and it can gatekeeper, and the gatekeeper chooses content to counter. And then of course, you can see here that essentially this is how a misinformation campaign is created. So you select the content which you want to counter, you create the counter article for the selected content, you create the comments, you create fake users, you create fake images, you create a fake sound clip, which you can easily do. You can create a fake journalist profile, you can, you know, write the article, which is, of course, really easy. And then, of course, you can search Twitter for relevant accounts and tweets, you can find the user's bio. Then, of course, you can just share the articles to those accounts and you can, along with posts that look like user commentary. And then, of course, you can influence people's opinions and stuff. And recently there has actually been a change to Twitter because allegedly, Elon Musk has recently stated that in order to fix this problem, what he's trying to do is actually make sure now that in order to sign up to Twitter, you have to pay an annual fee. And whilst that is pretty crazy for a social media account, most of them you can do that for free. I think he's doing this because he realizes that me, I don't really see it that much, but the bots on Twitter are getting out of hand and this is the only way to prevent that. So until there's some kind of crazy solution, it seems that that is going to have to be possibly the new way. And this is kind of important because whilst now we could have to do this probably manually in the future, the persuasiveness of these AIs are going to get more and more incredible. Here's where we actually look at responsible AI, and this is where they show that there's identical generation of Thanos. So this is from a movie, and you can see that this is the original, which is, of course the movie. So these are screenshots taken from the actual movie. And then this is, of course, midjourney version. You can see that this is a screenshot taken from the actual film, but this is essentially the prompt that the user has uses. The user has put Thanos Infinity War screenshot from a movie. And you can see that's what has been outputted, which is it's not looking the best in terms of mid journey because if, you know, I don't know, Disney or whatever were to file a lawsuit saying that, you know, you're unlawfully using our content, then they'd have a pretty solid case. So it's going to be kind of interesting to see how this kind of thing is done because this kind of issue is. It's really, really tricky because on one side you have the fact that AI systems don't necessarily store your images, they generate them and the same with text. And then on the other side you have like, if it doesn't store it, how comes it's just simply outputting this copyrighted image. So it's going to be a question that will like, like I said before, you know, there's going to be some kind of case which sets a precedent and I'm wondering if this is going to continue into the future. But I think that future updates, they're going to have more testing. We also had the Foundation Model transparency total scores of open source versus closed developers in 2023. And you can see that we have the newly introduced Foundation Model Transparency Index shows that AI developers lack transparency, especially regarding the disclosure of training data and methodologies. This lack of openness hinders efforts to further understand the robustness of safety AI systems, which is pretty interesting. Then of course, what we have here is extreme AI risks are quite difficult to analyze. You can see that over the past year a substantial debate has emerged among AI scholars and practitioners regarding the focus on immediate model risks like algorithmic discrimination versus potential long term existential threats. And it has become challenging to distinguish which claims are scientifically founded and should inform policymaking. This difficulty is compounded by the tangible nature of already present short term risks in contrast with the theoretical nature of, of existential threats. And they're basically saying that, look, you might be fixing the issues that are there now, but there are some issues that are going to be there in the future that we need to prepare for now. And here you can see that there is a difference in terms of those who believe that, you know, I guess what we have here is that companies that develop foundation models will be responsible for the mitigation of all associated risk rather than the organizations using these systems. And I think that is something that you do need to agree with. It's up to people who create the systems, not to people who use the systems, because you know, people are going to use the systems for pretty anything. And it's up to the people who develop that product to make sure it's actually safe before they put it out into the wild. Something that was interesting and another trend that was actually pretty fascinating was that the number of reported AI incidents is of course increasing. So it says that according to the AI Incident Database, which tracks incidents related to the misuse of AI, 123 incidents were reported in 2023, a 32.3 increase from 2023. And since 2013's AI incidents have grown over by 20 fold. What you can see here that this is actually something that's rather fascinating. Of course they also talk about. A notable example includes AI generated sexually explicit deepfakes of Taylor Swift that were widely shared online. And this trend, unfortunately I do think that this is going to get worse because something that I do do in my spare time is I go through all AI tools that have been released. I release tutorials on them because I really want to understand how I can improve my workflow. I don't want to be someone that is late to a new AI tool that's blowing up. I want to be very. And whilst looking at some of these tools, some of these image generation tools that are coming out, some of them that are open source, closed source, a lot of them actually allow you to generate content like this, like adult content and it's really, really easy as well. So I do think that it is something that's going to be a really, really big problem because in the future there will need to be some kind of regulations because there are just so many tools now that you can really use to just generate harmful content and there's no really regulations on, on it around online and such. So I do think that unfortunately this is going to be increasing in a trend which is definitely worrying because you know, I'm sure we will have a friend or something like that and as long as your social media profile is open, they can use those images, train an AI system really quickly and generate a consistent character, generating explicit images for whatever purpose that may be. And that's not something that you want. So it will be interesting to see how policymakers actually do combat that because the trend shows that, you know, the graph is going up and that's not good in this scenario. Now something that they also showed as well was that ChatGPT is politically biased. So you can see here that it says researchers find a significant bias in ChatGPT towards Democrats in the United States and towards the Labor Party in the uk. And this finding raises concerns about the tool's potential to influence users political views, particularly in a year marked by global elections. Now why Is this bad? Why is it bad if ChatGPT leans one way or another? Well, when you have 100 million people using something on a day to day basis, I think something needs to be as unbiased and possible. And if you have a system that is weighted to one side or the other, the problem is, is that you know, subtly you are influencing that user base's political opinions. And it's one of the ones that people have spoke about this before that, you know, how is it that these private companies are basically getting to decide the political views of certain individuals? And you might think that it doesn't matter because it's just ChatGPT, people just using it to write ESS. But sometimes these platforms and systems haven't managed to write about certain things. They refuse talking about certain issues. And if they are left leaning or right leaning, I definitely think that it's important to understand why and understand to have these systems be a bit more balanced in their opinions. This is where they actually talk about the economy. This is something that I'm going to be focusing on. And they said the integration of AI into the economy raises many compelling questions. Some predict that AI will drive productivity improvements, but the extent of its impact remains uncertain. A major concern is the potential for massive labor displacement, something I talk about on the second channel. But to what degree? Jobs will be automated versus augmented by AI and companies already utilizing AI in various ways across industries. But some regions of the world are witnessing greater investment inflows into the transformative technology. And it appears to be in specific AI fields like natural language processing and data management. So this is of course something that you want to pay attention to. So one of the craziest things about generative AI is that investment in those companies are just increasing. We can see that the trend moving forward is going to be, you know, continually going upwards. And while some of you might think that this is currently a bubble, I don't think it is a bubble at the moment because the applications for generative AI do exist and it seems that we're in the early phase. This doesn't seem like something that is in its.com phase just yet. Not that it couldn't be, but it does seem that right now there is a huge level of investment because you can see the things are okay, just kind of declining and then boom. After the release of ChatGPT, I think the global consciousness shifted to where people completely understand what this technology is able to do. Now you can see AI private investment surged last year and funding for generative AI surge nearly octupling from 2022 to reach 25.2 billion. And major players in the generative AI space, including OpenAI, anthropic hugging face and inflection, reported raising substantial funding rounds. And this was something that was true, you know, something that I covered quite a lot on this channel. And it's something that is not surprising, to say the least, because the transformative nature of this technology means that it can generate outsized returns. We can also see that the United States, in terms of the trend, is leading the private investment area. We know that the United States definitely has a very, very good, healthy system in terms of investments, and it's got a healthy economic area. So we know that that is no surprise. But other countries are definitely lagging behind in terms of their investments into AI. Now, this is one statistic that I'm not sure is particularly true. I'm not questioning the data at all, but I've seen a lot of different reports that do say different. They state that, you know, AI job postings by geographic area. It says AI related job positions made up 2% of all job postings in America, a figure that decreased to 1.6% in 2023. And this decline in AI job postings is attributed to fewer postings from leading AI firms and a reduced proportion of tech roles within these companies. I'm guessing that maybe overall.