Stanford's AI Index Report 2024

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to or are curious how AI related tech is changing the world, this is the show for you. Thank you to our partners at Fly IO, the home of changelog.com Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents so you can launch your app near your users. Learn more@fly IO.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder and CEO at Prediction Guard and I am joined today to talk about a very interesting report that we've, we've talked about on the show today, the AI index report from 2024 from the Human Centered Stanford University Human Centered AI Center. I'm joined by Nestor Masle, who is the Research manager at the Stanford Institute for Human Centered AI. Welcome Nestor.

I'm super excited to be here.

It's great to have you back. As I mentioned, we've talked on the show before about the AI Index Report in previous years, but for those that haven't had that background or listened to those episodes, could you just give a little bit of a sound bite about what the AI Index Report is?

Sure. So the AI Index is an annual report that currently is in its seventh edition that aims to tell the story of really what's going on in AI from a diversity of perspectives. We look at trends in technical performance. So what can the technology do now that it wasn't necessarily able to do five years ago? We look at trends in the economy. How are businesses integrating this tool, how much are investors investing in this tool. We look at trends in policymaking, how are policymakers responding to what's going on in the space? And we don't just look at those things. We study research and development, ethics, public opinion, diversity. And I think really we aim to be kind of a one stop shop, kind of an encyclopedia of what's happened with AI in the last year that policymakers, business leaders, or really anybody that needs to know and understand what's going on in the space can turn to when they have questions about artificial intelligence.

Interesting. Yeah. And could you tell me a little bit about the Institute for Human Centered AI? Kind of why the Institute is undertaking this and how it kind of fits into maybe the wider set of things that the Institute does?

Yeah, of course. I mean, funnily enough, it's actually very recently we celebrated the fifth anniversary of High.

So congratulations.

Founded in 2019 and we're kind of five years into this wonderful journey and I think the Institute really exists to try to advance AI research, education and policy in a way that will fundamentally improve the human condition. The creators of the Institute, I think, came together really because they felt that AI could be an incredibly groundbreaking technology. It could be something that could really elevate the potential of humans. But in order to do that, we'd have to think very carefully about how we actually want to develop some of these tools. And that's what we spend our time thinking about at the Institute. Realistically, when it comes to AI, the ways in which this tool is going to develop is not only going to depend on computer scientists and how hard they're working, but. But it's going to depend as well on policymakers, on business leaders and on the regular public. So those individuals as well need to be given a tool that allows them to identify and understand how the space is evolving and developing. And that's what we aim to do at the Index. The AI Index is the report that we feel can give these individuals the capacity to make the decisions that they need to be making about this technology.

Yeah, and the Index has been published for, like you say, a number of years. And of course, this. I'm imagining that this last year or maybe the last couple of years have been kind of interesting in that on this show, in our conversations, in conversations across the industry, generative AI has, has dominated a lot of those conversations, but doesn't necessarily mean that that is kind of the quote, unquote AI that is impacting humans in their everyday life.

Yeah, there's more to life than generative AI. There's more AIs.

Yeah, exactly. So how did you come at the report this kind of time around with the acknowledgment that of course this is transformative, what we're seeing in many ways. But also it's not, it's not the only, like, you know, when you talk about a report about AI, it's not necessarily just a report about large language models, I assume.

Yeah, I think that was important for us. I mean, we certainly added some new data points this year on generative AI because we felt it had kind of come to the surface and it was something that we needed to chat about. But in a lot of the chapters, I think for us, it's incredibly important to exactly, as you said, draw that distinction between foundation models, generative AI and non generative AI systems. And for example, in the research and development section, we have information not only on the number of foundation models that different countries are producing, but also how many notable machine learning models these countries produce. The kind of idea being that it is possible for a machine learning model to be notable and not necessarily be of the generative type. Similarly, when it comes to the economy section, we track total AI investment, not just generative AI investments. And we even included a completely new chapter this year on some of the ways in which AI is interfacing and engaging with science. And that touches on a lot of these kinds of developments that relate to some of the ways in which AI is used in non generative ways, but is still really moving us forward and leading to a lot of really exciting advancements.

Yeah, that's interesting. And to give a sense of how the index is put together, you know, I'm sure people have seen different surveys that are out there, different takes on, on AI and where it's at in terms of the approach of the, of the Institute. What is the kind of main research mechanism that goes into the report and like, how is it, how is it developed, what's involved, who's involved, that sort of thing?

Yeah, great question. It's kind of a two pronged effort in that we collect data ourselves for particular questions that we feel there isn't already good data for. So I think we try to be strategic and you know, we find that if there's someone else in the research community that is already collecting data that we find to be relevant and interesting, we don't try to necessarily duplicate their efforts. And as such we partner with a lot of data vendors like Accenture, GitHub, McKinsey, LinkedIn, the study portals, they all collect data that we find to be interesting and then we work to then included into the AI Index report. But again, for certain topics where we feel that there isn't enough data, for example, we felt that there wasn't a lot of good data on the number of AI policies and legislations that were being released on national levels, we endeavor to collect some of that data ourselves. And in terms of the research agenda, it's set by the AI Index steering committee. So we're very privileged to be advised by a diverse committee of AI thought leaders, people that were really are very influential in commenting on what kinds of things are going on in the AI space. People like Jack Clark, who's one of the co founders of Anthropic, people like Eric Brynjolfsson, who is arguably one of the world's leading economists on AI, and people like James Monica, who leads AI research efforts at Google. We work with these individuals to discuss and identify what kind of topics we want to track and we figure out as part of the process where each report should be going and what kinds of things we need to be chatting about. Awesome.

Yeah. So it's not just AI researchers that are or, you know, polling people that are, that are pouring into this. It's actual AI people from industry, academics, but also spread across other, other areas like economics.

Yeah, definitely. A lot of diverse perspectives and I think also perspectives from a lot of people that have been in AI for a long time because we see this with a lot of Technologies where AI announced itself in 2022, ChatGPT came out and then all of a sudden everybody and their mother became an expert on AI. I think some of the people that we really work with have been in the space for a very long time and have kind of seen it ebb and flow. So can especially offer a lot of very valuable perspective on where we are in this moment and kind of contextualize and situate that in a very nice way.

Yeah, Well, I do want to get into some of the specific points raised in the, in the report, but before I do that, I'd love to ask, especially with these sorts of things, because you've, you've been thinking about this deeply, looking at all the data and, and all that. Was there any one particular thing that stood out as kind of surprising or counterintuitive or just stood out generally in the kind of year over year progress of the report? Looking at this year's, anything that stood out as surprising to you in particular?

I think thinking about the cost of some of these frontier models. One of the things we did in this year's report is we partnered with Epoch AI, which is a great AI research institute, do some estimates on how much it costs to train some of these frontier AI systems. And I think we kind of all knew in the back of our minds that these systems were going to be very expensive. And we eventually crunched the numbers and you know, it's one thing to kind of anticipate that and you actually see the numbers that, you know, GPT4 is costing close to $80 million. Gemini is costing close to 190 million. And you kind of see this trend line which is just kind of going up and, you know, almost exponentially. So it really kind of puts into perspective how far we've come and I think it also poses a lot of interesting questions about the future. How much further can we go? I think a lot of these companies are betting very heavily that the scaling laws are going to hold, that they can continue pumping more and more data into these systems and that these systems are going to respond with improved capabilities and even new capabilities. And I think they're all making interesting bets and it's going to be very interesting to see if that's going to hold. And, you know, what the future may look like is something that I'm going to really be watching with a lot of open eyes and an interested perspective.

Yeah, yeah. It seems like it's getting to the point where, and this was something that was also highlighted that I saw in the report about industry dominating frontier AI research. And I guess that's not like that's connected in some ways to what you were just talking about because the. I'm thinking back to my own PhD and like the five of us in our grad student office. Right. It's no group like that that's going to go about kind of the training of one of these new models at this scale. Certainly there have been efforts, I guess from efforts like Bloom and others that have brought together researchers from around the world to work on models in almost like a collaborative way, like a CERN type effort or something like that, which requires a huge budget. But at least as far as I can tell from the data and what you're talking about is, you know, there's those budgets are increasing, these models are expensive both to create and to run at scale. And so you, I assume that that is connected then to that observation of industry kind of leading the front on the research side. Is that fair?

Yeah, I think pretty directly connected, yeah. I think it's, I think a lot of these companies know that feeding more data into the system leads to better performance, or at least it has so far. And I think these companies are betting that that trend is going to continue. So they're pouring money in and kind of hoping that we're going to see improved capabilities. And I think part of the reason these kinds of estimates were so eye popping to me is because even coming into the report, I was kind of aware that as you said, no grad student could build an LLM, train it on their laptop in a way that would compete with some of the language models that these big industrial players release. But we're kind of getting into territory soon where you'd even have to wonder of the kind of industrial giants who can really afford to be building these things if we're going to get into territories where they're costing a billion, if not more. I mean, 100 million is a lot of money, but it's a lot less money than a billion dollars. Right. And the scale of who can kind of get involved at that level is a lot different and varies fairly substantially. And that has a lot of important implications for how AI research is being done and what kind of topics people focus on. I mean, I think industry actors do a lot of really valuable research and they do contribute a lot of interesting insights. But I mean, at some point you need to pay off the investors and you need to make a product that is commercializable and that could kind of shift incentives in ways that again, coming back to this mission element that we talked about with Hive might not necessarily align with building AI systems that do the most to really further humanity. Foreign.

Hey friends, this episode of Practical AI is brought to you by our new friends over at Plum. Plum is a low code AI pipeline builder that helps you to build complex AI pipelines super fast. You can easily create AI pipelines using their node based editor iterate and deploy faster and more reliably build than coding by hand without sacrificing control. Deployment is easy. Pipelines are live API endpoints eliminate the need for constant code redeployment and debugging. By deploying complex AI pipelines as API endpoints, team collaboration is easy too. Plum's declarative node based editor enables you to build quickly while empowering non technical roles to iterate on what you've done without breaking it. You can build advanced AI features, get structured output every time, transform data and leverage validated JSON schema to create reliable, high quality structured output. So Plum is built for builders early stage product teams are using Plum to go from idea to validation in record time. To get started go to useplum.com that's plum with a B as in plumber 3 quest access today that's U-S-E P L U M B.com again useplumb.com.

Well Nestor, some of the things that you mentioned in the report are related to generative AI. Some of them are not. One of the things that I think is interesting about this particular index is talking about some of the things beyond the technology that might impact not only the direction of AI development but actually does potentially impact practitioners like like some of those listening to this podcast. And one of those things being the kind of sharp increase in regulation in particular in the United States as related to AI. I'm wondering from your perspective, especially as this index has has looked at things over a number of years, kind of every year we're talking on this show about like oh when, when and how are the regulations coming down? What from your perspective can we discern from from this kind of sharp increase in regulation in the, in the US and in the maybe the practicality of how that will actually influence developers.

Yeah. So I would say a couple of things. I think first, big takeaway for me is that we're probably going to see a lot more action on the state level, and we already are, than on the federal level. I mean, if you look at the total number of proposed AI related bills in the United States on the federal level and the state level, there has been a pretty massive increase in the last seven, eight years in both counts. But there are substantially more state level bills that have been put into law. I think looking at my notes here, close to 40 on the state level in 2023 compared to just 1 on the federal level in 2023. And I think this again is not surprising. I think it takes usually a little bit longer for there to be consensus on the federal level, but it might mean that we can usefully look to states as kind of being one of the first barometers of what's going on with the regulation and what we might see down the pipe in the federal level. I think a second point is that when you actually look at the regulatory agencies, they're all passing more AI related legislation and AI related regulation. And, but the regulation itself is coming from a diversity of bodies. So it's not just like the Copyright Office is kind of, you know, hogging all the attention and passing 30 AI related regulations. The Executive Office of the President is getting in on the fund, the Department of Commerce, the Department of Health and Human Security. There are so many of these regulatory agencies that are thinking very deeply about this tool. And I think this reflects the fact that AI is general purpose. It doesn't surprise me. But I think it's also a note that, you know, if you think to yourself that, oh, I'm, I'm not a computer scientist, therefore I probably don't need to worry about how this is going to affect my lives. I would kind of, you know, urge you to maybe walk back that assumption because regulators across different spaces are starting to become much more actively involved in passing AI related regulation. And I think a lot of us are going to be affected much sooner than, than we would like to believe. And I mean, coming back to the state level point as well, I think you're seeing now a lot of very kind of hot and contentious debates, especially in California around SB 1047. I think you're going to see that in similar states where, you know, I think four or five years ago, the regulation that we did see was quite kind of expansive and let's say a lot lower stakes. It was very often what we would call more expansive regulation that was saying let's explore how we could use AI or let's kind of empower AI researchers. Whereas now the regulation is starting to become a lot more restrictive. It's putting in rules about how these technologies can develop and how people can use these technologies. And obviously when you go with that approach, there tends to be, I think, much more debate about getting it right. And that leads to a lot of kind of fiery opinions on either sides of any particular regulation.

I guess there was this sharp increase in that in this past year that you, that you saw. There's probably some people out there, as you mentioned, there's a variety of mixed opinions about this of whether that's actually kind of drives more market share for these closed providers who have already built up some of that market and maybe have some of that influence and open access or open source community that maybe is doesn't have the ability to or the willingness to put in a lot of effort into this compliance side. Was there any thought amongst the group around how not only this would impact so regulations would come down, they would be maybe enforced potentially in impactful ways, but how that would kind of shape the development of this technology moving forward?

Yeah, I mean, I think that's an open question. We don't do a lot of, I would say predictive work in the index. I think we try to mostly look at what we know to be true so far. But I mean certainly, I think at least if you look in California with the debate going on with the bill that I had previously mentioned, SB 1047, I think, yeah, it's this kind of question of open versus of closed source, that thing. That particular bill wants to put certain requirements on models that are above a certain compute threshold. And some of these requirements are so stringent that conceivably a lot of open source developers wouldn't necessarily be able to kind of meet all of them. And whether or not these regulations are backed for commercial reasons, as you kind of mentioned, some of these industrial players could conceivably have commercial incentives to support regulation that maybe, you know, compromises their open source players. But I mean, I do think as well there's a lot of people in the kind of AI safety community that very ideologically believe that AI poses a serious existential risk and that we should really be hyper cautious about how we scale. And I don't think it's necessarily the position of the Index to kind of say we support one position versus another. But I do think what I would say is that it's now kind of. We're now getting into a moment where, again, policymakers are starting to think about this, business leaders are starting to think about this, and we really need to think carefully about what we do and what we put into law. Because, you know, once it's law, it's going to shape incentives and it's going to shape how the community develops. And I think there are compelling arguments made by people on both sides, but it's difficult to know what the future holds. And I think it's hard to know whether this is the right move or that is the right move. But I think we always have to try to hold ourselves to some kind of standard of intellectual accountability and kind of acknowledge that, you know, we can't always get it right, but we try to. We need to try to have as thoughtful and as well balanced of a perspective as possible.

Yeah, I think that's a really good perspective. You mentioned, when you were answering that about kind of some of these regulations being tied to kind of how AI will develop moving forward, and in particular as tied to the size and scale of models being developed. I was really interested that there is a portion of the report that is just titled will Models Run out of Data? Which I think is really interesting. Um, we talked a little bit about these models becoming more and more expensive. But of course, there's an element of this which also is the scaling of the data required, to train, particularly these generative models, but other models as well, maybe computer vision models or other models. And that is something that, you know, I. I definitely get the question every once in a while from. From people that have heard something like, oh, you know, these models have already sucked in all of the Internet of data. Right. Like, what's. What's left to train on? And then you've got another perspective that I hear sometimes, which is, well, now we're just going to fill up the Internet with generated data, which cycles back into training data sets. So, yeah, I'd love for you to bring any perspectives that you have on this question of will models run out of data? I think it's a question that's coming up frequently for people.

Yeah, it's hard to know. I mean, that's the problem. You want to kind of go on these podcasts and give kind of kind of clear, concise answers, but in a lot of cases, it's quite nuanced. I think there's reasons to be optimistic there's reasons to be pessimistic. I think when you kind of look at optimistic reasons for why we might not run out of data or why data might not be the bottleneck that we anticipate it to be. I think there are some papers that seem to suggest that synthetic data could meaningfully aid in training AI systems. I think as well, if you think about it, these language models are substantially less efficient than the human brain. They see millions times more text than any human would in their entire lives and in some cases can perform better than humans at certain tasks. But it's clear that there is kind of an architecture of the mind that being our own architecture that gets the job done with a lot less energy. And plausibly there's going to be more research being done on algorithmic efficiency that can maybe make it easier for models to perform platform at higher levels with less data. And I think as well, you know, we're always creating more data. We're kind of in an era in which more and more data is being kind of manifested and put out into the world. Some of that of course, is going to be generative and created by AI systems. But there's also new models like Meta segment anything which make it a lot easier to get segmentation masks. So AI could be used to also take more data from the world in a way that could help these systems. On the other hand, it does seem that if you kind of just look at the amount of tech stock that we have now, it seems like we might potentially run out of that relatively soon. So I think we cited estimates from Epoch, they've since updated those. They predict that kind of in four years we might potentially run out in terms of if we're going to kind of continue scaling models in the way that we're scaling their models. There's also other papers that are not necessarily as bullish on the potential of synthetic data kind of plugging this hole and filling in the gap. I guess for me the kind of really interesting thing is, you know, I'm really going to be curious to, to see what it's going to be like when GPT5 comes out. Because I mean all these models have, I think been now they're good, but they're kind of incrementally improving their, their capabilities at various tasks. But it still seems to me like they struggle on some tasks like planning, they struggle on some tasks like reasoning, they're still somewhat prone to hallucination and there are still kind of limits to what they can do. And I guess I would wonder, is scaling a transformer going to be sufficient to resolve some of those problems? Or do we potentially need a new architecture, a new way of building AI systems that could resolve some of those difficulties and resolve some of those challenges? And it's hard to know, but there's a variety of perspectives here. I'd be curious to get your perspective on it as well as someone that kind of thinks about this a lot too.

Yeah, yeah, I appreciate that. From my perspective, there's similar to yours. I think you can look at it like a mixed bag. I think the thing that I would maybe draw in here is that at least right now it seems like a promising route forward is not necessarily relying on these models to, you know, foundation models to have all the inbuilt knowledge that that is needed to for example, answer every question or have every fact or you know, be able to process any type of input necessarily. I think though certain things, like I've seen this in the progression of like function calling. For example, there was sort of a generation of these models where people figured out that oh, we can like use these to generate calls to APIs or to functions, but none of those prompts were in these sort of fine tuning data sets. And so like they kind of worked for that, but kind of not. And now like the ones that are coming out now that have that pre built into the fine tuning data sets, they do that much easier and can extend to a whole variety of function calling. So I think that these sorts of not like specific facts or specific knowledge about particular APIs or these sorts of things, but the ability to figure out what those, what those more general kind of building blocks of robust AI systems are and, and building the prompt data sets around those. My view is that there's going to be a lot more curation of that type of thing versus like just hoping that increasing data set size will solve a lot of those problems.

Yeah, it's a question of efficiency as well. Right. Because you know, for a lot of businesses it's also not super efficient to be kind of running these kind of very large, very computationally taxing and expensive models. So when you, it's, it's not only a question of maximizing performance, but what's that kind of sweet spot where you get good enough performance and the efficiency is kind of where it needs to be.

Well, speaking of performance and maybe utility of these models, we were talking a little bit about that, but maybe more generally, one of the things that's highlighted in the report is standardized evaluations for large language Models or maybe generative models. We also talked a little bit about this in relation to. There was a ML OPS community survey about AI quality and evaluation. And I think the results of that showed people still are having some issues figuring out the right evaluation, standardized evaluations, figuring out roi. Yeah. Any anything to highlight from your perspective in terms of this evaluation front and where the state of that is now or how that's changed?

Yeah, I mean, I think there is two things that I would say so the first is kind of when it comes more to general capabilities. I think one of the things that I've seen at the index is I don't necessarily know if the benchmarks that we have now, which I think are mostly academic, are sufficient for dealing with the realities of AI that we now face, which are industrial. And what I mean by this is that a lot of these benchmarks, and for the listeners that might be unfamiliar, the way I conceptualize is when, you know, a model developer launches a new model, like anthropic recently released Claude 3.5, they'll test it on a variety of benchmarks. These are tests of what AI systems can do, like a test of grade 8 math problems. And they'll say, you know, our model gets a 96% on this benchmark better than Gemini or these other models. Therefore we have the smartest and most capable model. Now I think the reason the community does this is because 10 years ago AI was strictly an academic problem. It was something that university researchers were thinking about and I think they wanted to know on an intellectual level, how could AI think? And these kind of benchmarks were useful. That's how you got things like imagenet. And it was useful then to kind of see how much better we could actually get at these systems. But businesses, you know, they're not solving grade 8 math problems or doing competition level math, as is tested on some of these benchmarks. They're using these AIs for wildly different purposes and they behave wildly differently depending on the context. I saw this firsthand. I was editing the report using different AI tools. I would use GPT4 and I would use Claude. And for whatever reason, I really preferred Claude over GPT4. I thought that it was a much better copy editor. GPT4 was sometimes suggesting to me words that I thought were very kind of gauche. And I would tell it like, don't use this word, and then it would use it again in two prompts. So I just kind of got a bit frustrated. But the point I'm making more broadly here is that we have evaluations for these models that test them on things that businesses aren't really doing. And I think there is an opportunity there for someone to kind of really identify how, how could we use these models from a productivity level and which ones are perhaps the best ones on that front. The second point that I would make about benchmarks and standardization, we talked about this in the responsible AI section that was co written by Anka Royal, who's one of the PhD collaborators at Stanford. With us was kind of looking at how these foundation model developers benchmark their models, juxtaposing general capabilities benchmarks with responsible AI benchmarks. And what you see when it comes to general capabilities is that a lot of these developers, they're all benchmarking on mmlu, which is a benchmark of general language understanding. They're all benchmarking on Codex, which is a benchmark of coding capabilities. They're all benchmarking on GSM8K, a benchmark of math. When it comes to responsible AI benchmarks, there really is no consensus. Some of them are testing on truthful qa, others are testing on real toxicity prompts. Others still are testing on bold. But really across the map, there is no consistency. And it's not clear to us if this lack of consistency reflects either a genuine belief that these developers have that, okay, certain responsible AI benchmarks are better than others, or if these developers are merely doing this as a means of kind of juicing their model performance performance, and they just choose the particular benchmarks that best suit them. But I mean, it is consequential because we're kind of now in an era in which AI is being widely used, and when something is being widely deployed, you need some kind of standardized evaluations or standardized comparisons of how these different things function. And it seems, at least when it comes to the responsible AI world, we're not really kind of at a stage where that's occurring.

Yeah, and one of the things I see highlighted under that section is also extreme AI risks are difficult to analyze. Could you dig into that a little bit? Because there's one element of this which is practically in the industry setting, as you were just mentioning, evaluating performance is maybe complicated and not standardized, and relies on these benchmarks, which might not be applicable in all cases. But then the other side, there's these risks, liabilities, harms that are, that are promoted across sort of academic and industry settings, which inform kind of the general discussion about AI and safety. But also for a practitioner, you know, maybe they're thinking, like, what do I focus on here? What what's an actual. Is, is this risk that these people are talking about a reality that I need to be considering, or is that sort of just something that should inform long term development or something I should consider today? Yeah, talk a little bit about that and like what you were seeing around the maybe safety and risk side of responsible AI.

Yeah, I think what we kind of meant here is that at least when you kind of talk about these AI risks, there seems to be kind of two categories. The more let's say short term risks, which are things that kind of bad things that AI can do now that we should be paying attention to, like its potential to be biased, its potential potential to be unfair or to violate privacy and more kind of long term existential risks, which I guess, you know, pardon my French, refer to the possibility of it killing us all at some point. Now, I think the kind of challenge here is that with the short term risks we already see this manifesting in the present and with the long term risks, I mean, some of the arguments these people make are theoretically plausible. Like you could imagine this happening, but it's hard to know how plausible it is. There are some people that feel very confident that AI at some point is going to become smarter than us, self improve and want to take control. There are others that are not as convinced and think that's just kind of lunacy. But it's kind of challenging with these long term risks because they're so theoretical and so in the future. And I mean, even if you could show that these models are sometimes deceptive, that they have the potential of, you know, being Machiavellian, it's still hard to kind of get to these longer term arguments because they depend a lot on these kinds of theoretical argumentative claims that are difficult to actually ground in reality in the future. And I think you see this kind of manifesting very tangibly with this bill SB 1047, where it seems to me that a lot of the people that are in favor of this bill, which again would impose some fairly stringent safety requirements on models above a certain compute threshold, they seem to be of the belief that AI could pose really serious safety risks. And there's a desire that, oh, if we scale up these models even further, we want to be able to kind of shut them down and ensure that they kind of don't pose a threat. Now if in fact it is the case that these models do have these safety risks, then that seems like a plausible argument. But if they don't, then you might in the process pass a law that could really cripple the ability of open source developers to create models. And the open source development community is very important for the startup ecosystem and the innovation ecosystem. So that's kind of what we mean when we talk about these risks are difficult to analyze. You know, there's a theoretical argument which is somewhat plausible, but how likely is that theoretical argument? That's, that's tough to know.

Yeah, along with the sort of risk discussion and evaluation discussion, I guess some of what's focused on in the report is also the perception of AI within maybe various demographics, but across various segments. And if I understand right, there's a sort of general pessimism about AI in terms of like its impact on people's jobs. But if I also understand it right, AI does seem to be making a positive impact in terms of people's quality of work and efficiency of work. Could you help us parse through some of those things a little bit in terms of maybe the more kind of general public impact and perception of this technology?

Yeah, I think there's two things that you're speaking to here. I think first is the fact that if you look at public opinion data, at least in a lot of countries like the United States, Canada, France, people are very bearish about AI. When kind of asked, do you feel that products and services using AI have more benefits than drawbacks in those three countries and also in a lot of other Western countries, the overwhelming majority of respondents seem to disagree. They don't think that AI is more beneficial than disadvantageous. Yet in a lot of other, let's say, developing countries like Indonesia, Thailand and Mexico, there is much more bullishness. People are very excited and very hopeful. We don't necessarily know why that's the case. That's obviously something that's very important to kind of unpack and continue thinking about as this technology develops and as it rolls out even further. Now, you mentioned this kind of point about economic advantage. I think what we're referencing here is that there's now a lot of new studies coming out in really different industrial sectors. Whether it's kind of looking at call centers, whether it's looking at legal work, whether it is looking at computer science work that shows that workers that use AI tend to be more productive than those that don't. At the minimum, they're able to do tasks faster, and at the maximum, they're not only able to do tasks faster, but they're able to submit work of higher quality. So what kind of explains the disconnect? Well, I mean, I still think we're kind of in the early days of AI integration, these kinds of studies that have looked at AI's positive impact were quite micro scale. I don't necessarily think businesses are kind of using this technology en masse yet. And I think second is the fact that it's hard to necessarily know where we might go even if AI is productive. Right. Because you know, if you're working 40 hours a week and all of a sudden you need to be working 20 hours a week because of AI, you know, you could either use that 20 hours to do perhaps different projects that will lead to more money and more value for your employer, or maybe your employer decides, hey, we just don't need you for those 20 hours and kind of scales back what they asks of you. So I think the kind of jury is still out on whether the integration of AI technologies is going to lead to widespread automation or augmentation. And I think when you kind of look at narratives of fear or if you at least try to understand why different people in different countries are as frightened by this technology as they are, I think a lot of it comes down to this, this element of just kind of uncertainty about what it's going to do to their, their jobs and their livelihoods.

Yeah, thanks for that, that perspective. As we kind of near, near the end of our conversation here, I'm wondering after seeing the results of the index in kind of this round and also working on it for some time, what is you look forward kind of to this next season of AI development and adoption and integration. As you were just talking about, what do you see as exciting or positive in terms of how things are moving forward and what's on your mind kind of looking towards the next. You mentioned not necessarily doing projections, which is fair, but what are you curious about kind of in terms of how, how things will develop going into this next season?

Yeah, I'd probably say three big things. I think the first one is, is scaling and a hold. So we talked about this already, but I think a lot of these companies are making these bets that they'll feed more data into these systems, they're going to get a lot better and I don't dispute that they're not going to improve. I think that the improvements are going to continue. Continue. I do wonder by how much and if in fact the improvements aren't as great as we anticipate them to be, what might that mean for the economics of AI? I think number two, looking at how are businesses actually integrating this technology? If you look at literature on a lot of other Technological transformations. A lot of economists argue that it typically takes decades from the launch of a technology to the point at which it actually registers positive productivity impacts, largely because very often you don't have infrastructure that's necessary to leverage a technology when it kind of appears out of the box. And I could imagine a similar thing with AI. Now, I don't think it's going to take decades for AI to kind of have its productivity impact being widely felt, but it will be curious to me to see if businesses start thinking a little bit more critically about how they want to use this tool and how they want to integrate this tool. And I think, third, kind of keeping an eye on what happens in the domain of policy you talked about, like, what am I, you know, what is something that I'm kind of encouraged by or kind of watching out for? I think it's. Yeah, what is the policymaking response going to be? And I'm encouraged because I know that I think looking back at the example of social media, you know, I think it took us close to a decade from when these tools were launched to when we started really kind of thinking about them on a political level in terms of kind of regulating them and ensuring that we built them with the right kind of incentives and in the right kinds of ways. And if you kind of point to 2022 as this moment where AI was kind of officially launched, I mean, we had pretty landmark legislation in 2023 with the EU AI act as well as Biden's executive order. So it's encouraging to me that policymakers are thinking about these things. I guess an open question for me is going to be where does the tone of policymaking go? And what comparative priorities do policymakers in different parts of the world have when it comes to launching these AI tools and models?

Awesome. Well, look forward to hearing maybe some of those, at least some data that's indicative of some of those trends in next year's index.

Yeah, you'll have to, you'll, you'll have to play some clips of some, some things that I said about the future here next year, and we can kind of revisit how accurate or inaccurate may have been.

How well did we do? Well, regardless, I definitely recommend people to. Of course, we'll link the index report in the show notes. So I encourage people to take a look at that. They've made it really easy to navigate. You can go by chapter and dig into particular sections and look at the key takeaways. So, yeah, definitely check it out. There's a amazing work that's been put together. So thank you, Nestor, for putting putting in your work to that and for all that the Institute is doing to help keep us informed. Appreciate it very much.

No, thank you guys for having me so much. It was a great conversation and hoping we can do this again next year. Take care.

Sounds great. Yeah. Bye.

All right, that is Practical AI for this week. Subscribe now if you haven't already. Head to PracticalAI FM for all the ways and join our free Slack team where you can hang out with Daniel, Chris and the entire Changelog community. Sign up today at PracticalAI FM. Community thanks again to our partners at Florida Fly IO, to our beat freaking Residents Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.