The real challenge of building agentic AI

All right. Thank you, everyone, for joining. Good morning and welcome to the Governing the Next Generation of AI virtual event in our Humans in the AI Loop virtual event series. Today, we'll explore AI agents, governance, and humans in the loop, looking at both the technical challenges of building autonomous systems and policy questions around keeping them safe, fair, and aligned with human values. We're really lucky to have 3 amazing speakers, Dr. Mark Esposito, MicroOne's Chief Economist and Harvard Professor, who's going to be moderating today's discussion. And then we have MicroOne experts, Dr. Amir Ascharod, Stanford researcher, MicroOne expert, sharing insights on AI alignment and the technical side of agentic AI, and Dr. José Ramón Enríquez, MicroOne expert and researcher at the Stanford Institute for Human-Centered AI. So over the next 45 minutes, we'll dive into what agentic AI is, its limits, and how the human-in-the-AI loop strategies guide the development. And we'll finish with some audience Q&A. So feel free to use the Q&A chatbot on the bottom, and we'll try and answer as many questions as possible. Similarly, if you'd like to enable captions for this, you can enable the captions icon on the bottom right-hand corner as well. So I'll pass it off to Mark, and thank you all for coming.

Thank you so much, Sydney. Very, very happy to keep on animating this conversation. I'm very, very lucky today. I have, as I mentioned in some of the social media posting, two, what I could say, the superstars, right? Researchers that are operating at the frontier of their respective fields. Both Amir and Jose, they're currently at Stanford in two very different capacity, and therefore we kind of building our conversation on today on, on the, the difference in what they see every single day in, in their dedicated area of research. We'll try to go a little bit more in details into kind of a driving narrative today that we'll try to eventually interlink and bridge. One is on, on agents, which is a big conversation these days. I think there's a lot of a bit of a buzz around the agentic AI, but somehow we miss on the conversation of where does it really come from, from a perspective of the construction of the algorithm that eventually interacts with agents. And so Amir will help us a lot with the kind of work he does at Stanford, as he has a very technical background. So he kind of looks at the inside of how the technology eventually performs. And Jose is working also Stanford, but His background in government and policy naturally opens him up to the big conversation on governance, which is becoming important around the world. It's been important in the US, is important across the continent, and I would say is going to become even more important as the technology continues to become more sophisticated. So this is great that we are dedicating our time with Amir and Jose. Besides the work they do at Stanford, what the three of us have in common is that we at one point in time joined MicroOne. We were becoming part of the ecosystem. And so they kind of feed into this series of humans in the AI loop, which is kind of showcasing the amazing amount of talent that we have in our community. So I'm here and I'm going to start really just for you to share maybe in a short few sentences what is that you're currently doing in your everyday job, just to give a sense about what keeps us busy. The experts at MicroOne, they do have MicroOne as a common platform, but they come from entirely different backgrounds sometimes. It's kind of nice to hear the story behind the character. So, Amir, I'm going to start from you, if that's okay. You are a computer scientist, but tell us more about what keeps us busy on a daily basis. What is your kind of bread and butter these days?

Yeah, sure. So first, I'm very happy, very glad to be here with you guys. I'm very excited about our discussion. And yeah, as a computer scientist/researcher/expert in Microbond, I would say, well, part of my daily work is just research. I would say AI research, and it's more like the core AI kind of research where we develop new algorithms and data structures and methods to basically enhance the performance of the current systems and LLMs and stuff that we have. And there is also another applied side to my work, which also is very close to what we do with MicroVon, which is basically along the lines of designing, let's say, new and efficient data pipelines and data structures and training mechanisms for these state-of-the-art models to improve and to become better than what they are right now.

Great. Great. Amir, look, it's great. Look, I'm going to try to hold my curiosity in asking you more questions, making sure that we are kind of creating a dialogue that we will share with our attendees right now. José, I go over to you. Fascinated about all your background. You and I share a bit more in common because we come from, well, in my case, economics and policy. You did your PhD at Harvard in government, right? And so there's a lot of what I think you had done during your doctoral studies that I understand. But then you decided to join Stanford, and you are in one of the state-of-the-art centers on the study on AI. So tell us more about what keeps us busy these days.

Yeah, of course. First of all, thank you so much, Mark, Sydney, for organizing. Everyone at MicroOne, it's just a great opportunity to be here and be talking about this very relevant challenges, also very timely. So I guess I would first want to start by saying the way I see the challenges that come with AI is 3 big challenges. One is big challenges of AI to governance. So this will be being governed by AI. You can think of this as all the threats to misinformation, fake news, chatbots being super persuasive. Then the second big challenge is how we govern AI, and this will be all the conversation around how we should regulate not only the companies themselves and the economic competition among those companies, but also the products that they're producing and how to, in a way, steer the conversation towards what is more useful to society. And the third one, where I mostly spend my time, is basically how to use this technology to improve governance. So this technology not only brings a lot of disruption for sure, And actually yesterday, a really cool paper was released by the Digital Economy Lab where I am currently working that indicates that there is a big disruption in entry-level jobs in the US and mostly for, for coders. So how can we take this technology and actually make something good out of it? And most of my time I'm working on using this technology to improve, for example, digital literacy and also increase the interactions and the quality of interactions between citizens and governments. So I'm currently working on a deliberation platform that does that.

That's great. So look, if that's okay to say, I'm going to stay with you as we go a bit deeper into the governance and then I'm going to switch back to Amir. How much progress do you think that we have made in this conversation on governance? And I ask you, you know, historically sometimes governance, especially governance of technology, tends to be a bit on the slower pace. By the nature how governance works, we can't expect it to go as fast as the development of technology, right? So it's kind of a tacit given that governance will eventually follow through. It won't necessarily anticipate. Some countries are talking about anticipatory governance. They're talking about agility. But one of the questions, like, can we even think about governance that can catch up with this, considering that the process of defining consensus in a governance structure requires way more temporal allocation, right, than what would happen, for example, in an AI lab, where within a range of few months, an entirely different design of a product will be launched. So what's your sense of pace and speed related to governance? Are we— do we stand a chance to have So like a commensurate fitting governance for the kind of technologies that we see develop these days.

Yeah, you're totally right. So it is a big contrast between how technology moves and how the institutions that by construction have to be a little bit more conservative and slower. So think about this big bureaucracy building that that are trying to adjust to the recent times. But at the same time, it is really important that we start addressing what seems to be the challenges. So I would say in general, we have two models where one is the European Union, that it's definitely on one side of the spectrum, and then the US version. The European Union basically is trying to forecast whatever the challenges are going to be regarding the use of technology. Obviously, that is a really, really hard task because you are asking, even if you ask the experts, experts do not have a consensus of how this technology would look like or what are the applications of this technology. And then on the other side, you have the United States that is trying to be as dynamic as possible. A lot of times this is fueled by all this dynamism in Palo Alto and kind of the vibe of doing more and faster, as fast as you can. So I think we haven't reached a consensus of what's the best model because one important thing is actually realizing what are the applications of these this model. So for example, MicroOne is an amazing example of what this technology can do and how we can create larger labor markets for people that I'm sure we couldn't imagine 2 years ago even, or even before the pandemic. So that's right, because remote work was something that we didn't even consider. So what I'm trying to say is there are a lot of challenges and there is a lot of need of this technical experts to be involved in the decisions of governments. First, to avoid overregulating, but also to enhance the dynamism that might happen with, with this technology companies because we don't want to basically destroy this innovation that might happen just because we think about certain threats to it. Of course, on the other side, and this is part of my work, of course, is trying to deal with potential threats to governance itself. And it is, for example, not a coincidence that OpenAI didn't release their Sora model that produces videos before the US election. So they are clearly like this. Companies have to take in consideration not only their maximization of revenues, but also they have to acknowledge that they're going to be a key part of society, and whatever the technology decisions that they take is going to impact a lot of people, not only in surrounding areas as not just the Bay Area, but everyone around the world.

That's right. So fascinating. I will keep on working on this kind of line of thinking with you, Jose. And my next iteration with you is this social ramification of technology, right? That sometimes we don't see when we're focusing entirely on the product. But Amir, when we were having our conversation yesterday, as we were preparing for this, so you mentioned several times humans in the loop from a perspective of designing algorithms that are taking into account people or experts or the training of the data. So if you're looking at the work you do, so you're basically working fundamentally with the technology itself, do you have a sense of governance on how do you govern an algorithm, right? Is it by the data that you're training? Is it on the quality of the dataset that you're choosing? Is it by the kind of people that you are basically involving in the production? 'Cause I think where you can help us, Amir. So when we talk about governance, I think a conversation like the one Jose is understanding is something that we can visualize. But what is governance of an algorithm from a perspective of a scientist like yourself, right? If you can help us with that conversation, I'd love for us to learn from you.

Yeah, sure. So I would say, I think it's, It's very insightful. It's very helpful to take a step back and kind of try to take a look under the hood and see what exactly we're dealing with here. Because I think it's a very common, I would say, misunderstanding that often people think about these AI agents and LLMs as things that they actually are not. So for example, I think this was like a very kind of like funny thing for people to do in the early days of ChatGPT where they would go and ask ChatGPT, like this, there, there's this famous example of how many Rs are there in the word strawberry, and then it would always, usually always get the answer wrong, and it would say like 2, but the answer is 3, or maybe they would ask, okay, like multiply these 2 numbers and tell me what the answer is, and then the answer would be wrong. But then when we try to see what these systems are and what's happening under the hood, we see that, okay, we have to kind of like understand what they're good for, what they're useful for, and have reasonable expectations from them. And like, if you look at an LLM as a next token predictor, as a next token generator, that's at the end of the day, that's all it is. It's like it looks at the history, gives you one next prediction, like from like creating, like, generating a sentence. What's going to be the next word that's going to come out of my mouth? That's what's happening. So, uh, when we talk about governing them, shaping them, trying to make them behave in a certain way, uh, trying to, uh, make them safe or, like, follow some sort of, like, regulations or guidelines, everything like that, we have to keep what they actually are, and we have to, like, know them for what they are. So With that in mind, then you can see how people can somehow govern them and shape the behavior out of them that they desire. So for example, the example that I mentioned earlier, like multiplying two numbers by each other, which the LLMs are not inherently good at, or like the how many Rs in strawberry, well, that results in people actually introducing tools to LLMs. So now still they cannot do that. They can never do that because they're not designed to do that. But now what they can do is that they They know which tool to call when they want to multiply two numbers or when they want to, I don't know, count some letters in a word or something. They can now call a tool that runs Python code. They can now call a calculator. They can now call another tool to do certain stuff. So now if you want to control them, I think we have to have the same kind of vision. If we want to make them act in some certain, like follow some certain safety or I don't know, some other kind of guideline. Well, we can do that with prompting. We can always do that with data. Data is usually very, very, very important. It's probably the most important component in designing these systems and managing them and controlling them and shaping their behavior. I would sometimes, that's not exactly true, but I would sometimes say that it's all about data. Good, high-quality data is basically all you need to get what you want out of the LM, out of these systems. But again, there is always some, you know, something that could be out of your control. You can have human in the loop to kind of like look at the outcomes of the system. I think that's, and this might be a point of discussion later today, but like if you have a physical AI system, like a self-driving car. Well, you always have the human in the loop. You don't really trust it without the human, but well, that's right. You know that you don't trust it. With the LLMs, it can hallucinate, it can give you very confident, very wrong answers. You can always do better with data. There's always room for improvement. But as far as I understand right now, there is always also no guarantee. And that's probably something that we always have to, uh, be to some healthy degree worried about.

Maybe. I mean, look, I think what you're helping us to understand is that it doesn't take away or reduce the value of a model to recognize the limitation of it. And I think it goes a lot in what Jose is mentioning. Governance is not designed to stifle innovation. It's designed to help the innovation to actually improve itself for a larger number of users or citizens, no matter what's the unit of measure. So somehow, So interesting. You, you start from two very different angles in terms of what keeps you busy, but you come across some comparable observations on the fact that this narrative about humans being completely displaced by technology is not in what we find every single day. So this is amazing. And I'll go back to you, Amir, to make sure that you help us understand when the prompting now gets into an automated pilot because it's becoming a gentile, right? What we're going to do. So I'm going to, I'm going to go to you, in a short while. So to kind of build on this narrative, if building on what Amir said, if the level of limitation now are getting unnoticed, and so we basically have a model that does not easily correspond to what we want it to be, there could be some social implication. And one of them might be against population that might be less represented, right, by the data that Amir was talking about. So one of the work you do is on representation. And I think it's important that we create a context. Not necessarily the data that we collect and we train the model represent the voices of everybody, right? There's, I think, a strong dependency on English being the major language of training. But what if the data does not take into account underprivileged backgrounds that have never been able to directly capture, right, their kind of sense of reality? So how do you govern that part of the conversation about representation. I know that it's not just a question, but it's actually part of the work you're doing at Stanford. Can you help us with that?

Yeah, of course. This is a very great question and really important topic because it's— I would say that this predates the actual conversations of AI. And one good example to illustrate this is a science paper that came up, I believe in 2018, where they are trying to predict healthcare spending in the US. And they show in the paper that if you take the wrong outcome, if you take actually spending, then you're gonna introduce a lot of biases in whatever prediction method you have. No matter how good the predictor, the prediction method is, you're going to have some biases because the data is reflecting some biases in itself. They actually show that there's some biases in terms of ethnicity in the US. What I'm saying is data is definitely the most important part, as Amir was saying, but also that we need to be very aware of what that data is actually representing. And part of my work is working on this new deliberation platform, deliberation.io. And the idea is now that we have the technology as AI chatbots, we can actually scale up and we can ask people about their preferences regarding public issues in their community. And we couldn't do this before because there, there weren't safe, accessible, anonymous channels for these people to interact. But also, now that we, we can have LLMs, they, the participants can actually engage in some meaningful conversations, not only with others so that they can learn other people's preferences, but also to reflect on their own opinions. So I've been currently working on an experiment that it's called Socratic Dialogue, where people are actually reflecting on their own thoughts regarding very controversial topics in the US, like gun control, abortion, use of voter ID. And the idea of this is to actually show that we are able to build systems that can include voices that are currently not represented in the debate, and that we can do this in a scalable, in, in very cheap ways. And we can do it not only in developed countries but also in the, in lower resource environments.

Great.

You know, also, you make me think about some of the things that you mentioned before about what kind of the striking of the Micro Wand community, right, is, uh, when I found out that some of, well, the interviews and the proctoring happens in multiple languages, right? And while it seems to be just, you know, a differentiation on the market by saying, yeah, there's more of, right? It's not just a differentiation in the market. It's really a representation of voices in the way that when you're training the model in different languages, you're equally fundamentally looking at the reasoning constructs that are coming from a specific conversation. I was having— yesterday I was teaching a class at night with a group online in Brazil. It was 45 of them. And I was kind of intrigued to say, well, do you use any Brazilian technology for your large language models? And of course, the answer is they use mostly US-based technology, right? And so again, the question will be over time on representation, not only in terms of the ethnicities or kind of what could be racial distribution in countries like the US, which is a multifactorial country, but also across the world in multiple regions. So I'm fascinated by the kind of work that experiment like the one you're running, right, are going to help us to basically improve the access. And you reminded me a little bit when you were talking about when we were using blockchain with voting rights, with the example of having access that was distributed, right? So I guess that's, that's a convergence of technology that we can definitely imagine will happen more and more.

Yeah. And let me comment on this. So there's there's this researcher, Daniel, uh, German at the University of Columbia. He has a really nice paper where he shows that in Sierra Leone, actually, teachers don't usually use Google to do their searches regarding content in for their classes, but they prefer some other AI-powered solutions because those solutions can be customized to whatever they're actually trying to the content that they're trying to consume. And this is because if people in Sierra Leone, they just Google stuff that is related to the curriculum in their schools, most of the findings are not going to be relevant to their context. So now that we're in this new era where AI is deployed in such a fast pace, we can think of the solutions to incorporate everyone, and hopefully all these tech companies will find it also in their interest, in their economic interest to also represent these people.

Yeah, great. Thanks so much, José, for this. Amir, back to our conversation. Look, it's fascinating to even have a chance to have a conversation with you on this. You mentioned a lot of things can go wrong. There is a need of internal governance. It's kind of like the vigilance required, right? Now, this implies that you are kind of looking at the conversation between a user that is prompting the machine that is generating an output. This is something we do understand to some degree. But what happens when you're automating the prompting to an agent? In other words, to what degree do you trust in kind of the planning phase, right? We say this is now agentic, that the agent will do a job that you can trust? And do you see agents to be entirely autonomous to the point that they will, they will basically manage entire functions on their own? Or do you see some oversight from humans also in agents? So how does it look like from your perspective?

Yeah, that's, that's a very good question. I would say a very important one and one that probably no one knows the exact answer to, but I'm gonna tell you my take on it. So I think before trying to actually answer the question, it's a good idea to kind of like look at the trend and like how the answer to this question might have changed over the past 2, 3 years. So like the first days that let's say ChatGPT was out and everyone was excited about it. Well, at that time, we didn't even have this kind of like agentic behavior. We would just like try to like chat with the LLM. Suppose you were a programmer, you wanted to do some Python coding. You would ask, okay, here is my code. It has this issue or I want to add this feature to it. And then it would help you with some new code. You would copy and paste that into your IDE and then you would run the code and then it would either work or you would see, okay, this is not working. It would go back to the LLM and then ask it to kind of like fix the error. So by things becoming agentic, one of the things that we mean is that, well, the LLM can now do this by itself, or you can actually like ask it, okay, build me this, I don't know, an Android app, an iOS app that does this thing, or like build me a website. And then, I mean, there are so many tools that people use. One of my favorites is Claude Code. And if you, if you use it or not. You just like run it on your computer and then just ask it, do this for me, like build this application for me, or like create this website for me. And then you can see it kind of like comes up with a list of action items that it has to do. And then like kind of like checks off each of the items that's done. And at the end, it tells you, okay, here's what I've done. I tried these methods, some of them didn't work. So I tried some alternative methods. Now it's working here as a functional app or website or whatever for you. And then when you look at it, sometimes it's actually exactly what you want it to be designed, sometimes it's not. So I would say in something like coding, like programming, we have made a lot of progress since even 2 years ago, since even like 6 months ago. I mean, it's crazy how fast things are getting better. Are we at a point where we can trust these autonomous systems with everything? Like let them write the prompts, do everything, do the planning, execute whatever they've planned? I think you can even, I mean, I think the answer at this point is no. If you want to just choose a binary answer, it's either a yes or no. I think we're in the right path, like we're on the right track and it's going to get better and better. But are we even close to the point where people would trust these systems with their lives? I think not. Like, are people Again, but physical AI, are people trusting their lives with autonomous driving, like self-driving cars or things like that? But these systems, I think the answer is clearly no. And I think it's like, but again, there were simpler tasks that were not doable 2 years ago, and they are doable right now. And we fully trust the system. But those tasks, because right now we, think that those certain tasks are easy enough for the system to actually fully implement and do and plan everything. So I think this is like a trajectory and we're kind of like making progress and every day there are more tasks being unlocked in the sense that we can fully trust the system with them. But to reach a point where we can like absolutely trust them with anything, I'm, I'm, I'm not sure if we're even close to that point.

That's right. I mean, let me, uh, do a follow-up with you, Amir. Um, I think your answer is so rich, right? You make us think a lot. And look, nobody can tell us what the future will look like, right? When people say, what will AI look like in 2 years' time? That's a question you can't ask. But my question is, do you see sometimes a limitation to the point that you're thinking this will not be a vertical that keeps on growing up? Eventually, we might reach a level where the technology slows down or consolidates or plateaus. Like, do you ever have not necessarily evidence, but a hunch that we might get to the point where this is not an infinite ladder, but we're going to get to a point where maybe the technology, the energy, the memory, the microchip, whatever it is, right, will get to a point of, of stall. And that is interesting because I guess many people, they think that the number in the sequence of the model they're coming is like the iPhone, right? Starting from iPhone and now we're in iPhone 16. And if you ask people, likely they'll expect 17. 18, 19. Will there be a moment where the technology can no longer improve in the same way it has been improving and might need to improve in a different way? What's your take on this? And again, not from a point of knowing, because who knows, but from a point of view or your kind of informed hunch, if I can place it like that.

Yeah, sure. So I think your iPhone example is actually a very, very good example for this. I'm not sure if, I mean, like 10 years ago, like when iPhone 5 was released, I don't know, maybe that's because I was a kid at the time, but I feel like everyone was like, they were seeing a lot more improvement compared to the previous version that what we were seeing right now. Like right now iPhone 16 is pretty much the same as iPhone 13, I would argue. So I think people have also realized that about GPT-5 compared to GPT-4 or something. You don't, like we're not really seeing that huge of a leap, like that big jump from GPT-4 to GPT-5, the way that we probably saw that from 2 to 3 or 3 to 4. And I think that that's very natural with technologies. When they're introduced, there are some easy steps to be taken at the first stages. And then when they become very good at certain things, well, there is not really easy to make more progress. So in that case, I think that the iPhone example is a very good example. And I think we should be expecting a breakthrough in the technology in the same way as we are expecting a very fundamental and fascinating breakthrough or update in the next versions of iPhone. Well, maybe we're not at the iPhone 16 level of the AI system, We were still around maybe 8, 10, or X, whatever it was called. Yeah. Trajectory is similar. And about what you said about like, is there like some threshold, some like ceiling that we cannot really move past, or if there is like an infinite ladder that we can always climb. I think this kind of, I mean, this reminds me of this philosophical question that I sometimes think about that like when you compare these systems with the human brain, Again, looking at the core of these systems that are next token generators, I think the question is, are we as humans and like, are our brains just very, very good, excellent next token generators? Or is there something fundamentally different that's happening in our heads that is not happening with these systems? I don't have any answer for that. I think I mean, I want to believe that we are something beyond that. I think that's something that all the humans want to think. But I would say that as long as we have proper data for the everyday tasks or the tasks that we want to solve with these models, we'll probably make progress. But again, I don't know if there will be a day when we realize that, okay, this structure, this next token generation, this mechanism is not enough to solve everything and we need another entirely different mechanism. If that is the case, if that turns out to be true, then I would say this is not an infinite ladder. It's not just about data, just about better training algorithms. It's probably about a different structure that we have not found yet. But if that's not the case, Maybe it is, then it's that latter.

That's right. I mean, fascinated because I think you're helping us to understand once the variables that are defining to some degree the access to the reasoning that has been populated in the models for any reason changes because we find there's a limitation on the reasoning, on the tokens, whatever, then that's a new formula that has to be eventually put together. So there could be an acceleration. You would make me think that we have in everyday language the sentence 0 to 100, but we don't have 100 to 200, right? And so the idea is, of course, the acceleration happens from 0 to 100 and you can measure it. But from 100 to 200 is an entirely different sort of like sense of priority that needs to be put in place. It's a different degree of acceleration. So that is great. And look, I think I see your hand is up and what we can do maybe if you have any reaction on this, but also looking at the Q&A, it happens that we have one question for me kind of and one question for you, Jose. So since you're on the mic, Maybe I can tell you what the question you can integrate into your answer. So es enar, who's basically asking, what do you think are kind of the trends in the governance that are going to generate access, not in democratic countries, but in no democratic countries? So one thing is to actually expect access in democratic country because of the sense of consensus. But what if you're in authoritarian countries, right? Would you still consider that the sector will nonetheless be able to generate governance regardless of the fact that you don't have representational voices like you would have in a democratic exercise. So that's the question for me, you know. But yeah, I'd love for you to either comment on what Amir was saying before and help us maybe with this question.

Yeah, I have two quick reactions to what Amir was saying with a little bit of risk of sounding too pessimistic. So the first thing is what he was saying about the agents and what I've been thinking now for some time is that maybe if we solve, even if we solve the alignment problem, which is that agents actually behave as humans want to or they represent the true preferences, then we should be thinking about a second layer, which is is some sort of agentic ecosystem where there are going to be a lot of agents interacting with themselves. And even if we are able to govern the good behaving agents, there are obviously going to be some bad actors that can interfere in the ecosystem and can mess everything up. So imagine having agents that are very comparative, but then you have some sort of bad actors that are trying to take advantage that everyone is very cooperative. So then you will have a second generation where you will kind of reproduce those bad agents that are getting more of the gains from that bad behavior. Then obviously, well, the second reaction is I feel like, and in part this is kind of fueled by Silicon Valley, that we're always thinking about the frontier of the technology. We're always thinking about, oh, what's the new iPhone? I love the analogy. I think that is totally right. But my reaction will be like, probably for a lot of applications still in developing countries, we will will need very, very simple models where just a very small amount of data, probably not super high quality, can help us get a lot of gains from that technology. And I was mentioning the case of Sierra Leone because that would be a lightweight model that people can access through a website or their phones or even SMS. As that was the case for that research. And just super quick on the question on non-democratic countries, there is what I see. I was a few months ago in DC at the World Bank. There was a conference with a lot of ministries of innovation there, and what I see is that all these ministries have a lot of FOMO on not participating in the conversation on being part of the AI deployment in their own countries. And I know that there's a lot of appetite for using these AI tools, not only just as some economic tools to improve their processes, but also to actually hear some voices from the population. So, for example, I've heard some demands from the Saudi Arabia government that they want to increase or they want to improve Arabic language LLMs because they see some opportunity there not only to to improve whatever the process is, like coding, but also actually reaching and hearing about the population. Then you can make a second argument whether that is going to actually help them, like non-democratic regimes, stay longer or not, whether giving a voice to people would be a good thing for democracy or not. But I will leave it there.

Yeah, that's great. I think, I think the gist of this says that in theory, the representation of power from a political perspective does not conflict with the access to an LLM if it's going to be a form of distribution of information, knowledge, right? Opportunity. So, but of course, the flip side of the story, this could be used to expand the propaganda around a specific argument, right? Or to obscure, right? A flow of information. This is something that is like everything, right? The kind of the side of the story is never monolithic, right? There's always different dimensions. There was a question, Amir, for you that I think is interesting. I'm going to pull it back from the Q&A. It was from Alexander, who basically, you know, asked, is there a consensus across the computer scientists and the researcher on what is how much data is needed, right, for you to feel that the model is now stable or reliable. In other words, when you feel, yeah, this model now can eventually go to this next phase, what defines that the data is stable or reliable that you can trust? What determines that? Is it a quantitative parameter? Is it a qualitative one? Is it on testing on multiple fronts? So what determines a data that you trust, Amir? To be eventually deployed?

Yeah, that's a good question. Before answering that, I also kind of want to comment on what you were just discussing about how these models can be used to promote certain views and certain values, because it's also very relevant to the data discussion. I just want to acknowledge the fact that This is very important because it's absolutely doable very easily with data, and you can just inject any form of bias that you want into huge models, even with a very small set of data. Some of these biases, they just exist because they existed in the dataset that the model was trained on without even us controlling that. These LLMs these days, this is one of the famous examples where they have some form of bias, for example, that a doctor is usually a he rather than a she. So if you ask it to talk about the doctor and then the nurse, there's a very good chance that it refers to the doctor as a he and then to the nurse as a she. So that's a bias. And you can introduce these— well, one of the very active research areas is to try to mitigate these biases, try to get rid of them. But if it's as a government, as a controlling entity, you want to introduce some sort of bias. You can very easily do that. So that's also something that I think people have to be careful about. As you can probably see that kind of bias, and if you guys have worked with DeepSeq, the famous model that was released, I guess, last year and got a lot of hype around it, it has a very clear and significant bias towards the government of China for some reason. I mean, kind of like we all know that. So yeah, that's one thing. And about how much data we need actually to get things like guarantees on safety and certain behavioral properties. I do not think that there is a very nice and beautiful consensus around this. I think, I mean, with simpler things like make the model behave in a certain way, have good instruction following or have some sort of bias or have some sort of perspective about one particular subject, well, in those cases, we do know that. It's in many cases a very small amount of data is enough to fine-tune the model towards doing that, having that certain feature or property or behavior. But when it comes to more high-level and like more, let's say, fundamental issues like safety and I would say us being confident enough in the performance of the model in a variety of aspects. I think it's usually done via just testing and testing and testing it again and then see if it's been enough or if we need to do this more and more and further and further until we get to that point.

So what is fascinating, Mary, is really the nature of the calibration that is required, right, before we get there, right? And look, if that's okay with you, Amir and Jose, as we are at time, there's one final question that happens to be perfect for the two of you, right? So maybe we can have your insight on both. And that will be Bruno's that is asking a fascinating question. I'll start from you, Amir. In terms of the agents, right, many of these agents, they are becoming more anthropomorphic. So what is your take? Should you think that eventually there is a way from the technology to identify the limits of that to the point that we want to create clear rules of engagement or what are the boundaries of an anthropomorphic model that looks at humans but somehow is still an algorithm? And then flipping this over to your side, Jose, right? Should we govern digital intimacy? Should we basically have, so like a way of PG-13, PG-18, right? We have done it with movies, right? So we kind of came up to say as a society, we want to protect minors from watching content, right? So should we govern digital intimacy or are we, are we stepping onto what will be free arbitry, right? Of where people have to decide whatever they want. So what's your take on this? And that will be our last conversation. So Amir. From your side?

So yeah, I think this is also, there are like different schools of thoughts around this. I personally am kind of against anthropomorphism for these models because I always try to keep in mind that at the end of the day, we have a text-to-speech generator under the hood. We can trick that text-to-speech generator. We can and design it to behave in a certain way using data, things like that. But we can see that, I mean, as these models are trained on more and more data and larger and larger datasets, well, they get better at showing the behavior that a regular human does. So I think, again, this goes probably to the same philosophical question of, What is it that is actually making us humans? So I would say that because the thing is that when people try to think about the models as humans, like very knowledgeable humans, I think there is a certain level of trust that comes with this because people start to think that, okay, these models now have this capability to think the way that the humans do. And I think that is not the case. I mean, At least one clue that would support my argument is that we as humans often need much less data to learn something or to generalize something that we already knew. Like we often read a few pages and then we learn a new mathematical concept, but you might see the LLM needing a lot, a lot, a lot more information to reach that point. And it does reach that point. So that's the thing. And when we just see that final outcome, that it's performing just as good as like a very well knowledgeable ex, like human expert, then we start to trust it and think that, okay, this is a human, it's actually better than a human. I think we should always be careful about where these things are good at, what they can do. And then also always acknowledge what they cannot do and what they are not designed for doing. And yeah, that's, I don't know if that answered the question. Spanish.

It does.

It does.

I mean, I love it because somehow you're telling us there is a bit of a responsibility in looking at what are the inherent features of a model rather than abstract things that do not really exist. And just for pleasing imagination of having the sort of like, you know, more human-looking like agent, which we'll have a conversation a couple of weeks with an expert in robotics, which is the very same narrative about humanoids, right? That we're trying to push for so long.

I just remembered an example about how these things could be very different from humans. In many research works, research papers, you often see that. I mean, it's very easy to, for example, design a backdoor for these models. You can just train the models so that if they see some very specific, some gibberish random word, then they would just enter a very different mode and behave in a very crazy manner. And well, you as the user of that model, well, you never know if such a backdoor exists or not, things like that. So I think it's very important to know that at the end of the day, these are algorithms and they can be designed by their designers to behave in certain ways. And at least we as the users of these models, well, we're not their designers. We don't know how exactly they were designed by by whoever designed them. We have to be careful that, well, there might be some backdoor, some certain behavior injected into these models that we are not aware of. So I think that's very related to what you guys were discussing on the governance side.

On the governance side, yes.

We have to be just careful about this. We have to know what dangers could exist. Well, they might exist. They might not exist. That's very important for us to keep that in mind.

Yeah.

You know, one of the things that comes to my mind, Amir, and I go to you, Jose, right? In a question in the chat from Ismael, right, that kind of say, who is the regulator of these ethics? And if I can kind of provide you an insight on that, Ismael, right? If you're looking at how José and Amir are coming from two very different fields of the AI ecosystem, but there's so much convergence in the understanding of the boundaries that defines that there is a much wider distribution of consensus on the limitation of the technology when you're working with the technology. We tend to lose sight of it. When you get away from the technology and you're going into this imaginary world that sometimes is becoming science fiction. So I'm very happy to kind of feel that we have this multi-stakeholder sense of epicenter around what technology is and what is not. And so to help us, maybe we finish it with this. To what degree should we eventually govern this kind of events that are kind of breaching the sense of normal? And the example digital intimacy is one of it.

Yeah, I think that the question about persuasiveness is also a really good one. Just last month in July, some researchers from MIT released a paper that shows that LLMs, conversational AI, can be really persuasive during election times. And they went and did an experiment with actual voters trying to convince them to switch sides. For the Canadian elections and US elections. So this is a thread that is there. And then you can think of, oh, what if we regulate and we impose that every conversation with an LLM should start from with the LLM saying, I'm an LLM. Then, well, that sounds like an easy fix, but then I'm sure that there will be be so many unintended consequences as people impersonating now LLMs just to extract information. For example, imagine financial information. So I think it's a very tricky question and that we should discuss. And that's what is the most important thing is like that all sides and all perspectives are being in discussion. Same with digital intimacy and restricting access. I'm sure that we can think of different models that are customized for children and that have really good guardrails. I think that in terms of technology, we're still developing some solutions as both the tech sector and the academia to make sure that the actual guardrails are in place. But I think that, again, the most important thing is that we're having this conversation.

That's right. You know, guys, we did 55 minutes of just pure power conversation. I have to say, amazing. The only regret is that these are not conversations we do over coffee in person. I still like the conversation that we can have when people meet. But imagine how many people will reach out throughout our kind of ecosystem by the inspiring on question that they have and And sometimes they become spirals, right? They don't easily find an answer. And the purpose of our series is not to give answer, but to instigate even better questions, right? And this is exactly what we have done with the conversation. So Amir, thank you so much for the work that you do at Stanford in being a researcher and a computer scientist. And it's a pleasure to have you as one of my colleagues at MicroOne. José, again, amazing that You, after your PhD, decided to continue to work in governance and decided to go to Stanford, continue to work in this important question and join a MicroOne entity that is by itself a company working on the ground, right? So this is, I think, quite great that we have people working in academia and in the industry without feeling that there is a trade-off between the two. Actually, they're very complementary. This has been great. So guys, thank you for your time. For any of you that were with us, we'll kind of generate a bit of social media clips that are kind of highlighting the conversation I had with these two amazing colleagues of mine. Next week, we're going to have two more superstars on our series coming from the MicroOne family, but again, two professors working in different institutions. We're helping to kind of tackle the conversation on AI and education. And not too long from now, we're going to have two more superstars. We like superstars. And two ladies who are working on redefining biases in AI, a computational scientist that is bridging between Paris and Los Angeles, and a robotics expert who is running a robotic lab in one of the biggest labs on the planet. And so It's just exciting what is ahead of us. Thank you so much for joining. And from Mark, Amir, and Josep, all the best now. Bye-bye.

Thank you.

Thank you. Bye, guys. Bye-bye.